R Notebook

Setup

In case the tidyverse is not installed on your computer already (you haven’t used it in R before) you need to install it before loading the library. This you need to do only once as long as you use the same computer or don’t need to update a loaded package. The library() function needs to be used whenever you start from a clean R state.

install.packages("tidyverse")

library(tidyverse)

Ditto for the tidytext package:

install.packages("tidytext")

library(tidytext)

Data import

We read the data with read_csv, not with the base package read.csv, to create a tibble.

text_df <- read_csv("survey-v1.csv")

The columns are named “Respondent” and Q1 to Q9

Later we’ll have to do some data cleaning, but lets’ do some analysis first to give us a feeling for what that means.

Text analysis for Q1

To get us started, we look first at one question only: the answers to Q1. We do so by creating a data frame (df) just for Q1. This is of course not how to do this in general, and we’ll later look at performing the analysis for all question answers contained in one data frame with a tidy structure.

Word frequencies

Select all Q1 answers:

text_q1 <- select(text_df, Respondent, Q1)

(Note. Minimally, to make the analysis more general, we would use names here for variables that are question neutral: text rather than text_q1, etc; this way, copy and modify would become easier.)

Get all the words from the answers into a df along with the respondent’s ID. The result is still a tibble:

words_q1 <- text_q1 %>%
  unnest_tokens(word, Q1)

Let’s look at the most frequent words:

words_q1 %>%
  count(word, sort = TRUE) %>%
  top_n(10)

Selecting by n

Clearly there are the typical stop words that we don’t want to have in counts. Let’s remove them. There’s a data frame that has stop words specified for us, as part of tidytext:

data(stop_words)

We can View(stop_words) to see them, and add our own ones, of course. Here are the first 10 stop words:

head(stop_words)

Note that it is important that stop_words has a column ‘word’, as words_q1 has. To remove them from the q1 answers, do this:

words_q1 <- anti_join(words_q1, stop_words)

This removes all rows in words_q1 that are matched by a stop word.

Now the frequencies are more informative (top ten only are shown):

words_q1 %>%
  count(word, sort = TRUE) %>%
  top_n(10)

Selecting by n

A bit of sentiment analysis

Tidytext comes with three lexicons that contains words for sentiments, overall more than 27,000 words:

head(sentiments)

Let’s use the nrc lexicon and look for joyfull words in Q1. First, get the joyful words into a df:

nrc_joy <- get_sentiments("nrc") %>% 
  filter(sentiment == "joy")

Then look for them in answers to Q1:

words_q1 %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)

Joining, by = "word"

So there, a bit of joy! You can perform additional (sentiment) analysis following the examples in this book–see in in particular the chapter on twitter analysis that is similar in data structure to a survey–but before that it is a good idea to change our data structure so that all the survey answers are in a tidy format.

Tidying the survey data

While it is sensible to analyse the answers separately for each question, it is less than elegant to do this based on a dataframe that has just one question in it as we did so far. It would mean to copy the analysis 8 times and replace all

To get the data into tidy format, we need ‘lengthen’ the original data. We gather the question responses all into one column instead of leaving them distributed over 9 columns. For this we need a new column questions. The gather function is part of the ‘tidy’ package, which we loaded when we loaded the tidyverse packages above.

text_df <- text_df %>%
  gather('Q1':'Q9', key="question", value = "words")

We know have a table with 207 rows in three columns: Respondent ID, Question number, and words (the response).

head(text_df)

Text mining on the tidy data set

Let’s get to the words:

words <- text_df %>%
  unnest_tokens(word, words)

and remove stopwords:

words <- anti_join(words, stop_words)
head(words)

We may also want to remove numbers and special characters. First we need a tibble to store these kind of stop symbols under:

stop_symbols <- tibble(word = c("1", "2", "3", "4", "5", "6", "7", "8", "9", "0", "#", "(", ")", "."))

Since the survey data is in one-word-per-row format, we can remove stop symbols with an anti_join (from dplyr):

words <- words %>%
  anti_join(stop_symbols)

The data are now cleaner, but not shown here.

This now also is the point were we also could do some stemming. For this we need this library:

library(SnowballC)

Which gives us a function ‘wordstem’:

words_stemmed <- mutate(words, word = wordStem(word))
head(words_stemmed, n=20L)

But we don’t want to use the stemmed words just now because it would intefere with the sentiment analysis. Stemming is useful, but only once we no longer need “real” words, such as for topic modelling and the likes.

Word frequencies

Now we can calculate word frequencies for each person. First, we group by person and count how many times each person used each word. Then we use left_join() to add a column of the total number of words used by each person. Finally, we calculate a frequency for each person and word. (If this is really meaningful across questions, as done here, is debatable but this is mainly a demonstration. See below for filtering and grouping.)

frequency <- words %>% 
  group_by(Respondent) %>% 
  count(word, sort = TRUE) %>% 
  left_join(words %>% 
              group_by(Respondent) %>% 
              summarise(total = n())) %>%
  mutate(freq = n/total)

Joining, by = "Respondent"

head(frequency)

Sentiment analysis

Sentiments across all questions

To repeat form the very first example, if we were interested in the joyfulness, we can get words of joy from for instance the nrc lexicon:

nrc_joy <- get_sentiments("nrc") %>% 
  filter(sentiment == "joy")

And then look for them in answers to from Q1 to Q9:

words %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE) %>%
  top_n(10)

Joining, by = "word"
Selecting by n

Sentiment anaysis on one question

Just for Q1:

words %>%
  filter(question == "Q1") %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)

Joining, by = "word"

Note that questions IDs are now a row value, so we need to filter (rows) rather than select (variables).

Sentiment on all questions

words %>%
  group_by(question) %>%
  inner_join(nrc_joy) %>%
  count(word, sort = FALSE)

Joining, by = "word"

We turned sorting on the count off here because we want to have the results grouped by question id. If Sorting was TRUE, the numeric count would be used first, then the question ID.

Sentiment analysis on a sub-set of all questions

Let’s pick Q1, 3 and 7:

words %>%
  filter(question == "Q1" | question == "Q3" | question == "Q7") %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)

Joining, by = "word"

Most common positive and negative sentiment words

Using the lexicon bing, we can find how words costribute to sentiments

bing_word_counts <- words %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

Joining, by = "word"

head(bing_word_counts)

Let’s graph this with ggplot2:

bing_word_counts %>%
  group_by(sentiment) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(y = "Contribution to sentiment",
       x = NULL) +
  coord_flip()

Selecting by n

The top_n(10) seems not to work but let’s worry about that later.

There’s much one one can do with sentiment analysis, obviously, see the tidytext book amongst others.

LS0tCnRpdGxlOiAiUiBOb3RlYm9vayIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKIyMgU2V0dXAKCkluIGNhc2UgdGhlIHRpZHl2ZXJzZSBpcyBub3QgaW5zdGFsbGVkIG9uIHlvdXIgY29tcHV0ZXIgYWxyZWFkeSAoeW91IGhhdmVuJ3QgdXNlZCBpdCBpbiBSIGJlZm9yZSkgeW91IG5lZWQgdG8gaW5zdGFsbCBpdCBiZWZvcmUgbG9hZGluZyB0aGUgbGlicmFyeS4gVGhpcyB5b3UgbmVlZCB0byBkbyBvbmx5IG9uY2UgYXMgbG9uZyBhcyB5b3UgdXNlIHRoZSBzYW1lIGNvbXB1dGVyIG9yIGRvbid0IG5lZWQgdG8gdXBkYXRlIGEgbG9hZGVkIHBhY2thZ2UuIFRoZSBgbGlicmFyeSgpYCBmdW5jdGlvbiBuZWVkcyB0byBiZSB1c2VkIHdoZW5ldmVyIHlvdSBzdGFydCBmcm9tIGEgY2xlYW4gUiBzdGF0ZS4gCgpgYGB7cn0KaW5zdGFsbC5wYWNrYWdlcygidGlkeXZlcnNlIikKYGBgCgpgYGB7cn0KbGlicmFyeSh0aWR5dmVyc2UpCmBgYAoKRGl0dG8gZm9yIHRoZSB0aWR5dGV4dCBwYWNrYWdlOgoKYGBge3J9Cmluc3RhbGwucGFja2FnZXMoInRpZHl0ZXh0IikKYGBgCgpgYGB7cn0KbGlicmFyeSh0aWR5dGV4dCkKYGBgCgoKIyMgRGF0YSBpbXBvcnQKCldlIHJlYWQgdGhlIGRhdGEgd2l0aCBgcmVhZF9jc3ZgLCBub3Qgd2l0aCB0aGUgYmFzZSBwYWNrYWdlIGByZWFkLmNzdmAsIHRvIGNyZWF0ZSBhIHRpYmJsZS4gCmBgYHtyIFJlYWRpbmcgZnJvbSBmaWxlLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQp0ZXh0X2RmIDwtIHJlYWRfY3N2KCJzdXJ2ZXktdjEuY3N2IikKYGBgCgpUaGUgY29sdW1ucyBhcmUgbmFtZWQgIlJlc3BvbmRlbnQiIGFuZCBRMSB0byBROQoKTGF0ZXIgd2UnbGwgaGF2ZSB0byBkbyBzb21lIGRhdGEgY2xlYW5pbmcsIGJ1dCBsZXRzJyBkbyBzb21lIGFuYWx5c2lzIGZpcnN0IHRvIGdpdmUgdXMgYSBmZWVsaW5nIGZvciB3aGF0IHRoYXQgbWVhbnMuIAoKIyMgVGV4dCBhbmFseXNpcyBmb3IgUTEgCgpUbyBnZXQgdXMgc3RhcnRlZCwgd2UgbG9vayBmaXJzdCBhdCBvbmUgcXVlc3Rpb24gb25seTogdGhlIGFuc3dlcnMgdG8gUTEuIFdlIGRvIHNvIGJ5IGNyZWF0aW5nIGEgZGF0YSBmcmFtZSAoZGYpIGp1c3QgZm9yIFExLiBUaGlzIGlzIG9mIGNvdXJzZSBub3QgaG93IHRvIGRvIHRoaXMgaW4gZ2VuZXJhbCwgYW5kIHdlJ2xsIGxhdGVyIGxvb2sgYXQgcGVyZm9ybWluZyB0aGUgYW5hbHlzaXMgZm9yIGFsbCBxdWVzdGlvbiBhbnN3ZXJzIGNvbnRhaW5lZCBpbiBvbmUgZGF0YSBmcmFtZSB3aXRoIGEgdGlkeSBzdHJ1Y3R1cmUuIAoKIyMjIFdvcmQgZnJlcXVlbmNpZXMKClNlbGVjdCBhbGwgUTEgYW5zd2VyczogCgpgYGB7cn0KdGV4dF9xMSA8LSBzZWxlY3QodGV4dF9kZiwgUmVzcG9uZGVudCwgUTEpCmBgYAoKKCpOb3RlLiBNaW5pbWFsbHksIHRvIG1ha2UgdGhlIGFuYWx5c2lzIG1vcmUgZ2VuZXJhbCwgd2Ugd291bGQgdXNlIG5hbWVzIGhlcmUgZm9yIHZhcmlhYmxlcyB0aGF0IGFyZSBxdWVzdGlvbiBuZXV0cmFsOiBgdGV4dGAgcmF0aGVyIHRoYW4gYHRleHRfcTFgLCBldGM7IHRoaXMgd2F5LCBjb3B5IGFuZCBtb2RpZnkgd291bGQgYmVjb21lIGVhc2llci4qKQoKR2V0IGFsbCB0aGUgd29yZHMgZnJvbSB0aGUgYW5zd2VycyBpbnRvIGEgZGYgYWxvbmcgd2l0aCB0aGUgcmVzcG9uZGVudCdzIElELiBUaGUgcmVzdWx0IGlzIHN0aWxsIGEgdGliYmxlOiAKCmBgYHtyfQp3b3Jkc19xMSA8LSB0ZXh0X3ExICU+JQogIHVubmVzdF90b2tlbnMod29yZCwgUTEpCmBgYAoKTGV0J3MgbG9vayBhdCB0aGUgbW9zdCBmcmVxdWVudCB3b3JkczoKCmBgYHtyfQp3b3Jkc19xMSAlPiUKICBjb3VudCh3b3JkLCBzb3J0ID0gVFJVRSkgJT4lCiAgdG9wX24oMTApCmBgYAoKQ2xlYXJseSB0aGVyZSBhcmUgdGhlIHR5cGljYWwgc3RvcCB3b3JkcyB0aGF0IHdlIGRvbid0IHdhbnQgdG8gaGF2ZSBpbiBjb3VudHMuIExldCdzIHJlbW92ZSB0aGVtLiBUaGVyZSdzIGEgZGF0YSBmcmFtZSB0aGF0IGhhcyBzdG9wIHdvcmRzIHNwZWNpZmllZCBmb3IgdXMsIGFzIHBhcnQgb2YgdGlkeXRleHQ6IAoKYGBge3J9CmRhdGEoc3RvcF93b3JkcykKYGBgCldlIGNhbiBgVmlldyhzdG9wX3dvcmRzKWAgdG8gc2VlIHRoZW0sIGFuZCBhZGQgb3VyIG93biBvbmVzLCBvZiBjb3Vyc2UuIEhlcmUgYXJlIHRoZSBmaXJzdCAxMCBzdG9wIHdvcmRzOgoKYGBge3J9CmhlYWQoc3RvcF93b3JkcykKYGBgCgoKTm90ZSB0aGF0IGl0IGlzIGltcG9ydGFudCB0aGF0IHN0b3Bfd29yZHMgaGFzIGEgY29sdW1uICd3b3JkJywgYXMgd29yZHNfcTEgaGFzLiAgVG8gcmVtb3ZlIHRoZW0gZnJvbSB0aGUgcTEgYW5zd2VycywgZG8gdGhpczogCgpgYGB7ciBtZXNzYWdlPUZBTFNFfQp3b3Jkc19xMSA8LSBhbnRpX2pvaW4od29yZHNfcTEsIHN0b3Bfd29yZHMpCmBgYApUaGlzIHJlbW92ZXMgYWxsIHJvd3MgaW4gd29yZHNfcTEgdGhhdCBhcmUgbWF0Y2hlZCBieSBhIHN0b3Agd29yZC4KCk5vdyB0aGUgZnJlcXVlbmNpZXMgYXJlIG1vcmUgaW5mb3JtYXRpdmUgKHRvcCB0ZW4gb25seSBhcmUgc2hvd24pOgoKYGBge3J9CndvcmRzX3ExICU+JQogIGNvdW50KHdvcmQsIHNvcnQgPSBUUlVFKSAlPiUKICB0b3BfbigxMCkKYGBgCgojIyMgQSBiaXQgb2Ygc2VudGltZW50IGFuYWx5c2lzCgpUaWR5dGV4dCBjb21lcyB3aXRoIHRocmVlIGxleGljb25zIHRoYXQgY29udGFpbnMgd29yZHMgZm9yIHNlbnRpbWVudHMsIG92ZXJhbGwgbW9yZSB0aGFuIDI3LDAwMCB3b3JkczoKYGBge3J9CmhlYWQoc2VudGltZW50cykKYGBgCgpMZXQncyB1c2UgdGhlIG5yYyBsZXhpY29uIGFuZCBsb29rIGZvciBqb3lmdWxsIHdvcmRzIGluIFExLiBGaXJzdCwgZ2V0IHRoZSBqb3lmdWwgd29yZHMgaW50byBhIGRmOgoKYGBge3J9Cm5yY19qb3kgPC0gZ2V0X3NlbnRpbWVudHMoIm5yYyIpICU+JSAKICBmaWx0ZXIoc2VudGltZW50ID09ICJqb3kiKQpgYGAKClRoZW4gbG9vayBmb3IgdGhlbSBpbiBhbnN3ZXJzIHRvIFExOgoKYGBge3J9CndvcmRzX3ExICU+JQogIGlubmVyX2pvaW4obnJjX2pveSkgJT4lCiAgY291bnQod29yZCwgc29ydCA9IFRSVUUpCmBgYApTbyB0aGVyZSwgYSBiaXQgb2Ygam95ISBZb3UgY2FuIHBlcmZvcm0gYWRkaXRpb25hbCAoc2VudGltZW50KSBhbmFseXNpcyBmb2xsb3dpbmcgdGhlIGV4YW1wbGVzIGluIFt0aGlzIGJvb2tdKGh0dHBzOi8vd3d3LnRpZHl0ZXh0bWluaW5nLmNvbS9pbmRleC5odG1sKS0tc2VlIGluIGluIHBhcnRpY3VsYXIgdGhlIGNoYXB0ZXIgb24gdHdpdHRlciBhbmFseXNpcyB0aGF0IGlzIHNpbWlsYXIgaW4gZGF0YSBzdHJ1Y3R1cmUgdG8gYSBzdXJ2ZXktLWJ1dCBiZWZvcmUgdGhhdCBpdCBpcyBhIGdvb2QgaWRlYSB0byBjaGFuZ2Ugb3VyIGRhdGEgc3RydWN0dXJlIHNvIHRoYXQgYWxsIHRoZSBzdXJ2ZXkgYW5zd2VycyBhcmUgaW4gYSB0aWR5IGZvcm1hdC4gCgojIyBUaWR5aW5nIHRoZSBzdXJ2ZXkgZGF0YQoKV2hpbGUgaXQgaXMgc2Vuc2libGUgdG8gYW5hbHlzZSB0aGUgYW5zd2VycyBzZXBhcmF0ZWx5IGZvciBlYWNoIHF1ZXN0aW9uLCBpdCBpcyBsZXNzIHRoYW4gZWxlZ2FudCB0byBkbyB0aGlzIGJhc2VkIG9uIGEgZGF0YWZyYW1lIHRoYXQgaGFzIGp1c3Qgb25lIHF1ZXN0aW9uIGluIGl0IGFzIHdlIGRpZCBzbyBmYXIuIEl0IHdvdWxkIG1lYW4gdG8gY29weSB0aGUgYW5hbHlzaXMgOCB0aW1lcyBhbmQgcmVwbGFjZSBhbGwgCgpUbyBnZXQgdGhlIGRhdGEgaW50byB0aWR5IGZvcm1hdCwgd2UgbmVlZCAnbGVuZ3RoZW4nIHRoZSBvcmlnaW5hbCBkYXRhLiBXZSBnYXRoZXIgdGhlIHF1ZXN0aW9uIHJlc3BvbnNlcyBhbGwgaW50byBvbmUgY29sdW1uIGluc3RlYWQgb2YgbGVhdmluZyB0aGVtIGRpc3RyaWJ1dGVkIG92ZXIgOSBjb2x1bW5zLiBGb3IgdGhpcyB3ZSBuZWVkIGEgbmV3IGNvbHVtbiBgcXVlc3Rpb25zYC4gVGhlIGBnYXRoZXJgIGZ1bmN0aW9uIGlzIHBhcnQgb2YgdGhlICd0aWR5JyBwYWNrYWdlLCB3aGljaCB3ZSBsb2FkZWQgd2hlbiB3ZSBsb2FkZWQgdGhlIHRpZHl2ZXJzZSBwYWNrYWdlcyBhYm92ZS4gCgpgYGB7ciB0aWR5aW5nIHRoZSBzdXJ2ZXl9CnRleHRfZGYgPC0gdGV4dF9kZiAlPiUKICBnYXRoZXIoJ1ExJzonUTknLCBrZXk9InF1ZXN0aW9uIiwgdmFsdWUgPSAid29yZHMiKQpgYGAKCldlIGtub3cgaGF2ZSBhIHRhYmxlIHdpdGggMjA3IHJvd3MgaW4gdGhyZWUgY29sdW1uczogUmVzcG9uZGVudCBJRCwgUXVlc3Rpb24gbnVtYmVyLCBhbmQgd29yZHMgKHRoZSByZXNwb25zZSkuIAoKYGBge3J9CmhlYWQodGV4dF9kZikKYGBgCgoKCiMjIFRleHQgbWluaW5nIG9uIHRoZSB0aWR5IGRhdGEgc2V0CgpMZXQncyBnZXQgdG8gdGhlIHdvcmRzOiAKCmBgYHtyfQp3b3JkcyA8LSB0ZXh0X2RmICU+JQogIHVubmVzdF90b2tlbnMod29yZCwgd29yZHMpCmBgYAphbmQgcmVtb3ZlIHN0b3B3b3JkczoKYGBge3IgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQp3b3JkcyA8LSBhbnRpX2pvaW4od29yZHMsIHN0b3Bfd29yZHMpCmhlYWQod29yZHMpCmBgYAoKCldlIG1heSBhbHNvIHdhbnQgdG8gcmVtb3ZlIG51bWJlcnMgYW5kIHNwZWNpYWwgY2hhcmFjdGVycy4gRmlyc3Qgd2UgbmVlZCBhIHRpYmJsZSB0byBzdG9yZSB0aGVzZSBraW5kIG9mIHN0b3Agc3ltYm9scyB1bmRlcjoKCmBgYHtyfQpzdG9wX3N5bWJvbHMgPC0gdGliYmxlKHdvcmQgPSBjKCIxIiwgIjIiLCAiMyIsICI0IiwgIjUiLCAiNiIsICI3IiwgIjgiLCAiOSIsICIwIiwgIiMiLCAiKCIsICIpIiwgIi4iKSkKYGBgCgpTaW5jZSB0aGUgc3VydmV5IGRhdGEgaXMgaW4gb25lLXdvcmQtcGVyLXJvdyBmb3JtYXQsIHdlIGNhbiByZW1vdmUgc3RvcCBzeW1ib2xzIHdpdGggYW4gYW50aV9qb2luIChmcm9tIGRwbHlyKToKCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CndvcmRzIDwtIHdvcmRzICU+JQogIGFudGlfam9pbihzdG9wX3N5bWJvbHMpCmBgYApUaGUgZGF0YSBhcmUgbm93IGNsZWFuZXIsIGJ1dCBub3Qgc2hvd24gaGVyZS4gCgoKVGhpcyBub3cgYWxzbyBpcyB0aGUgcG9pbnQgd2VyZSB3ZSBhbHNvIGNvdWxkIGRvIHNvbWUgc3RlbW1pbmcuIEZvciB0aGlzIHdlIG5lZWQgdGhpcyBsaWJyYXJ5OgpgYGB7cn0KbGlicmFyeShTbm93YmFsbEMpCmBgYApXaGljaCBnaXZlcyB1cyBhIGZ1bmN0aW9uICd3b3Jkc3RlbSc6IAoKYGBge3Igc3RlbW1pbmd9CndvcmRzX3N0ZW1tZWQgPC0gbXV0YXRlKHdvcmRzLCB3b3JkID0gd29yZFN0ZW0od29yZCkpCmhlYWQod29yZHNfc3RlbW1lZCwgbj0yMEwpCmBgYApCdXQgd2UgZG9uJ3Qgd2FudCB0byB1c2UgdGhlIHN0ZW1tZWQgd29yZHMganVzdCBub3cgIGJlY2F1c2UgaXQgd291bGQgaW50ZWZlcmUgd2l0aCB0aGUgc2VudGltZW50IGFuYWx5c2lzLiBTdGVtbWluZyBpcyB1c2VmdWwsIGJ1dCBvbmx5IG9uY2Ugd2Ugbm8gbG9uZ2VyIG5lZWQgInJlYWwiIHdvcmRzLCBzdWNoIGFzIGZvciB0b3BpYyBtb2RlbGxpbmcgYW5kIHRoZSBsaWtlcy4gCgoKIyMgV29yZCBmcmVxdWVuY2llcwoKTm93IHdlIGNhbiBjYWxjdWxhdGUgd29yZCBmcmVxdWVuY2llcyBmb3IgZWFjaCBwZXJzb24uIEZpcnN0LCB3ZSBncm91cCBieSBwZXJzb24gYW5kIGNvdW50IGhvdyBtYW55IHRpbWVzIGVhY2ggcGVyc29uIHVzZWQgZWFjaCB3b3JkLiBUaGVuIHdlIHVzZSBsZWZ0X2pvaW4oKSB0byBhZGQgYSBjb2x1bW4gb2YgdGhlIHRvdGFsIG51bWJlciBvZiB3b3JkcyB1c2VkIGJ5IGVhY2ggcGVyc29uLiBGaW5hbGx5LCB3ZSBjYWxjdWxhdGUgYSBmcmVxdWVuY3kgZm9yIGVhY2ggcGVyc29uIGFuZCB3b3JkLiAoSWYgdGhpcyBpcyByZWFsbHkgbWVhbmluZ2Z1bCBhY3Jvc3MgcXVlc3Rpb25zLCBhcyBkb25lIGhlcmUsIGlzIGRlYmF0YWJsZSBidXQgdGhpcyBpcyBtYWlubHkgYSBkZW1vbnN0cmF0aW9uLiBTZWUgYmVsb3cgZm9yIGZpbHRlcmluZyBhbmQgZ3JvdXBpbmcuKQoKYGBge3IgRnJlcXVlbmN5IHBlciByZXNwb25kZW50fQpmcmVxdWVuY3kgPC0gd29yZHMgJT4lIAogIGdyb3VwX2J5KFJlc3BvbmRlbnQpICU+JSAKICBjb3VudCh3b3JkLCBzb3J0ID0gVFJVRSkgJT4lIAogIGxlZnRfam9pbih3b3JkcyAlPiUgCiAgICAgICAgICAgICAgZ3JvdXBfYnkoUmVzcG9uZGVudCkgJT4lIAogICAgICAgICAgICAgIHN1bW1hcmlzZSh0b3RhbCA9IG4oKSkpICU+JQogIG11dGF0ZShmcmVxID0gbi90b3RhbCkKaGVhZChmcmVxdWVuY3kpCmBgYAojIyBTZW50aW1lbnQgYW5hbHlzaXMgCgojIyMgU2VudGltZW50cyBhY3Jvc3MgYWxsIHF1ZXN0aW9ucwoKVG8gcmVwZWF0IGZvcm0gdGhlIHZlcnkgZmlyc3QgZXhhbXBsZSwgaWYgd2Ugd2VyZSBpbnRlcmVzdGVkIGluIHRoZSBqb3lmdWxuZXNzLCB3ZSBjYW4gZ2V0IHdvcmRzIG9mIGpveSBmcm9tIGZvciBpbnN0YW5jZSB0aGUgbnJjIGxleGljb246CgpgYGB7cn0KbnJjX2pveSA8LSBnZXRfc2VudGltZW50cygibnJjIikgJT4lIAogIGZpbHRlcihzZW50aW1lbnQgPT0gImpveSIpCmBgYAoKQW5kIHRoZW4gbG9vayBmb3IgdGhlbSBpbiBhbnN3ZXJzIHRvIGZyb20gUTEgdG8gUTk6CgpgYGB7cn0Kd29yZHMgJT4lCiAgaW5uZXJfam9pbihucmNfam95KSAlPiUKICBjb3VudCh3b3JkLCBzb3J0ID0gVFJVRSkgJT4lCiAgdG9wX24oMTApCmBgYAojIyMgU2VudGltZW50IGFuYXlzaXMgb24gb25lIHF1ZXN0aW9uCgpKdXN0IGZvciBRMToKCmBgYHtyfQp3b3JkcyAlPiUKICBmaWx0ZXIocXVlc3Rpb24gPT0gIlExIikgJT4lCiAgaW5uZXJfam9pbihucmNfam95KSAlPiUKICBjb3VudCh3b3JkLCBzb3J0ID0gVFJVRSkKYGBgCk5vdGUgdGhhdCBxdWVzdGlvbnMgSURzIGFyZSBub3cgYSByb3cgdmFsdWUsIHNvIHdlIG5lZWQgdG8gZmlsdGVyIChyb3dzKSByYXRoZXIgdGhhbiBzZWxlY3QgKHZhcmlhYmxlcykuIAoKIyMjIFNlbnRpbWVudCBvbiBhbGwgcXVlc3Rpb25zCgpgYGB7cn0Kd29yZHMgJT4lCiAgZ3JvdXBfYnkocXVlc3Rpb24pICU+JQogIGlubmVyX2pvaW4obnJjX2pveSkgJT4lCiAgY291bnQod29yZCwgc29ydCA9IEZBTFNFKQpgYGAKV2UgdHVybmVkIHNvcnRpbmcgb24gdGhlIGNvdW50IG9mZiBoZXJlIGJlY2F1c2Ugd2Ugd2FudCB0byBoYXZlIHRoZSByZXN1bHRzIGdyb3VwZWQgYnkgcXVlc3Rpb24gaWQuIElmIFNvcnRpbmcgd2FzIFRSVUUsIHRoZSBudW1lcmljIGNvdW50IHdvdWxkIGJlIHVzZWQgZmlyc3QsIHRoZW4gdGhlIHF1ZXN0aW9uIElELiAKCiMjIyBTZW50aW1lbnQgYW5hbHlzaXMgb24gYSBzdWItc2V0IG9mIGFsbCBxdWVzdGlvbnMKCkxldCdzIHBpY2sgUTEsIDMgYW5kIDc6IAoKYGBge3J9CndvcmRzICU+JQogIGZpbHRlcihxdWVzdGlvbiA9PSAiUTEiIHwgcXVlc3Rpb24gPT0gIlEzIiB8IHF1ZXN0aW9uID09ICJRNyIpICU+JQogIGlubmVyX2pvaW4obnJjX2pveSkgJT4lCiAgY291bnQod29yZCwgc29ydCA9IFRSVUUpCmBgYAoKCgoKIyMjIE1vc3QgY29tbW9uIHBvc2l0aXZlIGFuZCBuZWdhdGl2ZSBzZW50aW1lbnQgd29yZHMKClVzaW5nIHRoZSBsZXhpY29uIGJpbmcsIHdlIGNhbiBmaW5kIGhvdyB3b3JkcyBjb3N0cmlidXRlIHRvIHNlbnRpbWVudHMKCmBgYHtyfQpiaW5nX3dvcmRfY291bnRzIDwtIHdvcmRzICU+JQogIGlubmVyX2pvaW4oZ2V0X3NlbnRpbWVudHMoImJpbmciKSkgJT4lCiAgY291bnQod29yZCwgc2VudGltZW50LCBzb3J0ID0gVFJVRSkgJT4lCiAgdW5ncm91cCgpCmhlYWQoYmluZ193b3JkX2NvdW50cykKYGBgCkxldCdzIGdyYXBoIHRoaXMgd2l0aCBnZ3Bsb3QyOiAKCmBgYHtyfQpiaW5nX3dvcmRfY291bnRzICU+JQogIGdyb3VwX2J5KHNlbnRpbWVudCkgJT4lCiAgdG9wX24oMTApICU+JQogIHVuZ3JvdXAoKSAlPiUKICBtdXRhdGUod29yZCA9IHJlb3JkZXIod29yZCwgbikpICU+JQogIGdncGxvdChhZXMod29yZCwgbiwgZmlsbCA9IHNlbnRpbWVudCkpICsKICBnZW9tX2NvbChzaG93LmxlZ2VuZCA9IEZBTFNFKSArCiAgZmFjZXRfd3JhcCh+c2VudGltZW50LCBzY2FsZXMgPSAiZnJlZV95IikgKwogIGxhYnMoeSA9ICJDb250cmlidXRpb24gdG8gc2VudGltZW50IiwKICAgICAgIHggPSBOVUxMKSArCiAgY29vcmRfZmxpcCgpCmBgYApUaGUgdG9wX24oMTApIHNlZW1zIG5vdCB0byB3b3JrIGJ1dCBsZXQncyB3b3JyeSBhYm91dCB0aGF0IGxhdGVyLiAKClRoZXJlJ3MgbXVjaCBvbmUgb25lIGNhbiBkbyB3aXRoIHNlbnRpbWVudCBhbmFseXNpcywgb2J2aW91c2x5LCBzZWUgW3RoZSB0aWR5dGV4dCBib29rXShodHRwczovL3d3dy50aWR5dGV4dG1pbmluZy5jb20vc2VudGltZW50Lmh0bWwpIGFtb25nc3Qgb3RoZXJzLiAKCgoK