Good morning! Ready to learn how to use R? Let’s get started.
Setting up R & RStudio before you start working
First, let’s get a few things set up in RStudio:
Click File > New File > R Script. This will open a new panel in the top left of your RStudio layout.
Click File > Save As. You can name the file “practice”. Make a new folder on the desktop called “workshop” and save it there.
We need to tell R where all our files are. We do this by setting the working directory. In the bottom-right panel in RStudio, click on the Files tab. Click the little box with three dots “…”, then navigate to your “workshop” folder. Last, click on the blue gear with “More” written next to it and select “Set as working directory”
Okay, now we are ready to start doing things. We are going to write our first lines of code to install and load a package. Packages are add-ons to R that provide specialized functions.
Type the following line of code in your Console (bottom-left window) and then press Enter.
install.packages("tidyverse")
You just downloaded a package. You also just wrote and executed your first line of code. Congratulations! You are now a coder.
Now, let’s load the package so we can use it. In the top-left panel, type the following line of code into your R Script. To run this code, you should press CTRL+Enter (Command+Enter on a Mac).
library(tidyverse)
package <U+393C><U+3E31>tidyverse<U+393C><U+3E32> was built under R version 3.5.3[30m-- [1mAttaching packages[22m --------------------------------------- tidyverse 1.2.1 --[39m
[30m[32mv[30m [34mggplot2[30m 3.1.0 [32mv[30m [34mpurrr [30m 0.3.0
[32mv[30m [34mtibble [30m 2.0.1 [32mv[30m [34mdplyr [30m 0.8.0.[31m1[30m
[32mv[30m [34mtidyr [30m 0.8.2 [32mv[30m [34mstringr[30m 1.4.0
[32mv[30m [34mreadr [30m 1.3.1 [32mv[30m [34mforcats[30m 0.4.0 [39m
[30m-- [1mConflicts[22m ------------------------------------------ tidyverse_conflicts() --
[31mx[30m [34mdplyr[30m::[32mfilter()[30m masks [34mstats[30m::filter()
[31mx[30m [34mdplyr[30m::[32mlag()[30m masks [34mstats[30m::lag()[39m
We’re going to download some data that we will use today. Copy and paste the following line of code in your Console:
download.file("https://raw.githubusercontent.com/gtlaflair/ltrc-2019/gh-pages/data/placement_1.csv",
"placement_1.csv", mode = "wb")
trying URL 'https://raw.githubusercontent.com/gtlaflair/ltrc-2019/gh-pages/data/placement_1.csv'
Content type 'text/plain; charset=utf-8' length 17869 bytes (17 KB)
downloaded 17 KB
Great, now we should be all set to go. We will use that package and that data a little bit later.
Some key basics of how R works
R is a calculator!
You can use R like a calculator. Try it out:
2+2
[1] 4
3*8
[1] 24
((2+2)*(3*8))/10
[1] 9.6
9^2
[1] 81
This isn’t the main reason to use R, but you will occasionally use mathematical operations in R. What if you want to take the square root of number?
You do things in R with functions
Functions are the powerhouse of R: These are commands you use to do things. Base R has many functions, and if you load additional packages, you can use other functions. install.packages() and library() are functions, so you are already an experienced function user. To calculate a square root, you can use a function called sqrt(). Let’s try:
sqrt(49)
[1] 7
But to really do work in R, you’ll need to use objects
Objects, like the name suggests, are things your can do stuff to in R. You use function to create and change objects, and you can use functions and objects together to create new objects. Objects are good ways of saving something you want for later. Let’s start making a few simple objects.
First, let’s save the number 7 as an object:
seven <- 7
And now lets do some math with it:
seven + 7
[1] 14
seven^2
[1] 49
seven/seven
[1] 1
seven
[1] 7
Yes, none of this is very interesting yet. But as you can see, you can use and manipulate objects. Importantly, if you just type the object name, R will simply show you the object - in whatever form makes the most sense.
I won’t spend too much more time on these kinds of simplistic objects- but feel free to ask questions. Instead, let’s make an object that has some real data we might actually be interested in working with.
data <- read_csv("placement_1.csv")
Parsed with column specification:
cols(
.default = col_double(),
names = [31mcol_character()[39m,
country = [31mcol_character()[39m,
admin_date = [34mcol_datetime(format = "")[39m
)
See spec(...) for full column specifications.
Creating new variables
This dataset is missing total scores. Let’s create some new variables for these using dplyr::mutate. First, we’ll make a reading total score.
data <- data %>% mutate(read_total = rowSums(.[40:74], na.rm = T))
Now we’ll do the same thing for listening:
data <- data %>% mutate(list_total = rowSums(.[5:39], na.rm = T))
Finally, let’s creat a whole-test total score. This will be easy!
data <- mutate(data, total = read_total + list_total)
Summarizing data
We can also use dplyr to summarize data in R. Let’s start simple by just looking at total scores.
data %>% summarise(total_mean = mean(total))
We can do more than one summary stat in the same function. We’ll add in SD, median, min, and max.
data %>% summarise(total_mean = mean(total),
total_sd = sd(total),
total_median = median(total),
total_min = min(total),
total_max = max(total))
What if we want to save those summary stats so that we can look at them later? Let’s create a new object that contains our calculations of summary stats.
total_summary <- data %>% summarise(total_mean = mean(total),
total_sd = sd(total),
total_median = median(total),
total_min = min(total),
total_max = max(total))
Now if we just type “total_summary” and run it (in a script or in the console), we get to see our values.
total_summary
One very powerful feature of dplyr is the ability to do split-apply-combine. Basically, you split up your data into groups, apply some function or calcuation, and then combine the results. We’ll try this out by using dplyr::group_by to produce summary statistics for learners’ country of origin.
data %>% group_by(country) %>% summarise(total_mean = mean(total),
total_sd = sd(total),
total_median = median(total),
total_min = min(total),
total_max = max(total))
Of course, we will probably want to save these summary stats. We can save to an object, and we can write that object to a .csv that we can send to our colleagues.
country_summary <- data %>% group_by(country) %>% summarise(total_mean = mean(total),
total_sd = sd(total),
total_median = median(total),
total_min = min(total),
total_max = max(total))
write_csv(country_summary, "country_summary.csv")
Basic Stats
Very briefly, here are ways to run some basic stats in R.
Correlation
R is great for correlations. There are packages and functions that can run correlations for a whole buch of variables at once and even automatically generate really cool graphics to illustrate correlations. We’ll start with a simple but very nice function, cor.test(), that provides a p value AND a confidence interval for a simple bivariate correlation.
cor.test(data$read_total, data$list_total, method = "pearson")
Pearson's product-moment correlation
data: data$read_total and data$list_total
t = 8.8627, df = 86, p-value = 9.368e-14
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.5629244 0.7865349
sample estimates:
cor
0.6909085
t-test
R, of course, can also do a simple t-test. Let’s compare the total scores of Russian and Chinese students.
First, we will use a tidyverse package called tidyr to filter the dataset.
china_russia <- data %>% filter(country %in% c("china", "russia"))
t-test time!
t.test(total ~ country, data = china_russia)
Welch Two Sample t-test
data: total by country
t = 0.59227, df = 54.286, p-value = 0.5561
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-4.048987 7.444821
sample estimates:
mean in group china mean in group russia
41.91667 40.21875
ANOVA
An ANOVA would let us compare the total scores of students from all 3 countries.
country_total_anova <- aov(total ~ country, data = data)
country_total_anova
Call:
aov(formula = total ~ country, data = data)
Terms:
country Residuals
Sum of Squares 269.936 10540.019
Deg. of Freedom 2 85
Residual standard error: 11.13554
Estimated effects may be unbalanced
summary(country_total_anova)
Df Sum Sq Mean Sq F value Pr(>F)
country 2 270 135 1.088 0.341
Residuals 85 10540 124
R can do many, many more statistical analyses- everything from Chi-square tests to mixed-effect regression to structural equation modeling to network analysis. This was just a taste of the basics!
Graphics with R
Graphics might be the biggest reason to start learning and using R. Compared to SPSS or Excel, graphics made in R generally look better and are actually easier to customize once you get familiar with the basics. I strongly recommend using the ggplot package (part of tidyverse) for creating graphics.
Histograms: an easy first plot
We’ll start by making a histogram of total scores.
ggplot(data = data, aes(x = total))+
geom_histogram()

That is a very basic, but very readable histogram. Let’s customize it a bit to make it prettier:
total_hist <- ggplot(data = data, aes(x = total))+
geom_histogram(binwidth = 5)+
labs(x = "Total Score", y = "Number of Test-Takers")+
theme_bw()
total_hist

Now this looks pretty nice! We can save a high-resolution version of this that will look great in a powerpoint or a paper:
ggsave("total_histogram.png", total_hist, width = 6, height = 4, units = "in", dpi = 300)
Go to your folder and open that - looks pretty good, right? We can also do some really neat stuff looking at subgroups with plots. Let’s try:
ggplot(data = data, aes(x = total))+
geom_histogram(binwidth = 5)+
labs(x = "Total Score", y = "Number of Test-Takers")+
theme_bw()+
facet_wrap(~country, nrow = 1)

Looks like Russian students might have a bimodal distribution! You can save this graph if you want - look at the previous example for hints.
Boxplots and variations
Boxplots are nice ways of summarizing different groups or different variables graphically. Here’s a simple one:
ggplot(data = data, aes(x = country, y = total))+
geom_boxplot()+
theme_bw()

That’s okay, but we could make it more informative. Violin plots are a variation that show the distributions in more detail. We’ll also add some color just to make things pop more.
ggplot(data = data, aes(x = country, y = total, fill = country))+
geom_violin()+
theme_bw()

We can also add individual points to boxplots (or violin plots)
ggplot(data = data, aes(x = country, y = total, fill = country))+
geom_boxplot()+
geom_dotplot(binaxis='y', stackdir='center', dotsize=1, binwidth = 1, fill = "black")+
theme_bw()

Lots of flexibility here!
Scatterplots
The last plot type we’ll take a look at today is the scatterplot. These are often the most informative plots, in my opinion, because they can show the relationship between variables in really interesting ways. Let’s start simple:
ggplot(data = data, aes(x = read_total, y = list_total))+
geom_point()

This shows a pretty clear relationship between reading and listening scores, just like our correlation from earlier indicated. In this plot, it doesn’t look like we have too much overplotting (i.e., when you have points that overlap each other). But if we did have some overplotting issues, we could use geom_jitter instead of geom_point, like this:
ggplot(data = data, aes(x = read_total, y = list_total))+
geom_jitter()

Looking at this, I think we should probably use geom_jitter. geom_jitter adds a little bit of random noise to the location of points, nudging them away from each other so you can see them better.
Let’s clean up this plot and add a regression line to show the relationship more clearly.
ggplot(data = data, aes(x = read_total, y = list_total))+
geom_jitter()+
geom_smooth(method = lm, se = FALSE)+
labs(y = "Listening Score", x = "Reading Score")+
theme_bw()

This is looking nice! We can also break down this relationship by groups according to country:
ggplot(data = data, aes(x = read_total, y = list_total, group = country, color = country))+
geom_jitter()+
geom_smooth(method = lm, se = FALSE)+
labs(y = "Listening Score", x = "Reading Score")+
theme_bw()

I encourage you to try using R to create graphics for your next presentation, paper, or thesis, even if you still want to use SPSS for your stats.
LS0tDQp0aXRsZTogIkludHJvIHRvIFIgJiBSU3R1ZGlvIGZvciBIVUZTIFRFU09MIg0Kb3V0cHV0OiBodG1sX25vdGVib29rDQotLS0NCg0KR29vZCBtb3JuaW5nISBSZWFkeSB0byBsZWFybiBob3cgdG8gdXNlIFI/IExldCdzIGdldCBzdGFydGVkLg0KDQojIyBTZXR0aW5nIHVwIFIgJiBSU3R1ZGlvIGJlZm9yZSB5b3Ugc3RhcnQgd29ya2luZw0KDQpGaXJzdCwgbGV0J3MgZ2V0IGEgZmV3IHRoaW5ncyBzZXQgdXAgaW4gUlN0dWRpbzoNCg0KMS4gQ2xpY2sgRmlsZSA+IE5ldyBGaWxlID4gUiBTY3JpcHQuIFRoaXMgd2lsbCBvcGVuIGEgbmV3IHBhbmVsIGluIHRoZSB0b3AgbGVmdCBvZiB5b3VyIFJTdHVkaW8gbGF5b3V0Lg0KDQoyLiBDbGljayBGaWxlID4gU2F2ZSBBcy4gWW91IGNhbiBuYW1lIHRoZSBmaWxlICJwcmFjdGljZSIuIE1ha2UgYSBuZXcgZm9sZGVyIG9uIHRoZSBkZXNrdG9wIGNhbGxlZCAid29ya3Nob3AiIGFuZCBzYXZlIGl0IHRoZXJlLg0KDQozLiBXZSBuZWVkIHRvIHRlbGwgUiB3aGVyZSBhbGwgb3VyIGZpbGVzIGFyZS4gV2UgZG8gdGhpcyBieSAqc2V0dGluZyB0aGUgd29ya2luZyBkaXJlY3RvcnkqLiBJbiB0aGUgYm90dG9tLXJpZ2h0IHBhbmVsIGluIFJTdHVkaW8sIGNsaWNrIG9uIHRoZSBGaWxlcyB0YWIuIENsaWNrIHRoZSBsaXR0bGUgYm94IHdpdGggdGhyZWUgZG90cyAiLi4uIiwgdGhlbiBuYXZpZ2F0ZSB0byB5b3VyICJ3b3Jrc2hvcCIgZm9sZGVyLiBMYXN0LCBjbGljayBvbiB0aGUgYmx1ZSBnZWFyIHdpdGggIk1vcmUiIHdyaXR0ZW4gbmV4dCB0byBpdCBhbmQgc2VsZWN0ICJTZXQgYXMgd29ya2luZyBkaXJlY3RvcnkiDQoNCk9rYXksIG5vdyB3ZSBhcmUgcmVhZHkgdG8gc3RhcnQgZG9pbmcgdGhpbmdzLiBXZSBhcmUgZ29pbmcgdG8gd3JpdGUgb3VyIGZpcnN0IGxpbmVzIG9mIGNvZGUgdG8gKmluc3RhbGwqIGFuZCAqbG9hZCogYSAqKnBhY2thZ2UqKi4gUGFja2FnZXMgYXJlIGFkZC1vbnMgdG8gUiB0aGF0IHByb3ZpZGUgc3BlY2lhbGl6ZWQgZnVuY3Rpb25zLg0KDQpUeXBlIHRoZSBmb2xsb3dpbmcgbGluZSBvZiBjb2RlIGluIHlvdXIgQ29uc29sZSAoYm90dG9tLWxlZnQgd2luZG93KSBhbmQgdGhlbiBwcmVzcyAqKkVudGVyKiouDQoNCmBgYHtyIGV2YWw9RkFMU0V9DQppbnN0YWxsLnBhY2thZ2VzKCJ0aWR5dmVyc2UiKQ0KYGBgDQoNCllvdSBqdXN0IGRvd25sb2FkZWQgYSBwYWNrYWdlLiBZb3UgYWxzbyBqdXN0IHdyb3RlIGFuZCBleGVjdXRlZCB5b3VyIGZpcnN0IGxpbmUgb2YgY29kZS4gQ29uZ3JhdHVsYXRpb25zISBZb3UgYXJlIG5vdyBhIGNvZGVyLg0KDQpOb3csIGxldCdzICpsb2FkKiB0aGUgcGFja2FnZSBzbyB3ZSBjYW4gdXNlIGl0LiBJbiB0aGUgdG9wLWxlZnQgcGFuZWwsIHR5cGUgdGhlIGZvbGxvd2luZyBsaW5lIG9mIGNvZGUgaW50byB5b3VyIFIgU2NyaXB0LiBUbyBydW4gdGhpcyBjb2RlLCB5b3Ugc2hvdWxkIHByZXNzICoqQ1RSTCtFbnRlcioqIChDb21tYW5kK0VudGVyIG9uIGEgTWFjKS4NCg0KYGBge3J9DQpsaWJyYXJ5KHRpZHl2ZXJzZSkNCmBgYA0KDQpXZSdyZSBnb2luZyB0byBkb3dubG9hZCBzb21lIGRhdGEgdGhhdCB3ZSB3aWxsIHVzZSB0b2RheS4gQ29weSBhbmQgcGFzdGUgdGhlIGZvbGxvd2luZyBsaW5lIG9mIGNvZGUgaW4geW91ciBDb25zb2xlOg0KDQpgYGB7cn0NCmRvd25sb2FkLmZpbGUoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9ndGxhZmxhaXIvbHRyYy0yMDE5L2doLXBhZ2VzL2RhdGEvcGxhY2VtZW50XzEuY3N2IiwNCiAgICAgICAgICAgICAgInBsYWNlbWVudF8xLmNzdiIsIG1vZGUgPSAid2IiKQ0KYGBgDQoNCg0KR3JlYXQsIG5vdyB3ZSBzaG91bGQgYmUgYWxsIHNldCB0byBnby4gV2Ugd2lsbCB1c2UgdGhhdCBwYWNrYWdlIGFuZCB0aGF0IGRhdGEgYSBsaXR0bGUgYml0IGxhdGVyLg0KDQojIyBTb21lIGtleSBiYXNpY3Mgb2YgaG93IFIgd29ya3MNCg0KIyMjIFIgaXMgYSBjYWxjdWxhdG9yIQ0KDQpZb3UgY2FuIHVzZSBSIGxpa2UgYSBjYWxjdWxhdG9yLiBUcnkgaXQgb3V0Og0KDQpgYGB7cn0NCjIrMg0KDQozKjgNCg0KKCgyKzIpKigzKjgpKS8xMA0KDQo5XjINCmBgYA0KDQpUaGlzIGlzbid0IHRoZSBtYWluIHJlYXNvbiB0byB1c2UgUiwgYnV0IHlvdSB3aWxsIG9jY2FzaW9uYWxseSB1c2UgbWF0aGVtYXRpY2FsIG9wZXJhdGlvbnMgaW4gUi4gV2hhdCBpZiB5b3Ugd2FudCB0byB0YWtlIHRoZSBzcXVhcmUgcm9vdCBvZiBudW1iZXI/IA0KDQojIyMgWW91IGRvIHRoaW5ncyBpbiBSIHdpdGggZnVuY3Rpb25zDQoNCioqRnVuY3Rpb25zKiogYXJlIHRoZSBwb3dlcmhvdXNlIG9mIFI6IFRoZXNlIGFyZSBjb21tYW5kcyB5b3UgdXNlIHRvIGRvIHRoaW5ncy4gQmFzZSBSIGhhcyBtYW55IGZ1bmN0aW9ucywgYW5kIGlmIHlvdSBsb2FkIGFkZGl0aW9uYWwgcGFja2FnZXMsIHlvdSBjYW4gdXNlIG90aGVyIGZ1bmN0aW9ucy4gaW5zdGFsbC5wYWNrYWdlcygpIGFuZCBsaWJyYXJ5KCkgYXJlIGZ1bmN0aW9ucywgc28geW91IGFyZSBhbHJlYWR5IGFuIGV4cGVyaWVuY2VkIGZ1bmN0aW9uIHVzZXIuIFRvIGNhbGN1bGF0ZSBhIHNxdWFyZSByb290LCB5b3UgY2FuIHVzZSBhIGZ1bmN0aW9uIGNhbGxlZCBzcXJ0KCkuIExldCdzIHRyeToNCg0KYGBge3J9DQpzcXJ0KDQ5KQ0KYGBgDQoNCiMjIyBCdXQgdG8gcmVhbGx5IGRvIHdvcmsgaW4gUiwgeW91J2xsIG5lZWQgdG8gdXNlIG9iamVjdHMNCg0KKipPYmplY3RzKiosIGxpa2UgdGhlIG5hbWUgc3VnZ2VzdHMsIGFyZSB0aGluZ3MgeW91ciBjYW4gZG8gc3R1ZmYgdG8gaW4gUi4gWW91IHVzZSBmdW5jdGlvbiB0byBjcmVhdGUgYW5kIGNoYW5nZSBvYmplY3RzLCBhbmQgeW91IGNhbiB1c2UgZnVuY3Rpb25zIGFuZCBvYmplY3RzIHRvZ2V0aGVyIHRvIGNyZWF0ZSBuZXcgb2JqZWN0cy4gT2JqZWN0cyBhcmUgZ29vZCB3YXlzIG9mICpzYXZpbmcqIHNvbWV0aGluZyB5b3Ugd2FudCBmb3IgbGF0ZXIuIExldCdzIHN0YXJ0IG1ha2luZyBhIGZldyBzaW1wbGUgb2JqZWN0cy4NCg0KRmlyc3QsIGxldCdzIHNhdmUgdGhlIG51bWJlciA3IGFzIGFuIG9iamVjdDoNCg0KYGBge3J9DQpzZXZlbiA8LSA3DQpgYGANCg0KQW5kIG5vdyBsZXRzIGRvIHNvbWUgbWF0aCB3aXRoIGl0Og0KDQpgYGB7cn0NCnNldmVuICsgNw0KDQpzZXZlbl4yDQoNCnNldmVuL3NldmVuDQoNCnNldmVuDQpgYGANCg0KWWVzLCBub25lIG9mIHRoaXMgaXMgdmVyeSBpbnRlcmVzdGluZyB5ZXQuIEJ1dCBhcyB5b3UgY2FuIHNlZSwgeW91IGNhbiB1c2UgYW5kIG1hbmlwdWxhdGUgb2JqZWN0cy4gSW1wb3J0YW50bHksIGlmIHlvdSBqdXN0IHR5cGUgdGhlIG9iamVjdCBuYW1lLCBSIHdpbGwgc2ltcGx5IHNob3cgeW91IHRoZSBvYmplY3QgLSBpbiB3aGF0ZXZlciBmb3JtIG1ha2VzIHRoZSBtb3N0IHNlbnNlLiANCg0KSSB3b24ndCBzcGVuZCB0b28gbXVjaCBtb3JlIHRpbWUgb24gdGhlc2Uga2luZHMgb2Ygc2ltcGxpc3RpYyBvYmplY3RzLSBidXQgZmVlbCBmcmVlIHRvIGFzayBxdWVzdGlvbnMuIEluc3RlYWQsIGxldCdzIG1ha2UgYW4gb2JqZWN0IHRoYXQgaGFzIHNvbWUgcmVhbCAqZGF0YSogd2UgbWlnaHQgYWN0dWFsbHkgYmUgaW50ZXJlc3RlZCBpbiB3b3JraW5nIHdpdGguDQoNCmBgYHtyfQ0KZGF0YSA8LSByZWFkX2NzdigicGxhY2VtZW50XzEuY3N2IikNCmBgYA0KDQojIyBDcmVhdGluZyBuZXcgdmFyaWFibGVzDQoNClRoaXMgZGF0YXNldCBpcyBtaXNzaW5nIHRvdGFsIHNjb3Jlcy4gTGV0J3MgY3JlYXRlIHNvbWUgbmV3IHZhcmlhYmxlcyBmb3IgdGhlc2UgdXNpbmcgZHBseXI6Om11dGF0ZS4gRmlyc3QsIHdlJ2xsIG1ha2UgYSByZWFkaW5nIHRvdGFsIHNjb3JlLg0KDQpgYGB7cn0NCmRhdGEgPC0gZGF0YSAlPiUgbXV0YXRlKHJlYWRfdG90YWwgPSByb3dTdW1zKC5bNDA6NzRdLCBuYS5ybSA9IFQpKQ0KYGBgDQoNCk5vdyB3ZSdsbCBkbyB0aGUgc2FtZSB0aGluZyBmb3IgbGlzdGVuaW5nOg0KDQpgYGB7cn0NCmRhdGEgPC0gZGF0YSAlPiUgbXV0YXRlKGxpc3RfdG90YWwgPSByb3dTdW1zKC5bNTozOV0sIG5hLnJtID0gVCkpDQpgYGANCg0KRmluYWxseSwgbGV0J3MgY3JlYXQgYSB3aG9sZS10ZXN0IHRvdGFsIHNjb3JlLiBUaGlzIHdpbGwgYmUgZWFzeSENCg0KYGBge3J9DQpkYXRhIDwtIG11dGF0ZShkYXRhLCB0b3RhbCA9IHJlYWRfdG90YWwgKyBsaXN0X3RvdGFsKQ0KYGBgDQoNCiMjIFN1bW1hcml6aW5nIGRhdGENCg0KV2UgY2FuIGFsc28gdXNlIGRwbHlyIHRvIHN1bW1hcml6ZSBkYXRhIGluIFIuIExldCdzIHN0YXJ0IHNpbXBsZSBieSBqdXN0IGxvb2tpbmcgYXQgdG90YWwgc2NvcmVzLg0KDQpgYGB7cn0NCmRhdGEgJT4lIHN1bW1hcmlzZSh0b3RhbF9tZWFuID0gbWVhbih0b3RhbCkpDQpgYGANCg0KV2UgY2FuIGRvIG1vcmUgdGhhbiBvbmUgc3VtbWFyeSBzdGF0IGluIHRoZSBzYW1lIGZ1bmN0aW9uLiBXZSdsbCBhZGQgaW4gU0QsIG1lZGlhbiwgbWluLCBhbmQgbWF4Lg0KDQpgYGB7cn0NCmRhdGEgJT4lIHN1bW1hcmlzZSh0b3RhbF9tZWFuID0gbWVhbih0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfc2QgPSBzZCh0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfbWVkaWFuID0gbWVkaWFuKHRvdGFsKSwNCiAgICAgICAgICAgICAgICAgICB0b3RhbF9taW4gPSBtaW4odG90YWwpLA0KICAgICAgICAgICAgICAgICAgIHRvdGFsX21heCA9IG1heCh0b3RhbCkpDQpgYGANCg0KV2hhdCBpZiB3ZSB3YW50IHRvIHNhdmUgdGhvc2Ugc3VtbWFyeSBzdGF0cyBzbyB0aGF0IHdlIGNhbiBsb29rIGF0IHRoZW0gbGF0ZXI/IExldCdzIGNyZWF0ZSBhIG5ldyBvYmplY3QgdGhhdCBjb250YWlucyBvdXIgY2FsY3VsYXRpb25zIG9mIHN1bW1hcnkgc3RhdHMuDQoNCmBgYHtyfQ0KdG90YWxfc3VtbWFyeSA8LSBkYXRhICU+JSBzdW1tYXJpc2UodG90YWxfbWVhbiA9IG1lYW4odG90YWwpLA0KICAgICAgICAgICAgICAgICAgIHRvdGFsX3NkID0gc2QodG90YWwpLA0KICAgICAgICAgICAgICAgICAgIHRvdGFsX21lZGlhbiA9IG1lZGlhbih0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfbWluID0gbWluKHRvdGFsKSwNCiAgICAgICAgICAgICAgICAgICB0b3RhbF9tYXggPSBtYXgodG90YWwpKQ0KYGBgDQoNCk5vdyBpZiB3ZSBqdXN0IHR5cGUgInRvdGFsX3N1bW1hcnkiIGFuZCBydW4gaXQgKGluIGEgc2NyaXB0IG9yIGluIHRoZSBjb25zb2xlKSwgd2UgZ2V0IHRvIHNlZSBvdXIgdmFsdWVzLg0KDQpgYGB7cn0NCnRvdGFsX3N1bW1hcnkNCmBgYA0KDQpPbmUgdmVyeSBwb3dlcmZ1bCBmZWF0dXJlIG9mIGRwbHlyIGlzIHRoZSBhYmlsaXR5IHRvIGRvICpzcGxpdC1hcHBseS1jb21iaW5lKi4gQmFzaWNhbGx5LCB5b3Ugc3BsaXQgdXAgeW91ciBkYXRhIGludG8gZ3JvdXBzLCBhcHBseSBzb21lIGZ1bmN0aW9uIG9yIGNhbGN1YXRpb24sIGFuZCB0aGVuIGNvbWJpbmUgdGhlIHJlc3VsdHMuIFdlJ2xsIHRyeSB0aGlzIG91dCBieSB1c2luZyBkcGx5cjo6Z3JvdXBfYnkgdG8gcHJvZHVjZSBzdW1tYXJ5IHN0YXRpc3RpY3MgZm9yIGxlYXJuZXJzJyBjb3VudHJ5IG9mIG9yaWdpbi4NCg0KYGBge3J9DQpkYXRhICU+JSBncm91cF9ieShjb3VudHJ5KSAlPiUgc3VtbWFyaXNlKHRvdGFsX21lYW4gPSBtZWFuKHRvdGFsKSwNCiAgICAgICAgICAgICAgICAgICB0b3RhbF9zZCA9IHNkKHRvdGFsKSwNCiAgICAgICAgICAgICAgICAgICB0b3RhbF9tZWRpYW4gPSBtZWRpYW4odG90YWwpLA0KICAgICAgICAgICAgICAgICAgIHRvdGFsX21pbiA9IG1pbih0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfbWF4ID0gbWF4KHRvdGFsKSkNCmBgYA0KDQpPZiBjb3Vyc2UsIHdlIHdpbGwgcHJvYmFibHkgd2FudCB0byBzYXZlIHRoZXNlIHN1bW1hcnkgc3RhdHMuIFdlIGNhbiBzYXZlIHRvIGFuIG9iamVjdCwgYW5kIHdlIGNhbiB3cml0ZSB0aGF0IG9iamVjdCB0byBhIC5jc3YgdGhhdCB3ZSBjYW4gc2VuZCB0byBvdXIgY29sbGVhZ3Vlcy4NCg0KYGBge3J9DQpjb3VudHJ5X3N1bW1hcnkgPC0gZGF0YSAlPiUgZ3JvdXBfYnkoY291bnRyeSkgJT4lIHN1bW1hcmlzZSh0b3RhbF9tZWFuID0gbWVhbih0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfc2QgPSBzZCh0b3RhbCksDQogICAgICAgICAgICAgICAgICAgdG90YWxfbWVkaWFuID0gbWVkaWFuKHRvdGFsKSwNCiAgICAgICAgICAgICAgICAgICB0b3RhbF9taW4gPSBtaW4odG90YWwpLA0KICAgICAgICAgICAgICAgICAgIHRvdGFsX21heCA9IG1heCh0b3RhbCkpDQoNCndyaXRlX2Nzdihjb3VudHJ5X3N1bW1hcnksICJjb3VudHJ5X3N1bW1hcnkuY3N2IikNCmBgYA0KDQoNCiMjIEJhc2ljIFN0YXRzDQoNClZlcnkgYnJpZWZseSwgaGVyZSBhcmUgd2F5cyB0byBydW4gc29tZSBiYXNpYyBzdGF0cyBpbiBSLg0KDQojIyMgQ29ycmVsYXRpb24NClIgaXMgZ3JlYXQgZm9yIGNvcnJlbGF0aW9ucy4gVGhlcmUgYXJlIHBhY2thZ2VzIGFuZCBmdW5jdGlvbnMgdGhhdCBjYW4gcnVuIGNvcnJlbGF0aW9ucyBmb3IgYSB3aG9sZSBidWNoIG9mIHZhcmlhYmxlcyBhdCBvbmNlIGFuZCBldmVuIGF1dG9tYXRpY2FsbHkgZ2VuZXJhdGUgcmVhbGx5IGNvb2wgZ3JhcGhpY3MgdG8gaWxsdXN0cmF0ZSBjb3JyZWxhdGlvbnMuIFdlJ2xsIHN0YXJ0IHdpdGggYSBzaW1wbGUgYnV0IHZlcnkgbmljZSBmdW5jdGlvbiwgY29yLnRlc3QoKSwgdGhhdCBwcm92aWRlcyBhIHAgdmFsdWUgQU5EIGEgY29uZmlkZW5jZSBpbnRlcnZhbCBmb3IgYSBzaW1wbGUgYml2YXJpYXRlIGNvcnJlbGF0aW9uLg0KDQpgYGB7cn0NCmNvci50ZXN0KGRhdGEkcmVhZF90b3RhbCwgZGF0YSRsaXN0X3RvdGFsLCBtZXRob2QgPSAicGVhcnNvbiIpDQpgYGANCg0KIyMjIHQtdGVzdA0KUiwgb2YgY291cnNlLCBjYW4gYWxzbyBkbyBhIHNpbXBsZSB0LXRlc3QuIExldCdzIGNvbXBhcmUgdGhlIHRvdGFsIHNjb3JlcyBvZiBSdXNzaWFuIGFuZCBDaGluZXNlIHN0dWRlbnRzLg0KDQpGaXJzdCwgd2Ugd2lsbCB1c2UgYSB0aWR5dmVyc2UgcGFja2FnZSBjYWxsZWQgdGlkeXIgdG8gZmlsdGVyIHRoZSBkYXRhc2V0Lg0KDQpgYGB7cn0NCmNoaW5hX3J1c3NpYSA8LSBkYXRhICU+JSBmaWx0ZXIoY291bnRyeSAlaW4lIGMoImNoaW5hIiwgInJ1c3NpYSIpKQ0KYGBgDQoNCnQtdGVzdCB0aW1lIQ0KDQpgYGB7cn0NCnQudGVzdCh0b3RhbCB+IGNvdW50cnksIGRhdGEgPSBjaGluYV9ydXNzaWEpDQpgYGANCg0KIyMjIEFOT1ZBDQpBbiBBTk9WQSB3b3VsZCBsZXQgdXMgY29tcGFyZSB0aGUgdG90YWwgc2NvcmVzIG9mIHN0dWRlbnRzIGZyb20gYWxsIDMgY291bnRyaWVzLg0KDQpgYGB7cn0NCmNvdW50cnlfdG90YWxfYW5vdmEgPC0gYW92KHRvdGFsIH4gY291bnRyeSwgZGF0YSA9IGRhdGEpDQoNCmNvdW50cnlfdG90YWxfYW5vdmENCg0Kc3VtbWFyeShjb3VudHJ5X3RvdGFsX2Fub3ZhKQ0KYGBgDQoNClIgY2FuIGRvIG1hbnksIG1hbnkgbW9yZSBzdGF0aXN0aWNhbCBhbmFseXNlcy0gZXZlcnl0aGluZyBmcm9tIENoaS1zcXVhcmUgdGVzdHMgdG8gbWl4ZWQtZWZmZWN0IHJlZ3Jlc3Npb24gdG8gc3RydWN0dXJhbCBlcXVhdGlvbiBtb2RlbGluZyB0byBuZXR3b3JrIGFuYWx5c2lzLiBUaGlzIHdhcyBqdXN0IGEgdGFzdGUgb2YgdGhlIGJhc2ljcyENCg0KIyMgR3JhcGhpY3Mgd2l0aCBSDQpHcmFwaGljcyBtaWdodCBiZSB0aGUgYmlnZ2VzdCByZWFzb24gdG8gc3RhcnQgbGVhcm5pbmcgYW5kIHVzaW5nIFIuIENvbXBhcmVkIHRvIFNQU1Mgb3IgRXhjZWwsIGdyYXBoaWNzIG1hZGUgaW4gUiBnZW5lcmFsbHkgbG9vayBiZXR0ZXIgYW5kIGFyZSBhY3R1YWxseSAqZWFzaWVyKiB0byBjdXN0b21pemUgb25jZSB5b3UgZ2V0IGZhbWlsaWFyIHdpdGggdGhlIGJhc2ljcy4gSSBzdHJvbmdseSByZWNvbW1lbmQgdXNpbmcgdGhlIGdncGxvdCBwYWNrYWdlIChwYXJ0IG9mIHRpZHl2ZXJzZSkgZm9yIGNyZWF0aW5nIGdyYXBoaWNzLg0KDQojIyMgSGlzdG9ncmFtczogYW4gZWFzeSBmaXJzdCBwbG90DQpXZSdsbCBzdGFydCBieSBtYWtpbmcgYSBoaXN0b2dyYW0gb2YgdG90YWwgc2NvcmVzLg0KDQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGF0YSwgYWVzKHggPSB0b3RhbCkpKw0KICBnZW9tX2hpc3RvZ3JhbSgpDQpgYGANCg0KVGhhdCBpcyBhIHZlcnkgYmFzaWMsIGJ1dCB2ZXJ5IHJlYWRhYmxlIGhpc3RvZ3JhbS4gTGV0J3MgY3VzdG9taXplIGl0IGEgYml0IHRvIG1ha2UgaXQgcHJldHRpZXI6DQoNCmBgYHtyfQ0KdG90YWxfaGlzdCA8LSBnZ3Bsb3QoZGF0YSA9IGRhdGEsIGFlcyh4ID0gdG90YWwpKSsNCiAgZ2VvbV9oaXN0b2dyYW0oYmlud2lkdGggPSA1KSsNCiAgbGFicyh4ID0gIlRvdGFsIFNjb3JlIiwgeSA9ICJOdW1iZXIgb2YgVGVzdC1UYWtlcnMiKSsNCiAgdGhlbWVfYncoKQ0KDQp0b3RhbF9oaXN0DQpgYGANCg0KTm93IHRoaXMgbG9va3MgcHJldHR5IG5pY2UhIFdlIGNhbiBzYXZlIGEgaGlnaC1yZXNvbHV0aW9uIHZlcnNpb24gb2YgdGhpcyB0aGF0IHdpbGwgbG9vayBncmVhdCBpbiBhIHBvd2VycG9pbnQgb3IgYSBwYXBlcjoNCg0KYGBge3J9DQpnZ3NhdmUoInRvdGFsX2hpc3RvZ3JhbS5wbmciLCB0b3RhbF9oaXN0LCB3aWR0aCA9IDYsIGhlaWdodCA9IDQsIHVuaXRzID0gImluIiwgZHBpID0gMzAwKQ0KYGBgDQoNCkdvIHRvIHlvdXIgZm9sZGVyIGFuZCBvcGVuIHRoYXQgLSBsb29rcyBwcmV0dHkgZ29vZCwgcmlnaHQ/IFdlIGNhbiBhbHNvIGRvIHNvbWUgcmVhbGx5IG5lYXQgc3R1ZmYgbG9va2luZyBhdCBzdWJncm91cHMgd2l0aCBwbG90cy4gTGV0J3MgdHJ5Og0KDQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGF0YSwgYWVzKHggPSB0b3RhbCkpKw0KICBnZW9tX2hpc3RvZ3JhbShiaW53aWR0aCA9IDUpKw0KICBsYWJzKHggPSAiVG90YWwgU2NvcmUiLCB5ID0gIk51bWJlciBvZiBUZXN0LVRha2VycyIpKw0KICB0aGVtZV9idygpKw0KICBmYWNldF93cmFwKH5jb3VudHJ5LCBucm93ID0gMSkNCmBgYA0KTG9va3MgbGlrZSBSdXNzaWFuIHN0dWRlbnRzIG1pZ2h0IGhhdmUgYSBiaW1vZGFsIGRpc3RyaWJ1dGlvbiEgWW91IGNhbiBzYXZlIHRoaXMgZ3JhcGggaWYgeW91IHdhbnQgLSBsb29rIGF0IHRoZSBwcmV2aW91cyBleGFtcGxlIGZvciBoaW50cy4NCg0KIyMjIEJveHBsb3RzIGFuZCB2YXJpYXRpb25zDQpCb3hwbG90cyBhcmUgbmljZSB3YXlzIG9mIHN1bW1hcml6aW5nIGRpZmZlcmVudCBncm91cHMgb3IgZGlmZmVyZW50IHZhcmlhYmxlcyBncmFwaGljYWxseS4gSGVyZSdzIGEgc2ltcGxlIG9uZToNCg0KYGBge3J9DQpnZ3Bsb3QoZGF0YSA9IGRhdGEsIGFlcyh4ID0gY291bnRyeSwgeSA9IHRvdGFsKSkrDQogIGdlb21fYm94cGxvdCgpKw0KICB0aGVtZV9idygpDQpgYGANClRoYXQncyBva2F5LCBidXQgd2UgY291bGQgbWFrZSBpdCBtb3JlIGluZm9ybWF0aXZlLiBWaW9saW4gcGxvdHMgYXJlIGEgdmFyaWF0aW9uIHRoYXQgc2hvdyB0aGUgZGlzdHJpYnV0aW9ucyBpbiBtb3JlIGRldGFpbC4gV2UnbGwgYWxzbyBhZGQgc29tZSBjb2xvciBqdXN0IHRvIG1ha2UgdGhpbmdzIHBvcCBtb3JlLg0KDQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGF0YSwgYWVzKHggPSBjb3VudHJ5LCB5ID0gdG90YWwsIGZpbGwgPSBjb3VudHJ5KSkrDQogIGdlb21fdmlvbGluKCkrDQogIHRoZW1lX2J3KCkNCmBgYA0KDQpXZSBjYW4gYWxzbyBhZGQgaW5kaXZpZHVhbCBwb2ludHMgdG8gYm94cGxvdHMgKG9yIHZpb2xpbiBwbG90cykNCmBgYHtyfQ0KZ2dwbG90KGRhdGEgPSBkYXRhLCBhZXMoeCA9IGNvdW50cnksIHkgPSB0b3RhbCwgZmlsbCA9IGNvdW50cnkpKSsNCiAgZ2VvbV9ib3hwbG90KCkrDQogIGdlb21fZG90cGxvdChiaW5heGlzPSd5Jywgc3RhY2tkaXI9J2NlbnRlcicsIGRvdHNpemU9MSwgYmlud2lkdGggPSAxLCBmaWxsID0gImJsYWNrIikrDQogIHRoZW1lX2J3KCkNCmBgYA0KTG90cyBvZiBmbGV4aWJpbGl0eSBoZXJlIQ0KDQojIyMgU2NhdHRlcnBsb3RzDQpUaGUgbGFzdCBwbG90IHR5cGUgd2UnbGwgdGFrZSBhIGxvb2sgYXQgdG9kYXkgaXMgdGhlIHNjYXR0ZXJwbG90LiBUaGVzZSBhcmUgb2Z0ZW4gdGhlIG1vc3QgaW5mb3JtYXRpdmUgcGxvdHMsIGluIG15IG9waW5pb24sIGJlY2F1c2UgdGhleSBjYW4gc2hvdyB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gdmFyaWFibGVzIGluIHJlYWxseSBpbnRlcmVzdGluZyB3YXlzLiBMZXQncyBzdGFydCBzaW1wbGU6DQoNCmBgYHtyfQ0KZ2dwbG90KGRhdGEgPSBkYXRhLCBhZXMoeCA9IHJlYWRfdG90YWwsIHkgPSBsaXN0X3RvdGFsKSkrDQogIGdlb21fcG9pbnQoKQ0KYGBgDQpUaGlzIHNob3dzIGEgcHJldHR5IGNsZWFyIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHJlYWRpbmcgYW5kIGxpc3RlbmluZyBzY29yZXMsIGp1c3QgbGlrZSBvdXIgY29ycmVsYXRpb24gZnJvbSBlYXJsaWVyIGluZGljYXRlZC4gSW4gdGhpcyBwbG90LCBpdCBkb2Vzbid0IGxvb2sgbGlrZSB3ZSBoYXZlIHRvbyBtdWNoICoqb3ZlcnBsb3R0aW5nKiogKGkuZS4sIHdoZW4geW91IGhhdmUgcG9pbnRzIHRoYXQgb3ZlcmxhcCBlYWNoIG90aGVyKS4gQnV0IGlmIHdlIGRpZCBoYXZlIHNvbWUgb3ZlcnBsb3R0aW5nIGlzc3Vlcywgd2UgY291bGQgdXNlIGdlb21faml0dGVyIGluc3RlYWQgb2YgZ2VvbV9wb2ludCwgbGlrZSB0aGlzOg0KDQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGF0YSwgYWVzKHggPSByZWFkX3RvdGFsLCB5ID0gbGlzdF90b3RhbCkpKw0KICBnZW9tX2ppdHRlcigpDQpgYGANCkxvb2tpbmcgYXQgdGhpcywgSSB0aGluayB3ZSBzaG91bGQgcHJvYmFibHkgdXNlIGdlb21faml0dGVyLiBnZW9tX2ppdHRlciBhZGRzIGEgbGl0dGxlIGJpdCBvZiByYW5kb20gbm9pc2UgdG8gdGhlIGxvY2F0aW9uIG9mIHBvaW50cywgbnVkZ2luZyB0aGVtIGF3YXkgZnJvbSBlYWNoIG90aGVyIHNvIHlvdSBjYW4gc2VlIHRoZW0gYmV0dGVyLg0KDQpMZXQncyBjbGVhbiB1cCB0aGlzIHBsb3QgYW5kIGFkZCBhIHJlZ3Jlc3Npb24gbGluZSB0byBzaG93IHRoZSByZWxhdGlvbnNoaXAgbW9yZSBjbGVhcmx5Lg0KDQpgYGB7cn0NCmdncGxvdChkYXRhID0gZGF0YSwgYWVzKHggPSByZWFkX3RvdGFsLCB5ID0gbGlzdF90b3RhbCkpKw0KICBnZW9tX2ppdHRlcigpKw0KICBnZW9tX3Ntb290aChtZXRob2QgPSBsbSwgc2UgPSBGQUxTRSkrDQogIGxhYnMoeSA9ICJMaXN0ZW5pbmcgU2NvcmUiLCB4ID0gIlJlYWRpbmcgU2NvcmUiKSsNCiAgdGhlbWVfYncoKQ0KYGBgDQpUaGlzIGlzIGxvb2tpbmcgbmljZSEgV2UgY2FuIGFsc28gYnJlYWsgZG93biB0aGlzIHJlbGF0aW9uc2hpcCBieSBncm91cHMgYWNjb3JkaW5nIHRvIGNvdW50cnk6DQoNCmBgYHtyfQ0KZ2dwbG90KGRhdGEgPSBkYXRhLCBhZXMoeCA9IHJlYWRfdG90YWwsIHkgPSBsaXN0X3RvdGFsLCBncm91cCA9IGNvdW50cnksIGNvbG9yID0gY291bnRyeSkpKw0KICBnZW9tX2ppdHRlcigpKw0KICBnZW9tX3Ntb290aChtZXRob2QgPSBsbSwgc2UgPSBGQUxTRSkrDQogIGxhYnMoeSA9ICJMaXN0ZW5pbmcgU2NvcmUiLCB4ID0gIlJlYWRpbmcgU2NvcmUiKSsNCiAgdGhlbWVfYncoKQ0KYGBgDQoNCg0KSSBlbmNvdXJhZ2UgeW91IHRvIHRyeSB1c2luZyBSIHRvIGNyZWF0ZSBncmFwaGljcyBmb3IgeW91ciBuZXh0IHByZXNlbnRhdGlvbiwgcGFwZXIsIG9yIHRoZXNpcywgZXZlbiBpZiB5b3Ugc3RpbGwgd2FudCB0byB1c2UgU1BTUyBmb3IgeW91ciBzdGF0cy4=