Introduction

Welcome to your first LASER badge! This LASER Orientation Badge is really a warm-up activitiy to introduce you to R Markdown and the coding case studies that we will be using in the machine learning, network analysis, and text mining labs. It is a chance to become familiar with how RStudio and R Markdown works.

You may have used R before-or you may not have! Either is fine as this task will be designed with the assumption that you have not used R before. It includes “reaches” for anyone who may want to do a bit more.

In the context of doing so, we’ll focus on the following tasks:

  1. Reading data into R (in the Prepare section)
  2. Preparing and “wrangling” data in table (think spreadsheet!) format (in the Wrangle section)
  3. Creating some plots (in the Explore section)
  4. Running a model - specifically, a regression model (in the Model section)
  5. Finally, creating a reproducible report of your work you can share with others (in the Communicate section)

The LASER Cycle

You may be wondering what these bolded terms above refer to; what’s so special about preparing, wrangling, exploring, and modeling data - and communicating results? We’re using these terms as a part of a framework, or model, for what we mean by doing learning in STEM education research.

The particular framework we are using comes from the work of Krumm et al.’s Learning Analytics Goes to School. You can check that out, but don’t feel any need to dive deep for now - we’ll be spending more time on this in first day of the summer institute. For now, know that this document is organized around three of the five components of what we’re referring to as the LASER cycle.

Click the green arrow to the right of the “code chunk” below to view the image (more on that process of clicking the green arrow and what it does, too, in a moment)!

knitr::include_graphics("img/laser-cycle.png")

How to use this R Markdown document

This is an R Markdown file as indicated by the .rmd extension at the end of the file name. R Markdown documents are fully reproducible and use a productive notebook interface to combine narrative text and “chunks” of code to produce a range of formatted outputs including: formats including HTML, PDF, MS Word, Beamer, HTML5 slides, Tufte-style handouts, books, dashboards, shiny applications, scientific articles, websites, and more.

There are two keys to your use of R Markdown for this activity:

  1. First, be sure that you are viewing the document in the “Visual Editor” mode. You can use this mode by clicking the word “Visual” on the left side of the toolbar above.
  2. Second, click “Knit” next to the yarn ball at the top of this screen to preview the document as you work through it. This will allow you to see your code and the input in a rendered - easy-to-read - document, just as others will see this document when shared. Try knitting the document now and see what happens.

Let’s get started! We are glad you are here and to begin this exciting (and challenging) journey together.

1. PREPARE

By preparing, we refer to developing a question or purpose for the analysis, which you likely know from your research can be difficult! This part of the process also involves developing an understanding of the data and what you may need to analyze the data. This often involves looking at the data and its documentation. For now, we’ll focus on just a few parts of this process, diving in much more deeply over the coming weeks.

Packages 📦

R uses “packages,” add-ons that enhance its functionality. One package that we’ll be using is the tidyverse. To load the tidyverse, click the green arrow in the right corner of the block-or “chunk”-of code that follows.

library(tidyverse)
## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──
## ✓ ggplot2 3.3.3     ✓ purrr   0.3.4
## ✓ tibble  3.1.2     ✓ dplyr   1.0.6
## ✓ tidyr   1.1.3     ✓ stringr 1.4.0
## ✓ readr   1.4.0     ✓ forcats 0.5.1
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

Please do not worry if you saw a number of messages: those probably mean that the tidyverse loaded just fine. If you see an error, though, try to interpret or search via your search engine the contents of the error, or reach out to us for assistance.

Loading (or reading in) data

Next, we’ll load data-specifically, a CSV file, the kind that you can export from Microsoft Excel or Google Sheets - into R, using the read_csv() function in the next chunk.

Clicking the green arrow runs the code; do that next to read the sci-online-classes.csv file stored in your data folder into your R environment:

d <- read_csv("data/sci-online-classes.csv")
## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   .default = col_double(),
##   course_id = col_character(),
##   subject = col_character(),
##   semester = col_character(),
##   section = col_character(),
##   Gradebook_Item = col_character(),
##   Grade_Category = col_logical(),
##   Gender = col_character()
## )
## ℹ Use `spec()` for the full column specifications.

Nice work! You should now see a new data “object” named d saved in your Environment pane. Try clicking on it and see what happens.

Viewing or inspecting data

Now let’s learn another way to inspect our data. Run the next chunk and look at the results, tabbing left or right with the arrows, or scanning through the rows by clicking the numbers at the bottom of the pane with the print-out of the data you loaded:

d
## # A tibble: 603 x 30
##    student_id course_id   total_points_possi… total_points_ear… percentage_earn…
##         <dbl> <chr>                     <dbl>             <dbl>            <dbl>
##  1      43146 FrScA-S216…                3280              2220            0.677
##  2      44638 OcnA-S116-…                3531              2672            0.757
##  3      47448 FrScA-S216…                2870              1897            0.661
##  4      47979 OcnA-S216-…                4562              3090            0.677
##  5      48797 PhysA-S116…                2207              1910            0.865
##  6      51943 FrScA-S216…                4208              3596            0.855
##  7      52326 AnPhA-S216…                4325              2255            0.521
##  8      52446 PhysA-S116…                2086              1719            0.824
##  9      53447 FrScA-S116…                4655              3149            0.676
## 10      53475 FrScA-S116…                1710              1402            0.820
## # … with 593 more rows, and 25 more variables: subject <chr>, semester <chr>,
## #   section <chr>, Gradebook_Item <chr>, Grade_Category <lgl>,
## #   FinalGradeCEMS <dbl>, Points_Possible <dbl>, Points_Earned <dbl>,
## #   Gender <chr>, q1 <dbl>, q2 <dbl>, q3 <dbl>, q4 <dbl>, q5 <dbl>, q6 <dbl>,
## #   q7 <dbl>, q8 <dbl>, q9 <dbl>, q10 <dbl>, TimeSpent <dbl>,
## #   TimeSpent_hours <dbl>, TimeSpent_std <dbl>, int <dbl>, pc <dbl>, uv <dbl>
str(d)
## spec_tbl_df [603 × 30] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ student_id           : num [1:603] 43146 44638 47448 47979 48797 ...
##  $ course_id            : chr [1:603] "FrScA-S216-02" "OcnA-S116-01" "FrScA-S216-01" "OcnA-S216-01" ...
##  $ total_points_possible: num [1:603] 3280 3531 2870 4562 2207 ...
##  $ total_points_earned  : num [1:603] 2220 2672 1897 3090 1910 ...
##  $ percentage_earned    : num [1:603] 0.677 0.757 0.661 0.677 0.865 ...
##  $ subject              : chr [1:603] "FrScA" "OcnA" "FrScA" "OcnA" ...
##  $ semester             : chr [1:603] "S216" "S116" "S216" "S216" ...
##  $ section              : chr [1:603] "02" "01" "01" "01" ...
##  $ Gradebook_Item       : chr [1:603] "POINTS EARNED & TOTAL COURSE POINTS" "ATTEMPTED" "POINTS EARNED & TOTAL COURSE POINTS" "POINTS EARNED & TOTAL COURSE POINTS" ...
##  $ Grade_Category       : logi [1:603] NA NA NA NA NA NA ...
##  $ FinalGradeCEMS       : num [1:603] 93.5 81.7 88.5 81.9 84 ...
##  $ Points_Possible      : num [1:603] 5 10 10 5 438 5 10 10 443 5 ...
##  $ Points_Earned        : num [1:603] NA 10 NA 4 399 NA NA 10 425 2.5 ...
##  $ Gender               : chr [1:603] "M" "F" "M" "M" ...
##  $ q1                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q2                   : num [1:603] 4 4 4 5 3 NA 5 3 3 NA ...
##  $ q3                   : num [1:603] 4 3 4 3 3 NA 3 3 3 NA ...
##  $ q4                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q5                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q6                   : num [1:603] 5 4 4 5 4 NA 5 4 3 NA ...
##  $ q7                   : num [1:603] 5 4 4 4 4 NA 4 3 3 NA ...
##  $ q8                   : num [1:603] 5 5 5 5 4 NA 5 3 4 NA ...
##  $ q9                   : num [1:603] 4 4 3 5 NA NA 5 3 2 NA ...
##  $ q10                  : num [1:603] 5 4 5 5 3 NA 5 3 5 NA ...
##  $ TimeSpent            : num [1:603] 1555 1383 860 1599 1482 ...
##  $ TimeSpent_hours      : num [1:603] 25.9 23 14.3 26.6 24.7 ...
##  $ TimeSpent_std        : num [1:603] -0.181 -0.308 -0.693 -0.148 -0.235 ...
##  $ int                  : num [1:603] 5 4.2 5 5 3.8 4.6 5 3 4.2 NA ...
##  $ pc                   : num [1:603] 4.5 3.5 4 3.5 3.5 4 3.5 3 3 NA ...
##  $ uv                   : num [1:603] 4.33 4 3.67 5 3.5 ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   student_id = col_double(),
##   ..   course_id = col_character(),
##   ..   total_points_possible = col_double(),
##   ..   total_points_earned = col_double(),
##   ..   percentage_earned = col_double(),
##   ..   subject = col_character(),
##   ..   semester = col_character(),
##   ..   section = col_character(),
##   ..   Gradebook_Item = col_character(),
##   ..   Grade_Category = col_logical(),
##   ..   FinalGradeCEMS = col_double(),
##   ..   Points_Possible = col_double(),
##   ..   Points_Earned = col_double(),
##   ..   Gender = col_character(),
##   ..   q1 = col_double(),
##   ..   q2 = col_double(),
##   ..   q3 = col_double(),
##   ..   q4 = col_double(),
##   ..   q5 = col_double(),
##   ..   q6 = col_double(),
##   ..   q7 = col_double(),
##   ..   q8 = col_double(),
##   ..   q9 = col_double(),
##   ..   q10 = col_double(),
##   ..   TimeSpent = col_double(),
##   ..   TimeSpent_hours = col_double(),
##   ..   TimeSpent_std = col_double(),
##   ..   int = col_double(),
##   ..   pc = col_double(),
##   ..   uv = col_double()
##   .. )

Your Turn ⤵

What do you notice about this data set? What do you wonder? Add one-two thoughts following the dashes next (you can add additional dashes if you like!):

  • The dataframe appears to essentially be a gradebook for science course(s) with some additional variables on student behavior, like the number of hours spent (studying, presumably?)
  • What is CEMS

There are other ways to inspect your data; the glimpse() function provides one such way. Run the code below to take a glimpse at your data.

glimpse(d)
## Rows: 603
## Columns: 30
## $ student_id            <dbl> 43146, 44638, 47448, 47979, 48797, 51943, 52326,…
## $ course_id             <chr> "FrScA-S216-02", "OcnA-S116-01", "FrScA-S216-01"…
## $ total_points_possible <dbl> 3280, 3531, 2870, 4562, 2207, 4208, 4325, 2086, …
## $ total_points_earned   <dbl> 2220, 2672, 1897, 3090, 1910, 3596, 2255, 1719, …
## $ percentage_earned     <dbl> 0.6768293, 0.7567261, 0.6609756, 0.6773345, 0.86…
## $ subject               <chr> "FrScA", "OcnA", "FrScA", "OcnA", "PhysA", "FrSc…
## $ semester              <chr> "S216", "S116", "S216", "S216", "S116", "S216", …
## $ section               <chr> "02", "01", "01", "01", "01", "03", "01", "01", …
## $ Gradebook_Item        <chr> "POINTS EARNED & TOTAL COURSE POINTS", "ATTEMPTE…
## $ Grade_Category        <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
## $ FinalGradeCEMS        <dbl> 93.45372, 81.70184, 88.48758, 81.85260, 84.00000…
## $ Points_Possible       <dbl> 5, 10, 10, 5, 438, 5, 10, 10, 443, 5, 12, 10, 5,…
## $ Points_Earned         <dbl> NA, 10.00, NA, 4.00, 399.00, NA, NA, 10.00, 425.…
## $ Gender                <chr> "M", "F", "M", "M", "F", "F", "M", "F", "F", "M"…
## $ q1                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 4, 3, 5, NA,…
## $ q2                    <dbl> 4, 4, 4, 5, 3, NA, 5, 3, 3, NA, NA, 5, 3, 3, NA,…
## $ q3                    <dbl> 4, 3, 4, 3, 3, NA, 3, 3, 3, NA, NA, 3, 3, 5, NA,…
## $ q4                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 5, 3, 5, NA,…
## $ q5                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 5, 4, 5, NA,…
## $ q6                    <dbl> 5, 4, 4, 5, 4, NA, 5, 4, 3, NA, NA, 5, 3, 5, NA,…
## $ q7                    <dbl> 5, 4, 4, 4, 4, NA, 4, 3, 3, NA, NA, 5, 3, 5, NA,…
## $ q8                    <dbl> 5, 5, 5, 5, 4, NA, 5, 3, 4, NA, NA, 4, 3, 5, NA,…
## $ q9                    <dbl> 4, 4, 3, 5, NA, NA, 5, 3, 2, NA, NA, 5, 2, 2, NA…
## $ q10                   <dbl> 5, 4, 5, 5, 3, NA, 5, 3, 5, NA, NA, 4, 4, 5, NA,…
## $ TimeSpent             <dbl> 1555.1667, 1382.7001, 860.4335, 1598.6166, 1481.…
## $ TimeSpent_hours       <dbl> 25.91944500, 23.04500167, 14.34055833, 26.643610…
## $ TimeSpent_std         <dbl> -0.18051496, -0.30780313, -0.69325954, -0.148446…
## $ int                   <dbl> 5.0, 4.2, 5.0, 5.0, 3.8, 4.6, 5.0, 3.0, 4.2, NA,…
## $ pc                    <dbl> 4.50, 3.50, 4.00, 3.50, 3.50, 4.00, 3.50, 3.00, …
## $ uv                    <dbl> 4.333333, 4.000000, 3.666667, 5.000000, 3.500000…

We have one more question to pose to you: What do rows and columns typically represent in your area of work and/or research?

Generally, rows typically represent “cases,” the units that we measure, or the units on which we collect data. This is not a trick question! What counts as a “case” (and therefore what is represented as a row) varies by (and within) fields. There may be multiple types or levels of units studied in your field; listing more than one is fine! Also, please consider what columns - which usually represent variables - represent in your area of work and/or research.

Your Turn ⤵

What rows typically (or you think may) represent:

  • I believe the rows here refer to students (so it’s a wide, rather than a long/tidy dataset).

What columns typically (or you think may) represent:

  • Outcomes or variables related to the student (e.g., their gender, their course, their score)

Next, we’ll use a few functions that are handy for preparing data in table form.

2. WRANGLE

By wrangle, we refer to the process of cleaning and processing data, and, in cases, merging (or joining) data from multiple sources. Often, this part of the process is very (surprisingly) time-intensive. Wrangling your data into shape can itself be an important accomplishment! There are great tools in R to do this, especially through the use of the {dplyr} R package.

Selecting variables

Let’s select only a few variables.

library(dplyr)
d %>%
  select(student_id, total_points_possible, total_points_earned)
## # A tibble: 603 x 3
##    student_id total_points_possible total_points_earned
##         <dbl>                 <dbl>               <dbl>
##  1      43146                  3280                2220
##  2      44638                  3531                2672
##  3      47448                  2870                1897
##  4      47979                  4562                3090
##  5      48797                  2207                1910
##  6      51943                  4208                3596
##  7      52326                  4325                2255
##  8      52446                  2086                1719
##  9      53447                  4655                3149
## 10      53475                  1710                1402
## # … with 593 more rows

Notice how the number of columns (variables) is now different.

Let’s include one additional variable in your select function.

First, we need to figure out what variables exist in our dataset (or be reminded of this - it’s very common in R to be continually checking and inspecting your data)!

You can use a function named glimpse() to do this.

glimpse(d)
## Rows: 603
## Columns: 30
## $ student_id            <dbl> 43146, 44638, 47448, 47979, 48797, 51943, 52326,…
## $ course_id             <chr> "FrScA-S216-02", "OcnA-S116-01", "FrScA-S216-01"…
## $ total_points_possible <dbl> 3280, 3531, 2870, 4562, 2207, 4208, 4325, 2086, …
## $ total_points_earned   <dbl> 2220, 2672, 1897, 3090, 1910, 3596, 2255, 1719, …
## $ percentage_earned     <dbl> 0.6768293, 0.7567261, 0.6609756, 0.6773345, 0.86…
## $ subject               <chr> "FrScA", "OcnA", "FrScA", "OcnA", "PhysA", "FrSc…
## $ semester              <chr> "S216", "S116", "S216", "S216", "S116", "S216", …
## $ section               <chr> "02", "01", "01", "01", "01", "03", "01", "01", …
## $ Gradebook_Item        <chr> "POINTS EARNED & TOTAL COURSE POINTS", "ATTEMPTE…
## $ Grade_Category        <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
## $ FinalGradeCEMS        <dbl> 93.45372, 81.70184, 88.48758, 81.85260, 84.00000…
## $ Points_Possible       <dbl> 5, 10, 10, 5, 438, 5, 10, 10, 443, 5, 12, 10, 5,…
## $ Points_Earned         <dbl> NA, 10.00, NA, 4.00, 399.00, NA, NA, 10.00, 425.…
## $ Gender                <chr> "M", "F", "M", "M", "F", "F", "M", "F", "F", "M"…
## $ q1                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 4, 3, 5, NA,…
## $ q2                    <dbl> 4, 4, 4, 5, 3, NA, 5, 3, 3, NA, NA, 5, 3, 3, NA,…
## $ q3                    <dbl> 4, 3, 4, 3, 3, NA, 3, 3, 3, NA, NA, 3, 3, 5, NA,…
## $ q4                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 5, 3, 5, NA,…
## $ q5                    <dbl> 5, 4, 5, 5, 4, NA, 5, 3, 4, NA, NA, 5, 4, 5, NA,…
## $ q6                    <dbl> 5, 4, 4, 5, 4, NA, 5, 4, 3, NA, NA, 5, 3, 5, NA,…
## $ q7                    <dbl> 5, 4, 4, 4, 4, NA, 4, 3, 3, NA, NA, 5, 3, 5, NA,…
## $ q8                    <dbl> 5, 5, 5, 5, 4, NA, 5, 3, 4, NA, NA, 4, 3, 5, NA,…
## $ q9                    <dbl> 4, 4, 3, 5, NA, NA, 5, 3, 2, NA, NA, 5, 2, 2, NA…
## $ q10                   <dbl> 5, 4, 5, 5, 3, NA, 5, 3, 5, NA, NA, 4, 4, 5, NA,…
## $ TimeSpent             <dbl> 1555.1667, 1382.7001, 860.4335, 1598.6166, 1481.…
## $ TimeSpent_hours       <dbl> 25.91944500, 23.04500167, 14.34055833, 26.643610…
## $ TimeSpent_std         <dbl> -0.18051496, -0.30780313, -0.69325954, -0.148446…
## $ int                   <dbl> 5.0, 4.2, 5.0, 5.0, 3.8, 4.6, 5.0, 3.0, 4.2, NA,…
## $ pc                    <dbl> 4.50, 3.50, 4.00, 3.50, 3.50, 4.00, 3.50, 3.00, …
## $ uv                    <dbl> 4.333333, 4.000000, 3.666667, 5.000000, 3.500000…

Your Turn ⤵

In the code chunk below, add a new variable to the code below, being careful to type the new variable name as it appears in the data. We’ve added some code to get you started. Consider how the names of the other variables are separated as you think about how to add an additional variable to this code.

d %>%
  select(student_id, total_points_possible, total_points_earned, pc)
## # A tibble: 603 x 4
##    student_id total_points_possible total_points_earned    pc
##         <dbl>                 <dbl>               <dbl> <dbl>
##  1      43146                  3280                2220   4.5
##  2      44638                  3531                2672   3.5
##  3      47448                  2870                1897   4  
##  4      47979                  4562                3090   3.5
##  5      48797                  2207                1910   3.5
##  6      51943                  4208                3596   4  
##  7      52326                  4325                2255   3.5
##  8      52446                  2086                1719   3  
##  9      53447                  4655                3149   3  
## 10      53475                  1710                1402  NA  
## # … with 593 more rows

Once added, the output should be different than in the code above - there should now be an additional variable included in the print-out.

Filtering variables

Next, let’s explore filtering variables. Check out and run the next chunk of code, imagining that we wish to filter our data to view only the rows associated with students who earned a final grade (as a percentage) of 70 - 70% - or higher.

d %>%
  filter(FinalGradeCEMS > 70)
## # A tibble: 438 x 30
##    student_id course_id   total_points_possi… total_points_ear… percentage_earn…
##         <dbl> <chr>                     <dbl>             <dbl>            <dbl>
##  1      43146 FrScA-S216…                3280              2220            0.677
##  2      44638 OcnA-S116-…                3531              2672            0.757
##  3      47448 FrScA-S216…                2870              1897            0.661
##  4      47979 OcnA-S216-…                4562              3090            0.677
##  5      48797 PhysA-S116…                2207              1910            0.865
##  6      52326 AnPhA-S216…                4325              2255            0.521
##  7      52446 PhysA-S116…                2086              1719            0.824
##  8      53447 FrScA-S116…                4655              3149            0.676
##  9      53475 FrScA-S216…                1209               977            0.808
## 10      54066 OcnA-S116-…                4641              3429            0.739
## # … with 428 more rows, and 25 more variables: subject <chr>, semester <chr>,
## #   section <chr>, Gradebook_Item <chr>, Grade_Category <lgl>,
## #   FinalGradeCEMS <dbl>, Points_Possible <dbl>, Points_Earned <dbl>,
## #   Gender <chr>, q1 <dbl>, q2 <dbl>, q3 <dbl>, q4 <dbl>, q5 <dbl>, q6 <dbl>,
## #   q7 <dbl>, q8 <dbl>, q9 <dbl>, q10 <dbl>, TimeSpent <dbl>,
## #   TimeSpent_hours <dbl>, TimeSpent_std <dbl>, int <dbl>, pc <dbl>, uv <dbl>
Your Turn ⤵

In the next code chunk, change the cut-off from 70% to some other value - larger or smaller (maybe much larger or smaller - feel free to play around with the code a bit!).

d %>%
  filter(FinalGradeCEMS <= 20)
## # A tibble: 30 x 30
##    student_id course_id   total_points_possi… total_points_ear… percentage_earn…
##         <dbl> <chr>                     <dbl>             <dbl>            <dbl>
##  1      71415 FrScA-S216…                3284              2574            0.784
##  2      85323 FrScA-S116…                4000              3136            0.784
##  3      85355 OcnA-S116-…                3173              2687            0.847
##  4      85390 PhysA-S116…                5246              4318            0.823
##  5      85889 BioA-S116-…                3838              2248            0.586
##  6      86588 AnPhA-S116…                4024              2812            0.699
##  7      87081 BioA-S116-…                4254              3315            0.779
##  8      87122 OcnA-S116-…                9066              5414            0.597
##  9      88874 FrScA-S116…                3370              2770            0.822
## 10      90995 AnPhA-S116…                3516              2878            0.819
## # … with 20 more rows, and 25 more variables: subject <chr>, semester <chr>,
## #   section <chr>, Gradebook_Item <chr>, Grade_Category <lgl>,
## #   FinalGradeCEMS <dbl>, Points_Possible <dbl>, Points_Earned <dbl>,
## #   Gender <chr>, q1 <dbl>, q2 <dbl>, q3 <dbl>, q4 <dbl>, q5 <dbl>, q6 <dbl>,
## #   q7 <dbl>, q8 <dbl>, q9 <dbl>, q10 <dbl>, TimeSpent <dbl>,
## #   TimeSpent_hours <dbl>, TimeSpent_std <dbl>, int <dbl>, pc <dbl>, uv <dbl>

What happens when you change the cut-off from 70 to something else? Add a thought (or more):

  • There are far fewer observations in the resulting dataframe

Arrange

The last function we’ll use for preparing tables is arrange.

We’ll combine this arrange() function with a function we used already - select(). We do this so we can view only the student ID and their final grade.

d %>%
  select(student_id, FinalGradeCEMS) %>% 
  arrange(FinalGradeCEMS)
## # A tibble: 603 x 2
##    student_id FinalGradeCEMS
##         <dbl>          <dbl>
##  1      90995          0    
##  2      92606          0.535
##  3      95684          0.903
##  4      90996          1.80 
##  5      94876          2.93 
##  6      92633          3.01 
##  7      85390          3.06 
##  8      94630          3.43 
##  9      90995          5.04 
## 10      96677          5.2  
## # … with 593 more rows

Note that arrange works by sorting values in ascending order (from lowest to highest); you can change this by using the desc() function with arrange, like the following:

d %>%
  select(student_id, FinalGradeCEMS) %>%
  arrange(desc(FinalGradeCEMS))
## # A tibble: 603 x 2
##    student_id FinalGradeCEMS
##         <dbl>          <dbl>
##  1      85650          100  
##  2      91067           99.8
##  3      66740           99.3
##  4      86792           99.1
##  5      78153           99.0
##  6      66689           98.6
##  7      88261           98.6
##  8      92740           98.6
##  9      92726           98.2
## 10      92741           98.2
## # … with 593 more rows

Your Turn ⤵

In the code chunk below, replace FinalGradeCEMS that is used with both the select() and arrange() functions with a different variable in the data set. Consider returning to the code chunk above in which you glimpsed at the names of all of the variables.

d %>%
  select(student_id, pc) %>%
  arrange(desc(pc))
## # A tibble: 603 x 2
##    student_id    pc
##         <dbl> <dbl>
##  1      58168     5
##  2      60186     5
##  3      70068     5
##  4      78000     5
##  5      84645     5
##  6      85482     5
##  7      85702     5
##  8      86002     5
##  9      86251     5
## 10      86365     5
## # … with 593 more rows

Reach 1 🎉

Can you compose a series of functions that include the select(), filter(), and arrange functions? Recall that you can “pipe” the output from one function to the next as when we used select() and arrange() together in the code chunk above.

This reach is not required/necessary to complete; it’s just for those who wish to do a bit more with these functions at this time (we’ll do more in class, too!)

...
## Error in eval(expr, envir, enclos): '...' used in an incorrect context

3. EXPLORE

Exploratory data analysis, or exploring your data, involves processes of describing your data (such as by calculating the means and standard deviations of numeric variables, or counting the frequency of categorical variables) and, often, visualizing your data prior. In this section, we’ll create a few plots to explore our data.

Histogram

The code below creates a histogram, or a distribution of the values, in this case for students’ final grades.

ggplot(d, aes(x = FinalGradeCEMS)) +
  geom_histogram()
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## Warning: Removed 30 rows containing non-finite values (stat_bin).

You can change the color of the histogram bars by specifying a color as follows:

ggplot(d, aes(x = FinalGradeCEMS)) +
  geom_histogram(fill = "blue")
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## Warning: Removed 30 rows containing non-finite values (stat_bin).

Changing colors

Your Turn ⤵

In the code chunk below, change the color to one of your choosing; consider this list of valid color names here: http://www.stat.columbia.edu/~tzheng/files/Rcolor.pdf

ggplot(d, aes(x = FinalGradeCEMS)) +
  geom_histogram(fill = "dodgerblue")
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## Warning: Removed 30 rows containing non-finite values (stat_bin).

Finally, we’ll make one more change; visualize the distribution of another variable in the data - one other than FinalGradeCEMS. You can do so by swapping out the name for another variable with FinalGradeCEMS. Also, change the color to one other than blue.

ggplot(d, aes(x = pc)) +
  geom_histogram(fill = "green")
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## Warning: Removed 75 rows containing non-finite values (stat_bin).

Reach 2 🎉

Completed the above? Nice job! Try for a “reach” by creating a scatter plot for the relationship between two variables. You will need to pass the names of two variables to the code below for what is now simply XXX (a placeholder).

str(d)
## spec_tbl_df [603 × 30] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ student_id           : num [1:603] 43146 44638 47448 47979 48797 ...
##  $ course_id            : chr [1:603] "FrScA-S216-02" "OcnA-S116-01" "FrScA-S216-01" "OcnA-S216-01" ...
##  $ total_points_possible: num [1:603] 3280 3531 2870 4562 2207 ...
##  $ total_points_earned  : num [1:603] 2220 2672 1897 3090 1910 ...
##  $ percentage_earned    : num [1:603] 0.677 0.757 0.661 0.677 0.865 ...
##  $ subject              : chr [1:603] "FrScA" "OcnA" "FrScA" "OcnA" ...
##  $ semester             : chr [1:603] "S216" "S116" "S216" "S216" ...
##  $ section              : chr [1:603] "02" "01" "01" "01" ...
##  $ Gradebook_Item       : chr [1:603] "POINTS EARNED & TOTAL COURSE POINTS" "ATTEMPTED" "POINTS EARNED & TOTAL COURSE POINTS" "POINTS EARNED & TOTAL COURSE POINTS" ...
##  $ Grade_Category       : logi [1:603] NA NA NA NA NA NA ...
##  $ FinalGradeCEMS       : num [1:603] 93.5 81.7 88.5 81.9 84 ...
##  $ Points_Possible      : num [1:603] 5 10 10 5 438 5 10 10 443 5 ...
##  $ Points_Earned        : num [1:603] NA 10 NA 4 399 NA NA 10 425 2.5 ...
##  $ Gender               : chr [1:603] "M" "F" "M" "M" ...
##  $ q1                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q2                   : num [1:603] 4 4 4 5 3 NA 5 3 3 NA ...
##  $ q3                   : num [1:603] 4 3 4 3 3 NA 3 3 3 NA ...
##  $ q4                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q5                   : num [1:603] 5 4 5 5 4 NA 5 3 4 NA ...
##  $ q6                   : num [1:603] 5 4 4 5 4 NA 5 4 3 NA ...
##  $ q7                   : num [1:603] 5 4 4 4 4 NA 4 3 3 NA ...
##  $ q8                   : num [1:603] 5 5 5 5 4 NA 5 3 4 NA ...
##  $ q9                   : num [1:603] 4 4 3 5 NA NA 5 3 2 NA ...
##  $ q10                  : num [1:603] 5 4 5 5 3 NA 5 3 5 NA ...
##  $ TimeSpent            : num [1:603] 1555 1383 860 1599 1482 ...
##  $ TimeSpent_hours      : num [1:603] 25.9 23 14.3 26.6 24.7 ...
##  $ TimeSpent_std        : num [1:603] -0.181 -0.308 -0.693 -0.148 -0.235 ...
##  $ int                  : num [1:603] 5 4.2 5 5 3.8 4.6 5 3 4.2 NA ...
##  $ pc                   : num [1:603] 4.5 3.5 4 3.5 3.5 4 3.5 3 3 NA ...
##  $ uv                   : num [1:603] 4.33 4 3.67 5 3.5 ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   student_id = col_double(),
##   ..   course_id = col_character(),
##   ..   total_points_possible = col_double(),
##   ..   total_points_earned = col_double(),
##   ..   percentage_earned = col_double(),
##   ..   subject = col_character(),
##   ..   semester = col_character(),
##   ..   section = col_character(),
##   ..   Gradebook_Item = col_character(),
##   ..   Grade_Category = col_logical(),
##   ..   FinalGradeCEMS = col_double(),
##   ..   Points_Possible = col_double(),
##   ..   Points_Earned = col_double(),
##   ..   Gender = col_character(),
##   ..   q1 = col_double(),
##   ..   q2 = col_double(),
##   ..   q3 = col_double(),
##   ..   q4 = col_double(),
##   ..   q5 = col_double(),
##   ..   q6 = col_double(),
##   ..   q7 = col_double(),
##   ..   q8 = col_double(),
##   ..   q9 = col_double(),
##   ..   q10 = col_double(),
##   ..   TimeSpent = col_double(),
##   ..   TimeSpent_hours = col_double(),
##   ..   TimeSpent_std = col_double(),
##   ..   int = col_double(),
##   ..   pc = col_double(),
##   ..   uv = col_double()
##   .. )
ggplot(d, aes(x = pc, y = total_points_earned)) +
  geom_point()
## Warning: Removed 75 rows containing missing values (geom_point).

4. MODEL

“Model” is one of those terms that has many different meanings. For our purpose, we refer to the process of simplifying and summarizing our data. Thus, models can take many forms; calculating means represents a legitimate form of modeling data, as does estimating more complex models, including linear regressions, and models and algorithms associated with machine learning tasks. For now, we’ll run a linear regression to predict students’ final grades.

Below, we predict students’ final grades (FinaGradeCEMS, which is on a 0-100 point scale) on the basis of the time they spent on the course (measured through their learning management system in minutes, TimeSpent, and the subject (one of five) of their specific course.

m1 <- lm(FinalGradeCEMS ~ TimeSpent + subject, data = d)
summary(m1)
## 
## Call:
## lm(formula = FinalGradeCEMS ~ TimeSpent + subject, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -70.378  -8.836   4.816  12.855  36.047 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  57.3931739  2.3382193  24.546  < 2e-16 ***
## TimeSpent     0.0071098  0.0006516  10.912  < 2e-16 ***
## subjectBioA  -1.5596482  3.6053075  -0.433    0.665    
## subjectFrScA 11.7306546  2.2143847   5.297 1.68e-07 ***
## subjectOcnA   1.0974545  2.5771474   0.426    0.670    
## subjectPhysA 16.0357213  3.0712923   5.221 2.50e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 19.8 on 567 degrees of freedom
##   (30 observations deleted due to missingness)
## Multiple R-squared:  0.213,  Adjusted R-squared:  0.2061 
## F-statistic: 30.69 on 5 and 567 DF,  p-value: < 2.2e-16

Your Turn ⤵

Notice how above the variables are separated by a + symbol. Below, add another - a third - variable to the regression model. Specifically, add a variable students’ initial, self-reported interest in science, int - and any other variable(s) you like! What do you notice about the results? We’re going to dive into this much more: if you have many questions now, you’re in the right spot!

m2 <- lm(FinalGradeCEMS ~ TimeSpent + subject + pc, data = d)
summary(m2)
## 
## Call:
## lm(formula = FinalGradeCEMS ~ TimeSpent + subject + pc, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -67.789  -9.282   4.481  13.214  42.651 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  66.8806101  5.7837037  11.564  < 2e-16 ***
## TimeSpent     0.0072313  0.0006955  10.398  < 2e-16 ***
## subjectBioA  -1.3543995  3.9057303  -0.347   0.7289    
## subjectFrScA 13.5446736  2.3958673   5.653 2.66e-08 ***
## subjectOcnA   3.8936648  2.7337716   1.424   0.1550    
## subjectPhysA 17.9305330  3.2420930   5.531 5.17e-08 ***
## pc           -3.2286969  1.3876604  -2.327   0.0204 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 19.82 on 497 degrees of freedom
##   (99 observations deleted due to missingness)
## Multiple R-squared:  0.2368, Adjusted R-squared:  0.2276 
## F-statistic:  25.7 on 6 and 497 DF,  p-value: < 2.2e-16

5. COMMUNICATE

Great job! Once you’ve finished your work, Upon doing so, you should see a new laser-orientation-badge.html.

Congratulations, you’ve completed your Models & Inference Badge! Complete the following steps to submit your work for review by

  1. Change the name of the author: in the YAML header at the very top of this document to your name. As noted in Reproducible Research in R, The YAML header controls the style and feel for knitted document but doesn’t actually display in the final output.

  2. Click the yarn icon above to “knit” your data product to a HTML file that will be saved in your R Project folder.

  3. Commit your changes in GitHub Desktop and push them to your online GitHub repository.

  4. Publish your HTML page the web using one of the following publishing methods:

    • Publish on RPubs by clicking the “Publish” button located in the Viewer Pane when you knit your document. Note, you will need to quickly create a RPubs account.

    • Publishing on GitHub using either GitHub Pages or the HTML previewer.

  5. Post a new discussion on GitHub to our Foundations Badges forum. In your post, include a link to your published web page and a short reflection highlighting one thing you learned from this lab and one thing you’d like to explore further.

LS0tCnRpdGxlOiAiTEFTRVIgQmFkZ2UiCnN1YnRpdGxlOiAiTEFTRVIgSW5zdGl0dXRlIE9yaWVudGF0aW9uIgphdXRob3I6ICJZT1VSIE5BTUUgSEVSRSIKZGF0ZTogImByIGZvcm1hdChTeXMuRGF0ZSgpLCclQiAlZSwgJVknKWAiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB5ZXMKICAgIHRvY19kZXB0aDogNAogICAgdG9jX2Zsb2F0OiB5ZXMKICAgIGNvZGVfZm9sZGluZzogc2hvdwogICAgY29kZV9kb3dubG9hZDogVFJVRQplZGl0b3Jfb3B0aW9uczoKICBtYXJrZG93bjoKICAgIHdyYXA6IDcyCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwgZXJyb3IgPSBUUlVFKQpgYGAKCiFbXShpbWcvTEFTRVJfSHgucG5nKXt3aWR0aD0iNDAlIn0KCiMjIEludHJvZHVjdGlvbgoKV2VsY29tZSB0byB5b3VyIGZpcnN0IExBU0VSIGJhZGdlISBUaGlzIExBU0VSIE9yaWVudGF0aW9uIEJhZGdlIGlzCnJlYWxseSBhIHdhcm0tdXAgYWN0aXZpdGl5IHRvIGludHJvZHVjZSB5b3UgdG8gUiBNYXJrZG93biBhbmQgdGhlIGNvZGluZwpjYXNlIHN0dWRpZXMgdGhhdCB3ZSB3aWxsIGJlIHVzaW5nIGluIHRoZSBtYWNoaW5lIGxlYXJuaW5nLCBuZXR3b3JrCmFuYWx5c2lzLCBhbmQgdGV4dCBtaW5pbmcgbGFicy4gSXQgaXMgYSBjaGFuY2UgdG8gYmVjb21lIGZhbWlsaWFyIHdpdGgKaG93IFJTdHVkaW8gYW5kIFIgTWFya2Rvd24gd29ya3MuCgpZb3UgbWF5IGhhdmUgdXNlZCBSIGJlZm9yZS1vciB5b3UgbWF5IG5vdCBoYXZlISBFaXRoZXIgaXMgZmluZSBhcyB0aGlzCnRhc2sgd2lsbCBiZSBkZXNpZ25lZCB3aXRoIHRoZSBhc3N1bXB0aW9uIHRoYXQgeW91IGhhdmUgbm90IHVzZWQgUgpiZWZvcmUuIEl0IGluY2x1ZGVzICJyZWFjaGVzIiBmb3IgYW55b25lIHdobyBtYXkgd2FudCB0byBkbyBhIGJpdCBtb3JlLgoKSW4gdGhlIGNvbnRleHQgb2YgZG9pbmcgc28sIHdlJ2xsIGZvY3VzIG9uIHRoZSBmb2xsb3dpbmcgdGFza3M6CgoxLiAgUmVhZGluZyBkYXRhIGludG8gUiAoaW4gdGhlICoqUHJlcGFyZSoqIHNlY3Rpb24pCjIuICBQcmVwYXJpbmcgYW5kICJ3cmFuZ2xpbmciIGRhdGEgaW4gdGFibGUgKHRoaW5rIHNwcmVhZHNoZWV0ISkgZm9ybWF0CiAgICAoaW4gdGhlICoqV3JhbmdsZSoqIHNlY3Rpb24pCjMuICBDcmVhdGluZyBzb21lIHBsb3RzIChpbiB0aGUgKipFeHBsb3JlKiogc2VjdGlvbikKNC4gIFJ1bm5pbmcgYSBtb2RlbCAtIHNwZWNpZmljYWxseSwgYSByZWdyZXNzaW9uIG1vZGVsIChpbiB0aGUgKipNb2RlbCoqCiAgICBzZWN0aW9uKQo1LiAgRmluYWxseSwgY3JlYXRpbmcgYSByZXByb2R1Y2libGUgcmVwb3J0IG9mIHlvdXIgd29yayB5b3UgY2FuIHNoYXJlCiAgICB3aXRoIG90aGVycyAoaW4gdGhlICoqQ29tbXVuaWNhdGUqKiBzZWN0aW9uKQoKIyMjIFRoZSBMQVNFUiBDeWNsZQoKWW91IG1heSBiZSB3b25kZXJpbmcgd2hhdCB0aGVzZSBib2xkZWQgdGVybXMgYWJvdmUgcmVmZXIgdG87IHdoYXQncyBzbwpzcGVjaWFsIGFib3V0IHByZXBhcmluZywgd3JhbmdsaW5nLCBleHBsb3JpbmcsIGFuZCBtb2RlbGluZyBkYXRhIC0gYW5kCmNvbW11bmljYXRpbmcgcmVzdWx0cz8gV2UncmUgdXNpbmcgdGhlc2UgdGVybXMgYXMgYSBwYXJ0IG9mIGEgZnJhbWV3b3JrLApvciBtb2RlbCwgZm9yIHdoYXQgd2UgbWVhbiBieSBkb2luZyBsZWFybmluZyBpbiBTVEVNIGVkdWNhdGlvbiByZXNlYXJjaC4KClRoZSBwYXJ0aWN1bGFyIGZyYW1ld29yayB3ZSBhcmUgdXNpbmcgY29tZXMgZnJvbSB0aGUgd29yayBvZiBLcnVtbSBldAphbC4ncyBbKkxlYXJuaW5nIEFuYWx5dGljcyBHb2VzIHRvClNjaG9vbCpdKGh0dHBzOi8vZ2l0aHViLmNvbS9sYXNlci1pbnN0aXR1dGUvZXNzZW50aWFsLXJlYWRpbmdzL2Jsb2IvbWFpbi9sYXNlci1vcmllbnRhdGlvbi9MZWFybmluZyUyMEFuYWx5dGljcyUyMEdvZXMlMjB0byUyMFNjaG9vbC5wZGYpKi4qCllvdSBjYW4gY2hlY2sgdGhhdCBvdXQsIGJ1dCBkb24ndCBmZWVsIGFueSBuZWVkIHRvIGRpdmUgZGVlcCBmb3Igbm93IC0Kd2UnbGwgYmUgc3BlbmRpbmcgbW9yZSB0aW1lIG9uIHRoaXMgaW4gZmlyc3QgZGF5IG9mIHRoZSBzdW1tZXIKaW5zdGl0dXRlLiBGb3Igbm93LCBrbm93IHRoYXQgdGhpcyBkb2N1bWVudCBpcyBvcmdhbml6ZWQgYXJvdW5kIHRocmVlIG9mCnRoZSBmaXZlIGNvbXBvbmVudHMgb2Ygd2hhdCB3ZSdyZSByZWZlcnJpbmcgdG8gYXMgdGhlICoqTEFTRVIgY3ljbGUqKi4KCkNsaWNrIHRoZSBncmVlbiBhcnJvdyB0byB0aGUgcmlnaHQgb2YgdGhlICJjb2RlIGNodW5rIiBiZWxvdyB0byB2aWV3IHRoZQppbWFnZSAobW9yZSBvbiB0aGF0IHByb2Nlc3Mgb2YgY2xpY2tpbmcgdGhlIGdyZWVuIGFycm93IGFuZCB3aGF0IGl0CmRvZXMsIHRvbywgaW4gYSBtb21lbnQpIQoKYGBge3J9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCJpbWcvbGFzZXItY3ljbGUucG5nIikKYGBgCgojIyMgSG93IHRvIHVzZSB0aGlzIFIgTWFya2Rvd24gZG9jdW1lbnQKClRoaXMgaXMgYW4gUiBNYXJrZG93biBmaWxlIGFzIGluZGljYXRlZCBieSB0aGUgLnJtZCBleHRlbnNpb24gYXQgdGhlIGVuZApvZiB0aGUgZmlsZSBuYW1lLiBSIE1hcmtkb3duIGRvY3VtZW50cyBhcmUgZnVsbHkgcmVwcm9kdWNpYmxlIGFuZCB1c2UgYQpwcm9kdWN0aXZlIFtub3RlYm9vawppbnRlcmZhY2VdKGh0dHBzOi8vYm9va2Rvd24ub3JnL3lpaHVpL3JtYXJrZG93bi9ub3RlYm9vay5odG1sKSB0bwpjb21iaW5lIG5hcnJhdGl2ZSB0ZXh0IGFuZCAiY2h1bmtzIiBvZiBjb2RlIHRvIHByb2R1Y2UgYSByYW5nZSBvZgpmb3JtYXR0ZWQgb3V0cHV0cyBpbmNsdWRpbmc6IGZvcm1hdHMKaW5jbHVkaW5nwqBbSFRNTF0oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duL2h0bWwtZG9jdW1lbnQuaHRtbCkswqBbUERGXShodHRwczovL2Jvb2tkb3duLm9yZy95aWh1aS9ybWFya2Rvd24vcGRmLWRvY3VtZW50Lmh0bWwpLMKgW01TCldvcmRdKGh0dHBzOi8vYm9va2Rvd24ub3JnL3lpaHVpL3JtYXJrZG93bi93b3JkLWRvY3VtZW50Lmh0bWwpLMKgW0JlYW1lcl0oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duL2JlYW1lci1wcmVzZW50YXRpb24uaHRtbCkswqBbSFRNTDUKc2xpZGVzXShodHRwczovL2Jvb2tkb3duLm9yZy95aWh1aS9ybWFya2Rvd24vaW9zbGlkZXMtcHJlc2VudGF0aW9uLmh0bWwpLMKgW1R1ZnRlLXN0eWxlCmhhbmRvdXRzXShodHRwczovL2Jvb2tkb3duLm9yZy95aWh1aS9ybWFya2Rvd24vdHVmdGUtaGFuZG91dHMuaHRtbCkswqBbYm9va3NdKGh0dHBzOi8vYm9va2Rvd24ub3JnLykswqBbZGFzaGJvYXJkc10oaHR0cHM6Ly9ybWFya2Rvd24ucnN0dWRpby5jb20vZmxleGRhc2hib2FyZC8pLMKgW3NoaW55CmFwcGxpY2F0aW9uc10oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duL3NoaW55LWRvY3VtZW50cy5odG1sKSzCoFtzY2llbnRpZmljCmFydGljbGVzXShodHRwczovL2dpdGh1Yi5jb20vcnN0dWRpby9ydGljbGVzKSzCoFt3ZWJzaXRlc10oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duL3JtYXJrZG93bi1zaXRlLmh0bWwpLAphbmQgbW9yZS4KClRoZXJlIGFyZSB0d28ga2V5cyB0byB5b3VyIHVzZSBvZiBSIE1hcmtkb3duIGZvciB0aGlzIGFjdGl2aXR5OgoKMS4gIEZpcnN0LCBiZSBzdXJlIHRoYXQgeW91IGFyZSB2aWV3aW5nIHRoZSBkb2N1bWVudCBpbiB0aGUgIlZpc3VhbAogICAgRWRpdG9yIiBtb2RlLiBZb3UgY2FuIHVzZSB0aGlzIG1vZGUgYnkgY2xpY2tpbmcgdGhlIHdvcmQgIlZpc3VhbCIgb24KICAgIHRoZSBsZWZ0IHNpZGUgb2YgdGhlIHRvb2xiYXIgYWJvdmUuCjIuICBTZWNvbmQsIGNsaWNrICJLbml0IiBuZXh0IHRvIHRoZSB5YXJuIGJhbGwgYXQgdGhlIHRvcCBvZiB0aGlzIHNjcmVlbgogICAgdG8gcHJldmlldyB0aGUgZG9jdW1lbnQgYXMgeW91IHdvcmsgdGhyb3VnaCBpdC4gVGhpcyB3aWxsIGFsbG93IHlvdQogICAgdG8gc2VlIHlvdXIgY29kZSBhbmQgdGhlIGlucHV0IGluIGEgcmVuZGVyZWQgLSBlYXN5LXRvLXJlYWQgLQogICAgZG9jdW1lbnQsIGp1c3QgYXMgb3RoZXJzIHdpbGwgc2VlIHRoaXMgZG9jdW1lbnQgd2hlbiBzaGFyZWQuIFRyeQogICAga25pdHRpbmcgdGhlIGRvY3VtZW50IG5vdyBhbmQgc2VlIHdoYXQgaGFwcGVucy4KCkxldCdzIGdldCBzdGFydGVkISBXZSBhcmUgZ2xhZCB5b3UgYXJlIGhlcmUgYW5kIHRvIGJlZ2luIHRoaXMgZXhjaXRpbmcKKGFuZCBjaGFsbGVuZ2luZykgam91cm5leSB0b2dldGhlci4KCiMjIDEuIFBSRVBBUkUKCkJ5IHByZXBhcmluZywgd2UgcmVmZXIgdG8gZGV2ZWxvcGluZyBhIHF1ZXN0aW9uIG9yIHB1cnBvc2UgZm9yIHRoZQphbmFseXNpcywgd2hpY2ggeW91IGxpa2VseSBrbm93IGZyb20geW91ciByZXNlYXJjaCBjYW4gYmUgZGlmZmljdWx0IQpUaGlzIHBhcnQgb2YgdGhlIHByb2Nlc3MgYWxzbyBpbnZvbHZlcyBkZXZlbG9waW5nIGFuIHVuZGVyc3RhbmRpbmcgb2YKdGhlIGRhdGEgYW5kIHdoYXQgeW91IG1heSBuZWVkIHRvIGFuYWx5emUgdGhlIGRhdGEuIFRoaXMgb2Z0ZW4gaW52b2x2ZXMKbG9va2luZyBhdCB0aGUgZGF0YSBhbmQgaXRzIGRvY3VtZW50YXRpb24uIEZvciBub3csIHdlJ2xsIGZvY3VzIG9uIGp1c3QKYSBmZXcgcGFydHMgb2YgdGhpcyBwcm9jZXNzLCBkaXZpbmcgaW4gbXVjaCBtb3JlIGRlZXBseSBvdmVyIHRoZSBjb21pbmcKd2Vla3MuCgojIyMgUGFja2FnZXMg8J+TpgoKUiB1c2VzICJwYWNrYWdlcywiIGFkZC1vbnMgdGhhdCBlbmhhbmNlIGl0cyBmdW5jdGlvbmFsaXR5LiBPbmUgcGFja2FnZQp0aGF0IHdlJ2xsIGJlIHVzaW5nIGlzIHRoZSB0aWR5dmVyc2UuIFRvIGxvYWQgdGhlIHRpZHl2ZXJzZSwgY2xpY2sgdGhlCmdyZWVuIGFycm93IGluIHRoZSByaWdodCBjb3JuZXIgb2YgdGhlIGJsb2NrLW9yICJjaHVuayItb2YgY29kZSB0aGF0CmZvbGxvd3MuCgpgYGB7cn0KbGlicmFyeSh0aWR5dmVyc2UpCmBgYAoKUGxlYXNlIGRvIG5vdCB3b3JyeSBpZiB5b3Ugc2F3IGEgbnVtYmVyIG9mIG1lc3NhZ2VzOiB0aG9zZSBwcm9iYWJseSBtZWFuCnRoYXQgdGhlIHRpZHl2ZXJzZSBsb2FkZWQganVzdCBmaW5lLiBJZiB5b3Ugc2VlIGFuIGVycm9yLCB0aG91Z2gsIHRyeSB0bwppbnRlcnByZXQgb3Igc2VhcmNoIHZpYSB5b3VyIHNlYXJjaCBlbmdpbmUgdGhlIGNvbnRlbnRzIG9mIHRoZSBlcnJvciwgb3IKcmVhY2ggb3V0IHRvIHVzIGZvciBhc3Npc3RhbmNlLgoKIyMjIExvYWRpbmcgKG9yIHJlYWRpbmcgaW4pIGRhdGEKCk5leHQsIHdlJ2xsIGxvYWQgZGF0YS1zcGVjaWZpY2FsbHksIGEgQ1NWIGZpbGUsIHRoZSBraW5kIHRoYXQgeW91IGNhbgpleHBvcnQgZnJvbSBNaWNyb3NvZnQgRXhjZWwgb3IgR29vZ2xlIFNoZWV0cyAtIGludG8gUiwgdXNpbmcgdGhlCmByZWFkX2NzdigpYCBmdW5jdGlvbiBpbiB0aGUgbmV4dCBjaHVuay4KCkNsaWNraW5nIHRoZSBncmVlbiBhcnJvdyBydW5zIHRoZSBjb2RlOyBkbyB0aGF0IG5leHQgdG8gcmVhZCB0aGUKYHNjaS1vbmxpbmUtY2xhc3Nlcy5jc3ZgIGZpbGUgc3RvcmVkIGluIHlvdXIgZGF0YSBmb2xkZXIgaW50byB5b3VyIFIKZW52aXJvbm1lbnQ6CgpgYGB7cn0KZCA8LSByZWFkX2NzdigiZGF0YS9zY2ktb25saW5lLWNsYXNzZXMuY3N2IikKYGBgCgpOaWNlIHdvcmshIFlvdSBzaG91bGQgbm93IHNlZSBhIG5ldyBkYXRhICJvYmplY3QiIG5hbWVkIGBkYCBzYXZlZCBpbgp5b3VyIEVudmlyb25tZW50IHBhbmUuIFRyeSBjbGlja2luZyBvbiBpdCBhbmQgc2VlIHdoYXQgaGFwcGVucy4KCiMjIyMgVmlld2luZyBvciBpbnNwZWN0aW5nIGRhdGEKCk5vdyBsZXQncyBsZWFybiBhbm90aGVyIHdheSB0byBpbnNwZWN0IG91ciBkYXRhLiBSdW4gdGhlIG5leHQgY2h1bmsgYW5kCmxvb2sgYXQgdGhlIHJlc3VsdHMsIHRhYmJpbmcgbGVmdCBvciByaWdodCB3aXRoIHRoZSBhcnJvd3MsIG9yIHNjYW5uaW5nCnRocm91Z2ggdGhlIHJvd3MgYnkgY2xpY2tpbmcgdGhlIG51bWJlcnMgYXQgdGhlIGJvdHRvbSBvZiB0aGUgcGFuZSB3aXRoCnRoZSBwcmludC1vdXQgb2YgdGhlIGRhdGEgeW91IGxvYWRlZDoKCmBgYHtyfQpkCnN0cihkKQpgYGAKCiMjIyMgWyoqWW91ciBUdXJuKipde3N0eWxlPSJjb2xvcjogZ3JlZW47In0gKiripLUqKgoKV2hhdCBkbyB5b3Ugbm90aWNlIGFib3V0IHRoaXMgZGF0YSBzZXQ/IFdoYXQgZG8geW91IHdvbmRlcj8gQWRkIG9uZS10d28KdGhvdWdodHMgZm9sbG93aW5nIHRoZSBkYXNoZXMgbmV4dCAoeW91IGNhbiBhZGQgYWRkaXRpb25hbCBkYXNoZXMgaWYgeW91Cmxpa2UhKToKCi0gVGhlIGRhdGFmcmFtZSBhcHBlYXJzIHRvIGVzc2VudGlhbGx5IGJlIGEgZ3JhZGVib29rIGZvciBzY2llbmNlIGNvdXJzZShzKSB3aXRoIHNvbWUgYWRkaXRpb25hbCB2YXJpYWJsZXMgb24gc3R1ZGVudCBiZWhhdmlvciwgbGlrZSB0aGUgbnVtYmVyIG9mIGhvdXJzIHNwZW50IChzdHVkeWluZywgcHJlc3VtYWJseT8pCi0gICBXaGF0IGlzIENFTVMKClRoZXJlIGFyZSBvdGhlciB3YXlzIHRvIGluc3BlY3QgeW91ciBkYXRhOyB0aGUgYGdsaW1wc2UoKWAgZnVuY3Rpb24KcHJvdmlkZXMgb25lIHN1Y2ggd2F5LiBSdW4gdGhlIGNvZGUgYmVsb3cgdG8gdGFrZSBhIGdsaW1wc2UgYXQgeW91cgpkYXRhLgoKYGBge3J9CmdsaW1wc2UoZCkKYGBgCgpXZSBoYXZlIG9uZSBtb3JlIHF1ZXN0aW9uIHRvIHBvc2UgdG8geW91OiBXaGF0IGRvIHJvd3MgYW5kIGNvbHVtbnMKdHlwaWNhbGx5IHJlcHJlc2VudCBpbiB5b3VyIGFyZWEgb2Ygd29yayBhbmQvb3IgcmVzZWFyY2g/CgpHZW5lcmFsbHksIHJvd3MgdHlwaWNhbGx5IHJlcHJlc2VudCAiY2FzZXMsIiB0aGUgdW5pdHMgdGhhdCB3ZSBtZWFzdXJlLApvciB0aGUgdW5pdHMgb24gd2hpY2ggd2UgY29sbGVjdCBkYXRhLiBUaGlzIGlzIG5vdCBhIHRyaWNrIHF1ZXN0aW9uIQpXaGF0IGNvdW50cyBhcyBhICJjYXNlIiAoYW5kIHRoZXJlZm9yZSB3aGF0IGlzIHJlcHJlc2VudGVkIGFzIGEgcm93KQp2YXJpZXMgYnkgKGFuZCB3aXRoaW4pIGZpZWxkcy4gVGhlcmUgbWF5IGJlIG11bHRpcGxlIHR5cGVzIG9yIGxldmVscyBvZgp1bml0cyBzdHVkaWVkIGluIHlvdXIgZmllbGQ7IGxpc3RpbmcgbW9yZSB0aGFuIG9uZSBpcyBmaW5lISBBbHNvLCBwbGVhc2UKY29uc2lkZXIgd2hhdCBjb2x1bW5zIC0gd2hpY2ggdXN1YWxseSByZXByZXNlbnQgdmFyaWFibGVzIC0gcmVwcmVzZW50IGluCnlvdXIgYXJlYSBvZiB3b3JrIGFuZC9vciByZXNlYXJjaC4KCiMjIyMgWyoqWW91ciBUdXJuKipde3N0eWxlPSJjb2xvcjogZ3JlZW47In0gKiripLUqKgoKV2hhdCByb3dzIHR5cGljYWxseSAob3IgeW91IHRoaW5rIG1heSkgcmVwcmVzZW50OgoKLSAgIEkgYmVsaWV2ZSB0aGUgcm93cyBoZXJlIHJlZmVyIHRvIHN0dWRlbnRzIChzbyBpdCdzIGEgd2lkZSwgcmF0aGVyIHRoYW4gYSBsb25nL3RpZHkgZGF0YXNldCkuCgpXaGF0IGNvbHVtbnMgdHlwaWNhbGx5IChvciB5b3UgdGhpbmsgbWF5KSByZXByZXNlbnQ6CgotIE91dGNvbWVzIG9yIHZhcmlhYmxlcyByZWxhdGVkIHRvIHRoZSBzdHVkZW50IChlLmcuLCB0aGVpciBnZW5kZXIsIHRoZWlyIGNvdXJzZSwgdGhlaXIgc2NvcmUpCgpOZXh0LCB3ZSdsbCB1c2UgYSBmZXcgZnVuY3Rpb25zIHRoYXQgYXJlIGhhbmR5IGZvciBwcmVwYXJpbmcgZGF0YSBpbgp0YWJsZSBmb3JtLgoKIyMgMi4gV1JBTkdMRQoKQnkgd3JhbmdsZSwgd2UgcmVmZXIgdG8gdGhlIHByb2Nlc3Mgb2YgY2xlYW5pbmcgYW5kIHByb2Nlc3NpbmcgZGF0YSwKYW5kLCBpbiBjYXNlcywgbWVyZ2luZyAob3Igam9pbmluZykgZGF0YSBmcm9tIG11bHRpcGxlIHNvdXJjZXMuIE9mdGVuLAp0aGlzIHBhcnQgb2YgdGhlIHByb2Nlc3MgaXMgdmVyeSAoc3VycHJpc2luZ2x5KSB0aW1lLWludGVuc2l2ZS4KV3JhbmdsaW5nIHlvdXIgZGF0YSBpbnRvIHNoYXBlIGNhbiBpdHNlbGYgYmUgYW4gaW1wb3J0YW50CmFjY29tcGxpc2htZW50ISBUaGVyZSBhcmUgZ3JlYXQgdG9vbHMgaW4gUiB0byBkbyB0aGlzLCBlc3BlY2lhbGx5CnRocm91Z2ggdGhlIHVzZSBvZiB0aGUge2RwbHlyfSBSIHBhY2thZ2UuCgojIyMgU2VsZWN0aW5nIHZhcmlhYmxlcwoKTGV0J3Mgc2VsZWN0IG9ubHkgYSBmZXcgdmFyaWFibGVzLgoKYGBge3J9CmxpYnJhcnkoZHBseXIpCmQgJT4lCiAgc2VsZWN0KHN0dWRlbnRfaWQsIHRvdGFsX3BvaW50c19wb3NzaWJsZSwgdG90YWxfcG9pbnRzX2Vhcm5lZCkKYGBgCgpOb3RpY2UgaG93IHRoZSBudW1iZXIgb2YgY29sdW1ucyAodmFyaWFibGVzKSBpcyBub3cgZGlmZmVyZW50LgoKTGV0J3MgKmluY2x1ZGUgb25lIGFkZGl0aW9uYWwgdmFyaWFibGUqIGluIHlvdXIgc2VsZWN0IGZ1bmN0aW9uLgoKRmlyc3QsIHdlIG5lZWQgdG8gZmlndXJlIG91dCB3aGF0IHZhcmlhYmxlcyBleGlzdCBpbiBvdXIgZGF0YXNldCAob3IgYmUKcmVtaW5kZWQgb2YgdGhpcyAtIGl0J3MgdmVyeSBjb21tb24gaW4gUiB0byBiZSBjb250aW51YWxseSBjaGVja2luZyBhbmQKaW5zcGVjdGluZyB5b3VyIGRhdGEpIQoKWW91IGNhbiB1c2UgYSBmdW5jdGlvbiBuYW1lZCBnbGltcHNlKCkgdG8gZG8gdGhpcy4KCmBgYHtyfQpnbGltcHNlKGQpCmBgYAoKIyMjIyBbKipZb3VyIFR1cm4qKl17c3R5bGU9ImNvbG9yOiBncmVlbjsifSAqKuKktSoqCgpJbiB0aGUgY29kZSBjaHVuayBiZWxvdywgYWRkIGEgbmV3IHZhcmlhYmxlIHRvIHRoZSBjb2RlIGJlbG93LCBiZWluZwpjYXJlZnVsIHRvIHR5cGUgdGhlIG5ldyB2YXJpYWJsZSBuYW1lIGFzIGl0IGFwcGVhcnMgaW4gdGhlIGRhdGEuIFdlJ3ZlCmFkZGVkIHNvbWUgY29kZSB0byBnZXQgeW91IHN0YXJ0ZWQuIENvbnNpZGVyIGhvdyB0aGUgbmFtZXMgb2YgdGhlIG90aGVyCnZhcmlhYmxlcyBhcmUgc2VwYXJhdGVkIGFzIHlvdSB0aGluayBhYm91dCBob3cgdG8gYWRkIGFuIGFkZGl0aW9uYWwKdmFyaWFibGUgdG8gdGhpcyBjb2RlLgoKYGBge3J9CmQgJT4lCiAgc2VsZWN0KHN0dWRlbnRfaWQsIHRvdGFsX3BvaW50c19wb3NzaWJsZSwgdG90YWxfcG9pbnRzX2Vhcm5lZCwgcGMpCmBgYAoKT25jZSBhZGRlZCwgdGhlIG91dHB1dCBzaG91bGQgYmUgZGlmZmVyZW50IHRoYW4gaW4gdGhlIGNvZGUgYWJvdmUgLQp0aGVyZSBzaG91bGQgbm93IGJlIGFuIGFkZGl0aW9uYWwgdmFyaWFibGUgaW5jbHVkZWQgaW4gdGhlIHByaW50LW91dC4KCiMjIyBGaWx0ZXJpbmcgdmFyaWFibGVzCgpOZXh0LCBsZXQncyBleHBsb3JlIGZpbHRlcmluZyB2YXJpYWJsZXMuIENoZWNrIG91dCBhbmQgcnVuIHRoZSBuZXh0CmNodW5rIG9mIGNvZGUsIGltYWdpbmluZyB0aGF0IHdlIHdpc2ggdG8gZmlsdGVyIG91ciBkYXRhIHRvIHZpZXcgb25seQp0aGUgcm93cyBhc3NvY2lhdGVkIHdpdGggc3R1ZGVudHMgd2hvIGVhcm5lZCBhIGZpbmFsIGdyYWRlIChhcyBhCnBlcmNlbnRhZ2UpIG9mIDcwIC0gNzAlIC0gb3IgaGlnaGVyLgoKYGBge3J9CmQgJT4lCiAgZmlsdGVyKEZpbmFsR3JhZGVDRU1TID4gNzApCmBgYAoKIyMjIyMgWyoqWW91ciBUdXJuKipde3N0eWxlPSJjb2xvcjogZ3JlZW47In0gKiripLUqKgoKSW4gdGhlIG5leHQgY29kZSBjaHVuaywgY2hhbmdlIHRoZSBjdXQtb2ZmIGZyb20gNzAlIHRvIHNvbWUgb3RoZXIKdmFsdWUgLSBsYXJnZXIgb3Igc21hbGxlciAobWF5YmUgbXVjaCBsYXJnZXIgb3Igc21hbGxlciAtIGZlZWwgZnJlZSB0bwpwbGF5IGFyb3VuZCB3aXRoIHRoZSBjb2RlIGEgYml0ISkuCgpgYGB7cn0KZCAlPiUKICBmaWx0ZXIoRmluYWxHcmFkZUNFTVMgPD0gMjApCmBgYAoKV2hhdCBoYXBwZW5zIHdoZW4geW91IGNoYW5nZSB0aGUgY3V0LW9mZiBmcm9tIDcwIHRvIHNvbWV0aGluZyBlbHNlPyBBZGQKYSB0aG91Z2h0IChvciBtb3JlKToKCi0gICBUaGVyZSBhcmUgZmFyIGZld2VyIG9ic2VydmF0aW9ucyBpbiB0aGUgcmVzdWx0aW5nIGRhdGFmcmFtZQoKIyMjIEFycmFuZ2UKClRoZSBsYXN0IGZ1bmN0aW9uIHdlJ2xsIHVzZSBmb3IgcHJlcGFyaW5nIHRhYmxlcyBpcyBhcnJhbmdlLgoKV2UnbGwgY29tYmluZSB0aGlzIGFycmFuZ2UoKSBmdW5jdGlvbiB3aXRoIGEgZnVuY3Rpb24gd2UgdXNlZCBhbHJlYWR5IC0Kc2VsZWN0KCkuIFdlIGRvIHRoaXMgc28gd2UgY2FuIHZpZXcgb25seSB0aGUgc3R1ZGVudCBJRCBhbmQgdGhlaXIgZmluYWwKZ3JhZGUuCgpgYGB7cn0KZCAlPiUKICBzZWxlY3Qoc3R1ZGVudF9pZCwgRmluYWxHcmFkZUNFTVMpICU+JSAKICBhcnJhbmdlKEZpbmFsR3JhZGVDRU1TKQpgYGAKCk5vdGUgdGhhdCBhcnJhbmdlIHdvcmtzIGJ5IHNvcnRpbmcgdmFsdWVzIGluIGFzY2VuZGluZyBvcmRlciAoZnJvbQpsb3dlc3QgdG8gaGlnaGVzdCk7IHlvdSBjYW4gY2hhbmdlIHRoaXMgYnkgdXNpbmcgdGhlIGRlc2MoKSBmdW5jdGlvbgp3aXRoIGFycmFuZ2UsIGxpa2UgdGhlIGZvbGxvd2luZzoKCmBgYHtyfQpkICU+JQogIHNlbGVjdChzdHVkZW50X2lkLCBGaW5hbEdyYWRlQ0VNUykgJT4lCiAgYXJyYW5nZShkZXNjKEZpbmFsR3JhZGVDRU1TKSkKYGBgCgojIyMjIFsqKllvdXIgVHVybioqXXtzdHlsZT0iY29sb3I6IGdyZWVuOyJ9ICoq4qS1KioKCkluIHRoZSBjb2RlIGNodW5rIGJlbG93LCByZXBsYWNlIEZpbmFsR3JhZGVDRU1TIHRoYXQgaXMgdXNlZCB3aXRoIGJvdGgKdGhlIHNlbGVjdCgpIGFuZCBhcnJhbmdlKCkgZnVuY3Rpb25zIHdpdGggYSBkaWZmZXJlbnQgdmFyaWFibGUgaW4gdGhlCmRhdGEgc2V0LiBDb25zaWRlciByZXR1cm5pbmcgdG8gdGhlIGNvZGUgY2h1bmsgYWJvdmUgaW4gd2hpY2ggeW91CmdsaW1wc2VkIGF0IHRoZSBuYW1lcyBvZiBhbGwgb2YgdGhlIHZhcmlhYmxlcy4KCmBgYHtyfQpkICU+JQogIHNlbGVjdChzdHVkZW50X2lkLCBwYykgJT4lCiAgYXJyYW5nZShkZXNjKHBjKSkKYGBgCgojIyMgUmVhY2ggMSDwn46JCgpDYW4geW91IGNvbXBvc2UgYSBzZXJpZXMgb2YgZnVuY3Rpb25zIHRoYXQgaW5jbHVkZSB0aGUgc2VsZWN0KCksCmZpbHRlcigpLCBhbmQgYXJyYW5nZSBmdW5jdGlvbnM/IFJlY2FsbCB0aGF0IHlvdSBjYW4gInBpcGUiIHRoZSBvdXRwdXQKZnJvbSBvbmUgZnVuY3Rpb24gdG8gdGhlIG5leHQgYXMgd2hlbiB3ZSB1c2VkIHNlbGVjdCgpIGFuZCBhcnJhbmdlKCkKdG9nZXRoZXIgaW4gdGhlIGNvZGUgY2h1bmsgYWJvdmUuCgoqVGhpcyByZWFjaCBpcyBub3QgcmVxdWlyZWQvbmVjZXNzYXJ5IHRvIGNvbXBsZXRlOyBpdCdzIGp1c3QgZm9yIHRob3NlCndobyB3aXNoIHRvIGRvIGEgYml0IG1vcmUgd2l0aCB0aGVzZSBmdW5jdGlvbnMgYXQgdGhpcyB0aW1lICh3ZSdsbCBkbwptb3JlIGluIGNsYXNzLCB0b28hKSoKCmBgYHtyfQouLi4KYGBgCgoKIyMgMy4gRVhQTE9SRQoKRXhwbG9yYXRvcnkgZGF0YSBhbmFseXNpcywgb3IgZXhwbG9yaW5nIHlvdXIgZGF0YSwgaW52b2x2ZXMgcHJvY2Vzc2VzIG9mCipkZXNjcmliaW5nKiB5b3VyIGRhdGEgKHN1Y2ggYXMgYnkgY2FsY3VsYXRpbmcgdGhlIG1lYW5zIGFuZCBzdGFuZGFyZApkZXZpYXRpb25zIG9mIG51bWVyaWMgdmFyaWFibGVzLCBvciBjb3VudGluZyB0aGUgZnJlcXVlbmN5IG9mCmNhdGVnb3JpY2FsIHZhcmlhYmxlcykgYW5kLCBvZnRlbiwgdmlzdWFsaXppbmcgeW91ciBkYXRhIHByaW9yLiBJbiB0aGlzCnNlY3Rpb24sIHdlJ2xsIGNyZWF0ZSBhIGZldyBwbG90cyB0byBleHBsb3JlIG91ciBkYXRhLgoKIyMjIEhpc3RvZ3JhbQoKVGhlIGNvZGUgYmVsb3cgY3JlYXRlcyBhIGhpc3RvZ3JhbSwgb3IgYSBkaXN0cmlidXRpb24gb2YgdGhlIHZhbHVlcywgaW4KdGhpcyBjYXNlIGZvciBzdHVkZW50cycgZmluYWwgZ3JhZGVzLgoKYGBge3J9CmdncGxvdChkLCBhZXMoeCA9IEZpbmFsR3JhZGVDRU1TKSkgKwogIGdlb21faGlzdG9ncmFtKCkKYGBgCgpZb3UgY2FuIGNoYW5nZSB0aGUgY29sb3Igb2YgdGhlIGhpc3RvZ3JhbSBiYXJzIGJ5IHNwZWNpZnlpbmcgYSBjb2xvciBhcwpmb2xsb3dzOgoKYGBge3J9CmdncGxvdChkLCBhZXMoeCA9IEZpbmFsR3JhZGVDRU1TKSkgKwogIGdlb21faGlzdG9ncmFtKGZpbGwgPSAiYmx1ZSIpCmBgYAoKIyMjIENoYW5naW5nIGNvbG9ycwoKIyMjIyBbKipZb3VyIFR1cm4qKl17c3R5bGU9ImNvbG9yOiBncmVlbjsifSAqKuKktSoqCgpJbiB0aGUgY29kZSBjaHVuayBiZWxvdywgY2hhbmdlIHRoZSBjb2xvciB0byBvbmUgb2YgeW91ciBjaG9vc2luZzsKY29uc2lkZXIgdGhpcyBsaXN0IG9mIHZhbGlkIGNvbG9yIG5hbWVzIGhlcmU6CjxodHRwOi8vd3d3LnN0YXQuY29sdW1iaWEuZWR1L350emhlbmcvZmlsZXMvUmNvbG9yLnBkZj4KCmBgYHtyfQpnZ3Bsb3QoZCwgYWVzKHggPSBGaW5hbEdyYWRlQ0VNUykpICsKICBnZW9tX2hpc3RvZ3JhbShmaWxsID0gImRvZGdlcmJsdWUiKQpgYGAKCkZpbmFsbHksIHdlJ2xsIG1ha2Ugb25lIG1vcmUgY2hhbmdlOyB2aXN1YWxpemUgdGhlIGRpc3RyaWJ1dGlvbiBvZgphbm90aGVyIHZhcmlhYmxlIGluIHRoZSBkYXRhIC0gb25lIG90aGVyIHRoYW4gRmluYWxHcmFkZUNFTVMuIFlvdSBjYW4gZG8Kc28gYnkgc3dhcHBpbmcgb3V0IHRoZSBuYW1lIGZvciBhbm90aGVyIHZhcmlhYmxlIHdpdGggRmluYWxHcmFkZUNFTVMuCkFsc28sIGNoYW5nZSB0aGUgY29sb3IgdG8gb25lIG90aGVyIHRoYW4gYmx1ZS4KCmBgYHtyfQpnZ3Bsb3QoZCwgYWVzKHggPSBwYykpICsKICBnZW9tX2hpc3RvZ3JhbShmaWxsID0gImdyZWVuIikKYGBgCgojIyMgUmVhY2ggMiDwn46JCgpDb21wbGV0ZWQgdGhlIGFib3ZlPyBOaWNlIGpvYiEgVHJ5IGZvciBhICJyZWFjaCIgYnkgY3JlYXRpbmcgYSBzY2F0dGVyCnBsb3QgZm9yIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiB0d28gdmFyaWFibGVzLiBZb3Ugd2lsbCBuZWVkIHRvIHBhc3MKdGhlIG5hbWVzIG9mIHR3byB2YXJpYWJsZXMgdG8gdGhlIGNvZGUgYmVsb3cgZm9yIHdoYXQgaXMgbm93IHNpbXBseSBYWFgKKGEgcGxhY2Vob2xkZXIpLgoKYGBge3J9CnN0cihkKQpnZ3Bsb3QoZCwgYWVzKHggPSBwYywgeSA9IHRvdGFsX3BvaW50c19lYXJuZWQpKSArCiAgZ2VvbV9wb2ludCgpCmBgYAoKIyMgNC4gTU9ERUwKCiJNb2RlbCIgaXMgb25lIG9mIHRob3NlIHRlcm1zIHRoYXQgaGFzIG1hbnkgZGlmZmVyZW50IG1lYW5pbmdzLiBGb3Igb3VyCnB1cnBvc2UsIHdlIHJlZmVyIHRvIHRoZSBwcm9jZXNzIG9mIHNpbXBsaWZ5aW5nIGFuZCBzdW1tYXJpemluZyBvdXIKZGF0YS4gVGh1cywgbW9kZWxzIGNhbiB0YWtlIG1hbnkgZm9ybXM7IGNhbGN1bGF0aW5nIG1lYW5zIHJlcHJlc2VudHMgYQpsZWdpdGltYXRlIGZvcm0gb2YgbW9kZWxpbmcgZGF0YSwgYXMgZG9lcyBlc3RpbWF0aW5nIG1vcmUgY29tcGxleAptb2RlbHMsIGluY2x1ZGluZyBsaW5lYXIgcmVncmVzc2lvbnMsIGFuZCBtb2RlbHMgYW5kIGFsZ29yaXRobXMKYXNzb2NpYXRlZCB3aXRoIG1hY2hpbmUgbGVhcm5pbmcgdGFza3MuIEZvciBub3csIHdlJ2xsIHJ1biBhIGxpbmVhcgpyZWdyZXNzaW9uIHRvIHByZWRpY3Qgc3R1ZGVudHMnIGZpbmFsIGdyYWRlcy4KCkJlbG93LCB3ZSBwcmVkaWN0IHN0dWRlbnRzJyBmaW5hbCBncmFkZXMgKGBGaW5hR3JhZGVDRU1TYCwgd2hpY2ggaXMgb24gYQowLTEwMCBwb2ludCBzY2FsZSkgb24gdGhlIGJhc2lzIG9mIHRoZSB0aW1lIHRoZXkgc3BlbnQgb24gdGhlIGNvdXJzZQoobWVhc3VyZWQgdGhyb3VnaCB0aGVpciBsZWFybmluZyBtYW5hZ2VtZW50IHN5c3RlbSBpbiBtaW51dGVzLApgVGltZVNwZW50YCwgYW5kIHRoZSBzdWJqZWN0IChvbmUgb2YgZml2ZSkgb2YgdGhlaXIgc3BlY2lmaWMgY291cnNlLgoKYGBge3J9Cm0xIDwtIGxtKEZpbmFsR3JhZGVDRU1TIH4gVGltZVNwZW50ICsgc3ViamVjdCwgZGF0YSA9IGQpCnN1bW1hcnkobTEpCmBgYAoKIyMjIyBbKipZb3VyIFR1cm4qKl17c3R5bGU9ImNvbG9yOiBncmVlbjsifSAqKuKktSoqCgpOb3RpY2UgaG93IGFib3ZlIHRoZSB2YXJpYWJsZXMgYXJlIHNlcGFyYXRlZCBieSBhICsgc3ltYm9sLiBCZWxvdywgYWRkCiphbm90aGVyKiAtIGEgdGhpcmQgLSB2YXJpYWJsZSB0byB0aGUgcmVncmVzc2lvbiBtb2RlbC4gU3BlY2lmaWNhbGx5LAphZGQgYSB2YXJpYWJsZSBzdHVkZW50cycgaW5pdGlhbCwgc2VsZi1yZXBvcnRlZCBpbnRlcmVzdCBpbiBzY2llbmNlLApgaW50YCAtIGFuZCBhbnkgb3RoZXIgdmFyaWFibGUocykgeW91IGxpa2UhIFdoYXQgZG8geW91IG5vdGljZSBhYm91dCB0aGUKcmVzdWx0cz8gV2UncmUgZ29pbmcgdG8gZGl2ZSBpbnRvIHRoaXMgKm11Y2gqIG1vcmU6IGlmIHlvdSBoYXZlIG1hbnkKcXVlc3Rpb25zIG5vdywgeW91J3JlIGluIHRoZSByaWdodCBzcG90IQoKYGBge3J9Cm0yIDwtIGxtKEZpbmFsR3JhZGVDRU1TIH4gVGltZVNwZW50ICsgc3ViamVjdCArIHBjLCBkYXRhID0gZCkKc3VtbWFyeShtMikKYGBgCgojIyA1LiBDT01NVU5JQ0FURQoKR3JlYXQgam9iISBPbmNlIHlvdSd2ZSBmaW5pc2hlZCB5b3VyIHdvcmssIFVwb24gZG9pbmcgc28sIHlvdSBzaG91bGQgc2VlCmEgbmV3IGBsYXNlci1vcmllbnRhdGlvbi1iYWRnZS5odG1sYC4KCkNvbmdyYXR1bGF0aW9ucywgeW91J3ZlIGNvbXBsZXRlZCB5b3VyIE1vZGVscyAmIEluZmVyZW5jZSBCYWRnZSEKQ29tcGxldGUgdGhlIGZvbGxvd2luZyBzdGVwcyB0byBzdWJtaXQgeW91ciB3b3JrIGZvciByZXZpZXcgYnkKCjEuICBDaGFuZ2UgdGhlIG5hbWUgb2YgdGhlIGBhdXRob3I6YCBpbiB0aGUgW1lBTUwKICAgIGhlYWRlcl0oaHR0cHM6Ly9tb25hc2hkYXRhZmx1ZW5jeS5naXRodWIuaW8vci1yZXAtcmVzL3lhbWwtaGVhZGVyLmh0bWwpCiAgICBhdCB0aGUgdmVyeSB0b3Agb2YgdGhpcyBkb2N1bWVudCB0byB5b3VyIG5hbWUuIEFzIG5vdGVkIGluCiAgICBbUmVwcm9kdWNpYmxlIFJlc2VhcmNoIGluCiAgICBSXShodHRwczovL21vbmFzaGRhdGFmbHVlbmN5LmdpdGh1Yi5pby9yLXJlcC1yZXMvaW5kZXguaHRtbCksIFRoZQogICAgWUFNTCBoZWFkZXIgY29udHJvbHMgdGhlIHN0eWxlIGFuZCBmZWVsIGZvciBrbml0dGVkIGRvY3VtZW50IGJ1dAogICAgZG9lc24ndCBhY3R1YWxseSBkaXNwbGF5IGluIHRoZSBmaW5hbCBvdXRwdXQuCgoyLiAgQ2xpY2sgdGhlIHlhcm4gaWNvbiBhYm92ZSB0byAia25pdCIgeW91ciBkYXRhIHByb2R1Y3QgdG8gYQogICAgW0hUTUxdKGh0dHBzOi8vYm9va2Rvd24ub3JnL3lpaHVpL3JtYXJrZG93bi9odG1sLWRvY3VtZW50Lmh0bWwpIGZpbGUKICAgIHRoYXQgd2lsbCBiZSBzYXZlZCBpbiB5b3VyIFIgUHJvamVjdCBmb2xkZXIuCgozLiAgQ29tbWl0IHlvdXIgY2hhbmdlcyBpbiBHaXRIdWIgRGVza3RvcCBhbmQgcHVzaCB0aGVtIHRvIHlvdXIgb25saW5lCiAgICBHaXRIdWIgcmVwb3NpdG9yeS4KCjQuICBQdWJsaXNoIHlvdXIgSFRNTCBwYWdlIHRoZSB3ZWIgdXNpbmcgb25lIG9mIHRoZSBmb2xsb3dpbmcKICAgIFtwdWJsaXNoaW5nIG1ldGhvZHNdKGh0dHBzOi8vcnB1YnMuY29tL2NhdGh5ZGF0YXNjaWVuY2UvNTE4NjkyKToKCiAgICAtICAgUHVibGlzaCBvbiBbUlB1YnNdKGh0dHBzOi8vcnB1YnMuY29tKSBieSBjbGlja2luZyB0aGUgIlB1Ymxpc2giCiAgICAgICAgYnV0dG9uIGxvY2F0ZWQgaW4gdGhlIFZpZXdlciBQYW5lIHdoZW4geW91IGtuaXQgeW91ciBkb2N1bWVudC4KICAgICAgICBOb3RlLCB5b3Ugd2lsbCBuZWVkIHRvIHF1aWNrbHkgY3JlYXRlIGEgUlB1YnMgYWNjb3VudC4KCiAgICAtICAgUHVibGlzaGluZyBvbiBHaXRIdWIgdXNpbmcgZWl0aGVyIFtHaXRIdWIKICAgICAgICBQYWdlc10oaHR0cHM6Ly9wYWdlcy5naXRodWIuY29tKSBvciB0aGUgW0hUTUwKICAgICAgICBwcmV2aWV3ZXJdKGh0dHA6Ly9odG1scHJldmlldy5naXRodWIuaW8pLgoKNS4gIFBvc3QgYSBuZXcgZGlzY3Vzc2lvbiBvbiBHaXRIdWIgdG8gb3VyIFtGb3VuZGF0aW9ucyBCYWRnZXMKICAgIGZvcnVtXShodHRwczovL2dpdGh1Yi5jb20vb3Jncy9sYXNlci1pbnN0aXR1dGUvdGVhbXMvZm91bmRhdGlvbnMvZGlzY3Vzc2lvbnMvMikuCiAgICBJbiB5b3VyIHBvc3QsIGluY2x1ZGUgYSBsaW5rIHRvIHlvdXIgcHVibGlzaGVkIHdlYiBwYWdlIGFuZCBhIHNob3J0CiAgICByZWZsZWN0aW9uIGhpZ2hsaWdodGluZyBvbmUgdGhpbmcgeW91IGxlYXJuZWQgZnJvbSB0aGlzIGxhYiBhbmQgb25lCiAgICB0aGluZyB5b3UnZCBsaWtlIHRvIGV4cGxvcmUgZnVydGhlci4K