Section 1 - Creating Our First Model

1.1

Load the training and testing sets using the read.csv() function, and save them as variables with the names pisaTrain and pisaTest.

How many students are there in the training set?

pisaTrain

Error: object 'pisaTrain' not found

1.2

Using tapply() on pisaTrain, what is the average reading test score of males?Of females?

A

       0        1 
512.9406 483.5325

1.3

Which variables are missing data in at least one observation in the training set? Select all that apply.

summary(pisaTrain)

     grade            male                      raceeth    
 Min.   : 8.00   Min.   :0.0000   White             :2015  
 1st Qu.:10.00   1st Qu.:0.0000   Hispanic          : 834  
 Median :10.00   Median :1.0000   Black             : 444  
 Mean   :10.09   Mean   :0.5111   Asian             : 143  
 3rd Qu.:10.00   3rd Qu.:1.0000   More than one race: 124  
 Max.   :12.00   Max.   :1.0000   (Other)           :  68  
                                  NA's              :  35  
   preschool      expectBachelors     motherHS    motherBachelors 
 Min.   :0.0000   Min.   :0.0000   Min.   :0.00   Min.   :0.0000  
 1st Qu.:0.0000   1st Qu.:1.0000   1st Qu.:1.00   1st Qu.:0.0000  
 Median :1.0000   Median :1.0000   Median :1.00   Median :0.0000  
 Mean   :0.7228   Mean   :0.7859   Mean   :0.88   Mean   :0.3481  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.00   3rd Qu.:1.0000  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.00   Max.   :1.0000  
 NA's   :56       NA's   :62       NA's   :97     NA's   :397     
   motherWork        fatherHS      fatherBachelors 
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.0000   1st Qu.:1.0000   1st Qu.:0.0000  
 Median :1.0000   Median :1.0000   Median :0.0000  
 Mean   :0.7345   Mean   :0.8593   Mean   :0.3319  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.0000  
 NA's   :93       NA's   :245      NA's   :569     
   fatherWork       selfBornUS      motherBornUS   
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:1.0000  
 Median :1.0000   Median :1.0000   Median :1.0000  
 Mean   :0.8531   Mean   :0.9313   Mean   :0.7725  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.0000  
 NA's   :233      NA's   :69       NA's   :71      
  fatherBornUS    englishAtHome    computerForSchoolwork
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000       
 1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:1.0000       
 Median :1.0000   Median :1.0000   Median :1.0000       
 Mean   :0.7668   Mean   :0.8717   Mean   :0.8994       
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000       
 Max.   :1.0000   Max.   :1.0000   Max.   :1.0000       
 NA's   :113      NA's   :71       NA's   :65           
 read30MinsADay   minutesPerWeekEnglish studentsInEnglish
 Min.   :0.0000   Min.   :   0.0        Min.   : 1.0     
 1st Qu.:0.0000   1st Qu.: 225.0        1st Qu.:20.0     
 Median :0.0000   Median : 250.0        Median :25.0     
 Mean   :0.2899   Mean   : 266.2        Mean   :24.5     
 3rd Qu.:1.0000   3rd Qu.: 300.0        3rd Qu.:30.0     
 Max.   :1.0000   Max.   :2400.0        Max.   :75.0     
 NA's   :34       NA's   :186           NA's   :249      
 schoolHasLibrary  publicSchool        urban          schoolSize  
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   : 100  
 1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:0.0000   1st Qu.: 712  
 Median :1.0000   Median :1.0000   Median :0.0000   Median :1212  
 Mean   :0.9676   Mean   :0.9339   Mean   :0.3849   Mean   :1369  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1900  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.0000   Max.   :6694  
 NA's   :143                                        NA's   :162   
  readingScore  
 Min.   :168.6  
 1st Qu.:431.7  
 Median :499.7  
 Mean   :497.9  
 3rd Qu.:566.2  
 Max.   :746.0

1.4

Linear regression discards observations with missing data, so we will remove all such observations from the training and testing sets. Later in the course, we will learn about imputation, which deals with missing data by filling in missing values with plausible information.

Type the following commands into your R console to remove observations with any missing value from pisaTrain and pisaTest:

nrow(pisaTest)

[1] 990

Section 2 - Factor variables

2.1

Factor variables are variables that take on a discrete set of values, like the “Region” variable in the WHO dataset from the second lecture of Unit 1. This is an unordered factor because there isn’t any natural ordering between the levels. An ordered factor has a natural ordering between the levels (an example would be the classifications “large,” “medium,” and “small”).

Which of the following variables is an unordered factor with at least 3 levels? (Select all that apply.)

Which of the following variables is an ordered factor with at least 3 levels? (Select all that apply.)

head(pisaTest)
#grade

2.2

Now, consider the variable “raceeth” in our problem, which has levels “American Indian/Alaska Native”, “Asian”, “Black”, “Hispanic”, “More than one race”, “Native Hawaiian/Other Pacific Islander”, and “White”. Because it is the most common in our population, we will select White as the reference level.

Which binary variables will be included in the regression model? (Select all that apply.)

#raceethAmerican Indian/Alaska Native
#raceethAsian 
#raceethBlack
#raceethHispanic
#raceethMore than one race 
#raceethNative Hawaiian/Other Pacific Islander

2.3

Consider again adding our unordered factor race to the regression model with reference level “White”.

For a student who is Asian, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)

#raceethAmerican Indian/Alaska Native
#raceethBlack
#raceethHispanic
#raceethMore than one race
#raceethNative Hawaiian/Other Pacific Islander

For a student who is white, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)

#raceethAmerican Indian/Alaska Native
#raceethAsian
#raceethBlack
#raceethHispanic
#raceethMore than one race
#raceethNative Hawaiian/Other Pacific Islander

Section 3 - Building a model

3.1

summary(lmScore)


Call:
lm(formula = readingScore ~ ., data = pisaTrain)

Residuals:
    Min      1Q  Median      3Q     Max 
-247.44  -48.86    1.86   49.77  217.18 

Coefficients:
                                                Estimate
(Intercept)                                   143.766333
grade                                          29.542707
male                                          -14.521653
raceethAmerican Indian/Alaska Native          -67.277327
raceethAsian                                   -4.110325
raceethBlack                                  -67.012347
raceethHispanic                               -38.975486
raceethMore than one race                     -16.922522
raceethNative Hawaiian/Other Pacific Islander  -5.101601
preschool                                      -4.463670
expectBachelors                                55.267080
motherHS                                        6.058774
motherBachelors                                12.638068
motherWork                                     -2.809101
fatherHS                                        4.018214
fatherBachelors                                16.929755
fatherWork                                      5.842798
selfBornUS                                     -3.806278
motherBornUS                                   -8.798153
fatherBornUS                                    4.306994
englishAtHome                                   8.035685
computerForSchoolwork                          22.500232
read30MinsADay                                 34.871924
minutesPerWeekEnglish                           0.012788
studentsInEnglish                              -0.286631
schoolHasLibrary                               12.215085
publicSchool                                  -16.857475
urban                                          -0.110132
schoolSize                                      0.006540
                                              Std. Error t value
(Intercept)                                    33.841226   4.248
grade                                           2.937399  10.057
male                                            3.155926  -4.601
raceethAmerican Indian/Alaska Native           16.786935  -4.008
raceethAsian                                    9.220071  -0.446
raceethBlack                                    5.460883 -12.271
raceethHispanic                                 5.177743  -7.528
raceethMore than one race                       8.496268  -1.992
raceethNative Hawaiian/Other Pacific Islander  17.005696  -0.300
preschool                                       3.486055  -1.280
expectBachelors                                 4.293893  12.871
motherHS                                        6.091423   0.995
motherBachelors                                 3.861457   3.273
motherWork                                      3.521827  -0.798
fatherHS                                        5.579269   0.720
fatherBachelors                                 3.995253   4.237
fatherWork                                      4.395978   1.329
selfBornUS                                      7.323718  -0.520
motherBornUS                                    6.587621  -1.336
fatherBornUS                                    6.263875   0.688
englishAtHome                                   6.859492   1.171
computerForSchoolwork                           5.702562   3.946
read30MinsADay                                  3.408447  10.231
minutesPerWeekEnglish                           0.010712   1.194
studentsInEnglish                               0.227819  -1.258
schoolHasLibrary                                9.264884   1.318
publicSchool                                    6.725614  -2.506
urban                                           3.962724  -0.028
schoolSize                                      0.002197   2.977
                                              Pr(>|t|)    
(Intercept)                                   2.24e-05 ***
grade                                          < 2e-16 ***
male                                          4.42e-06 ***
raceethAmerican Indian/Alaska Native          6.32e-05 ***
raceethAsian                                   0.65578    
raceethBlack                                   < 2e-16 ***
raceethHispanic                               7.29e-14 ***
raceethMore than one race                      0.04651 *  
raceethNative Hawaiian/Other Pacific Islander  0.76421    
preschool                                      0.20052    
expectBachelors                                < 2e-16 ***
motherHS                                       0.32001    
motherBachelors                                0.00108 ** 
motherWork                                     0.42517    
fatherHS                                       0.47147    
fatherBachelors                               2.35e-05 ***
fatherWork                                     0.18393    
selfBornUS                                     0.60331    
motherBornUS                                   0.18182    
fatherBornUS                                   0.49178    
englishAtHome                                  0.24153    
computerForSchoolwork                         8.19e-05 ***
read30MinsADay                                 < 2e-16 ***
minutesPerWeekEnglish                          0.23264    
studentsInEnglish                              0.20846    
schoolHasLibrary                               0.18749    
publicSchool                                   0.01226 *  
urban                                          0.97783    
schoolSize                                     0.00294 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 73.81 on 2385 degrees of freedom
Multiple R-squared:  0.3251,    Adjusted R-squared:  0.3172 
F-statistic: 41.04 on 28 and 2385 DF,  p-value: < 2.2e-16

3.2

What is the training-set root-mean squared error (RMSE) of lmScore?

sqrt(sum((lmScore$residuals^2))/nrow(pisaTrain))

[1] 73.36555

3.3

Consider two students A and B. They have all variable values the same, except that student A is in grade 11 and student B is in grade 9. What is the predicted reading score of student A minus the predicted reading score of student B?

29.542707*2

[1] 59.08541

3.4

What is the meaning of the coefficient associated with variable raceethAsian?

#Predicted difference in the reading score between an Asian student and a white student who is otherwise identical

3.5

Based on the significance codes, which variables are candidates for removal from the model? Select all that apply. (We’ll assume that the factor variable raceeth should only be removed if none of its levels are significant.)

summary(lmScore)


Call:
lm(formula = readingScore ~ ., data = pisaTrain)

Residuals:
    Min      1Q  Median      3Q     Max 
-247.44  -48.86    1.86   49.77  217.18 

Coefficients:
                                                Estimate
(Intercept)                                   143.766333
grade                                          29.542707
male                                          -14.521653
raceethAmerican Indian/Alaska Native          -67.277327
raceethAsian                                   -4.110325
raceethBlack                                  -67.012347
raceethHispanic                               -38.975486
raceethMore than one race                     -16.922522
raceethNative Hawaiian/Other Pacific Islander  -5.101601
preschool                                      -4.463670
expectBachelors                                55.267080
motherHS                                        6.058774
motherBachelors                                12.638068
motherWork                                     -2.809101
fatherHS                                        4.018214
fatherBachelors                                16.929755
fatherWork                                      5.842798
selfBornUS                                     -3.806278
motherBornUS                                   -8.798153
fatherBornUS                                    4.306994
englishAtHome                                   8.035685
computerForSchoolwork                          22.500232
read30MinsADay                                 34.871924
minutesPerWeekEnglish                           0.012788
studentsInEnglish                              -0.286631
schoolHasLibrary                               12.215085
publicSchool                                  -16.857475
urban                                          -0.110132
schoolSize                                      0.006540
                                              Std. Error t value
(Intercept)                                    33.841226   4.248
grade                                           2.937399  10.057
male                                            3.155926  -4.601
raceethAmerican Indian/Alaska Native           16.786935  -4.008
raceethAsian                                    9.220071  -0.446
raceethBlack                                    5.460883 -12.271
raceethHispanic                                 5.177743  -7.528
raceethMore than one race                       8.496268  -1.992
raceethNative Hawaiian/Other Pacific Islander  17.005696  -0.300
preschool                                       3.486055  -1.280
expectBachelors                                 4.293893  12.871
motherHS                                        6.091423   0.995
motherBachelors                                 3.861457   3.273
motherWork                                      3.521827  -0.798
fatherHS                                        5.579269   0.720
fatherBachelors                                 3.995253   4.237
fatherWork                                      4.395978   1.329
selfBornUS                                      7.323718  -0.520
motherBornUS                                    6.587621  -1.336
fatherBornUS                                    6.263875   0.688
englishAtHome                                   6.859492   1.171
computerForSchoolwork                           5.702562   3.946
read30MinsADay                                  3.408447  10.231
minutesPerWeekEnglish                           0.010712   1.194
studentsInEnglish                               0.227819  -1.258
schoolHasLibrary                                9.264884   1.318
publicSchool                                    6.725614  -2.506
urban                                           3.962724  -0.028
schoolSize                                      0.002197   2.977
                                              Pr(>|t|)    
(Intercept)                                   2.24e-05 ***
grade                                          < 2e-16 ***
male                                          4.42e-06 ***
raceethAmerican Indian/Alaska Native          6.32e-05 ***
raceethAsian                                   0.65578    
raceethBlack                                   < 2e-16 ***
raceethHispanic                               7.29e-14 ***
raceethMore than one race                      0.04651 *  
raceethNative Hawaiian/Other Pacific Islander  0.76421    
preschool                                      0.20052    
expectBachelors                                < 2e-16 ***
motherHS                                       0.32001    
motherBachelors                                0.00108 ** 
motherWork                                     0.42517    
fatherHS                                       0.47147    
fatherBachelors                               2.35e-05 ***
fatherWork                                     0.18393    
selfBornUS                                     0.60331    
motherBornUS                                   0.18182    
fatherBornUS                                   0.49178    
englishAtHome                                  0.24153    
computerForSchoolwork                         8.19e-05 ***
read30MinsADay                                 < 2e-16 ***
minutesPerWeekEnglish                          0.23264    
studentsInEnglish                              0.20846    
schoolHasLibrary                               0.18749    
publicSchool                                   0.01226 *  
urban                                          0.97783    
schoolSize                                     0.00294 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 73.81 on 2385 degrees of freedom
Multiple R-squared:  0.3251,    Adjusted R-squared:  0.3172 
F-statistic: 41.04 on 28 and 2385 DF,  p-value: < 2.2e-16

Section 4 - Predicting on unseen data

4.1

Using the “predict” function and supplying the “newdata” argument, use the lmScore model to predict the reading scores of students in pisaTest. Call this vector of predictions “predTest”. Do not change the variables in the model (for example, do not remove variables that we found were not significant in the previous part of this problem). Use the summary function to describe the test set predictions.

What is the range between the maximum and minimum predicted reading score on the test set?

max(predTest)-min(predTest)

[1] 284.4683

4.2

What is the sum of squared errors (SSE) of lmScore on the testing set?

SSE

[1] 5762082

4.3

What is the predicted test score used in the baseline model? Remember to compute this value using the training set and not the test set.

mean(pisaTrain$readingScore)

[1] 517.9629

What is the sum of squared errors of the baseline model on the testing set? HINT: We call the sum of squared errors for the baseline model the total sum of squares (SST).

SSE = sum((pisaTest$readingScore - baseline)^2)

4.4

What is the test-set R-squared value of lmScore?

SSE= sum((pisaTest$readingScore-predTest)^2) 
SST = sum ((pisaTest$readingScore - baseline)^2)
1-SSE/SST
[1] 0.2614944
LS0tDQp0aXRsZTogIkFTMi0yIFJlYWRpbmcgVGVzdCBTY29yZXMiDQphdXRob3I6ICI8546L5qyjPiA8TTA2NDExMTAzOT4iDQpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sNCi0tLQ0KIyMjIFNlY3Rpb24gMSAtIENyZWF0aW5nIE91ciBGaXJzdCBNb2RlbA0KDQojIyMjIDEuMSANCkxvYWQgdGhlIHRyYWluaW5nIGFuZCB0ZXN0aW5nIHNldHMgdXNpbmcgdGhlIHJlYWQuY3N2KCkgZnVuY3Rpb24sIGFuZCBzYXZlIHRoZW0gYXMgdmFyaWFibGVzIHdpdGggdGhlIG5hbWVzIHBpc2FUcmFpbiBhbmQgcGlzYVRlc3QuDQoNCkhvdyBtYW55IHN0dWRlbnRzIGFyZSB0aGVyZSBpbiB0aGUgdHJhaW5pbmcgc2V0Pw0KDQpgYGB7cn0NClN5cy5zZXRsb2NhbGUoIkxDX0FMTCIsIkMiKQ0KcGlzYVRyYWluID0gcmVhZC5jc3YoImRhdGEvcGlzYTIwMDl0cmFpbi5jc3YiKQ0KcGlzYVRlc3QgPSByZWFkLmNzdigiZGF0YS9waXNhMjAwOXRlc3QuY3N2IikNCm5yb3cocGlzYVRyYWluKQ0KYGBgDQoNCiMjIyMgMS4yDQoNClVzaW5nIHRhcHBseSgpIG9uIHBpc2FUcmFpbiwgd2hhdCBpcyB0aGUgYXZlcmFnZSByZWFkaW5nIHRlc3Qgc2NvcmUgb2YgbWFsZXM/T2YgZmVtYWxlcz8NCmBgYHtyfQ0KbmFtZXMocGlzYVRyYWluKQ0KQSA9IHRhcHBseShwaXNhVHJhaW4kcmVhZGluZ1Njb3JlLHBpc2FUcmFpbiRtYWxlLG1lYW4pDQpBDQpgYGANCg0KIyMjIyAxLjMNCldoaWNoIHZhcmlhYmxlcyBhcmUgbWlzc2luZyBkYXRhIGluIGF0IGxlYXN0IG9uZSBvYnNlcnZhdGlvbiBpbiB0aGUgdHJhaW5pbmcgc2V0PyBTZWxlY3QgYWxsIHRoYXQgYXBwbHkuDQpgYGB7cn0NCnN1bW1hcnkocGlzYVRyYWluKQ0KI3JhY2VldGgscHJlc2Nob29sLGV4cGVjdEJhY2hlbG9ycyxtb3RoZXJIUyxtb3RoZXJCYWNoZWxvcnMsbW90aGVyV29yayxmYXRoZXJIUyxmYXRoZXJCYWNoZWxvcnMsZmF0aGVyV29yayxzZWxmQm9yblVTLG1vdGhlckJvcm5VUyxmYXRoZXJCb3JuVVMsZW5nbGlzaEF0SG9tZSxjb21wdXRlckZvclNjaG9vbHdvcmsscmVhZDMwTWluc0FEYXkgICxtaW51dGVzUGVyV2Vla0VuZ2xpc2gsc3R1ZGVudHNJbkVuZ2xpc2gsc2Nob29sSGFzTGlicmFyeSxzY2hvb2xTaXplDQpgYGANCg0KIyMjIyAxLjQNCkxpbmVhciByZWdyZXNzaW9uIGRpc2NhcmRzIG9ic2VydmF0aW9ucyB3aXRoIG1pc3NpbmcgZGF0YSwgc28gd2Ugd2lsbCByZW1vdmUgYWxsIHN1Y2ggb2JzZXJ2YXRpb25zIGZyb20gdGhlIHRyYWluaW5nIGFuZCB0ZXN0aW5nIHNldHMuIExhdGVyIGluIHRoZSBjb3Vyc2UsIHdlIHdpbGwgbGVhcm4gYWJvdXQgaW1wdXRhdGlvbiwgd2hpY2ggZGVhbHMgd2l0aCBtaXNzaW5nIGRhdGEgYnkgZmlsbGluZyBpbiBtaXNzaW5nIHZhbHVlcyB3aXRoIHBsYXVzaWJsZSBpbmZvcm1hdGlvbi4NCg0KVHlwZSB0aGUgZm9sbG93aW5nIGNvbW1hbmRzIGludG8geW91ciBSIGNvbnNvbGUgdG8gcmVtb3ZlIG9ic2VydmF0aW9ucyB3aXRoIGFueSBtaXNzaW5nIHZhbHVlIGZyb20gcGlzYVRyYWluIGFuZCBwaXNhVGVzdDoNCmBgYHtyfQ0KcGlzYVRyYWluID0gbmEub21pdChwaXNhVHJhaW4pDQpwaXNhVGVzdCA9IG5hLm9taXQocGlzYVRlc3QpDQpucm93KHBpc2FUcmFpbikNCm5yb3cocGlzYVRlc3QpDQpgYGANCiMjIyBTZWN0aW9uIDIgLSBGYWN0b3IgdmFyaWFibGVzDQoNCiMjIyMgMi4xDQpGYWN0b3IgdmFyaWFibGVzIGFyZSB2YXJpYWJsZXMgdGhhdCB0YWtlIG9uIGEgZGlzY3JldGUgc2V0IG9mIHZhbHVlcywgbGlrZSB0aGUgIlJlZ2lvbiIgdmFyaWFibGUgaW4gdGhlIFdITyBkYXRhc2V0IGZyb20gdGhlIHNlY29uZCBsZWN0dXJlIG9mIFVuaXQgMS4gVGhpcyBpcyBhbiB1bm9yZGVyZWQgZmFjdG9yIGJlY2F1c2UgdGhlcmUgaXNuJ3QgYW55IG5hdHVyYWwgb3JkZXJpbmcgYmV0d2VlbiB0aGUgbGV2ZWxzLiBBbiBvcmRlcmVkIGZhY3RvciBoYXMgYSBuYXR1cmFsIG9yZGVyaW5nIGJldHdlZW4gdGhlIGxldmVscyAoYW4gZXhhbXBsZSB3b3VsZCBiZSB0aGUgY2xhc3NpZmljYXRpb25zICJsYXJnZSwiICJtZWRpdW0sIiBhbmQgInNtYWxsIikuDQoNCldoaWNoIG9mIHRoZSBmb2xsb3dpbmcgdmFyaWFibGVzIGlzIGFuIHVub3JkZXJlZCBmYWN0b3Igd2l0aCBhdCBsZWFzdCAzIGxldmVscz8gKFNlbGVjdCBhbGwgdGhhdCBhcHBseS4pDQoNCmBgYHtyfQ0KaGVhZChwaXNhVGVzdCkNCiNyYWNlZXRoDQpgYGANCldoaWNoIG9mIHRoZSBmb2xsb3dpbmcgdmFyaWFibGVzIGlzIGFuIG9yZGVyZWQgZmFjdG9yIHdpdGggYXQgbGVhc3QgMyBsZXZlbHM/IChTZWxlY3QgYWxsIHRoYXQgYXBwbHkuKQ0KYGBge3J9DQpoZWFkKHBpc2FUZXN0KQ0KI2dyYWRlDQpgYGANCg0KIyMjIyAyLjINCk5vdywgY29uc2lkZXIgdGhlIHZhcmlhYmxlICJyYWNlZXRoIiBpbiBvdXIgcHJvYmxlbSwgd2hpY2ggaGFzIGxldmVscyAiQW1lcmljYW4gSW5kaWFuL0FsYXNrYSBOYXRpdmUiLCAiQXNpYW4iLCAiQmxhY2siLCAiSGlzcGFuaWMiLCAiTW9yZSB0aGFuIG9uZSByYWNlIiwgIk5hdGl2ZSBIYXdhaWlhbi9PdGhlciBQYWNpZmljIElzbGFuZGVyIiwgYW5kICJXaGl0ZSIuIEJlY2F1c2UgaXQgaXMgdGhlIG1vc3QgY29tbW9uIGluIG91ciBwb3B1bGF0aW9uLCB3ZSB3aWxsIHNlbGVjdCBXaGl0ZSBhcyB0aGUgcmVmZXJlbmNlIGxldmVsLg0KDQpXaGljaCBiaW5hcnkgdmFyaWFibGVzIHdpbGwgYmUgaW5jbHVkZWQgaW4gdGhlIHJlZ3Jlc3Npb24gbW9kZWw/IChTZWxlY3QgYWxsIHRoYXQgYXBwbHkuKQ0KYGBge3J9DQojcmFjZWV0aEFtZXJpY2FuIEluZGlhbi9BbGFza2EgTmF0aXZlDQojcmFjZWV0aEFzaWFuIA0KI3JhY2VldGhCbGFjaw0KI3JhY2VldGhIaXNwYW5pYw0KI3JhY2VldGhNb3JlIHRoYW4gb25lIHJhY2UgDQojcmFjZWV0aE5hdGl2ZSBIYXdhaWlhbi9PdGhlciBQYWNpZmljIElzbGFuZGVyDQpgYGANCg0KIyMjIyAyLjMNCkNvbnNpZGVyIGFnYWluIGFkZGluZyBvdXIgdW5vcmRlcmVkIGZhY3RvciByYWNlIHRvIHRoZSByZWdyZXNzaW9uIG1vZGVsIHdpdGggcmVmZXJlbmNlIGxldmVsICJXaGl0ZSIuDQoNCkZvciBhIHN0dWRlbnQgd2hvIGlzIEFzaWFuLCB3aGljaCBiaW5hcnkgdmFyaWFibGVzIHdvdWxkIGJlIHNldCB0byAwPyBBbGwgcmVtYWluaW5nIHZhcmlhYmxlcyB3aWxsIGJlIHNldCB0byAxLiAoU2VsZWN0IGFsbCB0aGF0IGFwcGx5LikNCmBgYHtyfQ0KI3JhY2VldGhBbWVyaWNhbiBJbmRpYW4vQWxhc2thIE5hdGl2ZQ0KI3JhY2VldGhCbGFjaw0KI3JhY2VldGhIaXNwYW5pYw0KI3JhY2VldGhNb3JlIHRoYW4gb25lIHJhY2UNCiNyYWNlZXRoTmF0aXZlIEhhd2FpaWFuL090aGVyIFBhY2lmaWMgSXNsYW5kZXINCmBgYA0KRm9yIGEgc3R1ZGVudCB3aG8gaXMgd2hpdGUsIHdoaWNoIGJpbmFyeSB2YXJpYWJsZXMgd291bGQgYmUgc2V0IHRvIDA/IEFsbCByZW1haW5pbmcgdmFyaWFibGVzIHdpbGwgYmUgc2V0IHRvIDEuIChTZWxlY3QgYWxsIHRoYXQgYXBwbHkuKQ0KYGBge3J9DQojcmFjZWV0aEFtZXJpY2FuIEluZGlhbi9BbGFza2EgTmF0aXZlDQojcmFjZWV0aEFzaWFuDQojcmFjZWV0aEJsYWNrDQojcmFjZWV0aEhpc3BhbmljDQojcmFjZWV0aE1vcmUgdGhhbiBvbmUgcmFjZQ0KI3JhY2VldGhOYXRpdmUgSGF3YWlpYW4vT3RoZXIgUGFjaWZpYyBJc2xhbmRlcg0KYGBgDQoNCiMjIyBTZWN0aW9uIDMgLSBCdWlsZGluZyBhIG1vZGVsDQoNCiMjIyMgMy4xDQpgYGB7cn0NCnBpc2FUcmFpbiRyYWNlZXRoID0gcmVsZXZlbChwaXNhVHJhaW4kcmFjZWV0aCwgIldoaXRlIikNCnBpc2FUZXN0JHJhY2VldGggPSByZWxldmVsKHBpc2FUZXN0JHJhY2VldGgsICJXaGl0ZSIpDQpuYW1lcyhwaXNhVHJhaW4pDQpsbVNjb3JlID0gbG0ocmVhZGluZ1Njb3JlIH4gLixkYXRhPXBpc2FUcmFpbikNCnN1bW1hcnkobG1TY29yZSkNCiNsbVNjb3JlDQpgYGANCg0KIyMjIyAzLjINCldoYXQgaXMgdGhlIHRyYWluaW5nLXNldCByb290LW1lYW4gc3F1YXJlZCBlcnJvciAoUk1TRSkgb2YgbG1TY29yZT8NCmBgYHtyfQ0Kc3FydChzdW0oKGxtU2NvcmUkcmVzaWR1YWxzXjIpKS9ucm93KHBpc2FUcmFpbikpDQpgYGANCg0KIyMjIyAzLjMNCkNvbnNpZGVyIHR3byBzdHVkZW50cyBBIGFuZCBCLiBUaGV5IGhhdmUgYWxsIHZhcmlhYmxlIHZhbHVlcyB0aGUgc2FtZSwgZXhjZXB0IHRoYXQgc3R1ZGVudCBBIGlzIGluIGdyYWRlIDExIGFuZCBzdHVkZW50IEIgaXMgaW4gZ3JhZGUgOS4gV2hhdCBpcyB0aGUgcHJlZGljdGVkIHJlYWRpbmcgc2NvcmUgb2Ygc3R1ZGVudCBBIG1pbnVzIHRoZSBwcmVkaWN0ZWQgcmVhZGluZyBzY29yZSBvZiBzdHVkZW50IEI/DQpgYGB7cn0NCnN1bW1hcnkobG1TY29yZSkNCjI5LjU0MjcwNyoyDQpgYGANCg0KIyMjIyAzLjQNCldoYXQgaXMgdGhlIG1lYW5pbmcgb2YgdGhlIGNvZWZmaWNpZW50IGFzc29jaWF0ZWQgd2l0aCB2YXJpYWJsZSByYWNlZXRoQXNpYW4/DQpgYGB7cn0NCiNQcmVkaWN0ZWQgZGlmZmVyZW5jZSBpbiB0aGUgcmVhZGluZyBzY29yZSBiZXR3ZWVuIGFuIEFzaWFuIHN0dWRlbnQgYW5kIGEgd2hpdGUgc3R1ZGVudCB3aG8gaXMgb3RoZXJ3aXNlIGlkZW50aWNhbCANCmBgYA0KDQojIyMjIDMuNQ0KQmFzZWQgb24gdGhlIHNpZ25pZmljYW5jZSBjb2Rlcywgd2hpY2ggdmFyaWFibGVzIGFyZSBjYW5kaWRhdGVzIGZvciByZW1vdmFsIGZyb20gdGhlIG1vZGVsPyBTZWxlY3QgYWxsIHRoYXQgYXBwbHkuIChXZSdsbCBhc3N1bWUgdGhhdCB0aGUgZmFjdG9yIHZhcmlhYmxlIHJhY2VldGggc2hvdWxkIG9ubHkgYmUgcmVtb3ZlZCBpZiBub25lIG9mIGl0cyBsZXZlbHMgYXJlIHNpZ25pZmljYW50LikNCmBgYHtyfQ0Kc3VtbWFyeShsbVNjb3JlKQ0KI3ByZXNjaG9vbCxtb3RoZXJIUyxtb3RoZXJXb3JrLGZhdGhlckhTLGZhdGhlcldvcmssc2VsZkJvcm5VUyxtb3RoZXJCb3JuVVMsZmF0aGVyQm9yblVTLGVuZ2xpc2hBdEhvbWUsbWludXRlc1BlcldlZWtFbmdsaXNoLHN0dWRlbnRzSW5FbmdsaXNoLHNjaG9vbEhhc0xpYnJhcnksdXJiYW4NCmBgYA0KIyMjIFNlY3Rpb24gNCAtIFByZWRpY3Rpbmcgb24gdW5zZWVuIGRhdGENCg0KIyMjIyA0LjENClVzaW5nIHRoZSAicHJlZGljdCIgZnVuY3Rpb24gYW5kIHN1cHBseWluZyB0aGUgIm5ld2RhdGEiIGFyZ3VtZW50LCB1c2UgdGhlIGxtU2NvcmUgbW9kZWwgdG8gcHJlZGljdCB0aGUgcmVhZGluZyBzY29yZXMgb2Ygc3R1ZGVudHMgaW4gcGlzYVRlc3QuIENhbGwgdGhpcyB2ZWN0b3Igb2YgcHJlZGljdGlvbnMgInByZWRUZXN0Ii4gRG8gbm90IGNoYW5nZSB0aGUgdmFyaWFibGVzIGluIHRoZSBtb2RlbCAoZm9yIGV4YW1wbGUsIGRvIG5vdCByZW1vdmUgdmFyaWFibGVzIHRoYXQgd2UgZm91bmQgd2VyZSBub3Qgc2lnbmlmaWNhbnQgaW4gdGhlIHByZXZpb3VzIHBhcnQgb2YgdGhpcyBwcm9ibGVtKS4gVXNlIHRoZSBzdW1tYXJ5IGZ1bmN0aW9uIHRvIGRlc2NyaWJlIHRoZSB0ZXN0IHNldCBwcmVkaWN0aW9ucy4NCg0KV2hhdCBpcyB0aGUgcmFuZ2UgYmV0d2VlbiB0aGUgbWF4aW11bSBhbmQgbWluaW11bSBwcmVkaWN0ZWQgcmVhZGluZyBzY29yZSBvbiB0aGUgdGVzdCBzZXQ/DQpgYGB7cn0NCnByZWRUZXN0ID0gcHJlZGljdChsbVNjb3JlLHBpc2FUZXN0KQ0Kc3VtbWFyeShwcmVkVGVzdCkNCm1heChwcmVkVGVzdCktbWluKHByZWRUZXN0KQ0KYGBgDQoNCiMjIyMgNC4yDQpXaGF0IGlzIHRoZSBzdW0gb2Ygc3F1YXJlZCBlcnJvcnMgKFNTRSkgb2YgbG1TY29yZSBvbiB0aGUgdGVzdGluZyBzZXQ/DQpgYGB7cn0NClNTRT0gc3VtKChwaXNhVGVzdCRyZWFkaW5nU2NvcmUtcHJlZFRlc3QpXjIpIA0KUk1TRSA9IHNxcnQoU1NFL25yb3cocGlzYVRlc3QpKQ0KYGBgDQojIyMjIDQuMw0KV2hhdCBpcyB0aGUgcHJlZGljdGVkIHRlc3Qgc2NvcmUgdXNlZCBpbiB0aGUgYmFzZWxpbmUgbW9kZWw/IFJlbWVtYmVyIHRvIGNvbXB1dGUgdGhpcyB2YWx1ZSB1c2luZyB0aGUgdHJhaW5pbmcgc2V0IGFuZCBub3QgdGhlIHRlc3Qgc2V0Lg0KDQpgYGB7cn0NCmJhc2VsaW5lID0gbWVhbihwaXNhVHJhaW4kcmVhZGluZ1Njb3JlKQ0KYmFzZWxpbmUNCmBgYA0KV2hhdCBpcyB0aGUgc3VtIG9mIHNxdWFyZWQgZXJyb3JzIG9mIHRoZSBiYXNlbGluZSBtb2RlbCBvbiB0aGUgdGVzdGluZyBzZXQ/IEhJTlQ6IFdlIGNhbGwgdGhlIHN1bSBvZiBzcXVhcmVkIGVycm9ycyBmb3IgdGhlIGJhc2VsaW5lIG1vZGVsIHRoZSB0b3RhbCBzdW0gb2Ygc3F1YXJlcyAoU1NUKS4NCg0KYGBge3J9DQpTU0UgPSBzdW0oKHBpc2FUZXN0JHJlYWRpbmdTY29yZSAtIGJhc2VsaW5lKV4yKQ0KYGBgDQoNCiMjIyMgNC40DQpXaGF0IGlzIHRoZSB0ZXN0LXNldCBSLXNxdWFyZWQgdmFsdWUgb2YgbG1TY29yZT8NCmBgYHtyfQ0KU1NFPSBzdW0oKHBpc2FUZXN0JHJlYWRpbmdTY29yZS1wcmVkVGVzdCleMikgDQpTU1QgPSBzdW0gKChwaXNhVGVzdCRyZWFkaW5nU2NvcmUgLSBiYXNlbGluZSleMikNCjEtU1NFL1NTVA0KYGBg