AS2-2 Reading Test Scores

edX assignment link: http://bit.ly/2KE2g00

The Programme for International Student Assessment (PISA) is a test given every three years to 15-year-old students from around the world to evaluate their performance in mathematics, reading, and science. This test provides a quantitative way to compare the performance of students from different parts of the world. In this homework assignment, we will predict the reading scores of students from the United States of America on the 2009 PISA exam.

The datasets pisa2009train.csv and pisa2009test.csv contain information about the demographics and schools for American students taking the exam, derived from 2009 PISA Public-Use Data Files distributed by the United States National Center for Education Statistics (NCES). While the datasets are not supposed to contain identifying information about students taking the test, by using the data you are bound by the NCES data use agreement, which prohibits any attempt to determine the identity of any student in the datasets.

Each row in the datasets pisa2009train.csv and pisa2009test.csv represents one student taking the exam. The datasets have the following variables:

grade: The grade in school of the student (most 15-year-olds in America are in 10th grade)
male: Whether the student is male (1/0)
raceeth: The race/ethnicity composite of the student
preschool: Whether the student attended preschool (1/0)
expectBachelors: Whether the student expects to obtain a bachelor’s degree (1/0)
motherHS: Whether the student’s mother completed high school (1/0)
motherBachelors: Whether the student’s mother obtained a bachelor’s degree (1/0)
motherWork: Whether the student’s mother has part-time or full-time work (1/0)
fatherHS: Whether the student’s father completed high school (1/0)
fatherBachelors: Whether the student’s father obtained a bachelor’s degree (1/0)
fatherWork: Whether the student’s father has part-time or full-time work (1/0)
selfBornUS: Whether the student was born in the United States of America (1/0)
motherBornUS: Whether the student’s mother was born in the United States of America (1/0)
fatherBornUS: Whether the student’s father was born in the United States of America (1/0)
englishAtHome: Whether the student speaks English at home (1/0)
computerForSchoolwork: Whether the student has access to a computer for schoolwork (1/0)
read30MinsADay: Whether the student reads for pleasure for 30 minutes/day (1/0)
minutesPerWeekEnglish: The number of minutes per week the student spend in English class
studentsInEnglish: The number of students in this student’s English class at school
schoolHasLibrary: Whether this student’s school has a library (1/0)
publicSchool: Whether this student attends a public school (1/0)
urban: Whether this student’s school is in an urban area (1/0)
schoolSize: The number of students in this student’s school
readingScore: The student’s reading score, on a 1000-point scale

Section 1

1.1 Dataset size

Load the training and testing sets using the read.csv() function, and save them as variables with the names pisaTrain and pisaTest.

How many students are there in the training set?

pisaTrain = read.csv("D:/buiness_analytics/unit2/data/pisa2009train.csv")
pisaTest = read.csv("D:/buiness_analytics/unit2/data/pisa2009test.csv")
nrow(pisaTrain)

[1] 3663

1.2 Summarizing the dataset

Using tapply() on pisaTrain, what is the average reading test score of males?

tapply(pisaTrain$readingScore,pisaTrain$male,mean)

       0        1 
512.9406 483.5325

#males = 483.5325

Of females?

tapply(pisaTrain$readingScore,pisaTrain$male,mean)

       0        1 
512.9406 483.5325

#females = 512.9406

1.3 Locating missing values

Which variables are missing data in at least one observation in the training set? Select all that apply.

grade
male
raceeth
preschool
expectBachelors
motherHS
motherBachelors
motherWork
fatherHS
fatherBachelors
fatherWork
selfBornUS
motherBornUS
fatherBornUS
englishAtHome
computerForSchoolwork
read30MinsADay
minutesPerWeekEnglish
studentsInEnglish
schoolHasLibrary
publicSchool
urban
schoolSize
readingScore

summary(pisaTrain)

     grade            male                      raceeth       preschool     
 Min.   : 8.00   Min.   :0.0000   White             :2015   Min.   :0.0000  
 1st Qu.:10.00   1st Qu.:0.0000   Hispanic          : 834   1st Qu.:0.0000  
 Median :10.00   Median :1.0000   Black             : 444   Median :1.0000  
 Mean   :10.09   Mean   :0.5111   Asian             : 143   Mean   :0.7228  
 3rd Qu.:10.00   3rd Qu.:1.0000   More than one race: 124   3rd Qu.:1.0000  
 Max.   :12.00   Max.   :1.0000   (Other)           :  68   Max.   :1.0000  
                                  NA's              :  35   NA's   :56      
 expectBachelors     motherHS    motherBachelors    motherWork        fatherHS     
 Min.   :0.0000   Min.   :0.00   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:1.0000   1st Qu.:1.00   1st Qu.:0.0000   1st Qu.:0.0000   1st Qu.:1.0000  
 Median :1.0000   Median :1.00   Median :0.0000   Median :1.0000   Median :1.0000  
 Mean   :0.7859   Mean   :0.88   Mean   :0.3481   Mean   :0.7345   Mean   :0.8593  
 3rd Qu.:1.0000   3rd Qu.:1.00   3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000  
 Max.   :1.0000   Max.   :1.00   Max.   :1.0000   Max.   :1.0000   Max.   :1.0000  
 NA's   :62       NA's   :97     NA's   :397      NA's   :93       NA's   :245     
 fatherBachelors    fatherWork       selfBornUS      motherBornUS     fatherBornUS   
 Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.0000   1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:1.0000  
 Median :0.0000   Median :1.0000   Median :1.0000   Median :1.0000   Median :1.0000  
 Mean   :0.3319   Mean   :0.8531   Mean   :0.9313   Mean   :0.7725   Mean   :0.7668  
 3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.0000   Max.   :1.0000   Max.   :1.0000  
 NA's   :569      NA's   :233      NA's   :69       NA's   :71       NA's   :113     
 englishAtHome    computerForSchoolwork read30MinsADay   minutesPerWeekEnglish
 Min.   :0.0000   Min.   :0.0000        Min.   :0.0000   Min.   :   0.0       
 1st Qu.:1.0000   1st Qu.:1.0000        1st Qu.:0.0000   1st Qu.: 225.0       
 Median :1.0000   Median :1.0000        Median :0.0000   Median : 250.0       
 Mean   :0.8717   Mean   :0.8994        Mean   :0.2899   Mean   : 266.2       
 3rd Qu.:1.0000   3rd Qu.:1.0000        3rd Qu.:1.0000   3rd Qu.: 300.0       
 Max.   :1.0000   Max.   :1.0000        Max.   :1.0000   Max.   :2400.0       
 NA's   :71       NA's   :65            NA's   :34       NA's   :186          
 studentsInEnglish schoolHasLibrary  publicSchool        urban          schoolSize  
 Min.   : 1.0      Min.   :0.0000   Min.   :0.0000   Min.   :0.0000   Min.   : 100  
 1st Qu.:20.0      1st Qu.:1.0000   1st Qu.:1.0000   1st Qu.:0.0000   1st Qu.: 712  
 Median :25.0      Median :1.0000   Median :1.0000   Median :0.0000   Median :1212  
 Mean   :24.5      Mean   :0.9676   Mean   :0.9339   Mean   :0.3849   Mean   :1369  
 3rd Qu.:30.0      3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1.0000   3rd Qu.:1900  
 Max.   :75.0      Max.   :1.0000   Max.   :1.0000   Max.   :1.0000   Max.   :6694  
 NA's   :249       NA's   :143                                        NA's   :162   
  readingScore  
 Min.   :168.6  
 1st Qu.:431.7  
 Median :499.7  
 Mean   :497.9  
 3rd Qu.:566.2  
 Max.   :746.0

#raceeth,preschool,expectBachelors,motherHS,motherBachelors,motherWork ,fatherHS
#fatherBachelors,fatherWork,selfBornUS,motherBornUS,fatherBornUS,englishAtHome
#computerForSchoolwork,read30MinsADay,minutesPerWeekEnglish,studentsInEnglish
#schoolHasLibrary,schoolSize

1.4 Removing missing values

Linear regression discards observations with missing data, so we will remove all such observations from the training and testing sets. Later in the course, we will learn about imputation, which deals with missing data by filling in missing values with plausible information.

Type the following commands into your R console to remove observations with any missing value from pisaTrain and pisaTest:

pisaTrain = na.omit(pisaTrain)

pisaTest = na.omit(pisaTest)

How many observations are now in the training set?

pisaTrain = na.omit(pisaTrain)
str(pisaTrain)

'data.frame':   2414 obs. of  24 variables:
 $ grade                : int  11 10 10 10 10 10 10 10 11 9 ...
 $ male                 : int  1 0 1 0 1 0 0 0 1 1 ...
 $ raceeth              : Factor w/ 7 levels "American Indian/Alaska Native",..: 7 3 4 7 5 4 7 4 7 7 ...
 $ preschool            : int  0 1 1 1 1 1 1 1 1 1 ...
 $ expectBachelors      : int  0 1 0 1 1 1 1 0 1 1 ...
 $ motherHS             : int  1 0 1 1 1 1 1 0 1 1 ...
 $ motherBachelors      : int  1 0 0 0 1 0 0 0 0 1 ...
 $ motherWork           : int  1 1 1 0 1 1 1 0 0 1 ...
 $ fatherHS             : int  1 1 1 1 0 1 1 0 1 1 ...
 $ fatherBachelors      : int  0 0 0 0 0 0 1 0 1 1 ...
 $ fatherWork           : int  1 1 0 1 1 0 1 1 1 1 ...
 $ selfBornUS           : int  1 1 1 1 1 0 1 0 1 1 ...
 $ motherBornUS         : int  1 1 1 1 1 0 1 0 1 1 ...
 $ fatherBornUS         : int  1 1 0 1 1 0 1 0 1 1 ...
 $ englishAtHome        : int  1 1 1 1 1 0 1 0 1 1 ...
 $ computerForSchoolwork: int  1 1 1 1 1 0 1 1 1 1 ...
 $ read30MinsADay       : int  1 1 1 1 0 1 1 1 0 0 ...
 $ minutesPerWeekEnglish: int  450 200 250 300 294 232 225 270 275 225 ...
 $ studentsInEnglish    : int  25 23 35 30 24 14 20 25 30 15 ...
 $ schoolHasLibrary     : int  1 1 1 1 1 1 1 1 1 1 ...
 $ publicSchool         : int  1 1 1 1 1 1 1 1 1 0 ...
 $ urban                : int  0 1 1 0 0 0 0 1 1 1 ...
 $ schoolSize           : int  1173 2640 1095 1913 899 1733 149 1400 1988 915 ...
 $ readingScore         : num  575 458 614 439 466 ...
 - attr(*, "na.action")= 'omit' Named int  1 3 6 7 9 11 13 21 29 30 ...
  ..- attr(*, "names")= chr  "1" "3" "6" "7" ...

#2414

How many observations are now in the testing set?

pisaTest = na.omit(pisaTest)
str(pisaTest)

'data.frame':   990 obs. of  24 variables:
 $ grade                : int  10 10 10 10 11 10 10 10 10 10 ...
 $ male                 : int  0 0 0 0 0 1 0 1 1 0 ...
 $ raceeth              : Factor w/ 7 levels "American Indian/Alaska Native",..: 7 7 1 7 7 4 7 4 7 4 ...
 $ preschool            : int  1 1 1 1 0 1 0 1 1 1 ...
 $ expectBachelors      : int  0 1 0 0 0 1 1 0 1 1 ...
 $ motherHS             : int  1 1 1 1 1 1 1 1 1 1 ...
 $ motherBachelors      : int  1 0 0 0 1 1 0 0 1 0 ...
 $ motherWork           : int  1 0 0 1 1 1 0 1 1 1 ...
 $ fatherHS             : int  1 1 1 1 1 1 1 1 1 1 ...
 $ fatherBachelors      : int  0 1 0 0 1 0 0 0 1 1 ...
 $ fatherWork           : int  0 1 0 1 1 1 1 0 1 1 ...
 $ selfBornUS           : int  1 1 1 1 1 1 1 1 1 1 ...
 $ motherBornUS         : int  1 1 1 1 1 1 1 1 1 1 ...
 $ fatherBornUS         : int  1 1 1 1 1 1 1 1 1 1 ...
 $ englishAtHome        : int  1 1 1 1 1 1 1 1 1 1 ...
 $ computerForSchoolwork: int  1 1 1 1 1 1 1 1 1 1 ...
 $ read30MinsADay       : int  0 0 1 1 1 1 0 0 0 1 ...
 $ minutesPerWeekEnglish: int  240 240 240 270 270 350 350 360 350 360 ...
 $ studentsInEnglish    : int  30 30 30 35 30 25 27 28 25 27 ...
 $ schoolHasLibrary     : int  1 1 1 1 1 1 1 1 1 1 ...
 $ publicSchool         : int  1 1 1 1 1 1 1 1 1 1 ...
 $ urban                : int  0 0 0 0 0 0 0 0 0 0 ...
 $ schoolSize           : int  808 808 808 808 808 899 899 899 899 899 ...
 $ readingScore         : num  355 454 405 665 605 ...
 - attr(*, "na.action")= 'omit' Named int  2 3 4 6 12 16 17 19 22 23 ...
  ..- attr(*, "names")= chr  "2" "3" "4" "6" ...

#990

Section 2

2.1

Factor variables are variables that take on a discrete set of values, like the “Region” variable in the WHO dataset from the second lecture of Unit 1. This is an unordered factor because there isn’t any natural ordering between the levels. An ordered factor has a natural ordering between the levels (an example would be the classifications “large,” “medium,” and “small”).

Which of the following variables is an unordered factor with at least 3 levels? (Select all that apply.)

grade
male
raceeth

str(pisaTrain)

'data.frame':   2414 obs. of  24 variables:
 $ grade                : int  11 10 10 10 10 10 10 10 11 9 ...
 $ male                 : int  1 0 1 0 1 0 0 0 1 1 ...
 $ raceeth              : Factor w/ 7 levels "American Indian/Alaska Native",..: 7 3 4 7 5 4 7 4 7 7 ...
 $ preschool            : int  0 1 1 1 1 1 1 1 1 1 ...
 $ expectBachelors      : int  0 1 0 1 1 1 1 0 1 1 ...
 $ motherHS             : int  1 0 1 1 1 1 1 0 1 1 ...
 $ motherBachelors      : int  1 0 0 0 1 0 0 0 0 1 ...
 $ motherWork           : int  1 1 1 0 1 1 1 0 0 1 ...
 $ fatherHS             : int  1 1 1 1 0 1 1 0 1 1 ...
 $ fatherBachelors      : int  0 0 0 0 0 0 1 0 1 1 ...
 $ fatherWork           : int  1 1 0 1 1 0 1 1 1 1 ...
 $ selfBornUS           : int  1 1 1 1 1 0 1 0 1 1 ...
 $ motherBornUS         : int  1 1 1 1 1 0 1 0 1 1 ...
 $ fatherBornUS         : int  1 1 0 1 1 0 1 0 1 1 ...
 $ englishAtHome        : int  1 1 1 1 1 0 1 0 1 1 ...
 $ computerForSchoolwork: int  1 1 1 1 1 0 1 1 1 1 ...
 $ read30MinsADay       : int  1 1 1 1 0 1 1 1 0 0 ...
 $ minutesPerWeekEnglish: int  450 200 250 300 294 232 225 270 275 225 ...
 $ studentsInEnglish    : int  25 23 35 30 24 14 20 25 30 15 ...
 $ schoolHasLibrary     : int  1 1 1 1 1 1 1 1 1 1 ...
 $ publicSchool         : int  1 1 1 1 1 1 1 1 1 0 ...
 $ urban                : int  0 1 1 0 0 0 0 1 1 1 ...
 $ schoolSize           : int  1173 2640 1095 1913 899 1733 149 1400 1988 915 ...
 $ readingScore         : num  575 458 614 439 466 ...
 - attr(*, "na.action")= 'omit' Named int  1 3 6 7 9 11 13 21 29 30 ...
  ..- attr(*, "names")= chr  "1" "3" "6" "7" ...

#raceeth ->為Factor

Which of the following variables is an ordered factor with at least 3 levels? (Select all that apply.)

#factor(pisaTrain$grade)
#grade ->資料讀進來的時候會自動轉換成int,而grade會做為分類的依據，且滿足3個以上層級的條件，因此是Factor

2.2 Unordered factors in regression models

To include unordered factors in a linear regression model, we define one level as the “reference level” and add a binary variable for each of the remaining levels. In this way, a factor with n levels is replaced by n-1 binary variables. The reference level is typically selected to be the most frequently occurring level in the dataset.

As an example, consider the unordered factor variable “color”, with levels “red”, “green”, and “blue”. If “green” were the reference level, then we would add binary variables “colorred” and “colorblue” to a linear regression problem. All red examples would have colorred=1 and colorblue=0. All blue examples would have colorred=0 and colorblue=1. All green examples would have colorred=0 and colorblue=0.

Now, consider the variable “raceeth” in our problem, which has levels “American Indian/Alaska Native”, “Asian”, “Black”, “Hispanic”, “More than one race”, “Native Hawaiian/Other Pacific Islander”, and “White”. Because it is the most common in our population, we will select White as the reference level.

Which binary variables will be included in the regression model? (Select all that apply.)

summary(pisaTrain$raceeth)

         American Indian/Alaska Native                                  Asian 
                                    20                                     95 
                                 Black                               Hispanic 
                                   228                                    500 
                    More than one race Native Hawaiian/Other Pacific Islander 
                                    81                                     20 
                                 White 
                                  1470

#raceeth內又分成7種分類，如果要做回歸分析，可以新增變數(ex:raceethAsisan)，將其變成0/1的型態，即是否為該種族。

2.3 Example unordered factors

Consider again adding our unordered factor race to the regression model with reference level “White”.

For a student who is Asian, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)

raceethAmerican Indian/Alaska Native
raceethAsian
raceethBlack
raceethHispanic
raceethMore than one race
raceethNative Hawaiian/Other Pacific Islander

#將raceethAsian設定為1，其他設定為0

For a student who is white, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)

raceethAmerican Indian/Alaska Native
raceethAsian
raceethBlack
raceethHispanic
raceethMore than one race
raceethNative Hawaiian/Other Pacific Islander

#除raceethWhite設定為1之外，其他設定為0

Section 3

3.1 Building a model

Because the race variable takes on text values, it was loaded as a factor variable when we read in the dataset with read.csv() – you can see this when you run str(pisaTrain) or str(pisaTest). However, by default R selects the first level alphabetically (“American Indian/Alaska Native”) as the reference level of our factor instead of the most common level (“White”). Set the reference level of the factor by typing the following two lines in your R console:

pisaTrain$raceeth = relevel(pisaTrain$raceeth, "White")

pisaTest$raceeth = relevel(pisaTest$raceeth, "White")

Now, build a linear regression model (call it lmScore) using the training set to predict readingScore using all the remaining variables.

It would be time-consuming to type all the variables, but R provides the shorthand notation “readingScore ~ .” to mean “predict readingScore using all the other variables in the data frame.” The period is used to replace listing out all of the independent variables. As an example, if your dependent variable is called “Y”, your independent variables are called “X1”, “X2”, and “X3”, and your training data set is called “Train”, instead of the regular notation:

LinReg = lm(Y ~ X1 + X2 + X3, data = Train)

You would use the following command to build your model:

LinReg = lm(Y ~ ., data = Train)

What is the Multiple R-squared value of lmScore on the training set?

pisaTrain$raceeth = relevel(pisaTrain$raceeth, "White")
pisaTest$raceeth = relevel(pisaTest$raceeth, "White")
lmScore = lm(readingScore ~ .,data = pisaTrain)
summary(lmScore)


Call:
lm(formula = readingScore ~ ., data = pisaTrain)

Residuals:
    Min      1Q  Median      3Q     Max 
-247.44  -48.86    1.86   49.77  217.18 

Coefficients:
                                                Estimate Std. Error t value Pr(>|t|)    
(Intercept)                                   143.766333  33.841226   4.248 2.24e-05 ***
grade                                          29.542707   2.937399  10.057  < 2e-16 ***
male                                          -14.521653   3.155926  -4.601 4.42e-06 ***
raceethAmerican Indian/Alaska Native          -67.277327  16.786935  -4.008 6.32e-05 ***
raceethAsian                                   -4.110325   9.220071  -0.446  0.65578    
raceethBlack                                  -67.012347   5.460883 -12.271  < 2e-16 ***
raceethHispanic                               -38.975486   5.177743  -7.528 7.29e-14 ***
raceethMore than one race                     -16.922522   8.496268  -1.992  0.04651 *  
raceethNative Hawaiian/Other Pacific Islander  -5.101601  17.005696  -0.300  0.76421    
preschool                                      -4.463670   3.486055  -1.280  0.20052    
expectBachelors                                55.267080   4.293893  12.871  < 2e-16 ***
motherHS                                        6.058774   6.091423   0.995  0.32001    
motherBachelors                                12.638068   3.861457   3.273  0.00108 ** 
motherWork                                     -2.809101   3.521827  -0.798  0.42517    
fatherHS                                        4.018214   5.579269   0.720  0.47147    
fatherBachelors                                16.929755   3.995253   4.237 2.35e-05 ***
fatherWork                                      5.842798   4.395978   1.329  0.18393    
selfBornUS                                     -3.806278   7.323718  -0.520  0.60331    
motherBornUS                                   -8.798153   6.587621  -1.336  0.18182    
fatherBornUS                                    4.306994   6.263875   0.688  0.49178    
englishAtHome                                   8.035685   6.859492   1.171  0.24153    
computerForSchoolwork                          22.500232   5.702562   3.946 8.19e-05 ***
read30MinsADay                                 34.871924   3.408447  10.231  < 2e-16 ***
minutesPerWeekEnglish                           0.012788   0.010712   1.194  0.23264    
studentsInEnglish                              -0.286631   0.227819  -1.258  0.20846    
schoolHasLibrary                               12.215085   9.264884   1.318  0.18749    
publicSchool                                  -16.857475   6.725614  -2.506  0.01226 *  
urban                                          -0.110132   3.962724  -0.028  0.97783    
schoolSize                                      0.006540   0.002197   2.977  0.00294 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 73.81 on 2385 degrees of freedom
Multiple R-squared:  0.3251,    Adjusted R-squared:  0.3172 
F-statistic: 41.04 on 28 and 2385 DF,  p-value: < 2.2e-16

#Multiple R-squared = 0.3251

Note that this R-squared is lower than the ones for the models we saw in the lectures and recitation. This does not necessarily imply that the model is of poor quality. More often than not, it simply means that the prediction problem at hand (predicting a student’s test score based on demographic and school-related variables) is more difficult than other prediction problems (like predicting a team’s number of wins from their runs scored and allowed, or predicting the quality of wine from weather conditions).

3.2 Computing the root-mean squared error of the model

What is the training-set root-mean squared error (RMSE) of lmScore?

SSE = sum(lmScore$residuals^2)
n = nrow(pisaTrain)
RMSE = sqrt(SSE/n)
#RMSE =73.36555

3.3 Comparing predictions for similar students

Consider two students A and B. They have all variable values the same, except that student A is in grade 11 and student B is in grade 9. What is the predicted reading score of student A minus the predicted reading score of student B?

-59.09
-29.54
0
29.54
59.09
The difference cannot be determined without more information about the two students

29.542707*2

[1] 59.08541

#reading score與grade的Estimate為29.542707，而兩者差2，故答案為29.542707*2 = 59.08541，約59.09

3.4 Interpreting model coefficients

What is the meaning of the coefficient associated with variable raceethAsian?

Predicted average reading score of an Asian student
Difference between the average reading score of an Asian student and the average reading score of a white student
Difference between the average reading score of an Asian student and the average reading score of all the students in the dataset
Predicted difference in the reading score between an Asian student and a white student who is otherwise identical

#raceethAsian係數是來判斷是否為亞洲人，因此其值為0/1，lmScore是使用白人學生與其他學生族群比較，因此當raceethAsian有變動時，是否會影響表現，可作為預測亞洲學生與其他族群學生的差異。

3.5 Identifying variables lacking statistical significance

Based on the significance codes, which variables are candidates for removal from the model? Select all that apply. (We’ll assume that the factor variable raceeth should only be removed if none of its levels are significant.)

grade
male
raceeth
preschool
expectBachelors
motherHS
motherBachelors
motherWork
fatherHS
fatherBachelors
fatherWork
selfBornUS
motherBornUS
fatherBornUS
englishAtHome
computerForSchoolwork
read30MinsADay
minutesPerWeekEnglish
studentsInEnglish
schoolHasLibrary
publicSchool
urban
schoolSize

summary(lmScore)


Call:
lm(formula = readingScore ~ ., data = pisaTrain)

Residuals:
    Min      1Q  Median      3Q     Max 
-247.44  -48.86    1.86   49.77  217.18 

Coefficients:
                                                Estimate Std. Error t value Pr(>|t|)    
(Intercept)                                   143.766333  33.841226   4.248 2.24e-05 ***
grade                                          29.542707   2.937399  10.057  < 2e-16 ***
male                                          -14.521653   3.155926  -4.601 4.42e-06 ***
raceethAmerican Indian/Alaska Native          -67.277327  16.786935  -4.008 6.32e-05 ***
raceethAsian                                   -4.110325   9.220071  -0.446  0.65578    
raceethBlack                                  -67.012347   5.460883 -12.271  < 2e-16 ***
raceethHispanic                               -38.975486   5.177743  -7.528 7.29e-14 ***
raceethMore than one race                     -16.922522   8.496268  -1.992  0.04651 *  
raceethNative Hawaiian/Other Pacific Islander  -5.101601  17.005696  -0.300  0.76421    
preschool                                      -4.463670   3.486055  -1.280  0.20052    
expectBachelors                                55.267080   4.293893  12.871  < 2e-16 ***
motherHS                                        6.058774   6.091423   0.995  0.32001    
motherBachelors                                12.638068   3.861457   3.273  0.00108 ** 
motherWork                                     -2.809101   3.521827  -0.798  0.42517    
fatherHS                                        4.018214   5.579269   0.720  0.47147    
fatherBachelors                                16.929755   3.995253   4.237 2.35e-05 ***
fatherWork                                      5.842798   4.395978   1.329  0.18393    
selfBornUS                                     -3.806278   7.323718  -0.520  0.60331    
motherBornUS                                   -8.798153   6.587621  -1.336  0.18182    
fatherBornUS                                    4.306994   6.263875   0.688  0.49178    
englishAtHome                                   8.035685   6.859492   1.171  0.24153    
computerForSchoolwork                          22.500232   5.702562   3.946 8.19e-05 ***
read30MinsADay                                 34.871924   3.408447  10.231  < 2e-16 ***
minutesPerWeekEnglish                           0.012788   0.010712   1.194  0.23264    
studentsInEnglish                              -0.286631   0.227819  -1.258  0.20846    
schoolHasLibrary                               12.215085   9.264884   1.318  0.18749    
publicSchool                                  -16.857475   6.725614  -2.506  0.01226 *  
urban                                          -0.110132   3.962724  -0.028  0.97783    
schoolSize                                      0.006540   0.002197   2.977  0.00294 ** 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 73.81 on 2385 degrees of freedom
Multiple R-squared:  0.3251,    Adjusted R-squared:  0.3172 
F-statistic: 41.04 on 28 and 2385 DF,  p-value: < 2.2e-16

#preschool,motherHS,motherWork,fatherHS,fatherWork,selfBornUS,motherBornUS,fatherBornUS,englishAtHome,minutesPerWeekEnglish,studentsInEnglish,schoolHasLibrary,urban

Section 4

4.1 Predicting on unseen data

Using the “predict” function and supplying the “newdata” argument, use the lmScore model to predict the reading scores of students in pisaTest. Call this vector of predictions “predTest”. Do not change the variables in the model (for example, do not remove variables that we found were not significant in the previous part of this problem). Use the summary function to describe the test set predictions.

What is the range between the maximum and minimum predicted reading score on the test set?

predTest = predict(lmScore,newdata = pisaTest)
summary(predTest)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  353.2   482.0   524.0   516.7   555.7   637.7

637.7-353.2

[1] 284.5

#284.5

4.2 Test set SSE and RMSE

What is the sum of squared errors (SSE) of lmScore on the testing set?

SSE = sum((predTest -pisaTest$readingScore)^2 )
SSE

[1] 5762082

What is the root-mean squared error (RMSE) of lmScore on the testing set?

RMSE = sqrt(SSE/nrow(pisaTest))
RMSE

[1] 76.29079

4.3 Baseline prediction and test-set SSE

What is the predicted test score used in the baseline model? Remember to compute this value using the training set and not the test set.

baseline = mean(pisaTrain$readingScore)
baseline

[1] 517.9629

What is the sum of squared errors of the baseline model on the testing set? HINT: We call the sum of squared errors for the baseline model the total sum of squares (SST).

SST = sum((baseline- pisaTest$readingScore)^2)
SST

[1] 7802354

4.4 Test-set R-squared

What is the test-set R-squared value of lmScore?

R_squared = 1-SSE/SST
R_squared

[1] 0.2614944

LS0tDQp0aXRsZTogIkFTMi0yIFJlYWRpbmcgVGVzdCBTY29yZXMiDQphdXRob3I6ICLmlr3ph4flvaMgTTA2NDAyMDAxNyINCm91dHB1dDogaHRtbF9ub3RlYm9vaw0KLS0tDQoNCmVkWCBhc3NpZ25tZW50IGxpbms6IGh0dHA6Ly9iaXQubHkvMktFMmcwMA0KDQpUaGUgUHJvZ3JhbW1lIGZvciBJbnRlcm5hdGlvbmFsIFN0dWRlbnQgQXNzZXNzbWVudCAoUElTQSkgaXMgYSB0ZXN0IGdpdmVuIGV2ZXJ5IHRocmVlIHllYXJzIHRvIDE1LXllYXItb2xkIHN0dWRlbnRzIGZyb20gYXJvdW5kIHRoZSB3b3JsZCB0byBldmFsdWF0ZSB0aGVpciBwZXJmb3JtYW5jZSBpbiBtYXRoZW1hdGljcywgcmVhZGluZywgYW5kIHNjaWVuY2UuIFRoaXMgdGVzdCBwcm92aWRlcyBhIHF1YW50aXRhdGl2ZSB3YXkgdG8gY29tcGFyZSB0aGUgcGVyZm9ybWFuY2Ugb2Ygc3R1ZGVudHMgZnJvbSBkaWZmZXJlbnQgcGFydHMgb2YgdGhlIHdvcmxkLiBJbiB0aGlzIGhvbWV3b3JrIGFzc2lnbm1lbnQsIHdlIHdpbGwgcHJlZGljdCB0aGUgcmVhZGluZyBzY29yZXMgb2Ygc3R1ZGVudHMgZnJvbSB0aGUgVW5pdGVkIFN0YXRlcyBvZiBBbWVyaWNhIG9uIHRoZSAyMDA5IFBJU0EgZXhhbS4NCg0KVGhlIGRhdGFzZXRzIHBpc2EyMDA5dHJhaW4uY3N2IGFuZCBwaXNhMjAwOXRlc3QuY3N2IGNvbnRhaW4gaW5mb3JtYXRpb24gYWJvdXQgdGhlIGRlbW9ncmFwaGljcyBhbmQgc2Nob29scyBmb3IgQW1lcmljYW4gc3R1ZGVudHMgdGFraW5nIHRoZSBleGFtLCBkZXJpdmVkIGZyb20gMjAwOSBQSVNBIFB1YmxpYy1Vc2UgRGF0YSBGaWxlcyBkaXN0cmlidXRlZCBieSB0aGUgVW5pdGVkIFN0YXRlcyBOYXRpb25hbCBDZW50ZXIgZm9yIEVkdWNhdGlvbiBTdGF0aXN0aWNzIChOQ0VTKS4gV2hpbGUgdGhlIGRhdGFzZXRzIGFyZSBub3Qgc3VwcG9zZWQgdG8gY29udGFpbiBpZGVudGlmeWluZyBpbmZvcm1hdGlvbiBhYm91dCBzdHVkZW50cyB0YWtpbmcgdGhlIHRlc3QsIGJ5IHVzaW5nIHRoZSBkYXRhIHlvdSBhcmUgYm91bmQgYnkgdGhlIE5DRVMgZGF0YSB1c2UgYWdyZWVtZW50LCB3aGljaCBwcm9oaWJpdHMgYW55IGF0dGVtcHQgdG8gZGV0ZXJtaW5lIHRoZSBpZGVudGl0eSBvZiBhbnkgc3R1ZGVudCBpbiB0aGUgZGF0YXNldHMuDQoNCkVhY2ggcm93IGluIHRoZSBkYXRhc2V0cyBwaXNhMjAwOXRyYWluLmNzdiBhbmQgcGlzYTIwMDl0ZXN0LmNzdiByZXByZXNlbnRzIG9uZSBzdHVkZW50IHRha2luZyB0aGUgZXhhbS4gVGhlIGRhdGFzZXRzIGhhdmUgdGhlIGZvbGxvd2luZyB2YXJpYWJsZXM6DQoNCisgZ3JhZGU6IFRoZSBncmFkZSBpbiBzY2hvb2wgb2YgdGhlIHN0dWRlbnQgKG1vc3QgMTUteWVhci1vbGRzIGluIEFtZXJpY2EgYXJlIGluIDEwdGggZ3JhZGUpDQoNCisgbWFsZTogV2hldGhlciB0aGUgc3R1ZGVudCBpcyBtYWxlICgxLzApDQoNCisgcmFjZWV0aDogVGhlIHJhY2UvZXRobmljaXR5IGNvbXBvc2l0ZSBvZiB0aGUgc3R1ZGVudA0KDQorIHByZXNjaG9vbDogV2hldGhlciB0aGUgc3R1ZGVudCBhdHRlbmRlZCBwcmVzY2hvb2wgKDEvMCkNCg0KKyBleHBlY3RCYWNoZWxvcnM6IFdoZXRoZXIgdGhlIHN0dWRlbnQgZXhwZWN0cyB0byBvYnRhaW4gYSBiYWNoZWxvcidzIGRlZ3JlZSAoMS8wKQ0KDQorIG1vdGhlckhTOiBXaGV0aGVyIHRoZSBzdHVkZW50J3MgbW90aGVyIGNvbXBsZXRlZCBoaWdoIHNjaG9vbCAoMS8wKQ0KDQorIG1vdGhlckJhY2hlbG9yczogV2hldGhlciB0aGUgc3R1ZGVudCdzIG1vdGhlciBvYnRhaW5lZCBhIGJhY2hlbG9yJ3MgZGVncmVlICgxLzApDQoNCisgbW90aGVyV29yazogV2hldGhlciB0aGUgc3R1ZGVudCdzIG1vdGhlciBoYXMgcGFydC10aW1lIG9yIGZ1bGwtdGltZSB3b3JrICgxLzApDQoNCisgZmF0aGVySFM6IFdoZXRoZXIgdGhlIHN0dWRlbnQncyBmYXRoZXIgY29tcGxldGVkIGhpZ2ggc2Nob29sICgxLzApDQoNCisgZmF0aGVyQmFjaGVsb3JzOiBXaGV0aGVyIHRoZSBzdHVkZW50J3MgZmF0aGVyIG9idGFpbmVkIGEgYmFjaGVsb3IncyBkZWdyZWUgKDEvMCkNCg0KKyBmYXRoZXJXb3JrOiBXaGV0aGVyIHRoZSBzdHVkZW50J3MgZmF0aGVyIGhhcyBwYXJ0LXRpbWUgb3IgZnVsbC10aW1lIHdvcmsgKDEvMCkNCg0KKyBzZWxmQm9yblVTOiBXaGV0aGVyIHRoZSBzdHVkZW50IHdhcyBib3JuIGluIHRoZSBVbml0ZWQgU3RhdGVzIG9mIEFtZXJpY2EgKDEvMCkNCg0KKyBtb3RoZXJCb3JuVVM6IFdoZXRoZXIgdGhlIHN0dWRlbnQncyBtb3RoZXIgd2FzIGJvcm4gaW4gdGhlIFVuaXRlZCBTdGF0ZXMgb2YgQW1lcmljYSAoMS8wKQ0KDQorIGZhdGhlckJvcm5VUzogV2hldGhlciB0aGUgc3R1ZGVudCdzIGZhdGhlciB3YXMgYm9ybiBpbiB0aGUgVW5pdGVkIFN0YXRlcyBvZiBBbWVyaWNhICgxLzApDQoNCisgZW5nbGlzaEF0SG9tZTogV2hldGhlciB0aGUgc3R1ZGVudCBzcGVha3MgRW5nbGlzaCBhdCBob21lICgxLzApDQoNCisgY29tcHV0ZXJGb3JTY2hvb2x3b3JrOiBXaGV0aGVyIHRoZSBzdHVkZW50IGhhcyBhY2Nlc3MgdG8gYSBjb21wdXRlciBmb3Igc2Nob29sd29yayAoMS8wKQ0KDQorIHJlYWQzME1pbnNBRGF5OiBXaGV0aGVyIHRoZSBzdHVkZW50IHJlYWRzIGZvciBwbGVhc3VyZSBmb3IgMzAgbWludXRlcy9kYXkgKDEvMCkNCg0KKyBtaW51dGVzUGVyV2Vla0VuZ2xpc2g6IFRoZSBudW1iZXIgb2YgbWludXRlcyBwZXIgd2VlayB0aGUgc3R1ZGVudCBzcGVuZCBpbiBFbmdsaXNoIGNsYXNzDQoNCisgc3R1ZGVudHNJbkVuZ2xpc2g6IFRoZSBudW1iZXIgb2Ygc3R1ZGVudHMgaW4gdGhpcyBzdHVkZW50J3MgRW5nbGlzaCBjbGFzcyBhdCBzY2hvb2wNCg0KKyBzY2hvb2xIYXNMaWJyYXJ5OiBXaGV0aGVyIHRoaXMgc3R1ZGVudCdzIHNjaG9vbCBoYXMgYSBsaWJyYXJ5ICgxLzApDQoNCisgcHVibGljU2Nob29sOiBXaGV0aGVyIHRoaXMgc3R1ZGVudCBhdHRlbmRzIGEgcHVibGljIHNjaG9vbCAoMS8wKQ0KDQorIHVyYmFuOiBXaGV0aGVyIHRoaXMgc3R1ZGVudCdzIHNjaG9vbCBpcyBpbiBhbiB1cmJhbiBhcmVhICgxLzApDQoNCisgc2Nob29sU2l6ZTogVGhlIG51bWJlciBvZiBzdHVkZW50cyBpbiB0aGlzIHN0dWRlbnQncyBzY2hvb2wNCg0KKyByZWFkaW5nU2NvcmU6IFRoZSBzdHVkZW50J3MgcmVhZGluZyBzY29yZSwgb24gYSAxMDAwLXBvaW50IHNjYWxlDQoNCi0gLSAtDQoNCiMjIyBTZWN0aW9uIDENCiANCiMjIyMgMS4xIERhdGFzZXQgc2l6ZQ0KTG9hZCB0aGUgdHJhaW5pbmcgYW5kIHRlc3Rpbmcgc2V0cyB1c2luZyB0aGUgcmVhZC5jc3YoKSBmdW5jdGlvbiwgYW5kIHNhdmUgdGhlbSBhcyB2YXJpYWJsZXMgd2l0aCB0aGUgbmFtZXMgcGlzYVRyYWluIGFuZCBwaXNhVGVzdC4NCg0KSG93IG1hbnkgc3R1ZGVudHMgYXJlIHRoZXJlIGluIHRoZSB0cmFpbmluZyBzZXQ/DQoNCmBgYHtyfQ0KcGlzYVRyYWluID0gcmVhZC5jc3YoIkQ6L2J1aW5lc3NfYW5hbHl0aWNzL3VuaXQyL2RhdGEvcGlzYTIwMDl0cmFpbi5jc3YiKQ0KcGlzYVRlc3QgPSByZWFkLmNzdigiRDovYnVpbmVzc19hbmFseXRpY3MvdW5pdDIvZGF0YS9waXNhMjAwOXRlc3QuY3N2IikNCm5yb3cocGlzYVRyYWluKQ0KYGBgDQoNCg0KIyMjIyAxLjIgU3VtbWFyaXppbmcgdGhlIGRhdGFzZXQNClVzaW5nIHRhcHBseSgpIG9uIHBpc2FUcmFpbiwgd2hhdCBpcyB0aGUgYXZlcmFnZSByZWFkaW5nIHRlc3Qgc2NvcmUgb2YgbWFsZXM/DQoNCmBgYHtyfQ0KdGFwcGx5KHBpc2FUcmFpbiRyZWFkaW5nU2NvcmUscGlzYVRyYWluJG1hbGUsbWVhbikNCiNtYWxlcyA9IDQ4My41MzI1DQpgYGANCg0KT2YgZmVtYWxlcz8NCmBgYHtyfQ0KdGFwcGx5KHBpc2FUcmFpbiRyZWFkaW5nU2NvcmUscGlzYVRyYWluJG1hbGUsbWVhbikNCiNmZW1hbGVzID0gNTEyLjk0MDYNCmBgYA0KDQoNCiMjIyMxLjMgTG9jYXRpbmcgbWlzc2luZyB2YWx1ZXMNCldoaWNoIHZhcmlhYmxlcyBhcmUgbWlzc2luZyBkYXRhIGluIGF0IGxlYXN0IG9uZSBvYnNlcnZhdGlvbiBpbiB0aGUgdHJhaW5pbmcgc2V0PyBTZWxlY3QgYWxsIHRoYXQgYXBwbHkuDQoNCisgZ3JhZGUNCisgbWFsZQ0KKyByYWNlZXRoDQorIHByZXNjaG9vbA0KKyBleHBlY3RCYWNoZWxvcnMNCisgbW90aGVySFMNCisgbW90aGVyQmFjaGVsb3JzDQorIG1vdGhlcldvcmsNCisgZmF0aGVySFMNCisgZmF0aGVyQmFjaGVsb3JzDQorIGZhdGhlcldvcmsNCisgc2VsZkJvcm5VUw0KKyBtb3RoZXJCb3JuVVMNCisgZmF0aGVyQm9yblVTDQorIGVuZ2xpc2hBdEhvbWUNCisgY29tcHV0ZXJGb3JTY2hvb2x3b3JrDQorIHJlYWQzME1pbnNBRGF5DQorIG1pbnV0ZXNQZXJXZWVrRW5nbGlzaA0KKyBzdHVkZW50c0luRW5nbGlzaA0KKyBzY2hvb2xIYXNMaWJyYXJ5DQorIHB1YmxpY1NjaG9vbA0KKyB1cmJhbg0KKyBzY2hvb2xTaXplDQorIHJlYWRpbmdTY29yZQ0KDQpgYGB7cn0NCnN1bW1hcnkocGlzYVRyYWluKQ0KI3JhY2VldGgscHJlc2Nob29sLGV4cGVjdEJhY2hlbG9ycyxtb3RoZXJIUyxtb3RoZXJCYWNoZWxvcnMsbW90aGVyV29yayAsZmF0aGVySFMNCiNmYXRoZXJCYWNoZWxvcnMsZmF0aGVyV29yayxzZWxmQm9yblVTLG1vdGhlckJvcm5VUyxmYXRoZXJCb3JuVVMsZW5nbGlzaEF0SG9tZQ0KI2NvbXB1dGVyRm9yU2Nob29sd29yayxyZWFkMzBNaW5zQURheSxtaW51dGVzUGVyV2Vla0VuZ2xpc2gsc3R1ZGVudHNJbkVuZ2xpc2gNCiNzY2hvb2xIYXNMaWJyYXJ5LHNjaG9vbFNpemUNCmBgYA0KDQoNCiMjIyMxLjQgUmVtb3ZpbmcgbWlzc2luZyB2YWx1ZXMNCg0KTGluZWFyIHJlZ3Jlc3Npb24gZGlzY2FyZHMgb2JzZXJ2YXRpb25zIHdpdGggbWlzc2luZyBkYXRhLCBzbyB3ZSB3aWxsIHJlbW92ZSBhbGwgc3VjaCBvYnNlcnZhdGlvbnMgZnJvbSB0aGUgdHJhaW5pbmcgYW5kIHRlc3Rpbmcgc2V0cy4gTGF0ZXIgaW4gdGhlIGNvdXJzZSwgd2Ugd2lsbCBsZWFybiBhYm91dCBpbXB1dGF0aW9uLCB3aGljaCBkZWFscyB3aXRoIG1pc3NpbmcgZGF0YSBieSBmaWxsaW5nIGluIG1pc3NpbmcgdmFsdWVzIHdpdGggcGxhdXNpYmxlIGluZm9ybWF0aW9uLg0KDQpUeXBlIHRoZSBmb2xsb3dpbmcgY29tbWFuZHMgaW50byB5b3VyIFIgY29uc29sZSB0byByZW1vdmUgb2JzZXJ2YXRpb25zIHdpdGggYW55IG1pc3NpbmcgdmFsdWUgZnJvbSBwaXNhVHJhaW4gYW5kIHBpc2FUZXN0Og0KDQogICAgcGlzYVRyYWluID0gbmEub21pdChwaXNhVHJhaW4pDQoNCiAgICBwaXNhVGVzdCA9IG5hLm9taXQocGlzYVRlc3QpDQoNCkhvdyBtYW55IG9ic2VydmF0aW9ucyBhcmUgbm93IGluIHRoZSB0cmFpbmluZyBzZXQ/DQoNCmBgYHtyfQ0KcGlzYVRyYWluID0gbmEub21pdChwaXNhVHJhaW4pDQpzdHIocGlzYVRyYWluKQ0KIzI0MTQNCmBgYA0KDQpIb3cgbWFueSBvYnNlcnZhdGlvbnMgYXJlIG5vdyBpbiB0aGUgdGVzdGluZyBzZXQ/DQoNCmBgYHtyfQ0KcGlzYVRlc3QgPSBuYS5vbWl0KHBpc2FUZXN0KQ0Kc3RyKHBpc2FUZXN0KQ0KIzk5MA0KYGBgDQoNCg0KDQojIyMgU2VjdGlvbiAyDQoNCg0KIyMjIzIuMQ0KDQpGYWN0b3IgdmFyaWFibGVzIGFyZSB2YXJpYWJsZXMgdGhhdCB0YWtlIG9uIGEgZGlzY3JldGUgc2V0IG9mIHZhbHVlcywgbGlrZSB0aGUgIlJlZ2lvbiIgdmFyaWFibGUgaW4gdGhlIFdITyBkYXRhc2V0IGZyb20gdGhlIHNlY29uZCBsZWN0dXJlIG9mIFVuaXQgMS4gVGhpcyBpcyBhbiB1bm9yZGVyZWQgZmFjdG9yIGJlY2F1c2UgdGhlcmUgaXNuJ3QgYW55IG5hdHVyYWwgb3JkZXJpbmcgYmV0d2VlbiB0aGUgbGV2ZWxzLiBBbiBvcmRlcmVkIGZhY3RvciBoYXMgYSBuYXR1cmFsIG9yZGVyaW5nIGJldHdlZW4gdGhlIGxldmVscyAoYW4gZXhhbXBsZSB3b3VsZCBiZSB0aGUgY2xhc3NpZmljYXRpb25zICJsYXJnZSwiICJtZWRpdW0sIiBhbmQgInNtYWxsIikuDQoNCldoaWNoIG9mIHRoZSBmb2xsb3dpbmcgdmFyaWFibGVzIGlzIGFuIHVub3JkZXJlZCBmYWN0b3Igd2l0aCBhdCBsZWFzdCAzIGxldmVscz8gKFNlbGVjdCBhbGwgdGhhdCBhcHBseS4pDQoNCisgZ3JhZGUNCisgbWFsZQ0KKyByYWNlZXRoDQoNCmBgYHtyfQ0Kc3RyKHBpc2FUcmFpbikNCiNyYWNlZXRoIC0+54K6RmFjdG9yDQoNCmBgYA0KDQpXaGljaCBvZiB0aGUgZm9sbG93aW5nIHZhcmlhYmxlcyBpcyBhbiBvcmRlcmVkIGZhY3RvciB3aXRoIGF0IGxlYXN0IDMgbGV2ZWxzPyAoU2VsZWN0IGFsbCB0aGF0IGFwcGx5LikNCg0KYGBge3J9DQojZmFjdG9yKHBpc2FUcmFpbiRncmFkZSkNCiNncmFkZSAtPuizh+aWmeiugOmAsuS+hueahOaZguWAmeacg+iHquWLlei9ieaPm+aIkGludCzogIxncmFkZeacg+WBmueCuuWIhumhnueahOS+neaTmu+8jOS4lOa7v+i2szPlgIvku6XkuIrlsaTntJrnmoTmop3ku7bvvIzlm6DmraTmmK9GYWN0b3INCmBgYA0KDQojIyMjIDIuMiBVbm9yZGVyZWQgZmFjdG9ycyBpbiByZWdyZXNzaW9uIG1vZGVscw0KDQpUbyBpbmNsdWRlIHVub3JkZXJlZCBmYWN0b3JzIGluIGEgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWwsIHdlIGRlZmluZSBvbmUgbGV2ZWwgYXMgdGhlICJyZWZlcmVuY2UgbGV2ZWwiIGFuZCBhZGQgYSBiaW5hcnkgdmFyaWFibGUgZm9yIGVhY2ggb2YgdGhlIHJlbWFpbmluZyBsZXZlbHMuIEluIHRoaXMgd2F5LCBhIGZhY3RvciB3aXRoIG4gbGV2ZWxzIGlzIHJlcGxhY2VkIGJ5IG4tMSBiaW5hcnkgdmFyaWFibGVzLiBUaGUgcmVmZXJlbmNlIGxldmVsIGlzIHR5cGljYWxseSBzZWxlY3RlZCB0byBiZSB0aGUgbW9zdCBmcmVxdWVudGx5IG9jY3VycmluZyBsZXZlbCBpbiB0aGUgZGF0YXNldC4NCg0KQXMgYW4gZXhhbXBsZSwgY29uc2lkZXIgdGhlIHVub3JkZXJlZCBmYWN0b3IgdmFyaWFibGUgImNvbG9yIiwgd2l0aCBsZXZlbHMgInJlZCIsICJncmVlbiIsIGFuZCAiYmx1ZSIuIElmICJncmVlbiIgd2VyZSB0aGUgcmVmZXJlbmNlIGxldmVsLCB0aGVuIHdlIHdvdWxkIGFkZCBiaW5hcnkgdmFyaWFibGVzICJjb2xvcnJlZCIgYW5kICJjb2xvcmJsdWUiIHRvIGEgbGluZWFyIHJlZ3Jlc3Npb24gcHJvYmxlbS4gQWxsIHJlZCBleGFtcGxlcyB3b3VsZCBoYXZlIGNvbG9ycmVkPTEgYW5kIGNvbG9yYmx1ZT0wLiBBbGwgYmx1ZSBleGFtcGxlcyB3b3VsZCBoYXZlIGNvbG9ycmVkPTAgYW5kIGNvbG9yYmx1ZT0xLiBBbGwgZ3JlZW4gZXhhbXBsZXMgd291bGQgaGF2ZSBjb2xvcnJlZD0wIGFuZCBjb2xvcmJsdWU9MC4NCg0KTm93LCBjb25zaWRlciB0aGUgdmFyaWFibGUgInJhY2VldGgiIGluIG91ciBwcm9ibGVtLCB3aGljaCBoYXMgbGV2ZWxzICJBbWVyaWNhbiBJbmRpYW4vQWxhc2thIE5hdGl2ZSIsICJBc2lhbiIsICJCbGFjayIsICJIaXNwYW5pYyIsICJNb3JlIHRoYW4gb25lIHJhY2UiLCAiTmF0aXZlIEhhd2FpaWFuL090aGVyIFBhY2lmaWMgSXNsYW5kZXIiLCBhbmQgIldoaXRlIi4gQmVjYXVzZSBpdCBpcyB0aGUgbW9zdCBjb21tb24gaW4gb3VyIHBvcHVsYXRpb24sIHdlIHdpbGwgc2VsZWN0IFdoaXRlIGFzIHRoZSByZWZlcmVuY2UgbGV2ZWwuDQoNCg0KV2hpY2ggYmluYXJ5IHZhcmlhYmxlcyB3aWxsIGJlIGluY2x1ZGVkIGluIHRoZSByZWdyZXNzaW9uIG1vZGVsPyAoU2VsZWN0IGFsbCB0aGF0IGFwcGx5LikNCg0KDQpgYGB7cn0NCnN1bW1hcnkocGlzYVRyYWluJHJhY2VldGgpDQojcmFjZWV0aOWFp+WPiOWIhuaIkDfnqK7liIbpoZ7vvIzlpoLmnpzopoHlgZrlm57mrbjliIbmnpDvvIzlj6/ku6XmlrDlop7orormlbgoZXg6cmFjZWV0aEFzaXNhbinvvIzlsIflhbborormiJAwLzHnmoTlnovmhYvvvIzljbPmmK/lkKbngrroqbLnqK7ml4/jgIINCmBgYA0KDQojIyMjIDIuMyBFeGFtcGxlIHVub3JkZXJlZCBmYWN0b3JzIA0KDQpDb25zaWRlciBhZ2FpbiBhZGRpbmcgb3VyIHVub3JkZXJlZCBmYWN0b3IgcmFjZSB0byB0aGUgcmVncmVzc2lvbiBtb2RlbCB3aXRoIHJlZmVyZW5jZSBsZXZlbCAiV2hpdGUiLg0KDQpGb3IgYSBzdHVkZW50IHdobyBpcyBBc2lhbiwgd2hpY2ggYmluYXJ5IHZhcmlhYmxlcyB3b3VsZCBiZSBzZXQgdG8gMD8gQWxsIHJlbWFpbmluZyB2YXJpYWJsZXMgd2lsbCBiZSBzZXQgdG8gMS4gKFNlbGVjdCBhbGwgdGhhdCBhcHBseS4pDQoNCisgcmFjZWV0aEFtZXJpY2FuIEluZGlhbi9BbGFza2EgTmF0aXZlDQorIHJhY2VldGhBc2lhbg0KKyByYWNlZXRoQmxhY2sNCisgcmFjZWV0aEhpc3BhbmljDQorIHJhY2VldGhNb3JlIHRoYW4gb25lIHJhY2UNCisgcmFjZWV0aE5hdGl2ZSBIYXdhaWlhbi9PdGhlciBQYWNpZmljIElzbGFuZGVyDQoNCg0KYGBge3J9DQoj5bCHcmFjZWV0aEFzaWFu6Kit5a6a54K6Me+8jOWFtuS7luioreWumueCujANCg0KYGBgDQoNCg0KRm9yIGEgc3R1ZGVudCB3aG8gaXMgd2hpdGUsIHdoaWNoIGJpbmFyeSB2YXJpYWJsZXMgd291bGQgYmUgc2V0IHRvIDA/IEFsbCByZW1haW5pbmcgdmFyaWFibGVzIHdpbGwgYmUgc2V0IHRvIDEuIChTZWxlY3QgYWxsIHRoYXQgYXBwbHkuKQ0KDQorIHJhY2VldGhBbWVyaWNhbiBJbmRpYW4vQWxhc2thIE5hdGl2ZQ0KKyByYWNlZXRoQXNpYW4NCisgcmFjZWV0aEJsYWNrDQorIHJhY2VldGhIaXNwYW5pYw0KKyByYWNlZXRoTW9yZSB0aGFuIG9uZSByYWNlDQorIHJhY2VldGhOYXRpdmUgSGF3YWlpYW4vT3RoZXIgUGFjaWZpYyBJc2xhbmRlcg0KDQoNCmBgYHtyfQ0KI+mZpHJhY2VldGhXaGl0ZeioreWumueCujHkuYvlpJbvvIzlhbbku5boqK3lrprngrowDQpgYGANCg0KDQojIyMgU2VjdGlvbiAzDQoNCiMjIyMzLjEgQnVpbGRpbmcgYSBtb2RlbA0KDQpCZWNhdXNlIHRoZSByYWNlIHZhcmlhYmxlIHRha2VzIG9uIHRleHQgdmFsdWVzLCBpdCB3YXMgbG9hZGVkIGFzIGEgZmFjdG9yIHZhcmlhYmxlIHdoZW4gd2UgcmVhZCBpbiB0aGUgZGF0YXNldCB3aXRoIHJlYWQuY3N2KCkgLS0geW91IGNhbiBzZWUgdGhpcyB3aGVuIHlvdSBydW4gc3RyKHBpc2FUcmFpbikgb3Igc3RyKHBpc2FUZXN0KS4gSG93ZXZlciwgYnkgZGVmYXVsdCBSIHNlbGVjdHMgdGhlIGZpcnN0IGxldmVsIGFscGhhYmV0aWNhbGx5ICgiQW1lcmljYW4gSW5kaWFuL0FsYXNrYSBOYXRpdmUiKSBhcyB0aGUgcmVmZXJlbmNlIGxldmVsIG9mIG91ciBmYWN0b3IgaW5zdGVhZCBvZiB0aGUgbW9zdCBjb21tb24gbGV2ZWwgKCJXaGl0ZSIpLiBTZXQgdGhlIHJlZmVyZW5jZSBsZXZlbCBvZiB0aGUgZmFjdG9yIGJ5IHR5cGluZyB0aGUgZm9sbG93aW5nIHR3byBsaW5lcyBpbiB5b3VyIFIgY29uc29sZToNCg0KICAgIHBpc2FUcmFpbiRyYWNlZXRoID0gcmVsZXZlbChwaXNhVHJhaW4kcmFjZWV0aCwgIldoaXRlIikNCg0KICAgIHBpc2FUZXN0JHJhY2VldGggPSByZWxldmVsKHBpc2FUZXN0JHJhY2VldGgsICJXaGl0ZSIpDQoNCk5vdywgYnVpbGQgYSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCAoY2FsbCBpdCBsbVNjb3JlKSB1c2luZyB0aGUgdHJhaW5pbmcgc2V0IHRvIHByZWRpY3QgcmVhZGluZ1Njb3JlIHVzaW5nIGFsbCB0aGUgcmVtYWluaW5nIHZhcmlhYmxlcy4NCg0KSXQgd291bGQgYmUgdGltZS1jb25zdW1pbmcgdG8gdHlwZSBhbGwgdGhlIHZhcmlhYmxlcywgYnV0IFIgcHJvdmlkZXMgdGhlIHNob3J0aGFuZCBub3RhdGlvbiAicmVhZGluZ1Njb3JlIH4gLiIgdG8gbWVhbiAicHJlZGljdCByZWFkaW5nU2NvcmUgdXNpbmcgYWxsIHRoZSBvdGhlciB2YXJpYWJsZXMgaW4gdGhlIGRhdGEgZnJhbWUuIiBUaGUgcGVyaW9kIGlzIHVzZWQgdG8gcmVwbGFjZSBsaXN0aW5nIG91dCBhbGwgb2YgdGhlIGluZGVwZW5kZW50IHZhcmlhYmxlcy4gQXMgYW4gZXhhbXBsZSwgaWYgeW91ciBkZXBlbmRlbnQgdmFyaWFibGUgaXMgY2FsbGVkICJZIiwgeW91ciBpbmRlcGVuZGVudCB2YXJpYWJsZXMgYXJlIGNhbGxlZCAiWDEiLCAiWDIiLCBhbmQgIlgzIiwgYW5kIHlvdXIgdHJhaW5pbmcgZGF0YSBzZXQgaXMgY2FsbGVkICJUcmFpbiIsIGluc3RlYWQgb2YgdGhlIHJlZ3VsYXIgbm90YXRpb246DQoNCiAgICBMaW5SZWcgPSBsbShZIH4gWDEgKyBYMiArIFgzLCBkYXRhID0gVHJhaW4pDQoNCllvdSB3b3VsZCB1c2UgdGhlIGZvbGxvd2luZyBjb21tYW5kIHRvIGJ1aWxkIHlvdXIgbW9kZWw6DQoNCiAgICBMaW5SZWcgPSBsbShZIH4gLiwgZGF0YSA9IFRyYWluKQ0KDQpXaGF0IGlzIHRoZSBNdWx0aXBsZSBSLXNxdWFyZWQgdmFsdWUgb2YgbG1TY29yZSBvbiB0aGUgdHJhaW5pbmcgc2V0Pw0KDQpgYGB7cn0NCnBpc2FUcmFpbiRyYWNlZXRoID0gcmVsZXZlbChwaXNhVHJhaW4kcmFjZWV0aCwgIldoaXRlIikNCnBpc2FUZXN0JHJhY2VldGggPSByZWxldmVsKHBpc2FUZXN0JHJhY2VldGgsICJXaGl0ZSIpDQpsbVNjb3JlID0gbG0ocmVhZGluZ1Njb3JlIH4gLixkYXRhID0gcGlzYVRyYWluKQ0Kc3VtbWFyeShsbVNjb3JlKQ0KI011bHRpcGxlIFItc3F1YXJlZCA9IDAuMzI1MQ0KYGBgDQoNCg0KTm90ZSB0aGF0IHRoaXMgUi1zcXVhcmVkIGlzIGxvd2VyIHRoYW4gdGhlIG9uZXMgZm9yIHRoZSBtb2RlbHMgd2Ugc2F3IGluIHRoZSBsZWN0dXJlcyBhbmQgcmVjaXRhdGlvbi4gVGhpcyBkb2VzIG5vdCBuZWNlc3NhcmlseSBpbXBseSB0aGF0IHRoZSBtb2RlbCBpcyBvZiBwb29yIHF1YWxpdHkuIE1vcmUgb2Z0ZW4gdGhhbiBub3QsIGl0IHNpbXBseSBtZWFucyB0aGF0IHRoZSBwcmVkaWN0aW9uIHByb2JsZW0gYXQgaGFuZCAocHJlZGljdGluZyBhIHN0dWRlbnQncyB0ZXN0IHNjb3JlIGJhc2VkIG9uIGRlbW9ncmFwaGljIGFuZCBzY2hvb2wtcmVsYXRlZCB2YXJpYWJsZXMpIGlzIG1vcmUgZGlmZmljdWx0IHRoYW4gb3RoZXIgcHJlZGljdGlvbiBwcm9ibGVtcyAobGlrZSBwcmVkaWN0aW5nIGEgdGVhbSdzIG51bWJlciBvZiB3aW5zIGZyb20gdGhlaXIgcnVucyBzY29yZWQgYW5kIGFsbG93ZWQsIG9yIHByZWRpY3RpbmcgdGhlIHF1YWxpdHkgb2Ygd2luZSBmcm9tIHdlYXRoZXIgY29uZGl0aW9ucykuDQoNCiMjIyMgMy4yIENvbXB1dGluZyB0aGUgcm9vdC1tZWFuIHNxdWFyZWQgZXJyb3Igb2YgdGhlIG1vZGVsDQoNCldoYXQgaXMgdGhlIHRyYWluaW5nLXNldCByb290LW1lYW4gc3F1YXJlZCBlcnJvciAoUk1TRSkgb2YgbG1TY29yZT8NCg0KYGBge3J9DQpTU0UgPSBzdW0obG1TY29yZSRyZXNpZHVhbHNeMikNCm4gPSBucm93KHBpc2FUcmFpbikNClJNU0UgPSBzcXJ0KFNTRS9uKQ0KI1JNU0UgPTczLjM2NTU1DQpgYGANCg0KDQojIyMjIDMuMyBDb21wYXJpbmcgcHJlZGljdGlvbnMgZm9yIHNpbWlsYXIgc3R1ZGVudHMNCg0KQ29uc2lkZXIgdHdvIHN0dWRlbnRzIEEgYW5kIEIuIFRoZXkgaGF2ZSBhbGwgdmFyaWFibGUgdmFsdWVzIHRoZSBzYW1lLCBleGNlcHQgdGhhdCBzdHVkZW50IEEgaXMgaW4gZ3JhZGUgMTEgYW5kIHN0dWRlbnQgQiBpcyBpbiBncmFkZSA5LiBXaGF0IGlzIHRoZSBwcmVkaWN0ZWQgcmVhZGluZyBzY29yZSBvZiBzdHVkZW50IEEgbWludXMgdGhlIHByZWRpY3RlZCByZWFkaW5nIHNjb3JlIG9mIHN0dWRlbnQgQj8NCg0KKyAtNTkuMDkNCisgLTI5LjU0DQorIDANCisgMjkuNTQNCisgNTkuMDkNCisgVGhlIGRpZmZlcmVuY2UgY2Fubm90IGJlIGRldGVybWluZWQgd2l0aG91dCBtb3JlIGluZm9ybWF0aW9uIGFib3V0IHRoZSB0d28gc3R1ZGVudHMNCg0KYGBge3J9DQoyOS41NDI3MDcqMg0KI3JlYWRpbmcgc2NvcmXoiIdncmFkZeeahEVzdGltYXRl54K6MjkuNTQyNzA377yM6ICM5YWp6ICF5beuMu+8jOaVheetlOahiOeCujI5LjU0MjcwNyoyID0gNTkuMDg1NDHvvIzntIQ1OS4wOQ0KDQoNCmBgYA0KDQoNCiMjIyMgMy40IEludGVycHJldGluZyBtb2RlbCBjb2VmZmljaWVudHMNCg0KV2hhdCBpcyB0aGUgbWVhbmluZyBvZiB0aGUgY29lZmZpY2llbnQgYXNzb2NpYXRlZCB3aXRoIHZhcmlhYmxlIHJhY2VldGhBc2lhbj8NCg0KKyBQcmVkaWN0ZWQgYXZlcmFnZSByZWFkaW5nIHNjb3JlIG9mIGFuIEFzaWFuIHN0dWRlbnQNCisgRGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBhdmVyYWdlIHJlYWRpbmcgc2NvcmUgb2YgYW4gQXNpYW4gc3R1ZGVudCBhbmQgdGhlIGF2ZXJhZ2UgcmVhZGluZyAgc2NvcmUgb2YgYSB3aGl0ZSBzdHVkZW50DQorIERpZmZlcmVuY2UgYmV0d2VlbiB0aGUgYXZlcmFnZSByZWFkaW5nIHNjb3JlIG9mIGFuIEFzaWFuIHN0dWRlbnQgYW5kIHRoZSBhdmVyYWdlIHJlYWRpbmcgICAgc2NvcmUgb2YgYWxsIHRoZSBzdHVkZW50cyBpbiB0aGUgZGF0YXNldA0KKyBQcmVkaWN0ZWQgZGlmZmVyZW5jZSBpbiB0aGUgcmVhZGluZyBzY29yZSBiZXR3ZWVuIGFuIEFzaWFuIHN0dWRlbnQgYW5kIGEgd2hpdGUgc3R1ZGVudCB3aG8gaXMgb3RoZXJ3aXNlIGlkZW50aWNhbA0KDQpgYGB7cn0NCiNyYWNlZXRoQXNpYW7kv4LmlbjmmK/kvobliKTmlrfmmK/lkKbngrrkup7mtLLkurrvvIzlm6DmraTlhbblgLzngrowLzHvvIxsbVNjb3Jl5piv5L2/55So55m95Lq65a2455Sf6IiH5YW25LuW5a2455Sf5peP576k5q+U6LyD77yM5Zug5q2k55W2cmFjZWV0aEFzaWFu5pyJ6K6K5YuV5pmC77yM5piv5ZCm5pyD5b2x6Z+/6KGo54++77yM5Y+v5L2c54K66aCQ5ris5Lqe5rSy5a2455Sf6IiH5YW25LuW5peP576k5a2455Sf55qE5beu55Ww44CCDQoNCmBgYA0KDQoNCiMjIyMzLjUgSWRlbnRpZnlpbmcgdmFyaWFibGVzIGxhY2tpbmcgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlDQoNCkJhc2VkIG9uIHRoZSBzaWduaWZpY2FuY2UgY29kZXMsIHdoaWNoIHZhcmlhYmxlcyBhcmUgY2FuZGlkYXRlcyBmb3IgcmVtb3ZhbCBmcm9tIHRoZSBtb2RlbD8gU2VsZWN0IGFsbCB0aGF0IGFwcGx5LiAoV2UnbGwgYXNzdW1lIHRoYXQgdGhlIGZhY3RvciB2YXJpYWJsZSByYWNlZXRoIHNob3VsZCBvbmx5IGJlIHJlbW92ZWQgaWYgbm9uZSBvZiBpdHMgbGV2ZWxzIGFyZSBzaWduaWZpY2FudC4pDQoNCisgZ3JhZGUNCisgbWFsZQ0KKyByYWNlZXRoDQorIHByZXNjaG9vbA0KKyBleHBlY3RCYWNoZWxvcnMNCisgbW90aGVySFMNCisgbW90aGVyQmFjaGVsb3JzDQorIG1vdGhlcldvcmsNCisgZmF0aGVySFMNCisgZmF0aGVyQmFjaGVsb3JzDQorIGZhdGhlcldvcmsNCisgc2VsZkJvcm5VUw0KKyBtb3RoZXJCb3JuVVMNCisgZmF0aGVyQm9yblVTDQorIGVuZ2xpc2hBdEhvbWUNCisgY29tcHV0ZXJGb3JTY2hvb2x3b3JrDQorIHJlYWQzME1pbnNBRGF5DQorIG1pbnV0ZXNQZXJXZWVrRW5nbGlzaA0KKyBzdHVkZW50c0luRW5nbGlzaA0KKyBzY2hvb2xIYXNMaWJyYXJ5DQorIHB1YmxpY1NjaG9vbA0KKyB1cmJhbg0KKyBzY2hvb2xTaXplDQoNCmBgYHtyfQ0Kc3VtbWFyeShsbVNjb3JlKQ0KI3ByZXNjaG9vbCxtb3RoZXJIUyxtb3RoZXJXb3JrLGZhdGhlckhTLGZhdGhlcldvcmssc2VsZkJvcm5VUyxtb3RoZXJCb3JuVVMsZmF0aGVyQm9yblVTLGVuZ2xpc2hBdEhvbWUsbWludXRlc1BlcldlZWtFbmdsaXNoLHN0dWRlbnRzSW5FbmdsaXNoLHNjaG9vbEhhc0xpYnJhcnksdXJiYW4NCmBgYA0KDQoNCiMjI1NlY3Rpb24gNA0KDQoNCiMjIyMgNC4xIFByZWRpY3Rpbmcgb24gdW5zZWVuIGRhdGENCg0KVXNpbmcgdGhlICJwcmVkaWN0IiBmdW5jdGlvbiBhbmQgc3VwcGx5aW5nIHRoZSAibmV3ZGF0YSIgYXJndW1lbnQsIHVzZSB0aGUgbG1TY29yZSBtb2RlbCB0byBwcmVkaWN0IHRoZSByZWFkaW5nIHNjb3JlcyBvZiBzdHVkZW50cyBpbiBwaXNhVGVzdC4gQ2FsbCB0aGlzIHZlY3RvciBvZiBwcmVkaWN0aW9ucyAicHJlZFRlc3QiLiBEbyBub3QgY2hhbmdlIHRoZSB2YXJpYWJsZXMgaW4gdGhlIG1vZGVsIChmb3IgZXhhbXBsZSwgZG8gbm90IHJlbW92ZSB2YXJpYWJsZXMgdGhhdCB3ZSBmb3VuZCB3ZXJlIG5vdCBzaWduaWZpY2FudCBpbiB0aGUgcHJldmlvdXMgcGFydCBvZiB0aGlzIHByb2JsZW0pLiBVc2UgdGhlIHN1bW1hcnkgZnVuY3Rpb24gdG8gZGVzY3JpYmUgdGhlIHRlc3Qgc2V0IHByZWRpY3Rpb25zLg0KDQpXaGF0IGlzIHRoZSByYW5nZSBiZXR3ZWVuIHRoZSBtYXhpbXVtIGFuZCBtaW5pbXVtIHByZWRpY3RlZCByZWFkaW5nIHNjb3JlIG9uIHRoZSB0ZXN0IHNldD8NCg0KYGBge3J9DQpwcmVkVGVzdCA9IHByZWRpY3QobG1TY29yZSxuZXdkYXRhID0gcGlzYVRlc3QpDQpzdW1tYXJ5KHByZWRUZXN0KQ0KNjM3LjctMzUzLjINCiMyODQuNQ0KYGBgDQoNCiMjIyMgNC4yIFRlc3Qgc2V0IFNTRSBhbmQgUk1TRQ0KDQpXaGF0IGlzIHRoZSBzdW0gb2Ygc3F1YXJlZCBlcnJvcnMgKFNTRSkgb2YgbG1TY29yZSBvbiB0aGUgdGVzdGluZyBzZXQ/DQoNCmBgYHtyfQ0KU1NFID0gc3VtKChwcmVkVGVzdCAtcGlzYVRlc3QkcmVhZGluZ1Njb3JlKV4yICkNClNTRQ0KYGBgDQoNCldoYXQgaXMgdGhlIHJvb3QtbWVhbiBzcXVhcmVkIGVycm9yIChSTVNFKSBvZiBsbVNjb3JlIG9uIHRoZSB0ZXN0aW5nIHNldD8NCg0KYGBge3J9DQpSTVNFID0gc3FydChTU0UvbnJvdyhwaXNhVGVzdCkpDQpSTVNFDQpgYGANCg0KDQojIyMjIDQuMyBCYXNlbGluZSBwcmVkaWN0aW9uIGFuZCB0ZXN0LXNldCBTU0UNCg0KV2hhdCBpcyB0aGUgcHJlZGljdGVkIHRlc3Qgc2NvcmUgdXNlZCBpbiB0aGUgYmFzZWxpbmUgbW9kZWw/IFJlbWVtYmVyIHRvIGNvbXB1dGUgdGhpcyB2YWx1ZSB1c2luZyB0aGUgdHJhaW5pbmcgc2V0IGFuZCBub3QgdGhlIHRlc3Qgc2V0Lg0KDQpgYGB7cn0NCmJhc2VsaW5lID0gbWVhbihwaXNhVHJhaW4kcmVhZGluZ1Njb3JlKQ0KYmFzZWxpbmUNCmBgYA0KDQpXaGF0IGlzIHRoZSBzdW0gb2Ygc3F1YXJlZCBlcnJvcnMgb2YgdGhlIGJhc2VsaW5lIG1vZGVsIG9uIHRoZSB0ZXN0aW5nIHNldD8gSElOVDogV2UgY2FsbCB0aGUgc3VtIG9mIHNxdWFyZWQgZXJyb3JzIGZvciB0aGUgYmFzZWxpbmUgbW9kZWwgdGhlIHRvdGFsIHN1bSBvZiBzcXVhcmVzIChTU1QpLg0KDQpgYGB7cn0NClNTVCA9IHN1bSgoYmFzZWxpbmUtIHBpc2FUZXN0JHJlYWRpbmdTY29yZSleMikNClNTVA0KYGBgDQoNCiMjIyMgNC40IFRlc3Qtc2V0IFItc3F1YXJlZA0KDQpXaGF0IGlzIHRoZSB0ZXN0LXNldCBSLXNxdWFyZWQgdmFsdWUgb2YgbG1TY29yZT8NCmBgYHtyfQ0KUl9zcXVhcmVkID0gMS1TU0UvU1NUDQpSX3NxdWFyZWQNCmBgYA0KDQoNCg0KDQo=