edX assignment link: http://bit.ly/2KE2g00
The Programme for International Student Assessment (PISA) is a test given every three years to 15-year-old students from around the world to evaluate their performance in mathematics, reading, and science. This test provides a quantitative way to compare the performance of students from different parts of the world. In this homework assignment, we will predict the reading scores of students from the United States of America on the 2009 PISA exam.
The datasets pisa2009train.csv and pisa2009test.csv contain information about the demographics and schools for American students taking the exam, derived from 2009 PISA Public-Use Data Files distributed by the United States National Center for Education Statistics (NCES). While the datasets are not supposed to contain identifying information about students taking the test, by using the data you are bound by the NCES data use agreement, which prohibits any attempt to determine the identity of any student in the datasets.
Each row in the datasets pisa2009train.csv and pisa2009test.csv represents one student taking the exam. The datasets have the following variables:
grade: The grade in school of the student (most 15-year-olds in America are in 10th grade)
male: Whether the student is male (1/0)
raceeth: The race/ethnicity composite of the student
preschool: Whether the student attended preschool (1/0)
expectBachelors: Whether the student expects to obtain a bachelor’s degree (1/0)
motherHS: Whether the student’s mother completed high school (1/0)
motherBachelors: Whether the student’s mother obtained a bachelor’s degree (1/0)
motherWork: Whether the student’s mother has part-time or full-time work (1/0)
fatherHS: Whether the student’s father completed high school (1/0)
fatherBachelors: Whether the student’s father obtained a bachelor’s degree (1/0)
fatherWork: Whether the student’s father has part-time or full-time work (1/0)
selfBornUS: Whether the student was born in the United States of America (1/0)
motherBornUS: Whether the student’s mother was born in the United States of America (1/0)
fatherBornUS: Whether the student’s father was born in the United States of America (1/0)
englishAtHome: Whether the student speaks English at home (1/0)
computerForSchoolwork: Whether the student has access to a computer for schoolwork (1/0)
read30MinsADay: Whether the student reads for pleasure for 30 minutes/day (1/0)
minutesPerWeekEnglish: The number of minutes per week the student spend in English class
studentsInEnglish: The number of students in this student’s English class at school
schoolHasLibrary: Whether this student’s school has a library (1/0)
publicSchool: Whether this student attends a public school (1/0)
urban: Whether this student’s school is in an urban area (1/0)
schoolSize: The number of students in this student’s school
readingScore: The student’s reading score, on a 1000-point scale
Load the training and testing sets using the read.csv() function, and save them as variables with the names pisaTrain and pisaTest.
How many students are there in the training set?
#3663Which variables are significant in the model? We will consider a variable signficant only if the p-value is below 0.05. (Select all that apply.)
tapply(pisa2009train$readingScore, pisa2009train$male==1, mean) FALSE TRUE
529.4637 506.5191 Which variables are missing data in at least one observation in the training set? Select all that apply.
summary(pisa2009train) grade male raceeth
Min. : 8.00 Min. :0.0000 White :1470
1st Qu.:10.00 1st Qu.:0.0000 American Indian/Alaska Native : 20
Median :10.00 Median :1.0000 Asian : 95
Mean :10.13 Mean :0.5012 Black : 228
3rd Qu.:10.00 3rd Qu.:1.0000 Hispanic : 500
Max. :12.00 Max. :1.0000 More than one race : 81
Native Hawaiian/Other Pacific Islander: 20
preschool expectBachelors motherHS motherBachelors
Min. :0.0000 Min. :0.0000 Min. :0.000 Min. :0.0000
1st Qu.:0.0000 1st Qu.:1.0000 1st Qu.:1.000 1st Qu.:0.0000
Median :1.0000 Median :1.0000 Median :1.000 Median :0.0000
Mean :0.7274 Mean :0.8343 Mean :0.896 Mean :0.3637
3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:1.000 3rd Qu.:1.0000
Max. :1.0000 Max. :1.0000 Max. :1.000 Max. :1.0000
motherWork fatherHS fatherBachelors fatherWork
Min. :0.0000 Min. :0.0000 Min. :0.0000 Min. :0.0000
1st Qu.:0.0000 1st Qu.:1.0000 1st Qu.:0.0000 1st Qu.:1.0000
Median :1.0000 Median :1.0000 Median :0.0000 Median :1.0000
Mean :0.7357 Mean :0.8741 Mean :0.3484 Mean :0.8571
3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:1.0000
Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :1.0000
selfBornUS motherBornUS fatherBornUS englishAtHome
Min. :0.0000 Min. :0.00 Min. :0.0000 Min. :0.0000
1st Qu.:1.0000 1st Qu.:1.00 1st Qu.:1.0000 1st Qu.:1.0000
Median :1.0000 Median :1.00 Median :1.0000 Median :1.0000
Mean :0.9362 Mean :0.79 Mean :0.7854 Mean :0.8815
3rd Qu.:1.0000 3rd Qu.:1.00 3rd Qu.:1.0000 3rd Qu.:1.0000
Max. :1.0000 Max. :1.00 Max. :1.0000 Max. :1.0000
computerForSchoolwork read30MinsADay minutesPerWeekEnglish studentsInEnglish
Min. :0.0000 Min. :0.0000 Min. : 0.0 Min. : 1.00
1st Qu.:1.0000 1st Qu.:0.0000 1st Qu.: 225.0 1st Qu.:20.00
Median :1.0000 Median :0.0000 Median : 250.0 Median :25.00
Mean :0.9155 Mean :0.3016 Mean : 269.8 Mean :24.56
3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.: 300.0 3rd Qu.:30.00
Max. :1.0000 Max. :1.0000 Max. :1680.0 Max. :75.00
schoolHasLibrary publicSchool urban schoolSize
Min. :0.0000 Min. :0.0000 Min. :0.0000 Min. : 100
1st Qu.:1.0000 1st Qu.:1.0000 1st Qu.:0.0000 1st Qu.: 712
Median :1.0000 Median :1.0000 Median :0.0000 Median :1233
Mean :0.9714 Mean :0.9176 Mean :0.3629 Mean :1372
3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:1900
Max. :1.0000 Max. :1.0000 Max. :1.0000 Max. :6694
readingScore
Min. :244.5
1st Qu.:455.8
Median :520.2
Mean :518.0
3rd Qu.:581.4
Max. :746.0
# preschool expectBachelor motherHS motherBachelors motherWork
#fatherHS fatherBachelors fatherWork selfBornUS
#read30MinsADay minutesPerWeekEnglish studentsInEnglish schoolHasLibrary
#schoolSizeLinear regression discards observations with missing data, so we will remove all such observations from the training and testing sets. Later in the course, we will learn about imputation, which deals with missing data by filling in missing values with plausible information.
Type the following commands into your R console to remove observations with any missing value from pisaTrain and pisaTest:
pisaTrain = na.omit(pisaTrain)
pisaTest = na.omit(pisaTest)How many observations are now in the training set?
pisa2009train = na.omit(pisa2009train)
pisa2009test = na.omit(pisa2009test)How many observations are now in the testing set?
#990Factor variables are variables that take on a discrete set of values, like the “Region” variable in the WHO dataset from the second lecture of Unit 1. This is an unordered factor because there isn’t any natural ordering between the levels. An ordered factor has a natural ordering between the levels (an example would be the classifications “large,” “medium,” and “small”).
Which of the following variables is an unordered factor with at least 3 levels? (Select all that apply.)
table(pisa2009test$grade)
9 10 11 12
74 716 199 1 #raceethWhich of the following variables is an ordered factor with at least 3 levels? (Select all that apply.)
#gradeTo include unordered factors in a linear regression model, we define one level as the “reference level” and add a binary variable for each of the remaining levels. In this way, a factor with n levels is replaced by n-1 binary variables. The reference level is typically selected to be the most frequently occurring level in the dataset.
As an example, consider the unordered factor variable “color”, with levels “red”, “green”, and “blue”. If “green” were the reference level, then we would add binary variables “colorred” and “colorblue” to a linear regression problem. All red examples would have colorred=1 and colorblue=0. All blue examples would have colorred=0 and colorblue=1. All green examples would have colorred=0 and colorblue=0.
Now, consider the variable “raceeth” in our problem, which has levels “American Indian/Alaska Native”, “Asian”, “Black”, “Hispanic”, “More than one race”, “Native Hawaiian/Other Pacific Islander”, and “White”. Because it is the most common in our population, we will select White as the reference level.
Which binary variables will be included in the regression model? (Select all that apply.)
#"American Indian/Alaska Native", "Asian", "Black", "Hispanic", "More than one race", "Native Hawaiian/Other Pacific Islander"Consider again adding our unordered factor race to the regression model with reference level “White”.
For a student who is Asian, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)
#"American Indian/Alaska Native", "Black", "Hispanic", "More than one race", "Native Hawaiian/Other Pacific Islander"For a student who is white, which binary variables would be set to 0? All remaining variables will be set to 1. (Select all that apply.)
#"American Indian/Alaska Native", "Asian", "Black", "Hispanic", "More than one race", "Native Hawaiian/Other Pacific Islander"Because the race variable takes on text values, it was loaded as a factor variable when we read in the dataset with read.csv() – you can see this when you run str(pisaTrain) or str(pisaTest). However, by default R selects the first level alphabetically (“American Indian/Alaska Native”) as the reference level of our factor instead of the most common level (“White”). Set the reference level of the factor by typing the following two lines in your R console:
pisaTrain$raceeth = relevel(pisaTrain$raceeth, "White")
pisaTest$raceeth = relevel(pisaTest$raceeth, "White")Now, build a linear regression model (call it lmScore) using the training set to predict readingScore using all the remaining variables.
It would be time-consuming to type all the variables, but R provides the shorthand notation “readingScore ~ .” to mean “predict readingScore using all the other variables in the data frame.” The period is used to replace listing out all of the independent variables. As an example, if your dependent variable is called “Y”, your independent variables are called “X1”, “X2”, and “X3”, and your training data set is called “Train”, instead of the regular notation:
LinReg = lm(Y ~ X1 + X2 + X3, data = Train)You would use the following command to build your model:
LinReg = lm(Y ~ ., data = Train)What is the Multiple R-squared value of lmScore on the training set?
summary(lmScore)
Call:
lm(formula = readingScore ~ ., data = pisa2009train)
Residuals:
Min 1Q Median 3Q Max
-247.44 -48.86 1.86 49.77 217.18
Coefficients:
Estimate Std. Error t value
(Intercept) 143.766333 33.841226 4.248
grade 29.542707 2.937399 10.057
male -14.521653 3.155926 -4.601
raceethAmerican Indian/Alaska Native -67.277327 16.786935 -4.008
raceethAsian -4.110325 9.220071 -0.446
raceethBlack -67.012347 5.460883 -12.271
raceethHispanic -38.975486 5.177743 -7.528
raceethMore than one race -16.922522 8.496268 -1.992
raceethNative Hawaiian/Other Pacific Islander -5.101601 17.005696 -0.300
preschool -4.463670 3.486055 -1.280
expectBachelors 55.267080 4.293893 12.871
motherHS 6.058774 6.091423 0.995
motherBachelors 12.638068 3.861457 3.273
motherWork -2.809101 3.521827 -0.798
fatherHS 4.018214 5.579269 0.720
fatherBachelors 16.929755 3.995253 4.237
fatherWork 5.842798 4.395978 1.329
selfBornUS -3.806278 7.323718 -0.520
motherBornUS -8.798153 6.587621 -1.336
fatherBornUS 4.306994 6.263875 0.688
englishAtHome 8.035685 6.859492 1.171
computerForSchoolwork 22.500232 5.702562 3.946
read30MinsADay 34.871924 3.408447 10.231
minutesPerWeekEnglish 0.012788 0.010712 1.194
studentsInEnglish -0.286631 0.227819 -1.258
schoolHasLibrary 12.215085 9.264884 1.318
publicSchool -16.857475 6.725614 -2.506
urban -0.110132 3.962724 -0.028
schoolSize 0.006540 0.002197 2.977
Pr(>|t|)
(Intercept) 2.24e-05 ***
grade < 2e-16 ***
male 4.42e-06 ***
raceethAmerican Indian/Alaska Native 6.32e-05 ***
raceethAsian 0.65578
raceethBlack < 2e-16 ***
raceethHispanic 7.29e-14 ***
raceethMore than one race 0.04651 *
raceethNative Hawaiian/Other Pacific Islander 0.76421
preschool 0.20052
expectBachelors < 2e-16 ***
motherHS 0.32001
motherBachelors 0.00108 **
motherWork 0.42517
fatherHS 0.47147
fatherBachelors 2.35e-05 ***
fatherWork 0.18393
selfBornUS 0.60331
motherBornUS 0.18182
fatherBornUS 0.49178
englishAtHome 0.24153
computerForSchoolwork 8.19e-05 ***
read30MinsADay < 2e-16 ***
minutesPerWeekEnglish 0.23264
studentsInEnglish 0.20846
schoolHasLibrary 0.18749
publicSchool 0.01226 *
urban 0.97783
schoolSize 0.00294 **
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 73.81 on 2385 degrees of freedom
Multiple R-squared: 0.3251, Adjusted R-squared: 0.3172
F-statistic: 41.04 on 28 and 2385 DF, p-value: < 2.2e-16Note that this R-squared is lower than the ones for the models we saw in the lectures and recitation. This does not necessarily imply that the model is of poor quality. More often than not, it simply means that the prediction problem at hand (predicting a student’s test score based on demographic and school-related variables) is more difficult than other prediction problems (like predicting a team’s number of wins from their runs scored and allowed, or predicting the quality of wine from weather conditions).
Consider two students A and B. They have all variable values the same, except that student A is in grade 11 and student B is in grade 9. What is the predicted reading score of student A minus the predicted reading score of student B?
What is the meaning of the coefficient associated with variable raceethAsian?
Based on the significance codes, which variables are candidates for removal from the model? Select all that apply. (We’ll assume that the factor variable raceeth should only be removed if none of its levels are significant.)
Using the “predict” function and supplying the “newdata” argument, use the lmScore model to predict the reading scores of students in pisaTest. Call this vector of predictions “predTest”. Do not change the variables in the model (for example, do not remove variables that we found were not significant in the previous part of this problem). Use the summary function to describe the test set predictions.
What is the range between the maximum and minimum predicted reading score on the test set?
637.7-353.2
[1] 284.5What is the sum of squared errors (SSE) of lmScore on the testing set?
sum((predTest-pisa2009test$readingScore)^2)
[1] 5762082What is the root-mean squared error (RMSE) of lmScore on the testing set?
sqrt(mean((predTest-pisa2009test$readingScore)^2))
[1] 76.29079What is the predicted test score used in the baseline model? Remember to compute this value using the training set and not the test set.
baseline
[1] 517.9629What is the sum of squared errors of the baseline model on the testing set? HINT: We call the sum of squared errors for the baseline model the total sum of squares (SST).
sum((baseline-pisa2009test$readingScore)^2)
[1] 7802354What is the test-set R-squared value of lmScore?