HW12
Olga Shiligin
15/11/2019
The attached who.csv dataset contains real-world data from 2008. The variables included follow.
Country: name of the country
LifeExp: average life expectancy for the country in years
InfantSurvival: proportion of those surviving to one year or more
Under5Survival: proportion of those surviving to five years or more
TBFree: proportion of the population without TB.
PropMD: proportion of the population who are MDs
PropRN: proportion of the population who are RNs
PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate
GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate
TotExp: sum of personal and government expenditures.
- Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
who<- read.csv("./who.csv", header = TRUE)
head(who)## Country LifeExp InfantSurvival Under5Survival TBFree
## 1 Afghanistan 42 0.835 0.743 0.99769
## 2 Albania 71 0.985 0.983 0.99974
## 3 Algeria 71 0.967 0.962 0.99944
## 4 Andorra 82 0.997 0.996 0.99983
## 5 Angola 41 0.846 0.740 0.99656
## 6 Antigua and Barbuda 73 0.990 0.989 0.99991
## PropMD PropRN PersExp GovtExp TotExp
## 1 0.000228841 0.000572294 20 92 112
## 2 0.001143127 0.004614439 169 3128 3297
## 3 0.001060478 0.002091362 108 5184 5292
## 4 0.003297297 0.003500000 2589 169725 172314
## 5 0.000070400 0.001146162 36 1620 1656
## 6 0.000142857 0.002773810 503 12543 13046str(who)## 'data.frame': 190 obs. of 10 variables:
## $ Country : Factor w/ 190 levels "Afghanistan",..: 1 2 3 4 5 6 7 8 9 10 ...
## $ LifeExp : int 42 71 71 82 41 73 75 69 82 80 ...
## $ InfantSurvival: num 0.835 0.985 0.967 0.997 0.846 0.99 0.986 0.979 0.995 0.996 ...
## $ Under5Survival: num 0.743 0.983 0.962 0.996 0.74 0.989 0.983 0.976 0.994 0.996 ...
## $ TBFree : num 0.998 1 0.999 1 0.997 ...
## $ PropMD : num 2.29e-04 1.14e-03 1.06e-03 3.30e-03 7.04e-05 ...
## $ PropRN : num 0.000572 0.004614 0.002091 0.0035 0.001146 ...
## $ PersExp : int 20 169 108 2589 36 503 484 88 3181 3788 ...
## $ GovtExp : int 92 3128 5184 169725 1620 12543 19170 1856 187616 189354 ...
## $ TotExp : int 112 3297 5292 172314 1656 13046 19654 1944 190797 193142 ...summary(who)## Country LifeExp InfantSurvival
## Afghanistan : 1 Min. :40.00 Min. :0.8350
## Albania : 1 1st Qu.:61.25 1st Qu.:0.9433
## Algeria : 1 Median :70.00 Median :0.9785
## Andorra : 1 Mean :67.38 Mean :0.9624
## Angola : 1 3rd Qu.:75.00 3rd Qu.:0.9910
## Antigua and Barbuda: 1 Max. :83.00 Max. :0.9980
## (Other) :184
## Under5Survival TBFree PropMD PropRN
## Min. :0.7310 Min. :0.9870 Min. :0.0000196 Min. :0.0000883
## 1st Qu.:0.9253 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455
## Median :0.9745 Median :0.9992 Median :0.0010474 Median :0.0027584
## Mean :0.9459 Mean :0.9980 Mean :0.0017954 Mean :0.0041336
## 3rd Qu.:0.9900 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164
## Max. :0.9970 Max. :1.0000 Max. :0.0351290 Max. :0.0708387
##
## PersExp GovtExp TotExp
## Min. : 3.00 Min. : 10.0 Min. : 13
## 1st Qu.: 36.25 1st Qu.: 559.5 1st Qu.: 584
## Median : 199.50 Median : 5385.0 Median : 5541
## Mean : 742.00 Mean : 40953.5 Mean : 41696
## 3rd Qu.: 515.25 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :6350.00 Max. :476420.0 Max. :482750
## # checking missing values
sapply(who, function(y) sum(length(which(is.na(y)))))/nrow(who)*100## Country LifeExp InfantSurvival Under5Survival TBFree
## 0 0 0 0 0
## PropMD PropRN PersExp GovtExp TotExp
## 0 0 0 0 0Scatterplot of LifeExp~TotExp
scatter.smooth(who$LifeExp~who$TotExp, main = "Life Expectancy vs Total Expenditures", ylab = "Life Expectancy", xlab = "Total Expenditures")
linearity: the relationship of LifeExp VS TotExp is not linear.
model.1<-lm(LifeExp ~ TotExp, data = who)
summary(model.1)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14F statistics = 65.26. [F-test determines whether the relationship between the model and the response variable is statistically significant. The f-test compares the model with zero predictor variables (the intercept only model), and decides whether the added coefficients improved the model. If you get a significant result, then whatever coefficients you included in your model improved the model’s fit.]
R^2 = 0.2577. Total Expenditures can only explain 25% of Life Expectancy variation, which is quite small. [R-squared provides an estimate of the strength of the relationship between the model and the response variable.]
residual standard error = 9.371 [average error of a model, how well a model is doing to predict the data on average]
p-values = 7.714e-14 < 0.05 [Overall p value on the basis of F-statistic, normally p value less than 0.05 indicate that overall model is significant]
plot(model.1$residuals)
qqnorm(model.1$residuals)
qqline(model.1$residuals)
linearity: the relationship of LifeExp VS TotExp is not linear.
statistical independence: the errors are not statistically independent.
homoscedasticity: errors do not have constant varience.
normality: the errors are not normally distribution.
I can conclude that none of the linear regression assumptions are met. We can not consider the above model as a valid one.
- Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and r re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
scatter.smooth(I(who$LifeExp^4.6) ~ I(who$TotExp^0.06), main = "Life Expectancy vs Total Expenditures", ylab = "Life Expectancy", xlab = "Total Expenditures")
After variables transformation the relationship between LifeExp and TotExp is linear.
TotExp_0.06 <- (who$TotExp)^0.06
LifeExp_4.6<- (who$LifeExp)^4.6
model.2<-lm(LifeExp_4.6 ~ TotExp_0.06, data = who)
summary(model.2)##
## Call:
## lm(formula = LifeExp_4.6 ~ TotExp_0.06, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp_0.06 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16F statistics = 507.7. It is larger than in model.1. The further the F-statistic is from 1 the better it is.
R^2 = 0.7298. Total Expenditures can explain 73% of Life Expectancy variation, which is quite good. The strength of the relationship between the model and the response variable is better than in model.1
standard error = 90490000 is significantly larger than in model.1.
p-values = 2.2e-16 is smaller than in model.1 which indicates that model.2 is significant which means that the quality of a linear regression fit is better.
plot(model.2$residuals)
qqnorm(model.2$residuals)
qqline(model.2$residuals)
linearity: the relationship of LifeExp VS TotExp is linear.
statistical independence: the errors are statistically independent.
homoscedasticity: errors have constant varience.
normality: the errors are normally distribution.
Conclusions:
the assumptions of the linear regression are met.
Model.2 is significantly better than Model.1
- Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
model.2 is LifeExp4.6 = −736527910+620060216∗TotExp0.06
newdata<- data.frame(TotExp_0.06 = c(1.5,2.5))
predict(model.2, newdata, interval= "predict")^(1/4.6)## fit lwr upr
## 1 63.31153 35.93545 73.00793
## 2 86.50645 81.80643 90.43414- Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model? LifeExp = b0+b1 x PropMd + b2 x TotExp +b3 x PropMD x TotExp
model.3<-lm(LifeExp ~ PropMD + TotExp + PropMD:TotExp, data = who)
summary(model.3)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + PropMD:TotExp, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16plot(model.3$residuals)
qqnorm(model.3$residuals)
qqline(model.3$residuals)
F statistics = 34.49
R^2 = 0.3574
standard error = 8.765
p-values = 2.2e-16
The output of the model 3 indicates that model is weak. Also the residuals are not evenly distributed around 0. Residuals are not normally distributed.
- Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
newdata<- data.frame (PropMD = 0.03, TotExp = 14)
predict(model.3, newdata, interval = "predict")## fit lwr upr
## 1 107.696 84.24791 131.1441The prediction is unrealistic - 107 years. This is also confirming that model.3 is weak.