Data 605 Assignment 12
Sergio Ortega Cruz
April 17, 2019
Getting the Data
The attached who.csv dataset contains real-world data from 2008. The variables included follow.
Country: name of the country LifeExp: average life expectancy for the country in years InfantSurvival: proportion of those surviving to one year or more Under5Survival: proportion of those surviving to five years or more TBFree: proportion of the population without TB.PropMD: proportion of the population who are MDs PropRN: proportion of the population who are RNs PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate TotExp: sum of personal and government expenditures.
who <- read.csv('who.csv')
head(who)## Country LifeExp InfantSurvival Under5Survival TBFree
## 1 Afghanistan 42 0.835 0.743 0.99769
## 2 Albania 71 0.985 0.983 0.99974
## 3 Algeria 71 0.967 0.962 0.99944
## 4 Andorra 82 0.997 0.996 0.99983
## 5 Angola 41 0.846 0.740 0.99656
## 6 Antigua and Barbuda 73 0.990 0.989 0.99991
## PropMD PropRN PersExp GovtExp TotExp
## 1 0.000228841 0.000572294 20 92 112
## 2 0.001143127 0.004614439 169 3128 3297
## 3 0.001060478 0.002091362 108 5184 5292
## 4 0.003297297 0.003500000 2589 169725 172314
## 5 0.000070400 0.001146162 36 1620 1656
## 6 0.000142857 0.002773810 503 12543 13046Exploring the Data
summary(who)## Country LifeExp InfantSurvival
## Afghanistan : 1 Min. :40.00 Min. :0.8350
## Albania : 1 1st Qu.:61.25 1st Qu.:0.9433
## Algeria : 1 Median :70.00 Median :0.9785
## Andorra : 1 Mean :67.38 Mean :0.9624
## Angola : 1 3rd Qu.:75.00 3rd Qu.:0.9910
## Antigua and Barbuda: 1 Max. :83.00 Max. :0.9980
## (Other) :184
## Under5Survival TBFree PropMD PropRN
## Min. :0.7310 Min. :0.9870 Min. :0.0000196 Min. :0.0000883
## 1st Qu.:0.9253 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455
## Median :0.9745 Median :0.9992 Median :0.0010474 Median :0.0027584
## Mean :0.9459 Mean :0.9980 Mean :0.0017954 Mean :0.0041336
## 3rd Qu.:0.9900 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164
## Max. :0.9970 Max. :1.0000 Max. :0.0351290 Max. :0.0708387
##
## PersExp GovtExp TotExp
## Min. : 3.00 Min. : 10.0 Min. : 13
## 1st Qu.: 36.25 1st Qu.: 559.5 1st Qu.: 584
## Median : 199.50 Median : 5385.0 Median : 5541
## Mean : 742.00 Mean : 40953.5 Mean : 41696
## 3rd Qu.: 515.25 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :6350.00 Max. :476420.0 Max. :482750
## Question 1
Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
# Linear regression model build
lm <- lm(LifeExp ~ TotExp, data=who)
# Scatterplot of dependent and independent variables
plot(LifeExp~TotExp, data=who,
xlab="Total Expenditures", ylab="Life Expectancy",
main="Life Expectancy vs Total Expenditures")
abline(lm)
summary(lm)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14P-value for F-test and TotExp is less than 0.05 but Adjusted R-squared is way too low, 0.2537. Standard error, 9.371, is higher than what we want it to be. Since p-value is less than 0.05 for both F-statistics and TotExp, we reject the null hypothesis; it is statistically significant.
# Residuals variability plot
plot(lm$fitted.values, lm$residuals,
xlab="Fitted Values", ylab="Residuals",
main="Residuals Plot for Linear Model")
abline(h=0)
## Residuals Q-Q plot
plot(lm)
Comments
QQ plot line and residual vs fitted graph suggests that at least model 2 is closer to normal distribution than model 1. However, Since mean < median, we can say that the model is still negatively skewed.
F-statistics is 507.7 and adjusted R^2 is 0.7283 P-values both for F-statistics and TotExp_power is less than 0.05. Residual standard error is 90490000 but since variables are rescaled, we cannot really say residual standard error for model 2 is higher than model 1. Relative to the size of TotExp_power coefficient, residual standard error rather decreased in model 2.
Overall, model 2 is much better than model 1.
Question 3
Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
predictdata <- data.frame(TotExpNew=c(1.5,2.5))
predict(lmNew, predictdata,interval="predict")^(1/4.6)## fit lwr upr
## 1 63.31153 35.93545 73.00793
## 2 86.50645 81.80643 90.43414Predicting the values at 1.5 adn 2.5 provides the following results.
The prediction at 1.5 is 63 years with a CI(35.93545, 73.00793).
The prediction at 2.5 is 87 year with a CI(81.80643, 90.43414)
Question 4
Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model?
LifeExp = b0+b1 x PropMd + b2 x TotExp + b3 x PropMD x TotExp
# Multiple linear regression model build
lm4 <- lm(LifeExp ~ PropMD + TotExp + TotExp:PropMD, data=who)
# Linear regression model summary
summary(lm4)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + TotExp:PropMD, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16# Residuals variability plot
plot(lm4)
## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
Comments
P-values for each variable and F-statistics are all below 0.05 so the results are statistically significant. We reject null hypothesis. Adjusted R^2 is around 0.3471 and it is substantially higher than model 1 but less than model 2. Residual standard error is smaller than model 1 but not model 2, with respect to residual standard error divided by variable coefficients. The model is not normally distributed according to QQ plot and residual vs fitted graph test, similar to model 1. Since mean < median, we can say that the model is still negatively skewed.
Question 5
Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
newdata <- data.frame(PropMD=0.03, TotExp=14)
predict(lm4, newdata,interval="predict")## fit lwr upr
## 1 107.696 84.24791 131.1441Predicting the values at PropMD=0.03, TotExp=14 provides the following results.
The prediction is 108 years with a CI(84.24791, 131.1441).
The data maxes out about the 90-100 range. Seeing a prediction of 108 becomes unrealistic when the CI also shows 132 years.
The model does what it is supposed to which is predict but it’s up the data scientist to also interpret the results of the model.
Comments
P-value for F-test and TotExp is less than 0.05 but Adjusted R-squared is way too low, 0.2537. Standard error, 9.371, is higher than what we want it to be. Since p-value is less than 0.05 for both F-statistics and TotExp, we reject the null hypothesis; it is statistically significant.
From the diagnostic plot, we can say that many of residuals are not centered around mean = 0. Normal QQ values are not fitting the theoretical line fairly well. Residual vs fitted graph tells us constant variance condition also fails. Both QQ plot and residual vs. fitted value graph tell us that this model is not normally distributed.
Question 2
Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and r re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
Transforming the variables as noted in Question 2.