data 605 assignment 12
Jun Pan
April 20, 2019
`
The attached who.csv dataset contains real-world data from 2008. The variables included follow.
Country: name of the country LifeExp: average life expectancy for the country in years InfantSurvival: proportion of those surviving to one year or more Under5Survival: proportion of those surviving to five years or more TBFree: proportion of the population without TB. PropMD: proportion of the population who are MDs PropRN: proportion of the population who are RNs PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate TotExp: sum of personal and government expenditures.
- Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
life_exp <- read.csv("C:/Users/tbao/Desktop/CUNY MSDS notes/DATA 605/WK 12/who.csv")summary(life_exp)## Country LifeExp InfantSurvival
## Afghanistan : 1 Min. :40.00 Min. :0.8350
## Albania : 1 1st Qu.:61.25 1st Qu.:0.9433
## Algeria : 1 Median :70.00 Median :0.9785
## Andorra : 1 Mean :67.38 Mean :0.9624
## Angola : 1 3rd Qu.:75.00 3rd Qu.:0.9910
## Antigua and Barbuda: 1 Max. :83.00 Max. :0.9980
## (Other) :184
## Under5Survival TBFree PropMD PropRN
## Min. :0.7310 Min. :0.9870 Min. :0.0000196 Min. :0.0000883
## 1st Qu.:0.9253 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455
## Median :0.9745 Median :0.9992 Median :0.0010474 Median :0.0027584
## Mean :0.9459 Mean :0.9980 Mean :0.0017954 Mean :0.0041336
## 3rd Qu.:0.9900 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164
## Max. :0.9970 Max. :1.0000 Max. :0.0351290 Max. :0.0708387
##
## PersExp GovtExp TotExp
## Min. : 3.00 Min. : 10.0 Min. : 13
## 1st Qu.: 36.25 1st Qu.: 559.5 1st Qu.: 584
## Median : 199.50 Median : 5385.0 Median : 5541
## Mean : 742.00 Mean : 40953.5 Mean : 41696
## 3rd Qu.: 515.25 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :6350.00 Max. :476420.0 Max. :482750
## str(life_exp)## 'data.frame': 190 obs. of 10 variables:
## $ Country : Factor w/ 190 levels "Afghanistan",..: 1 2 3 4 5 6 7 8 9 10 ...
## $ LifeExp : int 42 71 71 82 41 73 75 69 82 80 ...
## $ InfantSurvival: num 0.835 0.985 0.967 0.997 0.846 0.99 0.986 0.979 0.995 0.996 ...
## $ Under5Survival: num 0.743 0.983 0.962 0.996 0.74 0.989 0.983 0.976 0.994 0.996 ...
## $ TBFree : num 0.998 1 0.999 1 0.997 ...
## $ PropMD : num 2.29e-04 1.14e-03 1.06e-03 3.30e-03 7.04e-05 ...
## $ PropRN : num 0.000572 0.004614 0.002091 0.0035 0.001146 ...
## $ PersExp : int 20 169 108 2589 36 503 484 88 3181 3788 ...
## $ GovtExp : int 92 3128 5184 169725 1620 12543 19170 1856 187616 189354 ...
## $ TotExp : int 112 3297 5292 172314 1656 13046 19654 1944 190797 193142 ...#scatter plot of single linear model
life_exp_lm <- lm(LifeExp ~ TotExp, data=life_exp)
plot(LifeExp~TotExp, data=life_exp,
xlab="Total Expenditures", ylab="Life Expectancy",
main="Life Expectancy vs Total Expenditures")
abline(life_exp_lm)
summary(life_exp_lm)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = life_exp)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14plot(life_exp_lm$fitted.values, life_exp_lm$residuals,
xlab="Fitted Values", ylab="Residuals",
main="Residuals Plot")
abline(h=0)
qqnorm(life_exp_lm$residuals)
qqline(life_exp_lm$residuals)
From the scatter plot, residual plot and qq plot, we can conclude that the relationship is not linear and the observers are not randomly distributed. Residual standard error is 9.371 and F-statistic is 65.26. Average life expectancy is 67.38, the SE is reasonable. R2 is only 0.2577 (so the model explains only 25.77% of variability). Although F-statistics is high. P-value is nearly 0, the relationship between the two variables is not due to random variation.
- Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and r re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
LifeExp4.6 <- life_exp$LifeExp^4.6
TotExp0.06 <- life_exp$TotExp^0.06
life_exp_lm2 <- lm(LifeExp4.6 ~ TotExp0.06)
plot(LifeExp4.6~TotExp0.06,
xlab="Total Expenditures", ylab="Life Expectancy",
main="Life Expectancy vs Total Expenditures (Transformed)")
abline(life_exp_lm)
summary(life_exp_lm2)##
## Call:
## lm(formula = LifeExp4.6 ~ TotExp0.06)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp0.06 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16plot(life_exp_lm2$fitted.values, life_exp_lm2$residuals,
xlab="Fitted Values", ylab="Residuals",
main="Residuals Plot")
abline(h=0)
qqnorm(life_exp_lm2$residuals)
qqline(life_exp_lm2$residuals)
From the scatter plot, we can see the retionship is close to linear. The residual plot shows the most of variables are randomly distributed around 0 although looks a little big left skewed. The qq plot shows a nearly linear relationship between the two variables.
Residual standard error is 90,490,000 and. SE is high compared to the non-transformed variables. R2 is 0.7298, which isexplains 72.98% of variability. F-statistic is 507.7, P-value is again nearly 0, the relationship is not due to random variation.
- Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
range <- data.frame(TotExp0.06 = c(1.5,2.5))
predict(life_exp_lm2, range,interval="predict")^(1/4.6)## fit lwr upr
## 1 63.31153 35.93545 73.00793
## 2 86.50645 81.80643 90.43414Predicting the values at 1.5 adn 2.5 provides the following results.
The prediction at 1.5 is 63 years with a CI(35.93545, 73.00793).
The prediction at 2.5 is 87 year with a CI(81.80643, 90.43414)
- Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model?
LifeExp = b0+b1 x PropMd + b2 x TotExp +b3 x PropMD x TotExp
life_exp_lm3 <- lm(LifeExp ~ PropMD + TotExp + TotExp:PropMD, data=life_exp)
summary(life_exp_lm3)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + TotExp:PropMD, data = life_exp)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16plot(life_exp_lm3$fitted.values, life_exp_lm3$residuals,
xlab="Fitted Values", ylab="Residuals",
main="Residuals Plot")
abline(h=0)
qqnorm(life_exp_lm3$residuals)
qqline(life_exp_lm3$residuals)
Residuals plots shows there is no constant variability and that residuals are not normally distributed. QQ plot shows not a very good linear fit. Residual standard error is 8.765 and F-statistic is 34.49. Compare to the average of life expectancy is 67.38, the SE is Ok. R2 equals to 0.3574 which means the model explains only 35.74% of variability. F-statistics is fairly high and P-value is nearly 0, so the relationship is not due to random variation.
- Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
condition <- data.frame(PropMD=0.03, TotExp=14)
predict(life_exp_lm3, condition,interval="predict")## fit lwr upr
## 1 107.696 84.24791 131.1441The prediction is 107.70 years with 95% confidence interval between 84.25 and 131.14. The prediction does not make sense in real world. There is nothing in our data to support this prediction and it goes against common sense. It is not a good and useful model.