Assignment 12
Week 12, Regression 2
Fundamentals of Computational Mathematics
CUNY MSDS DATA 605, Fall 2018
Rose Koh
11/17/2018
The attached who.csv dataset contains real-world data from 2008. The variables included follow.
Country: name of the country
LifeExp: average life expectancy for the country in years
InfantSurvival: proportion of those surviving to one year or more
Under5Survival: proportion of those surviving to five years or more
TBFree: proportion of the population without TB.
PropMD: proportion of the population who are MDs
PropRN: proportion of the population who are RNs
PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate
GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate
TotExp: sum of personal and government expenditures.
df <- read.csv("who.csv", stringsAsFactors = F, header = T)
str(df)## 'data.frame': 190 obs. of 10 variables:
## $ Country : chr "Afghanistan" "Albania" "Algeria" "Andorra" ...
## $ LifeExp : int 42 71 71 82 41 73 75 69 82 80 ...
## $ InfantSurvival: num 0.835 0.985 0.967 0.997 0.846 0.99 0.986 0.979 0.995 0.996 ...
## $ Under5Survival: num 0.743 0.983 0.962 0.996 0.74 0.989 0.983 0.976 0.994 0.996 ...
## $ TBFree : num 0.998 1 0.999 1 0.997 ...
## $ PropMD : num 2.29e-04 1.14e-03 1.06e-03 3.30e-03 7.04e-05 ...
## $ PropRN : num 0.000572 0.004614 0.002091 0.0035 0.001146 ...
## $ PersExp : int 20 169 108 2589 36 503 484 88 3181 3788 ...
## $ GovtExp : int 92 3128 5184 169725 1620 12543 19170 1856 187616 189354 ...
## $ TotExp : int 112 3297 5292 172314 1656 13046 19654 1944 190797 193142 ...head(df)## Country LifeExp InfantSurvival Under5Survival TBFree
## 1 Afghanistan 42 0.835 0.743 0.99769
## 2 Albania 71 0.985 0.983 0.99974
## 3 Algeria 71 0.967 0.962 0.99944
## 4 Andorra 82 0.997 0.996 0.99983
## 5 Angola 41 0.846 0.740 0.99656
## 6 Antigua and Barbuda 73 0.990 0.989 0.99991
## PropMD PropRN PersExp GovtExp TotExp
## 1 0.000228841 0.000572294 20 92 112
## 2 0.001143127 0.004614439 169 3128 3297
## 3 0.001060478 0.002091362 108 5184 5292
## 4 0.003297297 0.003500000 2589 169725 172314
## 5 0.000070400 0.001146162 36 1620 1656
## 6 0.000142857 0.002773810 503 12543 13046summary(df)## Country LifeExp InfantSurvival Under5Survival
## Length:190 Min. :40.00 Min. :0.8350 Min. :0.7310
## Class :character 1st Qu.:61.25 1st Qu.:0.9433 1st Qu.:0.9253
## Mode :character Median :70.00 Median :0.9785 Median :0.9745
## Mean :67.38 Mean :0.9624 Mean :0.9459
## 3rd Qu.:75.00 3rd Qu.:0.9910 3rd Qu.:0.9900
## Max. :83.00 Max. :0.9980 Max. :0.9970
## TBFree PropMD PropRN
## Min. :0.9870 Min. :0.0000196 Min. :0.0000883
## 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455
## Median :0.9992 Median :0.0010474 Median :0.0027584
## Mean :0.9980 Mean :0.0017954 Mean :0.0041336
## 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164
## Max. :1.0000 Max. :0.0351290 Max. :0.0708387
## PersExp GovtExp TotExp
## Min. : 3.00 Min. : 10.0 Min. : 13
## 1st Qu.: 36.25 1st Qu.: 559.5 1st Qu.: 584
## Median : 199.50 Median : 5385.0 Median : 5541
## Mean : 742.00 Mean : 40953.5 Mean : 41696
## 3rd Qu.: 515.25 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :6350.00 Max. :476420.0 Max. :482750attach(df)- Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
library(ggplot2)
p1 <- ggplot(df, aes(TotExp,LifeExp)) +
geom_point() +
geom_smooth(method='lm', se = FALSE, color = 'blue') +
labs(title = "Scatterplot of LifeExp~TotExp") +
scale_x_continuous(labels = scales::comma)
p1
lm1 <- lm(LifeExp~TotExp)
summary(lm1)##
## Call:
## lm(formula = LifeExp ~ TotExp)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14par(mfrow = c(2,2))
plot(lm1)
An F-Test compares a model with no predictors to the model that we specify. The F-statistic from this model is 65.26 (the model has one regression degree of freedom and 188 degrees of freedom). F-table value for one regression degree of freedom and 120 residual degrees of freedom is 6.851. Since our model’s F-statistic is much greater than the F-table value, this suggests we can reject the Null hypothesis, a regression model with a zero coefficient. The p-value is near 0. The \(R^2 = 0.2577\) tells us that 25.77% of the variation in the data is accounted for the model; The model does not strongly fit the data. The standard error is a reasonably small percentage of the coefficient.
The residual plots shows there is no constant variabiliity and the residuals are not normally distributed. The model is not a good fit to describe the relationship and it is clear that the relationship is not linear.
- Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
detach(df)
df$LifeExp4.6 <- df$LifeExp^4.6
df$TotExp0.06 <- df$TotExp^.06
attach(df)
p2 <- ggplot(df, aes(TotExp0.06, LifeExp4.6)) +
geom_point() +
geom_smooth(method='lm', se = FALSE, color = 'blue') +
labs(title = "Scatterplot of LifeExp~TotExp") +
scale_y_continuous(labels = scales::comma)
p2
lm2 <- lm(LifeExp4.6~TotExp0.06)
summary(lm2)##
## Call:
## lm(formula = LifeExp4.6 ~ TotExp0.06)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp0.06 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(lm2)
The transformed model shows F-statistic 507.7 that is better than the previous model. The P-value is also more statistically significant than the previous model. The \(R^2\) value is 0.7298, which is higher than the previous model. The standard error is 90,490,000 and is significantly higher than the previous model. The residual plot shows the variability is fairly constant with a few outliers. The distribution seems near normal with some deviation at the tails. It is a much better model to describe the relationship than the previous model.
- Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
The model is \(\widehat{LifeExp4.6} = 736527910 + 620060216 * TotExp0.06\)
predict.TotExp <- function(x){
n <- lm2$coefficients[1] + lm2$coefficients[2] * x
return(n ** (1/4.6)) # Must transform the data into a sensible number
}
#TotExp^.06=1.5
predict.TotExp(1.5)## (Intercept)
## 63.31153#TotExp^.06=2.5
predict.TotExp(2.5)## (Intercept)
## 86.50645- Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model?
LifeExp = b0+b1 x PropMd + b2 x TotExp +b3 x PropMD x TotExp
\(LifeExp = \beta0 + \beta1 \times PropMd + \beta2 \times TotExp + \beta3 \times PropMD \times TotExp\)
detach(df)
df$PropMD.TotExp <- df$PropMD * df$TotExp
attach(df)
lm4 <- lm(LifeExp ~ PropMD + TotExp + PropMD.TotExp)
summary(lm4)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + PropMD.TotExp)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD.TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16par(mfrow = c(2,2))
plot(lm4)## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
The F-statistics is fairly high but lower than the value of the first model. The P-value <2.2e-16 is statistically signifiicant. A residual standard error 8.765 seems to have improved compare to the first moddel. The \(R^2\) is only 0.3574, meaniing the model explains only 35.74% of variability which is pretty low. This model is somewhat similar to the first model and not as good as the previous transformation model. The residual plot shows there is no constant variability and the distribution doesn’t seem normal. This isn’t good model to desribe the relationship.
- Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
\(\widehat{LifeExp} = \beta0 + \beta1 \times PropMd + \beta2 \times TotExp + \beta3 \times PropMD \times TotExp\)
tbl4 <- summary(lm4)$coefficients
predict5 <- function(prop.md, tot.exp){
n <- tbl4[1] + tbl4[2] * prop.md + tbl4[3] * tot.exp + tbl4[4] * prop.md * tot.exp
return(n)
}
prop.md <- .03
tot.exp <- 14.
predict5(prop.md, tot.exp)## [1] 107.696# OR
data <- data.frame(PropMD=0.03, TotExp=14, PropMD.TotExp = 14*0.03)
predict(lm4, data, interval="predict")## fit lwr upr
## 1 107.696 84.24791 131.1441The predicted life expectancy is 107 years old with 95% confidende interval between 84 yo and 131 yo. The forrecast seem unrealiistic.
Look at the following data, Where the total Expenditure of $13 shows Life Expectancy of 49:
df[TotExp<15,]## Country LifeExp InfantSurvival Under5Survival TBFree PropMD
## 28 Burundi 49 0.891 0.819 0.99286 2.45e-05
## PropRN PersExp GovtExp TotExp LifeExp4.6 TotExp0.06 PropMD.TotExp
## 28 0.000164933 3 10 13 59552770 1.166371 0.0003185How does increasing $1 in Total Expenditure jumps the value from 49 to 107?
df[LifeExp == max(LifeExp),]## Country LifeExp InfantSurvival Under5Survival TBFree PropMD
## 85 Japan 83 0.997 0.996 0.99971 0.002113049
## PropRN PersExp GovtExp TotExp LifeExp4.6 TotExp0.06 PropMD.TotExp
## 85 0.009461544 2936 159192 162128 672603658 2.053958 342.5844The maximum life expectancy shows in Japan at 83, with Tot Exp of 162128.
I would say this is not a good model to support life expectancy forecast.