Data605: HW12
jfor: Josh Forster
2023-04-18
Provided dataset from 2008 with the following variables included: Country: name of the country LifeExp: average life expectancy for the country in years InfantSurvival: proportion of those surviving to one year or more Under5Survival: proportion of those surviving to five years or more TBFree: proportion of the population without TB. PropMD: proportion of the population who are MDs PropRN: proportion of the population who are RNs PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate TotExp: sum of personal and government expenditures.
df <- read.csv('https://raw.githubusercontent.com/jforster19/Data605/main/HW12_input.csv')
head(df)## Country LifeExp InfantSurvival Under5Survival TBFree PropMD
## 1 Afghanistan 42 0.835 0.743 0.99769 0.000228841
## 2 Albania 71 0.985 0.983 0.99974 0.001143127
## 3 Algeria 71 0.967 0.962 0.99944 0.001060478
## 4 Andorra 82 0.997 0.996 0.99983 0.003297297
## 5 Angola 41 0.846 0.740 0.99656 0.000070400
## 6 Antigua and Barbuda 73 0.990 0.989 0.99991 0.000142857
## PropRN PersExp GovtExp TotExp
## 1 0.000572294 20 92 112
## 2 0.004614439 169 3128 3297
## 3 0.002091362 108 5184 5292
## 4 0.003500000 2589 169725 172314
## 5 0.001146162 36 1620 1656
## 6 0.002773810 503 12543 13046summary(df)## Country LifeExp InfantSurvival Under5Survival
## Length:190 Min. :40.00 Min. :0.8350 Min. :0.7310
## Class :character 1st Qu.:61.25 1st Qu.:0.9433 1st Qu.:0.9253
## Mode :character Median :70.00 Median :0.9785 Median :0.9745
## Mean :67.38 Mean :0.9624 Mean :0.9459
## 3rd Qu.:75.00 3rd Qu.:0.9910 3rd Qu.:0.9900
## Max. :83.00 Max. :0.9980 Max. :0.9970
## TBFree PropMD PropRN PersExp
## Min. :0.9870 Min. :0.0000196 Min. :0.0000883 Min. : 3.00
## 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455 1st Qu.: 36.25
## Median :0.9992 Median :0.0010474 Median :0.0027584 Median : 199.50
## Mean :0.9980 Mean :0.0017954 Mean :0.0041336 Mean : 742.00
## 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164 3rd Qu.: 515.25
## Max. :1.0000 Max. :0.0351290 Max. :0.0708387 Max. :6350.00
## GovtExp TotExp
## Min. : 10.0 Min. : 13
## 1st Qu.: 559.5 1st Qu.: 584
## Median : 5385.0 Median : 5541
## Mean : 40953.5 Mean : 41696
## 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :476420.0 Max. :482750- Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
The scatterplot of total expendenditure vs life expectancy
ggplot(df,aes(x=TotExp,y=LifeExp)) +
geom_point()
The relationship of the variables does not appear to be linear perhaps warranting a quadratic term or further transformation
lm.out <- lm(LifeExp~TotExp,data=df)
summary(lm.out)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14The F-Statistic indicates that the model has a statistical difference from an intercept only model. The residuals are not centered on zero (only slightly skewed) or evenly spread as the left skew is much higher than the right, which based on the scatterplot makes some sense. The R-Squared is 25.77% which indicates that not much of the variation is explained by the model. The Standard error provided is about 8 times smaller than the coefficient estimate which indicates there is an acceptable level of variability within the calculated value of the coefficient. Lastly, the p-value for Total expenditures is statistically significant as a predictor even at a very low alpha level.
Diagnostic Plots
par(mfrow=c(2,2))
plot(lm.out)
The diagnostic plots indicate that several of the assumptions needed for a simple linear regression are violated. The fitted residuals show a clear pattern although it appear to be non-linear and the residuals do not appear to be normal with substantial skews on both ends of the distribution.
- Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and r re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
Simple Linear Regression and Diagnostic Plots
df$TotExp_pwr <- df$TotExp^0.06
df$LifeExp_pwr <- df$LifeExp^4.6
power_lm <- lm(LifeExp_pwr~TotExp_pwr,data=df)
summary(power_lm)##
## Call:
## lm(formula = LifeExp_pwr ~ TotExp_pwr, data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp_pwr 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16par(mfrow=c(2,2))
plot(power_lm)
Reviewing Scatterplot of Newly Transformed Fields
ggplot(df,aes(TotExp_pwr,LifeExp_pwr)) +
geom_point()
Similar to the prior version of the model the F-statistic is significant indicating the model is different than only using the intercept to predict the independent variable. From an initial scan of the residual it appears that required assumptions still seem to be violated given they are not centered around zero and do not seem to be evenly spread on both the higher and lower values of transformed total expenditures. The p-value seems to be significant and the standard error for the coefficient is 22 times smaller indicating lower variability in the coefficient estimate. The R-Squared term is much higher within this model as it explains far more variation in the dependent variable at 72.98%.
The fitted residuals do not appear to display a pattern across the range of values, but they are not centered around zero so it still violates a key assumption of linear models. The QQ plot does seem to show better approximation of normality although there is a potentially problematic left skew present.
Although both models do not appear to be valid forms of linear regression, the second transformed model does perform better in explaining the variation in life expectancy if one was to ignore the other shortcomings.
- Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
Forecasted value
\(\hat{y} = -736527910 + 620060216 * TotExp\)
Need to take the inverse of the power of 4.6 to calculate the underlying term for life expectancy
(620060216 * 1.5 - 736527910)^(1/4.6)## [1] 63.31153(620060216 * 2.5 - 736527910)^(1/4.6)## [1] 86.50645- Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model? LifeExp = b0+b1 x PropMd + b2 x TotExp +b3 x PropMD x TotExp
mlm.out <- lm(LifeExp~PropMD+TotExp+PropMD*TotExp,data=df)
summary(mlm.out)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + PropMD * TotExp, data = df)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16The F-Statistic indicates that the model explains enough of the variation in the response to warrant further review of the coefficients. Another key statistic in the output is the R-Squared and Adjusted R-Squared terms that show that approximately 35% of the variation can be explained using this multiple linear regression model. The standard error ratios to their coefficients appear to be within an acceptable range (e.g. 5 to 10) except for the interaction term, but it is close enough to the low end of the ratio. It may indicate there is more variability within the coefficient that might cause larger residuals. All of the p-values for each of the dependent variables show that they are statistically significant within the model even at very conservative alpha levels. Overall, the base summary statistics indicate the model’s residuals are somewhat close to zero, but have a slight leftward skew which we can review further with diagnostic plots to validate key assumptions.
Diagnostic Plots
par(mfrow=c(2,2))
plot(mlm.out)## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
## Warning in sqrt(crit * p * (1 - hh)/hh): NaNs produced
From a visual review of the plotted residuals it is more obvious that the residuals show a clear parabolic pattern and despite appearing to center around zero are not spread very evenly across the range of fitted values. The residuals appear to be somewhat normal except for potentially two large outliers that also are shown to be quite impactful to the model based on the leverage in the bottom right quadrant of the diagnostic plots.
The model itself is merely okay as it only explains about 9 percent more of the variation in life expectancy despite more complexity than the simple linear regression. It also violates the assumption for random distribution of fitted residuals which probably should disqualify someone from using this regression model as a forecasting tool.
- Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
\(\hat{y} = 6.277*10^1 + 1.497*10^{3}*PropMD +7.233*10^{-5}*TotExp - 6.026*10^{-3}*PropMD * TotExp\)
6.277*10^1 + 1.497*10^{3}*0.03 +7.233*10^{-5}*14 - 6.026*10^{-3}*0.03 * 14## [1] 107.6785The forecasted value for life expectancy is 107 which is far higher than any of the values available in the dataset for this response variable despite the fact that one of the inputs (TotExp) is close to it’s minimum value. It also is very unlikely that any person despite a high proportion of doctors would have such a high life expectancy so it is reasonable to say this forecast is unrealistic in providing an accurate estimate.