Assignment #12
Sie Siong Wong
11/8/2020
Problem Set
The attached who.csv dataset contains real-world data from 2008. The variables included follow.
- Country: name of the country
- LifeExp: average life expectancy for the country in years
- InfantSurvival: proportion of those surviving to one year or more
- Under5Survival: proportion of those surviving to five years or more
- TBFree: proportion of the population without TB.
- PropMD: proportion of the population who are MDs
- PropRN: proportion of the population who are RNs
- PersExp: mean personal expenditures on healthcare in US dollars at average exchange rate
- GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate
- TotExp: sum of personal and government expenditures.
Load the Packages
library(dplyr)Load the Data
who <- read.csv('https://raw.githubusercontent.com/SieSiongWong/DATA-605/master/who.csv')
head(who, 5)## Country LifeExp InfantSurvival Under5Survival TBFree PropMD
## 1 Afghanistan 42 0.835 0.743 0.99769 0.000228841
## 2 Albania 71 0.985 0.983 0.99974 0.001143127
## 3 Algeria 71 0.967 0.962 0.99944 0.001060478
## 4 Andorra 82 0.997 0.996 0.99983 0.003297297
## 5 Angola 41 0.846 0.740 0.99656 0.000070400
## PropRN PersExp GovtExp TotExp
## 1 0.000572294 20 92 112
## 2 0.004614439 169 3128 3297
## 3 0.002091362 108 5184 5292
## 4 0.003500000 2589 169725 172314
## 5 0.001146162 36 1620 1656Statistic Summary
str(who)## 'data.frame': 190 obs. of 10 variables:
## $ Country : Factor w/ 190 levels "Afghanistan",..: 1 2 3 4 5 6 7 8 9 10 ...
## $ LifeExp : int 42 71 71 82 41 73 75 69 82 80 ...
## $ InfantSurvival: num 0.835 0.985 0.967 0.997 0.846 0.99 0.986 0.979 0.995 0.996 ...
## $ Under5Survival: num 0.743 0.983 0.962 0.996 0.74 0.989 0.983 0.976 0.994 0.996 ...
## $ TBFree : num 0.998 1 0.999 1 0.997 ...
## $ PropMD : num 2.29e-04 1.14e-03 1.06e-03 3.30e-03 7.04e-05 ...
## $ PropRN : num 0.000572 0.004614 0.002091 0.0035 0.001146 ...
## $ PersExp : int 20 169 108 2589 36 503 484 88 3181 3788 ...
## $ GovtExp : int 92 3128 5184 169725 1620 12543 19170 1856 187616 189354 ...
## $ TotExp : int 112 3297 5292 172314 1656 13046 19654 1944 190797 193142 ...summary(who)## Country LifeExp InfantSurvival Under5Survival
## Afghanistan : 1 Min. :40.00 Min. :0.8350 Min. :0.7310
## Albania : 1 1st Qu.:61.25 1st Qu.:0.9433 1st Qu.:0.9253
## Algeria : 1 Median :70.00 Median :0.9785 Median :0.9745
## Andorra : 1 Mean :67.38 Mean :0.9624 Mean :0.9459
## Angola : 1 3rd Qu.:75.00 3rd Qu.:0.9910 3rd Qu.:0.9900
## Antigua and Barbuda: 1 Max. :83.00 Max. :0.9980 Max. :0.9970
## (Other) :184
## TBFree PropMD PropRN PersExp
## Min. :0.9870 Min. :0.0000196 Min. :0.0000883 Min. : 3.00
## 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455 1st Qu.: 36.25
## Median :0.9992 Median :0.0010474 Median :0.0027584 Median : 199.50
## Mean :0.9980 Mean :0.0017954 Mean :0.0041336 Mean : 742.00
## 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164 3rd Qu.: 515.25
## Max. :1.0000 Max. :0.0351290 Max. :0.0708387 Max. :6350.00
##
## GovtExp TotExp
## Min. : 10.0 Min. : 13
## 1st Qu.: 559.5 1st Qu.: 584
## Median : 5385.0 Median : 5541
## Mean : 40953.5 Mean : 41696
## 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :476420.0 Max. :482750
## Question 1
Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, R^2, standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
Answer:
Scatter plot of LifeExp~TotExp
plot(who$LifeExp ~ who$TotExp, main='Average Life Expectancy vs Total Expenditures.', ylab='LifeExp', xlab='TotExp')
Simple linear regression
life_exp_lm <- lm(LifeExp ~ TotExp, data = who)
summary(life_exp_lm)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14Using the qf() function to get the F critical value.
# Getting the F critical value
qf(0.95, 1, 188)## [1] 3.891398The linear model is \[LifeExp = 6.297e-05*TotExp + 64.75\]
Interpret the F statistic, p-value, standard error and R-squared value:
We see that the F value is 65.26 is much greater than the F critical value 3.89 and also the p-value 7.714e-14 is less than the 5% significant level. This indicates that the sample data (190 observations) provide enough evidence to reject the null hypothesis for the entire population and the model fits the data better than the intercept-only model. So, this suggests that there is a strong association between the changes in the sum of personal and government expenditures variable and the shifts in the average life expectancy variable.
For a good model, we typically would like to see a standard error that is at least five to ten times smaller than the corresponding coefficient. From the model summary, we can see that the standard error for the total expenditures is 8 times smaller than the corresponding coefficient (6.297e-05/7.795e-06=8) and the standard error for the intercept is 86 times smaller than the corresponding coefficient (6.475e+01/7.535e-01=86). This large ratio means that there is relatively little variability in the slope and intercept estimates, \({a}_{1}\) and \({a}_{0}\).
The reported R-squared of 0.2577 for this model means that the model explains 25.77 percent of the data’s variation. If we take the square root of the R-squared value, that will be the correlation coefficient, R (sqrt(0.2577)=0.508) of the model. This can be used to quantify the strength of the relationship between the variables. The larger the number of R the stronger the relationship between the variables. Generally, the R value between 0.5 and 0.7 is considered a moderate correlation.
Check whether all the 4 basic assumptions of linear regression is met:
par(mfrow=c(2,2))
# Linear regression plot with the best fit line
plot(who$TotExp, who$LifeExp, xlab = 'Expenditures',
ylab = 'Years',
main = 'Life Expectation vs Total Expenditures',
col = "red",
pch = 16)
abline(life_exp_lm)
# Residuals plot
plot(life_exp_lm$fitted.values, life_exp_lm$residuals,
xlab='Fitted Values', ylab='Residuals')
abline(h = 0, lty = 3, col="blue")
# Histogram plot
hist(life_exp_lm$residuals, xlab="Residuals")
qqnorm(life_exp_lm$residuals)
qqline(life_exp_lm$residuals)
Linearity: From the scatter plot, there does not seems to be linearity. Most data points seem to follow along some curve instead of falling along the best fit line.
Independence of Errors: From the residuals plot, we can see that the distribution of errors is not random and there is a pattern. Thus, we can say there is an autocorrelation effect among the residuals and the independence assumption is not valid.
Homoscedasticity: From the residuals plot, we can see that variation of residuals does not seem to be constant along the line drawn from 0 residual. In other words, the data points of residuals are not about the same distance from the line drawn from the residual mean equals to 0.
Normality of Error Distribution: The residuals histogram shows left-skewed and the qq plot also demonstrates that the data points form a curve instead of a straight line. Therefore, the residuals are not normally distributed.
To conclude, the model does not meet all the 4 basic assumptions of linear regression.
Question 2
Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”
Answer:
Simple linear regression with the transformed variables (LifeExp^4.6, TotExp^.06)
who_2 <- mutate(who, LifeExp=LifeExp^4.6, TotExp=TotExp^0.06)
life_exp_lm_2 <- lm(LifeExp ~ TotExp, data = who_2)
summary(life_exp_lm_2)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = who_2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16Using the qf() function to get the F critical value.
# Getting the F critical value
qf(0.95, 1, 188)## [1] 3.891398The linear model is \[LifeExp = 620060216*TotExp - 736527910\]
Interpret the F statistic, p-value, standard error and R-squared value:
Compare to the first model, we see that the F value 507.7 is much larger and the p-value 2.2e-16 is much smaller. So, transformation makes the association between the changes in the sum of personal and government expenditures variable and the shifts in the average life expectancy variable even stronger.
Also, we can see that the standard error for the total expenditures is 22.5 times smaller than the corresponding coefficient (620060216/27518940=22.5) and the standard error for the intercept is 15.7 times smaller than the corresponding coefficient (736527910/46817945=15.7). This large ratio means that there is relatively little variability in the slope and intercept estimates, \({a}_{1}\) and \({a}_{0}\).
The reported R-squared becomes 0.7298 for this new model. This means that this new model explains approximately 3 times more of the data’s variation. The correlation coefficient, R of this new model is 0.854 which is much higher than previous model. This indicates that the transformed variables has much stronger relationship compared to the original data. Generally, the R value greater than 0.7 is considered a strong correlation.
Furthermore, from the scatter plot with best fit line, residuals histogram, residuals plot, and qq plot below, we can see the new model meets all the 4 basic assumptions of linear regression.
To conclude, this new model with the transformed variables (LifeExp^4.6, TotExp^.06) is a better model.
par(mfrow=c(2,2))
# Linear regression plot with the best fit line
plot(who_2$TotExp, who_2$LifeExp, xlab = 'Expenditures',
ylab = 'Years',
main = 'Life Expectation vs Total Expenditures',
col = "red",
pch = 16)
abline(life_exp_lm_2)
# Residuals plot
plot(life_exp_lm_2$fitted.values, life_exp_lm_2$residuals,
xlab='Fitted Values', ylab='Residuals')
abline(h = 0, lty = 3, col="blue")
# Histogram plot
hist(life_exp_lm_2$residuals, xlab="Residuals")
qqnorm(life_exp_lm_2$residuals)
qqline(life_exp_lm_2$residuals)
Question 3
Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.
Answer:
From below calculations, the forecast life expectancy when TotExp^.06 =1.5 is 63.31 and the forecast life expectancy when TotExp^.06 =2.5 is 86.51.
life_exp_1_5 = 620060216*(1.5) - 736527910
life_exp_1_5^(1/4.6)## [1] 63.31153life_exp_2_5 = 620060216*(2.5) - 736527910
life_exp_2_5^(1/4.6)## [1] 86.50645Question 4
Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model?
LifeExp = b0+b1 x PropMd + b2 x TotExp +b3 x PropMD x TotExp
Answer:
Multiple regression model
life_exp_mrm <- lm(LifeExp ~ TotExp + PropMD + (TotExp*PropMD), data = who)
summary(life_exp_mrm)##
## Call:
## lm(formula = LifeExp ~ TotExp + PropMD + (TotExp * PropMD), data = who)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp:PropMD -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16Using the qf() function to get the F critical value.
# Getting the F critical value
qf(0.95, 3, (190*3-3))## [1] 2.620621The linear model is \[LifeExp = 7.233e-05*TotExp + 1.497e+03*PropMD - 6.026e-03*(TotExp*PropMD) + 6.277e+01\]
Interpret the F statistic, p-value, standard error and R-squared value:
We see that the F value is 34.49 is much greater than the F critical value 2.62 and also the p-value 2.2e-16 is less than the 5% significant level. This indicates that the sample data (570 observations) provide enough evidence to reject the null hypothesis for the entire population and the model fits the data better than the intercept-only model. So, this suggests that there is a strong association between the changes in the 3 variables (TotExp, PropMD, TotExp*PropMD) and the shifts in the average life expectancy variable.
For a good model, we typically would like to see a standard error that is at least five to ten times smaller than the corresponding coefficient. From the model summary, we can see that the standard error for all the variables including the intercept is at least 5 times smaller than the corresponding coefficients. This large ratio means that there is relatively little variability in the slope and intercept estimates.
The reported R-squared of 0.3574 for this model means that the model explains 35.74 percent of the data’s variation. If we take the square root of the R-squared value, that will be the correlation coefficient, R (sqrt(0.3574)=0.598) of the model. Generally, the R value between 0.5 and 0.7 is considered a moderate correlation.
Check whether all the 4 basic assumptions of linear regression is met:
par(mfrow=c(2,2))
# Residuals plot
plot(life_exp_mrm$fitted.values, life_exp_mrm$residuals,
xlab='Fitted Values', ylab='Residuals')
abline(h = 0, lty = 3, col="blue")
# Histogram plot
hist(life_exp_mrm$residuals, xlab="Residuals")
qqnorm(life_exp_mrm$residuals)
qqline(life_exp_mrm$residuals)
Independence of Errors: From the residuals plot, we can see that the distribution of errors is not random and there is a pattern. Thus, we can say there is an autocorrelation effect among the residuals and the independence assumption is not valid.
Homoscedasticity: From the residuals plot, we can see that variation of residuals does not seem to be constant along the line drawn from 0 residual. In other words, the data points of residuals are not about the same distance from the line drawn from the residual mean equals to 0.
Normality of Error Distribution: The residuals histogram shows left-skewed and the qq plot also demonstrates that the data points form a curve instead of a straight line. Therefore, the residuals are not normally distributed.
Linearity: Although we can’t plot a scatter plot with a best fit line to visualize whether linearity exist. But linearity required residuals of the regression to be normally distributed. We can see from residuals and qq plot that it is not normally distributed.
The model does not meet all the 4 basic assumptions of linear regression. Even though the F statistic, p-value, and R-squared show that this is a moderate model but it does not meet all the 4 basic assumptions of linear regression. Therefore, this is not an appropriate model.
Question 5
Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?
Answer:
This forecast does not seem realistic. With the little 14 (minimum value from statistic summary) total personal and government expenditures and the 0.03 (maximum value from statistic summary) high proportion of doctor in the population, the average life expectancy is 107.7 years. This life average life expectancy is not plausible today and also should be the greater the expenditures (e.g. health care) the longer the life expectancy. To conclude, the model is not an appropriate model.
life_exp_0314 <- 7.233e-05*(14) + 1.497e+03*(0.03) - 6.026e-03*(14*0.03) + 6.277e+01
life_exp_0314## [1] 107.6785