intro
the data set contains the data from 2008. The variables include follow.
Country: name of the country
LifeExp: average life expectancy for the country in years
InfantSurvival: proportion of those surviving to one year and more
Under5Survival: proportion of those surviving to five or more
TBFree: proportion of the population without TB
PropMD: proportion of the population who are MDs
PropRN: proportion of the population who are RNs
PersExp: mean personal ecpenditures on healthcare in US dollars at average exchange rate
GovtExp: mean government expenditures per capita on healthcare, US dollars at average exchange rate
TotExp: sum of personal and government expenditures
load data
df <- read.csv('who.csv')
head(df)## Country LifeExp InfantSurvival Under5Survival TBFree PropMD
## 1 Afghanistan 42 0.835 0.743 0.99769 0.000228841
## 2 Albania 71 0.985 0.983 0.99974 0.001143127
## 3 Algeria 71 0.967 0.962 0.99944 0.001060478
## 4 Andorra 82 0.997 0.996 0.99983 0.003297297
## 5 Angola 41 0.846 0.740 0.99656 0.000070400
## 6 Antigua and Barbuda 73 0.990 0.989 0.99991 0.000142857
## PropRN PersExp GovtExp TotExp
## 1 0.000572294 20 92 112
## 2 0.004614439 169 3128 3297
## 3 0.002091362 108 5184 5292
## 4 0.003500000 2589 169725 172314
## 5 0.001146162 36 1620 1656
## 6 0.002773810 503 12543 13046check data
- variables have its own value type
- no missing value
- other statistics look ok so far
# summary
summary(df)## Country LifeExp InfantSurvival Under5Survival
## Length:190 Min. :40.00 Min. :0.8350 Min. :0.7310
## Class :character 1st Qu.:61.25 1st Qu.:0.9433 1st Qu.:0.9253
## Mode :character Median :70.00 Median :0.9785 Median :0.9745
## Mean :67.38 Mean :0.9624 Mean :0.9459
## 3rd Qu.:75.00 3rd Qu.:0.9910 3rd Qu.:0.9900
## Max. :83.00 Max. :0.9980 Max. :0.9970
## TBFree PropMD PropRN PersExp
## Min. :0.9870 Min. :0.0000196 Min. :0.0000883 Min. : 3.00
## 1st Qu.:0.9969 1st Qu.:0.0002444 1st Qu.:0.0008455 1st Qu.: 36.25
## Median :0.9992 Median :0.0010474 Median :0.0027584 Median : 199.50
## Mean :0.9980 Mean :0.0017954 Mean :0.0041336 Mean : 742.00
## 3rd Qu.:0.9998 3rd Qu.:0.0024584 3rd Qu.:0.0057164 3rd Qu.: 515.25
## Max. :1.0000 Max. :0.0351290 Max. :0.0708387 Max. :6350.00
## GovtExp TotExp
## Min. : 10.0 Min. : 13
## 1st Qu.: 559.5 1st Qu.: 584
## Median : 5385.0 Median : 5541
## Mean : 40953.5 Mean : 41696
## 3rd Qu.: 25680.2 3rd Qu.: 26331
## Max. :476420.0 Max. :482750question 1
Provide a scatterplot of LifeExp~TotExp, and run simple linear regression. Do not transform the variables. Provide and interpret the F statistics, \(R^2\), standard error,and p-values only. Discuss whether the assumptions of simple linear regression met.
From scatter plot, it does have relationship, but it doesn’t seem to have linear relationship.
# scatter plot
plot(df$LifeExp, df$TotExp, main = 'life expectancy vs. total expenditures', xlab = 'life expectancy', ylab = 'total expenditures')
anyway, lets try to build a linear model to see if it works well.
# make a copy of df
dfc <- data.frame(df)# simple linear regression without transformation
dfc.lm <- lm(LifeExp ~ TotExp, data = dfc)- p-value is below threshold 0.05
- std.err is small indicates that observations are close to fitted line
- \(R^2\) means that this model explains 25% of data
- the p-value for f-statistic is below the threshold as well, which means that the data provides sufficient evidence that the model fits data better than the model with no independent variables since we only do the simple regression here.
# summary of model1
summary(dfc.lm)##
## Call:
## lm(formula = LifeExp ~ TotExp, data = dfc)
##
## Residuals:
## Min 1Q Median 3Q Max
## -24.764 -4.778 3.154 7.116 13.292
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.475e+01 7.535e-01 85.933 < 2e-16 ***
## TotExp 6.297e-05 7.795e-06 8.079 7.71e-14 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared: 0.2577, Adjusted R-squared: 0.2537
## F-statistic: 65.26 on 1 and 188 DF, p-value: 7.714e-14conclusion
from the interpretation without looking at the residuals. Except the \(R^2\), the rest of statistics are all showing that the linear model is a pretty good fit. However, if the model fits the data well, \(R^2\) should not be such low, which only explains 25% of data. which also means that the majority data cannot be explained by the model. Last but not least, in the very beginning of scatter plot shows that these two variale exist some kind of relationship, but not linear. Therefore, I would say that it does not meet the assumption of linear regression.
question 2
Raise life expectancy to the 4.6 power(i.e., \(LifeExp^{4.6}\)). Raise total expenditures to the 0.06 power (nearly a log transform, \(TotExp^{0.6}\)). Plot \(LifeExp^{4.6}\) as a function of \(TotExp^{0.6}\), and re-run the simple regression model using transformed variables. Provide and interpret the F statistics, \(R^2\), standard error, and p-values. Which model is “better?”
# add recaled columns
dfc.rescale <- cbind(dfc, LifeExp4.6 = dfc$LifeExp^4.6,TotExp0.06 = dfc$TotExp^0.06)compared to the previous plot, this one tends to be much more linear.
# scatter plot
plot(dfc.rescale$LifeExp4.6, dfc.rescale$TotExp0.06, main = 'rescaled life expantency vs. rescaled total expenditures', xlab = '(life expentancy)^4.6', ylab = '(total expenditure)^0.06')
# simple linear regression with transfromed variables
dfc.rescale.lm <- lm(LifeExp4.6 ~ TotExp0.06, data = dfc.rescale)from the summary:
- p-value of independent variable is below threshold
- standard error is 15021. It looks large individually, but it is pretty small relatively, compared to rescaled observations
- \(R^2\) in this model is 0.5705 which explains about 57% of data.
- p value of F statistic is even smaller than previous model.
# summary of model2
summary(dfc.rescale.lm)##
## Call:
## lm(formula = LifeExp4.6 ~ TotExp0.06, data = dfc.rescale)
##
## Residuals:
## Min 1Q Median 3Q Max
## -308616089 -53978977 13697187 59139231 211951764
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -736527910 46817945 -15.73 <2e-16 ***
## TotExp0.06 620060216 27518940 22.53 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared: 0.7298, Adjusted R-squared: 0.7283
## F-statistic: 507.7 on 1 and 188 DF, p-value: < 2.2e-16conclusion
compared to the previous model, under the circumstance of not looking into residual plot. the \(R^2\) has increased about 290%, F statistic and p-value for independent variable have decreased around two percentage points. As a result, I will say that this model is much better than the previous model.
question 3
using the results from 3, forecast life expentancy when \(TotExp^{0.06}\) = 1.5, then forecast life expentancy when \(TotExp^{0.06}\) = 2.5.
From the second model, we can conclude the linear function:
\[LifeExp^{4.6} = -736527910 + 620060216\times TotExp^{0.06}\]
# define function to calculate life expectancy
calculate_lifeExp <- function(a){
return (-736527910 + 620060216*a)
}# when TotExp^0.06 = 1.5, LifeExp^4.6 =
calculate_lifeExp(1.5)## [1] 193562414# when TotExp^0.06 = 2.5, LifeExp^4.6 =
calculate_lifeExp(2.5)## [1] 813622630question 4
build the following multiple regression model and interpret the F statistics, \(R^2\), standard error, and p-values. How good is the model?
\[LifeExp = b_0+b_1 \times PropMD + b_2 \times TotExp + b_3 \times PropMD \times TotExp\]
# multiple regression
dfc.multi.lm <- lm(LifeExp ~ PropMD + TotExp + PropMD * TotExp, data = dfc)from the summary of model3:
- all p-values of independent variables are below threshold, which indicates that these variables are significant for building the model.
- standard error are all small. the model fits well.
- p-value of F statistics is literally the same in model 2, but the F statistic is much smaller than model2. it indicates that the model fits data better than only use 2 variables.
- \(R^2\) is 0.3471 which explains around 35% of data
# summary model3
summary(dfc.multi.lm)##
## Call:
## lm(formula = LifeExp ~ PropMD + TotExp + PropMD * TotExp, data = dfc)
##
## Residuals:
## Min 1Q Median 3Q Max
## -27.320 -4.132 2.098 6.540 13.074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.277e+01 7.956e-01 78.899 < 2e-16 ***
## PropMD 1.497e+03 2.788e+02 5.371 2.32e-07 ***
## TotExp 7.233e-05 8.982e-06 8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03 1.472e-03 -4.093 6.35e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared: 0.3574, Adjusted R-squared: 0.3471
## F-statistic: 34.49 on 3 and 186 DF, p-value: < 2.2e-16conclusion
This model is not working so well I would say. It only explains 35% of data. I know that \(R^2\) is not the absolute factor to determine whether a model is good. However, I suspect that all the other statistics such as p-values all imply that this model significantly explains the data, how come \(R^2\) is so low? it should be some reasons, maybe these variables are significant, we should include some other important ones. anyhow, only speak to this model, it is not as good as I expected. But compared to model1, it seems a little better.
question 5
forecast LifeExp when PropMD = 0.3 and TotExp = 14. does this forecast seem realistic? why or why not?
PropMD = 0.3: this is close to the maximum value of the reported observations according to the summary of this variable.
TotExp = 14: this is literally the minimum of this independent variable from the summary
according to the scatter plot based on these two independent variables, the point(PropMD = 0.3, TotExp = 14) will be around the right bottom, which will be a extreme outlier. in other words that the probability of such event happen would be super low.
# summary of PropMD
summary(dfc$PropMD)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000196 0.0002444 0.0010474 0.0017954 0.0024584 0.0351290# summary of TotExp
summary(dfc$TotExp)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 13 584 5541 41696 26331 482750plot(dfc$PropMD, dfc$TotExp, main = 'PropMD vs TotExp', xlab = 'Proportion of the populatio who are MD', ylab = 'total expenditures')
I calculate the unrealistic forecast value anyway, like I mentioned before, the forecast value exceed more than 20 of the reported value, the probability of this even happen should close to zero.
# calculate the unrealistic forecast
as.double(sprintf('%.3f',6.277e+01)) + as.double(sprintf('%.3f',1.497e+03)) * 0.03 +
as.double(sprintf('%.3f',7.233e-05)) * 14 + as.double(sprintf('%.3f',-6.026e-03)) * 0.03 * 14## [1] 107.6775summary(dfc$LifeExp)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 40.00 61.25 70.00 67.38 75.00 83.00conclusion
unrealistic :(