CUNY DATA605 Wk12

setwd("C:/Users/ZacharyHerold/Desktop/DATA605/Wk12")
df <- data.frame(read.csv("who.csv"))
head(df)

##               Country LifeExp InfantSurvival Under5Survival  TBFree
## 1         Afghanistan      42          0.835          0.743 0.99769
## 2             Albania      71          0.985          0.983 0.99974
## 3             Algeria      71          0.967          0.962 0.99944
## 4             Andorra      82          0.997          0.996 0.99983
## 5              Angola      41          0.846          0.740 0.99656
## 6 Antigua and Barbuda      73          0.990          0.989 0.99991
##        PropMD      PropRN PersExp GovtExp TotExp
## 1 0.000228841 0.000572294      20      92    112
## 2 0.001143127 0.004614439     169    3128   3297
## 3 0.001060478 0.002091362     108    5184   5292
## 4 0.003297297 0.003500000    2589  169725 172314
## 5 0.000070400 0.001146162      36    1620   1656
## 6 0.000142857 0.002773810     503   12543  13046

1

Provide a scatterplot of LifeExp~TotExp, and run simple linear regression without transforming the variables.

mod1 <- lm(LifeExp~TotExp)
summary(mod1)

## 
## Call:
## lm(formula = LifeExp ~ TotExp)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -24.764  -4.778   3.154   7.116  13.292 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 6.475e+01  7.535e-01  85.933  < 2e-16 ***
## TotExp      6.297e-05  7.795e-06   8.079 7.71e-14 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 9.371 on 188 degrees of freedom
## Multiple R-squared:  0.2577, Adjusted R-squared:  0.2537 
## F-statistic: 65.26 on 1 and 188 DF,  p-value: 7.714e-14

Provide and interpret:

F statistics

The model with zero predictor variables is also called “Intercept Only Model”. F - Test for overall significance compares a intercept only regression model with the current model. In other words, it indicates whether your linear regression model provides a better fit to the data than a model that contains no independent variables.And then tries to comment on whether addition of these variables together is significant enough for them to be there or not.

The F-statistic of 65.26 on 1 and 188 DF, meaning the variance of the Intercept Only model is 65.26x greater than the variance for the univariate model.

R Squared

This is a measure which tells us how well our regression equation explains observed data values. R Squared = ( Explained Variation in Observed Values) / (Total variation in Observed Values)

Adjusted R-squared of 0.2577 means that 25% of the variance can be accounted for by the model

Standard Error

The standard error of the regression provides the absolute measure of the typical distance that the data points fall from the regression line. S is in the units of the dependent variable. Lower values of S because it signifies that the distances between the data points and the fitted values are smaller. The standard error for TotExp is very close to 0 in this model.

p-values

A p-value is the probability of obtaining an effect at least as extreme as the one in your sample data, assuming the truth of the null hypothesis. P-value that are greater than the level of significance (typically 0.05) invalidate the use of that variate in the regression model.

In the examle, the p-value of TotExp is nearly 0, rendering it highly significant.

Discuss whether the assumptions of simple linear regression met.

plot(fitted(mod1),resid(mod1))

For higher fitted values, the residuals show a decreasing trend, providing evidence against the normality assumption, in which the residuals would be randomly distributed about 0.

qqnorm(resid(mod1))
qqline(resid(mod1))

The Q-Q plot also provides evidence against normality, given the strong deviation from the QQ line.

2.

Raise life expectancy to the 4.6 power (i.e., LifeExp^4.6). Raise total expenditures to the 0.06 power (nearly a log transform, TotExp^.06). Plot LifeExp^4.6 as a function of TotExp^.06, and r re-run the simple regression model using the transformed variables. Provide and interpret the F statistics, R^2, standard error, and p-values. Which model is “better?”

LifeExp2 <- LifeExp^4.6
TotExp2 <- TotExp^.06
plot(LifeExp2,TotExp2)

mod2 <- lm(I(LifeExp2)~I(TotExp2))
summary(mod2)

## 
## Call:
## lm(formula = I(LifeExp2) ~ I(TotExp2))
## 
## Residuals:
##        Min         1Q     Median         3Q        Max 
## -308616089  -53978977   13697187   59139231  211951764 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -736527910   46817945  -15.73   <2e-16 ***
## I(TotExp2)   620060216   27518940   22.53   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 90490000 on 188 degrees of freedom
## Multiple R-squared:  0.7298, Adjusted R-squared:  0.7283 
## F-statistic: 507.7 on 1 and 188 DF,  p-value: < 2.2e-16

Provide and interpret: F-test improves to 507.7 on 1 and 188 DF, showing the greater ratio in variance from the intercept only model.

R^2 improves sharply to 0.7298, showing much more variation can be explained by the model.
standard error, the error is very high due to the exponent in the model, as such we consider the t value which rescales it as a ratio.

p-value only. The p-value is highly statistically significant in this model.

###3.
Using the results from 3, forecast life expectancy when TotExp^.06 =1.5. Then forecast life expectancy when TotExp^.06=2.5.

The forecasts are given here:

l1 <- coef(mod2)[1] + coef(mod2)[2] * 1.5
l2 <- coef(mod2)[1] + coef(mod2)[2] * 2.5
(c(l1^(1/4.6),l2^(1/4.6)))

## (Intercept) (Intercept) 
##    63.31153    86.50645

4.

Build the following multiple regression model and interpret the F Statistics, R^2, standard error, and p-values. How good is the model?

\(LifeExp = b_0+b_1 * PropMd + b_2 * TotExp + b_3 * PropMD * TotExp\)

mod3 <- lm(LifeExp ~ PropMD*TotExp)
summary(mod3)

## 
## Call:
## lm(formula = LifeExp ~ PropMD * TotExp)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -27.320  -4.132   2.098   6.540  13.074 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    6.277e+01  7.956e-01  78.899  < 2e-16 ***
## PropMD         1.497e+03  2.788e+02   5.371 2.32e-07 ***
## TotExp         7.233e-05  8.982e-06   8.053 9.39e-14 ***
## PropMD:TotExp -6.026e-03  1.472e-03  -4.093 6.35e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 8.765 on 186 degrees of freedom
## Multiple R-squared:  0.3574, Adjusted R-squared:  0.3471 
## F-statistic: 34.49 on 3 and 186 DF,  p-value: < 2.2e-16

interpret the : F Statistics, the ratio of variation worsens when compared to model2 R^2, also using the Adjusted R-squared, we see model3 loses predictive force when compared to model2 standard error, the standard errors are low, except in the acase of PropMD p-values, these are all highly significant.

5.

Forecast LifeExp when PropMD=.03 and TotExp = 14. Does this forecast seem realistic? Why or why not?

l3 <- coef(mod3)[1] + coef(mod3)[2] * .03 + coef(mod3)[3] * 14 + coef(mod3)[4] * .03 * 14
l3

## (Intercept) 
##     107.696

The predicted life expectancy is 107.7 years, well beyond the typical lifespan. This is due to the high coefficient for PropMD (1497) and the fact that the example value is well beyond the mean (expected) value for PropMD of 0.0018. This term is clrealy problematic as reflected in the higher Std. error for this covariant.

mean(df$PropMD)

## [1] 0.00179538