1. Multivariate Regression
data(swiss)
summary(swiss)## Fertility Agriculture Examination Education
## Min. :35.00 Min. : 1.20 Min. : 3.00 Min. : 1.00
## 1st Qu.:64.70 1st Qu.:35.90 1st Qu.:12.00 1st Qu.: 6.00
## Median :70.40 Median :54.10 Median :16.00 Median : 8.00
## Mean :70.14 Mean :50.66 Mean :16.49 Mean :10.98
## 3rd Qu.:78.45 3rd Qu.:67.65 3rd Qu.:22.00 3rd Qu.:12.00
## Max. :92.50 Max. :89.70 Max. :37.00 Max. :53.00
## Catholic Infant.Mortality
## Min. : 2.150 Min. :10.80
## 1st Qu.: 5.195 1st Qu.:18.15
## Median : 15.140 Median :20.00
## Mean : 41.144 Mean :19.94
## 3rd Qu.: 93.125 3rd Qu.:21.70
## Max. :100.000 Max. :26.60?swiss## starting httpd help server ... done#model 1 - all variables
model <- lm(Fertility~Agriculture+Examination+Education+Catholic+Infant.Mortality,data=swiss)
summary(model)##
## Call:
## lm(formula = Fertility ~ Agriculture + Examination + Education +
## Catholic + Infant.Mortality, data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -15.2743 -5.2617 0.5032 4.1198 15.3213
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 66.91518 10.70604 6.250 1.91e-07 ***
## Agriculture -0.17211 0.07030 -2.448 0.01873 *
## Examination -0.25801 0.25388 -1.016 0.31546
## Education -0.87094 0.18303 -4.758 2.43e-05 ***
## Catholic 0.10412 0.03526 2.953 0.00519 **
## Infant.Mortality 1.07705 0.38172 2.822 0.00734 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.165 on 41 degrees of freedom
## Multiple R-squared: 0.7067, Adjusted R-squared: 0.671
## F-statistic: 19.76 on 5 and 41 DF, p-value: 5.594e-10# model 2 - model without Examination and Agriculture
model_reduced <- lm(Fertility~Education+Catholic+Infant.Mortality,data=swiss)
summary(model_reduced)##
## Call:
## lm(formula = Fertility ~ Education + Catholic + Infant.Mortality,
## data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.4781 -5.4403 -0.5143 4.1568 15.1187
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 48.67707 7.91908 6.147 2.24e-07 ***
## Education -0.75925 0.11680 -6.501 6.83e-08 ***
## Catholic 0.09607 0.02722 3.530 0.00101 **
## Infant.Mortality 1.29615 0.38699 3.349 0.00169 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.505 on 43 degrees of freedom
## Multiple R-squared: 0.6625, Adjusted R-squared: 0.639
## F-statistic: 28.14 on 3 and 43 DF, p-value: 3.15e-10# model 3 - model without Examination
model_best <- lm(Fertility~Agriculture+Education+Catholic+Infant.Mortality,data=swiss)
summary(model_best)##
## Call:
## lm(formula = Fertility ~ Agriculture + Education + Catholic +
## Infant.Mortality, data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.6765 -6.0522 0.7514 3.1664 16.1422
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 62.10131 9.60489 6.466 8.49e-08 ***
## Agriculture -0.15462 0.06819 -2.267 0.02857 *
## Education -0.98026 0.14814 -6.617 5.14e-08 ***
## Catholic 0.12467 0.02889 4.315 9.50e-05 ***
## Infant.Mortality 1.07844 0.38187 2.824 0.00722 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.168 on 42 degrees of freedom
## Multiple R-squared: 0.6993, Adjusted R-squared: 0.6707
## F-statistic: 24.42 on 4 and 42 DF, p-value: 1.717e-10To find the best multiple regression model for Fertility, I began with a model that has all five independent variables. The full model’s R squared was 70.67%, meaning that the independent variables explained approximately 70.67% of the variation in Fertility.The adjusted R-squared was 67.10%. I noticed that Examination was not statistically significant (p=0.3155), while Agriculture had the highest p-value among the statistically significant variables (p = 0.0187).
I decided to try a reduced model, without Examination and Agriculture. The model consisted of Education, Catholic, and Infant.Mortality. However, it’s adjusted R squared decreased to 63.9%. Generally, we want the adjusted R squared to be high as possible (close to 100%, or 1 for probability) because it indicates how well the model explains the variation in Fertility while accounting for the number of independent variables. .Finally, I built a new model containing Agriculture, Education, Catholic, and Infant.Mortality.
The final model achieved an R squared of 0.6993, and adjusted R squared of 0.6707. All four independent variables were statistically significant at 5% significance level. By examining the estimates, Agriculture and Education had negative relationships with Fertility, while Catholic and Infant.Mortality had positive relationships.
I selected the four-variable model because it gives nearly the same adjusted R squared as the full model, while excluding the statistically insignificant Examination variable. The overall regression was also statistically significant (F=24.42, p= 1.717e-10). Therefore, I chose the four-variable model because it is simpler, all its variables are statistically significant, and it still explains about 70% of the variation in Fertility.
2. Logistic Regression
data(swiss)
# new variable for Fertility
#binary var:
# 1 = Fertility > 70.
# 0 = Fertility <= 70
swiss$Fertility70 <- as.numeric(swiss$Fertility> 70)
#check distribution
table(swiss$Fertility70)##
## 0 1
## 23 24# try full log model - all 5 independent variables
log_full <- glm(
Fertility70 ~ Agriculture + Examination +
Education + Catholic + Infant.Mortality,
data = swiss,
family = binomial(link = "logit")
)
summary(log_full)##
## Call:
## glm(formula = Fertility70 ~ Agriculture + Examination + Education +
## Catholic + Infant.Mortality, family = binomial(link = "logit"),
## data = swiss)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 4.82826 5.25607 0.919 0.3583
## Agriculture -0.09615 0.04011 -2.397 0.0165 *
## Examination -0.32116 0.13844 -2.320 0.0203 *
## Education -0.12078 0.08610 -1.403 0.1607
## Catholic 0.02078 0.01376 1.509 0.1312
## Infant.Mortality 0.29078 0.21051 1.381 0.1672
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 65.135 on 46 degrees of freedom
## Residual deviance: 32.887 on 41 degrees of freedom
## AIC: 44.887
##
## Number of Fisher Scoring iterations: 6# build the logistic regression model using the 4 variables from multiple regression
prev_best_log_model <- glm(Fertility70~ Agriculture + Education + Catholic + Infant.Mortality,data=swiss,family = binomial(link = "logit"))
summary(prev_best_log_model)##
## Call:
## glm(formula = Fertility70 ~ Agriculture + Education + Catholic +
## Infant.Mortality, family = binomial(link = "logit"), data = swiss)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.94524 4.13457 -0.229 0.8192
## Agriculture -0.04973 0.02675 -1.859 0.0631 .
## Education -0.21212 0.08825 -2.404 0.0162 *
## Catholic 0.03316 0.01187 2.795 0.0052 **
## Infant.Mortality 0.22203 0.17261 1.286 0.1983
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 65.135 on 46 degrees of freedom
## Residual deviance: 40.634 on 42 degrees of freedom
## AIC: 50.634
##
## Number of Fisher Scoring iterations: 5# new reduced model - only Education and Catholic variables
log_reduced <- glm(Fertility70~ Education + Catholic ,data=swiss,family = binomial(link = "logit"))
summary(log_reduced)##
## Call:
## glm(formula = Fertility70 ~ Education + Catholic, family = binomial(link = "logit"),
## data = swiss)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 0.377979 0.718516 0.526 0.5988
## Education -0.118823 0.059196 -2.007 0.0447 *
## Catholic 0.023428 0.009328 2.511 0.0120 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 65.135 on 46 degrees of freedom
## Residual deviance: 48.262 on 44 degrees of freedom
## AIC: 54.262
##
## Number of Fisher Scoring iterations: 5# finding best AIC using backward selection
log_best <- step(log_full, direction = "backward")## Start: AIC=44.89
## Fertility70 ~ Agriculture + Examination + Education + Catholic +
## Infant.Mortality
##
## Df Deviance AIC
## <none> 32.887 44.887
## - Infant.Mortality 1 35.149 45.149
## - Education 1 35.275 45.275
## - Catholic 1 35.315 45.315
## - Examination 1 40.634 50.634
## - Agriculture 1 41.504 51.504summary(log_best)##
## Call:
## glm(formula = Fertility70 ~ Agriculture + Examination + Education +
## Catholic + Infant.Mortality, family = binomial(link = "logit"),
## data = swiss)
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 4.82826 5.25607 0.919 0.3583
## Agriculture -0.09615 0.04011 -2.397 0.0165 *
## Examination -0.32116 0.13844 -2.320 0.0203 *
## Education -0.12078 0.08610 -1.403 0.1607
## Catholic 0.02078 0.01376 1.509 0.1312
## Infant.Mortality 0.29078 0.21051 1.381 0.1672
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 65.135 on 46 degrees of freedom
## Residual deviance: 32.887 on 41 degrees of freedom
## AIC: 44.887
##
## Number of Fisher Scoring iterations: 6# compare AIC
AIC(log_full, log_best)## df AIC
## log_full 6 44.88671
## log_best 6 44.88671To build a logistic regression model for predicting Fertility>70.0, I began by creating a binary variable where 1 represents Fertility greater than 70, and 0 represents Fertility less than 70.The dataset contained 24 observations with Fertility greater than 70 and 23 observations with Fertility less than or equal to 70.
I started with a full logistic regression model containing all five independent variables. Agriculture and Examination were statistically significant at the 5% level, with p-values of 0.0165 and 0.0203. Both variables had negative estimates, showing that higher values were associated with lower odds of Fertility being greater than 70. The other three variables were not statistically significant.
Next, I tested a four-variable model using Agriculture, Education, Catholic, and Infant.Mortality, which were the variables selected in my previous multiple linear regression. In this model, Education and Catholic were statistically significant, while Agriculture and Infant.Mortality were not.
Then I tested a reduced model containing only Education and Catholic. Both variables were statistically significant at the 5% significance level.
I compared the models’ AIC levels, which evaluates model fit while accounting for number of parameters. Generally, a lower AIC shows a better balance between model fit and complexity.The full model had the lowest AIC (44.887), followed by the four-variable model (50.634) and the two-variable model (54.262).
Finally, I used backward selection with the step() function to see if removing variables could improve the model’s AIC and provide a better balance between model fit and simplicity.
I selected the full model (5 variables) because it had the lowest AIC among the three models I tested.Although only two of its five independent variables were statistically significant, the full model provided the best balance between model fit and complexity based on AIC.