Import necessary packages
library(psych)
library(MASS)
library(ResourceSelection)
## ResourceSelection 0.3-6 2023-06-27
library(car)
## Loading required package: carData
##
## Attaching package: 'car'
## The following object is masked from 'package:psych':
##
## logit
data <- swiss
# Build best multiple regression model for Fertility Variable
# TESTING : Show normality of the dataset (for testing purposes)
kdepairs(data)
## Warning in par(usr): argument 1 does not name a graphical parameter
## Warning in par(usr): argument 1 does not name a graphical parameter
## Warning in par(usr): argument 1 does not name a graphical parameter
## Warning in par(usr): argument 1 does not name a graphical parameter
## Warning in par(usr): argument 1 does not name a graphical parameter
## Warning in par(usr): argument 1 does not name a graphical parameter
The distribution of the Fertility, and Infant Mortality appear roughly normal, the variables agriculture, examination are slightly skewed, and Education and Catholic are completely nonnormal.
# Use with unaltered DF
model <- lm(Fertility~., data = swiss)
summary(model)
##
## Call:
## lm(formula = Fertility ~ ., data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -15.2743 -5.2617 0.5032 4.1198 15.3213
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 66.91518 10.70604 6.250 1.91e-07 ***
## Agriculture -0.17211 0.07030 -2.448 0.01873 *
## Examination -0.25801 0.25388 -1.016 0.31546
## Education -0.87094 0.18303 -4.758 2.43e-05 ***
## Catholic 0.10412 0.03526 2.953 0.00519 **
## Infant.Mortality 1.07705 0.38172 2.822 0.00734 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.165 on 41 degrees of freedom
## Multiple R-squared: 0.7067, Adjusted R-squared: 0.671
## F-statistic: 19.76 on 5 and 41 DF, p-value: 5.594e-10
The R^2 with the complete set of variables input into the model is .71 with an adjusted R^2 of 0.67, both moderate linear correlation to the Fertility variable. However, the Examination variable is not statistically significant, so it will be removed from the model and rerun fitting the data.
# Use with unaltered DF
model <- lm(Fertility~.-Examination, data = swiss)
summary(model)
##
## Call:
## lm(formula = Fertility ~ . - Examination, data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.6765 -6.0522 0.7514 3.1664 16.1422
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 62.10131 9.60489 6.466 8.49e-08 ***
## Agriculture -0.15462 0.06819 -2.267 0.02857 *
## Education -0.98026 0.14814 -6.617 5.14e-08 ***
## Catholic 0.12467 0.02889 4.315 9.50e-05 ***
## Infant.Mortality 1.07844 0.38187 2.824 0.00722 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.168 on 42 degrees of freedom
## Multiple R-squared: 0.6993, Adjusted R-squared: 0.6707
## F-statistic: 24.42 on 4 and 42 DF, p-value: 1.717e-10
With an R^2 value of 0.70, the model performs slightly worse than the original with an equally performing Adjusted R^2 relative to the first model.
# Examination is not statistically significant to the model
# BASE R^2 - 0.71
# Use powerTransform to find optimal exponents for all variables, then create regression
exps <- powerTransform(as.matrix(data)~1)
testTransform(exps, exps$lambda)
## LRT df pval
## LR test, lambda = (1.5 1.05 0.63 0.33 0.13 1.84) 0 6 1
for(i in 1:ncol(data)){
data[,i] <- data[,i] ^ exps$lambda[i]
}
improvedModel <- lm(Fertility~., data = data)
summary(improvedModel)
##
## Call:
## lm(formula = Fertility ~ ., data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -196.944 -56.206 -5.782 74.089 177.714
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 766.5946 169.7202 4.517 5.23e-05 ***
## Agriculture -1.3227 0.7729 -1.711 0.09459 .
## Examination -30.7657 15.1102 -2.036 0.04824 *
## Education -129.0042 48.0501 -2.685 0.01043 *
## Catholic 113.4463 73.0863 1.552 0.12829
## Infant.Mortality 0.7608 0.2439 3.120 0.00331 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 100.2 on 41 degrees of freedom
## Multiple R-squared: 0.6235, Adjusted R-squared: 0.5775
## F-statistic: 13.58 on 5 and 41 DF, p-value: 7.886e-08
The powerTransform did more harm than good to the R^2 and adjusted R^2 values and should not be used in this case.
Overall, the original model performed better than the powerTransformed R^2 and the adjusted R^2. That was the best model I could render for this problem with a final R^2 of roughly 0.71, suggesting a moderate-to-strong linear correlation between Fertility and the rest of the variables.
# Save dataset
data <- swiss
# Create logistic regression model using all variables for best results
The Akaike Information Criterion (AIC) is an metric of error and should be minimized.
# Original AIC : 326.07
summary(model)
##
## Call:
## lm(formula = Fertility ~ . - Examination, data = swiss)
##
## Residuals:
## Min 1Q Median 3Q Max
## -14.6765 -6.0522 0.7514 3.1664 16.1422
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 62.10131 9.60489 6.466 8.49e-08 ***
## Agriculture -0.15462 0.06819 -2.267 0.02857 *
## Education -0.98026 0.14814 -6.617 5.14e-08 ***
## Catholic 0.12467 0.02889 4.315 9.50e-05 ***
## Infant.Mortality 1.07844 0.38187 2.824 0.00722 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 7.168 on 42 degrees of freedom
## Multiple R-squared: 0.6993, Adjusted R-squared: 0.6707
## F-statistic: 24.42 on 4 and 42 DF, p-value: 1.717e-10
data <- swiss
model <- glm(Fertility~., data=data)
pred = predict.glm(model, data, type='response')
pred[pred<70] = 0
pred[pred>=70] = 1
data$Fertility[data$Fertility < 70] = 0
data$Fertility[data$Fertility >= 70] = 1
summary(model)
##
## Call:
## glm(formula = Fertility ~ ., data = data)
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 66.91518 10.70604 6.250 1.91e-07 ***
## Agriculture -0.17211 0.07030 -2.448 0.01873 *
## Examination -0.25801 0.25388 -1.016 0.31546
## Education -0.87094 0.18303 -4.758 2.43e-05 ***
## Catholic 0.10412 0.03526 2.953 0.00519 **
## Infant.Mortality 1.07705 0.38172 2.822 0.00734 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for gaussian family taken to be 51.34251)
##
## Null deviance: 7178 on 46 degrees of freedom
## Residual deviance: 2105 on 41 degrees of freedom
## AIC: 326.07
##
## Number of Fisher Scoring iterations: 2
Because Examination is not statistically significant to the model, it shall be removed and the model remade
data <-swiss
model <- glm(Fertility~.-Examination, data=data)
pred = predict.glm(model, data, type='response')
pred[pred<70] = 0
pred[pred>=70] = 1
data$Fertility[data$Fertility < 70] = 0
data$Fertility[data$Fertility >= 70] = 1
summary(model)
##
## Call:
## glm(formula = Fertility ~ . - Examination, data = data)
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 62.10131 9.60489 6.466 8.49e-08 ***
## Agriculture -0.15462 0.06819 -2.267 0.02857 *
## Education -0.98026 0.14814 -6.617 5.14e-08 ***
## Catholic 0.12467 0.02889 4.315 9.50e-05 ***
## Infant.Mortality 1.07844 0.38187 2.824 0.00722 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for gaussian family taken to be 51.38261)
##
## Null deviance: 7178.0 on 46 degrees of freedom
## Residual deviance: 2158.1 on 42 degrees of freedom
## AIC: 325.24
##
## Number of Fisher Scoring iterations: 2
All of the variables are statistically significant in the model with a better AIC than the initial model, thus this model shall be kept
plot(model)
For the most part, all 4 graphs concerning normality are acceptable,
though the QQ plot shows some skewedness and the Scale-Location plot
does have an outlier around predicted value 35.
table(pred, data$Fertility)
##
## pred 0 1
## 0 16 4
## 1 7 20
The model predicts 36/47 (77%) Fertility rates to be below/above 70 correctly with a false positive rate of 25% and a false negative rate of 20%