Import necessary packages

library(psych)
library(MASS)
library(ResourceSelection)
## ResourceSelection 0.3-6   2023-06-27
library(car)
## Loading required package: carData
## 
## Attaching package: 'car'
## The following object is masked from 'package:psych':
## 
##     logit
data <- swiss

# Build best multiple regression model for Fertility Variable

# TESTING : Show normality of the dataset (for testing purposes)
kdepairs(data)
## Warning in par(usr): argument 1 does not name a graphical parameter

## Warning in par(usr): argument 1 does not name a graphical parameter

## Warning in par(usr): argument 1 does not name a graphical parameter

## Warning in par(usr): argument 1 does not name a graphical parameter

## Warning in par(usr): argument 1 does not name a graphical parameter

## Warning in par(usr): argument 1 does not name a graphical parameter

The distribution of the Fertility, and Infant Mortality appear roughly normal, the variables agriculture, examination are slightly skewed, and Education and Catholic are completely nonnormal.

# Use with unaltered DF
model <- lm(Fertility~., data = swiss)
summary(model)
## 
## Call:
## lm(formula = Fertility ~ ., data = swiss)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -15.2743  -5.2617   0.5032   4.1198  15.3213 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      66.91518   10.70604   6.250 1.91e-07 ***
## Agriculture      -0.17211    0.07030  -2.448  0.01873 *  
## Examination      -0.25801    0.25388  -1.016  0.31546    
## Education        -0.87094    0.18303  -4.758 2.43e-05 ***
## Catholic          0.10412    0.03526   2.953  0.00519 ** 
## Infant.Mortality  1.07705    0.38172   2.822  0.00734 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.165 on 41 degrees of freedom
## Multiple R-squared:  0.7067, Adjusted R-squared:  0.671 
## F-statistic: 19.76 on 5 and 41 DF,  p-value: 5.594e-10

The R^2 with the complete set of variables input into the model is .71 with an adjusted R^2 of 0.67, both moderate linear correlation to the Fertility variable. However, the Examination variable is not statistically significant, so it will be removed from the model and rerun fitting the data.

# Use with unaltered DF
model <- lm(Fertility~.-Examination, data = swiss)
summary(model)
## 
## Call:
## lm(formula = Fertility ~ . - Examination, data = swiss)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -14.6765  -6.0522   0.7514   3.1664  16.1422 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      62.10131    9.60489   6.466 8.49e-08 ***
## Agriculture      -0.15462    0.06819  -2.267  0.02857 *  
## Education        -0.98026    0.14814  -6.617 5.14e-08 ***
## Catholic          0.12467    0.02889   4.315 9.50e-05 ***
## Infant.Mortality  1.07844    0.38187   2.824  0.00722 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.168 on 42 degrees of freedom
## Multiple R-squared:  0.6993, Adjusted R-squared:  0.6707 
## F-statistic: 24.42 on 4 and 42 DF,  p-value: 1.717e-10

With an R^2 value of 0.70, the model performs slightly worse than the original with an equally performing Adjusted R^2 relative to the first model.

Try to create a better predictive model using the powerTransform function.

# Examination is not statistically significant to the model
# BASE R^2 - 0.71

# Use powerTransform to find optimal exponents for all variables, then create regression
exps <- powerTransform(as.matrix(data)~1)
testTransform(exps, exps$lambda)
##                                                  LRT df pval
## LR test, lambda = (1.5 1.05 0.63 0.33 0.13 1.84)   0  6    1
for(i in 1:ncol(data)){
  data[,i] <- data[,i] ^ exps$lambda[i]
}
improvedModel <- lm(Fertility~., data = data)
summary(improvedModel)
## 
## Call:
## lm(formula = Fertility ~ ., data = data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -196.944  -56.206   -5.782   74.089  177.714 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       766.5946   169.7202   4.517 5.23e-05 ***
## Agriculture        -1.3227     0.7729  -1.711  0.09459 .  
## Examination       -30.7657    15.1102  -2.036  0.04824 *  
## Education        -129.0042    48.0501  -2.685  0.01043 *  
## Catholic          113.4463    73.0863   1.552  0.12829    
## Infant.Mortality    0.7608     0.2439   3.120  0.00331 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 100.2 on 41 degrees of freedom
## Multiple R-squared:  0.6235, Adjusted R-squared:  0.5775 
## F-statistic: 13.58 on 5 and 41 DF,  p-value: 7.886e-08

Transformation Summary:

The powerTransform did more harm than good to the R^2 and adjusted R^2 values and should not be used in this case.

SUMMARY :

Overall, the original model performed better than the powerTransformed R^2 and the adjusted R^2. That was the best model I could render for this problem with a final R^2 of roughly 0.71, suggesting a moderate-to-strong linear correlation between Fertility and the rest of the variables.

Build best LOGISTIC regression model predicting Fertility > 70.0

# Save dataset
data <- swiss
# Create logistic regression model using all variables for best results

Quick note:

The Akaike Information Criterion (AIC) is an metric of error and should be minimized.

Show summary of the model

# Original AIC : 326.07
summary(model)
## 
## Call:
## lm(formula = Fertility ~ . - Examination, data = swiss)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -14.6765  -6.0522   0.7514   3.1664  16.1422 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      62.10131    9.60489   6.466 8.49e-08 ***
## Agriculture      -0.15462    0.06819  -2.267  0.02857 *  
## Education        -0.98026    0.14814  -6.617 5.14e-08 ***
## Catholic          0.12467    0.02889   4.315 9.50e-05 ***
## Infant.Mortality  1.07844    0.38187   2.824  0.00722 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 7.168 on 42 degrees of freedom
## Multiple R-squared:  0.6993, Adjusted R-squared:  0.6707 
## F-statistic: 24.42 on 4 and 42 DF,  p-value: 1.717e-10

Build initial model containing all variables

data <- swiss

model <- glm(Fertility~., data=data)
pred = predict.glm(model, data, type='response')

pred[pred<70] = 0
pred[pred>=70] = 1

data$Fertility[data$Fertility < 70] = 0
data$Fertility[data$Fertility >= 70] = 1

summary(model)
## 
## Call:
## glm(formula = Fertility ~ ., data = data)
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      66.91518   10.70604   6.250 1.91e-07 ***
## Agriculture      -0.17211    0.07030  -2.448  0.01873 *  
## Examination      -0.25801    0.25388  -1.016  0.31546    
## Education        -0.87094    0.18303  -4.758 2.43e-05 ***
## Catholic          0.10412    0.03526   2.953  0.00519 ** 
## Infant.Mortality  1.07705    0.38172   2.822  0.00734 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for gaussian family taken to be 51.34251)
## 
##     Null deviance: 7178  on 46  degrees of freedom
## Residual deviance: 2105  on 41  degrees of freedom
## AIC: 326.07
## 
## Number of Fisher Scoring iterations: 2

Because Examination is not statistically significant to the model, it shall be removed and the model remade

Set Results over 70 to 1, under to 0 for both initial data set and results

data <-swiss

model <- glm(Fertility~.-Examination, data=data)
pred = predict.glm(model, data, type='response')

pred[pred<70] = 0
pred[pred>=70] = 1

data$Fertility[data$Fertility < 70] = 0
data$Fertility[data$Fertility >= 70] = 1

summary(model)
## 
## Call:
## glm(formula = Fertility ~ . - Examination, data = data)
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      62.10131    9.60489   6.466 8.49e-08 ***
## Agriculture      -0.15462    0.06819  -2.267  0.02857 *  
## Education        -0.98026    0.14814  -6.617 5.14e-08 ***
## Catholic          0.12467    0.02889   4.315 9.50e-05 ***
## Infant.Mortality  1.07844    0.38187   2.824  0.00722 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for gaussian family taken to be 51.38261)
## 
##     Null deviance: 7178.0  on 46  degrees of freedom
## Residual deviance: 2158.1  on 42  degrees of freedom
## AIC: 325.24
## 
## Number of Fisher Scoring iterations: 2

All of the variables are statistically significant in the model with a better AIC than the initial model, thus this model shall be kept

Check normality of the model

plot(model)

For the most part, all 4 graphs concerning normality are acceptable, though the QQ plot shows some skewedness and the Scale-Location plot does have an outlier around predicted value 35.

Show the confusion table of the model for the given dataset

table(pred, data$Fertility)
##     
## pred  0  1
##    0 16  4
##    1  7 20

The model predicts 36/47 (77%) Fertility rates to be below/above 70 correctly with a false positive rate of 25% and a false negative rate of 20%