library(dplyr)
library(ggplot2)
library(forecast)
library(visualize)

Simple Linear Regression

1

library(carData)
data(Salaries)

2

glimpse(Salaries)
## Rows: 397
## Columns: 6
## $ rank          <fct> Prof, Prof, AsstProf, Prof, Prof, AssocProf, Prof, Prof,…
## $ discipline    <fct> B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, A, A,…
## $ yrs.since.phd <int> 19, 20, 4, 45, 40, 6, 30, 45, 21, 18, 12, 7, 1, 2, 20, 1…
## $ yrs.service   <int> 18, 16, 3, 39, 41, 6, 23, 45, 20, 18, 8, 2, 1, 0, 18, 3,…
## $ sex           <fct> Male, Male, Male, Male, Male, Male, Male, Male, Male, Fe…
## $ salary        <int> 139750, 173200, 79750, 115000, 141500, 97000, 175000, 14…

numeric:yrs.since.phd,yrs.service and salary

factor:rank,discipline and sex

3

set.seed(250)
index<-sample(1:nrow(Salaries),0.6*nrow(Salaries))
train<-Salaries[index,]
test<-Salaries[-index,]
rownames(train)<-c(1:nrow(train))

a.Dividing the data into a training set and a validation set helps evaluate model performance, prevent overfitting, and adjust parameters. This partitioning method ensures that the model can generalize to new data during training and improves the model’s reliability and accuracy.

4

ggplot(train,aes(x=yrs.since.phd,y=salary))+geom_point()+geom_smooth()+ggtitle("The relationship between years since Ph.D. and salary")
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'

As the years since Ph.D. increase, salary has a tendency to increase first and then decrease. This makes intuitive sense to me, because this picture shows that there is no obvious linear relationship between years since Ph.D. and salary, but a quadratic relationship.

5

cor.test(train$yrs.since.phd,train$salary)
## 
##  Pearson's product-moment correlation
## 
## data:  train$yrs.since.phd and train$salary
## t = 6.6408, df = 236, p-value = 2.127e-10
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.2839580 0.4987868
## sample estimates:
##       cor 
## 0.3967922

The correlation coefficient between years since Ph.D. and salary is 0.4434727, which is a medium correlation. The p value is 6.896e-13. Since the p value is less than the significance level 0.05, the null hypothesis that the correlation coefficient is 0 is rejected. Therefore the correlation between years since Ph.D. and salary is significant.

6

mod1<-lm(salary~yrs.since.phd,train)
summary(mod1)
## 
## Call:
## lm(formula = salary ~ yrs.since.phd, data = train)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -78821 -17071  -3029  14217  75822 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    92359.0     3357.7  27.506  < 2e-16 ***
## yrs.since.phd    867.9      130.7   6.641 2.13e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 25910 on 236 degrees of freedom
## Multiple R-squared:  0.1574, Adjusted R-squared:  0.1539 
## F-statistic:  44.1 on 1 and 236 DF,  p-value: 2.127e-10

7

residuals<-resid(mod1)
max(residuals)
## [1] 75821.96
min(residuals)
## [1] -78821.13

The minimum and maximum residual values in this model are -74091.8 and 99591.47.

a

residuals[which.max(residuals)]
##       43 
## 75821.96
train[as.numeric(names(residuals[which.max(residuals)])),"salary"]
## [1] 205500
mod1$fitted.values[as.numeric(names(residuals[which.max(residuals)]))]
##     43 
## 129678

The observation that generates the highest residual value is sample row 226, with an actual salary of 231545. The predicted salary is 131953.5, and the actual salary minus the predicted salary is the residual value.

b

residuals[which.min(residuals)]
##       139 
## -78821.13
train[as.numeric(names(residuals[which.min(residuals)])),"salary"]
## [1] 57800
mod1$fitted.values[as.numeric(names(residuals[which.min(residuals)]))]
##      139 
## 136621.1

The observation that generates the lowest residual value is sample row 227, with an actual salary of 76840. The predicted salary is 150931.8, and the actual salary minus the predicted salary is the residual value.

c

years since PhD. may not completely predict professor salary, possibly because years since PhD., although an important factor, is not the only factor that determines professor salary. A professor’s salary can be affected by a variety of factors. Therefore, predicting a professor’s salary based solely on years since PhD. may ignore these other important factors, causing the model to deviate.

9

\(salary=91888.3+1054.3*yrs.since.phd\)

When yrs.since.phd=30,\(salary=91888.3+1054.3*30=123517.3\)

10

pred_train<-predict(mod1,train)
pred_test<-predict(mod1,test)
accuracy(train$salary,pred_train)
##                    ME     RMSE      MAE        MPE     MAPE
## Test set 1.353256e-11 25799.34 19881.63 0.08660941 17.45646
accuracy(test$salary,pred_test)
##                 ME     RMSE      MAE       MPE     MAPE
## Test set -4945.647 30041.84 23374.49 -4.015951 20.56992

The accuracy of the model on the training and validation sets is evaluated to check the generalization ability and robustness of the model. For the training set, the RMSE is 26610.51 and the MAE is 20194.32; while for the test set, the RMSE is 28855.5 and the MAE is 23033.08. The RMSE and MAE of the test set are both smaller than the training set, indicating that the model performs better on the training set than on the test set. Indicates that the model may be overfitting the data and the model needs to be further adjusted.

11

sd(train$salary)
## [1] 28165.9

The standard deviation of salaries in the training set is 29752.26. When we compare this value with the RMSE of the model (26610.51), we can observe that the RMSE is slightly lower than the standard deviation of the salaries in the training set. Note that on average, the deviation between the model’s predictions and actual salaries is slightly less than the standard deviation of the salaries in the training set. This suggests that the model captures a substantial portion of the variation present in the data, but still needs improvement in reducing prediction error.

Multiple Linear Regression

1

cor(train[,c("yrs.since.phd","yrs.service","salary")])
##               yrs.since.phd yrs.service    salary
## yrs.since.phd     1.0000000   0.9047598 0.3967922
## yrs.service       0.9047598   1.0000000 0.3194599
## salary            0.3967922   0.3194599 1.0000000

The correlation coefficient between yrs.service and yrs.since.phd is 0.8982139, which is a strong correlation, thus suggesting that multicollinearity may be an issue. Since the correlation coefficient between salary and yrs.since.phd is 0.4434727, it is higher than the correlation coefficient between yrs.service and salary. Therefore the yrs.service variable should be removed.

2

Dummy variables are a coding method used to represent categorical variables. Usually 0 or 1 is used to represent different categories of a categorical variable. Each dummy variable corresponds to a category. The purpose of dummy variables is to convert categorical variables into numerical forms that can be used in the model, so that the model can better understand and process categorical information.

3

fullmodel<-lm(salary~rank+discipline+yrs.since.phd+sex,train)
mod2<-step(fullmodel, direction = "backward" ) 
## Start:  AIC=4746.5
## salary ~ rank + discipline + yrs.since.phd + sex
## 
##                 Df  Sum of Sq        RSS    AIC
## - yrs.since.phd  1 5.5152e+07 1.0379e+11 4744.6
## - sex            1 2.7794e+08 1.0401e+11 4745.1
## <none>                        1.0374e+11 4746.5
## - discipline     1 1.2307e+10 1.1604e+11 4771.2
## - rank           2 4.1335e+10 1.4507e+11 4822.3
## 
## Step:  AIC=4744.62
## salary ~ rank + discipline + sex
## 
##              Df  Sum of Sq        RSS    AIC
## - sex         1 2.5642e+08 1.0405e+11 4743.2
## <none>                     1.0379e+11 4744.6
## - discipline  1 1.3036e+10 1.1683e+11 4770.8
## - rank        2 7.6994e+10 1.8078e+11 4872.7
## 
## Step:  AIC=4743.21
## salary ~ rank + discipline
## 
##              Df  Sum of Sq        RSS    AIC
## <none>                     1.0405e+11 4743.2
## - discipline  1 1.3321e+10 1.1737e+11 4769.9
## - rank        2 7.9343e+10 1.8339e+11 4874.1
summary(mod2)
## 
## Call:
## lm(formula = salary ~ rank + discipline, data = train)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -63210 -11670  -1311   7423  90131 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      70127       3921  17.887  < 2e-16 ***
## rankAssocProf    14126       4796   2.945  0.00355 ** 
## rankProf         45241       3748  12.071  < 2e-16 ***
## disciplineB      15401       2814   5.473 1.13e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 21090 on 234 degrees of freedom
## Multiple R-squared:  0.4466, Adjusted R-squared:  0.4395 
## F-statistic: 62.95 on 3 and 234 DF,  p-value: < 2.2e-16

4

a

SST<-sum((train$salary-mean(train$salary))^2)
SST
## [1] 188016304377

b

SSR<-sum((mod2$fitted.values-mean(train$salary))^2)
SSR
## [1] 83969363810

c

SSR/SST
## [1] 0.4466068

The value of SSR/SST is 0.4712098,which is Multiple R-squared.

5

The t value of disciplineB is 5.021.

visualize.t(stat = c(-5.021,5.021), df = 233, section = "bounded") 

The shaded area is basically 100%. The shaded area is the probability that the t value is between -5.021 and 5.021, indicating that the p value of the predictor variable is very low.

6

The F statistic of the model is 51.91. The F statistic measures the overall fit of the linear regression model, that is, whether the independent variables in the regression equation significantly explain the variability of the dependent variable. The F statistic can test whether the model is overall significant. \(F = \frac{{\text{Mean Square Regression}}}{{\text{Mean Square Residual}}}\)

SSR <- sum((fitted(mod2) - mean(train$salary))^2) 
SSE <- sum(residuals(mod2)^2)                      
df_regression <- 4                                
df_residual <- 233                                

# Mean Square Regression  and Mean Square Residual
MSR <- SSR / df_regression
MSE <- SSE / df_residual

F_statistic <- MSR / MSE
F_statistic
## [1] 47.0097

7

rank:Full Professor discipline:B sex:Male

new_professor <- data.frame(rank = "Prof", discipline = "B", sex = "Male")

predicted_salary <- predict(mod2, newdata = new_professor)

print(predicted_salary)
##        1 
## 130769.4

The predicted value is 134961.6.

8

train_pred <- predict(mod2, newdata = train)
train_accuracy <- accuracy(train_pred, train$salary)
print(train_accuracy)
##                    ME     RMSE      MAE       MPE     MAPE
## Test set -1.09543e-10 20908.66 14865.93 -3.040128 13.18399
valid_pred <- predict(mod2, newdata = test)
valid_accuracy <- accuracy(valid_pred, test$salary)
print(valid_accuracy)
##                ME     RMSE      MAE     MPE    MAPE
## Test set 6516.274 25244.79 18522.03 1.94376 14.7454

Compared with the training set, the RMSE and MAE of the validation set have increased, MAPE has also increased slightly, and the average error ME is negative. This shows that the model’s prediction effect on the validation set is slightly worse than that on the training set, but it is still able to make reasonable predictions on the data overall. This situation implies a possible risk of overfitting. Compared with a simple linear regression model, a multiple linear regression model can better capture the impact of multiple independent variables on the dependent variable, and therefore may have a better fit on the training set. However, the complexity of multiple linear regression models increases the risk of overfitting, especially when sample sizes are small or when multicollinearity exists among independent variables.