library(dplyr)
library(ggplot2)
library(forecast)
library(visualize)
library(carData)
data(Salaries)
glimpse(Salaries)
## Rows: 397
## Columns: 6
## $ rank <fct> Prof, Prof, AsstProf, Prof, Prof, AssocProf, Prof, Prof,…
## $ discipline <fct> B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, B, A, A,…
## $ yrs.since.phd <int> 19, 20, 4, 45, 40, 6, 30, 45, 21, 18, 12, 7, 1, 2, 20, 1…
## $ yrs.service <int> 18, 16, 3, 39, 41, 6, 23, 45, 20, 18, 8, 2, 1, 0, 18, 3,…
## $ sex <fct> Male, Male, Male, Male, Male, Male, Male, Male, Male, Fe…
## $ salary <int> 139750, 173200, 79750, 115000, 141500, 97000, 175000, 14…
numeric:yrs.since.phd,yrs.service and salary
factor:rank,discipline and sex
set.seed(250)
index<-sample(1:nrow(Salaries),0.6*nrow(Salaries))
train<-Salaries[index,]
test<-Salaries[-index,]
rownames(train)<-c(1:nrow(train))
a.Dividing the data into a training set and a validation set helps evaluate model performance, prevent overfitting, and adjust parameters. This partitioning method ensures that the model can generalize to new data during training and improves the model’s reliability and accuracy.
ggplot(train,aes(x=yrs.since.phd,y=salary))+geom_point()+geom_smooth()+ggtitle("The relationship between years since Ph.D. and salary")
## `geom_smooth()` using method = 'loess' and formula = 'y ~ x'
As the years since Ph.D. increase, salary has a tendency to increase first and then decrease. This makes intuitive sense to me, because this picture shows that there is no obvious linear relationship between years since Ph.D. and salary, but a quadratic relationship.
cor.test(train$yrs.since.phd,train$salary)
##
## Pearson's product-moment correlation
##
## data: train$yrs.since.phd and train$salary
## t = 6.6408, df = 236, p-value = 2.127e-10
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## 0.2839580 0.4987868
## sample estimates:
## cor
## 0.3967922
The correlation coefficient between years since Ph.D. and salary is 0.4434727, which is a medium correlation. The p value is 6.896e-13. Since the p value is less than the significance level 0.05, the null hypothesis that the correlation coefficient is 0 is rejected. Therefore the correlation between years since Ph.D. and salary is significant.
mod1<-lm(salary~yrs.since.phd,train)
summary(mod1)
##
## Call:
## lm(formula = salary ~ yrs.since.phd, data = train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -78821 -17071 -3029 14217 75822
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 92359.0 3357.7 27.506 < 2e-16 ***
## yrs.since.phd 867.9 130.7 6.641 2.13e-10 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 25910 on 236 degrees of freedom
## Multiple R-squared: 0.1574, Adjusted R-squared: 0.1539
## F-statistic: 44.1 on 1 and 236 DF, p-value: 2.127e-10
residuals<-resid(mod1)
max(residuals)
## [1] 75821.96
min(residuals)
## [1] -78821.13
The minimum and maximum residual values in this model are -74091.8 and 99591.47.
residuals[which.max(residuals)]
## 43
## 75821.96
train[as.numeric(names(residuals[which.max(residuals)])),"salary"]
## [1] 205500
mod1$fitted.values[as.numeric(names(residuals[which.max(residuals)]))]
## 43
## 129678
The observation that generates the highest residual value is sample row 226, with an actual salary of 231545. The predicted salary is 131953.5, and the actual salary minus the predicted salary is the residual value.
residuals[which.min(residuals)]
## 139
## -78821.13
train[as.numeric(names(residuals[which.min(residuals)])),"salary"]
## [1] 57800
mod1$fitted.values[as.numeric(names(residuals[which.min(residuals)]))]
## 139
## 136621.1
The observation that generates the lowest residual value is sample row 227, with an actual salary of 76840. The predicted salary is 150931.8, and the actual salary minus the predicted salary is the residual value.
years since PhD. may not completely predict professor salary, possibly because years since PhD., although an important factor, is not the only factor that determines professor salary. A professor’s salary can be affected by a variety of factors. Therefore, predicting a professor’s salary based solely on years since PhD. may ignore these other important factors, causing the model to deviate.
\(salary=91888.3+1054.3*yrs.since.phd\)
When yrs.since.phd=30,\(salary=91888.3+1054.3*30=123517.3\)
pred_train<-predict(mod1,train)
pred_test<-predict(mod1,test)
accuracy(train$salary,pred_train)
## ME RMSE MAE MPE MAPE
## Test set 1.353256e-11 25799.34 19881.63 0.08660941 17.45646
accuracy(test$salary,pred_test)
## ME RMSE MAE MPE MAPE
## Test set -4945.647 30041.84 23374.49 -4.015951 20.56992
The accuracy of the model on the training and validation sets is evaluated to check the generalization ability and robustness of the model. For the training set, the RMSE is 26610.51 and the MAE is 20194.32; while for the test set, the RMSE is 28855.5 and the MAE is 23033.08. The RMSE and MAE of the test set are both smaller than the training set, indicating that the model performs better on the training set than on the test set. Indicates that the model may be overfitting the data and the model needs to be further adjusted.
sd(train$salary)
## [1] 28165.9
The standard deviation of salaries in the training set is 29752.26. When we compare this value with the RMSE of the model (26610.51), we can observe that the RMSE is slightly lower than the standard deviation of the salaries in the training set. Note that on average, the deviation between the model’s predictions and actual salaries is slightly less than the standard deviation of the salaries in the training set. This suggests that the model captures a substantial portion of the variation present in the data, but still needs improvement in reducing prediction error.
cor(train[,c("yrs.since.phd","yrs.service","salary")])
## yrs.since.phd yrs.service salary
## yrs.since.phd 1.0000000 0.9047598 0.3967922
## yrs.service 0.9047598 1.0000000 0.3194599
## salary 0.3967922 0.3194599 1.0000000
The correlation coefficient between yrs.service and yrs.since.phd is 0.8982139, which is a strong correlation, thus suggesting that multicollinearity may be an issue. Since the correlation coefficient between salary and yrs.since.phd is 0.4434727, it is higher than the correlation coefficient between yrs.service and salary. Therefore the yrs.service variable should be removed.
Dummy variables are a coding method used to represent categorical variables. Usually 0 or 1 is used to represent different categories of a categorical variable. Each dummy variable corresponds to a category. The purpose of dummy variables is to convert categorical variables into numerical forms that can be used in the model, so that the model can better understand and process categorical information.
fullmodel<-lm(salary~rank+discipline+yrs.since.phd+sex,train)
mod2<-step(fullmodel, direction = "backward" )
## Start: AIC=4746.5
## salary ~ rank + discipline + yrs.since.phd + sex
##
## Df Sum of Sq RSS AIC
## - yrs.since.phd 1 5.5152e+07 1.0379e+11 4744.6
## - sex 1 2.7794e+08 1.0401e+11 4745.1
## <none> 1.0374e+11 4746.5
## - discipline 1 1.2307e+10 1.1604e+11 4771.2
## - rank 2 4.1335e+10 1.4507e+11 4822.3
##
## Step: AIC=4744.62
## salary ~ rank + discipline + sex
##
## Df Sum of Sq RSS AIC
## - sex 1 2.5642e+08 1.0405e+11 4743.2
## <none> 1.0379e+11 4744.6
## - discipline 1 1.3036e+10 1.1683e+11 4770.8
## - rank 2 7.6994e+10 1.8078e+11 4872.7
##
## Step: AIC=4743.21
## salary ~ rank + discipline
##
## Df Sum of Sq RSS AIC
## <none> 1.0405e+11 4743.2
## - discipline 1 1.3321e+10 1.1737e+11 4769.9
## - rank 2 7.9343e+10 1.8339e+11 4874.1
summary(mod2)
##
## Call:
## lm(formula = salary ~ rank + discipline, data = train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -63210 -11670 -1311 7423 90131
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 70127 3921 17.887 < 2e-16 ***
## rankAssocProf 14126 4796 2.945 0.00355 **
## rankProf 45241 3748 12.071 < 2e-16 ***
## disciplineB 15401 2814 5.473 1.13e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 21090 on 234 degrees of freedom
## Multiple R-squared: 0.4466, Adjusted R-squared: 0.4395
## F-statistic: 62.95 on 3 and 234 DF, p-value: < 2.2e-16
SST<-sum((train$salary-mean(train$salary))^2)
SST
## [1] 188016304377
SSR<-sum((mod2$fitted.values-mean(train$salary))^2)
SSR
## [1] 83969363810
SSR/SST
## [1] 0.4466068
The value of SSR/SST is 0.4712098,which is Multiple R-squared.
The t value of disciplineB is 5.021.
visualize.t(stat = c(-5.021,5.021), df = 233, section = "bounded")
The shaded area is basically 100%. The shaded area is the probability that the t value is between -5.021 and 5.021, indicating that the p value of the predictor variable is very low.
The F statistic of the model is 51.91. The F statistic measures the overall fit of the linear regression model, that is, whether the independent variables in the regression equation significantly explain the variability of the dependent variable. The F statistic can test whether the model is overall significant. \(F = \frac{{\text{Mean Square Regression}}}{{\text{Mean Square Residual}}}\)
SSR <- sum((fitted(mod2) - mean(train$salary))^2)
SSE <- sum(residuals(mod2)^2)
df_regression <- 4
df_residual <- 233
# Mean Square Regression and Mean Square Residual
MSR <- SSR / df_regression
MSE <- SSE / df_residual
F_statistic <- MSR / MSE
F_statistic
## [1] 47.0097
rank:Full Professor discipline:B sex:Male
new_professor <- data.frame(rank = "Prof", discipline = "B", sex = "Male")
predicted_salary <- predict(mod2, newdata = new_professor)
print(predicted_salary)
## 1
## 130769.4
The predicted value is 134961.6.
train_pred <- predict(mod2, newdata = train)
train_accuracy <- accuracy(train_pred, train$salary)
print(train_accuracy)
## ME RMSE MAE MPE MAPE
## Test set -1.09543e-10 20908.66 14865.93 -3.040128 13.18399
valid_pred <- predict(mod2, newdata = test)
valid_accuracy <- accuracy(valid_pred, test$salary)
print(valid_accuracy)
## ME RMSE MAE MPE MAPE
## Test set 6516.274 25244.79 18522.03 1.94376 14.7454
Compared with the training set, the RMSE and MAE of the validation set have increased, MAPE has also increased slightly, and the average error ME is negative. This shows that the model’s prediction effect on the validation set is slightly worse than that on the training set, but it is still able to make reasonable predictions on the data overall. This situation implies a possible risk of overfitting. Compared with a simple linear regression model, a multiple linear regression model can better capture the impact of multiple independent variables on the dependent variable, and therefore may have a better fit on the training set. However, the complexity of multiple linear regression models increases the risk of overfitting, especially when sample sizes are small or when multicollinearity exists among independent variables.