Non-visual exploration

The data set contains 15 explanatory variables, and they can be categorized into four groups:

1. 7 batting metrics

2. 2 baserunning metrics

3. 2 fielding metrics

4. 4 pitching metrics

Visual Exploration

Creating Bins

From the histograms and summary statitics, TEAM_PITCHING_H seems to have the extreme outliers, so let’s put these values into quintile bins to mitigate their effect.

train_raw$TEAM_PITCHING_H <- bin_data(train_raw$TEAM_PITCHING_H, bins = 5, binType = "quantile") 

levels(train_raw$TEAM_PITCHING_H) <- c("One","Two","Three","Four","Five")

Missing Value Imputation

Let’s use the missForest package to do nonparametric missing-value imputation using Random Forest.

First, let’s try leaving in TEAM_BATTING_HPB, even though it has all of those NAs.

When we check the out-of-the-bag error, we would like to the normalized root mean-squares error (NRMSE) near zero, which would indicate a well-fitted imputation. However, as suspected, the normalized root mean-squares error (NRMSE) for TEAM_BATTING_HBP is nearly twice as large as the next highest NRMSE. Let’s remove it and impute the missing values again.

Source: Accuracy and Errors for Models

##   missForest iteration 1 in progress...done!
##   missForest iteration 2 in progress...done!
##   missForest iteration 3 in progress...done!
##   missForest iteration 4 in progress...done!
##   missForest iteration 5 in progress...done!

The remaining 5 imputed variables have NRMSE values have better fits and range from .02 to .11.

##   missForest iteration 1 in progress...done!
##   missForest iteration 2 in progress...done!
##   missForest iteration 3 in progress...done!
##   missForest iteration 4 in progress...done!
##   missForest iteration 5 in progress...done!

Model 1: All Variables

First, let’s build a model with all variables.

## 
## Call:
## lm(formula = TARGET_WINS ~ ., data = train_imputed)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -55.389  -8.298   0.195   8.167  54.913 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)    
## (Intercept)          28.6409850  5.7749125   4.960 7.59e-07 ***
## TEAM_BATTING_H        0.0467974  0.0036512  12.817  < 2e-16 ***
## TEAM_BATTING_2B      -0.0080999  0.0091590  -0.884  0.37659    
## TEAM_BATTING_3B       0.0248277  0.0166190   1.494  0.13533    
## TEAM_BATTING_HR       0.0655984  0.0277124   2.367  0.01801 *  
## TEAM_BATTING_BB       0.0123925  0.0052297   2.370  0.01789 *  
## TEAM_BATTING_SO      -0.0195892  0.0026891  -7.285 4.43e-13 ***
## TEAM_BASERUN_SB       0.0470115  0.0054356   8.649  < 2e-16 ***
## TEAM_BASERUN_CS       0.0262674  0.0156964   1.673  0.09437 .  
## TEAM_PITCHING_HTwo   -0.9872101  0.8882665  -1.111  0.26652    
## TEAM_PITCHING_HThree -1.0974513  0.9532332  -1.151  0.24973    
## TEAM_PITCHING_HFour  -4.3120963  1.1048503  -3.903 9.78e-05 ***
## TEAM_PITCHING_HFive  -3.1433220  1.4102628  -2.229  0.02592 *  
## TEAM_PITCHING_HR      0.0291162  0.0247491   1.176  0.23954    
## TEAM_PITCHING_BB     -0.0031523  0.0035931  -0.877  0.38041    
## TEAM_PITCHING_SO      0.0024559  0.0009026   2.721  0.00656 ** 
## TEAM_FIELDING_E      -0.0322373  0.0023362 -13.799  < 2e-16 ***
## TEAM_FIELDING_DP     -0.1123643  0.0140611  -7.991 2.11e-15 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 12.65 on 2258 degrees of freedom
## Multiple R-squared:  0.3602, Adjusted R-squared:  0.3554 
## F-statistic: 74.77 on 17 and 2258 DF,  p-value: < 2.2e-16

In the model output above, we notice the following:

• Of the 14 continuous & intercept variables, 9 have statistically significant p-values at the 5% significance level. In the 3rd model, we will explore which of the variables are actually necessary to a parsimonious model.

• Only 2 out of the 4 TEAM_PITCHING_H quintile dummy variables are statistically significant, so let’s check an ANOVA summary to make sure that the group means are significantly different. The F-statistic’s p-value is near zero, so we would reject the null hypothesis that the means are equal and keep the dummy variables in the model.

##                   Df Sum Sq Mean Sq F value Pr(>F)    
## TEAM_PITCHING_H    4  22687    5672   23.77 <2e-16 ***
## Residuals       2271 541810     239                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

• While we would expect negative coefficients for Fielding Errors, Walks Allowed & Batting Strikeouts, we wouldn’t expect negative values for Double Plays & Batting Doubles. Additionally, it’s surprising that Caught Stealing & Home Runs Allowed do not have negative coefficients. While none of the variables are perfectly collinear and were automatically removed from the model, these unexpected signs may indicate that we have some.

• At 0.3602, the adjusted \(R^2\) is not as high as we would prefer, and it indicates that only 36% of the variance in the response variable can be explained by the predictor variables.

• At 75, the F-statistic is large, and the model’s p-value is near zero. If the model’s diagnostics are sufficient, these values indicate that we would reject the null hypothesis that there is no relationship between the explanatory & response variables.

Model 2: Only Highly Correlated Variables

From our Correlated Variable Pairs table, let’s use only the 4 explanatory variables that were most correlated with the response variable.

##               Var1        Var2 Correlation
## 1  TEAM_PITCHING_H TARGET_WINS   0.4712343
## 2   TEAM_BATTING_H TARGET_WINS   0.4699467
## 3  TEAM_BATTING_BB TARGET_WINS   0.4686879
## 4 TEAM_PITCHING_BB TARGET_WINS   0.4683988

## 
## Call:
## lm(formula = TARGET_WINS ~ TEAM_PITCHING_H + TEAM_BATTING_H + 
##     TEAM_BATTING_BB + TEAM_PITCHING_BB, data = train_imputed)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -59.728  -8.963   0.394   9.085  65.012 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## (Intercept)          -6.052105   3.788542  -1.597  0.11030    
## TEAM_PITCHING_HTwo    0.092667   0.955466   0.097  0.92275    
## TEAM_PITCHING_HThree -0.034295   1.005300  -0.034  0.97279    
## TEAM_PITCHING_HFour  -3.315098   1.095721  -3.025  0.00251 ** 
## TEAM_PITCHING_HFive  -1.341073   1.301417  -1.030  0.30290    
## TEAM_BATTING_H        0.049052   0.002722  18.018  < 2e-16 ***
## TEAM_BATTING_BB       0.036671   0.003172  11.563  < 2e-16 ***
## TEAM_PITCHING_BB     -0.004882   0.002201  -2.218  0.02667 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 13.87 on 2268 degrees of freedom
## Multiple R-squared:  0.2275, Adjusted R-squared:  0.2251 
## F-statistic: 95.41 on 7 and 2268 DF,  p-value: < 2.2e-16

In the Model #2 output above, we notice the following:

• All 3 continuous variables are statistically significant, but this time the intercept is not.

• Only 1 of the 4 TEAM_PITCHING_H dummy variables is statistically significant.

• We would expect a negative coefficient for Walks Allowed and positive ones for Batting Walks & Batting Hits. However, 1 of the 4 Hits Allowed dummy variables does not have a negative coefficient, and this unexpected coefficient sign may indicate that we still have some collinearity.

• At 0.2251, the adjusted \(R^2\) indicates that this model explains less of variance in the response variable than Model #1.

• At 95, the F-statistic is larger than model #1, and the model’s p-value is near zero. If the model’s diagnostics are sufficient, these values indicate that we would reject the null hypothesis that there is no relationship between the explanatory & response variables.

Model 3: Backwards Elimination

• Let’s start with Model #1, but let’s remove the 4 variables that appeared to have multicollinearity issues because of their unexpected coefficient signs.

• Then we will remove one variable at a time by greatest coefficient p-value until we have a parsimonious model with only statistically significant variables.

• Along the way, we will take note if the removal of a variable causes a significant changes in adjusted R-squared and coefficient estimates.

• Lastly, TEAM_PITCHING_SO was removed because it didn’t have the expected sign.

## 
## Call:
## lm(formula = TARGET_WINS ~ TEAM_BATTING_H + TEAM_BATTING_HR + 
##     TEAM_BATTING_SO + TEAM_BASERUN_SB + TEAM_FIELDING_E, data = train_imputed)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -52.551  -8.678   0.128   8.514  53.122 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)    
## (Intercept)     25.629114   4.237642   6.048 1.71e-09 ***
## TEAM_BATTING_H   0.039267   0.002512  15.631  < 2e-16 ***
## TEAM_BATTING_HR  0.071958   0.008755   8.219 3.41e-16 ***
## TEAM_BATTING_SO -0.014925   0.002190  -6.814 1.21e-11 ***
## TEAM_BASERUN_SB  0.066361   0.003767  17.616  < 2e-16 ***
## TEAM_FIELDING_E -0.030775   0.001665 -18.482  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 12.93 on 2270 degrees of freedom
## Multiple R-squared:  0.3273, Adjusted R-squared:  0.3258 
## F-statistic: 220.9 on 5 and 2270 DF,  p-value: < 2.2e-16

In the Model #3 output above, we notice the following:

• All 5 continuous variables & and the intercept are statistically significant. Their p-values are near zero.

• Unlike the previous models, we do not have any unexpected coefficient signs, which means that we have removed at least some of the multilinearity.

• At 0.3258, the adjusted \(R^2\) indicates that this model explains 33% of the variance in the response variable.

• At 221, the F-statistic is larger than the other 2 models, and the model’s p-value is near zero. If the model’s diagnostics are sufficient, these values indicate that we would reject the null hypothesis that there is no relationship between the explanatory & response variables.

Evaluation

The table below summarizes the 3 models.

• While Model #1 has the highest adjusted R-squared, it also had 5 variables that were not statistically significant, and 4 variables with unexpected signs, which indicated multicollinearity issues.

• Model #2 used fewer variables, but the intercept coefficient was not statistically significant. We also saw some evidence of collinearity.

• Of the 3, Model #3 is the best. Removing the non-significant and collinear variables makes it more parsimonious with an adjusted R-squared that is less than Model #1 but much greater than Model #2. This model’s F-statistic is also significantly larger than the other 2. Additionally, the difference between Model #3’s predicted R-squared and R-squared is smaller than the other models, giving us more confidence that the model is not overfit.

Residual Plots

• In the Q-Q plot, we see some deviations for the normal distribution at the ends. Some of the extreme cases have been labeled. However, as the LMR text suggests, perhaps we can relax the normality assumption since we have more than 2200 observations.

• In the residuals-fitted and standardzied residuals-fitted plots, the best fitted line has a curve in the main cluster, which indicates that we do not have constant variance.

• Ideally, prior to making any inferences, we would want to attempt to transform the data in order to better meet the regression assumptions.

Test Model

##   missForest iteration 1 in progress...done!
##   missForest iteration 2 in progress...done!
##   missForest iteration 3 in progress...done!
##   missForest iteration 4 in progress...done!

test_results <- predict(mod3, newdata = test_imputed)