Adjusted R-Squared Evaluation

Introduction

Students at the beginning of PREDICT 411 (generalized linear models) tend to lean towards reporting Adjusted R-squared as their model evaluation criteria whereas grades are assigned according to some other, mysterious, error measurement conducted by the Professor.

The goal of adjusted R-squared is to account for the amount of variance explained by a given mdoel without encouraging over-fitting - when the model fits the sample data very well but is not as good of a fit out of sample. Unfortunately, sometimes over-fitting still happens when using adjusted r-squared as opposed to something like cross-validated error. The task here is to compare adjusted R-squared to this error measure and evaluate any patterns we see.

The data

Prof. Wilck has made 45 example records available that contain his error metric and past students’ self-reported adjusted r-squared.

library(readr)
library(dplyr)
setwd('C:/Users/joshy/Google Drive/northwestern/PREDICT-411/general-docs/arsq-eval/')
dat <- read_csv('Adjusted_Rsquared.csv')

Parsed with column specification:
cols(
  ADJ_R2 = col_double(),
  SCORE = col_double()
)

head(dat)

Plotting the model health scores (using log scales to normalize the distributions) demonstrates that students may actually be selling themselves short by reporting adjusted R-squared! Values for adjusted r-squared span a wide range of values while Prof. Wilck’s score is more consistently at the low end of it’s distribution.

library(ggplot2)
library(yaztheme)
library(ggridges)
# cor(log(dat))^2
scatter <- ggplot(dat, aes(log(ADJ_R2), log(SCORE)))+
  geom_point(size = 3, alpha = .7, color = yaz_cols[3])+
  theme_yaz()+
  labs(title = 'Distributions of Model Health Metrics')
scatter

At the risk of being too on-the-nose, the R-squared value for the line of best fit on the above scatter plot is just .14, meaning adjusted r-squared only accounts for 14% of Prof. Wilck’s out of sample error calculation!

Why does this matter?

By definition, predictive models are meant to be deployed out of sample. If the model fits in-sample and fails when applied to new data we risk making bad decisions. On the other hand, if a model appears to fit poorly in-sample but performs well out of sample, we risk the opportunity cost of passing up a good tool to use in our decision-making process. My preferred model health metric is some form of hold-out sample error testing like root mean squared error.