Analyzing Variability

library(tidyverse)
library(openintro)

Comment:

This is material I had planned to present Thursday night but couldn’t. So I am presenting it in an R Notebook. Please work through this material before doing the assignment, it will help.

Total variation

Suppose we have sets of \(n\) observations x and y. And we want to analyze the relatioship between them. Looking at a scatter plot of x and y we conclude that there may be a linear association between the variables. (WARNING: an association does not mean there is a causal relation.) Anyway, we perform a least squares regression and fit a model of the form: \[Y = \beta_0 + \beta_1X + \epsilon_i\] Where \(\epsilon \sim\mathcal{N}(0,\sigma^2).\) That is the \(\epsilon_i\) are normally distributed with mean 0 and variance \(\sigma^2\).

Now look at the variability of \(Y\). This is given by the variation of \(Y\) about \(\bar{Y}\) which is \[\text{Total variability} = \sum_{i=1}^{n}(Y_i - \bar{Y})^2\]

The Regression Variablity is the variability that is explained by adding the predictor (x). It is given by \[\text{Regression variability} = \sum_{i=1}^{n}(\hat{Y_i} - \bar{Y})^2\]

The Residual variablitity is what is left over around the regression line. \[\text{Residual variability}= \sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2\] An interesting fact is that: (proof beyond scope of this class)

\[\text{Total variability} = \text{Residual variability} + \text{Regression variability}\] Or in symbols:

\[\sum_{i=1}^{n}(Y_i - \bar{Y})^2 = \sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2 + \sum_{i=1}^{n}(\hat{Y_i} - \bar{Y})^2\]

Let’s check this with the diamond data:

fit <- lm(price ~ carat, data = diamond)
summary(fit)

Call:
lm(formula = price ~ carat, data = diamond)

Residuals:
    Min      1Q  Median      3Q     Max 
-85.159 -21.448  -0.869  18.972  79.370 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -259.63      17.32  -14.99   <2e-16 ***
carat        3721.02      81.79   45.50   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 31.84 on 46 degrees of freedom
Multiple R-squared:  0.9783,    Adjusted R-squared:  0.9778 
F-statistic:  2070 on 1 and 46 DF,  p-value: < 2.2e-16

price <-  diamond$price
carat <-  diamond$carat
df <- nrow(diamond) - 1
df

[1] 47

res_var = sum(residuals(fit)^2)
y_hat = predict(fit)
reg_var = sum((y_hat - mean(price))^2)
tot_var = sum((price - mean(price))^2)
tot_var

[1] 2145232

res_var + reg_var

[1] 2145232

tot_var - (res_var + reg_var)

[1] -9.313226e-10

\(R^2\)

R squared is the percentage of the total variability that is explained by the linear relationsip with the predictor. i.e., \[R^2 = \frac{\sum_{i=1}^{n}(\hat{Y}_i - \bar{Y})^2}{\sum_{i=1}^{n}(Y_i - \bar{Y})^2}\]

Looking at \(R^2\) in diamond, we get

Rsq = reg_var/tot_var
Rsq

[1] 0.9782608

cor(carat,price)^2 #square of the correlation coefficient

[1] 0.9782608

Rsq - cor(carat,price)^2

[1] 4.440892e-16

Interesting observations:

1. \(R^2\) is the percent of variation explained by the regression model.
2. \(R^2\) is the sample correlation squared
3. \(R^2\) can be a misleading summary of model fit. (deleting data may inflate the value)