What is Bias of an Estimator?

The bias of an estimator is the difference between an estimator’s expected value and the true value of the parameter being estimated. If an estimator is unbiased, its expected value matches the true parameter value. Bias indicates a systematic error in the estimator, where it consistently overestimates or underestimates the parameter.

Omitted Variable Bias (OVB)

Omitted Variable Bias occurs when a model leaves out one or more relevant variables. The bias does not disappear simply by increasing the sample size. Adding more variables can help, but only if the added variables are those that are actually missing and correlated both with the independent variable of interest and the dependent variable.

Conditions for Omitted Variable Bias

  1. The omitted variable is a determinant of the dependent variable (i.e., it has a non-zero coefficient in the true model).

  2. The omitted variable is correlated with one or more of the included independent variables.

Example

# Load the dataset
data(mtcars)

# Fit the full model
full_model <- lm(mpg ~ disp + wt + cyl, data = mtcars)

# Fit the reduced model (omitting the weight of the car)
reduced_model <- lm(mpg ~ disp + cyl, data = mtcars)

# Compare the coefficients
coef(full_model)
##  (Intercept)         disp           wt          cyl 
## 41.107677641  0.007472925 -3.635677016 -1.784943519
coef(reduced_model)
## (Intercept)        disp         cyl 
## 34.66099474 -0.02058363 -1.58727681
# You can also use the summary function to look at the details of each model
summary(full_model)
## 
## Call:
## lm(formula = mpg ~ disp + wt + cyl, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.4035 -1.4028 -0.4955  1.3387  6.0722 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 41.107678   2.842426  14.462 1.62e-14 ***
## disp         0.007473   0.011845   0.631  0.53322    
## wt          -3.635677   1.040138  -3.495  0.00160 ** 
## cyl         -1.784944   0.607110  -2.940  0.00651 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.595 on 28 degrees of freedom
## Multiple R-squared:  0.8326, Adjusted R-squared:  0.8147 
## F-statistic: 46.42 on 3 and 28 DF,  p-value: 5.399e-11
summary(reduced_model)
## 
## Call:
## lm(formula = mpg ~ disp + cyl, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.4213 -2.1722 -0.6362  1.1899  7.0516 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 34.66099    2.54700  13.609 4.02e-14 ***
## disp        -0.02058    0.01026  -2.007   0.0542 .  
## cyl         -1.58728    0.71184  -2.230   0.0337 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.055 on 29 degrees of freedom
## Multiple R-squared:  0.7596, Adjusted R-squared:  0.743 
## F-statistic: 45.81 on 2 and 29 DF,  p-value: 1.058e-09
# Check for correlation between the omitted variable (wt) and the dependent variable (mpg)
cor(mtcars$wt, mtcars$mpg)
## [1] -0.8676594
# Check for correlation between the omitted variable (wt) and the independent variable (disp)
cor(mtcars$wt, mtcars$disp)
## [1] 0.8879799

Full model: Coefficient for disp: 0.007473 (p-value = 0.53322, not statistically significant)

reduced model: Coefficient for disp: -0.02058 (p-value = 0.0542, marginally significant)

Interpreting OVB: The presence of OVB is further suggested by the change in magnitude and direction of the disp coefficient when wt is omitted. The conditions for OVB are met here:

  1. The omitted variable (wt) is significantly related to the dependent variable (mpg) as indicated by its coefficient in the full model.

  2. The omitted variable (wt) is likely correlated with the included independent variable (disp) since excluding wt leads to a significant change in the disp coefficient.

Given the significance of wt in predicting mpg, and the likelihood that heavier cars (with more weight) might also have larger engines (more displacement), it seems plausible that wt is correlated with both mpg and disp. This situation leads to OVB.

Direction of Bias: Since omitting wt leads to a negative coefficient for disp (compared to a positive but insignificant one when wt is included), it suggests that the bias is negative when wt is omitted. This might be because wt has a negative effect on mpg (heavier cars tend to have lower fuel efficiency), and when it’s omitted, its effect is partly attributed to disp, thus inflating the negative impact of disp on mpg.

Summary: The omitted weight (wt) of the cars is an important variable that affects fuel efficiency (mpg) and is correlated with engine size (disp). By not including it in the model, we risk attributing the effect that wt has on mpg to disp, which is what appears to have happened, leading to a negative bias in the estimate for disp. This results in the reduced model incorrectly suggesting that larger engines are associated with better fuel efficiency, contrary to what is indicated when wt is accounted for.