Robust Regression - Introduction

Usage of Robust Regression

• Robust regression can be used in any situation in which you would use least squares regression. When fitting a least squares regression, we might find some outliers or high leverage data points.
• We have decided that these data points are not data entry errors, neither they are from a different population than most of our data. So we have no compelling reason to exclude them from the analysis.
• Robust regression might be a good strategy since it is a compromise between excluding these points entirely from the analysis and including all the data points and treating all them equally in OLS regression. The idea of robust regression is to weigh the observations differently based on how well behaved these observations are. Roughly speaking, it is a form of weighted and reweighted least squares regression.
• Robust regression does not address issues of heterogeneity of variance.

Popularity of Robust Regression

Despite their superior performance over least squares estimation in many situations, robust methods for regression are still not widely used. Several reasons may help explain their unpopularity (Hampel et al. 1986, 2005). One possible reason is that there are several competing methods and the field got off to many false starts. Also, computation of robust estimates is much more computationally intensive than least squares estimation; in recent years however, this objection has become less relevant as computing power has increased greatly. Another reason may be that some popular statistical software packages failed to implement the methods (Stromberg, 2004). The belief of many statisticians that classical methods are robust may be another reason.

Fitting a robust model (rlm()}

The rlm() command in the MASS package command implements several versions of robust regression.

library(MASS)

Implementation of Robust Regression

• When fitting a least squares regression, we might find some outliers or high leverage data points. We have decided that these data points are not data entry errors, neither they are from a different population than most of our data. So we have no proper reason to exclude them from the analysis.

• Robust regression might be a good strategy since it is a compromise between excluding these points entirely from the analysis and including all the data points and treating all them equally in OLS regression. The idea of robust regression is to weigh the observations differently based on how well behaved these observations are.

• The idea of robust regression is to weigh the observations differently based on how well behaved these observations are. Roughly speaking, it is a form of weighted and reweighted least squares regression (i.e. a two step process , first fitting a linear model, then a robust model to correct for the influence of outliers).

• Robust regression is done by iterated re-weighted least squares (IRLS). The rlm command in the MASS package command implements several versions of robust regression.

• There are several weighting functions that can be used for IRLS. We are going to first use the Huber weights in this example. We will then look at the final weights created by the IRLS process. This can be very useful.

• Also we will look at an alternative weighting approach to Huber’s weighting – called Bisquare weighting.

Implementation with Bisquare Weighting

Implementing with bisquare weighting simply requires the specification of the additional argument, as per the code below)

FitAll.rr.2 = rlm(taste ~ Acetic + H2S + Lactic,
                  data=cheddar,
                  psi = psi.bisquare)

summary(FitAll.rr.2)

## 
## Call: rlm(formula = taste ~ Acetic + H2S + Lactic, data = cheddar, 
##     psi = psi.bisquare)
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -15.7034  -5.1552  -0.9793   5.6933  27.7661 
## 
## Coefficients:
##             Value    Std. Error t value 
## (Intercept) -17.7730  20.7031    -0.8585
## Acetic       -2.2650   4.6784    -0.4841
## H2S           4.0569   1.3096     3.0977
## Lactic       20.6885   9.0522     2.2855
## 
## Residual standard error: 7.878 on 26 degrees of freedom

\[Taste* = -17.77 -2.26 Acetic + 4.05 H2S + 20.68 Lactic\]

Weights using Bisquare estimator.

##hweights2[1:15, ]

Conclusion

• We can see that the weight given to some observations is dramatically lower using the bisquare weighting function than the Huber weighting function and the coefficient estimates from these two different weighting methods differ.
• When comparing the results of a regular OLS regression and a robust regression, if the results are very different, you will most likely want to use the results from the robust regression.
• Large differences suggest that the model parameters are being highly influenced by outliers.
• Different functions have advantages and drawbacks. Huber weights can have difficulties with severe outliers, and bisquare weights can have difficulties converging or may yield multiple solutions.

---