2024-03-12

Definition & Form

Simple Linear Regression is a statistical technique that models the relationship between one independent variable and one dependent.

The typical linear regression model appears in the following form:

\(Y = \beta_0 + \beta_1 X + \epsilon\),

where \(X\) is the independent variable, \(Y\) is the dependent variable, \(\beta_0\) is the slope intercept, \(\beta_1\) is the regression coefficient, and \(\epsilon\) is the residual error.

The value of regression parameter \(\hat\beta\) is such that the model sum of squared residuals is minimized.

Linear Regression Math

Theoretically, \(\beta_1 = \frac{Cov(X,Y)}{Var(x)}\) and \(\beta_0 = E[Y] - \beta_1E[X]\).

However, when working with a random sample, rather than the entire population, the true values of \(Cov(X,Y)\), \(Var(x)\), \(E[Y]\), and \(E[X]\) are not known. So, we must calculate \(\beta_1\), \(\beta_0\) in terms of sample analogs.

Hence, we derive the following estimators for \(\beta_1\) and \(\beta_0\):

  • \(\hat{\beta}_1 = \frac{\sum^{N}_{i=1}(x_i-\bar x)(y_i-\bar y)}{\sum^{N}_{i=1}(x_i-\bar x)^2}\)
  • \(\hat{\beta}_0 = \frac{1}{N}\sum^{N}_{i=1}y_i - \hat{\beta}_1\frac{1}{N}\sum^{N}_{i=1}x_i\)

Practical Example

Now, lets consider a practical example of simple linear regression in R. We will work with the \(\textbf{USArrests}\) data set, which contains data on violent crime arrests (per 100,000) by US State.

##            Murder Assault UrbanPop Rape
## Alabama      13.2     236       58 21.2
## Alaska       10.0     263       48 44.5
## Arizona       8.1     294       80 31.0
## Arkansas      8.8     190       50 19.5
## California    9.0     276       91 40.6
## Colorado      7.9     204       78 38.7

Lets conduct a simple linear regression analysis in R to model the relationship between assault arrests (per 100000) and percent urban population.

Plotly Scatter Plot

Before we build a model, lets plot assault arrests (per 100000) against percent urban population to determine whether there is in fact a linear form to the variables’ relationship.

There appears to be an extremely weak, positive linear relationship between assault arrests (per 100000) and percent urban population.

Linear Regression Ouput

The graph on the last slide did not provide a clear picture as to the relationship between assault arrests (per 100000) and percent urban population. Lets create and analyze the simple regression model to get a clearer picture.

simple_linear_model = lm(Assault ~ UrbanPop, data=USArrests)
coeftest(simple_linear_model)
## 
## t test of coefficients:
## 
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept) 73.07658   53.85084  1.3570  0.18112  
## UrbanPop     1.49044    0.80269  1.8568  0.06948 .
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Linear Regression Interpretation

The estimated regression coefficient for UrbanPop is \(1.490\) with a \(p-value\) of \(0.070\). We can interpret that a 1 unit increase in Urban Population percentage is associated with a 1.49 increase in assault arrests. Our intercept in \(73.077\), meaning, when an the percent of a state’s population that is urban is \(0\), we can expect \(73.077\) assault arrests to take place.

However, if we set our confidence level at \(95\%\), our model fails to reject the null hypothesis. That is to say, there is not sufficient evidence at the \(95\%\) confidence level to claim that there is a linear association between Urban Population percentage and the number of assault arrests in a given state.

GGPLOT for LM

Let’s graph our simple regression model.

## `geom_smooth()` using formula = 'y ~ x'

GGPLOT for Residuals

Finally, let’s plot and discuss our residuals.

Residuals Discussion

On a positive note, our model residuals are relatively normal, with no clear relationship between the fitted model and residuals. A normal distribution of residuals is one of the key assumptions of Simple Linear Regression models. Always check to make sure your errors are normally distributed.