March 09, 2024

What is linear regression?

Linear regression is a statistical model which estimates the relationship between variables.

It attempts to create a model between one or more independent variables and one dependent variable.

In this presentation, I will focus on the single independent variable case.

Through linear regression, we get an equation of the following form:

\(\hat{y} = b_0 + b_1 \cdot x\)

In this equation \(\hat{y}\) is the predicted value for the independent variable.

\(b_0\) is the intercept (our predicted y value given an x value of 0). \(b_1\) is the slope (how much we predict y to increase given an increase of 1 in x). \(x\) is our independent variable.

How do we find the terms for linear regression

To find the linear regression equation, we must find the equation which minimizes the value for SSR (sum of squared residuals) in the following equation:

\(\text{SSR} = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2\)

In effect, the above equation sums with square of the difference of the observed and expected values of the our equation.

Example Linear Regression

Using the following sample data, we can get the following linear regression model

x <- 1:10
y <- 2*x + rnorm(10, mean = 0, sd=8) #generate points that randomly deviate

When is linear regression suitable

Linear regression is suitable when you observe a linear relationship in your variable. It also requires that you can represent the This relationship can be either positive or negative. The following scatter plot shows a suitable example for a positive relationship.

The next scatter plot shows a suitable example for a negative relationship.

When is linear regression not suitable

Linear regression is not suitable when there is not a linear relationship. This can occur when the data points are scattered or when there is curvature in the graph. The next plot shows a case where there is no discernible relationship:

The next scatter plot shows that there is curvature in the graph, meaning that it is not suitable for linear regression.

How to do linear regression in R

The following R code shows how to do a simple linear regression as well as its output.

#sample data
x <- 1:20
y <- 3*x + 10 + rnorm(20, mean = 0, sd=8)
df = data.frame(x, y)

#generate the model
result <- lm(y ~ x, data = df) 
summary(result)
## 
## Call:
## lm(formula = y ~ x, data = df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -16.6984  -5.9515  -0.3997   7.3101  13.7752 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  11.0112     3.8761   2.841   0.0108 *  
## x             2.9488     0.3236   9.113 3.66e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 8.344 on 18 degrees of freedom
## Multiple R-squared:  0.8219, Adjusted R-squared:  0.812 
## F-statistic: 83.05 on 1 and 18 DF,  p-value: 3.655e-08

Additional outputs of linear regression and their interpretation

From the last slide, we can see that linear regression also gives many other outputs which helps us evaluate the utility of the model.

The intercept value represents the predicted value when x is zero. The x value (or slope) represents the predicted increase in y for every increase of 1 in x.

The t-value and associated p-value represent the t-test on the slope. This test has a null hypothesis that the slope is zero, and an alternative hypothesis that the slope is non-zero. It is used to determine if the slope is statistically significant.

The \(R^2\) value represents how much variation in y is accounted for by x.

Extensions of linear regression (multiple regression and polynomial regression)

Using simple linear regression as a base, we can also build more complicated models. For example, earlier in the slide show, we said that linear regression does not work in the curved case. However, we can extend simple linear regression by adding in quadratic terms (and even higher order terms) to account for the curvature.

As well, if we have multiple variables that we think are relevant, we can do multiple regression. An example multiple regression is shown below with the sample data:

myX = -5:5
myY = -5:5
df <- expand.grid(x = myX, y = myY) #this generates all combinations of x and y
random_offset = rnorm(nrow(df), mean = 0, sd = 5) #this generates a random offset for all the points
df$z = 2*df$x + 3*df$y + random_offset

Thank you!

Thank you for reading through my presentation!

I have included 8 total slides with the following: 1 plotly plot (in slide 9), multiple ggplots (in slides 4, 5, and 6), 2 slides with LaTEX script (in slides 2 and 3), and 1 slide with R code (in slide 7).