What is Linear Regression?

Linear Regression is a statistical analysis technique uses to calculate the relationship between two variables and extrapolate data. This is done by calculating a line of best fit for a set of data.

A line of best fit is drawn from the equation \(\hat{y} = \beta_{0} + \beta_{1}x\), and represents the approximate linear relationship between data.

Linear Regression is used to make predictions about variables based on their linear relationship.

Simple Linear Regression Example

In this sample data set, a line of best fit has been drawn based on the linear relationship between the length and width of flower petals.

Because the data does not include information about certain petal sizes, we use the line of best fit to interpolate/extrapolate and determine the expected width for these petals.

Calculating Linear Regression

The formula for a line of best fit is \(\hat{y} = \beta_{0} + \beta_{1}x\), where the coefficients \(\beta_{0}\) and \(\beta_{1}\) must be calculated using the following formulas:

\(\beta_{1} = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2}\)
\(\beta_{0} = \bar{y}-\beta_{1} \bar{x}\)

Where \(\bar{x}\) is the mean of x.

Linear Regression Example

Use linear regression to model the relationship between Weight and Miles per Gallon in the mtcars dataset.

This is a nonlinear relationship, so we set x = mpg and y = \(\frac{1}{wt}\). Linearizing nonlinear data is important for linear regression.

Linear Regression Example (cont.)

\(\beta_{1} = \frac{n\sum xy - \sum x \sum y}{n\sum x^2 - (\sum x)^2} = \frac{32 * 239.84 - 642.9*10.97}{32 * 1.40423\times 10^{4}- (642.9)^2} = 0.01735\)

\(\beta_{0} = \bar{y}-\beta_{1} \bar{x} = 0.3427 - 0.01735 * 20.091 = -0.00583\)

\(\hat{y} = -0.00583 + 0.01735x\)

Linear Regression in R

Calculating linear regression manually is very time-consuming, but R can make it much easier.

Let’s revisit the mtcars problem using the “lm” option in ggplot.

ggplot(aes(x = mpg, y = 1/wt), data = mtcars) + geom_point() + 
  geom_smooth(method = "lm", se = F) + xlab("") + ylab("")
## `geom_smooth()` using formula = 'y ~ x'

Interactive Regression Plot

When not to use Linear Regression

Linear Regression is a powerful tool for statistical analysis because it turns the associations between statistics into clear mathematical models. However, there are times when you should be cautious while using linear regression:

  • If you plot a scatterplot of your data and you don’t see a linear relationship, you either need to find some way to linearize the data or use some other method.
  • If the two variables have a low correlation coefficient, linear regression may give unpredictable and unhelpful results.
  • If your independent variable is not continuous, linear regression may or may not give a meaningful result.