• Linear Regression is a technique to determine a predictive model between a predictive (independent) variable and a continuous target (dependent) variable
• It is one of the most widely used statistical methods for analysis
• It is also one of the earliest to be rigourously defined and studied

• Linear regression works by finding a ‘best fit’ line that minimizes the total difference between the values predicted by a resulting linear function (the line) and the actual predicted values
• There are multiple ways to perform linear regression, but the most common is the method of ‘least squares’

• Predictor variable (or independent variable): The input variable(s) used to predict the target variable
• Target variable (or dependent variable): The output variable that is being predicted
• Coefficient: The parameters (b0 and b1) that define the linear relationship between the predictor and target variables
• Residual: The difference between the predicted value and the actual value for a given data point
• Ordinary Least Squares (OLS): The most common method used to estimate the coefficients, which minimizes the sum of the squared residuals

• The predictive line is defined by the equation: \(y = \beta_0 + \beta_1x\)
• Where:
  • \(y\) is the predicted value of the target variable
  • \(x\) is the value of the predictor variable
  • \(\beta_0\) is the y-intercept (the value of \(y\) when \(x = 0\))
  • \(\beta_1\) is the slope of the line (the change in \(y\) for a unit change in \(x\))

Calculating the predictive line using least squares involves the following steps:

1. Calculate the mean of the predictor and target variables
2. Calculate the slope and intercept of the predictive line

The slope (\(\beta_1\)) and intercept (\(\beta_0\)) are calculated with the following:

\[ \beta_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2} \]

\[ \beta_0 = \bar{y} - \beta_1\bar{x} \]

Where: - \(\bar{x}\) and \(\bar{y}\) are the mean values of the independent and dependent variables respectively, - \(n\) is the number of observations.

The goal of the least squares method is to find the values of \(\beta_0\) and \(\beta_1\) that minimize the sum of the squared residuals:

\[ \text{Minimize} \sum_{i=1}^{n} (y_i - (\beta_0 + \beta_1x_i))^2 \]

This results in a line of ‘best fit’ that accurately represents the trend in the data (assuming it is a linear relationship). The function determining line allows predictions of the dependent variable (\(y\)) given a value of the independent variable (\(x\)).

Using built-in mtcars dataset.

Again using built-in mtcars dataset.

For plotting purposes, doing a naive ‘regression plane’ plot of MPG as linearly dependent on HP and WT.

Learned of “Anscombe’s quartet” while reading Wikipedia page on Linear Regression, which illustrates need to analyze data in multiple ways. The quarter shows 4 sets of data with similar derived statistical characteristics (linear regression line in this case) but also shows that while the regression may be technically correct it may not actually describe the data relationship accurately.

library(ggplot2)
data <- data.frame(
  x = c(1, 2, 3, 4),
  y = c(2, 3, 5, 4)
)
# Linear Model
model <- lm(y ~ x, data = data)

# Predicted data points along the predictive line
data$predicted <- predict(model, newdata = data)

R provides a very easy to use built in function lm, that takes in a formula and data and returns a linear model object.

The predict function is used to predict the values of the dependent variable given a new value of the independent variable.

• Linear regression is a foundational statistical method that enables the prediction of a continuous target variable based on one or more predictor variables.