What is simple linear regression?

Simple linear regression is a statistical analysis technique that involves one independent variable and one dependent variable. It takes paired data samples between these variables and finds a line of best fit to most accurately predict a dependent variable’s value for a given value of the independent variable. The simple linear regression model is of slope-intercept form:
\(y = mx + b\)
where y represents the dependent variable, x represents the independent variable, m represents the slope calculated from the provided data, and b represents the y-intercept calculated from the provided data.

How is a linear regression model calculated?

Simple linear regression models hinge upon the idea of residuals. The definition of the residual is the difference in an observed dependent variable value and the predicted value of the model for a particular value of the independent variable. For ith dependent value yi and associated xi (independent) value from the data set, the residual resulting from a test linear model with approximate slope m and intercept b is:
\(y_i = mx_i + b +\varepsilon_i \implies \varepsilon_i = y_i - mx_i - b\)
The optimal linear regression model involves particular values for the slope and intercept that are in some way calculated to minimize the sum of squared residuals. This provides a line that best approximates the true underlying relationship between the variables.

Least squares method

The least squares method provides equations to calculate a slope and intercept for a model that minimize the sum of squared residuals for all points in the dataset. For N data pairs,
\(m = \frac{N\Sigma(xy) - \Sigma x \Sigma y}{N\Sigma(x^2) - (\Sigma x)^2}\)

\(b = \frac{\Sigma y - m \Sigma x}{N}\)

Example data set

Tabulated data is shown below along with its visualization in Cartesian coordinates.

##    Independent Dependent
## 1            0      10.5
## 2            1       9.2
## 3            2       9.0
## 4            3       7.3
## 5            4       6.0
## 6            5       5.9
## 7            6       4.6
## 8            7       3.4
## 9            8       2.7
## 10           9       0.9
## 11          10       0.7

Example linear regression

Using the data set from the previous slide, a simple linear regression can be calculated using the formulas for m and b.

# Assign data to vectors for calculation
x <- example_data$Independent
y <- example_data$Dependent
N <- length(x)

# Initialize vectors to be calculated using for loops
xy <- numeric(length(x))
xsquared <- numeric(length(x))

# Calculate x*y for all data pairs
for (i in 1:length(x)) {
    xy[i] <- x[i] * y[i]
}

# Calculate x^2 for all x
for (i in 1:length(x)) {
    xsquared[i] <- x[i]^2
}

# Create sums for calculation
sigma_x <- sum(x)
sigma_y <- sum(y)
sigma_xy <- sum(xy)
sigma_xsquared <- sum(xsquared)

# Calculate m
m <- (N * sigma_xy - sigma_x * sigma_y)/(N * sigma_xsquared -
    sigma_x^2)
b <- (sigma_y - m * sigma_x)/N

# Create simple linear regression and append to
# example_data
y_slr <- m * x + b
example_data$SLR <- y_slr

# Plot simple linear regression over scatterplot
ggplot(example_data) + geom_point(aes(x = Independent, y = Dependent)) +
    geom_line(aes(x = Independent, y = SLR), col = "blue")

The simple linear regression model is:


\(y = -1.0027x + 10.4864\)

Extrapolation

Extrapolation is the act of using a regression model based on a finite dataset to make predictions about the dependent variable for values of the independent variable that exceed the range of the original dataset. Conclusions made on the basis of extrapolated values do not necessarily reflect the true underlying relationship between the variables. For example, consider the tangent function. If x is restricted to the range [-.5, .5], the relationship appears reasonably linear. However, if a simple linear regression is calculated for this interval and used to make predications outside of the sampled range, it becomes quickly apparent that the predictions would misrepresent the true relationship defined by the tangent function.

Assumptions of simple linear regression models

Simple linear regression models make the following assumptions.