Simple Linear Regression

A statistical method that allows us to determine how one variable of interest is affected by changes in another variable.

It is defined as: \(y = \beta_0 + \beta_1\cdot x\)

  • \(y\) : dependent variable
  • \(x\) : independent variable
  • \(\beta_0\) : y-intercept
  • \(\beta_1\) : slope

Orange data set

Here we will use RStudio’s built-in Orange data set. Let’s see if there is a correlation between a tree’s age and circumference.

data("Orange"); head(Orange)
##   Tree  age circumference
## 1    1  118            30
## 2    1  484            58
## 3    1  664            87
## 4    1 1004           115
## 5    1 1231           120
## 6    1 1372           142

Tree: ID of measured tree

age: age of the tree in days since 1968/12/31

circumference: tree trunk measurement in mm

Scatter Plot (plotly)

Let’s create a scatter plot of age vs circumference.

xax <- list(title = "age (days)")
yax <- list(title = "circumference (mm)")
plot_ly(x=Orange$age, y=Orange$circumference, type="scatter", 
        mode="markers", color=Orange$Tree) %>%
  layout(xaxis=xax, yaxis=yax, title="Age vs Circumference")

Scatter Plot (plotly)

Here is a scatter plot of age vs circumference. There appears to be a positive correlation between the two variables.

Finding Regression Line

To find the regression line, also called “line of best fit”, the following process can be followed:

given a set of n points \((X_i ,Y_i)\) on a scatterplot,

find the best-fit line, \(\hat{Y_i} = \beta_0 + \beta_1\cdot X_i\)

such that the sum of squared errors in \(Y\), \(\sum{(Y_i - \hat{Y_i})^2}\) is minimized

Regression Line Code

In RStudio, we can utilize the ggplot2 library to create our graph with a regression line.

ggplot(data = Orange, aes(x = age, y = circumference)) + 
  geom_point() +
  geom_smooth(method="lm")

Our fitted-line will be made according to method="lm" (linear model).

Regression Line Graph

Here is our line of best fit:

Line of Best Fit

To get our line’s coefficients we can use lm().

#fit linear regression model
lm(circumference ~ age, data=Orange)
## 
## Call:
## lm(formula = circumference ~ age, data = Orange)
## 
## Coefficients:
## (Intercept)          age  
##     17.3997       0.1068

Regression Model: \(y = \beta_0 + \beta_1\cdot x\)

Orange data: \(y = 17.3997 + 0.1068\cdot x\)

Coefficent of Determination: R^2

How can we tell if there is a strong correlation between our variables? \(R^2\) is the fraction of variability that can be accounted for by the regression model.

\(R^2 = \frac{SSR}{SST}\)

\(SSR\) : Sum of Squared Residuals

\(SST\) : Total Sum of Squares

An \(R^2 > 0.7\) is generally considered a strong correlation.

Coefficent of Determination: R^2

Here we will get \(R^2\) from the summary() function, which takes in our linear model lm().

model = lm(circumference ~ age, data=Orange)
summary(model)$r.squared
## [1] 0.8345167

Utilizing the summary() method we are able to quickly see that our \(R^2\) value is 0.8345. Which indicates a strong correlation between our variables age and circumference.

Application Example

Let’s say we wanted to find which tree’s circumference was growing at the fastest rate.

Fastest Growing Tree

We can draw regression lines for each tree:

Tree 4 looks to have the steepest slope.

Fastest Growing Tree

To numerically verify, here are the slopes, \(\beta_1\), of each Tree:

  • Tree 1: \(0.08148\)
  • Tree 2: \(0.1251\)
  • Tree 3: \(0.08111\)
  • Tree 4: \(0.1352\)
  • Tree 5: \(0.111\)

Tree 4 has the largest value and therefore was the fastest growing tree!

Conclusion

Simple linear regression is used for three main purposes:

  1. To describe the linear dependence of one variable on another

  2. To predict values of one variable from values of another, for which more data are available

  3. To correct for the linear dependence of one variable on another, in order to clarify other features of its variability.