Simple Linear Regression is a tool in statistics that uses an independent variable (x) to predict a dependent variable (y) using a straight (linear) line.

This line is called the “best fit” line because it approximates the best (linear) fit to the data.

To find this best fit line, the least squares method is used. This method minimizes the difference between the line and the data.

Common examples include using an individuals height to predict weight, using a diamond’s carrat size to predict price, or using a vehicles weight to predict MPG’s.

Each of these examples relies on a correlation between the two variables. This will be explored further in the next slide containing the math of least squares.

\(\hat{y} = a + bx\)

where:

\(\hat{y}\) is the predicted value of y

\(a = \overline{y} - bx + \epsilon\)

\(b = \frac{\sum (x_i - \overline{x})(y_i - \overline{y})}{\sum (x_i - x)^2}\)

\(\overline{x} = \frac{\sum (x)}{n}\) - average of x

\(\overline{y} = \frac{\sum (y)}{n}\) - average of y

\(n\) = number of observations in dataset , \(\epsilon\) = error

## `geom_smooth()` using formula = 'y ~ x'

## (Intercept)       carat 
##   -2256.361    7756.426

The resulting equation of the model for the diamond dataset was \(\hat{y} = a + bx\) : substituting in our coefficients results in

\(\hat{y} = -2256.36 + 7756.43x\)

This means for every additional carat, the price goes up $7756.43 on average. There is a positive correlation between carat size and price.

The R code to produce the model and graph of the linear regression is below:

model <- lm(price ~ carat, data = diamonds)

ggplot(diamond, aes(x = carat, y = price)) + geom_point(alpha = 0.5) + # Add scatter plot geom_smooth(method = “lm”, col = “maroon”) + # Add regression line labs(title = “Linear Regression: Price vs. Carat”, x = “Carat”, y = “Price”)

coef(model)

## `geom_smooth()` using formula = 'y ~ x'
## `geom_smooth()` using formula = 'y ~ x'

## 
## Call:
## lm(formula = mpg ~ hp, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -5.7121 -2.1122 -0.8854  1.5819  8.2360 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
## hp          -0.06823    0.01012  -6.742 1.79e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.863 on 30 degrees of freedom
## Multiple R-squared:  0.6024, Adjusted R-squared:  0.5892 
## F-statistic: 45.46 on 1 and 30 DF,  p-value: 1.788e-07

## 
## Call:
## lm(formula = mpg ~ disp, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.8922 -2.2022 -0.9631  1.6272  7.2305 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 29.599855   1.229720  24.070  < 2e-16 ***
## disp        -0.041215   0.004712  -8.747 9.38e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.251 on 30 degrees of freedom
## Multiple R-squared:  0.7183, Adjusted R-squared:  0.709 
## F-statistic: 76.51 on 1 and 30 DF,  p-value: 9.38e-10

## (Intercept)          hp 
## 30.09886054 -0.06822828

## (Intercept)        disp 
## 29.59985476 -0.04121512

The equation of predicting MPG’s using Horsepower was

\(\hat{y} = 30.099 - 0.0682x\)

This means on average, for each additional horsepower, a car loses .0682MPG

The equation of predicting MPG’s using Displacement was

\(\hat{y} = 29.599 - 0.0412x\)

This means on average, for each additional unit of displacement, a car loses .0412MPG

In this case, MPG’s have a negative correlation with both Horsepower and Displacement

• Simple Linear Regression can be used on basic data sets to predict variables with high correlation. This is useful when exploring what, if any, relationship exists between variables.

• Confirming causation relationships. - If a SLR proves highly accurate in predicting the dependent variable, it is likely the independent in some way directly impacts the dependent variable. This direct impact is indicated by the high accuracy of the model.

• SLR can also be used in more complex data sets to predict data in areas where observations are missing. This allows for more accurate models, even though the data is incomplete. This is particularly useful in larger data sets, where observations are more likely to be missing.

• Assumes a linear relationship between the two variables. This can make it appear as though two variables are not related, when they are, but their relationship is not linear. If the two variables have an exponential, or other nonlinear relationship, the model will indicate a poor fit.

• Due to the simple nature of the linear regression, it is only possible to use two variables. This inherently limits the accuracy of the model, especially when compared to multiple regression. Multiple regression is more complicated, but allows multiple independent variables to be used in predicting the dependent variable. This would have proven especially useful in our MPG analysis. We would have been able to compare the accuracy of the two simple regression model to the multiple regression model.