March 16, 2024

What is Linear Regression?

  • Models the relationship between a dependent variable and one/more independent variables

  • Enables us to make predictions on how changes in the independent variable(s) affect the dependent variable

  • Simple Linear Regression Equation: \[ y = \beta_0 + \beta_1x + \varepsilon \]

    • \(y\): Dependent Variable
    • \(x\): Independent Variable
    • \((\beta_0)\): Intercept (Constant)
    • \((\beta_1)\): Coefficient (Slope)
    • \((\varepsilon)\): Error Term (Implicit - Not used in prediction)
  • Multiple Linear Regression Equation: \[ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \ldots + \beta_nx_n + \varepsilon \]

    • \(x_1, x_2, \ldots, x_n\): Independent Variables

    • \((\beta_0, \beta_1, \ldots, \beta_n)\): Coefficients

    • \((\varepsilon)\): Error Term (Implicit - Not used in prediction)

      -It represents the variability/noise between the observed values and the values predicted by linear regression

Example 1

  • Let’s use linear regression to predict the relationship between study hours and exam scores
  • Using the model summary below, the linear regression equation is: \[Score = 37.0667 + 5.6970*(Hours)\]
hours <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
exam_scores <- c(43, 49, 54, 60, 64, 72, 75, 85, 88, 94)
dataset <- data.frame(hours, exam_scores)
model <- lm(exam_scores ~ hours, data = dataset)
summary(model)
## 
## Call:
## lm(formula = exam_scores ~ hours, data = dataset)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.94545 -0.29394  0.05455  0.46364  2.35758 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.0667     0.8649   42.86 9.69e-11 ***
## hours         5.6970     0.1394   40.87 1.41e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.266 on 8 degrees of freedom
## Multiple R-squared:  0.9952, Adjusted R-squared:  0.9946 
## F-statistic:  1670 on 1 and 8 DF,  p-value: 1.414e-10

Example (Plotted)

ggplot(dataset, aes(x = hours, y = exam_scores)) +
  geom_point() +                              
  geom_smooth(method = "lm", se = FALSE) +
  labs(title = "Linear Regression",        
       x = "Hours", y = "Exam Scores")
`geom_smooth()` using formula = 'y ~ x'

Example 2

  • This examples also uses simple linear regression to predict a dependent variable (mpg) based on the independent variable of weight (wt)
  • Using the model summary below, the linear regression equation is: \[MPG = 37.22727 - 3.87783*WT\]
data(mtcars)
model <- lm(mpg ~ wt, data = mtcars)
summary(model)
## 
## Call:
## lm(formula = mpg ~ wt, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528, Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

Example 2 (Plotted)

  • The graph adds an extra dimension of horsepower (hp) to show how higher horsepower cars are associated with lower MPG and higher weight
## `geom_smooth()` using formula = 'y ~ x'

Example 3

  • This example uses multiple linear regression to predict home prices based on the number of bedrooms and square footage (2 independent variables).
  • Using the model summary below the multiple linear regression equation is: \[Price = 6666.7 + 24703.7*(bedrooms) + 95.56*(sqft)\]
houses <- data.frame(
  bedrooms = c(2, 3, 4, 2, 3, 4, 3, 4, 5),
  size_sqft = c(1500, 1800, 2000, 1200, 1600, 1900, 1700, 2000, 2300),
  price = c(200000, 250000, 300000, 180000, 220000, 280000, 240000, 310000, 350000)
  )

model <- lm(price ~ bedrooms + size_sqft, data = houses)
summary(model)
## 
## Call:
## lm(formula = price ~ bedrooms + size_sqft, data = houses)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -13667  -3222     37   3407  13407 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept)  6666.67   28244.96   0.236   0.8213  
## bedrooms    24703.70   11110.45   2.223   0.0679 .
## size_sqft      95.56      34.38   2.780   0.0320 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 9415 on 6 degrees of freedom
## Multiple R-squared:  0.9785, Adjusted R-squared:  0.9713 
## F-statistic: 136.3 on 2 and 6 DF,  p-value: 9.997e-06

Example 3 (Plotted)

  • We can plot the multiple linear regression equation using a 3d plot
## A line object has been specified, but lines is not in the mode
## Adding lines to the mode...

\(R^2\) in Linear Regression

  • In all 3 examples, the summary also mentions a \(R^2\) value which quantifies the model fit
  • It is the proportion of variance in the dependent variable that can be explained by the independent variable
  • R-squared ranges from 0 to 1, where 0 indicates that the model does not explain any variability in the dependent variable, and 1 indicates that the model explains all of the variability.
  • High \(R^2\) indicates that the regression model is a good fit
    • Example 1: \(R^2\) -> 0.9952; The independent variable explains 99.52% of the variability in the dependent variable, showing strong predictive capability

Conclusion: Linear Regression

  • Linear regression offers a powerful framework for modeling relationships between variables.
  • Through metrics like \(R^2\), it provides insights into the goodness-of-fit and predictive performance of the model.
  • It is widely applicable across diverse fields, linear regression serves as an important tool for data analysis and decision-making.