Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables.

🔹 Simple Linear Regression

Uses one independent variable to predict the dependent variable.
Example: Predicting house price based on square footage.

🔹 Multiple Linear Regression

Uses multiple independent variables to predict the dependent variable.
Example: Predicting house price based on square footage, number of rooms, and location.

The regression line is written as:

\[ y = a + bx \]

🔹 Coefficient Formulas

• Slope (b): \[ b = r \left( \frac{S_y}{S_x} \right) \]

• Intercept (a): \[ a = \bar{y} - b\bar{x} \]

Definitions

• \(r\): Pearson correlation coefficient
  
• \(S_y\), \(S_x\): Standard deviations of \(y\) and \(x\)
  
• \(\bar{y}\), \(\bar{x}\): Sample means of \(y\) and \(x\)
  
• \(a\): Intercept of the line
  
• \(b\): Slope of the line

The Pearson correlation coefficient \(r\) measures the strength and direction of a linear relationship between two variables.

Formula

\[ r = \frac{ \sum (x_i - \bar{x})(y_i - \bar{y}) } { \sqrt{ \sum (x_i - \bar{x})^2 } \cdot \sqrt{ \sum (y_i - \bar{y})^2 } } \]

Interpretation of \(r\)

• \(r > 0\): As \(x\) increases, \(y\) tends to increase
• \(r < 0\): As \(x\) increases, \(y\) tends to decrease

Examining the relationship between car weight and miles per gallon (MPG) using the built-in mtcars dataset.

# Load data
data(mtcars)
# View weight and mpg
head(mtcars[, c("wt", "mpg")], 3)

##                  wt  mpg
## Mazda RX4     2.620 21.0
## Mazda RX4 Wag 2.875 21.0
## Datsun 710    2.320 22.8

\[ r = \frac{ \sum (x_i - \bar{x})(y_i - \bar{y}) } { \sqrt{ \sum (x_i - \bar{x})^2 } \cdot \sqrt{ \sum (y_i - \bar{y})^2 } } \]

First, let’s find the value of the numerator:

\[ \sum (x_i - \bar{x})(y_i - \bar{y}) \]

# Calculate means
xbar <- mean(mtcars$wt)
ybar <- mean(mtcars$mpg)
# Calculate deviations
x_dev <- mtcars$wt - xbar
y_dev <- mtcars$mpg - ybar
# Calculate the summation of the product of deviations
numerator <- sum(x_dev * y_dev)
numerator

## [1] -158.6172

Next, find the value of the denominator:

\[ \sqrt{ \sum (x_i - \bar{x})^2 } \cdot \sqrt{ \sum (y_i - \bar{y})^2 } \]

# Calculate sum of squared deviations for x and y
sum_sq_x <- sum(x_dev^2)
sum_sq_y <- sum(y_dev^2)

# Calculate denominator
denominator <- sqrt(sum_sq_x) * sqrt(sum_sq_y)
denominator

## [1] 182.8105

Calculating the Pearson Correlation Coefficient \(r\)

Recall the formula: \[ r = \frac{ \sum (x_i - \bar{x})(y_i - \bar{y}) } { \sqrt{ \sum (x_i - \bar{x})^2 } \times \sqrt{ \sum (y_i - \bar{y})^2 } } \]

Now, compute \(r\) by dividing the numerator by the denominator:

# Assuming numerator and denominator are calculated as:
# numerator <- sum(x_dev * y_dev)
# denominator <- sqrt(sum(x_dev^2)) * sqrt(sum(y_dev^2))
r <- numerator / denominator
r

## [1] -0.8676594

We use:

• \(b = r \cdot \frac{s_y}{s_x}\) , \(a = \bar{y} - b \cdot \bar{x}\)

s_x <- sd(mtcars$wt)
s_y <- sd(mtcars$mpg)

b <- r * (s_y / s_x)
a <- ybar - b * xbar

a  # Intercept

## [1] 37.28513

b  # Slope

## [1] -5.344472

From our calculations:

## [1] "ŷ =  37.29  +  -5.34 x"

Preview