2025-10-16

What is Simple Linear Regression?

Simple linear regression is a statistical method that models the relationship between a dependent and an independent variable (X and Y) by fitting a straight line to the data. It helps us Y based on X.

\[ Y_i = B_0 + B_1X_i+ \epsilon_i \] where
\(B_0\) is the intercept
\(B_1\) is the slope and
\(\epsilon_i\) is the random error term (difference between actual and predicted values of Y)

Understanding the Parameters

\(B_0\) is the average value of Y when X is 0
\[ B_0 = \bar{Y} - B_1\bar{X} \] where
\(\bar{Y}\) is the mean of Y
\(B_1\) is the slope and
\(\bar{X}\) is the mean of X

\(B_1\) is the expected (average) change in Y
\[ B_1 = R(S_Y/S_X) \] where
\(R\) is the Pearson’s correlation coefficient between X and Y
\(S_Y\) is the standard deviation of Y and
\(S_X\) is the standard deviation of X

\(\epsilon_i\) is the error
\[ \epsilon_i = Y_i - \hat{Y}_i \] where
\(Y_i\) is the actual value of Y and
\(\hat{Y}_i\) is the predicted value of Y

Real Life Ex 1: Predicting Home Prices

Real estate analysts use house size to predict the future cost of a home. By plotting house size (i.e. square footage) against the cost, they can fit a line to see how much price changes with size.

Real Life Ex 2: Study Hours and Grades

Professors/teachers can use linear regression to see the correlation between amount of time spent studying and test scores. If students who spend more time studying tend to score higher, the regression line will show an upward slope.

Real Life Ex 3: Advertising and Sales

Business spend money on advertising in order to get more customers and increase sales. By running a linear regression between ad spending and revenue, they can estimate the expected rise in sales for for how much they spend. This way, they know how much to spend on their advertising budget.

mtcars

Let’s look at the mtcars dataset

head(mtcars, n = 10)
                   mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant           18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360        14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D         24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230          22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280          19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4

MPG vs Car Weight

Using simple linear regression, we can see the relationship between the weight of the car and the miles per gallon.

MPG vs Horsepower

This one shows the relationship between horsepower and the miles per gallon. We can use the same dataset to gain different insights.

MPG vs Car Weight vs Horsepower

This 3D model can show us the scatter plot with MPG, car weight, and horsepower. Because a simple linear regression is 2D, there is no regression line for this one.