2024-04-07

Introduction to Simple Linear Regression

  • Purpose: To examine the relationship between two continuous variables

  • Definition: A statistical method to model the linear relationship between a dependent variable (Y) and an independent variable (X).

The Simple Linear Regression Model

Formula: \[Y_{i} = \beta_{0} + \beta_{1} X_{i} + \epsilon_{i}\]

Terms explained:

  • \(Y_i\) is the dependent variable
  • \(X_i\) is the independent variable
  • \(\beta_0\) is the intercept
  • \(\beta_1\) is the slope
  • \(\epsilon_i\) is the error term

Assumptions of Linear Regression

Linearity:

The relationship between the dependent and independent variable should be linear.

Independence:

Observations (data points) should be independent of each other.

Homoscedasticity:

The variance of the errors (residuals) should be constant across different values of the independent variable.

Normality of Errors:

The errors (residuals) should follow a normal distribution.

Example Dataset (mtcars)

Shows scatter plot relationship between MPG and Weight of a vehicle. The linear nature allows the use of linear regression models.

Model Fitting in R

A low P-value shows a significant relationship between weight and mpg.

Call:
lm(formula = mpg ~ wt, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.5432 -2.3647 -0.1252  1.4096  6.8727 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
wt           -5.3445     0.5591  -9.559 1.29e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3.046 on 30 degrees of freedom
Multiple R-squared:  0.7528,    Adjusted R-squared:  0.7446 
F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

Interpreting Results

A linear regression line graphed onto the previous scatter plot.

3D Plot using Weight and HP as variables

For fun, added a 3d plot using weight and horse power compared to mpg, then graphed a linear regression plane to show significance.

Applications and Conclusion

Applications:

Prediction: Predict future data, given a model trained on a similar linear datasets.

Understanding causal factors: While regression doesn’t prove strict causation, it isolates the effect of a particular variable.

Forecasting: If the model includes a time-dependent variable, it could be used for forecasting trends.

Conclusion:

Simple linear regression is a building block for more complex statistical modeling techniques and has a wide application to many fields of study.