2024-05-31

Introduction

Simple linear regression is a statistical method that allows us to summarize and study relationships between two continuous (quantitative) variables. It is one of the most widely used techniques in data analysis and predictive modeling.

Importance and Uses

Predicting outcomes (e.g., sales, growth rates)

Understanding relationships between variables (e.g., height and weight)

Evaluating trends over time (e.g., stock prices, temperature changes)

Theory

Simple linear regression is a statistical method used to model the relationship between two continuous variables. One variable, typically denoted by x, is considered the independent variable (predictor or explanatory variable). The other variable, denoted by y, is the dependent variable (response or outcome variable).

Mathematical Model

The simple linear regression model takes the form: \[ y = \beta_0 + \beta_1 x + \epsilon \]

Components

\(y\): Dependent variable (response)

\(x\): Independent variable (predictor)

\(\beta_0\): Y-intercept

\(\beta_1\): Slope of the regression line

\(\epsilon\): Error term (residual)

Assumptions

To ensure the validity of the regression model, the following assumptions should be checked: - Linearity: The relationship between \(x\) and \(y\) is linear. - Independence: Observations are independent of each other. - Homoscedasticity: The variance of residuals is constant. - Normality: Residuals are normally distributed.

Example Data

Generate and visualize example data:

set.seed(123)
x <- rnorm(100)
y <- 2*x + rnorm(100)
data <- data.frame(x, y)
head(data)
##             x          y
## 1 -0.56047565 -1.8313579
## 2 -0.23017749 -0.2034713
## 3  1.55870831  2.8707247
## 4  0.07050839 -0.2065258
## 5  0.12928774 -0.6930431
## 6  1.71506499  3.3851022

Scatter Plot with Regression Line

Visualize the relationship between the predictor and response variables, and add a regression line:

library(ggplot2)
ggplot(data, aes(x=x, y=y)) +
  geom_point() +
  geom_smooth(method="lm", col="pink")
## `geom_smooth()` using formula = 'y ~ x'

Regression Analysis

Fit the linear model and display the summary:

model <- lm(y ~ x, data=data)
summary(model)
## 
## Call:
## lm(formula = y ~ x, data = data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1.9073 -0.6835 -0.0875  0.5806  3.2904 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -0.10280    0.09755  -1.054    0.295    
## x            1.94753    0.10688  18.222   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9707 on 98 degrees of freedom
## Multiple R-squared:  0.7721, Adjusted R-squared:  0.7698 
## F-statistic:   332 on 1 and 98 DF,  p-value: < 2.2e-16

Interpretation of Coefficients

Intercept (\(\beta_0\)): The expected value of \(y\) when \(x\) is 0.

Slope (\(\beta_1\)): The change in \(y\) for a one-unit change in \(x\).

Real-World Examples

Predicting salary based on years of experience

Estimating house prices based on square footage

Goodness-of-Fit

The \(R^2\) statistic provides a measure of how well the regression model fits the data:

\[ R^2 = \frac{SS_{regression}}{SS_{total}} \]

An \(R^2\) value close to 1 indicates a good fit.

Residual Plot

Plot residuals vs. fitted values to check model assumptions:

data$.fitted <- model$fitted.values
data$.resid <- model$residuals

ggplot(data, aes(x=.fitted, y=.resid)) +
  geom_point() +
  geom_hline(yintercept=0, linetype="dashed", col="blue") +
  labs(x="Fitted values", y="Residuals")

Diagnostic Plots

Introduce other diagnostic plots like the Q-Q plot for normality check:

# Necessary libraries
library(ggplot2)

data(mtcars)
model <- lm(mpg ~ wt, data = mtcars)
mtcars$resid <- residuals(model)

# Create Q-Q plot
ggplot(mtcars, aes(sample = resid)) +
  stat_qq() +
  stat_qq_line() +
  labs(title = "Q-Q Plot of Residuals",
       x = "Theoretical Quantiles",
       y = "Sample Quantiles")

  • Residuals vs Fitted: Checks linearity and homoscedasticity.
  • Q-Q Plot: Checks normality of residuals.

3D Plot

Provide a 3D scatter plot using plotly:

library(plotly)
## 
## Attaching package: 'plotly'
## The following object is masked from 'package:ggplot2':
## 
##     last_plot
## The following object is masked from 'package:stats':
## 
##     filter
## The following object is masked from 'package:graphics':
## 
##     layout
plot_ly(data, x=~x, y=~y, z=~data$.resid, type="scatter3d", mode="markers") %>%
  layout(title="3D Scatter Plot of Residuals", scene = list(
    xaxis = list(title = "Predictor (x)"),
    yaxis = list(title = "Response (y)"),
    zaxis = list(title = "Residuals")))

Applications

Simple linear regression is widely used in various fields, such as:

Economics: To predict consumer spending.

Biology: To understand the relationship between dosage and effect.

Engineering: To predict material strength based on composition.

Specific Examples

Estimating fuel efficiency based on engine size

Predicting crop yield based on rainfall

Conclusion

Simple linear regression is a powerful tool for modeling relationships between two continuous variables.

It helps in making predictions and understanding the impact of one variable on another.

Bibliography

Bevans, Rebecca. “Simple Linear Regression | an Easy Introduction & Examples.” Scribbr, 1 June 2023, www.scribbr.com/statistics/simple-linear-regression.