Simple Linear Regression

A Fundamental Statistical Method

Introduction to Linear Regression

Linear regression is one of the most fundamental and widely used statistical methods.

Key Applications:

Linear regression forms the foundation for many advanced statistical techniques.

The Mathematical Model

Linear regression models the relationship between a dependent variable \(y\) and one or more independent variables \(X\).

For simple linear regression with one predictor:

\[y = \beta_0 + \beta_1 x + \varepsilon\]

Where: - \(y\) is the dependent variable - \(x\) is the independent variable - \(\beta_0\) is the y-intercept - \(\beta_1\) is the slope coefficient - \(\varepsilon\) is the error term (residual)

Estimating the Parameters

The method of Ordinary Least Squares (OLS) is used to estimate the parameters \(\beta_0\) and \(\beta_1\).

The goal is to minimize the sum of squared residuals:

\[\min_{\beta_0, \beta_1} \sum_{i=1}^{n} (y_i - \beta_0 - \beta_1 x_i)^2\]

The resulting estimators are:

\[\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}\]

\[\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}\]

Example Dataset: Cars

Let’s examine the relationship between a car’s speed and stopping distance using the built-in cars dataset.

# Load the cars dataset
data(cars)

# Display the first few rows
head(cars)
##   speed dist
## 1     4    2
## 2     4   10
## 3     7    4
## 4     7   22
## 5     8   16
## 6     9   10
# Summary statistics
summary(cars)
##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00

Visualizing the Relationship (ggplot2)

# Create a scatter plot with ggplot2
ggplot(cars, aes(x = speed, y = dist)) +
  geom_point(color = "#3498DB", size = 3, alpha = 0.7) +
  geom_smooth(method = "lm", color = "#E74C3C", fill = "#F9E79F", alpha = 0.3) +
  labs(
    title = "Relationship Between Speed and Stopping Distance",
    x = "Speed (mph)",
    y = "Stopping Distance (ft)"
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(face = "bold"),
    axis.title = element_text(face = "bold")
  )

Fitting the Linear Model

# Fit a linear model
car_model <- lm(dist ~ speed, data = cars)

# Display the model summary
summary(car_model)
## 
## Call:
## lm(formula = dist ~ speed, data = cars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -29.069  -9.525  -2.272   9.215  43.201 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -17.5791     6.7584  -2.601   0.0123 *  
## speed         3.9324     0.4155   9.464 1.49e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 15.38 on 48 degrees of freedom
## Multiple R-squared:  0.6511, Adjusted R-squared:  0.6438 
## F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

The equation of our fitted line is: \[\text{Distance} = -17.58 + 3.93 \times \text{Speed}\]

Interactive Visualization (plotly)

# Create a 3D visualization with plotly
# We'll add a synthetic variable to demonstrate 3D capabilities
set.seed(123)
cars_3d <- cars
cars_3d$weight <- cars_3d$speed * 100 + rnorm(nrow(cars_3d), mean = 0, sd = 200)

# Create the 3D plot
plot_ly(cars_3d, x = ~speed, y = ~weight, z = ~dist, 
        type = "scatter3d", mode = "markers",
        marker = list(size = 5, color = ~dist, colorscale = "Viridis", 
                      opacity = 0.8, showscale = TRUE)) %>%
  layout(
    title = "3D Visualization of Car Data",
    scene = list(
      xaxis = list(title = "Speed (mph)"),
      yaxis = list(title = "Weight (synthetic)"),
      zaxis = list(title = "Stopping Distance (ft)")
    )
  )

Residual Analysis (ggplot2)

# Create a data frame with fitted values and residuals
model_data <- data.frame(
  Speed = cars$speed,
  Fitted = fitted(car_model),
  Residuals = residuals(car_model)
)

# Create residual plot
ggplot(model_data, aes(x = Fitted, y = Residuals)) +
  geom_point(color = "#2ECC71", size = 3, alpha = 0.7) +
  geom_hline(yintercept = 0, linetype = "dashed", color = "#E74C3C", size = 1) +
  labs(
    title = "Residual Plot",
    x = "Fitted Values",
    y = "Residuals"
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(face = "bold"),
    axis.title = element_text(face = "bold")
  )

Interpretation and Conclusion

Key Findings:

Applications:

References and Resources

Further Reading:

Online Resources: