Linear regression is a statistical method used to model the relationship between:

• Dependent variable (Y): The outcome we want to predict
• Independent variable (X): The predictor variable

The goal is to find the best-fitting straight line through the data points that minimizes the sum of squared residuals.

Applications:

• Predicting house prices based on square footage
• Forecasting sales based on advertising spend
• Estimating exam scores based on study hours

The simple linear regression model is expressed as:

\[Y_i = \beta_0 + \beta_1 X_i + \epsilon_i\]

where:

• \(Y_i\) = observed value of the dependent variable
• \(\beta_0\) = y-intercept (population parameter)
• \(\beta_1\) = slope (population parameter)
• \(X_i\) = value of the independent variable
• \(\epsilon_i\) = random error term, where \(\epsilon_i \sim N(0, \sigma^2)\)

The least squares estimates of the parameters are:

\[\hat{\beta}_1 = \frac{\sum_{i=1}^{n}(X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^{n}(X_i - \bar{X})^2}\]

\[\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\bar{X}\]

The fitted regression line is:

\[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i\]

Let’s examine the relationship between study hours and exam scores for a group of students.

Dataset preview:

##   study_hours exam_scores
## 1           2    54.19762
## 2           3    59.34911
## 3           4    71.79354
## 4           5    67.85254
## 5           6    71.64644

# Fit linear regression model
model <- lm(exam_scores ~ study_hours, data = study_data)

# Display model summary
summary(model)

## 
## Call:
## lm(formula = exam_scores ~ study_hours, data = study_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -7.0872 -2.8848 -0.3694  1.5616  7.6441 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  51.5237     2.6064   19.77 4.40e-11 ***
## study_hours   3.4154     0.2611   13.08 7.37e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 4.369 on 13 degrees of freedom
## Multiple R-squared:  0.9294, Adjusted R-squared:  0.924 
## F-statistic: 171.1 on 1 and 13 DF,  p-value: 7.372e-09

The line shows the predicted relationship: more study hours lead to higher exam scores.

Residuals should be randomly scattered around zero with no clear pattern.

Key metrics for assessing model fit:

R-squared (\(R^2\)): Proportion of variance explained by the model

\[R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum(Y_i - \hat{Y}_i)^2}{\sum(Y_i - \bar{Y})^2}\]

• Ranges from 0 to 1
• Higher values indicate better fit

## R-squared: 0.9294

## Adjusted R-squared: 0.9240

## RMSE: 4.0670

We test whether there’s a significant linear relationship:

\[H_0: \beta_1 = 0 \text{ (no relationship)}\] \[H_A: \beta_1 \neq 0 \text{ (relationship exists)}\]

Test statistic:

\[t = \frac{\hat{\beta}_1 - 0}{SE(\hat{\beta}_1)}\]

Results for our example:

## Slope estimate: 3.4154

## Standard error: 0.2611

## t-value: 13.0819

## p-value: 0.000000

Key Takeaways:

• Simple linear regression models the linear relationship between two variables
• The least squares method provides optimal parameter estimates
• Model diagnostics (residual plots, \(R^2\)) help assess model adequacy
• Hypothesis testing determines statistical significance

Limitations:

• Assumes linear relationship
• Sensitive to outliers
• Requires independence of observations

Next Steps: Explore multiple regression, polynomial regression, and other advanced modeling techniques.