Simple Linear Regression (SLR) is used to model the relationship between a single predictor variable \(X\) and a response variable \(Y\).

The population regression model is expressed as:

\[Y = \beta_0 + \beta_1 X + \epsilon\]

Where: - \(\beta_0\) is the y-intercept (the expected value of \(Y\) when \(X = 0\)). - \(\beta_1\) is the slope parameter (the change in \(Y\) associated with a one-unit increase in \(X\)). - \(\epsilon\) is the random error term, representing the deviation of an individual observation from the true regression line.

For SLR inferences to be valid, the error term \(\epsilon\) must satisfy several key assumptions:

• Linearity: The relationship between the predictor \(X\) and the mean of \(Y\) is linear.
• Independence: The errors are independent of each other.
• Normality: The errors are normally distributed for any given value of \(X\).
• Homoscedasticity: The errors have a constant variance (\(\sigma^2\)) across all values of \(X\).

Mathematically, we summarize these assumptions as: \[\epsilon_i \overset{i.i.d.}{\sim} N(0, \sigma^2)\]

We estimate the unknown population parameters \(\beta_0\) and \(\beta_1\) from sample data using the Ordinary Least Squares (OLS) method. OLS minimizes the Sum of Squared Residuals (SSR):

\[SSR = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (Y_i - (\hat{\beta}_0 + \hat{\beta}_1 X_i))^2\]

Minimizing this function yields the following closed-form estimators:

\[\hat{\beta}_1 = \frac{\sum_{i=1}^n (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^n (X_i - \bar{X})^2}\]

\[\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}\]

To demonstrate SLR, we simulate a dataset representing Advertising Budget (\(X\), in thousands of dollars) and Product Sales (\(Y\), in thousands of units). We assume true parameters \(\beta_0 = 5\) and \(\beta_1 = 2.5\).

set.seed(42)

# Simulate 100 observations
budget <- runif(100, min = 10, max = 100)
errors <- rnorm(100, mean = 0, sd = 15)
sales <- 5 + 2.5 * budget + errors

# Combine into a dataframe
data <- data.frame(Budget = budget, Sales = sales)
head(data, 5)

##     Budget    Sales
## 1 92.33254 240.6602
## 2 94.33679 229.0844
## 3 35.75256 118.0173
## 4 84.74029 226.4942
## 5 67.75710 175.7392

We can visualize the relationship and check the linear trend using the following ggplot2 code:

library(ggplot2)

ggplot(data, aes(x = Budget, y = Sales)) +
  geom_point(color = "#4A90E2", alpha = 0.7) +
  geom_smooth(method = "lm", color = "#8C1D40", 
              fill = "#8C1D40", alpha = 0.15) +
  theme_minimal() +
  labs(
    title = "Sales vs. Advertising Budget",
    x = "Advertising Budget ($ thousands)",
    y = "Sales (thousands of units)"
  )

We fit the linear model using R’s lm() function and check the estimated coefficients.

model <- lm(Sales ~ Budget, data = data)
summary(model)$coefficients

##              Estimate Std. Error    t value     Pr(>|t|)
## (Intercept) 0.4666771   3.248809  0.1436456 8.860753e-01
## Budget      2.5726220   0.051347 50.1026695 1.208113e-71

summary(model)$r.squared

## [1] 0.9624273

The estimated intercept is \(\hat{\beta}_0 \approx 0.47\) (true value is 5) and the estimated slope is \(\hat{\beta}_1 \approx 2.57\) (true value is 2.5). Every $1k increase in budget is associated with an estimated increase of \(2.57\) units of sales.

To verify homoscedasticity and linearity, we plot the residuals (\(e_i = Y_i - \hat{Y}_i\)) against the fitted values (\(\hat{Y}_i\)).

Using the fitted model, we construct a 95% prediction interval for the sales of a new advertising budget (\(X_h\)).

Formulas: \[\text{Confidence Interval: } \hat{Y}_h \pm t_{\alpha/2, n-2} \cdot s\{\hat{Y}_h\}\] \[\text{Prediction Interval: } \hat{Y}_h \pm t_{\alpha/2, n-2} \cdot s\{\text{pred}\}\]

Prediction Output in R:

new_obs <- data.frame(Budget = c(50, 75))
predict(model, newdata = new_obs, interval = "prediction", level = 0.95)

##        fit      lwr      upr
## 1 129.0978 101.3988 156.7967
## 2 193.4133 165.6648 221.1618