I.

1. State the 5 assumptions

The Gauss-Markov theorem outlines conditions under which ordinary least squares (OLS) regression estimates are the best linear unbiased estimators (BLUE). The five key assumptions are linearity, randomness in sampling, non-collinearity among explanatory variables, exogeneity (zero conditional mean), and homoscedasticity. These assumptions ensure the reliability and accuracy of OLS estimates without bias and with minimum variance among all linear estimators.

2. Non-technical Explanation

Linearity is about the relationship between what we are studying (like hours studied) and the outcome (grades) being straight-line. Adding a fixed amount of study time always improves the grade by the same amount.

Random means everyone has a fair chance to be chosen, which helps make sure our findings apply to everyone, not just a specific group.

For Non-Collinearity, imagine if we were trying to figure out what makes a vehicle go faster and we looked at both the engine size and the horsepower, but they’re basically telling you the same thing. We want each thing we measure (like engine size, weight, color) to tell us something new and unique, not just repeat what we learned from something else.

Exogeneity (Zero conditional mean) assumes there’s no secret, hidden factor that messes up our findings. If you’re studying whether studying more leads to better grades, there shouldn’t be something else, like a student’s natural intelligence, that affects both how much they study and their grades, without us knowing.

Homoscedasticity: Think of this as expecting your popcorn to pop in a uniform way, regardless of whether you’re making a small or large batch. In our studies, it means the difference between what we predict and what we observe should be consistent, no matter if we’re looking at small or big examples.

3.

Linearity ensures that the relationship between independent variables and the dependent variable can be accurately modeled with a straight line in a multidimensional space. This linearity is crucial for the mathematical formulation of OLS, facilitating interpretation and computation.

Random underpins the statistical inference process, ensuring that our sample estimates can be generalized to the population. This randomness is fundamental to establishing the distributional properties required for hypothesis testing and confidence intervals.

Non-Collinearity prevents the issue of multicollinearity, where independent variables are highly correlated, making it difficult to isolate their individual effects on the dependent variable. High correlation among variables can inflate the variance of coefficient estimates, undermining the statistical significance and stability of the model.

Exogeneity ensures that the error term is uncorrelated with the independent variables, a critical condition for unbiasedness of OLS estimates. Violations of exogeneity, such as omitted variable bias or simultaneity, lead to endogeneity, contaminating estimates and inferential statistics.

Homoscedasticity or equal variance of error terms, is fundamental for the efficiency of OLS estimates and the validity of standard errors, t-tests, and F-tests. Heteroscedasticity complicates error variance estimation and can lead to inefficient and potentially biased coefficient estimates, making corrective measures or alternative estimation techniques necessary.

II.

1.

In mtcars dataset, I am going to use mpg (miles per gallon) as the dependent variable and hp (horsepower) as the independent variable. This analysis aims to understand how a car’s fuel efficiency (measured in mpg) is influenced by its horsepower.

data(mtcars)
my_reg <- lm(mpg ~ hp, data = mtcars)

# Display the regression summary
summary(my_reg)
## 
## Call:
## lm(formula = mpg ~ hp, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -5.7121 -2.1122 -0.8854  1.5819  8.2360 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
## hp          -0.06823    0.01012  -6.742 1.79e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.863 on 30 degrees of freedom
## Multiple R-squared:  0.6024, Adjusted R-squared:  0.5892 
## F-statistic: 45.46 on 1 and 30 DF,  p-value: 1.788e-07

  • Hence, the equation is:

  • \[ \text{mpg} = 30.0989 + -0.06823 \times \text{hp} + \epsilon \]

Intercept: The estimated intercept is 30.0989, with a standard error of 1.634. This suggests that, theoretically, a car with 0 horsepower would have an estimated fuel efficiency of 30.0989 miles per gallon. This intercept is statistically significant (p < 0.0001), indicating a high level of confidence in this estimate.

Slope: The estimated slope coefficient for horsepower (hp) is -0.0682 with a standard error of 0.010, indicating that for each additional unit of horsepower, the fuel efficiency decreases by 0.0682 miles per gallon. This coefficient is also statistically significant (p < 0.0001), suggesting a strong inverse relationship between horsepower and fuel efficiency.

Statistical Significance: Both the intercept and the slope coefficient are statistically significant at a very low alpha level (well below the conventional 0.05 threshold), indicating strong evidence against the null hypothesis (which would suggest no effect).

Economic Magnitude: The magnitude of the slope coefficient (-0.0682) suggests that increases in horsepower are associated with relatively small decreases in miles per gallon, on average. While statistically significant, the economic or practical significance of this finding depends on the context—such as the range of horsepower across cars analyzed and the priorities of consumers or manufacturers. For example, a 100-horsepower increase in a car’s engine power would be associated with a decrease in fuel efficiency of approximately 6.82 miles per gallon, which could be considered substantial depending on fuel economy standards and consumer preferences.

R-squared: The R-squared value is 0.602, indicating that approximately 60.2% of the variance in miles per gallon can be explained by the car’s horsepower. This suggests a strong model fit, though other factors not included in this model also influence fuel efficiency.

III.

# Display diagnostic plots
plot(my_reg)

# Transforming the independent variable (hp) with a log transformation as an example
my_reg_log <- lm(mpg ~ log(hp), data = mtcars)
summary(my_reg_log)
## 
## Call:
## lm(formula = mpg ~ log(hp), data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.9427 -1.7053 -0.4931  1.7194  8.6460 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   72.640      6.004  12.098 4.55e-13 ***
## log(hp)      -10.764      1.224  -8.792 8.39e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.239 on 30 degrees of freedom
## Multiple R-squared:  0.7204, Adjusted R-squared:  0.7111 
## F-statistic:  77.3 on 1 and 30 DF,  p-value: 8.387e-10
# Checking if the model fit improves with transformation
plot(my_reg_log)

  1. Residuals vs Fitted: This plot helps to check the homoscedasticity and linearity assumptions. Ideally, you want to see a random scatter of points without any discernible pattern. A funnel shape or a clear pattern suggests non-constant variance (heteroscedasticity) or non-linearity.

    The plot with the log transformation of horsepower indicates a more random distribution of residuals and fewer signs of non-linear relationships.

  2. Normal Q-Q: This quantile-quantile plot compares the distribution of residuals to a normal distribution. If the residuals are normally distributed, the points should closely follow the 45-degree line. Deviations from this line indicate departures from normality.

  3. Scale-Location (or Spread-Location): Similar to the first, this plot shows residuals spread against fitted values but focuses on checking the homoscedasticity of residuals. A horizontal line with equally (randomly) spread points around it indicates homoscedasticity.

    The improvement is subtle, and further measures, like weighted least squares or a different transformation, may be necessary to fully address the issue. The log transformation does seem to reduce the heteroscedasticity problem to an extent but may not completely resolve it.

  4. Residuals vs Leverage: This plot helps identify influential data points that have a disproportionate impact on the model’s parameters. Points far from the center (high leverage) or with large residuals (outliers) can unduly influence the model.

Trends/Rules of Thumb for Analyzing Model Fit:

  • No clear patterns or systematic structures in the Residuals vs Fitted plot.

  • Points closely following the reference line in the Normal Q-Q plot.

  • A horizontal, even spread of points in the Scale-Location plot.

  • No points with high leverage and large residuals in the Residuals vs Leverage plot.