Simulated Regression to Inspect Assumptions

library(faraway)

Error in library(faraway) : there is no package called ‘faraway’

The goal of this article is to present a brief overview of simple linear regression and its assumptions. We will be using simulated data to run properly sepcified regression, look how their diagnostic tests look like ie: assumptions are met. I then proceed to adjust the model such that one of the assumptions is no longer met and look at how that reflects on the diagnostics plots and overall regression result.

The general regression assumptions are subject to a lot of debate, for the purposes of this post, the regression assumptions we will be assuming and testing are the following:

• Linear relationship between Y and X
• Homoscedasticity of the error term
• Exogeneity of the error. Mean of the error term is zero

In the example below we simulate a regression between two variables perfectly linear i.e no error component

In the example below we simulate a regression between two variables with a noise component ie error

set.seed(1994)
obs <- 5000
x2 <- runif(obs, min=-20,max=20)
e1 <- rnorm(obs, mean=1,sd=10)
y2 <- 2*x2 + e1
plot(x2,y2)

model2 <- lm(y2~x2)
summary(model2)

Call:
lm(formula = y2 ~ x2)

Residuals:
    Min      1Q  Median      3Q     Max 
-36.324  -6.689   0.055   6.620  36.585 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.63324    0.14020   4.517 6.42e-06 ***
x2           1.99326    0.01206 165.238  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 9.909 on 4998 degrees of freedom
Multiple R-squared:  0.8453,    Adjusted R-squared:  0.8452 
F-statistic: 2.73e+04 on 1 and 4998 DF,  p-value: < 2.2e-16

plot(model2)

In the example below we simulate a mispecified regression. We achieve this by using a log function, with a normally distributed error component

set.seed(1994) #set seed for reproducible random number generation
obs <- 5000 #number of observatoins to generate 
x3 <- runif(obs, min=1,max=20) #Randomly generate 5000 obs from a uniform distribution
e1 <- rnorm(obs, mean=0,sd=0.2)
y3 <- log(x3) + e1 #create a quadratic function from the the data variable 
plot(x3,y3) #plot the graph of the relationship of Y with X

model3 <- lm(y3~x3) #run a linear regression of the quadratic function Y on the varible X
summary(model3) #Summary stats of previous regression model 

Call:
lm(formula = y3 ~ x3)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.42171 -0.17977  0.02925  0.21195  0.83003 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.8545813  0.0092315   92.57   <2e-16 ***
x3          0.1227817  0.0007882  155.77   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3075 on 4998 degrees of freedom
Multiple R-squared:  0.8292,    Adjusted R-squared:  0.8292 
F-statistic: 2.426e+04 on 1 and 4998 DF,  p-value: < 2.2e-16

plot(model3) #diagnostic plots for the regression 

In the example below, we simulate a regression with heteroskedasticity of the error term.

set.seed(1994)
obs <- 5000
x4 <- runif(obs, min=0,max=20)
e1 <- rnorm(obs, mean=1,sd=sqrt(x4)) 
y4 <- 2*x4 + e1
plot(x4,y4)

model4 <- lm(y4~x4)
summary(model4)

Call:
lm(formula = y4 ~ x4)

Residuals:
     Min       1Q   Median       3Q      Max 
-16.2464  -1.7825   0.0146   1.6955  14.9166 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.981376   0.086711   11.32   <2e-16 ***
x4          1.990039   0.007599  261.87   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.121 on 4998 degrees of freedom
Multiple R-squared:  0.9321,    Adjusted R-squared:  0.9321 
F-statistic: 6.858e+04 on 1 and 4998 DF,  p-value: < 2.2e-16

plot(model4)

The following example introduces collinearity between the variables x1 and x2 and x3. We simulated two variables x1 and x2 and we summed the two with a noise component to get x3.

vif(model1)

Error in vif(model1) : could not find function "vif"

Logistic Regression

References:

https://stats.stackexchange.com/questions/258485/simulate-linear-regression-with-heteroscedasticity How to simulate heteroscedasdity of the error term in R
https://aosmith.rbind.io/2018/08/29/getting-started-simulating-data/ How to generate random data from various distributions
https://www.r-bloggers.com/simulating-endogeneity/ Simluating endogeneity http://people.duke.edu/~rnau/testing.htm Regression assumptions https://math.stackexchange.com/questions/1530571/linear-regression-model-assumptions Regression Assumptions https://stats.stackexchange.com/questions/276831/assumptions-of-multiple-regression-how-is-normality-assumption-different-from-c/276888 Regression Asusmptions https://nwfsc-timeseries.github.io/atsa-labs/sec-tslab-random-walks-rw.html Simulating a Random Walk https://daviddalpiaz.github.io/appliedstats/collinearity.html simulating multicolinearity in R https://www.econometrics-with-r.org/6-4-ols-assumptions-in-multiple-regression.html simulating multicolinearity in R (imperfect multi colinearity)

https://stats.stackexchange.com/questions/46523/how-to-simulate-artificial-data-for-logistic-regression/46525 Simulating logistic regression data

https://daviddalpiaz.github.io/appliedstats/logistic-regression.html#binary-response

https://stats.stackexchange.com/questions/321770/simulating-data-for-an-ordered-logit-model simulate ordinal logistic regression