Multivariate Regression

Load Data

A data frame with 32 observations on 11 (numeric) variables.

[, 1]	mpg	Miles/(US) gallon
[, 2]	cyl	Number of cylinders
[, 3]	disp	Displacement (cu.in.)
[, 4]	hp	Gross horsepower
[, 5]	drat	Rear axle ratio
[, 6]	wt	Weight (1000 lbs)
[, 7]	qsec	1/4 mile time
[, 8]	vs	Engine (0 = V-shaped, 1 = straight)
[, 9]	am	Transmission (0 = automatic, 1 = manual)
[,10]	gear	Number of forward gears
[,11]	carb	Number of carburetors

remove(list=ls())
df <- mtcars
?mtcars
library(dplyr)

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

glimpse(mtcars)

Rows: 32
Columns: 11
$ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19.2, 17.8,…
$ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, 4, 4, 8,…
$ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 140.8, 16…
$ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, 180, 180…
$ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.92, 3.92,…
$ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3.150, 3.…
$ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 22.90, 18…
$ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0,…
$ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0,…
$ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, 4, 3, 3,…
$ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, 1, 1, 2,…

Run MULTIVARIATE Regression

reg1<- 
lm(data = df, formula = mpg ~ cyl + disp)

kitchen_sink <- 
lm(data = df, formula = mpg ~ . )

library(stargazer)

Please cite as: 

 Hlavac, Marek (2022). stargazer: Well-Formatted Regression and Summary Statistics Tables.

 R package version 5.2.3. https://CRAN.R-project.org/package=stargazer 

stargazer(reg1, kitchen_sink,
          type = "text"
          )

==================================================================
                                 Dependent variable:              
                    ----------------------------------------------
                                         mpg                      
                             (1)                     (2)          
------------------------------------------------------------------
cyl                        -1.587**                -0.111         
                           (0.712)                 (1.045)        
                                                                  
disp                       -0.021*                  0.013         
                           (0.010)                 (0.018)        
                                                                  
hp                                                 -0.021         
                                                   (0.022)        
                                                                  
drat                                                0.787         
                                                   (1.635)        
                                                                  
wt                                                 -3.715*        
                                                   (1.894)        
                                                                  
qsec                                                0.821         
                                                   (0.731)        
                                                                  
vs                                                  0.318         
                                                   (2.105)        
                                                                  
am                                                  2.520         
                                                   (2.057)        
                                                                  
gear                                                0.655         
                                                   (1.493)        
                                                                  
carb                                               -0.199         
                                                   (0.829)        
                                                                  
Constant                  34.661***                12.303         
                           (2.547)                (18.718)        
                                                                  
------------------------------------------------------------------
Observations                  32                     32           
R2                          0.760                   0.869         
Adjusted R2                 0.743                   0.807         
Residual Std. Error    3.055 (df = 29)         2.650 (df = 21)    
F Statistic         45.808*** (df = 2; 29) 13.932*** (df = 10; 21)
==================================================================
Note:                                  *p<0.1; **p<0.05; ***p<0.01

Choose the best multivariate regression

n <- 11           # n = total number of variables y and Xs
(n * (n-1) ) / 2  # total number of possible regressions

[1] 55

• BIG AIC IS BAD.

  • As you add more parameters k, AIC increases. We like parsimonious (simple) models.

  • Large L (likelihood is good) -> Small -L is good !

• LOW AIC is GOOD.

library(MASS)

Attaching package: 'MASS'

The following object is masked from 'package:dplyr':

    select

?stepAIC()

best_model <-
stepAIC(object = kitchen_sink,
        direction = "backward"
        )

Start:  AIC=70.9
mpg ~ cyl + disp + hp + drat + wt + qsec + vs + am + gear + carb

       Df Sum of Sq    RSS    AIC
- cyl   1    0.0799 147.57 68.915
- vs    1    0.1601 147.66 68.932
- carb  1    0.4067 147.90 68.986
- gear  1    1.3531 148.85 69.190
- drat  1    1.6270 149.12 69.249
- disp  1    3.9167 151.41 69.736
- hp    1    6.8399 154.33 70.348
- qsec  1    8.8641 156.36 70.765
<none>              147.49 70.898
- am    1   10.5467 158.04 71.108
- wt    1   27.0144 174.51 74.280

Step:  AIC=68.92
mpg ~ disp + hp + drat + wt + qsec + vs + am + gear + carb

       Df Sum of Sq    RSS    AIC
- vs    1    0.2685 147.84 66.973
- carb  1    0.5201 148.09 67.028
- gear  1    1.8211 149.40 67.308
- drat  1    1.9826 149.56 67.342
- disp  1    3.9009 151.47 67.750
- hp    1    7.3632 154.94 68.473
<none>              147.57 68.915
- qsec  1   10.0933 157.67 69.032
- am    1   11.8359 159.41 69.384
- wt    1   27.0280 174.60 72.297

Step:  AIC=66.97
mpg ~ disp + hp + drat + wt + qsec + am + gear + carb

       Df Sum of Sq    RSS    AIC
- carb  1    0.6855 148.53 65.121
- gear  1    2.1437 149.99 65.434
- drat  1    2.2139 150.06 65.449
- disp  1    3.6467 151.49 65.753
- hp    1    7.1060 154.95 66.475
<none>              147.84 66.973
- am    1   11.5694 159.41 67.384
- qsec  1   15.6830 163.53 68.200
- wt    1   27.3799 175.22 70.410

Step:  AIC=65.12
mpg ~ disp + hp + drat + wt + qsec + am + gear

       Df Sum of Sq    RSS    AIC
- gear  1     1.565 150.09 63.457
- drat  1     1.932 150.46 63.535
<none>              148.53 65.121
- disp  1    10.110 158.64 65.229
- am    1    12.323 160.85 65.672
- hp    1    14.826 163.35 66.166
- qsec  1    26.408 174.94 68.358
- wt    1    69.127 217.66 75.350

Step:  AIC=63.46
mpg ~ disp + hp + drat + wt + qsec + am

       Df Sum of Sq    RSS    AIC
- drat  1     3.345 153.44 62.162
- disp  1     8.545 158.64 63.229
<none>              150.09 63.457
- hp    1    13.285 163.38 64.171
- am    1    20.036 170.13 65.466
- qsec  1    25.574 175.67 66.491
- wt    1    67.572 217.66 73.351

Step:  AIC=62.16
mpg ~ disp + hp + wt + qsec + am

       Df Sum of Sq    RSS    AIC
- disp  1     6.629 160.07 61.515
<none>              153.44 62.162
- hp    1    12.572 166.01 62.682
- qsec  1    26.470 179.91 65.255
- am    1    32.198 185.63 66.258
- wt    1    69.043 222.48 72.051

Step:  AIC=61.52
mpg ~ hp + wt + qsec + am

       Df Sum of Sq    RSS    AIC
- hp    1     9.219 169.29 61.307
<none>              160.07 61.515
- qsec  1    20.225 180.29 63.323
- am    1    25.993 186.06 64.331
- wt    1    78.494 238.56 72.284

Step:  AIC=61.31
mpg ~ wt + qsec + am

       Df Sum of Sq    RSS    AIC
<none>              169.29 61.307
- am    1    26.178 195.46 63.908
- qsec  1   109.034 278.32 75.217
- wt    1   183.347 352.63 82.790

sum(3:10)

[1] 52

stargazer(reg1, kitchen_sink, best_model, type="text" )

=========================================================================================
                                             Dependent variable:                         
                    ---------------------------------------------------------------------
                                                     mpg                                 
                             (1)                     (2)                    (3)          
-----------------------------------------------------------------------------------------
cyl                        -1.587**                -0.111                                
                           (0.712)                 (1.045)                               
                                                                                         
disp                       -0.021*                  0.013                                
                           (0.010)                 (0.018)                               
                                                                                         
hp                                                 -0.021                                
                                                   (0.022)                               
                                                                                         
drat                                                0.787                                
                                                   (1.635)                               
                                                                                         
wt                                                 -3.715*               -3.917***       
                                                   (1.894)                (0.711)        
                                                                                         
qsec                                                0.821                 1.226***       
                                                   (0.731)                (0.289)        
                                                                                         
vs                                                  0.318                                
                                                   (2.105)                               
                                                                                         
am                                                  2.520                 2.936**        
                                                   (2.057)                (1.411)        
                                                                                         
gear                                                0.655                                
                                                   (1.493)                               
                                                                                         
carb                                               -0.199                                
                                                   (0.829)                               
                                                                                         
Constant                  34.661***                12.303                  9.618         
                           (2.547)                (18.718)                (6.960)        
                                                                                         
-----------------------------------------------------------------------------------------
Observations                  32                     32                      32          
R2                          0.760                   0.869                  0.850         
Adjusted R2                 0.743                   0.807                  0.834         
Residual Std. Error    3.055 (df = 29)         2.650 (df = 21)        2.459 (df = 28)    
F Statistic         45.808*** (df = 2; 29) 13.932*** (df = 10; 21) 52.750*** (df = 3; 28)
=========================================================================================
Note:                                                         *p<0.1; **p<0.05; ***p<0.01