6.3.1.1 Significant correlation

Even when there is no correlation between two variables the correlation coefficient calculated based on a sample will not be 0. That’s why it is quite common to use a significance test to test whether the r value differs significantly from 0. More about this test can be found here.

6.3.2.1 MS Excel regression output

Figure 3 shows the MS Excel regression output for the Dutch car examples (see abve).

Figure 3. Regression output created with MS Excel, menu: data/Data Analysis/Regression.

The regression model in the example output in figure 4 can be described by the equation:
CATALOG PRICE = -70,849 + 81.30 x MASS, where
MASS is the mass of the car in kg, and
CATALOG PRICE is the catalog price in euro.

Call:
lm(formula = Catalog_price ~ Mass, data = rdw)

Residuals:
   Min     1Q Median     3Q    Max 
-47701  -4551   -765   4342  50948 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -70849.017   4167.561  -17.00   <2e-16 ***
Mass            81.305      2.971   27.36   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 9246 on 205 degrees of freedom
Multiple R-squared:  0.7851,    Adjusted R-squared:  0.784 
F-statistic: 748.7 on 1 and 205 DF,  p-value: < 2.2e-16

Figure 4. Regression output RDW example, using R.

6.3.2.2 Assessing the regression model

After generating a regression model it is important to assess the model. This is done to judge whether the model fits the data and, in case of comparing models, which model fits the data best.

Coefficient of determination R²
In case of simple linear regression, i.e. with just one X-variable, the R2 value is the correlation coefficient squared. In a multiple regression setting it’s more complicated.

In general, the following equation holds: R² = \(\frac{variation\ of\ the\ model\ Yvalues\ around\ the\ mean\ Yvalue}{variation\ of\ the\ Yvalues\ around\ the\ mean\ Yvalue}\) = \(\frac{\Sigma(\widehat{Y} - \bar{Y})^{2}} {\Sigma(Y - \bar{Y})^{2}}\)

The denominator of this expression, \(\Sigma(Y - \bar{Y})^{2}\), measures the total variation in the Y-values around the mean Y-value, while the numerator, \(\Sigma(\widehat{Y} - \bar{Y})^{2}\), measures the variation of the model Y-values (the \(\widehat{Y}\)-values) around the mean Y-value. The latter is the variation in the Y-values that is explained by the regression model.
So R² measures the proportion of the variation in the Y-values that is explained by the regression model. In all cases: 0 <= R² <= 1.

The standard error of the estimate s_e
The standard error of the estimate is a metric for the spread around the regression line.
The value can be used to estimate the so-called margin of error if the model is used to predict a Y-value given the X-value(s). As a rule of thumb, the margin of error that should be taken in account when predicting a Y-value is 2xs_e.
So the smaller the s_e value the better the model can predict a Y-value.
To evaluate the value of s_e, it is compared with \(\bar{Y}\), commonly by dividing s_e by \(\bar{Y}\).

P-values of the coefficient(s)
The P-values of the model coefficients are used to determine whether the different X-variables are significant in the model. Most common interpretation: a P-value less than .05 indicates that the X-variable is significant. The test for which this P-value is calculated is:
H₀: \(\beta\) = 0
H_A: \(\beta\) <> 0, where \(\beta\) is the coefficient of the X-variable in question.
Not rejecting H₀ means the coefficient of the X-variable doesn’t differ significantly from 0, in other words Y doesn’t depend significantly on X.

6.3.4.1 Data description

The sample contains 400 observations on 15 variables.

As a first analysis the correlation coefficients between some of the numeric variables has been calculated. MS Excel: Data/Data Analysis/Correlation.

Figure 4. Correlation matrix generated with MS Excel.

As can be seen in Figure 5 the variable SPECIAL_TAX has the highest correlation with CATALOG_PRICE. That’s why the first regression model is a simple linear refgression model with CATALOG_PRICE as explanatory variable.

Figure 5. Scatterplot, SPECIAL_TAX in euro against CATALOG_PRICE in euro.

6.3.4.2 Simple linear models

A first model uses CATALOG_PRICE as explanatory variable.

Figure 6. MS Excel output simple linear regression model with SPECIAL_TAX as response variable and CATALOG_PRICE as explanatory variable.

A second model uses MASS as explanatory variable.

Call:
lm(formula = SPECIAL_TAX ~ MASS, data = toyota)

Residuals:
    Min      1Q  Median      3Q     Max 
-5453.7 -2541.8  -234.7  1467.7 26311.6 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -6396.8549   702.5876  -9.105   <2e-16 ***
MASS            7.6394     0.5753  13.279   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3080 on 398 degrees of freedom
Multiple R-squared:  0.307, Adjusted R-squared:  0.3053 
F-statistic: 176.3 on 1 and 398 DF,  p-value: < 2.2e-16

6.3.4.3 Multiple linear model: SPECIAL_TAX ~ MASS + CATALOG_PRICE

In the second model two explanaory variables ares used: MASS and CATALOG_PRICE. This model has to be

Figure 7. MS Excel output linear regression model with SPECIAL_TAX as response variable and CATALOG_PRICE and MASS as explanatory variable.

Although this model uses two explanatory variables which are both moderately correlated with the response variable SPECIAL_TAX, the model is not much better than the simple linear models. The reason for this is that the two explanatory variables are highly correlated with each other. In general it is preferable to use explanatory variables which are not correlated to each other.

6.3.4.4 Multiple linear model: SPECIAL_TAX ~ CATALOG_PRICE + ELECTRIC

The third model makes use of a dummy variable ELECTRIC (1 = FUEL_DESCRPTION=“ELECTRICITY”, 0 = FUEL_DESCRPTION<>“ELECTRICITY”).

Figure 8. MS Excel output linear regression model with SPECIAL_TAX as response variable and CATALOG_PRICE and ELECTRIC as explanatory variables. ELECTRIC is a dummy variable which takes on value 1 if the car is an electric car and 0 otherwise.

This model is really an improvement of the simple regression model with CATALOG_PRICE as the only explanatory variable.

Exercise
As can be seen in Figure 5 the data set contains a couple of outliers. Remove these outliers from the data set and generate the different OLS models without these outliers.