The significance of the model

In the previous section, we built a linear model of sales as a function of youtube advertising budget: sales = 8.44 + 0.048*youtube.

Before using this formula to predict future sales, you should make sure that this model is statistically significant, that is:

• There is a statistically significant relationship between the predictor and the outcome variables
• The model that we built fits very well the data in our hand. Now we’ll try to check the quality of a linear regression model

help(summary)

## démarrage du serveur d'aide httpd ... fini

summary(model)$coefficient[,4]

##       (Intercept) marketing$youtube 
##       1.40630e-35       1.46739e-42

confint(model)

##                        2.5 %     97.5 %
## (Intercept)       7.35566312 9.52256140
## marketing$youtube 0.04223072 0.05284256

Model accuracy:

Once you identified that, at least, one predictor variable is significantly associated the outcome, you should continue the diagnostic by checking how well the model fits the data. This process is also referred to as the goodness-of-fit

The overall quality of the linear regression fit can be assessed using the following three quantities, displayed in the model summary:

• The Residual Standard Error (RSE)
• The R-squared (R2)
• Fstatistic

model_summary = summary(model)
model_summary

## 
## Call:
## lm(formula = marketing$sales ~ marketing$youtube)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -10.0632  -2.3454  -0.2295   2.4805   8.6548 
## 
## Coefficients:
##                   Estimate Std. Error t value Pr(>|t|)    
## (Intercept)       8.439112   0.549412   15.36   <2e-16 ***
## marketing$youtube 0.047537   0.002691   17.67   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.91 on 198 degrees of freedom
## Multiple R-squared:  0.6119, Adjusted R-squared:  0.6099 
## F-statistic: 312.1 on 1 and 198 DF,  p-value: < 2.2e-16

model_summary$sigma;

## [1] 3.910388

model_summary$r.squared;

## [1] 0.6118751

model_summary$fstatistic[1]

##   value 
## 312.145

The RSE (also known as the model sigma) is the residual variation, representing the average variation of the observations points around the fitted regression line. This is the standard deviation of residual errors.

RSE provides an absolute measure of patterns in the data that can’t be explained by the model. When comparing two models, the model with the small RSE is a good indication that this model fits the best the data.

Dividing the RSE by the average value of the outcome variable will give you the prediction error rate, which should be as small as possible.

In our example, RSE = 3.91, meaning that the observed sales values deviate from the true regression line by approximately 3.9 units in average.

Whether or not an RSE of 3.9 units is an acceptable prediction error is subjective and depends on the problem context. This is why we usually normalise it with respect to the predicted variable.

RSE = model_summary$sigma
y= marketing$sales
error =  RSE/mean(y)
error

## [1] 0.2323877

The R-squared (R2) ranges from 0 to 1 and represents the proportion of information (i.e. variation) in the data that can be explained by the model. The adjusted R-squared adjusts for the degrees of freedom.

The R2 measures, how well the model fits the data. For a simple linear regression, R2 is the square of the Pearson correlation coefficient.

A high value of R2 is a good indication. However, as the value of R2 tends to increase when more predictors are added in the model, such as in multiple linear regression model, you should mainly consider the adjusted R-squared, which is a penalized R2 for a higher number of predictors.

An (adjusted) R2 that is close to 1 indicates that a large proportion of the variability in the outcome has been explained by the regression model. A number near 0 indicates that the regression model did not explain much of the variability in the outcome. F-Statistic: The F-statistic gives the overall significance of the model. It assess whether at least one predictor variable has a non-zero coefficient.

In a simple linear regression, this test is not really interesting since it just duplicates the information in given by the t-test, available in the coefficient table. In fact, the F test is identical to the square of the t test: 312.1 = (17.67)^2. This is true in any model with 1 degree of freedom.

The F-statistic becomes more important once we start using multiple predictors as in multiple linear regression.

A large F-statistic will corresponds to a statistically significant p-value (p < 0.05). In our example, the F-statistic equal 312.14 producing a p-value of 1.46e-42, which is highly significant.

#Le F nous donne un p <0.05 donc c'est significatif

Make sure your data meet the assumptions: We can use R to check that our data meet the four main assumptions for linear regression. This can be done only after the model conructiion as we will see.

#There is 4 main assumptions:
#Linearity, Homoscedasticity, Independenc, Normality
res = resid(model)
plot(fitted(model),res)
abline(0,0)

qqnorm(res)
qqline(res)

plot(density(res))

plot(model)

#A residuals vs. leverage plot is a type of diagnostic plot that allows us to identify influential observations in a regression model.Each observation from the dataset is shown as a single point within the plot. The x-axis shows the leverage of each point and the y-axis shows the standardized residual of each point.