Suppose we want to predict a random variable \(y\) from a set of random variables \(\mathbf{x}\). Let \(\mu(\mathbf{x})\) be the predictor. The prediction error is \(y - \mu(\mathbf{x})\).
Argue that the mean squared error \(MSE = \mathbb{E}([y - \mu(\mathbf{x})]^2)\) is a sensible metric for evaluating the quality of a predictor \(\mu(\mathbf{x})\).
\(MSE = \mathbb{E}([y - \mu(\mathbf{x})]^2)\) is a sensible metric because the raw error \(y - \mu(\mathbf{x})\) can be positive or negative, so averaging it directly, \(\mathbb{E}(y - \mu(\mathbf{x}))\), would let errors of opposite sign cancel out and mask a poor predictor. Squaring the error solves this by penalising deviations of either sign, and it also penalises large errors more heavily than small ones. This gives a single non-negative number summarising the average error magnitude, which is zero only when \(\mu(\mathbf{x}) = y\) with probability 1. ### Part 2
Show \(\mathbb{E}( y - \mathbb{E}(y|\mathbf{x}) | \mathbf{x}) = 0\). Interpret.
Since \(\mathbb{E}(y|\mathbf{x})\) is already a function of \(\mathbf{x}\), conditioning on \(\mathbf{x}\) again leaves it unchanged, so by linearity of expectation:
\[\mathbb{E}(y - \mathbb{E}(y|\mathbf{x}) \mid \mathbf{x}) = \mathbb{E}(y|\mathbf{x}) - \mathbb{E}(\mathbb{E}(y|\mathbf{x}) \mid \mathbf{x}) = \mathbb{E}(y|\mathbf{x}) - \mathbb{E}(y|\mathbf{x}) = 0\]
Interpretation: the prediction error from using the conditional mean, \(y - \mathbb{E}(y|\mathbf{x})\), has conditional mean zero at every value of \(\mathbf{x}\). In other words, \(\mathbb{E}(y|\mathbf{x})\) never systematically over- or under-predicts \(y\) for any given \(\mathbf{x}\) — the errors are “balanced” locally, not just on average across the whole sample.
Show \(\mathbb{E}([y - \mu(\mathbf{x})]^2|\mathbf{x}) = \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2|\mathbf{x}) + \mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2|\mathbf{x})\)
Add and subtract \(\mathbb{E}(y|\mathbf{x})\) inside the squared error term:
\[y - \mu(\mathbf{x}) = \big[y - \mathbb{E}(y|\mathbf{x})\big] + \big[\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})\big]\]
Let \(A = y - \mathbb{E}(y|\mathbf{x})\) and \(B = \mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})\), so \(y - \mu(\mathbf{x}) = A + B\). Squaring:
\[(A+B)^2 = A^2 + 2AB + B^2\]
Taking \(\mathbb{E}(\cdot \mid \mathbf{x})\) of both sides:
\[\mathbb{E}([y - \mu(\mathbf{x})]^2 \mid \mathbf{x}) = \mathbb{E}(A^2 \mid \mathbf{x}) + 2\,\mathbb{E}(AB \mid \mathbf{x}) + \mathbb{E}(B^2 \mid \mathbf{x})\]
Since \(B = \mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})\) is a function of \(\mathbf{x}\) only, it is constant given \(\mathbf{x}\) and can be pulled out of the conditional expectation:
\[\mathbb{E}(AB \mid \mathbf{x}) = B \cdot \mathbb{E}(A \mid \mathbf{x}) = B \cdot \mathbb{E}(y - \mathbb{E}(y|\mathbf{x}) \mid \mathbf{x}) = B \cdot 0 = 0\]
using the result from Part 2. So the cross term vanishes, leaving:
\[\mathbb{E}([y - \mu(\mathbf{x})]^2 \mid \mathbf{x}) = \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2 \mid \mathbf{x}) + \mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2 \mid \mathbf{x})\]
Show \(\mathbb{E}([y - \mu(\mathbf{x})]^2) \geq \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2)\). Argue \(\mathbb{E}(y|\mathbf{x})\) is the best predictor.
Hint: Make use of the Law of Iterated Expectations
From Part 3, for any value of \(\mathbf{x}\):
\[\mathbb{E}([y - \mu(\mathbf{x})]^2 \mid \mathbf{x}) = \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2 \mid \mathbf{x}) + \mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2 \mid \mathbf{x})\]
Taking the unconditional expectation of both sides:
\[\mathbb{E}\Big[\mathbb{E}([y - \mu(\mathbf{x})]^2 \mid \mathbf{x})\Big] = \mathbb{E}\Big[\mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2 \mid \mathbf{x})\Big] + \mathbb{E}\Big[\mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2 \mid \mathbf{x})\Big]\]
By the Law of Iterated Expectations, \(\mathbb{E}[\mathbb{E}(Z \mid \mathbf{x})] = \mathbb{E}(Z)\) for any random variable \(Z\), so this becomes:
\[\mathbb{E}([y - \mu(\mathbf{x})]^2) = \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2) + \mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2)\]
The final term is the expectation of a squared quantity, so it is non-negative:
\[\mathbb{E}([\mathbb{E}(y|\mathbf{x}) - \mu(\mathbf{x})]^2) \geq 0\]
Therefore:
\[\mathbb{E}([y - \mu(\mathbf{x})]^2) = \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2) + (\text{a non-negative term}) \geq \mathbb{E}([y - \mathbb{E}(y|\mathbf{x})]^2)\]
Interpretation: the left-hand side is the MSE of an arbitrary predictor \(\mu(\mathbf{x})\), and the right-hand side is the MSE of the conditional mean \(\mathbb{E}(y|\mathbf{x})\). The inequality shows that the MSE of any predictor can never be smaller than the MSE of \(\mathbb{E}(y|\mathbf{x})\). Equality holds only when the extra term equals zero, which requires \(\mu(\mathbf{x}) = \mathbb{E}(y|\mathbf{x})\) with probability 1. Hence \(\mathbb{E}(y|\mathbf{x})\) achieves the minimum possible MSE among all predictors and is therefore the best predictor of \(y\) given \(\mathbf{x}\).
Load the data:
In lecture 1, we worked with the regression model
\[\begin{equation}\label{reg0} \log(sales) = \beta_{0,b} + \beta_{1,b} \log(price) + \varepsilon \end{equation}\]
where \(b \in \{do,mm,tr\}\) indicate “brand”, so \(\beta_{0,b}\) is a brand-specific intercept and \(\beta_{1,b}\) is the brand-specific price elasticity of orange juice sales.
Fit \(\eqref{reg0}\) to the
oj data. Report estimated coefficients and SEs of your
fitted model, and the implied brand-specific elasticities. What do you
learn about the different brands of orange juice from these
elasticities?
reg <- glm(log(sales) ~ brand + log(price), data=oj) summary(reg)
# Fit the regression model here
reg1 <- glm(log(sales) ~ brand * log(price), data = oj)
summary(reg1)##
## Call:
## glm(formula = log(sales) ~ brand * log(price), data = oj)
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 10.95468 0.02070 529.136 <2e-16 ***
## brandminute.maid 0.88825 0.04155 21.376 <2e-16 ***
## brandtropicana 0.96239 0.04645 20.719 <2e-16 ***
## log(price) -3.37753 0.03619 -93.322 <2e-16 ***
## brandminute.maid:log(price) 0.05679 0.05729 0.991 0.322
## brandtropicana:log(price) 0.66576 0.05352 12.439 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for gaussian family taken to be 0.6258833)
##
## Null deviance: 30079 on 28946 degrees of freedom
## Residual deviance: 18114 on 28941 degrees of freedom
## AIC: 68592
##
## Number of Fisher Scoring iterations: 2Coefficients: Estimate (Intercept) 10.82882 brandminute.maid 0.87017 brandtropicana 1.52994 log(price) -3.13869 Std. Error (Intercept) 0.01453 brandminute.maid 0.01293 brandtropicana 0.01631
Sales drop around 3% for every 1% increase in price.There is not a brand specific elasticity as they all share the same 3%, you would need to introduce interaction terms to separate by brand.The intercept does differ by brand. Dominick’s baseline log-sales is 10.829 while minute maids is 11.699 and Tropicana is 12.359 (at the same price). Tropicana sells the most, then Minute Maid then Dominick’s.
Now consider how advertisement campaigns may moderate sales volumes and the price sensitivity of the featured product:
\[\begin{equation}\label{reg3} \log(sales) = \beta_{0,b,f} + \beta_{1,b,f} \log(price) + \varepsilon \end{equation}\]
where \(b\) indicates “brand” and \(f\) indicates the brand was “featured” in a local advertisement campaign.
Describe how the three-way regression model \(\eqref{reg3}\) allow advertisement to affect sales volumes, and represent the model in terms of brand-dummies \(1_{b}\) and a featured-dummy \(1_{f}\).
Hint: The dummy variable versions of \(\eqref{reg3}\) involves 12 coefficients.
There are two effects. Being featured in an ad can shift sales up or down at any given price, captured by the feat-dummy’s coefficient and its interaction with the brand dummies (intercept terms) captures the extra shift specific to that brand when featured, beyond the generic feat effect and the generic brand effect. Also there is a second effect: being featured can also change how sales respond to price changes eg could be deal driven which alters consumer behaviour or brings in new customers who are less loyal to the brand and more price sensitive. This is captured by letting log(price) slope depend on feat and brand. Both the feat level effect and the sensitivity effect can vary by brand also. In terms of being three-way and having 12 coefficients that is because there is 3 brands and 2 states they can be in so 6 groups each with its own intercept and slope. Dominick’s, not features is baseline.
\[ \log(\text{sales}) = \beta_0 + \beta_1 \mathbb{1}_{mm} + \beta_2 \mathbb{1}_{tr} + \beta_3 \mathbb{1}_{f} + \beta_4 (\mathbb{1}_{mm}\mathbb{1}_{f}) + \beta_5 (\mathbb{1}_{tr}\mathbb{1}_{f}) \] \[ + \left[\gamma_0 + \gamma_1 \mathbb{1}_{mm} + \gamma_2 \mathbb{1}_{tr} + \gamma_3 \mathbb{1}_{f} + \gamma_4 (\mathbb{1}_{mm}\mathbb{1}_{f}) + \gamma_5 (\mathbb{1}_{tr}\mathbb{1}_{f})\right] \log(\text{price}) + \varepsilon \]
Fit \(\eqref{reg3}\) to the
oj data. Report estimated coefficients and SEs of your
fitted model, and the implied brand- and featured-specific elasticities.
How does advertisement impact price sensitivity? Why does advertisement
impact price sensitivity? How may your insights aid a marketing campaign
team?
##
## Call:
## glm(formula = log(sales) ~ log(price) * brand * feat, data = oj)
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 10.40658 0.02335 445.668 < 2e-16 ***
## log(price) -2.77415 0.03883 -71.445 < 2e-16 ***
## brandminute.maid 0.04720 0.04663 1.012 0.311
## brandtropicana 0.70794 0.05080 13.937 < 2e-16 ***
## feat 1.09441 0.03810 28.721 < 2e-16 ***
## log(price):brandminute.maid 0.78293 0.06140 12.750 < 2e-16 ***
## log(price):brandtropicana 0.73579 0.05684 12.946 < 2e-16 ***
## log(price):feat -0.47055 0.07409 -6.351 2.17e-10 ***
## brandminute.maid:feat 1.17294 0.08196 14.312 < 2e-16 ***
## brandtropicana:feat 0.78525 0.09875 7.952 1.90e-15 ***
## log(price):brandminute.maid:feat -1.10922 0.12225 -9.074 < 2e-16 ***
## log(price):brandtropicana:feat -0.98614 0.12411 -7.946 2.00e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for gaussian family taken to be 0.4829706)
##
## Null deviance: 30079 on 28946 degrees of freedom
## Residual deviance: 13975 on 28935 degrees of freedom
## AIC: 61094
##
## Number of Fisher Scoring iterations: 2Minute Maid: not featured, a 1% increase in price drops MM sales by 2%. Featured drops sales by roughly 3.6%. For tropicana not featured, a 1% increase in price drops sales by 2.0%. For featured, the same price drop causes a 3.5% drop in sales. For Dominick’s, not featured: a 1% inc leads to a 2.8% drop in sales and for featured a 3.2% drop.For every brand, featuring makes sales more price sensitive (elasticity more negative). The effect is not even across brands, Minute Maid’s roughly doubles when featured while Dominick’s barely moves. Most likely reasoning would be that being featured draws in a different, more price-attentive segment of shoppers: people who showed up because of a deal and not of brand loyalty so they may be more likely to switch away if the price increases. Dominick’s probably less hit by feature because they are already the cheapest option so not many options to switch to.
Compute, interpret and compare the \(R^2\) for \(\eqref{reg0}\) and \(\eqref{reg3}\). Comment on the difference between the two \(R^2\) values.
Hint: Use the deviance and null deviance statistics provided in
the regression output. If your regression output is myreg,
then you can access deviance as myreg$deviance and null
deviance as myreg$null.deviance
reg1 <- glm(log(sales) ~ brand + log(price), data = oj)
R2_reg1 <- 1 - reg1$deviance / reg1$null.deviance
R2_reg1## [1] 0.3940951ojreg <- glm(log(sales) ~ log(price) * brand * feat, data = oj)
R2_ojreg <- 1 - ojreg$deviance / ojreg$null.deviance
R2_ojreg## [1] 0.5353939## [1] 0.1412988Model 1 explains 39% of the variation in log sales. Model 2 explains 54%. Also mechanically guarenteed to go up when adding more terms (brand/ feat interactions); nothing was removed, only added. Model 1 was an overly simple model which led to confounding between advertisement and brand effects. For example, minute maid could have been featured in more ads than tropicana leading to more price sensitivity, must correct this by including feat in the regression.
Use the fitted models \(\eqref{reg0}\) and \(\eqref{reg3}\) to make out-of-sample
predictions for the sale of Dominicks orange juice
(brand = "dominicks") when the price is four dollars
(price = 4) and the juice is featured in an advertisement
(feat = 1). How would you assess the quality of the
out-of-sample predictions of the two models?
Hint I: Use the predict() function, which takes the
fitted model and a data frame of new data,
e.g. mypred <- predict(mymodel, mynewdata).
Hint II: Build mynewdata with
data.frame(brand = "dominicks", price = 4, feat = 1).
# Out-of-sample predictions here
mynewdata <- data.frame(brand = "dominicks", price = 4, feat = 1)
mynewdata ## brand price feat
## 1 dominicks 4 1## 1
## 6.477671## 1
## 7.002862## 1
## 650.4545## 1
## 1099.777The two models give substantially different predictions. Model 2 predicts 70% more units sold than model 1. Because model 1 has no feat term, it predicts sales for Dominick’s at 4$ without distinguishing between featured and not, so ignores that this observation was an ad week. Model 2 does use feat = 1, and since featuring is associated with an increase in sales, this explains model 2’s higher prediction,