ScPoEconometrics 3

• Suppose we had data on car speed and stopping dist (ance):

head(cars)

  speed dist
1     4    2
2     4   10
3     7    4
4     7   22
5     8   16
6     9   10

• How are speed and dist related?

• What's a reasonable summary of this relationship?

• A line. Great. But which line? This one?
• That's a flat line. But dist is increasing. :-(

• That one?
• Slightly better. Has a slope and an intercept.

• We observe \( (y_i,x_i) \) in the data.
• Here is a formula that describes their relationship: \[ y_i = b_0 + b_1 x_i + e_i \]
• How to choose \( b_0 \) and \( b_1 \) s.t. this is best line?

• We need to denote the output of the line.
• We call the value \( \widehat{y}_i \) the predicted value, after we choose values for \( b_0,b_1 \) and \( x \). \[ \widehat{y}_i = b_0 + b_1 x_i \]
• In general, we expect to make errors!

• Red Arrows are errors or residuals (often \( u \) or \( e \))

library(ScPoEconometrics) # load our library
launchApp('reg_simple_arrows')
aboutApp('reg_simple_arrows')

• choose \( (b_0,b_1) \) s.t. the sum \( e_1^2 + \dots + e_N^2 \) is as small as possible
• \( e_1^2 + \dots + e_N^2 \) is the sum of squared residuals, or SSR.
• Wait a moment… Why squared residuals?!

• In previous plot, errors of different sign (\( +/- \)) cancel out!
• The line with minimal SSR need not be a very good summary of our data.
• Squaring each \( e_i \) solves that issue as \( e_i^2 \geq 0, \forall i \)

launchApp('reg_simple')
aboutApp('reg_simple')

• OLS estimates the best line for us.
• In our single regressor case, there is a simple formula for the slope: \[ b_1 = \frac{cov(x,y)}{var(x)} \]
• and for the intercept \[ b_0 = \bar{y} - b_1\bar{x} \]

How does OLS actually perform the minimization problem?

launchApp('SSR_cone')
aboutApp('SSR_cone')  # after

Let's do some more OLS!

launchApp('reg_full')
aboutApp('reg_full')  # after

• There are some common flavors of OLS.
• We will go through some of them.
• E.g. what happens without an intercept?
• Or, what happens if we demean both \( y \) and \( x \)?

• Our line is flat at level \( b_0 \): \[ y = b_0 \]
• Our optimization problem is now \[ b_0 = \arg\min_{\text{int}} \sum_{i=1}^N \left[y_i - \text{int}\right]^2, \]
• With solution \[ b_0 = \frac{1}{N} \sum_{i=1}^N y_i = \overline{y}. \]

• Now we have a line anchored at the origin.
• The OLS slope estimate becomes \[ \begin{align} b_1 &= \arg\min_{\text{slope}} \sum_{i=1}^N \left[y_i - \text{slope } x_i \right]^2\\ \mapsto b_1 &= \frac{\frac{1}{N}\sum_{i=1}^N x_i y_i}{\frac{1}{N}\sum_{i=1}^N x_i^2} = \frac{\bar{x} \bar{y}}{\overline{x^2}} \end{align} \]

launchApp('reg_constrained')

• centering or demeaning means to substract the mean.
• We get \( \widetilde{y}_i = y_i - \bar{y} \).
• Let's run a regression without intercept as above using \( \widetilde{y}_i,\widetilde{x}_i \)
• We get \[ \begin{align} b_1 &= \frac{\frac{1}{N}\sum_{i=1}^N \widetilde{x}_i \widetilde{y}_i}{\frac{1}{N}\sum_{i=1}^N \widetilde{x}_i^2}\\ &= \frac{cov(x,y)}{var(x)} \end{align} \]

launchApp('demeaned_reg')

• To standardize \( z \) means to do \( \breve{z}=\frac{z-\bar{z}}{sd(z)} \)
• I.e. substract the variable's mean and divide by its standard deviation.
• Proceed as above, but with \( \breve{y}_i,\breve{x}_i \)
• We get \[ \begin{align} b_1 &= \frac{\frac{1}{N}\sum_{i=1}^N \breve{x}_i \breve{y}_i}{\frac{1}{N}\sum_{i=1}^N \breve{x}_i^2}\\ &= \frac{Cov(x,y)}{\sigma_x \sigma_y}=Corr(x,y) \end{align} \]

launchApp('reg_standardized')

1. The error is \( e_i = y_i - \widehat{y}_i \)
2. The average of \( \widehat{y}_i \) is equal to \( \bar{y} \). \[ \begin{align} \frac{1}{N} \sum_{i=1}^N \widehat{y}_i &= \frac{1}{N} \sum_{i=1}^N b_0 + b_1 x_i \\ &= b_0 + b_1 \bar{x} = \bar{y}\\ \end{align} \]
3. Then, \[ \frac{1}{N} \sum_{i=1}^N e_i = \bar{y} - \frac{1}{N} \sum_{i=1}^N \widehat{y}_i = 0 \] i.e. the average of errors is zero.

• It's important to keep in mind that Var, Cov, Corr and Regression measure linear relationships between two variables.
• Two datasets with identical correlations could look vastly different.
• They would have the same regression line.
• Same correlation coefficient.
• Is that even possible?

• So, be wary of only looking a linear summary stats.
• Also look at plots.

• Dinosaurs?

  launchApp("datasaurus")
  aboutApp("datasaurus")
  

• We can accomodate non-linear relationships in regressions.
• We'd just add a higher order term like this: \[ y_i = b_0 + b_1 x_i + b_2 x_i^2 + e_i \]
• This is multiple regression (next chapter!)

• For example, suppose we had this data and fit the above regression:

• Remember that \( y_i = \widehat{y}_i + e_i \).
• We have the following decomposition: \[ \begin{align} Var(y) &= Var(\widehat{y} + e)\\ &= Var(\widehat{y}) + Var(e) + 2 Cov(\widehat{y},e)\\ &= Var(\widehat{y}) + Var(e) \end{align} \]
• Because: \( Cov(\hat{y},e)=0 \)
• Total variation (SST) = Model explained (SSE) + Unexplained (SSR)

• The \( R^2 \) measures how good the model fits the data.
• \( R^2=1 \) is very good, \( R^2=0 \) is very poorly. \[ R^2 = \frac{\text{variance explained}}{\text{total variance}} = \frac{SSE}{SST} = 1 - \frac{SSR}{SST}\in[0,1] \]
• Small \( R^2 \) doesn't mean it's a useless model!

• Let's look at the worked example in the book!

• Suppose outcome \( y \) is income in Euros
• \( x \) be years of schooling \[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \]
• Assume that \( \beta_1 = 2000 \), s.t. each additional year of schooling gives 2000 Euros more income.
• What is \( \beta_1 \) if we measure \( y \) in thousands of euros instead?

• The result depends on whether we rescale \( y \), \( x \) or both.
• If we have rescale \( x \) by \( c \) and \( y \) by \( d \), one can show that \[ \begin{align} b_1' &= \frac{Cov(dx,cy)}{Var(dx)}\\ &= \frac{d}{c} b \end{align} \]
• and the intercept \[ \begin{align} b_0' &= \mathbb{E}[d\cdot Y] - \frac{Cov(cX, dY)}{Var(cX)} \mathbb{E}[c\cdot X] \\ &= d b_0 \end{align} \]

library(ScPoEconometrics)
launchApp('Rescale')

library(ScPoEconometrics)
runTutorial('rescaling')