Regression Analysis

双变量关系及建模

• 两个变量都是数值型（numeric）

• 应变量（response, dependent variable）

• 解释变量（explanatory）

  • 你认为与应变量有关的东西
  • 也可以叫做，independent variable, predictor

一般的模型表达式

\[应变量=f(解释变量)+噪音\]

线性回归模型表达式

\[应变量=截距+斜率*解释变量+噪音\]

线性回归模型–为什么这一条线最合适？

\[y_i= a + b*x_i+e\]

最小二乘法（least squares method ）

简单地说，最小二乘的思想就是要使得

** 观测点和估计点的距离的平方和达到最小 **.

这里的“二乘”指的是用平方来度量观测点与估计点的远近（在古汉语中“平方”称为“二乘”），“最小”指的是参数的估计值要保证各个观测点与估计点的距离的平方和达到最小。

最小二乘法（least squares method ）

线性回归模型:

\[y_i= a + b*x_i+e_i\]

\[e_i ~ N(0, \sigma)\]

预测值：

\[\hat{y_i} = \hat{a} + \hat{b}x_i\]

残差（error）:

\[e_i = y_i-\hat{y_i}\]

拟合的过程：有n个样本量，也就是n个数据对 \((x_i,y_i)\), 找到 截距 \(a\) 和斜率 \(b\) ，使得残差平方和最小:

\[\sum_1^{n} e_i^2\]

** 关键概念 ** :

\(\hat{y_i}\) 是给定 \(x_i\) 值时， \(y\) 的期望值

\(\hat{b}\) 是真实但是未知的 \(b\) 的估计值

残差 \(e\)：residual, error, noise

最小二乘法求线性回归系数

\[y_i= a + b*x_i+e_i\]

\[\hat{y_i} = \hat{a} + \hat{b}x_i\]

\[e_i = y_i-\hat{y_i}\]

\[\sum_1^{n} e_i^2 = \sum (y_i-\hat{y_i})^2=\sum (y_i- \hat{a} -\hat{b}x_i)^2\]

最小二乘法求线性回归系数:

\[f=\sum_1^{n} e_i^2 =\sum (y_i- \hat{a} -\hat{b}x_i)^2\]

一阶导数等于 0：

\[\frac{\partial f}{\partial\hat{a}} = -2 \sum (y_i- \hat{a} -\hat{b}x_i)=0\]

\[\frac{\partial f}{\partial\hat{b}} = -2 \sum (y_i- \hat{a} -\hat{b}x_i)x_i=0\]

联立上面的二元一次方程组，求解 \(\hat{a}\) 与 \(\hat{b}\)

\[\hat{b}=\frac{\sum (x_i-\bar{x}) (y_i-\bar{y})}{\sum (x_i-\bar{x})^2}\]

\[\hat{a}= \bar{y} - \hat{b}\bar{x}\]

最小二乘法的特点:

• 唯一性

• 简单（相对于非线性）

• 残差之和等于0

• 回归线必过

\((\bar{x}, \bar{y})\)

轶事：回归的由来

Regression to the mean is a concept attributed to Sir Francis Galton. The basic idea is that extreme random observations will tend to be less extreme upon a second trial. This is simply due to chance alone. Note that “regression to the mean” and “linear regression” are not the same thing.

• 孩子的身高和父母的身高：高个子的父母一般生的孩子个子也高，但是孩子一般没有父母那么高，孩子的身高会趋向于平均数。

• 这就是我们没有看到乔丹的儿子能够和乔丹一样成为飞人的原因

\[y_i= a + b*x_i+e_i\]

回归模型的目标是量化变量之间的关系：

• 关键系数的大小

• 关键系数的显著度

• 模型的解释力

• 模型的预测力

## 
## Call:
## lm(formula = mpg ~ wt, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.5432 -2.3647 -0.1252  1.4096  6.8727 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
## wt           -5.3445     0.5591  -9.559 1.29e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.046 on 30 degrees of freedom
## Multiple R-squared:  0.7528, Adjusted R-squared:  0.7446 
## F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

理解模型：

1. 残差 （Residuals）： Residuals show if the predicted response values are close or not to the response values that the model predicts.

\[e_i = y_i-\hat{y_i}\]

2. 系数估计 （Estimate coefficient）： How y changes when x changes by one unit

\[\hat{b}=\frac{\sum (x_i-\bar{x}) (y_i-\bar{y})}{\sum (x_i-\bar{x})^2}\]

\[\hat{a}= \bar{y} - \hat{b}\bar{x}\]

3. 系数估计的标准差（Standard error）： Standard error measures how the coefficient estimates can vary from the actual average value of the response variable (i.e. if the model is run more times). 每一次数据不同，结果都会有差异，但是该差异在一定范围内。

4. 显著度与P值 （Significance and P value）： Test of significance of the model shows that there is strong evidence of a linear relationship between the variables. This is visually interpreted by the significance stars *** at the end of the row. This level is a threshold that is used to indicate real findings, and not the ones by chance alone.

For each estimated regression coefficient, the variable’s p-Value Pr(>|t|) provides an estimate of the probability that the true coefficient is zero given the value of the estimate. More the number of stars near the p-Value are, more significant is the variable.

With the presence of the p-value, there is a test of hypothesis associated with it. In Linear Regression, the Null Hypothesis is that the coefficient associated with the variables is equal to zero. Instead, the alternative hypothesis is that the coefficient is not equal to zero and then exists a relationship between the independent variable and the dependent variable.

So, if p-values are less than significance level (typically, a p-value < 0.05 is a good cut-off point), null hypothesis can be safely rejected. In the current case, p-values are well below the 0.05 threshold, so the model is indeed statistically significant.

5. R-squared 以及 adjusted R-squared For the simple linear regression, R-squared is the square of the correlation between two variables. Its value can vary between 0 and 1: a value close to 0 means that the regression model does not explain the variance in the response variable, while a number close to 1 that the observed variance in the response variable is well explained. In the current case, R-squared suggests the linear model fit explains about 75% of the variance observed in the data.

\[R^2=1-\frac{var(e)}{var(y)}\]

6.F statistic Basically, F-test compares the model with zero predictor variables (the intercept only mod1el), and suggests whether the added coefficients improve the model. If a significant result is obtained, then the coefficients (in the current case, being a simple regression, only one predictor is entered) included in the model improve the model’s fit. So, F statistic defines the collective effect of all predictor variables on the response variable. In this model, F=119.8 is far greater than 1.

模型评估：adjusted R squared, BIC, AIC 从最全的模型开始 工资～学科+性别+级别+工作年数+年数平方 工资～学科+性别+级别+取得博士之后的年数+年数平方

回归诊断作图