3-1 Find the minimizer of the following estimation criterion \(Q(\beta)=\frac{1}{N}\sum_{n=1}^{N}|y_n-\beta x_n|^2+\lambda|\beta|^2\)

將原式對 \(\beta\) 微分

\(\frac{dQ(\beta)}{d\beta}=\frac{1}{N}\sum_{n=1}^N(-2y_nx_n+2\beta x_n^2)+2\beta\lambda=-2(\frac{1}{N}\sum_{n=1}^{N}(y_nx_n-\beta x_n^2)-\beta\lambda)\)

為了求 minimizer，令上式為 0，求 \(\beta\)，

\(\frac{1}{N}\sum_{n=1}^{N}(y_nx_n-\beta x_n^2)-\beta\lambda=0\)

\(\sum_{n=1}^{N}(y_nx_n-\beta x_n^2)=N\beta\lambda\)

\(\sum_{n=1}^{N}(y_nx_n)=N\beta\lambda+\sum_{n=1}^{N}(\beta x_n^2)\)

\(\sum_{n=1}^{N}(y_nx_n)=\beta(N\lambda+\sum_{n=1}^{N}(x_n^2))\)

\(\beta=\frac{\sum_{n=1}^{N}(y_nx_n)}{N\lambda+\sum_{n=1}^{N}(x_n^2)}\)

3-2 Now consider the case of P = 2. The corresponding criterion is \(Q(\beta_1,\beta_2)=\frac{1}{N}\sum_{n=1}^{N}|y_n-\beta_1x_{1n}-\beta_2x_{2n}|^2+\lambda\sum_{p=1}^{P}|\beta_p|^2\) If we fix the value of \(\beta_2\) at \(\hat{\beta}_2^{(t)}\) and define \(Q_1^{(t+1)}(\beta_1)=Q(\beta_1,\hat{\beta}_2^{(t)})\) as a function of only \(\beta_1\). Please find the expression of \(\hat{\beta}_1^{(t+1)}\), the minimizer of \(Q_1^{(t)}(\beta_1)\).

\(Q(\beta_1,\hat{\beta_2}^{(t)})=\frac{1}{N}\sum_{n=1}^{N}(y_n-\beta_1x_{1n}-\hat{\beta_2}^{(t)}x_{2n})^2+\lambda(\beta_1)^2+\lambda(\hat{\beta_2}^{(t)})^2\)，求 \(\hat{\beta_1}^{(t+1)}\)，先將上式對 \(\beta_1\) 微分，

\(\frac{dQ(\beta_1,\hat{\beta_2}^{(t)})}{d\beta_1}=\frac{1}{N}\sum_{n=1}^{N}(2\beta_1x_{1n}^2-2y_nx_{1n}+2x_{1n}\hat{\beta_2}^{(t)}x_{2n})+2\lambda\beta_1=\frac{2}{N}\sum_{n=1}^{N}(x_{1n})(\beta_1x_{1n}-y_n+\hat{\beta_2}^{(t)}x_{2n})+2\lambda\beta_1\), 令該式為 0，求 \(\beta_1\)

\(\frac{2}{N}\sum_{n=1}^{N}(x_{1n})(\beta_1x_{1n}-y_n+\hat{\beta_2}^{(t)}x_{2n})+2\lambda\beta_1=0\)

\(\frac{1}{N}\sum_{n=1}^{N}(x_{1n})(\beta_1x_{1n}-y_n+\hat{\beta_2}^{(t)}x_{2n})+\lambda\beta_1=0\)，將不含 \(\beta_1\) 項移至等號左邊

\(\beta_1(\frac{\sum_{n=1}^{N}x_{1n}}{N}^2)+\lambda\beta_1=\frac{\sum_{n=1}^{N}y_nx_{1n}}{N}-\frac{\sum_{n=1}^{N}\hat{\beta_2}^{(t)}x_{2n}x_{1n}}{N}\)，同乘 N 整理 \(\beta_1\)

\(\beta_1(\sum_{n=1}^{N}x_{1n}^2+N\lambda)=\sum_{n=1}^{N}y_nx_{1n}-\sum_{n=1}^{N}\hat{\beta_2}^{(t)}x_{2n}x_{1n}\)

\(\hat{\beta_1}^{(t+1)}=\frac{\sum_{n=1}^{N}(y_nx_{1n}-\hat{\beta_2}^{(t)}x_{2n}x_{1n})}{\sum_{n=1}^{N}x_{1n}^2+N\lambda}\)

3-3 After obtaining \(\hat{\beta}_1^{(t+1)}\) , we define \(Q_2^{(t+1)}(\beta_2)=Q(\hat{\beta}_1^{(t+1)},\beta_2)\) as a function of only \(\beta_2\). What is the expression of \(\hat{\beta}_2^{(t+1)}\), the minimizer of \(Q_2^{(t+1)}(\beta_2)\)?

\(Q(\hat{\beta_1}^{(t+1)},\beta_2)=\frac{1}{N}\sum_{n=1}^{N}(y_n-\hat{\beta_1}^{(t+1)}x_{1n}-\beta_2x_{2n})^2+\lambda(\hat{\beta_1}^{(t+1)})^2+\lambda(\beta_2)^2\)

\(\frac{dQ(\hat{\beta_1}^{(t+1)},\beta_2)}{d\beta_2}=\frac{1}{N}\sum_{n=1}^{N}(2\beta_2x_{2n}^2-2y_nx_{2n}+2x_{1n}\hat{\beta_1}^{(t+1)}x_{2n})+2\lambda\beta_2=\frac{2}{N}\sum_{n=1}^{N}(x_{2n})(\beta_2x_{2n}-y_n+\hat{\beta_1}^{(t+1)}x_{1n})+2\lambda\beta_2\), 令該式為 0，求 \(\beta_2\)，因與上題類似，直接把結果套進來

\(\hat{\beta_2}^{(t+1)}=\frac{\sum_{n=1}^{N}(y_nx_{2n}-\hat{\beta_1}^{(t+1)}x_{2n}x_{1n})}{\sum_{n=1}^{N}x_{2n}^2+N\lambda}\)

3-4 Suppose that we set \(\hat{\beta_1}^{(0)}=\hat{\beta_2}^{(0)}=0\) and \(\lambda=0.1\). Please sequentially calculate the values of \(\hat{\beta_1}^{(1)}\), \(\hat{\beta_2}^{(1)}\), \(\hat{\beta_1}^{(2)}\), \(\hat{\beta_2}^{(2)}\) according to the expressions of \(\hat{\beta_1}^{(t+1)}\) and \(\hat{\beta_2}^{(t+1)}\) that you derived in 2 and 3.

\(\hat{\beta_1}^{(1)}=\frac{\sum_{n=1}^{4}(y_nx_{1n})}{\sum_{n=1}^{4}x_{1n}^2+4*0.1}=12/4.4=2.727\)

\(\hat{\beta_2}^{(1)}=\frac{\sum_{n=1}^{4}(y_nx_{2n}-\hat{\beta_1}^{(1)}x_{2n}x_{1n})}{\sum_{n=1}^{4}x_{2n}^2+4*0.1}=4/4.4=0.227\)

\(\hat{\beta_1}^{(2)}=\frac{\sum_{n=1}^{4}(y_nx_{1n}-\hat{\beta_2}^{(1)}x_{2n}x_{1n})}{\sum_{n=1}^{4}x_{1n}^2+4*0.1}=12/4.4=2.727\)

\(\hat{\beta_2}^{(2)}=\frac{\sum_{n=1}^{4}(y_nx_{2n}-\hat{\beta_1}^{(2)}x_{2n}x_{1n})}{\sum_{n=1}^{4}x_{2n}^2+4*0.1}=4/4.4=0.227\)