Question 1

Let \(Y=e^{\beta_0 + \beta_1 X}\), we have: \[ p(X) = \frac{Y}{1 + Y} \\ \therefore p(X) + Yp(X) = Y \\ p(X) = [1 - p(X)] Y \\ \frac{p(X)}{1 - p(X)} = Y = e^{\beta_0 + \beta_1 X} \]

Question 2

When proving the same \(k\) produce both maximum \(p_k(x)\) and maximum of \(\delta_k(x)\), we have assumed that \(\sigma_1^2=\dots=\sigma_K^2\), the only variable is \(k\) in equation (4.12) and (4.13). So let \[ \frac { \frac {1} {\sqrt{2 \pi} \sigma} \exp(- \frac {1} {2 \sigma^2} x^2) } {\sum_{l=1}^K { \pi_l \frac {1} {\sqrt{2 \pi} \sigma} \exp(- \frac {1} {2 \sigma^2} (x - \mu_l)^2) }} = C \]

Take it into equation (4.12) and (4.13), we have: \[ p_k(x) = \exp (x \frac {\mu_k} {\sigma^2} - \frac {\mu_k^2} {2 \sigma^2}) \pi_k C \\ \therefore log(p_k(x)) = x\frac{\mu_k}{\sigma^2} - \frac{\mu_k^2}{2\sigma^2} + log(\pi_k) + log(C) = \delta_k(x) + log(C) \]

For logarithm function is monotonically increasing, when \(\delta_k(x)\) get its maximum, \(p_k(x)\) get its maximum, too.

Question 3

Like above question, but without assumption that \(\sigma_1^2 = \dots = \sigma_K^2\), let: \[ \frac {\frac {1} {\sqrt{2 \pi}}} {\sum_{l=1}^K { \pi_l \frac {1} {\sqrt{2 \pi} \sigma} \exp(- \frac {1} {2 \sigma^2} (x - \mu_l)^2) }} = C \]

Take it into equation (4.12) and (4.13), we have: \[ p_k(x) = \exp(-\frac{(x - \mu_k)^2}{2 \sigma_k^2}) \frac{\pi_k}{\sigma_k} C \\ \therefore log(p_k(x)) = \delta_k(x) = -\frac{(x - \mu_k)^2}{2 \sigma_k^2} + log(\frac{\pi_k}{\sigma_k}) + log(C) \]

So \(\delta_k(x)\) is a quadratic function of \(x\).

Question 4

4a ~ 4d

0.1 (10% in other words), 0.01, and \(10^{-100}\). As the increase of \(p\), the near points decrease exponentially.

4e

The length of each side for \(p\) dimensional hypercube is \(0.1^{\frac1p}\).

Question 5

5a: When the Bayes decision boundary is linear, QDA performs better than QDA on training set. LDA performs better than QDA on test set.

5b: When the Bayes boundary is non-linear, QDA performs better than LDA on both training and test sets.

5c: When \(n\) increase, QDA predicts more accurately than LDA, because the bias of QDA decrease faster than LDA. See the first paragraph of page 150 for reference.

5d: False. The bias of QDA can be smaller than LDA, which produces higher variance than LDA. The higher variance produces higher error rate in test data. See figure 4.9 for reference.

Question 6

According to equation (4.7): \[ p(X) = \frac{e^{\beta_0 + \beta_1X_1 + \beta_2X_2}}{1 + e^{\beta_0 + \beta_1X_1 + \beta_2X_2}} \\ = \frac{e^{-6 + 0.05 \times 40 + 3.5}}{1 + e^{-6 + 0.05 \times 40 + 3.5}} \\ = 0.378 \]

Let the hours needed to study as \(h\), we have: \[ p(X) = \frac{e^{-6 + 0.05h + 3.5}}{1 + e^{-6 + 0.05h + 3.5}} = 0.5 \\ \Rightarrow -6 + 0.05h + 3.5 = 0 \\ h = 50 \]

The student need 50 hours to have a 50% chance of getting an A in the class.

Question 7

Let 1 denotes “Yes” and 2 denotes "No’, with equation (4.15) we have: \(\mu_1 = 10, \mu_2 = 0, \pi_1 = 0.8, \pi_2 = 0.2, \sigma^2 = 36\). Take them into equation (4.12) with \(x = 4\):

item1 = 0.8 / sqrt(2*pi) / 6 * exp(-1 / (2 * 36) * (4 - 10)^2)
item2 = 0.2 / sqrt(2*pi) / 6 * exp(-1 / (2 * 36) * (4 - 0)^2)
p1x = item1 / (item1 + item2)
p1x
[1] 0.7518525

The probability issuing a dividend this year is 75.2%.

Question 8

For KNN with \(k = 1\), the training error rate is 0%, because for any training observation, the response is the nearest predictor itself. So the test error is 36% for KNN, which is higher than that of logistic regression (30%). So far the latter is better. But I prefer using higer \(k\) value to find out better solutions.

Question 9

9a

Let \(p\) denotes the default probability, we have: \[ \frac{p}{1-p} = 0.37 \Rightarrow p = \frac{0.37}{1+0.37} = 0.27 \]

9b

The odds of default is: \[ \frac{0.16}{1-0.16} = 0.19 \]

LS0tCnRpdGxlOiAiQ29uY2VwdHVhbCBFeGVyY2lzZXMgb2YgQ2hhcHRlciA0IgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgojIFF1ZXN0aW9uIDEKCkxldCAkWT1lXntcYmV0YV8wICsgXGJldGFfMSBYfSQsIHdlIGhhdmU6CiQkCnAoWCkgPSBcZnJhY3tZfXsxICsgWX0gXFwKXHRoZXJlZm9yZSBwKFgpICsgWXAoWCkgPSBZIFxcCnAoWCkgPSBbMSAtIHAoWCldIFkgXFwKXGZyYWN7cChYKX17MSAtIHAoWCl9ID0gWSA9IGVee1xiZXRhXzAgKyBcYmV0YV8xIFh9CiQkCgojIFF1ZXN0aW9uIDIKCldoZW4gcHJvdmluZyB0aGUgc2FtZSAkayQgcHJvZHVjZSBib3RoIG1heGltdW0gJHBfayh4KSQgYW5kIG1heGltdW0gb2YgJFxkZWx0YV9rKHgpJCwKd2UgaGF2ZSBhc3N1bWVkIHRoYXQgJFxzaWdtYV8xXjI9XGRvdHM9XHNpZ21hX0teMiQsIHRoZSBvbmx5IHZhcmlhYmxlIGlzICRrJCBpbiBlcXVhdGlvbiAoNC4xMikgYW5kICg0LjEzKS4gU28gbGV0CiQkClxmcmFjIHsgXGZyYWMgezF9IHtcc3FydHsyIFxwaX0gXHNpZ21hfSBcZXhwKC0gXGZyYWMgezF9IHsyIFxzaWdtYV4yfSB4XjIpIH0ge1xzdW1fe2w9MX1eSyB7IFxwaV9sIFxmcmFjIHsxfSB7XHNxcnR7MiBccGl9IFxzaWdtYX0gXGV4cCgtIFxmcmFjIHsxfSB7MiBcc2lnbWFeMn0gKHggLSBcbXVfbCleMikgfX0gPSBDCiQkCgpUYWtlIGl0IGludG8gZXF1YXRpb24gKDQuMTIpIGFuZCAoNC4xMyksIHdlIGhhdmU6CiQkCnBfayh4KSA9IFxleHAgKHggXGZyYWMge1xtdV9rfSB7XHNpZ21hXjJ9IC0gXGZyYWMge1xtdV9rXjJ9IHsyIFxzaWdtYV4yfSkgXHBpX2sgQyBcXApcdGhlcmVmb3JlCmxvZyhwX2soeCkpCj0geFxmcmFje1xtdV9rfXtcc2lnbWFeMn0gLSBcZnJhY3tcbXVfa14yfXsyXHNpZ21hXjJ9ICsgbG9nKFxwaV9rKSArIGxvZyhDKQo9IFxkZWx0YV9rKHgpICsgbG9nKEMpCiQkCgpGb3IgbG9nYXJpdGhtIGZ1bmN0aW9uIGlzIG1vbm90b25pY2FsbHkgaW5jcmVhc2luZywgd2hlbiAkXGRlbHRhX2soeCkkIGdldCBpdHMgbWF4aW11bSwKJHBfayh4KSQgZ2V0IGl0cyBtYXhpbXVtLCB0b28uCgojIFF1ZXN0aW9uIDMKCkxpa2UgYWJvdmUgcXVlc3Rpb24sIGJ1dCB3aXRob3V0IGFzc3VtcHRpb24gdGhhdCAkXHNpZ21hXzFeMiA9IFxkb3RzID0gXHNpZ21hX0teMiQsCmxldDoKJCQKXGZyYWMge1xmcmFjIHsxfSB7XHNxcnR7MiBccGl9fX0ge1xzdW1fe2w9MX1eSyB7IFxwaV9sIFxmcmFjIHsxfSB7XHNxcnR7MiBccGl9IFxzaWdtYX0gXGV4cCgtIFxmcmFjIHsxfSB7MiBcc2lnbWFeMn0gKHggLSBcbXVfbCleMikgfX0gPSBDCiQkCgpUYWtlIGl0IGludG8gZXF1YXRpb24gKDQuMTIpIGFuZCAoNC4xMyksIHdlIGhhdmU6CiQkCnBfayh4KSA9IFxleHAoLVxmcmFjeyh4IC0gXG11X2spXjJ9ezIgXHNpZ21hX2teMn0pIFxmcmFje1xwaV9rfXtcc2lnbWFfa30gQyBcXApcdGhlcmVmb3JlCmxvZyhwX2soeCkpID0gXGRlbHRhX2soeCkKPSAtXGZyYWN7KHggLSBcbXVfayleMn17MiBcc2lnbWFfa14yfSArIGxvZyhcZnJhY3tccGlfa317XHNpZ21hX2t9KSArIGxvZyhDKQokJAoKU28gJFxkZWx0YV9rKHgpJCBpcyBhIHF1YWRyYXRpYyBmdW5jdGlvbiBvZiAkeCQuCgojIFF1ZXN0aW9uIDQKCiMjIDRhIH4gNGQKMC4xICgxMCUgaW4gb3RoZXIgd29yZHMpLCAwLjAxLCBhbmQgJDEwXnstMTAwfSQuCkFzIHRoZSBpbmNyZWFzZSBvZiAkcCQsIHRoZSBuZWFyIHBvaW50cyBkZWNyZWFzZSBleHBvbmVudGlhbGx5LgoKIyMgNGUKVGhlIGxlbmd0aCBvZiBlYWNoIHNpZGUgZm9yICRwJCBkaW1lbnNpb25hbCBoeXBlcmN1YmUgaXMgJDAuMV57XGZyYWMxcH0kLgoKIyBRdWVzdGlvbiA1Cgo1YTogV2hlbiB0aGUgQmF5ZXMgZGVjaXNpb24gYm91bmRhcnkgaXMgbGluZWFyLCBRREEgcGVyZm9ybXMgYmV0dGVyIHRoYW4gUURBIG9uIHRyYWluaW5nIHNldC4KTERBIHBlcmZvcm1zIGJldHRlciB0aGFuIFFEQSBvbiB0ZXN0IHNldC4KCjViOiBXaGVuIHRoZSBCYXllcyBib3VuZGFyeSBpcyBub24tbGluZWFyLCBRREEgcGVyZm9ybXMgYmV0dGVyIHRoYW4gTERBIG9uIGJvdGggdHJhaW5pbmcgYW5kIHRlc3Qgc2V0cy4KCjVjOiBXaGVuICRuJCBpbmNyZWFzZSwgUURBIHByZWRpY3RzIG1vcmUgYWNjdXJhdGVseSB0aGFuIExEQSwgYmVjYXVzZSB0aGUgYmlhcyBvZiBRREEgZGVjcmVhc2UgZmFzdGVyIHRoYW4gTERBLgpTZWUgdGhlIGZpcnN0IHBhcmFncmFwaCBvZiBwYWdlIDE1MCBmb3IgcmVmZXJlbmNlLgoKNWQ6IEZhbHNlLiBUaGUgYmlhcyBvZiBRREEgY2FuIGJlIHNtYWxsZXIgdGhhbiBMREEsIHdoaWNoIHByb2R1Y2VzIGhpZ2hlciB2YXJpYW5jZSB0aGFuIExEQS4KVGhlIGhpZ2hlciB2YXJpYW5jZSBwcm9kdWNlcyBoaWdoZXIgZXJyb3IgcmF0ZSBpbiB0ZXN0IGRhdGEuIFNlZSBmaWd1cmUgNC45IGZvciByZWZlcmVuY2UuCgojIFF1ZXN0aW9uIDYKCkFjY29yZGluZyB0byBlcXVhdGlvbiAoNC43KToKJCQKcChYKSA9IFxmcmFje2Vee1xiZXRhXzAgKyBcYmV0YV8xWF8xICsgXGJldGFfMlhfMn19ezEgKyBlXntcYmV0YV8wICsgXGJldGFfMVhfMSArIFxiZXRhXzJYXzJ9fSBcXAo9IFxmcmFje2Veey02ICsgMC4wNSBcdGltZXMgNDAgKyAzLjV9fXsxICsgZV57LTYgKyAwLjA1IFx0aW1lcyA0MCArIDMuNX19IFxcCj0gMC4zNzgKJCQKCkxldCB0aGUgaG91cnMgbmVlZGVkIHRvIHN0dWR5IGFzICRoJCwgd2UgaGF2ZToKJCQKcChYKSA9IFxmcmFje2Veey02ICsgMC4wNWggKyAzLjV9fXsxICsgZV57LTYgKyAwLjA1aCArIDMuNX19ID0gMC41IFxcClxSaWdodGFycm93IC02ICsgMC4wNWggKyAzLjUgPSAwIFxcCmggPSA1MAokJAoKVGhlIHN0dWRlbnQgbmVlZCA1MCBob3VycyB0byBoYXZlIGEgNTAlIGNoYW5jZSBvZiBnZXR0aW5nIGFuIEEgaW4gdGhlIGNsYXNzLgoKIyBRdWVzdGlvbiA3CgpMZXQgMSBkZW5vdGVzICJZZXMiIGFuZCAyIGRlbm90ZXMgIk5vJywgd2l0aCBlcXVhdGlvbiAoNC4xNSkgd2UgaGF2ZTogJFxtdV8xID0gMTAsIFxtdV8yID0gMCwgXHBpXzEgPSAwLjgsIFxwaV8yID0gMC4yLCBcc2lnbWFeMiA9IDM2JC4gClRha2UgdGhlbSBpbnRvIGVxdWF0aW9uICg0LjEyKSB3aXRoICR4ID0gNCQ6CmBgYHtyfQppdGVtMSA9IDAuOCAvIHNxcnQoMipwaSkgLyA2ICogZXhwKC0xIC8gKDIgKiAzNikgKiAoNCAtIDEwKV4yKQppdGVtMiA9IDAuMiAvIHNxcnQoMipwaSkgLyA2ICogZXhwKC0xIC8gKDIgKiAzNikgKiAoNCAtIDApXjIpCnAxeCA9IGl0ZW0xIC8gKGl0ZW0xICsgaXRlbTIpCnAxeApgYGAKClRoZSBwcm9iYWJpbGl0eSBpc3N1aW5nIGEgZGl2aWRlbmQgdGhpcyB5ZWFyIGlzIDc1LjIlLgoKIyBRdWVzdGlvbiA4CkZvciBLTk4gd2l0aCAkayA9IDEkLCB0aGUgdHJhaW5pbmcgZXJyb3IgcmF0ZSBpcyAwJSwgYmVjYXVzZSBmb3IgYW55IHRyYWluaW5nIG9ic2VydmF0aW9uLCB0aGUgcmVzcG9uc2UgaXMgdGhlIG5lYXJlc3QgcHJlZGljdG9yIGl0c2VsZi4gU28gdGhlIHRlc3QgZXJyb3IgaXMgMzYlIGZvciBLTk4sIHdoaWNoIGlzIGhpZ2hlciB0aGFuIHRoYXQgb2YgbG9naXN0aWMgcmVncmVzc2lvbiAoMzAlKS4KU28gZmFyIHRoZSBsYXR0ZXIgaXMgYmV0dGVyLiBCdXQgSSBwcmVmZXIgdXNpbmcgaGlnZXIgJGskIHZhbHVlIHRvIGZpbmQgb3V0IGJldHRlciBzb2x1dGlvbnMuCgojIFF1ZXN0aW9uIDkKIyMgOWEKTGV0ICRwJCBkZW5vdGVzIHRoZSBkZWZhdWx0IHByb2JhYmlsaXR5LCB3ZSBoYXZlOgokJApcZnJhY3twfXsxLXB9ID0gMC4zNyBcUmlnaHRhcnJvdyBwID0gXGZyYWN7MC4zN317MSswLjM3fSA9IDAuMjcKJCQKCiMjIDliClRoZSBvZGRzIG9mIGRlZmF1bHQgaXM6CiQkClxmcmFjezAuMTZ9ezEtMC4xNn0gPSAwLjE5CiQk