Exercise 1

Using a little bit of algebra, prove that (4.2) is equivalent to (4.3). In other words, the logistic function representation and logit representation for the logistic regression model are equivalent.

Exercise 4

When the number of features p is large, there tends to be a deterioration in the performance of KNN and other local approaches that perform prediction using only observations that are near the test observation for which a prediction must be made. This phenomenon is known as the curse of dimensionality, and it ties into the fact that non-parametric approaches often perform poorly when p is large. We will now investigate this curse.

4.a)

Suppose that we have a set of observations, each with measurements on p = 1 feature, X. We assume that X is uniformly (evenly) distributed on [0, 1]. Associated with each observation is a response value. Suppose that we wish to predict a test observation’s response using only observations that are within 10 % of the range of X closest to that test observation. For instance, in order to predict the response for a test observation with X = 0.6, we will use observations in the range [0.55, 0.65]. On average, what fraction of the available observations will we use to make the prediction?

4.b)

Now suppose that we have a set of observations, each with measurements on p = 2 features, X1 and X2. We assume that (X1, X2) are uniformly distributed on [0, 1] × [0, 1]. We wish to predict a test observation’s response using only observations that are within 10 % of the range of X1 and within 10 % of the range of X2 closest to that test observation. For instance, in order to predict the response for a test observation with X1 = 0.6 and X2 = 0.35, we will use observations in the range [0.55, 0.65] for X1 and in the range [0.3, 0.4] for X2. On average, what fraction of the available observations will we use to make the prediction?

Less than 0.01.

4.c)

Now suppose that we have a set of observations on p = 100 features. Again the observations are uniformly distributed on each feature, and again each feature ranges in value from 0 to 1. We wish to predict a test observation’s response using observations within the 10 % of each feature’s range that is closest to that test observation. What fraction of the available observations will we use to make the prediction?

Less than (1/10)100.

4.d)

Using your answers to parts (a)–(c), argue that a drawback of KNN when p is large is that there are very few training observations “near” any given test observation.

The larger p is, the less observations are near test observations.

4.e)

Now suppose that we wish to make a prediction for a test observation by creating a p-dimensional hypercube centered around the test observation that contains, on average, 10 % of the training observations. For p = 1, 2, and 100, what is the length of each side of the hypercube? Comment on your answer.

Exercise 5

We now examine the differences between LDA and QDA

5.a)

If the Bayes decision boundary is linear, do we expect LDA or QDA to perform better on the training set? On the test set?

We expect both to do similar on the training and test set, but depending on the data set QDA might overfit and do worse on the test set.

5.b)

*If the Bayes decision boundary is non-linear, do we expect LDA or QDA to perform better on the training set? On the test set?

We expect QDA to perform better on both due to it’s inherit properties.

5.c)

In general, as the sample size n increases, do we expect the test prediction accuracy of QDA relative to LDA to improve, decline, or be unchanged? Why?

We expect QDA accuracy to improve relative to LDA because more samples lower bias.

5.d)

True or False: Even if the Bayes decision boundary for a given problem is linear, we will probably achieve a superior test error rate using QDA rather than LDA because QDA is flexible enough to model a linear decision boundary. Justify your answer

In general True. Something to note is that the larger number of parameters and classes are, the larger data set we need. If we don’t have a data set of sufficient size, LDA might be the better choice.

Exercise 6

Suppose we collect data for a group of students in a statistics class with variables X1 = hours studied, X2 = undergrad GPA, and Y = receive an A. We fit a logistic regression and produce estimated coefficient, βˆ0 = −6, βˆ1 = 0.05, βˆ2 = 1.

6.a)

Estimate the probability that a student who studies for 40 h and has an undergrad GPA of 3.5 gets an A in the class.

6.b)

How many hours would the student in part (a) need to study to have a 50 % chance of getting an A in the class?

Exercise 7

Suppose that we wish to predict whether a given stock will issue a dividend this year (“Yes” or “No”) based on X, last year’s percent profit. We examine a large number of companies and discover that the mean value of X for companies that issued a dividend was X¯ = 10, while the mean for those that didn’t was X¯ = 0. In addition, the variance of X for these two sets of companies was ˆσ2 = 36. Finally, 80 % of companies issued dividends. Assuming that X follows a normal distribution, predict the probability that a company will issue a dividend this year given that its percentage profit was X = 4 last year.

Exercise 8

Suppose that we take a data set, divide it into equally-sized training and test sets, and then try out two different classification procedures. First we use logistic regression and get an error rate of 20 % on the training data and 30 % on the test data. Next we use 1-nearest neighbors (i.e. K = 1) and get an average error rate (averaged over both test and training data sets) of 18 %. Based on these results, which method should we prefer to use for classification of new observations? Why?

I would use the classifier that does better on the test set. In this case that is the Logistic regression.

Exercise 9

This problem has to do with odds

9.a)

On average, what fraction of people with an odds of 0.37 of defaulting on their credit card payment will in fact default?

9.b)

Suppose that an individual has a 16 % chance of defaulting on her credit card payment. What are the odds that she will default?

Exercise 10

10.a)

library(ISLR)
summary(Weekly)
##       Year           Lag1               Lag2               Lag3         
##  Min.   :1990   Min.   :-18.1950   Min.   :-18.1950   Min.   :-18.1950  
##  1st Qu.:1995   1st Qu.: -1.1540   1st Qu.: -1.1540   1st Qu.: -1.1580  
##  Median :2000   Median :  0.2410   Median :  0.2410   Median :  0.2410  
##  Mean   :2000   Mean   :  0.1506   Mean   :  0.1511   Mean   :  0.1472  
##  3rd Qu.:2005   3rd Qu.:  1.4050   3rd Qu.:  1.4090   3rd Qu.:  1.4090  
##  Max.   :2010   Max.   : 12.0260   Max.   : 12.0260   Max.   : 12.0260  
##       Lag4               Lag5              Volume            Today         
##  Min.   :-18.1950   Min.   :-18.1950   Min.   :0.08747   Min.   :-18.1950  
##  1st Qu.: -1.1580   1st Qu.: -1.1660   1st Qu.:0.33202   1st Qu.: -1.1540  
##  Median :  0.2380   Median :  0.2340   Median :1.00268   Median :  0.2410  
##  Mean   :  0.1458   Mean   :  0.1399   Mean   :1.57462   Mean   :  0.1499  
##  3rd Qu.:  1.4090   3rd Qu.:  1.4050   3rd Qu.:2.05373   3rd Qu.:  1.4050  
##  Max.   : 12.0260   Max.   : 12.0260   Max.   :9.32821   Max.   : 12.0260  
##  Direction 
##  Down:484  
##  Up  :605  
##            
##            
##            
## 
pairs(Weekly, col = c('blue', 'red')[unclass(Weekly$Direction)],
      horInd = 1:(ncol(Weekly) - 2), verInd = 1:(ncol(Weekly) - 2))

We can observe that year and volume are correlated, and the chart below confirms that. Nothing else jumps out.

library(corrplot)
## corrplot 0.84 loaded
corrplot(cor(Weekly[, 1:(ncol(Weekly) - 1)]), type = "upper", tl.col = "black",
         tl.srt = 45)

glm.fit = glm(Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + Volume,
              data = Weekly, family = binomial)
summary(glm.fit)
## 
## Call:
## glm(formula = Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 + 
##     Volume, family = binomial, data = Weekly)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.6949  -1.2565   0.9913   1.0849   1.4579  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)   
## (Intercept)  0.26686    0.08593   3.106   0.0019 **
## Lag1        -0.04127    0.02641  -1.563   0.1181   
## Lag2         0.05844    0.02686   2.175   0.0296 * 
## Lag3        -0.01606    0.02666  -0.602   0.5469   
## Lag4        -0.02779    0.02646  -1.050   0.2937   
## Lag5        -0.01447    0.02638  -0.549   0.5833   
## Volume      -0.02274    0.03690  -0.616   0.5377   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1496.2  on 1088  degrees of freedom
## Residual deviance: 1486.4  on 1082  degrees of freedom
## AIC: 1500.4
## 
## Number of Fisher Scoring iterations: 4
glm.pred <- ifelse(predict(glm.fit, type = "response") > 0.5, "Up", "Down")
library(httk)
fourfoldplot(table(glm.pred, Weekly$Direction), color = c("#CC6666", "#99CC99"),
             conf.level = 0)

testSplit = (Weekly$Year >= 2009)

glm.fit = glm(Direction ~ Lag2, data = Weekly[!testSplit, ], family = binomial)
glm.pred <- ifelse(
  predict(glm.fit, Weekly[testSplit, ], type = "response") > 0.5, "Up", "Down")
fourfoldplot(table(glm.pred, Weekly[testSplit, ]$Direction),
             color = c("#CC6666", "#99CC99"), conf.level = 0)

library(MASS)
lda.fit = lda(Direction ~ Lag2, data = Weekly[!testSplit, ])
lda.pred <- predict(lda.fit, Weekly[testSplit, ])$class
fourfoldplot(table(lda.pred, Weekly[testSplit, ]$Direction),
             color = c("#CC6666", "#99CC99"), conf.level = 0)

qda.fit = qda(Direction ~ Lag2, data = Weekly[!testSplit, ])
qda.pred <- predict(qda.fit, Weekly[testSplit, ])$class
fourfoldplot(table(qda.pred, Weekly[testSplit, ]$Direction),
             color = c("#CC6666", "#99CC99"), conf.level = 0)

library(class)
fourfoldplot(
  table(
    knn(
      as.matrix(Weekly$Lag2[!testSplit]),
      as.matrix(Weekly$Lag2[testSplit]),
      Weekly$Direction[!testSplit],
      k = 1),
    Weekly[testSplit, ]$Direction),
  color = c("#99CC99", "#CC6666"),
  conf.level = 0)

We can observe that logistic regression and LDA perform best on this problem, with KNN being close. QDA on the other hand classifies all samples in the “Up” class and does not do very well on this problem.

Exercise 12

12.a), 12.b), 12.c) and 12.d)

Power = function(x, n){
  return(x ^ n)
}
list(Power(2, 3), Power(3, 8), Power(10, 3), Power(8, 17), Power(131, 3))
## [[1]]
## [1] 8
## 
## [[2]]
## [1] 6561
## 
## [[3]]
## [1] 1000
## 
## [[4]]
## [1] 2.2518e+15
## 
## [[5]]
## [1] 2248091

12.e) and 12.f)

PlotPower = function(numList, pow) {
  plot(numList, Power(numList, pow), log = "y")
}
PlotPower(1:10, 2)

PlotPower(1:10, 3)