Assignment #3
Jared Chappelle
2022-09-25
Problem 13
This question should be answered using the Weekly data set, which is part of the ISLR2 package. This data is similar in nature to the Smarket data from this chapter’s lab, except that it contains 1, 089 weekly returns for 21 years, from the beginning of 1990 to the end of 2010.
(a) Produce some numerical and graphical summaries of the Weekly data. Do there appear to be any patterns?
summary(Weekly)## Year Lag1 Lag2 Lag3
## Min. :1990 Min. :-18.1950 Min. :-18.1950 Min. :-18.1950
## 1st Qu.:1995 1st Qu.: -1.1540 1st Qu.: -1.1540 1st Qu.: -1.1580
## Median :2000 Median : 0.2410 Median : 0.2410 Median : 0.2410
## Mean :2000 Mean : 0.1506 Mean : 0.1511 Mean : 0.1472
## 3rd Qu.:2005 3rd Qu.: 1.4050 3rd Qu.: 1.4090 3rd Qu.: 1.4090
## Max. :2010 Max. : 12.0260 Max. : 12.0260 Max. : 12.0260
## Lag4 Lag5 Volume Today
## Min. :-18.1950 Min. :-18.1950 Min. :0.08747 Min. :-18.1950
## 1st Qu.: -1.1580 1st Qu.: -1.1660 1st Qu.:0.33202 1st Qu.: -1.1540
## Median : 0.2380 Median : 0.2340 Median :1.00268 Median : 0.2410
## Mean : 0.1458 Mean : 0.1399 Mean :1.57462 Mean : 0.1499
## 3rd Qu.: 1.4090 3rd Qu.: 1.4050 3rd Qu.:2.05373 3rd Qu.: 1.4050
## Max. : 12.0260 Max. : 12.0260 Max. :9.32821 Max. : 12.0260
## Direction
## Down:484
## Up :605
##
##
##
## pairs(Weekly)
Here, the only plots that look important are the two year and volume ones. These two variables seem to have a relationship whereas the other variables don’t.
cor(Weekly[,-9])## Year Lag1 Lag2 Lag3 Lag4
## Year 1.00000000 -0.032289274 -0.03339001 -0.03000649 -0.031127923
## Lag1 -0.03228927 1.000000000 -0.07485305 0.05863568 -0.071273876
## Lag2 -0.03339001 -0.074853051 1.00000000 -0.07572091 0.058381535
## Lag3 -0.03000649 0.058635682 -0.07572091 1.00000000 -0.075395865
## Lag4 -0.03112792 -0.071273876 0.05838153 -0.07539587 1.000000000
## Lag5 -0.03051910 -0.008183096 -0.07249948 0.06065717 -0.075675027
## Volume 0.84194162 -0.064951313 -0.08551314 -0.06928771 -0.061074617
## Today -0.03245989 -0.075031842 0.05916672 -0.07124364 -0.007825873
## Lag5 Volume Today
## Year -0.030519101 0.84194162 -0.032459894
## Lag1 -0.008183096 -0.06495131 -0.075031842
## Lag2 -0.072499482 -0.08551314 0.059166717
## Lag3 0.060657175 -0.06928771 -0.071243639
## Lag4 -0.075675027 -0.06107462 -0.007825873
## Lag5 1.000000000 -0.05851741 0.011012698
## Volume -0.058517414 1.00000000 -0.033077783
## Today 0.011012698 -0.03307778 1.000000000This correlation matrix confirms what I said about the pairs. The volume and year variables have a positive correlation but the other variables don’t seem to have any meaningful correlation.
(b) Use the full data set to perform a logistic regression with Direction as the response and the five lag variables plus Volume as predictors. Use the summary function to print the results. Do any of the predictors appear to be statistically significant? If so, which ones?
glm.fit=glm(Direction~Lag1+Lag2+Lag3+Lag4+Lag5+Volume,data=Weekly,family=binomial)
summary(glm.fit)##
## Call:
## glm(formula = Direction ~ Lag1 + Lag2 + Lag3 + Lag4 + Lag5 +
## Volume, family = binomial, data = Weekly)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.6949 -1.2565 0.9913 1.0849 1.4579
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 0.26686 0.08593 3.106 0.0019 **
## Lag1 -0.04127 0.02641 -1.563 0.1181
## Lag2 0.05844 0.02686 2.175 0.0296 *
## Lag3 -0.01606 0.02666 -0.602 0.5469
## Lag4 -0.02779 0.02646 -1.050 0.2937
## Lag5 -0.01447 0.02638 -0.549 0.5833
## Volume -0.02274 0.03690 -0.616 0.5377
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1496.2 on 1088 degrees of freedom
## Residual deviance: 1486.4 on 1082 degrees of freedom
## AIC: 1500.4
##
## Number of Fisher Scoring iterations: 4Yes, the only one that appears to be statistically significant is Lag2. Ideally we would want the p-value to be lower though.
(c) Compute the confusion matrix and overall fraction of correct predictions. Explain what the confusion matrix is telling you about the types of mistakes made by logistic regression.
glm.probs=predict(glm.fit,type='response')
glm.pred=rep("Down",length(glm.probs))
glm.pred[glm.probs>.5]="Up"
table(glm.pred, Direction) ## Direction
## glm.pred Down Up
## Down 54 48
## Up 430 557(54+557)/1089## [1] 0.5610652The logistic regression model we used was correct about 56% of the time on whether or not the market would go up or down. We guessed incorrectly 48 times that the market would go down when it actually went up, and incorrectly 430 times that it would go up when it actually went down. It seems that our model guesses much more accurately when the market will go up versus when it will go down.
(d) Now fit the logistic regression model using a training data period from 1990 to 2008, with Lag2 as the only predictor. Compute the confusion matrix and the overall fraction of correct predictions for the held out data (that is, the data from 2009 and 2010).
train=(Year<2009)
Weekly.09and10=Weekly[!train,]
Direction.09and10=Direction[!train]
glm.fits=glm(Direction~Lag2, data=Weekly,subset=train,family=binomial)
glm.probss=predict(glm.fits,Weekly.09and10,type='response')
glm.preds=rep('Down',length(glm.probss))
glm.preds[glm.probss>.5]='Up'
table(glm.preds,Direction.09and10)## Direction.09and10
## glm.preds Down Up
## Down 9 5
## Up 34 56(56+9)/104## [1] 0.625For the held out years of 2009 and 2010, the model was correct 62.5%
of the time which is a big step up than it was for all of the data
set.
(e) Repeat (d) using LDA.
library(MASS)
lda.fit=lda(Direction~Lag2,data=Weekly,subset=train)
lda.pred=predict(lda.fit, Weekly.09and10)
table(lda.pred$class,Direction.09and10)## Direction.09and10
## Down Up
## Down 9 5
## Up 34 56(56+9)/104## [1] 0.625The results here are the same as they were with logistic regression.
(f) Repeat (d) using QDA.
qda.fit=qda(Direction~Lag2,data=Weekly,subset=train)
qda.pred=predict(qda.fit,Weekly.09and10)$class
table(qda.pred,Direction.09and10)## Direction.09and10
## qda.pred Down Up
## Down 0 0
## Up 43 6161/104## [1] 0.5865385QDA was less accurate than logistic regression and LDA, only being correct 58.7% of the time. QDA only predicted Up on every single observation though.
(g) Repeat (d) using KNN with K = 1.
library(class)
train.Weekly=as.matrix(Lag2[train])
test.Weekly=as.matrix(Lag2[!train])
train.Direction=Direction[train]
set.seed(5)
knn.Weekly=knn(train.Weekly,test.Weekly,train.Direction,k=1)
table(knn.Weekly,Direction.09and10)## Direction.09and10
## knn.Weekly Down Up
## Down 21 30
## Up 22 31(31+21)/104## [1] 0.5KNN predicted which way the market would go correctly 50% of the
time. This is the least accurate model so far, and we might as well do a
coin flip.
(h) Repeat (d) using naive Bayes.
library(e1071)
bayes.weekly=naiveBayes(Direction~Lag2,data=Weekly,subset=train)
bayes.pred=predict(bayes.weekly,Weekly.09and10)
table(bayes.pred, Direction.09and10)## Direction.09and10
## bayes.pred Down Up
## Down 0 0
## Up 43 6161/104## [1] 0.5865385The naive Bayes classifier results turned out to be the same as the results for QDA, it was accurate 58.7% of the time, but only predicted up each time.
(i) Which of these methods appears to provide the best
results on this data?
LDA and logistic regression both appear to provide the best results on
this data. Both predict which way the market will go correctly 62.5% of
the time.
(j) Experiment with different combinations of predictors, including possible transformations and interactions, for each of the methods. Report the variables, method, and associated confusion matrix that appears to provide the best results on the held out data. Note that you should also experiment with values for K in the KNN classifier.
#Logistic Regression with interaction Lag2:Volume
train=(Year<2009)
Weekly.09and10=Weekly[!train,]
Direction.09and10=Direction[!train]
glm.fits2=glm(Direction~Lag2:Volume+Lag2, data=Weekly,subset=train,family=binomial)
glm.probs2=predict(glm.fits2,Weekly.09and10,type='response')
glm.preds2=rep('Down',length(glm.probs2))
glm.preds2[glm.probs2>.5]='Up'
table(glm.preds2,Direction.09and10)## Direction.09and10
## glm.preds2 Down Up
## Down 9 5
## Up 34 56(56+9)/104## [1] 0.625This model is essentially the same as it was without the volume. The confusion matrix is the same as it was in (d). Next I will try the interaction between Lag2 and Lag1 in addition to Lag2.
#Logistic Regression with interaction Lag2:Lag1
train=(Year<2009)
Weekly.09and10=Weekly[!train,]
Direction.09and10=Direction[!train]
glm.fits3=glm(Direction~Lag2:Lag1+Lag2, data=Weekly,subset=train,family=binomial)
glm.probs3=predict(glm.fits3,Weekly.09and10,type='response')
glm.preds3=rep('Down',length(glm.probs3))
glm.preds3[glm.probs3>.5]='Up'
table(glm.preds3,Direction.09and10)## Direction.09and10
## glm.preds3 Down Up
## Down 3 3
## Up 40 58(58+3)/104## [1] 0.5865385This time, the model slightly less accurate, only predicting the direction correctly about 58.7% of the time.
#LDA with interaction Lag2:Lag1
lda.fit2=lda(Direction~Lag2:Lag1+Lag2,data=Weekly,subset=train)
lda.pred2=predict(lda.fit2, Weekly.09and10)
table(lda.pred2$class,Direction.09and10)## Direction.09and10
## Down Up
## Down 3 3
## Up 40 58(58+3)/104## [1] 0.5865385As before, the LDA results with are the same as the logistic regression results.
#QDA with interaction Lag2:Lag1
qda.fit2=qda(Direction~Lag2:Lag1+Lag2,data=Weekly,subset=train)
qda.pred2=predict(qda.fit2,Weekly.09and10)$class
table(qda.pred2,Direction.09and10)## Direction.09and10
## qda.pred2 Down Up
## Down 24 37
## Up 19 24(24+24)/104## [1] 0.4615385This QDA model is now worse than a coin flip, predicting Direction correctly on 46.2% of the time.
#Same KNN as in (g) but with k=10
train.Weekly=as.matrix(Lag2[train])
test.Weekly=as.matrix(Lag2[!train])
train.Direction=Direction[train]
set.seed(5)
knn.Weekly2=knn(train.Weekly,test.Weekly,train.Direction,k=10)
table(knn.Weekly2,Direction.09and10)## Direction.09and10
## knn.Weekly2 Down Up
## Down 17 20
## Up 26 41(41+17)/104## [1] 0.5576923It seems that with k=10, this KNN classifier model improved, going from correctly predicting prediction 50% of the time to 55.8% of the time.
#Same KNN as in (g) but with k=20
train.Weekly=as.matrix(Lag2[train])
test.Weekly=as.matrix(Lag2[!train])
train.Direction=Direction[train]
set.seed(5)
knn.Weekly3=knn(train.Weekly,test.Weekly,train.Direction,k=20)
table(knn.Weekly3,Direction.09and10)## Direction.09and10
## knn.Weekly3 Down Up
## Down 20 22
## Up 23 39(39+20)/104## [1] 0.5673077With K=20, the model improved again to being correct 56.7% of the time.
#Same KNN as in (g) but with k=25
train.Weekly=as.matrix(Lag2[train])
test.Weekly=as.matrix(Lag2[!train])
train.Direction=Direction[train]
set.seed(5)
knn.Weekly4=knn(train.Weekly,test.Weekly,train.Direction,k=25)
table(knn.Weekly4,Direction.09and10)## Direction.09and10
## knn.Weekly4 Down Up
## Down 20 24
## Up 23 37(37+20)/104## [1] 0.5480769With K=20, the KNN classifier worsened, going down to being correct 54.8% of the time.
It seems that the logistic regression and LDA models we had originally used were the most accurate.
Problem 14
In this problem, you will develop a model to predict whether a given car gets high or low gas mileage based on the Auto data set.
(a) Create a binary variable, mpg01, that contains a 1 if mpg contains a value above its median, and a 0 if mpg contains a value below its median. You can compute the median using the median() function. Note you may find it helpful to use the data.frame() function to create a single data set containing both mpg01 and the other Auto variables.
detach(Weekly)
attach(Auto)
median(mpg)## [1] 22.75mpg01=rep(0,length(mpg))
mpg01[mpg>22.75]=1
Auto=data.frame(Auto,mpg01)(b) Explore the data graphically in order to investigate the association between mpg01 and the other features. Which of the other features seem most likely to be useful in predicting mpg01? Scatterplots and boxplots may be useful tools to answer this question. Describe your findings.
pairs(Auto[,-9])
It is difficult to tell anything here with the plots function, mpg01 can only be 0 or 1. It does seem like weight, acceleration, and horsepower seem to have some sort of relationship with mpg01.
plot(mpg01,weight)
With this plot, vehicles with higher weights tend to have mpg’s that are under 22.75, and vehicles with lower weights typically have mpg’s that are above 22.75.
(c) Split the data into a training set and a test set.
idx=sample(1:nrow(Auto),0.75*nrow(Auto))
train.Auto=Auto[idx,]
dim(train.Auto)## [1] 294 10test.Auto=Auto[-idx,]
dim(test.Auto)## [1] 98 10(d) Perform LDA on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained?
lda.Auto=lda(mpg01~weight+acceleration+horsepower,data=train.Auto)
pred.Auto=predict(lda.Auto,test.Auto)
table(pred.Auto$class,test.Auto$mpg01)##
## 0 1
## 0 39 3
## 1 5 51(3+5)/98## [1] 0.08163265The test error of this LDA model is 8.16%.
(e) Perform QDA on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained?
qda.Auto=qda(mpg01~weight+acceleration+horsepower,data=train.Auto)
pred2.Auto=predict(qda.Auto,test.Auto)
table(pred2.Auto$class,test.Auto$mpg01)##
## 0 1
## 0 40 5
## 1 4 49(4+5)/98## [1] 0.09183673The test error of this QDA model is 9.18%.
(f) Perform logistic regression on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01 in (b). What is the test error of the model obtained?
glm.Auto=glm(mpg01~weight+acceleration+horsepower,data=train.Auto,family=binomial)
probs3.Auto=predict(glm.Auto,test.Auto,type="response")
pred3.Auto=rep(0,length(probs3.Auto))
pred3.Auto[probs3.Auto>.5]=1
table(pred3.Auto,test.Auto$mpg01)##
## pred3.Auto 0 1
## 0 41 5
## 1 3 49(3+5)/98## [1] 0.08163265The test error for this logistic regression model is 8.16%.
(g) Perform naive Bayes on the training data in order to predict mpg01 using the variables that seemed most associated with mpg01in (b). What is the test error of the model obtained?
bayes.Auto=naiveBayes(mpg01~weight+acceleration+horsepower,data=train.Auto)
pred4.bayes=predict(bayes.Auto,test.Auto)
table(pred4.bayes,test.Auto$mpg01)##
## pred4.bayes 0 1
## 0 39 3
## 1 5 51(5+3)/98## [1] 0.08163265The naive Bayes test error is 8.16%.
Problem 16
Using the Boston data set, fit classification models in order
to predict whether a given census tract has a crime rate above or below
the median. Explore logistic regression, LDA, naive Bayes, and KNN
models using various subsets of the predictors. Describe your
findings.
Hint: You will have to create the response variable yourself, using the
variables that are contained in the Boston data set.
#Summary of the Boston Data Set
summary(Boston)## crim zn indus chas
## Min. : 0.00632 Min. : 0.00 Min. : 0.46 Min. :0.00000
## 1st Qu.: 0.08205 1st Qu.: 0.00 1st Qu.: 5.19 1st Qu.:0.00000
## Median : 0.25651 Median : 0.00 Median : 9.69 Median :0.00000
## Mean : 3.61352 Mean : 11.36 Mean :11.14 Mean :0.06917
## 3rd Qu.: 3.67708 3rd Qu.: 12.50 3rd Qu.:18.10 3rd Qu.:0.00000
## Max. :88.97620 Max. :100.00 Max. :27.74 Max. :1.00000
## nox rm age dis
## Min. :0.3850 Min. :3.561 Min. : 2.90 Min. : 1.130
## 1st Qu.:0.4490 1st Qu.:5.886 1st Qu.: 45.02 1st Qu.: 2.100
## Median :0.5380 Median :6.208 Median : 77.50 Median : 3.207
## Mean :0.5547 Mean :6.285 Mean : 68.57 Mean : 3.795
## 3rd Qu.:0.6240 3rd Qu.:6.623 3rd Qu.: 94.08 3rd Qu.: 5.188
## Max. :0.8710 Max. :8.780 Max. :100.00 Max. :12.127
## rad tax ptratio black
## Min. : 1.000 Min. :187.0 Min. :12.60 Min. : 0.32
## 1st Qu.: 4.000 1st Qu.:279.0 1st Qu.:17.40 1st Qu.:375.38
## Median : 5.000 Median :330.0 Median :19.05 Median :391.44
## Mean : 9.549 Mean :408.2 Mean :18.46 Mean :356.67
## 3rd Qu.:24.000 3rd Qu.:666.0 3rd Qu.:20.20 3rd Qu.:396.23
## Max. :24.000 Max. :711.0 Max. :22.00 Max. :396.90
## lstat medv
## Min. : 1.73 Min. : 5.00
## 1st Qu.: 6.95 1st Qu.:17.02
## Median :11.36 Median :21.20
## Mean :12.65 Mean :22.53
## 3rd Qu.:16.95 3rd Qu.:25.00
## Max. :37.97 Max. :50.00#Creating the binary crim variable
crime01=rep(0,length(crim))
crime01[crim>median(crim)]=1
Boston=data.frame(Boston,crime01)#Splitting the dataset
train=1:(dim(Boston)[1]/2)
test=(dim(Boston)[1]/2 + 1):dim(Boston)[1]
Boston.train=Boston[train, ]
Boston.test=Boston[test, ]
crime01.test=crime01[test]#Seeing what variables may be associated with crime01
cor(Boston)## crim zn indus chas nox
## crim 1.00000000 -0.20046922 0.40658341 -0.055891582 0.42097171
## zn -0.20046922 1.00000000 -0.53382819 -0.042696719 -0.51660371
## indus 0.40658341 -0.53382819 1.00000000 0.062938027 0.76365145
## chas -0.05589158 -0.04269672 0.06293803 1.000000000 0.09120281
## nox 0.42097171 -0.51660371 0.76365145 0.091202807 1.00000000
## rm -0.21924670 0.31199059 -0.39167585 0.091251225 -0.30218819
## age 0.35273425 -0.56953734 0.64477851 0.086517774 0.73147010
## dis -0.37967009 0.66440822 -0.70802699 -0.099175780 -0.76923011
## rad 0.62550515 -0.31194783 0.59512927 -0.007368241 0.61144056
## tax 0.58276431 -0.31456332 0.72076018 -0.035586518 0.66802320
## ptratio 0.28994558 -0.39167855 0.38324756 -0.121515174 0.18893268
## black -0.38506394 0.17552032 -0.35697654 0.048788485 -0.38005064
## lstat 0.45562148 -0.41299457 0.60379972 -0.053929298 0.59087892
## medv -0.38830461 0.36044534 -0.48372516 0.175260177 -0.42732077
## crime01 0.40939545 -0.43615103 0.60326017 0.070096774 0.72323480
## rm age dis rad tax ptratio
## crim -0.21924670 0.35273425 -0.37967009 0.625505145 0.58276431 0.2899456
## zn 0.31199059 -0.56953734 0.66440822 -0.311947826 -0.31456332 -0.3916785
## indus -0.39167585 0.64477851 -0.70802699 0.595129275 0.72076018 0.3832476
## chas 0.09125123 0.08651777 -0.09917578 -0.007368241 -0.03558652 -0.1215152
## nox -0.30218819 0.73147010 -0.76923011 0.611440563 0.66802320 0.1889327
## rm 1.00000000 -0.24026493 0.20524621 -0.209846668 -0.29204783 -0.3555015
## age -0.24026493 1.00000000 -0.74788054 0.456022452 0.50645559 0.2615150
## dis 0.20524621 -0.74788054 1.00000000 -0.494587930 -0.53443158 -0.2324705
## rad -0.20984667 0.45602245 -0.49458793 1.000000000 0.91022819 0.4647412
## tax -0.29204783 0.50645559 -0.53443158 0.910228189 1.00000000 0.4608530
## ptratio -0.35550149 0.26151501 -0.23247054 0.464741179 0.46085304 1.0000000
## black 0.12806864 -0.27353398 0.29151167 -0.444412816 -0.44180801 -0.1773833
## lstat -0.61380827 0.60233853 -0.49699583 0.488676335 0.54399341 0.3740443
## medv 0.69535995 -0.37695457 0.24992873 -0.381626231 -0.46853593 -0.5077867
## crime01 -0.15637178 0.61393992 -0.61634164 0.619786249 0.60874128 0.2535684
## black lstat medv crime01
## crim -0.38506394 0.4556215 -0.3883046 0.40939545
## zn 0.17552032 -0.4129946 0.3604453 -0.43615103
## indus -0.35697654 0.6037997 -0.4837252 0.60326017
## chas 0.04878848 -0.0539293 0.1752602 0.07009677
## nox -0.38005064 0.5908789 -0.4273208 0.72323480
## rm 0.12806864 -0.6138083 0.6953599 -0.15637178
## age -0.27353398 0.6023385 -0.3769546 0.61393992
## dis 0.29151167 -0.4969958 0.2499287 -0.61634164
## rad -0.44441282 0.4886763 -0.3816262 0.61978625
## tax -0.44180801 0.5439934 -0.4685359 0.60874128
## ptratio -0.17738330 0.3740443 -0.5077867 0.25356836
## black 1.00000000 -0.3660869 0.3334608 -0.35121093
## lstat -0.36608690 1.0000000 -0.7376627 0.45326273
## medv 0.33346082 -0.7376627 1.0000000 -0.26301673
## crime01 -0.35121093 0.4532627 -0.2630167 1.00000000It seems that the variables indus, nox, age, dis, rad, and tax have a decent amount of correlation with crime01. Their correlation values are all at least .60. The highest of all of them is nox with .72.
#Logistic regression with indus, nox, age, dis, rad, and tax as predictors.
set.seed(5)
glm.Boston=glm(crime01~indus+nox+age+dis+rad+tax,data=Boston.train,family=binomial)
probs.Boston=predict(glm.Boston,Boston.test,type="response")
pred.Boston=rep(0,length(probs.Boston))
pred.Boston[probs.Boston>0.5]=1
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 75 8
## 1 15 155(15+8)/253## [1] 0.09090909Here we can see that the error rate of this model is 9.09%.
summary(glm.Boston)##
## Call:
## glm(formula = crime01 ~ indus + nox + age + dis + rad + tax,
## family = binomial, data = Boston.train)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.97810 -0.21406 -0.03454 0.47107 3.04502
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -42.214032 7.617440 -5.542 2.99e-08 ***
## indus -0.213126 0.073236 -2.910 0.00361 **
## nox 80.868029 16.066473 5.033 4.82e-07 ***
## age 0.003397 0.012032 0.282 0.77772
## dis 0.307145 0.190502 1.612 0.10690
## rad 0.847236 0.183767 4.610 4.02e-06 ***
## tax -0.013760 0.004956 -2.777 0.00549 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 329.37 on 252 degrees of freedom
## Residual deviance: 144.44 on 246 degrees of freedom
## AIC: 158.44
##
## Number of Fisher Scoring iterations: 8It appears that the variables age and dis are not statistically significant predictors of crime01. I will perform another logistic regression without these two variables.
#Logistic regression with indus, nox, rad, and tax as predictors.
glm.Boston=glm(crime01~indus+nox+rad+tax,data=Boston.train,family=binomial)
probs.Boston=predict(glm.Boston,Boston.test,type="response")
pred.Boston=rep(0,length(probs.Boston))
pred.Boston[probs.Boston>0.5]=1
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 71 6
## 1 19 157(6+19)/253## [1] 0.09881423With age and dis taken out, the error rate is 9.88%. This is slightly higher than the last model, even with the statistically insignificant predictors removed.
#LDA with indus, nox, age, dis, rad, and tax as predictors.
lda.Boston=lda(crime01~indus+nox+age+dis+rad+tax,data=Boston.train,family=binomial)
pred.Boston=predict(lda.Boston,Boston.test)
table(pred.Boston$class,crime01.test)## crime01.test
## 0 1
## 0 81 18
## 1 9 145(9+18)/253## [1] 0.1067194The error rate for this LDA model is 10.67%. I am going to attempt a similar model but with age and dis taken out.
#LDA with indus, nox, rad, and tax as predictors.
lda.Boston=lda(crime01~indus+nox+rad+tax,data=Boston.train,family=binomial)
pred.Boston=predict(lda.Boston,Boston.test)
table(pred.Boston$class,crime01.test)## crime01.test
## 0 1
## 0 80 18
## 1 10 145(10+18)/253## [1] 0.1106719The error rate for the LDA model without age and dis is 11.07%. It is slightly worse than the previous model.
#Naive Bayes with indus, nox, age, dis, rad, and tax as predictors.
bayes.Boston=naiveBayes(crime01~indus+nox+age+dis+rad+tax,data=Boston.train)
pred.bayes=predict(bayes.Boston,Boston.test)
table(pred.bayes,Boston.test$crime01)##
## pred.bayes 0 1
## 0 77 13
## 1 13 150(13+13)/253## [1] 0.1027668With this Naive Bayes model, our error rate is 10.28%. I am once again going to attempt a similar model but with age and dis taken out.
#Naive Bayes with indus, nox, rad, and tax as predictors.
bayes.Boston=naiveBayes(crime01~indus+nox+rad+tax,data=Boston.train)
pred.bayes=predict(bayes.Boston,Boston.test)
table(pred.bayes,Boston.test$crime01)##
## pred.bayes 0 1
## 0 80 18
## 1 10 145(10+18)/253## [1] 0.1106719The error rate for this model is 11.07%. Once again, the model without age and dis performs worse than the original.
#KNN with indus, nox, age, dis, rad, and tax as predictors at k=1.
train.K=cbind(indus,nox,age,dis,rad,tax)[train,]
test.K=cbind(indus,nox,age,dis,rad,tax)[test,]
pred.Boston=knn(train.K,test.K,crime01.test,k=1)
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 31 155
## 1 59 8(59+155)/253## [1] 0.8458498The error rate of this model is terrible at a whopping 84.58%. I am going to increase k to k=50.
#KNN with indus, nox, age, dis, rad, and tax as predictors at k=50.
train.K=cbind(indus,nox,age,dis,rad,tax)[train,]
test.K=cbind(indus,nox,age,dis,rad,tax)[test,]
pred.Boston=knn(train.K,test.K,crime01.test,k=50)
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 38 14
## 1 52 149(54+13)/253## [1] 0.2648221The error rate of the model improved a lot at k=50 to 26.48%, but I will still increase it a lot to see that effect.
#KNN with indus, nox, age, dis, rad, and tax as predictors at k=80.
train.K=cbind(indus,nox,age,dis,rad,tax)[train,]
test.K=cbind(indus,nox,age,dis,rad,tax)[test,]
pred.Boston=knn(train.K,test.K,crime01.test,k=80)
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 33 12
## 1 57 151(56+13)/253## [1] 0.2727273The error rate at k=80 increased to 27.27%, so I will drop it to try and reverse that.
#KNN with indus, nox, age, dis, rad, and tax as predictors at k=65.
train.K=cbind(indus,nox,age,dis,rad,tax)[train,]
test.K=cbind(indus,nox,age,dis,rad,tax)[test,]
pred.Boston=knn(train.K,test.K,crime01.test,k=65)
table(pred.Boston,crime01.test)## crime01.test
## pred.Boston 0 1
## 0 33 12
## 1 57 151(57+12)/253## [1] 0.2727273The error rate remained at 27.27%, I’m not sure if KNN is the best model here.
After testing out these models, the one that was the best was the first one attempted, the logistic regression with indus, nox, age, dis, rad, and tax as the predictors with an error rate of only 9.09%. Each type of model seemed to perform progressively worse, with KNN being the worst for this data set. Additionally, removing the variables age and dis made the model perform worse, so in general those six predictors together seem to perform the best.