This is a smaller machine learning project exploring simple use of decision tree and also random forest. The dataset we will look at is the “college”" dataset from the library “ISLR”.

The task is to predict whether a school is classified as private or public based on the features listed.

Setup

Loading libraries.

library(ggplot2)
library(ISLR)
library(dplyr)
library(caTools)
library(rpart)
library(rpart.plot)

The structure of the dataset “college”:

df <- College
head(df)
##                              Private Apps Accept Enroll Top10perc
## Abilene Christian University     Yes 1660   1232    721        23
## Adelphi University               Yes 2186   1924    512        16
## Adrian College                   Yes 1428   1097    336        22
## Agnes Scott College              Yes  417    349    137        60
## Alaska Pacific University        Yes  193    146     55        16
## Albertson College                Yes  587    479    158        38
##                              Top25perc F.Undergrad P.Undergrad Outstate
## Abilene Christian University        52        2885         537     7440
## Adelphi University                  29        2683        1227    12280
## Adrian College                      50        1036          99    11250
## Agnes Scott College                 89         510          63    12960
## Alaska Pacific University           44         249         869     7560
## Albertson College                   62         678          41    13500
##                              Room.Board Books Personal PhD Terminal
## Abilene Christian University       3300   450     2200  70       78
## Adelphi University                 6450   750     1500  29       30
## Adrian College                     3750   400     1165  53       66
## Agnes Scott College                5450   450      875  92       97
## Alaska Pacific University          4120   800     1500  76       72
## Albertson College                  3335   500      675  67       73
##                              S.F.Ratio perc.alumni Expend Grad.Rate
## Abilene Christian University      18.1          12   7041        60
## Adelphi University                12.2          16  10527        56
## Adrian College                    12.9          30   8735        54
## Agnes Scott College                7.7          37  19016        59
## Alaska Pacific University         11.9           2  10922        15
## Albertson College                  9.4          11   9727        55

Data exploration

Basic data exploration before we begin building our models.

Scatterplot of graduation rate vs room and boarding costs.

ggplot(df,aes(Room.Board,Grad.Rate)) + geom_point(aes(color=Private))

As one could expect to see that Private colleges have higher Room and boarding costs.

Histogram of full time undergrad students, colored by private.

ggplot(df,aes(F.Undergrad)) + geom_histogram(aes(fill=Private),color="black",bins=50)

Histogram of graduation rate, colored by private.

ggplot(df,aes(Grad.Rate)) + geom_histogram(aes(fill=Private),color="black",bins=50)

In the histogram above, we see there is a college with above 100% graduation rate, we fix that by setting it to 100% instead.

subset(df,Grad.Rate>100)
##                   Private Apps Accept Enroll Top10perc Top25perc
## Cazenovia College     Yes 3847   3433    527         9        35
##                   F.Undergrad P.Undergrad Outstate Room.Board Books
## Cazenovia College        1010          12     9384       4840   600
##                   Personal PhD Terminal S.F.Ratio perc.alumni Expend
## Cazenovia College      500  22       47      14.3          20   7697
##                   Grad.Rate
## Cazenovia College       118

Fix:

df['Cazenovia College','Grad.Rate'] <- 100

Train-Test split

Splitting the dataset in 70% training and 30% testing.

set.seed(3)

sample <- sample.split(df$Private,SplitRatio=0.7)
train <- subset(df,sample==T)
test <- subset(df,sample==F)

Decision Tree

Building the model and predicting whether a college is private or not by assigning a probability, and prediction is set to Yes if that probability is higher that 50%.

tree <- rpart(Private ~ . ,method="class",data=train)
tree.preds <- as.data.frame(predict(tree,test))

Creating a column with our predictions, which we use to create our confusion matrix.

tree.preds$Private <- ifelse(tree.preds$Yes > 0.5,"Yes","No")
table(tree.preds$Private,test$Private)
##      
##        No Yes
##   No   57  13
##   Yes   7 156

Visualization of our tree.

prp(tree)

We see that the decision tree created here preforms fairly well. To see if this prediction can be improved we create a random forest model. ## Random Forest

library(randomForest)

Creating the model.

rf.model <- randomForest(Private ~ ., data=train,importance=TRUE)

Making predictions:

p <- predict(rf.model,test)
table(p,test$Private)
##      
## p      No Yes
##   No   58   6
##   Yes   6 163

The model using random forest performed a bit better than the decision tree.