The Goal

We have been learning a lot of code in our quest to explore cross validation techniques. Today, we are going to dig more deeply into the code behind 10-fold CV.

The Data

Our data from today is the cell phone price data from Lab 2. Recall that the goal is to predict \(Y\) = cell phone prices. In addition to price, we have 14 features:

• battery_power: Total energy a battery can store in one time measured in mAh
• clock_speed: speed at which microprocessor executes instructions
• fc: Front Camera mega pixels
• int_memory: Internal Memory in Gigabytes
• m_dep: Mobile Depth in cm
• mobile_wt: Weight of mobile phone
• n_cores: Number of cores of processor
• pc: Primary Camera mega pixels
• px_height: Pixel Resolution Height
• px_width: Pixel Resolution Width
• ram: Random Access Memory in Mega Bytes
• sc_h: Screen Height of mobile in cm
• sc_w: Screen Width of mobile in cm
• talk_time: longest time that a single battery charge will last w

To load both the test and training data sets, put the code below into a chunk and press play.

test <- read.csv("https://www.dropbox.com/scl/fi/y1hjakf9e3pkkohkot4sf/test_phones.csv?rlkey=7ujlcxnuvu0lemcoit39t6mcd&st=awcl33xl&dl=1")
train  <- read.csv("https://www.dropbox.com/scl/fi/8iitmby2trytf65pvmk96/train_phones.csv?rlkey=y5rhlbd26d6b20mu56oba4xg7&st=50a35x9h&dl=1")

We will also need the caret and ggplot2 libraries today.

suppressMessages( library(ggplot2) )
suppressMessages( library(caret) ) 

We are going to use KNN to predict cell phone prices, as we saw in Lab 2 that KNN predicted better than regression for these data. However, we need to choose \(K\) for KNN! In Lab 2, we only explored a few choices of \(K\). Today, we are going to explore all options \(K = 2, 3, 4, ...., 50\).

We have test data today, so it is tempting to use the test data to choose \(K\). However, we do not want to use test data to train or tune models or algorithms. Instead, we will use 10-fold CV to create validation data that we can use to choose \(K\). With that choice of \(K\), we than can fit KNN to our train data and make predictions on our test data to see how well that choice of \(K\) would work in practice!

This means validation data has two uses. Validation data can be used to estimate test metrics like the test RMSE when there is no test data present. When test data is present, validation data can be used to choose tuning parameters (like \(K\) in KNN) that are needed for a model or algorithm to run.

10-fold CV: Small Data Example

Before we get into actually using 10-fold CV for choosing \(K\), we are going to walk through the steps needed to run 10-fold CV.

10-fold CV involves dividing our train data set into 10 different subsets called folds. Each fold will contain roughly the same number of rows, and each row from the original data set will be assigned to exactly one fold.

Once we have determined how many rows (cell phones) need to go in to each fold, we need to complete the process of determining which rows will be assigned to each fold. Doing this involves sampling without replacement.

Before we try this on the real data set, let’s create a smaller data set to work with to illustrate the code.

tickets <- rep(1:5, 4)

tickets

##  [1] 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

tickets_Q4 <- 

# Set the random seed
set.seed(363663)

# Draw tickets from the box
foldsSmall <- sample(tickets, 20, replace = FALSE)

which(foldsSmall == 1)

fold1 <- trainSmall[which(foldsSmall==1), ]

10-fold CV on the original Cell Phone Data

Okay, now that we have seen an example of how to assign the rows in a data set to folds, let’s try it with all \(n=1500\) rows in the train data set using 10-fold CV.

# Create storage space 
yhat <- data.frame("price" = rep(NA,1500))

compute_RMSE <- function(truth, predictions){

  # Part 1
  part1 <- truth - predictions
  
  #Part 2
  part2 <- part1^2
  
  #Part 3: RSS 
  part3 <- sum(part2)
  
  #Part4: MSE
  part4 <- 1/length(predictions)*part3
  
  # Part 5:RMSE
  sqrt(part4)
}

Considering K

Now, the work we have done so far just found the validation RMSE using 10-fold CV. However, we have test data, so we didn’t really need to create validation data to estimate the test RMSE. What we do need validation for is to help us choose \(K\), and the work we have just done is a big part of the code we need for that process. We are going to (1) use 10-fold CV to create validation data to choose \(K\), (2) train KNN with train, and (3) assess our predictive ability with test.

In the work we have done so far, we have used 5-NN. However, is that the best choice for the number of neighbors? Could we end up with a smaller validation RMSE if we choose a different number of neighbors \(K\)? To determine this, we actually need to try out different options of \(K\).

Ideally, we want to be able to do what we did in Question 13 (compare choices of \(K\)) for many options of \(K\). Our data set for today is small enough that we can consider a lot of options and see which one works best, so let’s try \(K = 2, 3, 4, 5, 6, \dots, 49, 50\).

We already have one for loop to run 10-fold CV. To change \(K\), we need to wrap another for loop around that for loop. The code structure we need for this is called a Nested for-loop.

A Nested for-loop looks like this:

# Outer Loop
for( firstindex in SOMETHING){

  # Inner Loop 
  for( secondindex in SOMETHING ELSE){
  
 
  }

}

Example: Let’s suppose for each number 1 - 5, I want to add the numbers 3 - 10. To do this using a nested for-loop we use the following:

# Outer Loop
for( i in 1:5){

  # Inner Loop 
  for( j in 3:10){
    
    print( i + j)
 
  }

}

Generally, when I write nested loops I start with the inner loop to make sure it runs. Our goal is to find a validation RMSE for KNN using 10-fold CV with \(K = 2, 3, 4, 5, 6, \dots, 49, 50\). This means inner loop is 10-fold CV, and we already know that code is:

yhat <- data.frame("price" = rep(NA,1500))

for( i in 1:10){
  
  # Step 1 
  infold <- which(folds == i)

  # Step 2 
  newTrain <- train[ -infold,]
  validation  <- train[infold, ]

  # Step 3
  yhat[infold,"price"] <- knnregTrain( newTrain[,-15] , validation[,-15], newTrain[ , 15], k = 5)

}

compute_RMSE( train$Y, yhat$price)

I then think about the outer loop as wrapping around that inner loop and allowing something (in this case \(K\)) within the inner loop to change.

# Outer loop 
for(k in SOMETHING){ 

  yhat <- data.frame("price" = rep(NA,1500))
  
  # Inner loop 
  for( i in 1:10){
  
    # Step 1 
    infold <- which(folds == i)

    # Step 2 
    newTrain <- train[ -infold,]
    validation  <- train[infold, ]

    # Step 3
    yhat[infold,"price"] <- knnregTrain( newTrain[,-15] , validation[,-15], newTrain[ , 15], k = SOMETHING ELSE )

  }
  # Close inner loop

  compute_RMSE( train$Y, yhat$price)
  
} 
# Close outer loop

The only trick now is the compute_RMSE part. This part of the code will compute the validation RMSE for each choice of \(K\). However, as written now the loop will print the validation RMSE as the code runs. We do not really want 49 values of the RMSE to print out- that would make it really hard to figure out which choice of \(K\) gives us the lowest validation RMSE. Instead, we want to store the RMSE each time through the outer loop.

At this point, we have done essentially everything except adapt the code so that we store the validation RMSE in RMSE_storage rather than printing it out each time the outer loop runs. Let’s try our hand at running the whole loop.

smallestRMSE <- c(2:50)[which.min(RMSE_storage$RMSE)] 
  
ggplot( data.frame(RMSE_storage), aes( x = K, y = RMSE)) + geom_point() + geom_vline( xintercept = smallestRMSE, col = "blue" , lty = 2 ) +  labs(caption = paste("Validation RMSE values for Different Tuning Parameters. Smallest RMSE at K = ", smallestRMSE), y = "Validation RMSE", x = "Tuning Parameter")

Back to the test data

Okay, so we used 10-fold CV to create validation data that allowed us to choose \(K\) for KNN. Now that we have our choice of \(K\), let’s use it to make predictions on our test data.

Before you submit

A few last steps before we knit, and then you will be done with Lab 4!

• 1. Find the top of this file (the little tab), and look under it. You should see something with ABC and a check mark. This is for checking spelling! Click this to check your spelling before you do a final knit and submit.
• 2. If you are working with a partner, make sure their name and yours is on the top of the file.
• 3. Look at the very first chunk in your Markdown file. You should see something like:

knitr::opts_chunk$set(echo = TRUE)

Change this to:

knitr::opts_chunk$set(echo = FALSE, message = FALSE, warning = FALSE)

• 4. You must submit a PDF or HTML file. If you submit any other file type, it cannot be graded. Let me know if you have any questions.

Once you’ve done this, knit your file. This will create the PDF or html you need to submit. If you get stuck, let Dr. Dalzell know!