Using Random Seeds

To randomly sample \(n_{train}\) rows from our original \(n\) rows, where \(n_{train} = \lfloor(n \times .8)\rfloor\), we use the following:

set.seed(100)
trainRows <- sample(1:333, 333*.8 ,replace = FALSE)

The code chunk above has two lines. The first line of code, set.seed(100), is one we will use quite a bit in this course. We are going to use random sampling to choose the rows for our training data. This means that we are going to ask the computer to randomly sample 266 of our 333 rows.

However, suppose you close your Markdown file and come back to it later. We want the computer to choose the SAME 266 rows when you run your code again. Otherwise, you would get completely different results each time you drew a random sample! The set.seed() function is what ensures that each time you run your chunk, you will get the same random sample. Let’s try that.

You should notice that every time you hit play on this chunk, you get a different sample. This is what would happen if you closed your Markdown and re-opened it, or if you gave your code to someone else to run. This is not something we want to have happen, so we set a random seed to fix this problem.

Now we notice that no matter how often we run the chunk, we get the same values. Yes!! This means that we can close our R and come back to it later, and our results will not change. This also means that we can send our code to someone else, and they will get the same random sample that we did. This means that setting a seed can help make your code reproducible.

Setting a random seed (which is what we will call using the set.seed() command) will prove very useful for any kind of random sampling we do in this course.

Sampling in R

Now, let’s go back to the code chunk we started with:

set.seed(100)
trainRows <- sample(1:333, 333*.8 ,replace = FALSE)

We have discussed the first line, but what does the second line do? Well, sample(1:333, 333*.8, replace = FALSE) will draw 266 random numbers between 1 and 333 (1:333). The form of the code is

sample( What options am I sampling from?,
        How many values do I want to sample?,
        Do I want to sample with replacement?)

Once we understand what a code is doing, the next step is to understand the type of output the function produces. Once we have drawn our 266 random numbers, we store that output in a vector called trainRows.

trainRows

Okay, so right now trainRows is a vector that contains 266 numbers between 1:333. We don’t want a list of numbers; we want a training data set. This means that the next step is to go into the training data set and grab the rows specified in trainRows.

At this point, we have created our new training data. This means that 266 of the original 333 rows have been used. What do we do with the 67 rows that were not chosen for training data? The rows that were not selected for our new training data set will form our validation data set.

Before moving on, check to make sure newTrain has 266 rows and validation has 67 rows.

Now that we have our validation data and training data, we have created a few objects we no longer need. To clean up our work space in R, we can remove these objects if we wish. To do so, we use the code rm:

rm(trainRows,trainPractice, penguins_raw)

Using the New Data

Now that we have successfully split our data into a validation set and a new training set, let’s use them!

compute_RMSE <- function(truth, predictions){

  # Part 1
  part1 <- truth - predictions
  
  #Part 2
  part2 <- part1^2
  
  #Part 3: RSS 
  part3 <- sum(part2)
  
  #Part4: MSE
  part4 <- 1/length(predictions)*part3
  
  # Part 5:RMSE
  sqrt(part4)
}

Great!!! Are we done?? Can we report this as our predictive accuracy??

It turns out that we shouldn’t. There is one issue with how we have created our validation data.

Hmm…this isn’t the same number we got in Question 9. Is this a fluke?

It turns out that with a data set that this is this small, splitting the data into two data sets creates what is called high variance estimation. This means that our estimate of the validation RMSE is highly dependent on the split of the data into the new training and validation data sets, which means our validation RSME estimate is highly dependent on the random seeds that we choose. This is not ideal; we don’t want our results to be left up to random chance. Because of this, we are going to try a different technique to create validation data.

First Iteration

The first step in LOOCV is to remove the first row from the data set and treat it as validation data.

validation <- penguins[1, ]

We then treat the rest of the data as training data.

newTrain <- penguins[-1, ]

Remember that - means “except” or “excluding”, so this code means that all the rows except row 1 are assigned to the new training set.

This is the predicted body mass for the first penguin, so we want to store this predicted value in the first row of YHat.

I would strongly recommend that at this point you open the YHat data set and make sure that the first row is filled in with the number you expect, and the rest of the rows are NAs.

Now that we have our prediction, we recall that the penguin in our validation set was actually the first row in the penguins data set, so we know the true body mass for this penguin.

For Loop

At this point, we have completed just one iteration of LOOCV. In other words, we have run the process for just the first row in the data set. Now, we are ready to perform the process on all \(n = 333\) rows in the original penguins data set.

At this point it is a good idea to look over YHat to make sure there is no missing data, meaning that you have filled in every row. I usually use the summary command for this:

summary(YHat[ , "body_mass_g"])

Now we have a data set containing a predicted value of body mass for every penguin in the penguins data set. We also know the true body mass for every penguin in the penguins data set. This means that we can compute the validation RMSE.

Next Steps

LOOCV is a very powerful tool for allowing us to assess predictive accuracy. However, it has a drawback. If you use LOOCV on large data sets, the process is computationally expensive, meaning it is slow and takes a lot of computer memory.

So, it sounds like LOOCV is good for small to medium data sets, and creating validation data by splitting the data is good for very large data sets (like hundreds of thousands of rows). Isn’t there something in the middle we can use?? Yep. We will learn k-fold CV next class.

References