The Goal

In class, we have started to work on classification techniques, which mean situations where our response variable \(Y\) is categorical. Today, we are going to practice using R to apply these classification techniques. We will also start to explore the idea of tuning a model.

The Data

Spam emails are unwanted emails, generally sent to masses of people by corporations or other entities attempting to sway you into purchasing a good or service. Want a cruise to Europe?? Have we got a deal for you!! Modern email clients like GMail work to identify spam messages and keep them from appearing in your inbox. This is called filtering. Filters take a look at each email that appears in your inbox. Based on what the filter sees, it classifies an email as spam, and tucks it away in your spam folder, or not spam, in which case the email is placed in your inbox. The better the filter, the fewer unwanted emails you have to sift through in your inbox each day.

Today, we have a client who asks you to build a spam filter for a particular set of emails. They are going to provide training data for you to use to build your model, as well as a test data set that you can use to evaluate how well your model identifies spam.

You can download the test data by putting the following into your browser:

• https://drive.google.com/file/d/1Oe7PE3VIpR9fgNVJGmp1RONP1BKzNhZP

and the training data by putting the following into your browser:

• https://drive.google.com/file/d/1OjQ-dz5AU7mZf9al4cKtmCVFBx8rFU9Y

You then need to load both of these data sets into R. If you have never loaded data from a csv file into R before, or if you need a refresher, follow these steps in the “Moving the data into RStudio” section.

The response variable of interest is \(Y=\) spam.

• spam: “notspam” if the email is not spam and “spam” if the email is spam.

There are 18 possible features in this data set. They are:

• to_multiple: Indicator for whether the email was addressed to more than one recipient.
• from: Whether the message was listed as from anyone (this is usually set by default for regular outgoing email).
• cc: Indicator for whether anyone was CCed.
• sent_email: Indicator for whether the sender had been sent an email in the last 30 days.
• image: Indicates whether any images were attached.
• attach: Indicates whether any files were attached.
• dollar: Indicates whether a dollar sign or the word ‘dollar’ appeared in the email.
• winner: Indicates whether “winner” appeared in the email.
• inherit: Indicates whether “inherit” (or an extension, such as inheritance) appeared in the email.
• password: Indicates whether “password” appeared in the email.
• num_char: The number of characters in the email, in thousands.
• line_breaks: The number of line breaks in the email (does not count text wrapping).
• format: Indicates whether the email was written using HTML (e.g. may have included bolding or active links) or plaintext.
• re_subj: Indicates whether the subject started with “Re:”, “RE:”, “re:”, or “rE”.
• exclaim_subj: Indicates whether there was an exclamation point in the subject.
• urgent_subj: Indicates whether the word “urgent” was in the email subject.
• exclaim_mess: The number of exclamation points in the email message.
• number: Factor variable saying whether there was no number, a small number (under 1 million), or a big number.

Classification Approach 1: KNN

Our client has asked us to use two numeric features in our model: number of characters in the email (num_char) and the number of line breaks in the email (line_breaks). The goal is to build a model that can use these two features (and only these two features) to predict whether or not an email is spam.

We are going to start by using k-Nearest Neighbors (kNN) to predict whether or not an email is spam. KNN has a large advantage that there are not many conditions you need to check before using it. There is no determined shape defined by the method, as there is in something like logistic regression. The only thing we need is at least two numeric features, which we have. The first step is to visualize the relationship between these two features.

Now, this scatter plot is a little unusual because right now, our Y variable is not represented on the plot.

Okay, so with the graph, we can look at a point, find its nearest neighbors, and determine what value \(\hat{Y}\) we would predict for that point based on the value of \(Y\) for these neighbors. However, we don’t want to do this one point at a time, nor do we want to try to guess at which neighbors are closest when we have a lot of different points on the graph. This means we want to use R to help us use kNN to estimate \(Y\).

suppressMessages(library(class))

results <- knn(train = , test =  , cl = , k = )

knitr::kable(table(results, test$spam), caption= "Table 1", col = c("True Not Spam", "True Spam"))

Classification Approach 2: Logistic Regression

The next classification approach we are going to try is logistic regression. We have the same goal (prediction) and the same two features (number of characters and line breaks) to use to predict whether or not an email is spam.

When we switch from kNN to logistic regression, we change the modeling assumptions we are making. In logistic regression, we assume that the the relationship between each numeric feature and the log odds of an email being spam is linear. To check this, we create a special scatter plot called an empirical logit plot.

To make an empirical logit plot in R, we need to teach R a special function. Create an R chunk, copy the function below, paste it into an R chunk in your Markdown, and run it. You will notice that nothing seems to happen, but if you check the top right panel of your RStudio session, you should notice that a new function called emplogitPlot has appeared. R allows individual users to create functions as needed, and in essence you have just ``taught” R a new function. This function was written by Alex Schell (http://alexschell.github.io/emplogit.html).

emplogitPlot <- function(x, y, binsize = NULL, ci = FALSE, probit = FALSE,
prob = FALSE, main = NULL, xlab = "", ylab = "", lowess.in = FALSE){
  # x         vector with values of the independent variable
  # y         vector of binary responses
  # binsize   integer value specifying bin size (optional)
  # ci        logical value indicating whether to plot approximate
  #           confidence intervals (not supported as of 02/08/2015)
  # probit    logical value indicating whether to plot probits instead
  #           of logits
  # prob      logical value indicating whether to plot probabilities
  #           without transforming
  #
  # the rest are the familiar plotting options
  
  if(class(y) =="character"){
   y <- as.numeric(as.factor(y))-1
   }
  
  if (length(x) != length(y))
    stop("x and y lengths differ")
  if (any(y < 0 | y > 1))
    stop("y not between 0 and 1")
  if (length(x) < 100 & is.null(binsize))
    stop("Less than 100 observations: specify binsize manually")
  
  if (is.null(binsize)) binsize = min(round(length(x)/10), 50)
  
  if (probit){
    link = qnorm
    if (is.null(main)) main = "Empirical probits"
  } else {
    link = function(x) log(x/(1-x))
    if (is.null(main)) main = "Empirical logits"
  }
  
  sort = order(x)
  x = x[sort]
  y = y[sort]
  a = seq(1, length(x), by=binsize)
  b = c(a[-1] - 1, length(x))
  
  prob = xmean = ns = rep(0, length(a)) # ns is for CIs
  for (i in 1:length(a)){
    range = (a[i]):(b[i])
    prob[i] = mean(y[range])
    xmean[i] = mean(x[range])
    ns[i] = b[i] - a[i] + 1 # for CI 
  }
  
  extreme = (prob == 1 | prob == 0)
  prob[prob == 0] = min(prob[!extreme])
  prob[prob == 1] = max(prob[!extreme])
  
  g = link(prob) # logits (or probits if probit == TRUE)
  
  linear.fit = lm(g[!extreme] ~ xmean[!extreme])
  b0 = linear.fit$coef[1]
  b1 = linear.fit$coef[2]
  
  loess.fit = loess(g[!extreme] ~ xmean[!extreme])
  
  plot(xmean, g, main=main, xlab=xlab, ylab=ylab)
  abline(b0,b1)
  if(lowess.in ==TRUE){
  lines(loess.fit$x, loess.fit$fitted, lwd=2, lty=2)
  }
}

To use the function, we need two things: a response variable, an explanatory variable. For example, for line breaks, we can use:

emplogitPlot(x=train$line_breaks, y=train$spam, 
             xlab = "Line Breaks", 
             ylab = "Log Odds of Being Spam", 
             main = "Figure 3")

In cases where the empirical logit plot suggests that linearity might not be valid, a good first step is to try transforming the explanatory variables. One common transformation is taking the log.

Comparing Approaches

We now have two different classification approaches, and we have applied them both to the same data set.

We have now started the process of comparing different approaches to the same statistical learning task, as well as exploring the process of tuning models by choosing the values of certain constants, like \(k\) in kNN. We will continue this process next class!