The Data

The data you need are on Canvas, under Lab 5. Remember that you need to load this data into RStudio, and then again into your Markdown file.

• spam: Y variable; Indicator for whether the email was spam.
• to_multiple: Indicator for whether the email was addressed to more than one recipient.
• from: Whether the message was listed as from anyone (this is usually set by default for regular outgoing email).
• cc: Indicator for whether anyone was CCed.
• sent_email: Indicator for whether the sender had been sent an email in the last 30 days.
• image: Indicates whether any images were attached.
• attach: Indicates whether any files were attached.
• dollar: Indicates whether a dollar sign or the word ‘dollar’ appeared in the email.
• winner: Indicates whether “winner” appeared in the email.
• inherit: Indicates whether “inherit” (or an extension, such as inheritance) appeared in the email.
• password: Indicates whether “password” appeared in the email.
• num_char: The number of characters in the email, in thousands.
• line_breaks: The number of line breaks in the email (does not count text wrapping).
• format: Indicates whether the email was written using HTML (e.g. may have included bolding or active links) or plaintext.
• re_subj: Indicates whether the subject started with “Re:”, “RE:”, “re:”, or “rE”.
• exclaim_subj: Indicates whether there was an exclamation point in the subject.
• urgent_subj: Indicates whether the word “urgent” was in the email subject.
• exclaim_mess: The number of exclamation points in the email message.
• number: Factor variable saying whether there was no number, a small number (under 1 million), or a big number.

Our response variable is binary, where 1 means that an email is spam and a 0 means that the email is not spam. Before we try to model the data, our first step is always to visualize or summarize our data in some way. With binary data, a good first step is determine how many 0s and 1s we actually have in this data set. Are they all spam? Do we have a good mix? The table command is useful for this:

knitr::kable(table(email$spam))

We can also make a graph to visualize the distribution of Y.

Checking Conditions

Our X variable is now numeric. When we were working with LSLR and MR models (meaning when our Y was numeric), we generally needed to check the shape of the relationship between X and Y. We used a scatter plot of X versus Y to do this.

\[Y = \beta_0 + \beta_1 X + \epsilon\]

For logistic regression we need to check the shape of the relationship between X and the log odds that an email is spam.

\[log\left( \frac{\pi}{1-\pi} \right) = \beta_0 + \beta_1X \]

We are still going to use a scatter plot to check this assumption, but this time we will have a scatter plot with X on the x-axis and the log odds that Y is spam on the y-axis. This special scatter plot is called an empirical log odds plot. Page 474 in your book provides specific details on plot construction, but we are going to use a function in R to produce the plot.

Create an R chunk, copy the function below, paste it into an R chunk in your Markdown, and run it. You will notice that nothing seems to happen, but if you check the top right panel of your RStudio session, you should notice that a new function called emplogit has appeared. R allows individual users to create functions as needed, and in essence you have just ``taught” R a new function. This function was written by Alex Schell (http://alexschell.github.io/emplogit.html).

EmpLogOddsPlot<- function(x, y, binsize = NULL, ci = FALSE, probit = FALSE,prob = FALSE, main = NULL, xlab = "", ylab = "", lowess.in = FALSE){
  # x         vector with values of the independent variable
  # y         vector of binary responses
  # binsize   integer value specifying bin size (optional)
  # ci        logical value indicating whether to plot approximate
  #           confidence intervals (not supported as of 02/08/2015)
  # probit    logical value indicating whether to plot probits instead
  #           of logits
  # prob      logical value indicating whether to plot probabilities
  #           without transforming
  #
  # the rest are the familiar plotting options
  
  if (length(x) != length(y))
    stop("x and y lengths differ")
  if (any(y < 0 | y > 1))
    stop("y not between 0 and 1")
  if (length(x) < 100 & is.null(binsize))
    stop("Less than 100 observations: specify binsize manually")
  
  if (is.null(binsize)) binsize = min(round(length(x)/10), 50)
  
  if (probit){
    link = qnorm
    if (is.null(main)) main = "Empirical probits"
  } else {
    link = function(x) log(x/(1-x))
    if (is.null(main)) main = "Empirical logits"
  }
  
  sort = order(x)
  x = x[sort]
  y = y[sort]
  a = seq(1, length(x), by=binsize)
  b = c(a[-1] - 1, length(x))
  
  prob = xmean = ns = rep(0, length(a)) # ns is for CIs
  for (i in 1:length(a)){
    range = (a[i]):(b[i])
    prob[i] = mean(y[range])
    xmean[i] = mean(x[range])
    ns[i] = b[i] - a[i] + 1 # for CI 
  }
  
  extreme = (prob == 1 | prob == 0)
  prob[prob == 0] = min(prob[!extreme])
  prob[prob == 1] = max(prob[!extreme])
  
  g = link(prob) # logits (or probits if probit == TRUE)
  
  linear.fit = lm(g[!extreme] ~ xmean[!extreme])
  b0 = linear.fit$coef[1]
  b1 = linear.fit$coef[2]
  
  loess.fit = loess(g[!extreme] ~ xmean[!extreme])
  
  plot(xmean, g, main=main, xlab=xlab, ylab=ylab)
  abline(b0,b1)
  if(lowess.in ==TRUE){
  lines(loess.fit$x, loess.fit$fitted, lwd=2, lty=2)
  }
}

To use the function, we need two things: a response variable, an explanatory variable. You can decide on the number of bins (binsize =), or the function automatically decides based on those inputs how to best draw the plot.

EmpLogOddsPlot(x=email$line_breaks, y=email$spam, xlab = "Line Breaks", ylab = "Empirical log odds", main = "Figure 3")

In cases where the empirical log odds plots suggest that a linear might not the correct shape to use, we know that we have other shapes we can use for \(f(X)\) rather than just a line! We are going to try a new one today: \(f(X) = log(X)\).

Predictions

To make predictions using a logistic regression model, we use the predict.glm function in R. We know that to make a prediction, we need a particular value of the X variable. Suppose we have a message with 400 line breaks. Will our filter label it spam? That all depends on the probability suggested by the model. So let’s find out! The code below provides the predicted probability for a message with 400 line breaks according to our second model.

predict.glm(spamModel2,data.frame(line_breaks=400), type = "resp")

Now, a probability is not a decision. We have just determined the probability of declaring a message spam if it has 400 line breaks, or 5 line breaks. But this does not actually declare the message spam.

In order to make the ultimate decision, we essentially will be spinning a spinner where the probability of getting “spam” is defined by the model. The larger the probability, the larger “piece” of the spinner that is labelled spam. Suppose the model says the probability of being spam for a particular email is 40%. To make a decision for this email using R, we use the following code:

sample(c("spam","notspam"),1, prob= c(.4,.6))

Running this will spit out a decision for our email. Do we declare it spam, or not spam? Now remember, with logistic regression, we are modeling probabilities. If we run the code again, we are in essence taking another round at the spinner, and we may get a different answer. (Try it and see!) That is okay!! And better than okay, it’s expected.

In the code, the prob section holds the probabilities associated with the outcomes “spam” and “notspam”, respectively. These can be changed to suit the prediction you need to make.