Summary of the Data Files

We will now provide some basic information on the data files that we will be working with. This will include the size of the file in megabytes, the total number of lines, the total words, and some summary statistics on the words per line in each file.

 file_sizes <- round(file.info(cons)$size / 1024^2)
 total_lines <- sapply(list(blogs, news, twitter), length)
 total_words <- sapply(list(blogs, news, twitter), stri_stats_latex)[4,]
 wpl <- lapply(list(blogs, news, twitter), function(x) stri_count_words(x))
 wpl_summ = sapply(list(blogs, news, twitter),
                     function(x) summary(stri_count_words(x))[c('Min.', 'Mean', 'Max.')])

Words per Line

 par(mfrow=c(3,1))
 hist(wpl[[1]], 
      breaks = 50, 
      main = "Words per Line: Blogs", 
      xlab = "Words per Line", 
      col = "blue")
 hist(wpl[[2]], breaks = 50, 
      main = "Words per Line: News", 
      xlab = "Words per Line", 
      col = "red")
 hist(wpl[[3]], breaks = 50, 
      main = "Words per Line: Twitter", 
      xlab = "Words per Line", 
      col = "green")

Sample the Data

Before we can process the data, we are going to need to take a sample of it. We can still make good predictions with a relatively small sample of the data and this will make our processing and analysis much easier.

set.seed(666)
s <- 0.01 #sample size

blogs_s <- sample(blogs, length(blogs)*s, replace = F)
news_s <- sample(news, length(news)*s, replace = F)
twitter_s <- sample(twitter, length(twitter)*s, replace = F)
sample_data <- c(blogs_s, news_s, twitter_s)
rm(blogs, blogs_s, news, news_s, twitter, twitter_s)

Build a Corpus

Now that our data has been sampled, we will transform it into a corpus we can work with using the tm package in R.

corpus <- VCorpus(VectorSource(sample_data))

Data Cleaning

With the data in a corpus format, we can now begin cleaning and processing our data.
Here we will remove:

• Common stop words (i.e. “the”, “a”, “an”, “in”)
• Numbers
• Punctuation
• URLs
• Twitter handles
• Email address patterns
• Profanity
• Whitespace

To remove profanity we will use a list of offensive language made available by the Carnegie Mellon School of Computer Science. We will also convert all characters to the lower case.

# Define a function to turn a specific pattern into a blank string
DeleteText <- content_transformer(function(x, patt) gsub(patt, " ", x))

# Remove URLs
corpus <- tm_map(corpus, DeleteText, "(f|ht)tp(s?)://(.*)[.][a-z]+")

# Remove Twitter handles
corpus <- tm_map(corpus, DeleteText, "@[^\\s]+")

# Remove email address patterns
corpus <- tm_map(corpus, DeleteText, "\\b[A-Z a-z 0-9._ - ]*[@](.*?)[.]{1,3} \\b")

# Send characters to lowers
corpus <- tm_map(corpus, content_transformer(tolower))

# Remove stop words
corpus <- tm_map(corpus, removeWords, stopwords(kind = "en"))

# Remove numbers
corpus <- tm_map(corpus, removeNumbers)

# Remove Punctuation
corpus <- tm_map(corpus, removePunctuation)

# Remove profanity
## Load in profanity data
con <- file("profanity.txt", open = "r")
profanity <- readLines(con = con, skipNul = T)
close(con)
## Remove offensive language
corpus <- tm_map(corpus, removeWords, profanity)

# Remove whitespace
corpus <- tm_map(corpus, stripWhitespace)

saveRDS(corpus, "english_corpus.rds")

N-Grams

First, we will need to create functions to tokenize our data into one, two, and three grams.

one_gram <- function(x){
  NGramTokenizer(x, Weka_control(min = 1, max = 1))
} 
two_gram <- function(x){
  NGramTokenizer(x, Weka_control(min = 2, max = 2))
} 
three_gram <- function(x){
  NGramTokenizer(x, Weka_control(min = 3, max = 3))
}