Executive Summary

The goal of the Capstone project is to develop a text predictive model using a large, unstructured database of the English language. The data is from a corpus called HC Corpora which is collected from publicly available sources (tweets, blogs and news) by a web crawler.

This milestone report focuses on exploratory analysis of the data set.

  1. We first read in the English blogs, tweets and news data and look at its word count, unique words, linecounts and median, maximum number of characters.
  2. We split the combined data into training, validation and test sets.
  3. Then we create the corpus for the training set and clean it.
  4. After that, we tokenize the corpus into ngrams and create its data term matrix. Because of memory issues with the large data set, we only keep terms that occur more than 100 times.
  5. We then create frequency charts of the top words for the 1-grams, 2-grams, 3-grams, 4-grams and 5-grams.
  6. Finally we discuss next steps.

The final product is a shiny web application where, based on user input, five next-word candidates will be presented. The details for this will be covered in a subsequent report.

## Warning: package 'tm' was built under R version 4.3.3
## Loading required package: NLP
## Warning: package 'caTools' was built under R version 4.3.3

1. Read in the Data Set

#######################################################################################
# Read in the data
#######################################################################################
blogs <- readLines("en_US.blogs.txt", warn=FALSE, encoding="UTF-8", skipNul=TRUE)
news <- readLines("en_US.news.txt", warn=FALSE, encoding="UTF-8", skipNul=TRUE)
twitter <- readLines("en_US.twitter.txt", warn=FALSE, encoding="UTF-8", skipNul=TRUE)
combine <- c(blogs, news, twitter)

#######################################################################################
# Explore the data
#######################################################################################
# Function to compute the number of words
wordCount <- function(x) sum(stringr::str_count(x,"\\S+"))

# Function to compute the number of unique words
uniqueCount <- function(x){
  sum(stringr::str_count(toString(unique(unlist(strsplit(x,"\\s+|[[:punct:]]")))),"\\S+"))
}

# Function to compute the median and maximum number of characters
medNChar <- function(x) median(nchar(x))
maxNChar <- function(x) max(nchar(x))


# Compute no. of lines, words, unique words and ratio of unique to total words
list <- list(blogs, news, twitter, combine)
LineCount <- sapply(list,length)
WordCount <- sapply(list,wordCount)
UniqueCount <- sapply(list,uniqueCount)
UniqueRatio <- UniqueCount/WordCount
MedNChar <- sapply(list, medNChar)
MaxNchar <- sapply(list, maxNChar)
##             Lines      Words Uniques Uniques_Words MedNChar MaxNchar
## Blogs     899,288 37,334,131 450,711         1.21%      156    40833
## News       77,259  2,643,969  95,150          3.6%      186     5760
## Twitter 2,360,148 30,373,583 462,731         1.52%       64      140
## Combine 3,336,695 70,351,683 774,838          1.1%       73    40833

The bulk of the 70+M words from the English corpus is contributed by blogs then tweets in roughly equal proportions, followed by a much smaller proportion from news articles.

2. Split into Training, Validation and Test Data Sets

We split the data into training set (60%), validation set (20%) and test sets (20%).

combine <- readRDS("combine.rds")
# Randomize the combine data
set.seed(123)
combine <- sample(combine, length(combine))

set.seed(123)
split <- sample.split(combine, SplitRatio=0.6)
train <- subset(combine, split==TRUE)
rest <- subset(combine, split==FALSE)
split2 <- sample.split(rest, SplitRatio=0.5)
validate <- subset(rest, split2==TRUE)
test <- subset(rest, split2==FALSE)

3. Create and Clean Corpus

We create the training corpus using VCorpus (volatile corpus) instead of just Corpus (simple corpus) as the latter resulted in 1 grams being returned in the document term matrix when I create 2-grams using TM and RWeka packages. This is also reported in Creating N-Grams with tm & RWeka - works with VCorpus but not Corpus.

The steps taken to clean the corpus like removing punctuation marks, convert to lower case, etc. are indicated in the comment lines below. I remove profanity by removing banned words maintained at the site http://www.bannedwordlist.com/.

I did not remove stopwords or stem the document since we want to predict the next word based on the user input string and doing so will result in ngrams that lose contextual information.

# Make corpus
corpus.train <- VCorpus(VectorSource(train))

# Convert to unicode
convertUnicode <- function (x) stringi::stri_trans_general(x, "latin-ascii")
corpus.train <- tm_map(corpus.train, content_transformer(convertUnicode))

# Separate words separated by "-" or "/"
toSpace <- content_transformer(function(x, pattern) gsub(pattern," ", x, perl=TRUE))
corpus.train <- tm_map(corpus.train, toSpace, "-")
corpus.train <- tm_map(corpus.train, toSpace, "/")

# Remove all punctuations except apostrophe
removeSpecial <- function(x) gsub(".*?($|'|[^[:punct:]]).*?", "\\1", x, perl=TRUE)
corpus.train <- tm_map(corpus.train, content_transformer(removeSpecial))

# Remove emojis
corpus.train <- tm_map(corpus.train, toSpace, "[^[:graph:]']")

# Convert to lower case
corpus.train <- tm_map(corpus.train, content_transformer(tolower))

# Remove numbers
corpus.train <- tm_map(corpus.train, removeNumbers)

# Remove banned word list from http://www.bannedwordlist.com/
swearwords <- VectorSource(readLines("swearWords.txt", warn=FALSE, 
                                     encoding="UTF-8", skipNul=TRUE))
corpus.train <- tm_map(corpus.train, removeWords, swearwords)

# Remove errant "'s" introduced by the above steps
corpus.train <- tm_map(corpus.train, toSpace, " 's")

# Remove errant " ' " introduced by the above steps
corpus.train <- tm_map(corpus.train, toSpace, " ' ")

# Strip whitespace
corpus.train <- tm_map(corpus.train, stripWhitespace)

saveRDS(corpus.train, "corpus.train.rds")