Introduction

This the goal of this project is to learn some aspects of natural language processing (NLP) to develop a basic model that can accurately predict the next word based on the previous word(or words) written. This will ultimately lead to an an app in which one can type some words and the app will use the model to predict the next word in the sequence. This model requires the use of n-grams, which are small sequences of words commonly found in text documents. For the model to work, the specific n-gram sequences are required, but the number of times that any given n-gram occurs in a particular document is also required, known as the frequency. This milestone report describes the techniques used to preprocess and explore words and n-grams from documents containing blog entries, tweets and news article summaries. In addition, some discussion is provided about how to determine how many unique words are needed to reach a certain percentage of the total number of words in a given text document. Intuitively, not as many unique words would be needed as one would think, since most people use mainly fairly common words in a language day-to-day. In addition, some discussion is provided regarding how to investigate foreign words in a document and how to get the most out of a fewer number of words and smaller datasets.

Preprocessing

This section outlines some of the steps that were taken to load and pre-process the text data so that it is in a form that can be more easily used for analysis.

Load the necessary libraries:

library(tm)
library(ggplot2)
library(ngram)
library(gridExtra)
library(SnowballC)
library(wordcloud)
library(RColorBrewer)

Load the twitter, blogs and news documents into a corpus.Note that all the preprocessing in this section has already been done, so these code segments won’t be evaluated to save time.

dir_docs<- "C:/Capstone/final/"
setwd(dir_docs)
docs<- VCorpus(DirSource(directory = dir_docs), readerControl=list(readPlain, language="en", load=TRUE))

Remove all non-ASCII characters from the document. All the English text, punctuation and numbers is covered by ASCII, so this is one step for cleaning up the text documents. In addition, punctuation is removed and the letters are converted to all lower case. I decided to keep common English stopwords and also not to stem words with suffixes, since I felt that removing the stopwords and stemming words would cause them to lose some written context, which might make word prediction more difficult.

for(i in 1:length(docs)){
docs[[i]][[1]] <- gsub("[^\x01-\x7F]","",docs[[i]][[1]])
}
docs <- tm_map(docs,removePunctuation)
docs <- tm_map(docs, content_transformer(tolower))

Remove the bad language from the documents using a profanity list. The documents in the corpus have the format where the first list element describes which document is being looked at (document 1 to 3 in this case), the second list element describes which part of the document is being viewed (1 = text body, 2 = meta data) and the final element can be used to select which lines of the text document to access (assuming 1 is chosen as the text body for the previous element.). Finally after all the unwanted words have been removed, the excess whitespace is stripped away.

dir_profanity <- "C:/Capstone/profanity"
setwd(dir_profanity)

bad_words <- read.csv(paste0(dir_profanity,"/profanity_list.csv"),header=FALSE, stringsAsFactors = FALSE)
profanity_vector <- bad_words$V1

for (i in 1:length(docs)){
  for (j in 1:length(profanity_vector)){
    docs[[i]][[1]] <- gsub(profanity_vector[j],"",docs[[i]][[1]])
  }
}

docs<- tm_map(docs, stripWhitespace)

Now that the documents have been cleaned, write each text document to a separate folder containing only clean documents. This has already been done, so this code chunk won’t be evaluated here. Since clean text documents have been generated, we will then remove the original corpus from memory.

dir_clean_docs<- "C:/Capstone/clean2/"  
setwd(dir_clean_docs)

write(docs[[1]][[1]], "en_US_blogs_clean.txt", sep="\n")
write(docs[[2]][[1]], "en_US_news_clean.txt", sep="\n")
write(docs[[3]][[1]], "en_US_twitter_clean.txt", sep="\n")

rm(docs)

Exploratory Analysis

Now it is feasible to explore the content of the documents in more detail now that they have been cleaned. In this section we will initally load the cleaned documents into a corpus and generate a table of the number of words, lines and characters in each document. A document term matrix of word frequencies in each document will also be generated. Part of this document term matrix will be shown.

dir_clean_docs<- "C:/Capstone/clean2/"  
setwd(dir_clean_docs)

clean_docs<- VCorpus(DirSource(directory = dir_clean_docs), 
readerControl=list(readPlain, language="en", load=TRUE))

dtm <- DocumentTermMatrix(clean_docs, control=list(tolower=FALSE))

inspect(dtm)
## <<DocumentTermMatrix (documents: 3, terms: 831149)>>
## Non-/sparse entries: 1041723/1451724
## Sparsity           : 58%
## Maximal term length: 384
## Weighting          : term frequency (tf)
## Sample             :
##                          Terms
## Docs                          and    are    for   have   that     the
##   en_US_blogs_clean.txt   1085739 193628 362783 218530 459477 1855625
##   en_US_news_clean.txt      68213  10720  26973  10997  26358  151518
##   en_US_twitter_clean.txt  433618 158095 384392 168024 232872  934047
##                          Terms
## Docs                        this    was   with    you
##   en_US_blogs_clean.txt   257945 277996 286162 296819
##   en_US_news_clean.txt      9446  17625  19753   7420
##   en_US_twitter_clean.txt 162699 117013 172971 543062

Determine the number of words in each document and put the results in a document summary data frame.

nwords <- rowSums(as.matrix(dtm))

doc_summary <- data.frame(matrix(ncol =3, nrow = length(clean_docs)))
doc_names <- names(nwords) 
rownames(doc_summary) <- doc_names
colnames(doc_summary) <- c("Word Count", "Character Count", "Line Count")


words_col <- unname(nwords)
doc_summary$`Word Count` <- words_col

Now add the number of lines and characters in each document into the document summary data frame. Show the data frame.

lines_col <- c(length(clean_docs[[1]][[1]]),length(clean_docs[[2]][[1]]),
               length(clean_docs[[3]][[1]]))
doc_summary$`Line Count` <- lines_col


char_col <- c(sum(nchar(clean_docs[[1]][[1]])),sum(nchar(clean_docs[[2]][[1]])),
              sum(nchar(clean_docs[[3]][[1]])))

doc_summary$`Character Count` <- char_col

print(doc_summary)
##                         Word Count Character Count Line Count
## en_US_blogs_clean.txt     28973975       199984229     899288
## en_US_news_clean.txt       2136795        15080570      77259
## en_US_twitter_clean.txt   22744726       153120630    2360148

At this point we can create a frequency table of word counts across all three documents and show the results in a bar plot of the most common 25 words across all three documents.

word_matrix <- sort(colSums(as.matrix(dtm)), decreasing = TRUE)
word_freq_data <- data.frame(word = names(word_matrix),frequency=word_matrix)
rownames(word_freq_data) <- seq(1, nrow(word_freq_data),1)
word_freq_data$word <- as.character(word_freq_data$word)

Generate the barplot of different word frequencies.

word_freq_plot <- ggplot(data=word_freq_data[1:25,], aes(reorder(word,-frequency), frequency)) +
               geom_bar(stat="identity", color="darkgreen", fill="pink") + 
  labs(title = "Total Single Word Counts",x = "Word",y= "Frequency") + 
  theme(text = element_text(size=16),axis.text.x = element_text(angle = 60, hjust = 1))
word_freq_plot

Here is the corresponding word cloud for the same data.

pal2 <- brewer.pal(9,"Set1")
wordcloud(clean_docs, max.words = 100, random.order = FALSE, colors = pal2)

Write the word frequency data frame to a .csv file for later use. This has already been done, so this code chunk won’t be evaluated.

write.csv(word_freq_data, "C:/Capstone/OneGram.csv")

For later use, each text document now need to be tokenized into different frequency tables of n-gram. An n-gram is a sequence of n words that appear at least once in a given document. Certain sequences can appear many times. For 1-grams, which is just one word, a frequency table for each document was generated using the word matrix code already shown above. For n-grams where n > 1, this was more complex. The following function was used to generate 2,3, and 4-gram frequency tables for each document. The function takes two arguments; a) the document to be analyzed and b) nval, which is the n value for the n-gram. The function takes a document, converts it to a single very large string, finds the n-grams for a specific n value, generates the frequency table of all n-grams found in the document, converts this table to a data frame and sorts it in the order of decreasing frequency. Finally the data frame is then returned. For large documents and a large value of n, this became more time consuming (up to around 25 minutes or so on my PC), so this highlights some of the challenges of manipulating very large text documents. Using a smaller portion of a given document could definitely reduce this computation time and free up memory. Also an arguement will later be given to reduce the size of the frequency table, since rare n-grams (n-grams of low frequency counts) may not necessarily be needed for the prediction model. Removing low frequency n-grams could drastically reduce the size of the frequency table.

NgramTokenizer <- function(document, nval){
str <- concatenate(document, collapse = " ",rm.space = FALSE)

n_gram <- ngram_asweka(str, min =nval , max = nval, sep = " ")

n_gram_data <- table(n_gram)

n_gram_data <- as.data.frame(n_gram_data)

n_gram_data <- n_gram_data[order(n_gram_data$Freq, decreasing=T),]

rownames(n_gram_data) <- seq(1, nrow(n_gram_data),1)

return(n_gram_data)
}

An example of a function call to create an n-gram frequency table is given as follows. This was repeated to generate 2,3 and 4 n-gram tables for the blog, news and twitter document. This was done previously and won’t be re-evaluated here. The frequency tables were then saved for later use. This example generates a 3-gram frequency table for the blogs document and then saves the data.

three_grams_blogs <- NgramTokenizer(clean_docs[[1]][[1]],3)
write.csv(three_grams_blogs, "C:/Capstone/ThreeGram_blogs.csv")

The frequency tables are huge, particulary for the twitter and blog higher n-grams, often containing millions of entries. A large number of these n-grams show up only once in a given document. Therefore, in NLP, people often refer to an n-gram frequency cutoff, which describes a count number below which n-grams that have this count number or less are ignored (T. Hawker et. al, “Practical Queries of a Massive n-gram Database”, Proceedings of the Australasian Language Technology Workshop 2007, pp. 40-48). This is done since the low frequency n-grams don’t add much value to a prediction model compared to the n-grams that occur more frequently. The cutoff I chose was a frequency of 10, meaning that any n-grams that were seen less than 10 times were removed from the dataset. This dramatically reduced the size of these tables so they were easier to work with. This choice was based on a similar approach used by the Communication and Information Science (CIW) Department at the University of Groningen in the Netherlands, who created an app to run interactive n-gram queries of an indexed version of the Dutch Twitter corpus (http://www.let.rug.nl/gosse/Ngrams/).

The following code sequence demonstrates how to load the large n-gram frequency tables and then truncate them such that n-grams that occur less than ten times are removed from the tables. They can then be saved as tables of reduced size. This was done for all three documents for ngrams ranging from n = 1 to 4. The example here is for the single words in the blogs text document.

data_cutoff <- function(data, cutoff){
data <- data[data[,2]>=cutoff, ]
return(data)
}

one_grams_blogs <- read.csv("C:/Capstone/OneGram_blogs.csv")
one_grams_blogs <- data_cutoff(one_grams_blogs,10)
write.csv(one_grams_blogs, "C:/Capstone/OneGram_blogs_reduced.csv")

The following code will generate a plot of the most frequent fifteen 3-grams in each text document.

main_folder <-"C:/Capstone/"
three_grams_blogs <- read.csv(paste0(main_folder,"ThreeGram_blogs_reduced.csv"))
three_grams_news <- read.csv(paste0(main_folder,"ThreeGram_news_reduced.csv"))
three_grams_twitter <- read.csv(paste0(main_folder,"ThreeGram_twitter_reduced.csv"))

blogs_plot <- ggplot(data=three_grams_blogs[1:12,], aes(reorder(n_gram,-Freq), Freq)) +
               geom_bar(stat="identity", color="darkgreen", fill="orange") + 
  labs(title = "Top 12 3-Grams:Blogs",x = "3-Gram",y= "Frequency") +
  theme(text = element_text(size=16),axis.text.x = element_text(angle = 60, hjust = 1))


news_plot <- ggplot(data=three_grams_news[1:12,], aes(reorder(n_gram,-Freq), Freq)) +
               geom_bar(stat="identity", color="darkgreen", fill="magenta") + 
  labs(title = "Top 12 3-Grams:News",x = "3-Gram",y= "Frequency") +
  theme(text = element_text(size=16),axis.text.x = element_text(angle = 60, hjust = 1))

twitter_plot <- ggplot(data=three_grams_twitter[1:12,], aes(reorder(n_gram,-Freq), Freq)) +
               geom_bar(stat="identity", color="darkgreen", fill="turquoise") + 
  labs(title = "Top 12 3-Grams:Twitter",x = "3-Gram",y= "Frequency") + 
  theme(text = element_text(size=16),axis.text.x = element_text(angle = 60, hjust = 1))


plot_3grams <- grid.arrange(blogs_plot, news_plot, twitter_plot, ncol= 3)

From these results, the top 3-grams from twitter look quite personal. For the news, the 3-grams appear more formal and impersonal, while the blogs are somewhere in between. This makes sense given the context of the content in each of these types of documents.

Unique Words for Word Instances

The following code provides an example of how to find the number of unique words to get 50% and 90% of all word instances in a document. The number of words is kept track of with nwords, a counter for each unique word is given by i, which corresponds to a row index of the one word frequency table across all documents. The second column of this table is added up to get total word instances across all three documents. Coverage is given as a percentage of the total word instances. While the current word count is less than the coverage, a while loop will execute until this changes, cumulatively adding to the total number of word instances in nwords. The row index counter is also incremented. Once the condition is no longer met, the while loop breaks and the number of unique words (most recent row index) to get coverage is returned by the function. The function is called for 50% and 90% coverage.

one_grams <- read.csv(paste0(main_folder,"OneGram.csv"))


uniquewords <- function(percentage){
i<- 1
nwords <- 0
tot_words <- sum(one_grams$frequency) 
coverage <- percentage*tot_words

while(nwords < coverage){
  nwords = nwords + one_grams[i,2]
  i = i + 1
}

return(i)
}

uni_percent_50 <- uniquewords(0.5)
uni_percent_50
## [1] 262
uni_percent_90 <- uniquewords(0.9)
uni_percent_90
## [1] 10170

From these results, only 262 and 10170 unique words are required for 50% and 90% word instance coverage, respectively. Not a lot of words are needed to get instance coverage since only a small number of words are used regularly for communication in a language.

Words from Other Languages

Some languages use non-ASCII symbols and characters, so not removing these from the datasets could help you to filter for unusual words in your document after it has been loaded into a frequency table. Some languages also use unusual letter pairings and accents (ie. ‘cz’, ‘gt’ etc.), so it could be possible to filter for these character sequences as well to look for possible foreign words in a word frequency table.

Increasing coverage

To get more word coverage and expand on context for your model, it might be worthwile to add documents to your existing corpus that are of a different context than what is used here (ie. personal letters, work emails, page from a book, academic journal articles and reports, etc.) This would provide a larger range of n-gram types that would be available and enable your model to predict words for different kinds of documents. Since many basic words have already been found from the three documents provided, any additional tables added from these documents would only need to include more novel words and n-grams that haven’t been tabulated yet.

Next steps

The next steps will involve developing a predictive model based on n-grams to be able to guess the next word following a few previous ones. This could possibly involve a probabilty (Markov) approach which predicts the probability of the next word being a specific value given the previous words. For an n-gram, each word depends on the previous n-1 words that preceeded it.