Predict a Sequence of Words

Tokenization and nGram Generation

NGrams are a popular method introduced by Jurafsky and Martin (2000) for NLP when applied to text prediction (and other tasks) problems. The ngram is a sequence of words found in the text that can be further sorted by frequency of appearance. For example UniGrams are 1 word ngrams, while TriGrams are sequences of three words as “I will go” or “Happy New Year”. NGrams are very useful for simple text prediction models, because, ngrams that appear with higher frequency are the ones that have a higher chance (probabilistically speaking) of appearing in prediction problems out of sample. For example a sequence of two words “I will” (first and second word) can be the base for 100s if not 1000s TriGrams (possible third word). However, among the possible third words in the training sample that form TriGrams with “I will”, some appear more often than others and will be selected with higher probability in prediction out of sample.

In case a TriGram that contains the first two words “I will” is not found in the model, then the model can refer to BiGrams containing a first word “will” and use this information to predict the third word in “I will …”. Naturally if no BiGrams exist, the model can refer to the stored UniGrams and suggest the highest probability single word as found in the training sample.

One obvious disadvantage to NGrams modeling is that not all possible third words to “I will …” are available in the training sets. Smoothing techniques can be applied to the available and unavailable three grams containing “I will …”, where probability mass is transferred from available in the training sample TriGrams to the ones that have not appeared even once. There are a few less or more sophisticated techniques for probability smoothing, and some of these techniques will be used in modeling to improve the performance of the model out of sample.

One quickly materializing problem with higher order ngrams, like 3ngrams and 4ngrams, is that very few 3ngrams and 4ngrams appear with higher frequency, and most 3/4ngrams appear just once (called singletons). This leads to enormous dataframes and potential problems with memory availability to process these dataframes. A known method is to remove the singletons when storing the 3/4ngrams.

NGrams are generated and are used for the moment to explore the data in a subsequent section. The R code used to generate and store 1/2/3/4grams is presented in Appendix C. For the moment the ngrams are stored as they are generated. In a later step I will look into storing the ngrams based on indexing of single words to see if this approach can reduce the memory demands of the subsequent modeling tasks. This will also likely be necessary to be able to operate the shiny app that has to be developed once the model is ready.

Exploratory analysis

## [1] "Word Types to tokens ratio (TTR) for corpora with NO stop words removed"

##    Source   Tokens Word.Types         TTR
## 1   Blogs 26296381     260377 0.009901629
## 2    News 23990968     195090 0.008131810
## 3 Twitter 20942380     331649 0.015836261
## 4  Corpus 71229729     574816 0.008069889

## [1] "Word Types to tokens ratio (TTR) for corpora with stop words removed"

##    Source   Tokens Word.Types        TTR
## 1   Blogs 13641106     260254 0.01907866
## 2    News 13851582     194967 0.01407543
## 3 Twitter 11814786     331526 0.02806026
## 4  Corpus 39307474     574693 0.01462045

## [1] "number of words comprising 49.99 to 50.01% of the dictionary:  966 966"

## [1] "number of words comprising 50.01 to 60.01% of the dictionary:  967 1720"

## [1] "number of words comprising 60.01 to 70.01% of the dictionary:  1721 3108"

## [1] "number of words comprising 70.01 to 80.01% of the dictionary:  3109 6067"

## [1] "number of words comprising 80.01 to 90.01% of the dictionary:  6068 15048"

## [1] "number of words comprising 90.01 to 95.01% of the dictionary:  15048 32788"

Appendix A: Data subsampling

I sub sample the data into 3 main samples - training (70%), validation/development (15%), testing (15%). I further take a random 10% sample coming from the 70% training sample to use for testing my code. Working with a large corpus is very time consuming.

# set working folder
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

# A function to extract samples
textFileSample <- function(infile, outfile, selection) { 
  conn <- file(infile, "r")
  fulltext <- readLines(conn)
  nlines <- length(fulltext)
  close(conn)
  
  conn <- file(outfile, "w")

  for (i in 1:nlines) {
    if (selection[i]==1) { cat(fulltext[i], file=conn, sep="\n") }
  }
  close(conn)
  
  paste("Saved", sum(selection), "lines to file", outfile)
}

###################
## blogs ##########
###################

# read the txt file and count the number of lines in blogs document
conn <- file("./data/en_US/en_US.blogs.txt", "r")
fulltext <- readLines(conn)
numrows <- length(fulltext)
close(conn)
rm(fulltext)

# create indexes for samples for BLOGS
set.seed(42)
fraction1 <- 0.15      # 15% size of test sample
fraction2 <- 0.17      # approx 15% size of validation sample
fraction3 <- 0.14      # approx 10% size of training subsample out of full 70% train sample

# draw random samples based on rbinom
test.index <- rbinom(numrows, 1, fraction1)            # test sample index
val_train.index <- 1-test.index                        # full training + validation sample

val.subindex <- rbinom(numrows, 1, fraction2)          
val.index <- val_train.index*val.subindex              # validation sample index
train.index <- 1- test.index - val.index               # full training sample index

trainsample.subindex <- rbinom(numrows, 1, fraction3)
trainsample.index <- train.index*trainsample.subindex  # training sub-sample index

# extract subsamples and store
textFileSample("./data/en_US/en_US.blogs.txt", "./data/en_US/sample/blogs.test15pct.txt",
               test.index)
textFileSample("./data/en_US/en_US.blogs.txt", "./data/en_US/sample/blogs.val15pct.txt",
               val.index)
textFileSample("./data/en_US/en_US.blogs.txt", "./data/en_US/sample/blogs.train70pct.txt",
               train.index)                  
textFileSample("./data/en_US/en_US.blogs.txt", "./data/en_US/sample/blogs.train10pct.txt",
               trainsample.index)   

###################
## news ##########
###################

# read the txt file and count the number of lines in blogs document
conn <- file("./data/en_US/en_US.news.txt", "r")
fulltext <- readLines(conn)
numrows <- length(fulltext)
close(conn)
rm(fulltext)

# create indexes for samples for NEWS
set.seed(42)
fraction1 <- 0.15                                    # 15% size of test sample
fraction2 <- 0.17                                    # approx 15% size of validation sample
fraction3 <- 0.14                                    # approx 10% size of training subsample out of full train sample

test.index <- rbinom(numrows, 1, fraction1)            # test sample index
val_train.index <- 1-test.index                        # full training + validation sample

val.subindex <- rbinom(numrows, 1, fraction2)          
val.index <- val_train.index*val.subindex              # validation sample index
train.index <- 1- test.index - val.index               # full training sample index

trainsample.subindex <- rbinom(numrows, 1, fraction3)
trainsample.index <- train.index*trainsample.subindex  # training sub-sample index

# extract subsamples and store
textFileSample("./data/en_US/en_US.news.txt", "./data/en_US/sample/news.test15pct.txt",
               test.index)
textFileSample("./data/en_US/en_US.news.txt", "./data/en_US/sample/news.val15pct.txt",
               val.index)
textFileSample("./data/en_US/en_US.news.txt", "./data/en_US/sample/news.train70pct.txt",
               train.index)                  
textFileSample("./data/en_US/en_US.news.txt", "./data/en_US/sample/news.train10pct.txt",
               trainsample.index) 

###################
## twitter ########
###################

# load data and count number of lines in twitter document
conn <- file("./data/en_US/en_US.twitter.txt", "r")
fulltext <- readLines(conn)
numrows <- length(fulltext)
close(conn)
rm(fulltext)

# create indexes for samples for TWITTER
set.seed(42)
fraction1 <- 0.15                                    # 15% size of test sample
fraction2 <- 0.17                                    # approx 15% size of validation sample
fraction3 <- 0.14                                    # approx 10% size of training subsample out of full train sample

test.index <- rbinom(numrows, 1, fraction1)            # test sample index
val_train.index <- 1-test.index                        # full training + validation sample

val.subindex <- rbinom(numrows, 1, fraction2)          
val.index <- val_train.index*val.subindex              # validation sample index
train.index <- 1- test.index - val.index               # full training sample index

trainsample.subindex <- rbinom(numrows, 1, fraction3)
trainsample.index <- train.index*trainsample.subindex  # training sub-sample index

# extract subsamples
textFileSample("./data/en_US/en_US.twitter.txt", "./data/en_US/sample/twitter.test15pct.txt",
               test.index)
textFileSample("./data/en_US/en_US.twitter.txt", "./data/en_US/sample/twitter.val15pct.txt",
               val.index)
textFileSample("./data/en_US/en_US.twitter.txt", "./data/en_US/sample/twitter.train70pct.txt",
               train.index)                  
textFileSample("./data/en_US/en_US.twitter.txt", "./data/en_US/sample/twitter.train10pct.txt",
               trainsample.index) 

Next i combine blogs-news-tweets sub samples into combined-corpus sub samples:

##############################################
## combine a 10% sample of all threee sources
##############################################

# Combine train 70% sub-samples into one
conn <- file("./data/en_US/sample/blogs.train70pct.txt", "r")
        file1 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/news.train70pct.txt", "r")
        file2 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/twitter.train70pct.txt", "r")
        file3 <- readLines(conn)
        close(conn)
        
conn <- file("./data/en_US/sample/train70pct.txt", "w")
        cat(c(file1,file2,file3), file=conn, sep="\n")
        close(conn)
rm(file1, file2, file3)

# Combine train 10% sub-samples into one
conn <- file("./data/en_US/sample/blogs.train10pct.txt", "r")
        file1 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/news.train10pct.txt", "r")
        file2 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/twitter.train10pct.txt", "r")
        file3 <- readLines(conn)
        close(conn)
        
conn <- file("./data/en_US/sample/train10pct.txt", "w")
        cat(c(file1,file2,file3), file=conn, sep="\n")
        close(conn)
rm(file1, file2, file3)


# Combine test 15% sub-samples into one
conn <- file("./data/en_US/sample/blogs.test15pct.txt", "r")
        file1 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/news.test15pct.txt", "r")
        file2 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/twitter.test15pct.txt", "r")
        file3 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/test15pct.txt", "w")
        cat(c(file1,file2,file3), file=conn, sep="\n")
        close(conn)
rm(file1, file2, file3)

# Combine validation 15% sub-samples into one
conn <- file("./data/en_US/sample/blogs.val15pct.txt", "r")
        file1 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/news.val15pct.txt", "r")
        file2 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/twitter.val15pct.txt", "r")
        file3 <- readLines(conn)
        close(conn)

conn <- file("./data/en_US/sample/val15pct.txt", "w")
        cat(c(file1,file2,file3), file=conn, sep="\n")
        close(conn)
rm(file1, file2, file3)

The resulting data sets from this sub sampling are:

• blogs.train70pct.text, news.train70pct.text, twitter.train70pct.text
• blogs.train10pct.text, news.train10pct.text, twitter.train10pct.text
• blogs.test15pct.text, news.test15pct.text, twitter.test15pct.text
• blogs.val15pct.text, news.val15pct.text, twitter.val15pct.text
• train70pct.text, train10pct.text, test15pct.text, val15pct.text

Appendix B: Data pre-processing; Corpus prep for analysis

Set up pre-processing first:

library(tm) #load text mining library
library(textclean) #load auxiliary library 1

setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10") #sets R's working directory 

# using a DirSource it loads a txt file into a VCorpus for preprocessing
# the files that need pre-processing in general are the sub-samples 
# from the previous section. But we start with the training samples
# blogs.train70pct.txt, news.train70pct.txt, twitter.train70pct.txt, train10pct.txt

# for this example code I use the train10pct.txt sample, whihc i place in the "./data/en_US/train.sample" folder 

sample.data <- VCorpus(DirSource("./data/en_US/train.sample", encoding = "UTF-8"), 
                 readerControl=list(language="en"))
summary(sample.data)
docs <- sample.data

# content-transformer custom functions
# Some custom functions that are needed to transform the corpus using content_transformer 
# functionality of tm package
toEmpty <- content_transformer(function(x, pattern) {return (gsub(pattern, "", x))}) # pattern to empty
toSpace <- content_transformer(function(x, pattern) {return (gsub(pattern, " ", x))}) # pattern to space

# turns ? and ! and . into an end of sentence identifier EEOSS. The purpose is that when forming ngrams we 
# discard ngrams that are formed on the edges of sentences and combine words from more than one sentence
toEOS1 <- content_transformer(function(x, pattern) {return (gsub("\\? |\\?$|\\! |\\!$", " EEOSS ", x))})
toEOS2 <- content_transformer(function(x, pattern) {return (gsub("\\. |\\.$", " EEOSS ", x))})

# turns abbreviations as H.S.B.C. into an identifier AABRR. The purpose is that when forming ngrams we 
# discard ngrams that are formed on the edges of such abbreviations as these ngrams are incorrect 
# from information perspective
toABR <- content_transformer(function(x, pattern) {return (gsub("[A-Za-z]\\.[A-Za-z]\\.[A-Za-z]\\.[A-Za-z]\\. |[A-Za-z]\\.[A-Za-z]\\.[A-Za-z]\\. |[A-Za-z]\\.[A-Za-z]\\. ", " AABRR ", x))})

# turns numbers into an identifier NNUMM. The purpose is that when forming ngrams we 
# discard ngrams that are formed on the edges of such numbers as these ngrams are incorrect 
# from information perspective. A common approach is the 'remove numbers' with tm package.
# However, this will lead to formation of incorrect ngrams later on
toNUM <- content_transformer(function(x, pattern) {return (gsub("[0-9]+"," NNUMM ",x))})

# expand & and @ 
toAnd <- content_transformer(function(x) {return (gsub("&", "and", x))})
toAt <- content_transformer(function(x) {return (gsub("@", "at", x))})

# replace contractions using replace_contractions function from library library(textclean)
replaceContraction <- content_transformer(function(x) {return (replace_contraction(x))})
toHaveNot <- content_transformer(function(x) {return (gsub("haven't", "have not", x))})
toHadNot <- content_transformer(function(x) {return (gsub("hadn't", "had not", x))})

# remove duplicates
removeDup <- content_transformer(function(x) {return (unique(x))})

Apply a sequence of pre-processing steps on each text sample:

# 1. Remove duplicates
docs <- tm_map(docs, removeDup)

# 2. Separate words connected with - or /
# this helps later when we remove punctuation, so that two words do not stick to each otehr
docs <- tm_map(docs, toSpace, "-")
docs <- tm_map(docs, toSpace, "/")

# 3. Establish end of sentence
# the sequence of application of the functions is important here
docs <- tm_map(docs, toEOS1)
docs <- tm_map(docs, toABR)
docs <- tm_map(docs, toEOS2)
docs <- tm_map(docs, toNUM)

# 4. Remove email and http/s
docs <- tm_map(docs, toEmpty, "\\S+@\\S+") # email
docs <- tm_map(docs, toEmpty, "[Hh}ttp([^ ]+)") # html links

# 5. Remove retweet entries, Remove @ people, twitter usernames
docs <- tm_map(docs, toEmpty, "RT | via") # retweets
docs <- tm_map(docs, toEmpty, "@([^ ]+)") # people
docs <- tm_map(docs, toEmpty, "[@][a - zA - Z0 - 9_]{1,15}") # usernames

# 6. Transform to lower case (need to wrap in content_transformer) (time consuming procedure)
docs <- tm_map(docs,content_transformer(tolower))

# 7. Remove/replace &, @, 'm, 's, 'are, 'll, etc...
docs <- tm_map(docs, toAnd) 
docs <- tm_map(docs, toAt) 
docs <- tm_map(docs, replaceContraction)

        # remove this contraction in say "mother's day"
        # before it is removed by remove_punctuation
        docs <- tm_map(docs, toEmpty, "'s") 

        # i found that havn't and hadn't did not get replaced with replace_contractions
        docs <- tm_map(docs, toHaveNot)
        docs <- tm_map(docs, toHadNot)

# 8. Remove emoji's, emoticons like ❤#RelayForLife ❤"😖😰💔💔"
docs <- tm_map(docs, toEmpty, "[^\x01-\x7F]")

# 9. Remove g, mg, lbs etc; removes all single letters except "a" and "i"

docs <- tm_map(docs, toSpace, " [1-9]+g ") # grams
docs <- tm_map(docs, toSpace, " [1-9]+mg ") # miligrams, etc
docs <- tm_map(docs, toSpace, " [1-9]+kg ")
docs <- tm_map(docs, toSpace, " [1-9]+lbs ")
docs <- tm_map(docs, toSpace, " [1-9]+s ") # seconds, etc
docs <- tm_map(docs, toSpace, " [1-9]+m ")
docs <- tm_map(docs, toSpace, " [1-9]+h ")
docs <- tm_map(docs, toSpace, " +g ") # grams
docs <- tm_map(docs, toSpace, " +mg ") # miligrams, etc
docs <- tm_map(docs, toSpace, " +kg ")
docs <- tm_map(docs, toSpace, " +lbs ")
docs <- tm_map(docs, toSpace, " +s ") # seconds, etc
docs <- tm_map(docs, toSpace, " +m ")
docs <- tm_map(docs, toSpace, " +h ")
docs <- tm_map(docs, toSpace, " +lbs ")
docs <- tm_map(docs, toSpace, " +kg ")

docs <- tm_map(docs, toSpace, " [b-hj-z] ") # all single-letter words except a and i

# 10. Remove numbers
docs <- tm_map(docs, removeNumbers)

# 11. Remove punctuation
# Note that this removes hashtags from tweets. I examined the data and most often the
# hashtag precedes a word of meaning and not a name of a person. If it were the latter
# it would be logical to remove the whole word that is preceded by a hashtag. However, 
# this strategy would lead to loss in meaning in the sentences and potentially affect
# the predictive power of the model. In addition, hopefully words with hashtags 
# that dont have a general meaning will not repeat often, and will have a low weight
# in calibrating the model via the ngrams. 
# Result: REMOVE HASHTAGS only, not words w/ hashtag
docs <- tm_map(docs, removePunctuation, preserve_intra_word_dashes = TRUE)

        # 11.1. Remove ": removing punctuation did not remove these
        docs <- tm_map(docs, toEmpty, "“")
        docs <- tm_map(docs, toEmpty, "”")
        docs <- tm_map(docs, toEmpty, "‘")
        docs <- tm_map(docs, toEmpty, "’")

# 12. Remove profanity (time consuming procedure)
# I don't want my predictions to suggest profanity words. For this purpose i may
# train a model with these words in and remove them in the end by recalibrating the 
# model on non-profanity data. Or else, which makes more sense, is to remove them
# from the beginning and not train the model on those. The latter choice is applied. 

# Donwload a list with bad words - we need this

# install.packages("RCurl")
library(RCurl)

profanity.source<-"https://raw.githubusercontent.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/master/en"
if (!file.exists("./data/en_US/profanity/en_bws.txt")){download.file(url=profanity.source , 
                destfile = "./data/en_US/profanity/en_bws.txt", method='libcurl')}

# remove bad words from corpora
bw.data<-read.table("./data/en_US/profanity/en_bws.txt", header=FALSE, sep="\n", strip.white=TRUE) # load bad words
names(bw.data)<-"bad.words"
docs<- tm_map(docs, removeWords, bw.data[,1]) # removal of profanity

# 13. Remove stop words
# From this point forward i keep two types of corpora - one with stop words included
# and one with stop words removed. The latter sample always has "nonstop" in the name of the file
docs.nonstop<-tm_map(docs, removeWords, stopwords("english"))

# 14. Remove whitespace

# The previous transformations generate a lot of extra whitespace in the lines of text; 
# applying the stripWhitespace transformation deals with this problem.
# I have tried different methods for this, but finally decided to use the tm package
docs <- tm_map(docs, stripWhitespace)
docs.nonstop <- tm_map(docs.nonstop, stripWhitespace)

# 15. Stem documents
# For the purposes of this porject I have made a choice to not use a stemmed corpus
# Logically we need to work with word types, not stems from them to form correct ngrams
# However, I offer the code i have tested for this step

# # install.packages("SnowballC")
# library("SnowballC")
# docs.stem <- tm_map(docs,stemDocument)
# docs.nonstop.stem <- tm_map(docs.nonstop,stemDocument)

# 16. Save data
# At this point i export the corpus with two versions - stop words included or not
# The reason is that pre-processing is very time consuming and shold be done once

save(docs, docs.nonstop, 
     file = "./data/en_US/train.sample/Rdata_output/train10pct.preprocsd.Rdata")

write(docs[[1]][[1]],"./data/en_US/train.sample/txt_output/train10pct.docs.preprocsd.txt")
write(docs.nonstop[[1]][[1]],"./data/en_US/train.sample/txt_output/train10pct.docs.nonstop.preprocsd.txt")

The resulting training files from this pre-processing step are (pre-processed in name:

• blogs.train70pct.docs.pre-processed.text (also news., twitter. versions)
• blogs.train70pct.docs.nonstop.pre-processed.text (also news., twitter. versions)
• train10pct.docs.pre-processed.text, train10pct.docs.nonstop.pre-processed.text (to be used for testing code)
• twitter.train10pct.docs.pre-processed.text, twitter.train10pct.docs.nonstop.pre-processed.text (to be used for testing code)

Appendix C: Tokenization and nGram Generation

I use package stylo to do tokenization. Since the ngrams can become very large, especially for 3grams and 4 grams, I tokenize the train70pct samples separately for blogs/news/tweets and aggregate the ngrams in a subsequent step.

# install.packages("stylo")
library(stylo)

###*********************** convert text to words with stylo **********************

### this step is done consecutively for docs[1] and doc.nonstop[1]
### and is applied to the train70pct and train10pct corpora from the previous step

### here we provide the example code for docs[1]
docs.words<- txt.to.words(docs[1])
###*******************************************************************************

###
# create data frames of one, two and three ngrams
###

###
# unigram
###
df1grm<-data.frame(table(make.ngrams(docs.words[[1]], ngram.size = 1))) # create 1gram

# remove rows with eeoss, nnumm, aabrr as these are not useful
df1grm <- df1grm[which(df1grm$Var1!="eeoss"),]
df1grm <- df1grm[which(df1grm$Var1!="nnumm"),]
df1grm <- df1grm[which(df1grm$Var1!="aabrr"),]

# calculate relative frequencies
df1grm$RelFreq <- round(100*(df1grm$Freq/sum(df1grm$Freq)),2)

# Create a sorted table "sdf*" by decending frequency count
sdf1grm<-df1grm[order(df1grm$Freq, decreasing = TRUE),];

# Calculate relative cumulative frequencies, and column with row numbers.
# this allows one to find how many words are e.g. 50% of dictionary to answer
# a question like this later on
sdf1grm$CumSum <- cumsum(sdf1grm$Freq); sdf1grm$RelCumSum <- 100*(sdf1grm$CumSum/sum(sdf1grm$Freq))
sdf1grm$RowNum <- 1:dim(sdf1grm)[1]
colnames(sdf1grm)<-c("OneGram","Frequency","RelFrequency", "CumFreq", "RelCumFreq", "RowNum")

# Build frequency of frequency table for subsequent Good-Turing smoothing
uni.freqfreq<-data.frame(Uni=table(sdf1grm$Frequency))

###
# bigram
###

df2grm<-data.frame(table(make.ngrams(docs.words[[1]], ngram.size = 2))) # create 2gram

# calculate relative frequencies
df2grm$RelFreq <- round(100*(df2grm$Freq/sum(df2grm$Freq)),2)

# remove rows with eeoss, nnumm, aabrr as these are not useful
# note that any bigrams, tri/fourgrams that contain any of these three words are incorrect
# because they are not sequences of words that were actually used in the text
# grepl looks for a pattern in the text of each row
eos.index<-grepl("eeoss",df2grm$Var1)
df2grm<-df2grm[!eos.index,]
num.index<-grepl("nnumm",df2grm$Var1)
df2grm<-df2grm[!num.index,]
abr.index<-grepl("aabrr",df2grm$Var1)
df2grm<-df2grm[!abr.index,]

# Create a sorted table "sdf*" by decending frequency count
sdf2grm<-df2grm[order(df2grm$Freq, decreasing = TRUE),]; colnames(sdf2grm)<-c("TwoGram","Frequency","RelFrequency")

# Build frequency of frequency table for Good-Turing smoothing
di.freqfreq<-data.frame(Di=table(sdf2grm$Frequency))

###
# trigram
###

df3grm<-data.frame(table(make.ngrams(docs.words[[1]], ngram.size = 3))) # create 3gram

# calculate relative frequencies
df3grm$RelFreq <- round(100*(df3grm$Freq/sum(df3grm$Freq)),2)

# remove rows with eeoss, nnumm, aabrr as these are not useful
eos.index<-grepl("eeoss",df3grm$Var1)
df3grm<-df3grm[!eos.index,]
num.index<-grepl("nnumm",df3grm$Var1)
df3grm<-df3grm[!num.index,]
abr.index<-grepl("aabrr",df3grm$Var1)
df3grm<-df3grm[!abr.index,]

# Create a sorted table "sdf*" by decending frequency count
sdf3grm<-df3grm[order(df3grm$Freq, decreasing = TRUE),]; colnames(sdf3grm)<-c("TriGram","Frequency","RelFrequency")

# separate singletons
sdf3grm.2pl<-sdf3grm[which(sdf3grm$Frequency>=2),]
sdf3grm.singl<-sdf3grm[which(sdf3grm$Frequency==1),]

# Build frequency of frequency table for Good-Turing smoothing
tri.freqfreq<-data.frame(Tri=table(sdf3grm$Frequency))

###
# fourgram
###

df4grm<-data.frame(table(make.ngrams(docs.words[[1]], ngram.size = 4)))  # create 4gram

# calculate relative frequencies
df4grm$RelFreq <- round(100*(df4grm$Freq/sum(df4grm$Freq)),2)

# remove rows with eeoss, nnumm, aabrr as these are not useful
eos.index<-grepl("eeoss",df4grm$Var1)
df4grm<-df4grm[!eos.index,]
num.index<-grepl("nnumm",df4grm$Var1)
df4grm<-df4grm[!num.index,]
abr.index<-grepl("aabrr",df4grm$Var1)
df4grm<-df4grm[!abr.index,]

# Create a sorted table "sdf*" by decending frequency count
sdf4grm<-df4grm[order(df4grm$Freq, decreasing = TRUE),]; colnames(sdf4grm)<-c("FourGram","Frequency","RelFrequency")

# separate singletons
sdf4grm.2pl<-sdf4grm[which(sdf4grm$Frequency>=2),]
sdf4grm.singl<-sdf4grm[which(sdf4grm$Frequency==1),]

# Build frequency of frequency table for Good-Turing smoothing
four.freqfreq<-data.frame(Four=table(sdf4grm$Frequency))

###
# Save ngrams data
###

save(docs, docs.words, sdf1grm, sdf2grm, sdf3grm, sdf3grm.2pl, sdf3grm.singl, 
     sdf4grm, sdf4grm.2pl, sdf4grm.singl, uni.freqfreq, di.freqfreq, tri.freqfreq, four.freqfreq,
     file = "./data/en_US/train.sample/Rdata_output/train10pct_ngram_docs.Rdata")

# (for illustrative purposes only)
# save ngram data for the alternative nonstop corpus 
save(docs.nonstop, docs.words, sdf1grm, sdf2grm, sdf3grm, sdf3grm.2pl, sdf3grm.singl, 
     sdf4grm, sdf4grm.2pl, sdf4grm.singl, uni.freqfreq, di.freqfreq, tri.freqfreq, four.freqfreq,
     file = "./data/en_US/train.sample/Rdata_output/train10pct_ngram_docs.nonstop.Rdata")

Input files for the last step were:

• blogs.train70pct.pre-processed.Rdata, news.train70pct.pre-processed.Rdata, twitter.train70pct.pre-processed.Rdata
• train10pct.pre-processed.Rdata

Output files for the last step were:

• blogs.train70pct_ngram_docs.Rdata, news.train70pct_ngram_docs.Rdata, twitter.train70pct_ngram_docs.Rdata
• blogs.train70pct_ngram_docs.nonstop.Rdata, news.train70pct_ngram_docs.nonstop.Rdata, twitter.train70pct_ngram_docs.nonstop.Rdata
• train10pct_ngram_docs.Rdata, train10pct_ngram_docs.nonstop.Rdata

In the next step I aggregate the blogs/news/twitter train70pct samples ngrams into combined ngrams. I provide an example of the code used for files with included stop words. The same code is applied to files with stop words excluded.

###*************************************
### aggregate large training sample ####
###*************************************

####
## with stop words
####

# To load RData, one object at a time, we use library R.utils
# install.packages("R.utils")

library(R.utils); 

###
## unigram
###

setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

sdf1grm.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.Rdata")[["sdf1grm"]] 
sdf1grm.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.Rdata")[["sdf1grm"]]
sdf1grm.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.Rdata")[["sdf1grm"]]

# vertically bind the ngrams across three sources
df1grm <- rbind(sdf1grm.blogs[,1:2], sdf1grm.news[,1:2], sdf1grm.twitter[,1:2])

rm(sdf1grm.blogs,sdf1grm.news,sdf1grm.twitter)

## group and summarize using dplyr library
library(dplyr)

sdf1grm <- df1grm %>% group_by(OneGram) %>% summarize(Frequency = sum(Frequency) )
rm(df1grm)
sdf1grm <- as.data.frame(sdf1grm); sdf1grm <- sdf1grm[order(-sdf1grm$Frequency),]

# calculate relative frequencies
sdf1grm$RelFreq <- round(100*(sdf1grm$Frequency/sum(sdf1grm$Frequency)),2)

## Calculate relative cumulative frequencies, and column with row numbers.
## this allows one to find how many words are e.g. 50% of dictionary
sdf1grm$CumSum <- cumsum(sdf1grm$Freq); sdf1grm$RelCumSum <- 100*(sdf1grm$CumSum/sum(sdf1grm$Freq))
sdf1grm$RowNum <- 1:dim(sdf1grm)[1]
colnames(sdf1grm)<-c("OneGram","Frequency","RelFrequency", "CumFreq", "RelCumFreq", "RowNum")

# Build frequency of frequency table for Good-Turing smoothing
uni.freqfreq<-data.frame(Uni=table(sdf1grm$Frequency))

# save combined data for 1gram
save(sdf1grm, uni.freqfreq,
     file = "./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.Rdata")
rm(sdf1grm, uni.freqfreq)

###
## bigram
###

setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

sdf2grm.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.Rdata")[["sdf2grm"]] 
sdf2grm.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.Rdata")[["sdf2grm"]]
sdf2grm.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.Rdata")[["sdf2grm"]]

# vertically bind the ngrams across three sources
df2grm <- rbind(sdf2grm.blogs[,1:2], sdf2grm.news[,1:2], sdf2grm.twitter[,1:2])

rm(sdf2grm.blogs,sdf2grm.news,sdf2grm.twitter)

## group and summarize using dplyr library
library(dplyr)

sdf2grm <- df2grm %>% group_by(TwoGram) %>% summarize(Frequency = sum(Frequency) )
rm(df2grm)
sdf2grm <- as.data.frame(sdf2grm); sdf2grm <- sdf2grm[order(-sdf2grm$Frequency),]

# calculate relative frequencies
sdf2grm$RelFreq <- round(100*(sdf2grm$Frequency/sum(sdf2grm$Frequency)),2)

# Build frequency of frequency table for Good-Turing smoothing
di.freqfreq<-data.frame(Di=table(sdf2grm$Frequency))

# save combined data for 2gram
save(sdf2grm, di.freqfreq,
     file = "./data/en_US/train.sample/Rdata_output/train70pct_2gram_docs.Rdata")
rm(sdf2grm, di.freqfreq)

###
## trigram
###

setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

sdf3grm.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.Rdata")[["sdf3grm"]] 
sdf3grm.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.Rdata")[["sdf3grm"]]
sdf3grm.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.Rdata")[["sdf3grm"]]

# vertically bind the ngrams across three sources
df3grm <- rbind(sdf3grm.blogs[,1:2], sdf3grm.news[,1:2], sdf3grm.twitter[,1:2])

rm(sdf3grm.blogs,sdf3grm.news,sdf3grm.twitter)

## group and summarize using dplyr library
library(dplyr)

sdf3grm <- df3grm %>% group_by(TriGram) %>% summarize(Frequency = sum(Frequency) )
rm(df3grm)
sdf3grm <- as.data.frame(sdf3grm); sdf3grm <- sdf3grm[order(-sdf3grm$Frequency),]

# calculate relative frequencies
sdf3grm$RelFreq <- round(100*(sdf3grm$Frequency/sum(sdf3grm$Frequency)),2)

# Build frequency of frequency table for Good-Turing smoothing
tri.freqfreq<-data.frame(Tri=table(sdf3grm$Frequency))

# separate singletons
sdf3grm.2pl<-sdf3grm[which(sdf3grm$Frequency>=2),]
sdf3grm.singl<-sdf3grm[which(sdf3grm$Frequency==1),]

# save combined data for 3gram
save(sdf3grm, tri.freqfreq, 
     file = "./data/en_US/train.sample/Rdata_output/train70pct_3gram_docs.Rdata")
# save pre-processed data
save(sdf3grm.2pl, tri.freqfreq,
     file = "./data/en_US/train.sample/Rdata_output/train70pct_3gram.2pl_docs.Rdata")
# save pre-processed data
save(sdf3grm.singl, 
     file = "./data/en_US/train.sample/Rdata_output/train70pct_3gram.singl_docs.Rdata")

rm(sdf3grm, tri.freqfreq, sdf3grm.2pl, sdf3grm.singl)

###
## fourgram
###

setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

sdf4grm.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.Rdata")[["sdf4grm"]] 
sdf4grm.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.Rdata")[["sdf4grm"]]
sdf4grm.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.Rdata")[["sdf4grm"]]

# vertically bind the ngrams across three sources
df4grm <- rbind(sdf4grm.blogs[,1:2], sdf4grm.news[,1:2], sdf4grm.twitter[,1:2])

rm(sdf4grm.blogs,sdf4grm.news,sdf4grm.twitter)

## group and summarize using dplyr library
library(dplyr)

sdf4grm <- df4grm %>% group_by(FourGram) %>% summarize(Frequency = sum(Frequency) )
rm(df4grm)
sdf4grm <- as.data.frame(sdf4grm); sdf4grm <- sdf4grm[order(-sdf4grm$Frequency),]

# calculate relative frequencies
sdf4grm$RelFreq <- round(100*(sdf4grm$Frequency/sum(sdf4grm$Frequency)),2)

# Build frequency of frequency table for Good-Turing smoothing
four.freqfreq<-data.frame(Four=table(sdf4grm$Frequency))

# separate singletons
sdf4grm.2pl<-sdf4grm[which(sdf4grm$Frequency>=2),]
sdf4grm.singl<-sdf4grm[which(sdf4grm$Frequency==1),]

# save combined data for 4gram
save(sdf4grm, four.freqfreq, 
     file = "./data/en_US/train.sample/Rdata_output/train70pct_4gram_docs.Rdata")
# save pre-processed data
save(sdf4grm.2pl, four.freqfreq, 
     file = "./data/en_US/train.sample/Rdata_output/train70pct_4gram.2pl_docs.Rdata")
# save pre-processed data
save(sdf4grm.singl, 
     file = "./data/en_US/train.sample/Rdata_output/train70pct_4gram.singl_docs.Rdata")

rm(sdf4grm, four.freqfreq, sdf4grm.2pl, sdf4grm.singl)

Output files for the last step were:

• blogs.train70pct_ngram_docs.Rdata, news.train70pct_ngram_docs.Rdata, twitter.train70pct_ngram_docs.Rdata
• blogs.train70pct_ngram_docs.nonstop.Rdata, news.train70pct_ngram_docs.nonstop.Rdata, twitter.train70pct_ngram_docs.nonstop.Rdata

Output files from the last step were:

• train70pct_1gram_docs.Rdata, train70pct_2gram_docs.Rdata
• train70pct_3gram_docs.Rdata, train70pct_3gram.2pl_docs.Rdata, train70pct_3gram.singl_docs.Rdata
• train70pct_4gram_docs.Rdata, train70pct_4gram.2pl_docs.Rdata, train70pct_4gram.singl_docs.Rdata
• except for the train70pct_3gram.singl and train70pct_4gram.singl, the Rdata files also include the frequence of frequence tables

# TriGrams
 sdf3grm.2pl.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.Rdata")[["sdf3grm.2pl"]] 
 sdf3grm.2pl.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.Rdata")[["sdf3grm.2pl"]]
 sdf3grm.2pl.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.Rdata")[["sdf3grm.2pl"]]
 save(sdf3grm.2pl.blogs, file = "./data/en_US/train.sample/Rdata_output/blogs.train70pct_3gram.2pl_docs.Rdata")
 save(sdf3grm.2pl.news, file = "./data/en_US/train.sample/Rdata_output/news.train70pct_3gram.2pl_docs.Rdata")
 save(sdf3grm.2pl.twitter, file = "./data/en_US/train.sample/Rdata_output/twitter.train70pct_3gram.2pl_docs.Rdata")
 
 sdf3grm.2pl.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.nonstop.Rdata")[["sdf3grm.2pl"]] 
 sdf3grm.2pl.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.nonstop.Rdata")[["sdf3grm.2pl"]]
 sdf3grm.2pl.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.nonstop.Rdata")[["sdf3grm.2pl"]]
 
 save(sdf3grm.2pl.blogs, file = "./data/en_US/train.sample/Rdata_output/blogs.train70pct_3gram.2pl_docs.nonstop.Rdata")
 save(sdf3grm.2pl.news, file = "./data/en_US/train.sample/Rdata_output/news.train70pct_3gram.2pl_docs.nonstop.Rdata")
 save(sdf3grm.2pl.twitter, file = "./data/en_US/train.sample/Rdata_output/twitter.train70pct_3gram.2pl_docs.nonstop.Rdata")

# UniGrams  
sdf1grm.blogs <- loadToEnv("./data/en_US/train.sample/Rdata_output/blogs.train70pct_ngram_docs.nonstop.Rdata")[["sdf1grm"]] 
sdf1grm.news <- loadToEnv("./data/en_US/train.sample/Rdata_output/news.train70pct_ngram_docs.nonstop.Rdata")[["sdf1grm"]]
sdf1grm.twitter <- loadToEnv("./data/en_US/train.sample/Rdata_output/twitter.train70pct_ngram_docs.nonstop.Rdata")[["sdf1grm"]]

save(sdf1grm.blogs, file = "./data/en_US/train.sample/Rdata_output/blogs.train70pct_1gram_docs.nonstop.Rdata")
save(sdf1grm.news, file = "./data/en_US/train.sample/Rdata_output/news.train70pct_1gram_docs.nonstop.Rdata")
save(sdf1grm.twitter, file = "./data/en_US/train.sample/Rdata_output/twitter.train70pct_1gram_docs.nonstop.Rdata")

Appendix D: Data Characteristics and Exporatory Analysis

datafolder <- "/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10/data/en_US"
flist <- list.files(path=datafolder, recursive=F, pattern=".*en_.*.txt")

# lapply returns a list
l <- lapply(paste(datafolder, flist, sep="/"), function(f) {
        fsize <- file.info(f)[1]/1024/1024
        con <- file(f, open="r")
        lines <- readLines(con)
#        nchars <- lapply(lines, nchar)
#        maxchars <- which.max(nchars)
        nwords <- sum(sapply(strsplit(lines, "\\s+"), length))
        close(con)
        # round to two decimal points, but if the first two decimals are 00, then still display them
        return(c(f, format(round(fsize, 2), nsmall=2), length(lines), nwords))
})

# unlist the list and display the results
df <- data.frame(matrix(unlist(l), nrow=length(l), byrow=T))
colnames(df) <- c("file", "size(MB)", "num.of.lines", "num.of.words")

filenames <- data.frame(file.name = c("en_US.blogs.txt", "en_US.news.txt", "en_US.twitter.txt"))
df <- cbind(filenames,df[,2:4])
df

###
# ngram plots including stop words
###

rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

# load 1/2/3grams 
load("./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_2gram_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_3gram.2pl_docs.Rdata", verbose = FALSE)

# extract top 20 ngrams by frequency
top20oneg<-sdf1grm[1:20,]; top20oneg$OneGram <- as.character(top20oneg$OneGram)
top20twog<-sdf2grm[1:20,]; top20twog$TwoGram <- as.character(top20twog$TwoGram)
top20trig<-sdf3grm.2pl[1:20,]; top20trig$TriGram <- as.character(top20trig$TriGram)

rm(sdf1grm, sdf2grm, sdf3grm)

library(ggplot2)
#install.packages("ggpubr")
library(ggpubr) # use to align three plots in one row

p1 <- ggplot (top20oneg, aes(x = reorder(OneGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "lightgreen" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "OneGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

p2 <- ggplot (top20twog, aes(x = reorder(TwoGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "lightblue" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFreq, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "TwoGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

p3 <- ggplot (top20trig, aes(x = reorder(TriGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFreq, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "TriGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

# Arrange plots (3,1)
 ggarrange(p1, p2, p3 + rremove("x.text"), 
#           labels = c("A", "B", "C"),
          labels = NULL,
          ncol = 3, nrow = 1)

###
# ngram plots excluding stop words
###

rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

# load 1/2/3grams  
load("./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_2gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_3gram.2pl_docs.nonstop.Rdata", verbose = FALSE)
 
# extract top 20 ngrams by frequency
top20oneg<-sdf1grm[1:20,]; top20oneg$OneGram <- as.character(top20oneg$OneGram)
top20twog<-sdf2grm[1:20,]; top20twog$TwoGram <- as.character(top20twog$TwoGram)
top20trig<-sdf3grm.2pl[1:20,]; top20trig$TriGram <- as.character(top20trig$TriGram)

rm(sdf1grm, sdf2grm, sdf3grm)

library(ggplot2)
#install.packages("ggpubr")
library(ggpubr) # use to align three plots in one row

p1 <- ggplot (top20oneg, aes(x = reorder(OneGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "lightgreen" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "OneGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

p2 <- ggplot (top20twog, aes(x = reorder(TwoGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "lightblue" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFreq, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "TwoGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

p3 <- ggplot (top20trig, aes(x = reorder(TriGram, Frequency), y= Frequency )) + 
        geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
        coord_flip() +
        geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
        geom_text( aes (label = paste(RelFreq, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
        xlab( "TriGram List" ) +
        ylab( "Freq & RelFreq %" ) +
        theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
        theme_bw()

# Arrange plots (3,1)
 ggarrange(p1, p2, p3 + rremove("x.text"), 
#           labels = c("A", "B", "C"),
          labels = NULL,
          ncol = 3, nrow = 1)

###
# wordcloud plots of 3grams with excluded stop words
###
 
# install.packages("wordcloud")
library(wordcloud)
 
load("./data/en_US/train.sample/Rdata_output/train70pct_3gram.2pl_docs.nonstop.Rdata", verbose = FALSE)

wordcloud(words = sdf3grm.2pl$TriGram, freq = sdf3grm.2pl$Freq, min.freq = 3,
          max.words=100, random.order=FALSE, rot.per=0.35, 
          colors=brewer.pal(8, "Dark2")) #,scale=c(2, 1.2)) 

 ###
 # 3grams from each source including stop words
 ###

rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

load("./data/en_US/train.sample/Rdata_output/blogs.train70pct_3gram.2pl_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/news.train70pct_3gram.2pl_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/twitter.train70pct_3gram.2pl_docs.Rdata", verbose = FALSE)
 
 # extract top 20 by freq
 top20trig.blogs<-sdf3grm.2pl.blogs[1:20,]; top20trig.blogs$TriGram <- as.character(top20trig.blogs$TriGram)
 top20trig.news<-sdf3grm.2pl.news[1:20,]; top20trig.news$TriGram <- as.character(top20trig.news$TriGram)
 top20trig.twitter<-sdf3grm.2pl.twitter[1:20,]; top20trig.twitter$TriGram <- as.character(top20trig.twitter$TriGram)

 library(ggplot2)
 #install.packages("ggpubr")
 library(ggpubr)
 
 pp1 <- ggplot (top20trig.blogs, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram Blogs" ) +
         ylab( "Freq & RelFreq %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()
 
 pp2 <- ggplot (top20trig.news, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram News" ) +
         ylab( "Freq & RelFreq %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()
 
 pp3 <- ggplot (top20trig.twitter, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram Twits" ) +
         ylab( "Freq & RelFrequency %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()

# Arrange plots (3,1)  
 ggarrange(pp1, pp2, pp3 + rremove("x.text"), 
           #labels = c("A", "B", "C"),
           labels = NULL,
           ncol = 3, nrow = 1)

###
# 3grams from each source excluding stop words
###
 
rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")
  
load("./data/en_US/train.sample/Rdata_output/blogs.train70pct_3gram.2pl_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/news.train70pct_3gram.2pl_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/twitter.train70pct_3gram.2pl_docs.nonstop.Rdata", verbose = FALSE)
 
 # extract top 20 by freq
 top20trig.blogs<-sdf3grm.2pl.blogs[1:20,]; top20trig.blogs$TriGram <- as.character(top20trig.blogs$TriGram)
 top20trig.news<-sdf3grm.2pl.news[1:20,]; top20trig.news$TriGram <- as.character(top20trig.news$TriGram)
 top20trig.twitter<-sdf3grm.2pl.twitter[1:20,]; top20trig.twitter$TriGram <- as.character(top20trig.twitter$TriGram)

 library(ggplot2)
 #install.packages("ggpubr")
 library(ggpubr)
 
 pp1 <- ggplot (top20trig.blogs, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram Blogs" ) +
         ylab( "Freq & RelFreq %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()
 
 pp2 <- ggplot (top20trig.news, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram News" ) +
         ylab( "Freq & RelFreq %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()
 
 pp3 <- ggplot (top20trig.twitter, aes(x = reorder(TriGram, - Frequency), y= Frequency )) + 
         geom_bar( stat = "Identity" , fill = "mistyrose" ) +  
         coord_flip() +
         geom_text( aes (label = Frequency ) , vjust = - 0.3, hjust = 1.1, size = 2.5) +
         geom_text( aes (label = paste(RelFrequency, "%") ) , vjust = + 1.30, hjust = 1.1, size = 2.5) +
         xlab( "TriGram Twits" ) +
         ylab( "Freq & RelFrequency %" ) +
         theme ( axis.text.x = element_text ( angle = 45 , hjust = 1 ) ) +
         theme_bw()

# Arrange plots (3,1)  
 ggarrange(pp1, pp2, pp3 + rremove("x.text"), 
           #labels = c("A", "B", "C"),
           labels = NULL,
           ncol = 3, nrow = 1)

###
# Cumulative frequences for 1grams excluding stop words
###
 
# How many unique words do you need in a frequency sorted dictionary to cover 50% 
# of all word instances in the language? 90%?
 
rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")


load("./data/en_US/train.sample/Rdata_output/blogs.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/news.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/twitter.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)

# plot number of words vs culative relative frequency.
# this shows how dictinary coverage increases with addition of more words
# it is an increasing with a decreasing rate process. first deriv+, second deriv-
plot(sdf1grm$RowNum[1:50000], sdf1grm$RelCumFreq[1:50000], type="l", lty=1, 
     ylab = "Cumulative Relative Frequency (0%-100% coverage)", xlab = "Top Frequency Words")
lines(sdf1grm.blogs$RelCumFreq, type="l", lty=1, col = "blue")
lines(sdf1grm.news$RelCumFreq, type="l", lty=1, col = "green")
lines(sdf1grm.twitter$RelCumFreq, type="l", lty=1, col = "red")
legend("bottomright", 
       legend = c("Combined", "Blogs", "News", "Tweets"), 
       col = c("black","blue", "green", "red" ), 
       pch = c(19,19, 19, 19), bty = "n", pt.cex = 2, cex = 1.2, 
       text.col = c("black","blue", "green", "red" ), horiz = F 
       #       inset = c(0.1, 0.1))
)

# find some quantiles, that show how quickly the number of words needed rises to 
# achieve more covereage of the dictionary
paste("number of words comprising 49.99 to 50.01% of the dictionary: ", range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 49.99 & sdf1grm$RelCumFreq < 50.01])[1], range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 49.99 & sdf1grm$RelCumFreq < 50.01])[2])
paste("number of words comprising 50.01 to 60.01% of the dictionary: ",range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 50.01 & sdf1grm$RelCumFreq < 60.01])[1],range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 50.01 & sdf1grm$RelCumFreq < 60.01])[2])
paste("number of words comprising 60.01 to 70.01% of the dictionary: ",range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 60.01 & sdf1grm$RelCumFreq < 70.01])[1],range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 60.01 & sdf1grm$RelCumFreq < 70.01])[2])
paste("number of words comprising 70.01 to 80.01% of the dictionary: ",range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 70.01 & sdf1grm$RelCumFreq < 80.01])[1],range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 70.01 & sdf1grm$RelCumFreq < 80.01])[2])
paste("number of words comprising 80.01 to 90.01% of the dictionary: ",range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 80.01 & sdf1grm$RelCumFreq < 90.01])[1],range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 80.01 & sdf1grm$RelCumFreq < 90.01])[2])
paste("number of words comprising 90.01 to 95.01% of the dictionary: ",range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 89.99 & sdf1grm$RelCumFreq < 90.01])[2],range(sdf1grm$RowNum[sdf1grm$RelCumFreq > 90.01 & sdf1grm$RelCumFreq < 95.01])[2])

###
# Generates TTR: word.type to token ratio for 1gram including stop words. 
# It demonstrates how rich the language is in a particular corpora. 
# The higher TTR the more unique words are used to achieve to number of tokens
###

rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

load("./data/en_US/train.sample/Rdata_output/blogs.train70pct_1gram_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/news.train70pct_1gram_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/twitter.train70pct_1gram_docs.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.Rdata", verbose = FALSE)

# calculate TTR by source and combined
blogs.tokens <- sum(sdf1grm.blogs$Frequency); blogs.types <- dim(sdf1grm.blogs)[1];
blogs.TTR <- blogs.types/blogs.tokens;
news.tokens <- sum(sdf1grm.news$Frequency); news.types <- dim(sdf1grm.news)[1];
news.TTR <- news.types/news.tokens;
twitter.tokens <- sum(sdf1grm.twitter$Frequency); twitter.types <- dim(sdf1grm.twitter)[1];
twitter.TTR <- twitter.types/twitter.tokens;
tokens <- sum( sdf1grm$Frequency); types <- dim(sdf1grm)[1];
TTR <- types/tokens;

TTR <- data.frame("Source" = c("Blogs", "News", "Twitter", "Corpus"),
                          "Tokens" = c(blogs.tokens,news.tokens,twitter.tokens,tokens), 
                          "Word.Types" = c(blogs.types,news.types,twitter.types,types),
                          "TTR" = c(blogs.TTR,news.TTR,twitter.TTR,TTR))
print("Word Types to tokens ratio (TTR) for corpora with NO stop words removed")
TTR

###
# Generates TTR: word.type to token ratio for 1gram excluding stop words. 
# It demonstrates how rich the language is in a particular corpora. 
# The higher TTR the more unique words are used to achieve to number of tokens
###

rm(list=ls())
setwd("/Users/nikolaydobrinov/Documents/work/Courses/R/WorkDirectory/Course10")

load("./data/en_US/train.sample/Rdata_output/blogs.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/news.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/twitter.train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)
load("./data/en_US/train.sample/Rdata_output/train70pct_1gram_docs.nonstop.Rdata", verbose = FALSE)

# calculate TTR by source and combined
blogs.nonstop.tokens <- sum(sdf1grm.blogs$Frequency); blogs.nonstop.types <- dim(sdf1grm.blogs)[1];
blogs.nonstop.TTR <- blogs.nonstop.types/blogs.nonstop.tokens;
news.nonstop.tokens <- sum(sdf1grm.news$Frequency); news.nonstop.types <- dim(sdf1grm.news)[1];
news.nonstop.TTR <- news.nonstop.types/news.nonstop.tokens;
twitter.nonstop.tokens <- sum(sdf1grm.twitter$Frequency); twitter.nonstop.types <- dim(sdf1grm.twitter)[1];
twitter.nonstop.TTR <- twitter.nonstop.types/twitter.nonstop.tokens;
nonstop.tokens <- sum( sdf1grm$Frequency); nonstop.types <- dim(sdf1grm)[1];
nonstop.TTR <- nonstop.types/nonstop.tokens;


TTR.nonstop <- data.frame("Source" = c("Blogs", "News", "Twitter", "Corpus"),
                "Tokens" = c(blogs.nonstop.tokens,news.nonstop.tokens,twitter.nonstop.tokens,nonstop.tokens), 
                "Word.Types" = c(blogs.nonstop.types,news.nonstop.types,twitter.nonstop.types,nonstop.types),
                "TTR" = c(blogs.nonstop.TTR,news.nonstop.TTR,twitter.nonstop.TTR,nonstop.TTR))
print("Word Types to tokens ratio (TTR) for corpora with stop words removed")
TTR.nonstop