This is an original work prepared for the capstone of the Johns Hopkins Universty specialization in data science offered through Cousera.org.

Objectives & Scope

The ultimate goal of the analysis is to enable users to quickly enter text by predicting the next word and providing it as a one-touch option rather than having the user type the word. This application would be useful in almost every mobile application, as well as in computer-based tools such as search engines and social media.

To illustrate our techniques and provide possible investors, buyers and the public with “try it before you buy it” access, we will be creating and hosting a website that allows one to enter any phrase and press a button to see both the best predicton of the next word and the next five top predictions.

For simplicity we will be limiting our prediction targets to English only. Possibilities do exist to extend the algorithms to other languages in future.

We also assume that profanity will not be predicted. The likelihood of offending users is assumed to be higher than the usefulness of such predications.

We assume that users of our tools are typical of other internet users. For example, we do not specialize for users who are native or non-native English-speakers. Further, we do not specialize within specific dialects of English, although we do focus on American English rather than another flavour of English such as British or Australian.

Our initial algorithm and its testing will be based around three available data sets derived from American-English-based tweets, blogs and news captions. This data is easily available, and extensive. It is also focused on internet-based information - tweets and blogs, as well as less formal news headlines. This means that it is quite representative of our target usage, making it ideal for our initial purposes.

Time permitting we will explore other sources of data to enrich our algorithm and/or test its efficacy. These sources have not yet been selected. Numerous issues are involved in selecting additional data sources including (1) ownership of the data for this purpose and (2) suitability for prediction of social media phrases. Possible areas for research include: Facebook, P-Interest, and Redit. Other commonly used options such as wikipedia or news agencies have been rejected because of the relative formality of their writing compared to typical social media and interpersonal communications.

Planned Approach

Stage one of the plan was to study the selected data and understand what it has to offer, as well as some of the available analysis tools. The next section provides some insights into the data we plan to use. In the second stage, we will look in more detail at text structures, examining common combinations of words and word ordering.

Based on work presented in a lecture series at the University of Toronto by Geoffrey Hinton, we look at a simple predication model. Words often appear in similar phrases over and over again. If we have a large amount of text, we can examine all of it in three-word strings. For example, in the sentence: “Farewell cruel world, I leave you now.” there are these five three-word strings: “Farewell cruel world”, “cruel world, I”, “world I leave”, “I leave you” and “leave you now.” We can locate likely next words based on the two previous words simply by calculating how often the two words have appeared together in our data, and then looking at what comes next as the third word, and how often. This is a simple, effective method. However, it is dependent on (1) having seen the previous two words together before in our data set, and (2) being able to choose from potentially hundreds of equally-likely choices for the third word.

While we will be implementing the algorithm above, we will also be looking for techniques that address its key limitations. For example, Stanford University provides a good overview of both the technique we plan to use (called “n-grams”) and some of the modifications and extensions to it that have been attempted. These include smoothing algorithms to enable predictions of heretofore unseen combinations and the use of a thesaurus to enrich the array or predictions using synonyms for specific predicted words.

Data Analysis

Our initial data supporting the analysis consists of three files, each representing a single source of social text and containing thousands of examples. The files are:

Type of Info File Size Number of Samples
Blogs 200 MB 899288
News Headlines 196 MB 1010242
Twitter Tweets 159 MB 2360148

So, we have approximately four million samples of social text upon which we can base our analysis. Clearly, while extensive, it is unlikely that this list will prove exhaustive. It is also clear that extensive processing would need to be done to perform analysis of so much text. To get familiar with our data, we therefore chose to select about 10% of it at random. The code shown in Appendix 1 was used to separate each file into 100 smaller files. From these, 10 were chosen at random, by hand, to form the data in the next stage of the analysis.

Twitter - 10% Sample

Taking 10% of the twitter file available, at random, we perform a basic analysis of the data and produce some statistics. This analysis, including code and format, borrows heavily from a blog post by Kailash Awati and has been reworked to fit this data and our purposes.

  • Number of identified terms is 86116
  • Number of words occurring 1 time is 56657
  • The number of words occurring more than 100 times is 2166

The most common word, and its number of occurrences are:

##  just 
## 15019

A short list of the 20 highest-frequency terms is as follows:

##  [1] "can"    "day"    "dont"   "follow" "get"    "good"   "great" 
##  [8] "just"   "know"   "like"   "love"   "make"   "new"    "now"   
## [15] "one"    "see"    "thank"  "time"   "today"  "will"

To help visualize the most common words, consider the following word cloud where size and colour are used to indicate frequency.

To better see how words occur in tweets, we examine the following histogram. It is representative of the entire distribution of word frequencies, but for clarity of visualization shows only words that occure more than 100 times and fewer than 1/20 the number of times of the most frequent word. Clearly, the frequency of words declines rapidly and continuously.

All Types of Social Text - 10% Sample

Taking 10% of each file available, at random, we create a single, combined data set and perform a basic analysis of the data and produce some statistics. This analysis, including code and format, borrows heavily from a blog post by Kailash Awati and has been reworked to fit this data and our purposes.

  • Number of identified terms is 168024
  • Number of words occurring 1 time is 105164
  • The number of words occurring more than 100 times is 5193

The most common word, and its number of occurrences are:

##  will 
## 30759

A short list of the 20 highest-frequency terms is as follows:

##  [1] "can"   "day"   "dont"  "get"   "good"  "just"  "know"  "like" 
##  [9] "love"  "make"  "new"   "now"   "one"   "peopl" "said"  "see"  
## [17] "time"  "will"  "work"  "year"

To help visualize the most common words, consider the following chart indicating frequency.

Below we see a histogram of all words with frequency of at least 5% of that of the most common word. Here we see a pattern that is largely as expected. However, there seem to be fewer words at the lowest frequencies. This is just an artifact of how we cut off the data to show only high-frequency terms. So, the expected pattern of the number of words with a frequency being inversely related to the level of the frequency is maintained.

Final Thoughts on Data Analysis

Of interest is the significant similarity in the “Top 20” lists from the two analyses. They differ by only four terms. This supports the ideas that (1) 10% was likely a suitable sample size and (2) the three sources of data are largely similar. However, it is not conclusive.

Conclusion

Having understood our data and reviewed it, we are now ready to take on the second stage, constructing a model to look at n-grams, those three-word phrases that we will use for prediction.

Appendix: Sample Code

Showing the Random Splitting of Files

In this appendix, we present the code used to split the twitter corpus into one hundred files. From those 100 files, 10 were randomly selected (manually) to form the basis of the analysis. Thus, approximately 10% of the corpus was used to represent the entire population of the corpus for the purposes of this report.

maxlines <- 250
numfiles <- 100

numlines <- as.numeric(gsub(" ", "", gsub("_", " ", gsub("\\.+", " ", gsub("[a-z]+", 
    " ", tolower(system("wc -l en_US.twitter.txt", intern = TRUE)))))))

confile <- file("en_US.twitter.txt", "rt")

read <- 0
while (read < numlines) {
    print(read)
    lr <- sample(1:maxlines, 1)
    txt <- readLines(con = confile, n = lr, ok = TRUE, skipNul = TRUE)
    read <- read + lr
    rf <- sample(1:numfiles, 1)
    fileout <- file(paste("./corpus/twitter/", as.character(rf), ".txt", sep = ""), 
        open = "at")
    writeLines(text = txt, con = fileout)
    flush(fileout)
    sink()
    close(fileout)
}

close(confile)

Showing the Data Analysis Code

############# Analysis of the Corpus from 'Twitter' - One Word at a Time The analysis in
############# this section heavily borrows from an online tutorial: 'A Gentle
############# Introduction to Text Mining Using R' by Kailash Awati Link:
############# https://eight2late.wordpress.com/2015/05/27/a-gentle-introduction-to-text-mining-using-r/

docs <- Corpus(DirSource("./final/en_US/corpus/combinedsmall"))
docs <- tm_map(docs, removePunctuation)
docs <- tm_map(docs, removeNumbers)
docs <- tm_map(docs, content_transformer(tolower))
docs <- tm_map(docs, removeWords, stopwords("english"))

swears = c("fuck", "shit", "ass", "god", "damn", "fuckyou", "BAMF", "shitty", 
    "shite", "shity", "fuckoff", "goddamn", "penis", "dick", "piss", "suck", 
    "cunt", "bootycall", "fck", "sht", "pss", "dck", "fucking", "fcking", "fking", 
    "pissing", "shitting", "shiting", "damning", "damned", "fucked", "shat", 
    "fcked", "pissed", "sucking", "sucked", "asses", "cunts", "motherfucker")
docs <- tm_map(docs, removeWords, swears)
docs <- tm_map(docs, stripWhitespace)

stems <- tm_map(docs, stemDocument)
stems <- tm_map(stems, removeWords, swears)
stems <- tm_map(stems, stripWhitespace)

dtm <- DocumentTermMatrix(stems)
dtmdocs <- DocumentTermMatrix(docs)

############### Frequencies #

freq <- colSums(as.matrix(dtm))
ord <- order(freq, decreasing = TRUE)
hifreq <- findFreqTerms(dtm, lowfreq = freq[[ord[20]]])
usual <- findFreqTerms(dtm, lowfreq = 5)
forcorr <- findFreqTerms(dtm, lowfreq = round(freq[[ord[1]]]/12.5))

print("Number of identified terms is ")
print(length(freq))
print("Most common word, and its number of occurrences are: ")
print(freq[ord[1]])
print("The number of words appearing four or fewer times is: ")
print(length(freq) - length(usual))
print("  ")
print("A short list of the 20 highest-frequency terms is as follows ")
print(hifreq)

df = data.frame(term = names(freq), occurrences = freq)
p <- ggplot(subset(df, freq > freq[[ord[20]]] - 1), aes(term, occurrences))
p <- p + geom_bar(stat = "identity")
p <- p + theme(axis.text.x = element_text(angle = 45, hjust = 1))
p

set.seed(42403)
wordcloud(names(freq), freq, min.freq = freq[[ord[20]]], colors = brewer.pal(8, 
    "Dark2"))

print("The total number of unique words is: ")
print(length(freq))
print("Number of words occurring 1 time is: ")
print(length(freq[freq <= 1]))
print("Number of words occurring 10 times or fewer is: ")
print(length(freq[freq <= 10]))
print("Number of words occurring 100 times or fewer is: ")
print(length(freq[freq <= 100]))
print("The number of words occurring more than 100 times is: ")
print(length(freq[freq > 100]))

hif <- freq[freq > freq[[ord[1]]]/20]
lowf <- freq[freq <= freq[[ord[1]]]/20]
lowf <- lowf[lowf > 100]

hist(hif)
hist(lowf)

################# Correlations

print("  ")
print("Words most highly correlated with the most common term are: ")
print(findAssocs(dtm, hifreq[1], 0.875))