I used the following code to read in the data and, using the quanteda package, turn each data set into a corpus:

##load libraries
library(tidyr)
library(stringr)
library(dplyr)
library(ggplot2)
library(quanteda)
library(quanteda.textstats)
library(quanteda.textplots)
library(text2vec)
library(tidytext)
library(readtext)
library(tibble)
library(knitr)

## Define zip file path
# Specify the path to the zip file
zip_path <- "~/R/Coursera/Capstone/Coursera-SwiftKey.zip"
# Specify the directory where you want to extract the contents
extracted_dir <- "~/R/Coursera/Capstone"
# Unzip the contents of the zip file
unzip(zip_path, exdir = extracted_dir)

# Read in the three data sets
USblogs <- readLines("~/R/Coursera/Capstone/final/en_US/en_US.blogs.txt")
USnews <- readLines("~/R/Coursera/Capstone/final/en_US/en_US.news.txt")
UStwitter <- readLines("~/R/Coursera/Capstone/final/en_US/en_US.twitter.txt")

## Use quanteda to turn each data set into a corpus
corp_twitter <- corpus(UStwitter)
corp_news <- corpus(USnews)
corp_blogs <- corpus(USblogs)

The basic data table below summarizes the characteristics of each corpus or data set:

Number of Lines Number of Words
US Twitter 2360148 36647566
US Blogs 899288 42776489
US News Articles 77259 3064980

Because these data sets are so large, I am taking a sample of each data set in order to conduct further exploratory analysis and eventually modeling. The following code samples 30,000 total lines, sampling proportionally from each data set corpus (that is, in accordance with their respective proportion of the total sum of lines). The code combines these three samples into a single corpus, cleans that corpus by removing punctuation, numbers, and symbols, and finally creates a dfm from the result.

##sample 30,000 lines proportionally from the data sets
set.seed(123)
total <- length(corp_twitter)+length(corp_blogs)+length(corp_news)
twitter_sample <- round(length(corp_twitter)/total*10000, 0)
news_sample <- round(length(corp_news)/total*10000, 0)
blogs_sample <- round(length(corp_blogs)/total*10000, 0)
corpus_news_sample <- corpus_sample(corp_news, size = news_sample)
docnames(corpus_news_sample) <- (1:news_sample)
corpus_blogs_sample <- corpus_sample(corp_blogs, size = blogs_sample)
docnames(corpus_blogs_sample)<- ((news_sample+1):(news_sample+blogs_sample))
corpus_twitter_sample <- corpus_sample(corp_twitter, size = twitter_sample)
docnames(corpus_twitter_sample) <- ((news_sample+blogs_sample+1):(news_sample+blogs_sample+twitter_sample))
##combine into single corpus
combined_corpus <- corpus(c(corpus_news_sample, 
                            corpus_blogs_sample, 
                            corpus_twitter_sample))
## remove punctuation, numbers, and symbols
corpus_clean <- combined_corpus %>%
     tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE)

## create dfm
dfm <- dfm(corpus_clean)

These steps allow for further exploration of the data.The chart below shows the frequency of the top 25 most frequently used words in the combined and cleaned sample corpus:

Unsurprisingly, the most frequently used words are articles (“the” and “a”), propositions (“to”, “of”, “in”), conjunctions (“and”), and pronouns (“i”, “you”). These words are also the most prominent in a wordcloud of the 100 most commonly used words:

This is basic but still useful information, and we can next look at n-gram frequencies to explore combinations of words. The following charts show the frequency of the top 25 2-, 3-, and 4-grams, respectively:

Finally, we can do a bit more technical exploration by looking at collocations through the textstats_collocations() function in the quanteda package. This reveals combinations of words that tend to occur together more often than would be expected by chance, and the lambda value reflects the strength of the association between words. As an example, the following chart shows the lambda values of two-word collocations:

This image is not very useful for actually seeing which two-word collocations have the strongest association, but the shape of the curve is somewhat revealing for exploratory analysis. The curve represents a trade-off between strength of association and frequency of the collocation, with high-lambda values having extremely strong association because they rarely appear in the sampled data set. Because of this, we can view the point at which this curve starts to rise rapidly as a lambda value that strikes a helpful balance between capturing meaningful collocations and filtering out noise. In this case, that optimal lambda value might be in the 1-2 range. This may come in handy later on during more complex modeling steps.

Next Steps for Modeling

My plans to create a prediction algorithm and Shiny App are as follows:

  1. Sampling: Because the data is so large and modelling is computationally taxing, I will create an even smaller sub-sample, maintaining proportional representation from each of the original three datasets.

  2. Cross-Validation: In order to mitigate the risks of a small sample, I will use a tiered cross-validation system. I will have in-sample training and validation sets which I will use to train and test the model, respectively, during early and model-engineering stages. In order to avoid overfitting the model to relatively small samples, I will also have an out-of-sample validation set which will help me identify overfitting design flaws. Finally, I will have an out-of-sample test set which I use only once, as the final test of the finished model.

  3. Model Design: I will use a Katz Back-Off model with Good-Turing Smoothing in order to model 3-gram word combinations. This will allow me to account for 3-word combinations that are both observed and unobserved in the data set, using 2-grams to help identify unobserved 3-grams. Given an input word, the model will produce the probabilities that each word in the data set is the following word. I may use lambda values from collocations, as shown above, to help evaluate and/or fine-tune this model.

  4. Shiny App Functionality: My app will allow the user to input a single word, along with trigram and bigram discount factors that influence the model outputs. Based on these three input factors, the app will display the three most likely next words, along with associated probability scores.