Introduction

The goal of this project is to build a predictive model in the form of a Shiny App that suggests the next word based on previously entered text. As a first step, an exploratory analysis was conducted on the corpora provided by SwiftKey, three files in the form of blogs, tweets, and news articles. The objectives of this analysis were to understand the distribution of words and phrases, identify common language patterns, and develop a strategy for constructing an efficient prediction algorithm.

Summary of the Data

The first step was downloading the data and getting it into R. To reduce computation time, I randomly sampled 10% of each of the three files for exploratory analysis. This took the total length of all three datasets down from millions of entries to hundreds of thousands. You’ll see below that, as expected, the tweets contain fewer words on average than the blogs and articles.

## Package version: 4.4
## Unicode version: 14.0
## ICU version: 71.1
## Parallel computing: disabled
## See https://quanteda.io for tutorials and examples.
##    Corpus Entries Avg_Words Avg_Characters Avg_Characters_Per_Word
## 1   Blogs   89852      41.6          230.8                    4.70
## 2    News  100984      34.1          201.5                    5.02
## 3 Twitter  235642      12.9           68.6                    4.48

Exploratory Analysis

The next step is looking at word frequency and creating n-grams. To do this, I first combined and tokenized the data, before creating bigrams and trigrams and then computing frequencies. You will see from the plots below that the most common words, bigrams, and trigrams, tend to be what I would call “filler” words. This suggests that the n_grams will be very useful in connecting those filler words to each other when someone is typing to come up with a good suggestion. The more challenging and unpredictable part will be finding suggestions for the words in between those filler phrases that are often the action verbs, subjects, and descriptive words of the sentences.

## Unique Words and Distribution

The next step is looking at the dictionary of the combined dataset sample and assessing the cumulative distribution of words in that dataset. From the plot below, you will see that 50% of the dataset can be contained in within the 145 most frequent words. As we saw from the tables above, many of those words are the ones that make up those filler phrases. Looking further down the distribution, we see that to get to 90% of the dataset, we need over 8000 unique words. That is in line with my earlier observation that the filler phrases will be easy to finish, but the rest of the sentences will be much harder to predict, as they’ll contain a much wider array of unique words.

## Next steps and Planning to Model

Now that I have amassed the data and conducted exploratory analysis, I can begin thinking through how to create the prediction model. As laid out in the module, this will be a basic n-gram model, using the previous typed words to search for the most common n-grams starting with those words and then suggesting the words that finish the most common n-grams. The idea will be to create a back-up model that starts with the longest n-gram, looking through that table for a match and then the model will back up to shorter and shorter n-grams until it matches the previously typed set of words.

There are several next steps I can consider. First, the tradeoff between runtime and accuracy/reach will be important to consider. Part of that will be figuring out the longest n-gram to start with. If I start with an n-gram of 8, that may make the model highly accurate when it matches the previously typed 8 words to the training data. However, when there isn’t a match, the model will take much more time to back up through the shorter n-grams until it finds a match. Another consideration will be how many n-grams to fill up the tables with. It may make the model much more faster without sacrificing much accuracy to set a minimum number of times an n-gram appears in the dataset for it to be included in the tables the model searches through.

Another way to shorten runtime and improve accuracy is to identify the root of a word and apply that to its different forms. For instance, consider the bigram, “run away.” As far as a predictive model is concerned, that really is no different from the bigrams, “ran away,” and “running away.” Finding a way to combine those into just one bigram will shorten the lookup tables and improve accuracy of predictions, as “away” will be given the proper weight in the table regardless of the form of “run” that a user types. A similar strategy can be applied to typos.

Finally, in order to test the effectiveness of the model, I will split the data into a train/test set. This way, we will have data that the model didn’t “see” that we can use to test how we can expect our model to perform once it’s in action.