Overview

The Word Prediction Mini App is a Shiny application that predicts the next word in a phrase or sentence, given a prompt.

This version of the app is “mini” because it utilizes the free tier of the Shinyapps.io hosting site, where data storage and memory is limited.

To use the app, navigate to the app page. Wait about about 10 seconds for the app to load in the background, then simply type a sentence or phrase into the box labeled “Type Phrase Here”. The app will instantly return up to three predictions for the next word.

hint: The get the most use out of the app, type a prompt that isn’t likely to end in a sentence conjunction such as “a”, “the”, or “and”. For example, “rice and” is a better prompt than “eat rice”.



How was the App Built? Part 1

The app was built using de-identified text extracts from three sources: Twitter, News Media, & Blogs. Construction of the app can be generalized into three stages: Data preparation, predictive model development, app construction.

Data preparation

1. Read in the raw text, convert to data tables, convert all text to lower case
2. Remove any lines of text with profanity, using a modified version of a published list
3. Store the minimally cleaned data in three files, one for each source.
4. Additional cleaning: remove phrases with numbers, non-apostrophe punctuation, social media tags, phrases with repetitive letters, such as “aaaaaa”

Initial Model Development Steps

1. Sample each data source: 50% Twitter, 70% Blogs, 90% News
2. For each sample, use the quanteda package to build a corpus, a formalized data format for further processing of text data.
3. Tokenize each corpus; for every line of text in each corpus, break the text down into individual words.
4. For each tokenized corpus, create N-grams: groups of words that appear sequentially. For this project six N-grams were built for each corpus.



How was the App Built? Part 2

What is an N-Gram?

An N-gram is a a data set consisting of consisting of the consecutive n-words that appear in a corpus. For example, a 3-gram is built by taking a corpus of text and decomposing it to consecutive three-words phrases. Imagine I have a line of text as follows:

## [1] "I am almost done with this data science course"

A 3-gram version of this text would result as:

## [1] "I am almost"         "am almost done"      "almost done with"   
## [4] "done with this"      "with this data"      "data science course"

A 5-gram version of this text would result as

## [1] "I am almost done with"         "am almost done with this"     
## [3] "almost done with this data"    "done with this data science"  
## [5] "with this data science course"

Larger “N” N-grams take up more memory. This is because there are relatively fewer unique two-word combinations in the English language compared to six-word combinations. The plot below shows that the number of unique word combinations in N-grams grows considerably after the two-gram, and then starts to taper at the four, five, and six-gram level. The fact that there are more unique four-gram combinations that six-gram may be a characteristic of having a large Twitter dataset; there aren’t as many long phases from which to compile six-grams.

How was the App Built? Part 3

Final Model Development Steps

5. For each corpus N-gram, break rows up into a stem and a prediction. The stem is the first N-1 parts of the phrases and the prediction is the Nth word.
6. For each corpus N-gram, find the most common prediction for each stem.
7. For each N-gram, combine the corpori. Each final N-gram therefore contains up to three predictions for each stem; one each from Twitter, News, and Blogs.

App Development

The app was developed by first programming a backoff text prediction model using all six N-grams, then modifying this programming so that it would run on the Shinyapp.io server under the free tier. Using the profvis function from the same-named package helped to identify the memory-consuming steps in the code. In the end, it was necessary to sacrifice accuracy for speed. An initial app was built using all six N-grams, but the final app, called a mini-app, used only the two and three-gram data. The result app is reasonably fast after initial load, but isn’t as accurate as a full model built with all six N-grams.

What is a Backoff Model?

The basic backoff model used in this app:

1. Starts by truncating a user-defined text prompt such that the final number of words of the truncated prompt match the number of words in stem data of the highest N-gram.
2. The prompt is compared against the stems in the highest N-gram.
3. If the prompt does not match any stem in the highest N gram, the first word of the prompt is truncated and a new prompt is generated.
4. This new prompt is compared to the stems in the next-order down N-gram. The model continues to “back off” down through the N-grams in order until it finds a matching stem or gets to the one-gram. If it reaches the one-gram, it simply returns the word “the”, which was the most common single word in the corpori.

For example, if the user entered prompt is “Mary had a little lamb whose fleece was white as”, and the highest order N-gram in the model is a six-gram, the model would truncate the prompt to “whose fleece was white as” and then see whether this phrase exists among the six-gram stems. If it exists, the model returns up to three predictions (one from each corpus). If it does not exist, the model truncates the prompt to “fleece was white as” and looks at the five-gram stems. If it continues to not find a match, the model works all the way down to the one-gram, where it returns “the”.