For this final submission, we had to:

• Create a data product to show off the prediction algorithm. We created a Shiny app that accepts an n-gram and predicts the next word.
• Create a slide deck promoting the application using RStudio Presenter.

The corpus used for this predictive text algorithm was constructed from the three original files that include text from blogs, tweets, and news, taking a 10% sample from each file. This was accomplished through a processing of the data which included:

• Removing non-english characters
• Removing profanity
• Converting text to lowervase
• Removing special symbols (e.g. #, @, //)
• Removing punctuation
• Removing extra blank spaces

Once the data was cleaned, the corpus was generated. Then, the data was tokenized to generate three RData files that included n-gram frequencies for unigrams, bigrams, and trigrams.These files were what the model used!

The algorithm is designed to take an input, and then iterate the text from a trigram to a unigram, trying to find a match. Once it has found the longest, matching n-gram, it uses a Katz back-off, as shown below, designed to estimate the conditional probability of each word given its history in the n-gram (its maximum likelihood state of that n-gram according to our frequency files). The model backs off through progressively shorter models (n-1)-gram, trying to find the one with the most reliable information.

Back off is very inexpensive to calculate, which ensures out app runs fast without losing too much precision in its predictive power!