Data Science Capstone: SwiftKey Predictor Using Back-Off Good-Turing

An application that predicting the next word.

The main goal is to build a shiny application that consisting of a custom Natural Language Processing (NLP) model to predict the most probable next word to be typed, based upon analysis of a large corpus of raw text.

The tasks are:

• Understanding the problem
• Data acquisition and cleaning
• Exploratory analysis
• Statistical modeling
• Predictive modeling

This project is started with by analyzing a large corpus of text documents which is prepared from 3 raw text files, sourced from blogs, Twitter, and news, that is approximately 560MB in total.

2% of data set was sampled, cleaned, analyzed, tokenized, and n-gramified using a custom process into unigram, bigrams and trigrams.

A hashing algorithm was used to translate contextual n-grams into large integers for the prediction matrix. The probabilities of each n-gram were calculated based upon their frequency.

A prediction algorithm was developed based upon the Back-Off Good-Turing model, where the probability of a word is estimated based upon the context of preceding words.