Data Science Capstone: Word Prediction

This data product takes in a word or a sentence and predicts the next word. The model is trained on 70% of the data and uses a stupid backoff model with ngrams ranging from 1-5. The application is deployed at this link https://chrishan.shinyapps.io/finalwordprediction/

Method

The algorithm uses a stupid backoff model. First the model starts with a 5-gram match, given the sentence is long enough. If there is a match, the probability of the word is calculated based on the 5-gram match. If there is not a match, it moves onto 4-gram, to 3-gram, and so on.

The shiny application consists of the following elements:

• input textbox
• checkbox to indicate whether to remove stopwords
• predict button
• data table that shows all the ngram matches and their probabilities

Using the benchmark provided here Benchmark, we observed how the model performs on a test set.

The 5-gram model provides the best overall top-1 precision with being able to predict the next word on the first try 13.45% of the time. The final deployed application uses the 5-gram model on the basis of this result.