Overview of the model algorithm

• This presentation briefly explains an NLP model that predicts the next word to followed by a phrase.
• This presentation will:

1. Explain the core algorithm of the model.
2. Summarize the performance of the prediction algorith.
3. Explain how to use the Shiny application that runs the model.

• The model introduced in this presentation will take last two words of a given phrase to predict the next (third) word using Katz’s back off model algorthm.
• According to wikipedia: https://en.wikipedia.org/wiki/Katz%27s_back-off_model

Katz’s back off model is a generative n-gram language model that estimates the conditional probability of a word given its history in the n-gram. It accomplishes this estimation by “backing-off” to models with smaller histories under certain conditions. By doing so, the model with the most reliable information about a given history is used to provide the better results.

Details of the model algorithm

• N-gram model predicts \(x_i\) based on \(x_{i-(n-1)},...,x_{i-1}\). Maximum likelihood estimate assigns probability on each word that exist in corpus. The probability is the number of n-gram events divided by the number of total n-gram events.
• In the model, the independence assumptions (Markov assumption) are made so that each word depends only on the last \(n-i\) words. In probability terms, this is \(P(x_{i}\mid x_{i-(n-1)},...,x_{i-1})\). Simply the model count up the number of particular word or combination of words and divide by the total number of occurances.
• However this probability on each word is higher than the actual probability because no probability mass is assigned to unobserved n-grams. Therefore we will use discounting technique to “steal” some probability assigned to the observed n-grams to distribute to unobserved n-grams.
• For more details on the construction of the model, please refer to: http://rpubs.com/salus0324/484129

Performance of the model

The performance of the model was measure in three criterias: size, speed, and accuracy.

• Size: 579 KB The model was able to run in shinyapps.io. Hence the size is quite compact
• Speed: 6 seconds The model gives next word prediction within 6 seconds. This is not lightening fast, however reasonable considering the corpus it’s going through.
• Accuracy: 78% This is a quite accurate model compared to its prototype version (stupid back off model). The full explanation on the accuracy measurements can be found at

1. Katz’s Back-off Model: http://rpubs.com/salus0324/484129
2. Stupid Back-off Model: http://rpubs.com/salus0324/484133

Shiny application

• The shiny app with the prediction model can be found in https://salus0324.shinyapps.io/N-gramPrediction/
• Screen shot of the Shiny App, the instruction is embbed in the app.