Next Word Prediction

Presentation for the Coursera Data Science Specialisation Capstone Project

Algorithm for predicting the next word:

NGram model with stupid back off method

deployed in a shiny web application

Go to the app

Inputs:

• Over 500MB in text files containing:
  - over 4 million lines of text.
  - over 76 million words.
  
• 1 million most frequent 2, 3, 4 and 5 grams from the
  Corpus of Contemporary American English.
  

Steps taken:

1. Clean the textfiles.
   
2. Extract NGrams.
   
3. Create frequency counts of NGrams.
   
4. Add NGrams from the Corpus of Contemporary American English.

Steps in algorithm

1. Clean the input string.
   
2. Count the number of words in the cleaned string to identify the highest possible NGram.
   
3. Use stupid backoff to find the best matching prediction Stupid backof starts with highest NGram, if frequency of NGram is less than 5, go back one NGram back off again if needed.
   
4. Calculate probability based on highest NGram and number of times a back off was needed.
   
5. Return the predicted words, sorted by probability, maximum number of words to return is defined by the input.

Results in the app

• Predicted words and probabilities are returned to the app with a maximum number of words defined by the input value.
  
• The model returns the predictions in less than a second, which is pretty fast. Needed time to calculate the results is shown in the app.
  
• Results are shown in 3 boxes showing the top 3 results, a bar chart, a table (including probabilities) and a Word Cloud.
  
• Highest possible NGram to return results is the Five-Gram.
  

Check out the app here.