Next Word Prediction

This is a pitch for a Shiny application that make use of the Natural Language Processing framework and predict the next word given an user input phrase.

It was constructed as the final project of the Coursera Data Science Specialization.

The data used for training the algorithm was downloaded from here. The great amount of data was cleaned and sampled to construct the quadgrams used for the application, which then were classiffied and aggregated into five features:

• Base input word 3
• Base input word 2
• Base input word 1
• Prediction word
• Frequency of each unique base - prediction word combination

More details about the dataset can be found in this Rpubs presentation.

For predicting, a modification of the back off model was implemented, the algorithm workflow is like this:

1. The user input is cleaned with the same function that the original data was, and the three rightmost words are selected.

2. This words are searched in the app database, and if there is a match the prediction word is returned.

3. If no exact match is founded, the app database is filtered, when possible, by each one of the three input words and the prediction word with the greatest quadgram frequency is returned.