Predicting Next Word Application - Capstone Final Project

Data Science Specialization

The Capstone dataset used includes twitter, news and blogs from HC Corpora. The data performing cleansing, sampling and sub-setting, before gather all data in R data frames. Applying some Text Mining (TM) and NLP techniques, is created some set of word combinations (N-grams). These are the main support to Katz Backoff algorithm predicts the next word. Some adaptations and heuristics were specially developed to enhance this Shiny application.

Just type a word, phrase or sentence. The app shows what the user has entered, followed by cleansed form. As the main result, until the top five (more probable) n-grams predictions are displayed in a list control. The user can review or swap your input data, and the app will turn back to present more hints to predict. Another tab offers a more extensive documentation.

Main steps to achieve next word(s) predictions:

1. Loading 4 data frames contained n-grams combinations with 4-words, 3-words, 2-words, and 1-word previously generated.
2. Reading user input (a word or sentence)
3. Cleansing of user input (lowering, profanities removing, tokenization of input words: the last four)
4. Call to prediction model function, basically, the Stupid backoff algorithm (a more simplified approach to Katz Backoff):
• search in the fourgram data frame, if found, shows top 5 most probable matches. Otherwise;
• search in the trigram data frame, by the same way above. Otherwise;
• search in bigram data frame, by the same way above.
• else, at last, if none matching, displays the most frequent words in the unigram data frame.

See 5 lines of “bigrams” and “trigrams” data frames which are loaded by Shiny App.