Capstone Project: Next Word Text Prediction ShinyApp

Background: This capstone project is part of the Johns Hopkins Data Science Specialization offered by Coursera.

Aim: Creating a ShinyApp that takes as input a phrase in a text box input and outputs a prediction of the next word prediction.

The presentation gives an overview of:

• How the provided data was preprocessed.
• How the final prediction model was build.
• The basic functionalities of the ShinyApp.

The training data stems from English news, blogs and twitter messages. They provide the basis for building a text corpus.

To analyze n-gram frequencies, the following preprocessing steps were performed:

• Remove punctuations from text corpus.
• Transform words to lower case.
• Strip text of additional whitespaces.
• Stopwords and numbers are intentionally included.

Steps for building a prediction model:

• Use preprocessed text corpus to calculate bigrams, trigrams and fourgrams.
• Consider all bigrams, where the first word occurs at least 10 times.
• Consider all trigrams and fourgrams, where the first 2(3) words occur at least 5 times.
• Use a simplified Katz back-off model for the next word prediction.
• Optimise algorithm by saving and loading data as R data formats.

I would like to thank the entire team from Johns Hopkins University and Coursera for offering a very interesting and inspiring specialization in Data Science. A special thanks goes to C.H. Lampert for introducing me to the Python Natural Language Toolkit.

Useful References for the Capstone:

• John Fry's notes on “Bigrams and Trigrams” with R
• Michael Coollin's notes on Language Models
• Andy Lyons' presentation on Top Five CSS Customizations
• Shiny Cheat Sheet for Interactive Web Apps