Next Word Prediction

by Chan Chee-Foong on 1 Oct 2016
for Data Science Capstone Project of the Data Science Specialisation by Johns Hopkins University

Summary

Objective of the project is to build a predictive text model like those used by SwifyKey.
Data are downloaded from HC Corpora and the Exploratory Data Analysis can be found here.

1. Being an English-only 4-grams model means that the application will only use up to 3 words in the input to predict the next word.
   
2. Together with Natural Processing Language techniques, the prediction scores for the next word are computed using 2 methods: Stupid Backoff and Knesar-Ney Interpolation. The most likely next word has the highest score. Each method has pros and cons in terms of speed and accuracy.
3. Due to system limitation, only samples of the corpus are used in building the model and validation testing.
4. Records with non-English words, bad words, repeated words, etc are excluded from the sampling.
5. Only proper English words are used in building the prediction model.

The application developed using Shinyapp can be found here.

Instructions

1. Click on Dashboard to begin and test the application. Follow the instructions in the red box.
2. You should end the message with a blank space for the program to give a next word prediction.
3. Click on the Model Results to see details for the N-grams prediction scores.
4. Introduction, Notes, Appendix and References include information and considerations made when building the prediction model.