NLP Word Prediction Application

• This application takes a phrase and predicts the next word
• Users may choose from two different data sources for the predictions
  • HC Corpora data set
  • Microsoft Web Language Model
• Ultimately, this could be used as the foundation for a text prediction keyboard

• Create 1-, 2-, and 3-gram tokens from the blogs, news, and Twitter data sets
  • Profane words are removed using Google's list of bad words
  • Numbers and punctuation are removed
  • Stop words are included and may optionally be removed from the prediction results
• Calculate Good-Turing discount for each order token to allow for unseen words
• Calculate the probability for a word using Katz back-off method

There are two data sources which can be used:

1. HC Corpora data set
   • Obtained from scraping websites with a web crawler
   • Divided into 3 categories by type of website: blogs, news, and Twitter
   • Accuracy on a test set is around 12%
2. Microsoft Web Language Model
   • Utilizes a large corpus obtained from websites
   • Contains 1- to 5-gram tokens

1. Go to the Prediction Application on Shinyapps.io
2. Enter a phrase or sentence in the text box
3. Choose your data source for prediction
4. Choose whether to include common stop words in the results
   • Not available when using the Microsoft Web Language Model
5. Select the number of predictions to return
6. Click the Predict button and get your predictions!