Johns Hopkins University/Coursera

The Word Predictor App

Motivations:

• Around the world, people are spending an increasing amount of time on their mobile devices for email, social networking, banking and a whole range of other activities. But typing on mobile devices can be a serious pain. Accurately predicting the next word can alleviate some of the effort in typing.
• Natural Language Processing is key to unlocking insights from unstructured text data, from word prediction to aid with typing and the improve the accuracy of transcriptions, to topic modeling, sentiment analysis and summarizations to understand the meaning of conversations and text.

The objectives of the capstone project are to:

• Develop a predictive model of text starting with a large, unstructured database of the English language
• Deploy the model in a Shiny app that takes as input a phrase (multiple words) and it predicts the next word
• Create a presentation that pitches the algorithm and app

The corpora are collected from publicly available sources by a web crawler. The crawler checks for language, so as to mainly get texts consisting of the desired language. (The data was downloaded from the Coursera course site for the purposes of this project.)

The data consists of three text files containing blogs, news and tweets:

Prior to processing select profanities were removed from the data. Also, in an attempt to limit non-english words non-ASCII characters were also removed. (The frequency of non-english ascii based languages was tested and believed to be immaterial, so they were ignored.) For details on the data see the Capstone Checkpoint report.

The model is an n-gram model that matches the input last n-1 words in the phase against the n-grams and returns the nth word with the highest frequency. If an insufficient number of matches is found with 4-grams, the model moves to 3-grams, 2-grams and, finally, uni-grams. The quanteda r package was used to tokenize the data and create n-grams, removing punctuation, hash-tags, numbers, hyphens, symbols and urls. The text was also converted to all lower case. Finally to save space, n-grams that had a frequency of 1 were removed and only the 3 most frequent occurrences of root-grams ((n-1)-grams) where kept.

The table below shows some of the sample sizes chosen along with the resulting file size and prediction accuracy.

A sample size off 40% was used in the final app.
(Note: the sampled data was further split into train (80%) and text (20%) sets.)

Reference: Chapter 4: Speech and Language Processing. Daniel Jurafsky & James H. Martin.