Coursera Data Science Capstone Project

This presentation pitches a Shiny application for predicting the three most probable next words after the user types a partial input sentence.

This prototype application has been developed as part of the capstone project for the Johns Hopkins University and Coursera Data Science Specialization in collaboration with SwiftKey.

• Next word prediction is the task of suggesting the most probable word a user will type next. Current approaches are based on the empirical analysis of corpora (large text files) resulting in probability distributions over the different sequences that occur in the corpus. The resulting language models are then used for predicting the most likely next word.

• The English language corpus used in this case to build the 'n-gram' model was collected from publicly available sources - blogs, twitter and news by a web crawler.

• In order to reduce the size of the data and allow the prototype to run smoothly, about 10 % of the data was sampled, cleaned and tokenized into 6,5,4,3,2, and 1 word n-grams.

• After suitable data exploration, A 'Stupid Back-off' model was used to determine the three most probable next words based on the probabilities of occurrences of the relevant n-grams.

• The model was developed with 6, 5, 4, 3, 2 and 1 word 'ngrams' - sequences of words drawn from the sampled corpus that occur consecutively.

• The input sentence typed in the app is first cleaned of special characters,numbers, punctuations, swear words, etc. Then the last five words in the sentence (or fewer if the sentence is of smaller length ) are taken and matched with the corresponding 'n-order' n-gram list to find a match. If one is found the last word in the n-gram is returned as the predicted next word. If not the input string is further shortened and the next lower-order n-gram list is searched for a match.

• For example let us consider the input to be “All roads lead to…”. The model will then search the list of all 5-word n-grams and if a 5-word n-gram completing the sentence is found, the last word will be returned as next word predicted. In case no such n-gram is found the input will be shortened to “roads lead to…” and the model will search the list of 4-grams for a match. If one is found the last word is returned. If not the input is further shortened by a word and the next lower order n-gram list searched for a match. Finally if no match is returned up to the bigram level the model simply returns the top most probable unigram.

• The shiny app developed is hosted here. A screenshot of the hosted app is appended below
• In view of the large data files involved please allow the app about a minute to load onto your screen.

• All code files pertaining to the app are available at this github repository

- Thanks