Coursera Data Science Capstone: Course Project

This presentation is created as part of the requirement for the Coursera Data Science Capstone Course. The goal of the project is to build a predictive text model combined with a shiny app that predict the next word as the user types a sentence similar to the way most smart phone keyboards are implemented today.

The Shiny App

• https://jpmoraga.shinyapps.io/Shiny_Capstone/

Github Repo

• https://github.com/jpmoraga/DataScienceCapstone

Milestone Report

• https://rpubs.com/jpmoraga/DSCapstone_MilestoneReport

Before building the word prediction algorithm, data are first processed and cleaned as steps below:

• A subset of the original data was sampled from the three sources (blogs,twitter and news) which is then merged into one
• Next, data cleaning is done by rewirting contraction, conversion to lowercase, strip white spaces, removing punctuation and emoticons
• The corresponding n-grams are then created (fourgram,Trigram and Bigram)
• Next, the term-count tables are extracted from the N-Grams and sorted according to the frequency in descending order

The prediction model for next word is based on the Katz Back-off algorithm. Explanation of the next word prediction flow is as below:

• Compressed data sets containing descending frequency sorted n-grams are first loaded.
• User input words are cleaned in the similar way as before prior to prediction of the next word.
• For prediction of the next word, Quadgram is first used (first three words of Quadgram are the last three words of the user provided sentence).
• If no Quadgram is found, back off to Trigram (first two words of Trigram are the last two words of the sentence).
• If no Trigram is found, back off to Bigram (first word of Bigram is the last word of the sentence)