Data science Capstone Project: Word Prediction

Presentation of Capstone project in partnership with:

• John Hopkins Bloomberg School of Public Health - Pr Brian Caffo, Pr Roger Peng, Pr Jeff Leek and Coursera
• Swiftkey Corporation provides the filesets
• RStudio Corporation provides the hosting and development tool platforms

From Swiftkey files Twitters, News, Blogs in the English Create a data product to predict the next word. Tasks:

• Explore the data
• Train the data Natural Language Process
• Build a shiny apps with the model

• Train data with Markov Ngram: tokenize and weight word occurences - https://www.youtube.com/watch?v=o-CvoOkVrnY
• Additional filtering for scalability: 7 millions+ texts leads to performance problems to product with limited resources. Discounted Kneser-Ney smoothing criteria - http://mkoerner.de/media/bachelor-thesis.pdf helps in filtering using criterias on word variabilities, e.g. prior words 3,4 are fixed probability of word 5. The dataset is reduced to 100,000 lines
• Online Backoff mechanism implemented to find the match first with 5-gram, 4-gram until Uni-gram - https://www.youtube.com/watch?v=t-TZ0YrrIDA

• In the sidebar enter your text, initialization will be indicated by the word “words” in the main panel in a reasonable time.
• Prediction of the highest ranked 5 words will display below your text.
• In the main panel, a maximum of 30 highest probable words will displayed in a cloud plot.
• Overall performance should be lesser than .5 second for each reaction after initialization.
• Access to the product using: https://yuhrvfr.shinyapps.io/wordpredict.

• The training of dataset is key, the algorithm can be extended for more sophisticated filtering of data
• Self learning, words and texts unknown can be stored and evaluated to become potential new entries for the training dataset
• Specialization in text prediction mechanism based on the type of texts being analyzed