Jonathan Mallia
27th May, 2017
Coursera Data Science Capstone Project by John Hopkins University
The Word Predictor application predicts the next word based on the last words in the sentence.
Firstly, the last 4 words in the sentence are observed and determine if there is a suitable prediction for that phrase. If unsuccessful the last 3 words are then considered and fed to the prediction function to find a match. This process repeats until there is a match.
The application shows the top 3 words suggested from the predictions with the respective probablilty.
The biggest challenge in this project is the amount of data. The combination of words/phrases in a language are extensive so we need big corpuses (collections of text). This was a limitation due to the computer resources (especially mine with only 4Gb of RAM).
Therefore 6% of original corpus dataset (blogs, news and tweets) was used for building the prediction engine using R's tm package. This low sample percentage can have an impact on the accuracy of the prediction.
Katz model with Simple Good Turing smoothing was used for this project. Katz lets probabilities compete - the probabiliy of an N-gram vs a N-1-gram, adjusted by a factor. If there is an unobserved N-gram, we fall back to N-1 grams.
Simple Good-Turing (SGT) smoothing was implemented in order to get better probability estimates and most importantly get a reasonable probability estimate for unseen words and N-grams.
The application performs very fast due to the way the phrase cleaning and predictions models were written.