FinalProject-DataScienceCapstone

The objective of the Captone project is to build an application to predict the next word(s) from a partial sententce entered by the user.In this project, a n-gram model combining with both Good-Turing backoff and Kneser-Ney algorithm was used for predicting the next word based on the previous partial sentence.

Raw data files:

• en_US.blogs.txt
• en_US.news.txt
• en_US.twitter.txt

Library used:

• plyr
• data.table
• tm
• openNLP
• reshape2

1. A subset of the original data was sampled from the three sources (blogs,twitter and news) and merged into one corpus

2. Data cleaning is done by conversion to lowercase, strip white space, and removing punctuation and numbers.

3. The corresponding n-grams (Quadgram, Trigram, Bigram and Unigram) were then created by using “RWeka” package

4. The term-count tables are extracted, sorted and presented according to the frequency in descending order.

Here is the link to the milestone report: http://rpubs.com/song_hw/324239

Back-off algorithm:In a backoff N-gram model, if the N-gram we need has zero counts, we approximate it by backing off to the (N-1)-gram. We continue backing off until we reach a history that has some counts. In this project:

1. Quadgram is first used (If the first three words of Quadgram are the last three words of the user provided sentence).

2. If no Quadgram is found, back off to Trigram (If the first two words of Trigram are the last two words of the sentence).

3. If no Trigram is found, back off to Bigram (If the first word of Bigram is the last word of the sentence)

4. If no Bigram is found, back off to the most common word with highest frequency is returned.

Kneser-Ney algorithm:

1. create a unigram model as continuation probability

2. Estimate of the medel on the the number of bigram types it completes

3. Counting the highest-order N-gram being interpolated