For this project, the course provided three datasets (news, twitter, and blogs) in text format. I joined the three datasets and extracted a sample with 20% of the whole data, this sample was then divided into train and test sets. The train set was used to find bi-grams, three-grams, four-grams and five-grams. Here we can see the most frequent bi-grams and three-grams.

In order to make the model faster and reduce memory consumption some decisions were taken during the development of the model. I will list the final desitions.

1. The words that appear once and are present in the English dictionary were considered unknown words, these were replaced by “UNK” and their probability was computed the same way that the rest of the words.
2. From the remaining words I removed all the words with a frequency lower than 220.
3. Some n-grams were also deleted in the next way:
• bigram with a frequency less or equal to 60.
• three-gram with a frequency less or equal to 18

Four-gram and five gram were also reduced in the process, however, they were not considered at all in the final model.

I used the Katz Backk-off approach with good turing as smoothing technique. First, I created a function to perform Good Turing, it redistributes the counts of the n-grams. After calling the function we have a table with n-gram count, the new estimated probability, and a variable d or amount of discounting. I did this for bi-gram, three-grams, four-grams and five-grams.

In the next step a function was created to predict the next word given a phrase, this function takes the phrase and the n-grams previously created and assigns a probability to every word present in the corpus, this is the probability of being the next word given its history, using the Katz back-off approach. In the shiny app application, the model will check for the history of the phrase in the three-gram, when it is not present, it will then check the bi-gram.