1. Thorough text data cleaning using the quanteda package
2. Cross-validation using a subset of the combined dataset, with in-sample training and validation samples, and out-of-sample validation and final test samples
3. Implement a Katz Back-Off Model using Good-Turing Smoothing.
4. An R Shiny app that allows users to enter up to 9 words and provides the three most likely next words based on the model.

The Katz Back-Off Model estimates the probability of a word based on its context. This model predicts the next word in a sentence based on the words that came before it. If no examples of that combination of words are present in the dataset, then the model calculates probabilities based on shorter n-grams. This allows the model to account both for word combinations found within the data set and for those not found, which helps to avoid the over-fitting that would arise otherwise and improves the accuracy and robustness of the model. It applies this method iteratively until it finds an n-gram with sufficient frequency to estimate probability. The model also uses ‘smoothing’ to deal with uncertainties in the training data, ensuring a more robust probability estimate. Smoothing discount values allow for fine-tuning of the model.

Below is a screenshot of the app:

The user enters up to 9 words, and the app produces the three most likely next words in the context of the sentence. In order to make the app more useable and time-efficient, I’ve taken just a sample of the overall data set. This sacrifices some accuracy and robustness, but ensures that users do not have to spend minutes waiting for results.

You know as well as I that this has been a very challenging Capstone Project. I know my model isn’t perfect, but I’ve put significant effort into it and demonstrated the proficiency of R coding and statistical analysis I had hoped to gain in this Specialization. I appreciate your kind, forgiving, and empathetic grading!

Be well, and congratulations on getting through the courses!