1. Thorough text data cleaning using the quanteda package
2. Cross-validation using a subset of the combined dataset, with in-sample training and validation samples, and out-of-sample validation and final test samples
3. Implement a Katz Back-Off Model using Good-Turing Smoothing.
4. An R Shiny app that allows users to enter in two words as well as the smoothing discount values, and shows the top 3 most likely next word along with their probabilities.

The Katz Back-Off Model estimates the probability of a word based on its context. This model predicts the third word of a trigram based on the first two words. The model calculates the probabilties of each possible trigram, and when probabilities are very low or zero it ‘backs off’ by calculating probabilities based on shorter n-grams. This allows the model to account both for word combinations found within the data set and for those not found, which helps to avoid the severe over-fitting that would arise otherwise and improves the accuracy and robustness of the model. It applies this method iteratively until it finds an n-gram with sufficient frequency to estimate probability. The model also uses ‘smoothing’ to deal with uncertainties in the training data, ensuring a more robust probability estimate. Smoothing discount values allow for fine-tuning of the model.

Below is a screenshot of the app:

The user enters the first and second word of the bigram, as well as the bigram and trigram discount values. The app produces the three most likely next words, along with their probabilities. By adjusting the discount values, the user can explore the model’s functionality and tweak its settings.

You know as well as I that this has been a very challenging Capstone Project. I know my model isn’t perfect, but I’ve put significant effort into it and demonstrated the proficiency of R coding and statistical analysis I had hoped to gain in this Specialization. I appreciate your kind, forgiving, and empathetic grading!

Be well, and congratulations on getting through the courses!