Next word predictor
Pitch slides for John Hopkins Data Science Specialisation Capstone Project
2024-04-22
Introduction
This pitch presentation is part of the Capstone Project in the John Hopkins Data Science Specialisation course. The ‘pitch’ here is to promote an R Shiny application that can predict the next word in a sentence, in the same way that SwiftKey functions.
The challenge in this project was to learn and apply Natural Language Processing to a vague problem statement, with little instruction, mimicking a common experience as a data scientist.
The data provided from this project was sourced from news articles, blog posts and Twitter posts. All together the data consisted of 4,269,678 lines of text, 102,090,204 words and 870MB. Most words appeared only once, so that 17,641 unique words covered 90% of word occurrences. All available data were used, split into test and train datasets.
Links
Training Data and Approach
A key feature of this project was balancing size, speed and accuracy. A compromise was found using targets, tidytext and data.table packages to build ngram models for 1-6 ngrams.
Processing the data and creating the ngram models involved:
- removing profanity,
- keeping stop words,
- removing special characters and numbers and then
- tokenising the data into 1-6 ngrams.
In each ngram, the probability of the next word given the history text was calculated using a stupid Katz backoff approach. Next words with frequencies less than three were removed for efficiency and only the top 20 words per history text was retained.
Using the App
The R Shiny app searches the history text in a combined ngram model for the last 5,4,3,2,1 words of sentence, retrieving predicted words and probabilities. These are sorted on their probability and the top five are returned. If there are less than five (e.g. the history text is not found), then it returns the top 5 unigrams.
Features and next steps
This app has a number of useful features:
Predictions are presented as buttons, which allow the user to add the predicted word to the sentance and predict new words. Continually clicking on predicted words in this way allows the creation of ‘hallucination’ text.
The ngram model is stored as a data.table object with indexed search text, meaning that it is relatively very quick.
The app also includes the full milestone report, providing useful context about the training data and its attributes.
Ultimately, the prediction performance could be improved. Both with more data and more sophisticated prediction algorithms and deep learning models. This application serves as a proof of concept designed to secure further funding in this pitch scenario.
