2024-06-16

How my Predictive Model works

My code base primarily leverages the quanteda R package (reference: https://quanteda.io/).

  1. Take in a string of words (for example, don’t forget to buy bread and a case of)
  2. Take last 4 words of the above string of words (or less if less than 4 words are provided)
  3. Start with 4-gram (or trigram or bigram or unigram, if less than 4 words are provided)
  4. Check to see if result exists
  • If no results, then stupid back off to n-1 gram, n-2 gram until there’s only one word
    – If nothing if found, not even for one word, it just return a period (.)

Performance Prediction algorithm

The most important performance optimization was to build each N-Gram file for only the top 1/2 million features. Key line of code to implement:

  • nGramUSFreqSummary <- textstat_frequency(dfmGramUS, n=500000)

Performance before optimizations were:
- 1 minute to just load the data into the application.
- 5-10 seconds to look up the top n predicted words for a given n-gram.

Performance after optimizations were:
- 2 second delay to load application.
- Sub-second delay for prediction algorithm.

How the Product Works

  1. In Enter an n-gram (for example, ‘in the beginning’): enter an n-gram.
  2. In Maximum # of words to predict: enter the maximum number of next words to predict by rank (frequency).

The output (i.e., predicted words and how many words are predicted by rank (determined by frequency)) will automatically change based on any changes to the two steps above.