Word Prediction

In desining this machine learning model, we faced 3 key performance challenges:

• 1) how to read/analyze a large data corpus with limited time and processing power,
• 2) how to maximize the accuracy of the predictions, and
• 3) how to optimize performance for hosting on a public web server.

It would take significant computing resources to extract 100% of all n-grams from every corpus. However, using regression and statistical inference, I can aproximate the minimum number of random sampled texts required to obtain 95% of all n-grams. This allows us to achieve nearly complete analysis, with only a small fraction of the work.

The algorithm uses an n-gram step-down model to find the best prediction. If it can't find a 4-gram match, then it will look for 3-gram, 2-gram and finally 1-gram. When several matches are found, it will select the one with the highest occurrence frequency.

Once we collect n-gram samples from the 3 corpuses, we can't combine them or it will create sampling errors. Instead, we compile a set of random selections from all 3 corpuses, and test each against all 3 n-gram sets. Based on this, we create a probabilistic model for deciding which of the 3 answers will be chosen for any given test.

3) Optimize Performance

We run the algorithm for every unique n-gram across all 3 corpuses. This will serve as a pre-compiled cache that can be saved as a text file and loaded at runtime. Using cached results provides significantly faster search time, since the heavy lifting was done in advance.