Executive summary

The English text data have been downloaded, loaded, cleaned, sampled, and analyzed successfully. The three sources contain more than 4,269,642 lines and approximately 100,887,032 words. The main finding is that language is highly concentrated: a small number of common words cover much of everyday text, while many words and phrases occur rarely.

This supports a lightweight prediction approach that first uses the most recent three words and then falls back to shorter context when a phrase has not been seen. A working prototype has already been built and will form the core of the eventual Shiny application.

Data successfully loaded

The project uses English blogs, news articles, and Twitter messages. Processing was performed in chunks so the full files did not need to be held in memory at once.

Source File size (MB) Lines Approximate words
blogs.txt 200.4 899,288 37,179,893
news.txt 196.3 1,010,206 33,967,787
twitter.txt 159.4 2,360,148 29,739,352

For rapid development, a reproducible 0.1% sample was drawn from every source. After cleaning, this sample contained 100,325 word instances and 14,642 unique word types. Sampling keeps iteration fast while preserving representation from all three types of writing.

Major findings

Word frequencies have a long tail

A small number of words appear extremely often, while thousands appear only a few times. This is expected in natural language and means that storing every rare word and phrase would add substantial size with limited predictive value.

Dictionary coverage

Coverage of observed words Dictionary words required
50% 145
90% 5,469
95% 9,626
99% 13,639

Only 145 words cover half of all observed word instances, but 5,469 words are needed to reach 90%. This suggests keeping common words and phrases in the main model while treating very rare terms more selectively.

Common phrases

The sample contains 64,866 distinct two-word phrases and 86,722 distinct three-word phrases. The most common combinations are ordinary structures such as “of the”, “in the”, and “a lot of”.

Longer phrases are much less frequent than individual words. Consequently, the application cannot rely only on exact phrase matches.

Prediction algorithm plan

The prediction engine will:

  1. Clean and tokenize the user’s text using the same rules as the training data.
  2. Look for the last three words in the stored phrase table.
  3. Suggest the most likely next words when that context has been observed.
  4. Fall back to the last two words, then one word, and finally overall word frequency when a phrase is new.
  5. Exclude profanity and return only a small set of useful suggestions.

The current prototype uses this approach. It occupies approximately 2.59 MB in memory, has a compressed size of 94 KB, and produces a prediction in about 50 ms. On held-out text, the correct next word appears among the first three suggestions in 19.6% of tested cases. This is a baseline for improvement, not a final performance claim.

Shiny application plan

The Shiny app will have one text-entry box and display the three best next-word suggestions immediately. Selecting a suggestion will append it to the text. The interface will stay deliberately simple so attention remains on prediction quality and response time.

Before release, model settings will be tuned on untouched test data. The main success measures will be top-three accuracy, response time, and model memory. The next iteration will also compare results across blogs, news, and Twitter to ensure the model works reasonably well across writing styles.

Feedback requested

Feedback would be most valuable on two decisions: