This report summarizes our exploratory analysis of the US English text corpora (blogs, news articles, and Twitter posts) and outlines our validated architecture for a fast, lightweight next-word prediction application.
Key Findings & Readiness Milestones:
Data Ingested & Audited: All three datasets (~556 MB on disk; ~4.27 million lines total) have been loaded and verified.
Format & Length Differences: Blogs and news articles feature long-form, multi-sentence entries, whereas Twitter entries are short and frequent due to the 140-character limit.
High Dictionary Efficiency: Analysis of a 5% representative sample shows that just 140 unique words cover 50% of all word usage, and 7,921 unique words cover 90%.
Lightweight Model Validated: By pruning rare single-occurrence phrases and filtering offensive terms, we have already built a working 1-to-4-gram backoff lookup model occupying just 24.03 MB of RAM—well within web-hosting constraints.
The raw dataset comes from publicly crawled web text across three distinct communication styles. Note that the news file required binary-mode reading to prevent premature truncation from embedded special characters, and embedded null bytes were stripped from Twitter records.
| Source | Disk Size (MB) | Line Count | Total Characters | Mean Words / Line | Median Words / Line | Max Words / Line |
|---|---|---|---|---|---|---|
| Blogs | 200.42 | 899,288 | 206,824,382 | 41.8 | 28 | 6,726 |
| News | 196.28 | 1,010,242 | 203,223,154 | 34.4 | 32 | 1,796 |
| 159.36 | 2,360,148 | 162,096,241 | 12.8 | 12 | 47 |
Volume vs. Entry Count: All three files contain roughly comparable character volumes (162M to 207M characters), yet Twitter contains more than double the number of lines (2.36M) compared to Blogs (899k) and News (1.01M).
Line Independence: Because the source corpus splits documents into anonymized individual lines, each line is treated as an independent boundary during modeling so word sequences never bleed across unrelated entries.
The histograms below compare entry lengths across sources (capped at 150 words on the horizontal axis to highlight core distributions). Blogs and news show right-tailed distributions stretching into long articles, while Twitter clusters tightly below 30 words.
Using a 5% random sample (~213,000 lines), we normalized punctuation and smart quotes, converted text to lowercase, and replaced profanity with boundary markers so offensive phrases are never learned or suggested. We retained common stopwords (the, to, and) because they are essential for natural keyboard prediction.
| Rank | Top Unigram | Unigram Count | Top Bigram | Bigram Count | Top Trigram | Trigram Count |
|---|---|---|---|---|---|---|
| 1 | the | 238,106 | of the | 21,133 | one of the | 1,713 |
| 2 | to | 137,365 | in the | 20,503 | a lot of | 1,541 |
| 3 | and | 120,646 | to the | 10,578 | thanks for the | 1,227 |
| 4 | a | 119,734 | for the | 10,063 | to be a | 894 |
| 5 | of | 100,445 | on the | 9,620 | going to be | 829 |
| 6 | i | 83,475 | to be | 7,953 | the end of | 738 |
| 7 | in | 82,126 | at the | 6,983 | it was a | 736 |
| 8 | for | 55,293 | and the | 6,316 | i want to | 724 |
| 9 | is | 53,786 | in a | 5,958 | out of the | 722 |
| 10 | that | 52,025 | with the | 5,314 | some of the | 687 |
Why This Matters for Model Efficiency:
Everyday English follows a steep diminishing-returns curve: fewer than 8,000 words account for 90% of all word instances, while the remaining 10% consists of tens of thousands of rare terms, typos, and foreign words.
Pruning single-occurrence phrases (frequency == 1)
eliminates noise and shrinks memory overhead without sacrificing
everyday prediction quality.
Our objective is to deploy a responsive, smartphone-style next-word prediction app on shinyapps.io.
Multi-Level N-Gram Lookup (1- to 4-Grams): The
model stores the top 3 most probable next words for observed 1-, 2-, and
3-word contexts in indexed data.table structures, keeping
the entire model under 25 MB of RAM.
Backoff Logic for Unseen Phrases: When a user types a phrase, the engine checks 4-gram matches first (using the last 3 words). If fewer than 3 suggestions are found, it automatically “backs off” to 3-grams, 2-grams, and finally high-frequency unigrams so a sensible prediction is always returned.
Noise & Profanity Safeguards: Single-character artifacts (other than “a” and “i”) and offensive words are excluded prior to building lookup tables.
Interactive Text Input & Quick-Select Buttons: Users type into a clean text box and see the top 3 predicted next words rendered as clickable buttons that append directly to their sentence.
Benchmarking Before Launch: Prior to final deployment, we will evaluate the model against a held-out validation set to verify Top-1 and Top-3 prediction accuracy alongside millisecond-level response latency.