Executive Summary

This report summarizes our exploratory analysis of the US English text corpora (blogs, news articles, and Twitter posts) and outlines our validated architecture for a fast, lightweight next-word prediction application.

Key Findings & Readiness Milestones:

  • Data Ingested & Audited: All three datasets (~556 MB on disk; ~4.27 million lines total) have been loaded and verified.

  • Format & Length Differences: Blogs and news articles feature long-form, multi-sentence entries, whereas Twitter entries are short and frequent due to the 140-character limit.

  • High Dictionary Efficiency: Analysis of a 5% representative sample shows that just 140 unique words cover 50% of all word usage, and 7,921 unique words cover 90%.

  • Lightweight Model Validated: By pruning rare single-occurrence phrases and filtering offensive terms, we have already built a working 1-to-4-gram backoff lookup model occupying just 24.03 MB of RAM—well within web-hosting constraints.


1. Corpus Overview & Summary Statistics

The raw dataset comes from publicly crawled web text across three distinct communication styles. Note that the news file required binary-mode reading to prevent premature truncation from embedded special characters, and embedded null bytes were stripped from Twitter records.

Table 1: Summary Metrics Across US English Text Corpora
Source Disk Size (MB) Line Count Total Characters Mean Words / Line Median Words / Line Max Words / Line
Blogs 200.42 899,288 206,824,382 41.8 28 6,726
News 196.28 1,010,242 203,223,154 34.4 32 1,796
Twitter 159.36 2,360,148 162,096,241 12.8 12 47

Key Data Observations

  • Volume vs. Entry Count: All three files contain roughly comparable character volumes (162M to 207M characters), yet Twitter contains more than double the number of lines (2.36M) compared to Blogs (899k) and News (1.01M).

  • Line Independence: Because the source corpus splits documents into anonymized individual lines, each line is treated as an independent boundary during modeling so word sequences never bleed across unrelated entries.


2. Visual Exploration: Entry Length & Vocabulary Coverage

Word Count Distributions by Source

The histograms below compare entry lengths across sources (capped at 150 words on the horizontal axis to highlight core distributions). Blogs and news show right-tailed distributions stretching into long articles, while Twitter clusters tightly below 30 words.

N-Gram Frequencies & Dictionary Coverage

Using a 5% random sample (~213,000 lines), we normalized punctuation and smart quotes, converted text to lowercase, and replaced profanity with boundary markers so offensive phrases are never learned or suggested. We retained common stopwords (the, to, and) because they are essential for natural keyboard prediction.

Table 2: Top 10 Most Frequent Word Sequences (5% Sample)
Rank Top Unigram Unigram Count Top Bigram Bigram Count Top Trigram Trigram Count
1 the 238,106 of the 21,133 one of the 1,713
2 to 137,365 in the 20,503 a lot of 1,541
3 and 120,646 to the 10,578 thanks for the 1,227
4 a 119,734 for the 10,063 to be a 894
5 of 100,445 on the 9,620 going to be 829
6 i 83,475 to be 7,953 the end of 738
7 in 82,126 at the 6,983 it was a 736
8 for 55,293 and the 6,316 i want to 724
9 is 53,786 in a 5,958 out of the 722
10 that 52,025 with the 5,314 some of the 687

Why This Matters for Model Efficiency:

  • Everyday English follows a steep diminishing-returns curve: fewer than 8,000 words account for 90% of all word instances, while the remaining 10% consists of tens of thousands of rare terms, typos, and foreign words.

  • Pruning single-occurrence phrases (frequency == 1) eliminates noise and shrinks memory overhead without sacrificing everyday prediction quality.


3. Prediction Algorithm & Shiny App Roadmap

Our objective is to deploy a responsive, smartphone-style next-word prediction app on shinyapps.io.

Predictive Engine Design

  1. Multi-Level N-Gram Lookup (1- to 4-Grams): The model stores the top 3 most probable next words for observed 1-, 2-, and 3-word contexts in indexed data.table structures, keeping the entire model under 25 MB of RAM.

  2. Backoff Logic for Unseen Phrases: When a user types a phrase, the engine checks 4-gram matches first (using the last 3 words). If fewer than 3 suggestions are found, it automatically “backs off” to 3-grams, 2-grams, and finally high-frequency unigrams so a sensible prediction is always returned.

  3. Noise & Profanity Safeguards: Single-character artifacts (other than “a” and “i”) and offensive words are excluded prior to building lookup tables.

Planned Shiny App Experience & Validation

  • Interactive Text Input & Quick-Select Buttons: Users type into a clean text box and see the top 3 predicted next words rendered as clickable buttons that append directly to their sentence.

  • Benchmarking Before Launch: Prior to final deployment, we will evaluate the model against a held-out validation set to verify Top-1 and Top-3 prediction accuracy alongside millisecond-level response latency.