Executive Summary
This report summarizes our initial exploratory analysis of the HC
Corpora English text dataset (comprising Blogs, News articles, and
Twitter posts) and outlines our roadmap for building a lightweight, fast
next-word prediction Shiny application.
Key Takeaways for Stakeholders:
- Scale & Diversity: The raw dataset contains
over 4.2 million lines and 100+ million words across three distinct
writing styles—ranging from informal, abbreviated tweets to formal news
reporting.
- Efficiency Strategy: Because processing all 100+
million words in real time is impractical for a responsive web app, a 5%
random sample captures the core vocabulary and common word sequences
while keeping memory usage low.
- Next Steps: We will build an n-gram language model
with backoff smoothing to predict the most likely next word as a user
types, deployed via an intuitive Shiny interface.
1. Data Acquisition & Summary Statistics
The English language corpus was downloaded, extracted, and loaded
from three source files: en_US.blogs.txt, en_US.news.txt, and
en_US.twitter.txt.
Table 1: Summary Statistics of the US English Training
Corpus
| Blogs |
200.4 |
899,288 |
37,546,239 |
41.8 |
| News |
196.3 |
1,010,242 |
34,762,395 |
34.4 |
| Twitter |
159.4 |
2,360,148 |
30,093,410 |
12.8 |
Key Observations from Summary Table
- Twitter has the highest volume of entries (2.36 million lines), yet
each entry is short (averaging 12.8 words) due to character limits.
- Blogs contain the longest entries (averaging 41.8 words per post)
and the largest total word count (37.5 million words), providing rich
sentence structures for predictive modeling.
2. Exploratory Data Visualizations
To understand text length distributions and the most frequent words
and phrases (n-grams), we cleaned a representative random sample
(converting to lowercase and removing punctuation, numbers, and
profanity).
Distribution of Line Lengths Across Sources

Top Unigrams (Single Words) and Bigrams (Two-Word Phrases)
Analyzing word frequencies reveals how everyday English concentrates
heavily around a small core of connecting words and common phrases.

Interesting Findings
- High Vocabulary Concentration: Just a few hundred
unique words account for over 50% of all word occurrences, meaning we
can prune rare typos and obscure words to drastically reduce model size
without hurting prediction accuracy.
- Domain Differences Matter: Twitter introduces
conversational first-person pronouns (“I”, “you”) and contractions,
whereas News articles rely heavily on prepositional phrases (“of the”,
“in the”). Combining all three sources creates a balanced predictor
suitable for general typing.
3. Plan for Prediction Algorithm & Shiny App
Our goal is to deliver a fast, user-friendly text prediction tool
that runs smoothly within Shiny’s memory constraints:
- N-Gram Language Model (1-grams to 4-grams): We will
extract frequency tables for single words (unigrams), 2-word (bigrams),
3-word (trigrams), and 4-word (quadgrams) sequences from a cleaned 10%
training sample.
- Stupid Backoff Smoothing: When a user types a
phrase, the algorithm will first check for matching 4-grams. If no match
is found, it “backs off” to 3-grams, then 2-grams, and finally falls
back to the most probable single words—ensuring the app always returns a
prediction in milliseconds.
- Shiny Application Design:
- Clean Input Box: Users type a sentence into a text
field and immediately see the top 3 predicted next words as clickable
buttons or instant suggestions.
- Low Latency: Pre-computed, compressed lookup tables
(.rds format) will keep startup time under 2 seconds and memory usage
well below 100 MB.