Executive Summary

This report summarizes our initial exploratory analysis of the HC Corpora English text dataset (comprising Blogs, News articles, and Twitter posts) and outlines our roadmap for building a lightweight, fast next-word prediction Shiny application.

Key Takeaways for Stakeholders:

  • Scale & Diversity: The raw dataset contains over 4.2 million lines and 100+ million words across three distinct writing styles—ranging from informal, abbreviated tweets to formal news reporting.
  • Efficiency Strategy: Because processing all 100+ million words in real time is impractical for a responsive web app, a 5% random sample captures the core vocabulary and common word sequences while keeping memory usage low.
  • Next Steps: We will build an n-gram language model with backoff smoothing to predict the most likely next word as a user types, deployed via an intuitive Shiny interface.

1. Data Acquisition & Summary Statistics

The English language corpus was downloaded, extracted, and loaded from three source files: en_US.blogs.txt, en_US.news.txt, and en_US.twitter.txt.

Table 1: Summary Statistics of the US English Training Corpus
Data Source File Size (MB) Total Lines Total Words Avg. Words / Line
Blogs 200.4 899,288 37,546,239 41.8
News 196.3 1,010,242 34,762,395 34.4
Twitter 159.4 2,360,148 30,093,410 12.8

Key Observations from Summary Table

  • Twitter has the highest volume of entries (2.36 million lines), yet each entry is short (averaging 12.8 words) due to character limits.
  • Blogs contain the longest entries (averaging 41.8 words per post) and the largest total word count (37.5 million words), providing rich sentence structures for predictive modeling.

2. Exploratory Data Visualizations

To understand text length distributions and the most frequent words and phrases (n-grams), we cleaned a representative random sample (converting to lowercase and removing punctuation, numbers, and profanity).

Distribution of Line Lengths Across Sources

Top Unigrams (Single Words) and Bigrams (Two-Word Phrases)

Analyzing word frequencies reveals how everyday English concentrates heavily around a small core of connecting words and common phrases.

Interesting Findings

  • High Vocabulary Concentration: Just a few hundred unique words account for over 50% of all word occurrences, meaning we can prune rare typos and obscure words to drastically reduce model size without hurting prediction accuracy.
  • Domain Differences Matter: Twitter introduces conversational first-person pronouns (“I”, “you”) and contractions, whereas News articles rely heavily on prepositional phrases (“of the”, “in the”). Combining all three sources creates a balanced predictor suitable for general typing.

3. Plan for Prediction Algorithm & Shiny App

Our goal is to deliver a fast, user-friendly text prediction tool that runs smoothly within Shiny’s memory constraints:

  1. N-Gram Language Model (1-grams to 4-grams): We will extract frequency tables for single words (unigrams), 2-word (bigrams), 3-word (trigrams), and 4-word (quadgrams) sequences from a cleaned 10% training sample.
  2. Stupid Backoff Smoothing: When a user types a phrase, the algorithm will first check for matching 4-grams. If no match is found, it “backs off” to 3-grams, then 2-grams, and finally falls back to the most probable single words—ensuring the app always returns a prediction in milliseconds.
  3. Shiny Application Design:
    • Clean Input Box: Users type a sentence into a text field and immediately see the top 3 predicted next words as clickable buttons or instant suggestions.
    • Low Latency: Pre-computed, compressed lookup tables (.rds format) will keep startup time under 2 seconds and memory usage well below 100 MB.