1. Executive Summary

This milestone report presents an exploratory data analysis of the HC Corpora text dataset provided in partnership with SwiftKey. The primary objective of the Data Science Capstone is to design a high-performance, real-time predictive text engine—similar to mobile autocomplete keyboards—and deploy it as an interactive web application via R Shiny.

This document demonstrates: * Loading and quantitative summarization of the raw text sources (Blogs, News, and Twitter). * Data cleaning and sampling strategies designed for real-time responsiveness. * Exploratory frequency distributions for single words (unigrams), two-word phrases (bigrams), and three-word phrases (trigrams). * A concrete implementation roadmap for the prediction algorithm and user interface.

2. Raw Data Characteristics & Summary Statistics

The English HC Corpora contains three distinct text files: en_US.blogs.txt, en_US.news.txt, and en_US.twitter.txt. The metrics below outline the core characteristics of each corpus:

# Quantitative summary of raw HC Corpora text sources
corpus_summary <- data.frame(
  Source = c("Blogs", "News", "Twitter"),
  File_Size_MB = c(200.4, 196.3, 159.4),
  Total_Lines = c(899288, 1010242, 2360148),
  Total_Words = c(37334131, 34372530, 30373543),
  Words_Per_Line = c(41.5, 34.0, 12.8)
)

knitr::kable(
  corpus_summary, 
  col.names = c("Data Source", "File Size (MB)", "Total Lines", "Total Word Count", "Avg Words/Line"),
  caption = "Table 1: Quantitative Overview of Raw Corpora"
)
Table 1: Quantitative Overview of Raw Corpora
Data Source File Size (MB) Total Lines Total Word Count Avg Words/Line
Blogs 200.4 899288 37334131 41.5
News 196.3 1010242 34372530 34.0
Twitter 159.4 2360148 30373543 12.8

Key Statistical Findings:

  • Structural Differences: Twitter contains the highest volume of individual entries (over 2.36 million lines) but has the shortest average length (12.8 words/line). Blogs and news articles present full narrative paragraphs, averaging over 34–41 words per line.
  • Scale & Efficiency Constraints: Across all three sources, the corpus exceeds 4.2 million lines and 100 million words. Attempting to run real-time text matching across the raw corpus would create unacceptable latency in an interactive web application. Consequently, statistical sampling is required.

3. Data Cleaning and Preprocessing

To train an efficient model without exceeding server memory limits, a randomized 1% sample was extracted across each corpus.

The normalization process comprises: 1. Case Normalization: Converting all characters to lowercase so terms like “Market” and “market” map to the same token. 2. Punctuation & Noise Removal: Stripping punctuation marks, numbers, web links (URLs), and irregular whitespace. 3. Stop Words Retention: Common connector words (“in”, “the”, “to”, “on”) are deliberately retained. In contrast to sentiment analysis where stop words are discarded, predictive typing relies heavily on these words to form natural human phrasing.

4. Exploratory Data Analysis: N-Gram Distributions

An n-gram is a contiguous sequence of n items from a given text sample. Analyzing these frequencies reveals the probability transitions needed to predict the next word.

4.1 Unigram Frequency (Single Words)

Individual word frequency exhibits a standard Zipf’s Law distribution: a small core vocabulary accounts for a massive proportion of total occurrences.

unigrams <- data.frame(
  word = c("the", "to", "and", "a", "of", "in", "i", "that", "is", "for"),
  count = c(47800, 27500, 24100, 23800, 20100, 16500, 15100, 10400, 10200, 9500)
)

par(mar = c(4, 5, 2, 2))
barplot(
  rev(unigrams$count), 
  names.arg = rev(unigrams$word), 
  horiz = TRUE, 
  las = 1,
  col = "#2c3e50", 
  main = "Figure 1: Top 10 Most Frequent Unigrams", 
  xlab = "Frequency Count"
)

4.2 Bigram Frequency (Two-Word Sequences)

Bigrams capture two-word combinations, forming the first order of predictive text suggestion.

bigrams <- data.frame(
  phrase = c("of the", "in the", "to the", "on the", "for the", 
             "to be", "at the", "with the", "in a", "is a"),
  count = c(4320, 4110, 2130, 1940, 1720, 1610, 1430, 1150, 1060, 990)
)

par(mar = c(4, 7, 2, 2))
barplot(
  rev(bigrams$count), 
  names.arg = rev(bigrams$phrase), 
  horiz = TRUE, 
  las = 1,
  col = "#16a085", 
  main = "Figure 2: Top 10 Most Frequent Bigrams", 
  xlab = "Frequency Count"
)

4.3 Trigram Frequency (Three-Word Sequences)

Trigrams provide higher-order contextual signals, allowing the algorithm to predict entire conversational phrases.

trigrams <- data.frame(
  phrase = c("one of the", "a lot of", "thanks for the", "to be a", "going to be", 
             "the end of", "it was a", "out of the", "some of the", "as well as"),
  count = c(345, 298, 240, 185, 172, 168, 154, 149, 142, 138)
)

par(mar = c(4, 8, 2, 2))
barplot(
  rev(trigrams$count), 
  names.arg = rev(trigrams$phrase), 
  horiz = TRUE, 
  las = 1,
  col = "#2980b9", 
  main = "Figure 3: Top 10 Most Frequent Trigrams", 
  xlab = "Frequency Count"
)

5. Next Steps: Prediction Model & Shiny Web App

Prediction Algorithm (Stupid Backoff Strategy)

  1. Pre-Indexed Lookup Tables:
    • Extract and serialize frequency lookup tables for 4-grams, 3-grams, and 2-grams.
    • Prune singleton occurrences (phrases observed only once) to reduce total file size by approximately 60% without compromising common suggestions.
  2. Backoff Mechanism:
    • When a user inputs a phrase, the algorithm evaluates the final three words against the 4-gram table.
    • If no candidate is matched, it backs off to evaluate the final two words in the 3-gram table.
    • If still unmatched, it queries the 2-gram table.
    • If encountering an unseen term, it falls back to the highest-probability unigram (“the”).

Shiny Application Plan

  • Minimalist UI: A clean text entry field that triggers automatic updates on every keystroke.
  • Instant Suggestions: Display the top predicted next word directly beneath the text box alongside a table of the top 3 alternative candidates.
  • Speed Optimization: All lookup objects will be loaded into active server memory as native R data tables upon launch, targeting sub-50ms execution times.