Executive Summary

This report presents the exploratory data analysis for the Coursera Data Science Capstone project and outlines the strategy for building a predictive text application. Using the HC Corpora dataset provided by SwiftKey (comprising English Blogs, News, and Twitter feeds), we evaluate dataset volume, word distributions, and common multi-word phrases. Written for both technical and managerial stakeholders, this document demonstrates how raw unstructured text is transformed into a lightweight, fast next-word prediction engine deployed via R Shiny.

1. Automated Data Acquisition & Summary Metrics

To ensure full reproducibility inside a new project environment, the script below automatically checks for the dataset, downloads and extracts it if missing, and computes core summary statistics.

library(stringi)
library(knitr)
library(dplyr)

# Define expected paths
zip_file <- "Coursera-SwiftKey.zip"
zip_url  <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"

if (file.exists("en_US.blogs.txt")) {
  path_prefix <- ""
} else if (file.exists("final/en_US/en_US.blogs.txt")) {
  path_prefix <- "final/en_US/"
} else {
  if (!file.exists(zip_file)) {
    options(timeout = 1200)
    download.file(zip_url, destfile = zip_file, method = "curl", mode = "wb")
  }
  unzip(zip_file)
  path_prefix <- "final/en_US/"

  
  unzip(zip_file)
  path_prefix <- "final/en_US/"
}

file_blogs   <- paste0(path_prefix, "en_US.blogs.txt")
file_news    <- paste0(path_prefix, "en_US.news.txt")
file_twitter <- paste0(path_prefix, "en_US.twitter.txt")

# Read lines safely across platforms
blogs <- readLines(file_blogs, encoding = "UTF-8", skipNul = TRUE)

con_news <- file(file_news, open = "rb")
news <- readLines(con_news, encoding = "UTF-8", skipNul = TRUE)
close(con_news)

twitter <- readLines(file_twitter, encoding = "UTF-8", skipNul = TRUE)

# Compile word counts, line counts, and file sizes
word_blogs   <- stri_count_words(blogs)
word_news    <- stri_count_words(news)
word_twitter <- stri_count_words(twitter)

summary_table <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  File_Size_MB = c(
    round(file.info(file_blogs)$size / 1024^2, 1),
    round(file.info(file_news)$size / 1024^2, 1),
    round(file.info(file_twitter)$size / 1024^2, 1)
  ),
  Total_Lines = c(length(blogs), length(news), length(twitter)),
  Total_Words = c(sum(word_blogs), sum(word_news), sum(word_twitter)),
  Mean_Words_Per_Line = c(
    round(mean(word_blogs), 1),
    round(mean(word_news), 1),
    round(mean(word_twitter), 1)
  )
)

kable(summary_table, format.args = list(big.mark = ","), caption = "Table 1: Summary of Raw English Text Corpora")
Table 1: Summary of Raw English Text Corpora
Dataset File_Size_MB Total_Lines Total_Words Mean_Words_Per_Line
Blogs 200.4 899,288 37,546,806 41.8
News 196.3 1,010,242 34,762,658 34.4
Twitter 159.4 2,360,148 30,096,690 12.8

Distribution of Words per Line (Histogram)

Before tokenizing individual words, examining how many words appear per entry highlights the structural differences between long-form blogs, structured news articles, and short-form social posts.

library(ggplot2)

set.seed(123)
dist_df <- data.frame(
  Source = rep(c("Blogs", "News", "Twitter"), each = 5000),
  Words = c(
    sample(word_blogs, 5000),
    sample(word_news, 5000),
    sample(word_twitter, 5000)
  )
)

ggplot(dist_df %>% filter(Words <= 150), aes(x = Words, fill = Source)) +
  geom_histogram(bins = 30, alpha = 0.85, color = "white") +
  facet_wrap(~Source, scales = "free_y") +
  scale_fill_manual(values = c("#2c3e50", "#18bc9c", "#e74c3c")) +
  labs(
    title = "Distribution of Word Counts per Line Across Sources (Capped at 150 Words)",
    x = "Words per Line",
    y = "Number of Lines"
  ) +
  theme_minimal() +
  theme(legend.position = "none")

2. Sampling & Preprocessing Pipeline

Processing over 4.2 million lines (~570 MB) directly in memory is slow and unnecessary for discovering overarching language patterns. We extract a random 1% sample across all three sources, combine them, and standardize the text by:

  1. Converting all text to lowercase.
  2. Stripping URLs, numbers, punctuation, and special symbols.
  3. Retaining common English stopwords (e.g., the, to, and), because a real-world typing assistant must predict everyday connecting words just as accurately as content words.
library(quanteda)
library(quanteda.textstats)

set.seed(42)
sample_text <- c(
  sample(blogs, length(blogs) * 0.01),
  sample(news, length(news) * 0.01),
  sample(twitter, length(twitter) * 0.01)
)

# Free raw objects from RAM
rm(blogs, news, twitter, word_blogs, word_news, word_twitter, dist_df)
invisible(gc())

# Build cleaned token object
corp <- corpus(sample_text)
toks <- tokens(
  corp,
  remove_punct = TRUE,
  remove_numbers = TRUE,
  remove_symbols = TRUE,
  remove_url = TRUE
) %>% tokens_tolower()

3. N-Gram Frequency Distributions

An N-gram is a sequence of \(N\) consecutive words. Examining the most frequent single words (Unigrams), two-word pairs (Bigrams), and three-word sequences (Trigrams) shows the foundational building blocks of our prediction engine.

# Helper function to compute and plot top N-grams
plot_top_ngrams <- function(tokens_obj, n_val, title_text, bar_color) {
  dfm_obj <- tokens_ngrams(tokens_obj, n = n_val, concatenator = " ") %>% dfm()
  top_df  <- textstat_frequency(dfm_obj, n = 15)
  
  ggplot(top_df, aes(x = reorder(feature, frequency), y = frequency)) +
    geom_col(fill = bar_color) +
    coord_flip() +
    labs(title = title_text, x = NULL, y = "Frequency in 1% Sample") +
    theme_minimal()
}

plot_top_ngrams(toks, 1, "Top 15 Unigrams (Single Words)", "#2c3e50")

plot_top_ngrams(toks, 2, "Top 15 Bigrams (Two-Word Sequences)", "#18bc9c")

plot_top_ngrams(toks, 3, "Top 15 Trigrams (Three-Word Sequences)", "#e67e22")

4. Interesting Findings from Exploratory Analysis

5. Roadmap for Prediction Algorithm & Shiny Application

Algorithm Design: Stupid Backoff Model

To run smoothly within the strict memory and latency limits of a free web server (shinyapps.io), the final model will use an N-gram Stupid Backoff architecture:

  1. Compressed Lookup Tables: We will pre-compute frequency tables up to 4-grams, storing the first \(N-1\) words as the input context and the final word as the predicted output. Singletons (frequency = 1) will be pruned.
  2. Backoff Logic: When a user enters text, the algorithm extracts the last three words and searches the 4-gram table. If no match is found, it “backs off” to the last two words (3-gram table) applying a standard discount factor (\(\alpha = 0.4\)), then to the last single word (2-gram table), and finally defaults to the top unigrams (the, to, and).

Shiny App User Experience

The web interface will be designed for executive clarity and instant usability: * Live Text Input: A clean input box where users type a phrase and receive real-time predictions without needing to click a “Submit” button. * Top 3 Candidates: The interface will display the primary predicted word alongside two alternative recommendations as clickable buttons that append directly to the text box. * Sub-100ms Response Time: By indexing our N-gram lookup tables using R’s data.table package, queries will execute near-instantaneously.