Milestone Report: Exploratory Analysis for a Predictive Text Keyboard

Overview

SwiftKey builds a smart keyboard that predicts the next word as you type — for example, after “I went to the” it might suggest gym, store, or restaurant. This report is the first milestone toward building a similar predictive text model and Shiny app. It shows that the training data has been downloaded and loaded successfully, summarizes its basic properties, highlights a few interesting patterns, and lays out the plan for the prediction algorithm and app.

About the Data

The data comes from the HC Corpora, a collection of text scraped from public blogs, news sites, and Twitter by a web crawler, filtered to include mostly a single language per file. This report uses the English (en_US) subset: blogs, news, and Twitter files. Duplicate entries have been removed and roughly half of each original entry deleted before release, so the text cannot be traced back to a specific original post. A small number of lines may still contain non-English or foreign-language text despite the language filter — this is expected and not treated as an error.

Getting and Loading the Data

The code below downloads and unzips the source data (skipped if already present) and reads the three English files into R.

data_dir <- "data/en_US"
zip_url  <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
zip_path <- "data/Coursera-SwiftKey.zip"
files    <- c("en_US.blogs.txt", "en_US.news.txt", "en_US.twitter.txt")

if (!all(file.exists(file.path(data_dir, files)))) {
  dir.create(data_dir, recursive = TRUE, showWarnings = FALSE)
  if (!file.exists(zip_path)) download.file(zip_url, zip_path, mode = "wb")
  unzip(zip_path, files = file.path("final/en_US", files),
        exdir = "data", junkpaths = TRUE)
}

read_corpus <- function(path) {
  con <- file(path, "rb")
  on.exit(close(con))
  readLines(con, encoding = "UTF-8", skipNul = TRUE, warn = FALSE)
}

blogs   <- read_corpus(file.path(data_dir, "en_US.blogs.txt"))
news    <- read_corpus(file.path(data_dir, "en_US.news.txt"))
twitter <- read_corpus(file.path(data_dir, "en_US.twitter.txt"))

Basic Summary of the Three Files

File size, line count, word count, and words-per-line statistics for the full files (no sampling needed for these counts):

Basic summary statistics for the three English corpora
File	Size (MB)	Lines	Words	Mean Words/Line	Max Words/Line
Blogs	200.4	899,288	37,334,131	41.5	6630
News	196.3	1,010,242	34,372,530	34.0	1792
Twitter	159.4	2,360,148	30,373,583	12.9	47

Sampling for Deeper Analysis

The full corpora together contain several million lines, which is more than is needed to explore word patterns. For the word- and phrase-frequency analysis below, a random 2% sample of lines is drawn from each file — large enough to be representative, small enough to process quickly.

set.seed(2026)
sample_frac <- 0.02
sample_lines <- function(x) x[sample(seq_along(x), size = floor(length(x) * sample_frac))]

blogs_s   <- sample_lines(blogs)
news_s    <- sample_lines(news)
twitter_s <- sample_lines(twitter)

Distribution of Line Lengths

Words per line, by source (99th percentile trimmed for readability)

Twitter is tightly clustered at short line lengths (its historical character limit), while blogs and news skew toward longer, more variable lines.

Most Frequent Words

Common English “stop words” (the, and, to, …) and profanity have been removed so the chart highlights more meaningful vocabulary.

Top 20 words across the sampled corpora

Most Frequent Word Pairs (Bigrams)

Looking at two-word sequences (keeping stop words this time, since phrases like “of the” or “going to” are exactly the kind of pattern a next-word predictor relies on):

Top 15 two-word sequences across the sampled corpora

Interesting Findings

A small vocabulary covers most of the language. Just 1,617 unique words account for 50% of all word instances in the sample, and 18,451 words cover 90% — out of 70,882 unique words seen overall. This is good news for a prediction model: it doesn’t need to know every word perfectly to be useful most of the time.
Twitter writing is short and punchy. Its average line length (12.9 words) is well under half that of blogs and news, consistent with the platform’s historical character limit.
Blogs contain the longest outliers, with a maximum line length of 6,630 words — likely long-form posts with minimal paragraph breaks.

Plan for the Prediction Algorithm and Shiny App

The next-word predictor will be built from n-gram frequency tables — counts of how often each one-word, two-word, and three-word sequence occurs in the full training corpus (not just today’s sample). To predict the next word after a typed phrase, the model looks up the most common word that has followed similar phrases before. If the exact phrase was never seen, the model “backs off” to a shorter phrase (a technique called Stupid Backoff) so it can still offer a reasonable guess instead of giving up. Rare sequences will be trimmed from the tables to keep the model small and fast enough to run inside a web app, and the same profanity list used above will be applied to filter out inappropriate suggestions. Before finalizing the model, its predictions will be checked against text it hasn’t seen, to make sure it generalizes rather than just memorizing the training data.

The final product will be a Shiny web app: the user types a phrase into a text box, clicks Submit, and the app displays the top three predicted next words — the same experience as a phone keyboard’s suggestion bar.

Summary

The three English data sources have been downloaded, loaded, and explored. Word and phrase counts show clear, learnable structure in the data — exactly the kind of pattern an n-gram model can exploit — and line-length patterns already hint at real differences between blogs, news, and Twitter text. The next steps are building the n-gram model with a backoff strategy and wrapping it in a Shiny app for interactive next-word prediction.