1. Executive Summary

This milestone report outlines the initial exploratory data analysis of the English text corpora provided by SwiftKey (en_US.blogs.txt, en_US.news.txt, and en_US.twitter.txt). The objective is to evaluate data structures, extract basic descriptive statistics, clean the text, and examine word frequencies (unigrams, bigrams, and trigrams). Finally, we describe our roadmap for developing a predictive text algorithm and deploying an interactive Shiny web application.

2. Dataset Overview and Summary Statistics

Each file was analyzed to determine its uncompressed disk size, total line count, and total word count.

# Adjust file paths if located inside a subfolder (e.g., 'final/en_US/')
blogs_file   <- "en_US.blogs.txt"
news_file    <- "en_US.news.txt"
twitter_file <- "en_US.twitter.txt"

# File size in Megabytes
size_blogs   <- file.info(blogs_file)$size / (1024^2)
size_news    <- file.info(news_file)$size / (1024^2)
size_twitter <- file.info(twitter_file)$size / (1024^2)

# Read lines (binary connection handles special EOF characters cleanly)
con_blogs   <- file(blogs_file, "rb"); blogs <- readLines(con_blogs, encoding = "UTF-8", skipNul = TRUE); close(con_blogs)
con_news    <- file(news_file, "rb"); news <- readLines(con_news, encoding = "UTF-8", skipNul = TRUE); close(con_news)
con_twitter <- file(twitter_file, "rb"); twitter <- readLines(con_twitter, encoding = "UTF-8", skipNul = TRUE); close(con_twitter)

# Word counts using stringi
words_blogs   <- sum(stri_count_words(blogs))
words_news    <- sum(stri_count_words(news))
words_twitter <- sum(stri_count_words(twitter))

# Summary table
summary_df <- data.frame(
  Source = c("Blogs", "News", "Twitter"),
  File_Size_MB = round(c(size_blogs, size_news, size_twitter), 2),
  Total_Lines = c(length(blogs), length(news), length(twitter)),
  Total_Words = c(words_blogs, words_news, words_twitter),
  Avg_Words_Per_Line = round(c(words_blogs/length(blogs), words_news/length(news), words_twitter/length(twitter)), 1)
)

kable(summary_df, format.args = list(big.mark = ","), caption = "Table 1: Corpus Summary Statistics")
Table 1: Corpus Summary Statistics
Source File_Size_MB Total_Lines Total_Words Avg_Words_Per_Line
Blogs 200.42 899,288 37,581,907 41.8
News 196.28 1,010,242 34,858,293 34.5
Twitter 159.36 2,360,148 30,162,832 12.8

Blogs contain the highest average word count per line, whereas Twitter messages are significantly shorter due to character constraints.

3. Data Sampling & Text Cleaning

Because the combined corpus exceeds 4 million lines and 100 million words, a 1% pseudo-random sample was taken from each source to build the exploratory vocabulary efficiently.

set.seed(2026)
sample_size <- 0.01

sample_blogs   <- sample(blogs, length(blogs) * sample_size)
sample_news    <- sample(news, length(news) * sample_size)
sample_twitter <- sample(twitter, length(twitter) * sample_size)

raw_sample <- c(sample_blogs, sample_news, sample_twitter)
sample_df  <- tibble(text = raw_sample)

# Clean text: convert to lowercase, remove punctuation, numbers, and URLs
sample_df <- sample_df %>%
  mutate(text = gsub("http[^[:space:]]*", "", text)) %>%
  mutate(text = gsub("[^a-zA-Z' ]", "", text))

4. Exploratory Frequency Analysis (N-Grams)

Understanding word combinations (\(N\)-grams) is foundational for predictive modeling.

Most Frequent Single Words (Unigrams)

unigrams <- sample_df %>%
  unnest_tokens(word, text) %>%
  filter(!is.na(word) & word != "") %>%
  count(word, sort = TRUE)

ggplot(head(unigrams, 15), aes(x = reorder(word, n), y = n)) +
  geom_col(fill = "#2c3e50") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Words (Unigrams)",
       x = "Word", y = "Frequency Count") +
  theme_minimal(base_size = 12)

Stopwords (such as the, to, and, a) dominate individual token frequencies, which is expected for natural language sequencing.

Most Frequent Two-Word Phrases (Bigrams)

bigrams <- sample_df %>%
  unnest_tokens(bigram, text, token = "ngrams", n = 2) %>%
  filter(!is.na(bigram)) %>%
  count(bigram, sort = TRUE)

ggplot(head(bigrams, 15), aes(x = reorder(bigram, n), y = n)) +
  geom_col(fill = "#2980b9") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Bigrams",
       x = "Word Pair", y = "Frequency Count") +
  theme_minimal(base_size = 12)

Common transitional phrases like of the, in the, and to the appear with the highest recurring frequencies.

5. Strategy for Prediction Algorithm and Shiny App

  1. N-Gram Lookup & Katz’s Back-Off:
    • The core prediction engine will use frequency tables of trigrams, bigrams, and unigrams.
    • If the user types two preceding words, the model queries matching trigrams first. If no match is found, it backs off to bigrams, and finally to the highest-probability unigram.
  2. Efficiency & Pruning:
    • Single-occurrence \(N\)-grams (sparse tokens) will be pruned to reduce memory footprints and ensure real-time responsiveness (< 100 ms).
  3. Shiny Application Interface:
    • A clean UI featuring a single text input box.
    • Dynamic real-time prediction displaying the top 3 most likely next words as the user types.