1. Summary

The goal of this project is to build a smartphone-keyboard app that predicts the next word a person is about to type — similar to the auto-suggest feature on most phone keyboards. To do that, we first need to understand the raw material the app will learn from: a large collection of real-world English text pulled from blogs, news articles, and Twitter.

This report is an early checkpoint. It shows that:

  • The data has been downloaded and loads correctly.
  • We understand its basic shape and size (how much text, how long the entries are).
  • We’ve identified some interesting patterns (most common words and phrases).
  • We have a concrete, non-technical plan for turning this into a working prediction algorithm and app.

No modeling has happened yet — this is purely “getting to know the data” before we build anything.

2. The Data

The dataset comes from the HC Corpora collection distributed for this project, and contains three English-language text sources:

Source What it is
Blogs Long-form personal and topical blog posts
News Snippets from news articles
Twitter Short, informal tweets 
files <- c(
  Blogs   = "data/en_US.blogs.txt",
  News    = "data/en_US.news.txt",
  Twitter = "data/en_US.twitter.txt"
)

read_file <- function(path) {
  con <- file(path, "r")
  lines <- readLines(con, encoding = "UTF-8", skipNul = TRUE)
  close(con)
  lines
}

raw <- lapply(files, read_file)

2.1 Basic file statistics

The table below confirms the files loaded successfully and summarizes their size. Word counts, line counts, and average sentence/entry length are the kind of basic sanity check every data project starts with.

summary_tbl <- data.frame(
  Source = names(raw),
  `File size (MB)` = sapply(files, function(f) round(file.info(f)$size / 1024^2, 1)),
  `Lines` = sapply(raw, length),
  `Words` = sapply(raw, function(x) comma(sum(stri_count_words(x)))),
  check.names = FALSE
)
summary_tbl$Words <- as.character(summary_tbl$Words)
words_num <- sapply(raw, function(x) sum(stri_count_words(x)))
summary_tbl$`Avg. words / line` <- round(words_num / summary_tbl$Lines, 1)
summary_tbl$Lines <- comma(summary_tbl$Lines)

kable(summary_tbl, row.names = FALSE,
      caption = "Table 1: Size and volume of each data source")
Table 1: Size and volume of each data source
Source File size (MB) Lines Words Avg. words / line
Blogs 200.4 899,288 37,546,806 41.8
News 196.3 1,010,206 34,761,151 34.4
Twitter 159.4 2,360,148 30,096,690 12.8

A few things jump out immediately:

  • Twitter has by far the most entries (over 2.3 million tweets) but the shortest entries — about 13 words per tweet on average, consistent with the platform’s short-form nature.
  • Blogs have the longest entries — roughly 42 words per post on average — since blog posts are free-form writing.
  • Altogether the three files add up to over 100 million words, which is a substantial amount of real-world text to learn language patterns from.

2.2 Line-length distribution

set.seed(1234)
len_df <- bind_rows(lapply(names(raw), function(nm) {
  data.frame(source = nm, words = stri_count_words(raw[[nm]]))
}))

ggplot(len_df %>% filter(words <= 100), aes(x = words, fill = source)) +
  geom_histogram(binwidth = 2, alpha = 0.75, position = "identity") +
  facet_wrap(~source, scales = "free_y") +
  labs(title = "Figure 1: Distribution of entry length (word count)",
       x = "Words per line/entry", y = "Number of entries") +
  theme_minimal() +
  theme(legend.position = "none")

This confirms the pattern from Table 1 visually: Twitter entries cluster tightly under ~30 words (largely due to the historical 140-character limit), while blogs and news have a much wider spread, including some very long entries.

3. Sampling and Cleaning

The full dataset is too large (over 100 million words) to explore comfortably on a laptop, so — as is standard practice — we work with a random sample rather than the whole file. This report uses a random sample of 15,000 entries from each source (45,000 total) as a representative subset. The final prediction model will likely use a larger sample, but the patterns found here are expected to hold.

Before counting words, we cleaned the sampled text by:

  • Converting everything to lowercase (so “The” and “the” count as the same word)
  • Removing web links, @-mentions, and #hashtags
  • Removing numbers and punctuation
  • Collapsing extra whitespace
set.seed(1234)
n_sample <- 15000

sample_lines <- function(lines, n) lines[sample(seq_along(lines), min(n, length(lines)))]
samp <- lapply(raw, sample_lines, n = n_sample)

clean_text <- function(x) {
  x <- stri_trans_tolower(x)
  x <- stri_replace_all_regex(x, "http\\S+|www\\.\\S+", " ")
  x <- stri_replace_all_regex(x, "@\\w+", " ")
  x <- stri_replace_all_regex(x, "#\\w+", " ")
  x <- stri_replace_all_regex(x, "[^a-z' ]", " ")
  x <- stri_replace_all_regex(x, "\\s+", " ")
  stri_trim_both(x)
}

df <- bind_rows(lapply(names(samp), function(nm) {
  data.frame(source = nm, line_id = seq_along(samp[[nm]]),
             text = clean_text(samp[[nm]]), stringsAsFactors = FALSE)
}))

4. Most Frequent Words, Word Pairs, and Word Triples

To predict “the next word,” the algorithm ultimately needs to know which words and short phrases commonly follow one another. As a first look, we counted:

  • Unigrams — single words
  • Bigrams — two-word sequences (e.g., “of the”)
  • Trigrams — three-word sequences (e.g., “one of the”)
unigrams <- df %>% unnest_tokens(word, text, token = "words") %>% filter(word != "")
bigrams  <- df %>% unnest_tokens(bigram, text, token = "ngrams", n = 2) %>% filter(!is.na(bigram))
trigrams <- df %>% unnest_tokens(trigram, text, token = "ngrams", n = 3) %>% filter(!is.na(trigram))

top_unigrams <- unigrams %>% count(word, sort = TRUE) %>% head(15)
top_bigrams  <- bigrams  %>% count(bigram, sort = TRUE) %>% head(15)
top_trigrams <- trigrams %>% count(trigram, sort = TRUE) %>% head(15)
plot_top <- function(data, label_col, title) {
  ggplot(data, aes(x = reorder(.data[[label_col]], n), y = n)) +
    geom_col(fill = "#2c7fb8") +
    coord_flip() +
    labs(title = title, x = NULL, y = "Count") +
    theme_minimal(base_size = 11)
}

plot_top(top_unigrams, "word", "Figure 2: Top 15 most common single words")

Unsurprisingly, the most common words are short “function” words like the, to, and, and a — words that appear in virtually every sentence. This is expected and matches everyday English usage.

plot_top(top_bigrams, "bigram", "Figure 3: Top 15 most common two-word phrases")

plot_top(top_trigrams, "trigram", "Figure 4: Top 15 most common three-word phrases")

Common phrases like “of the,” “a lot of,” and “one of the” are exactly the kind of predictable pattern a next-word predictor can exploit — if someone has typed “a lot,” the model should strongly favor suggesting “of” next.

5. How Much Vocabulary Do We Actually Need?

An important practical question for the app: how large does the “dictionary” need to be? A small dictionary keeps the app fast and lightweight; too small, and it won’t cover enough of what people actually type.

uni_counts <- unigrams %>% count(word, sort = TRUE) %>%
  mutate(cum_pct = cumsum(n) / sum(n), rank = row_number())

n50 <- which(uni_counts$cum_pct >= 0.5)[1]
n90 <- which(uni_counts$cum_pct >= 0.9)[1]
total_unique <- nrow(uni_counts)
ggplot(uni_counts %>% filter(rank <= 10000), aes(x = rank, y = cum_pct)) +
  geom_line(color = "#2c7fb8", linewidth = 1) +
  geom_hline(yintercept = c(0.5, 0.9), linetype = "dashed", color = "grey40") +
  scale_y_continuous(labels = percent) +
  labs(title = "Figure 5: How many unique words cover most of the text?",
       x = "Number of unique words (ranked by frequency)",
       y = "Cumulative % of all word instances covered") +
  theme_minimal()

Key finding: out of 56,814 unique words seen in our sample, just 138 of the most common words cover half of everything people write, and about 6,961 words cover 90% of all word instances. This is a very encouraging sign — it means a relatively compact vocabulary can capture the vast majority of real-world usage, which keeps the eventual app fast and small.

6. Interesting Findings So Far

  • Twitter text is short and informal; blogs and news are longer and more structured. The model will likely need to handle both styles.
  • A small set of common words dominates — the top 138 words already account for half of all word usage.
  • Common short phrases are highly predictable (e.g., “of the,” “going to be”), which is good news: even a fairly simple model should do well on everyday phrasing.
  • Raw text contains noise — URLs, @-mentions, hashtags, and inconsistent punctuation — all of which need to be cleaned before modeling, as demonstrated in Section 3.

7. Plan for the Prediction Algorithm

In plain terms, here’s the approach:

  1. Build a “phrase book” from the data. Using the cleaned text, we’ll count how often every word, pair of words, and triple of words appears — similar to what we did above, but using a larger sample (and eventually the whole dataset).
  2. Predict using “what usually comes next.” When a user types a word or two, the app looks up what word most often follows that exact phrase in our phrase book (e.g., after “thanks for the,” the top candidate is “thanks for the support” or similar).
  3. Handle phrases we’ve never seen before. No dataset covers every possible phrase. When the app encounters an unfamiliar combination, it will fall back to a shorter phrase (three words → two words → one word) — a standard, well-tested technique called “backoff.” This keeps the app from getting stuck when it hits something new.
  4. Balance size, speed, and accuracy. Section 5’s finding — that a fairly small vocabulary covers most real usage — means we can likely keep the phrase book compact enough to run quickly on a phone-sized app, without sacrificing much accuracy.

8. Plan for the Shiny App

The final deliverable will be a simple, interactive web app (built with R’s Shiny framework) that mimics a phone keyboard’s predictive text:

  • Input box where the user types a word or partial sentence.
  • Live suggestions — as the user types, the app shows the top 3 predicted next words, updating in real time.
  • Tap-to-complete — clicking a suggested word adds it to the sentence, just like a phone keyboard.
  • A simple, uncluttered interface so it’s clear and usable by a non-technical audience, with a short “how it works” note for context.

9. Next Steps

  • Scale the n-gram analysis from our 45,000-entry sample up to a much larger portion (or all) of the data.
  • Build and evaluate the backoff prediction model for accuracy and speed.
  • Build and test the Shiny interface, then optimize for response time.
  • Get feedback on this plan before moving into full model-building.

This report was generated from the raw en_US.blogs.txt, en_US.news.txt, and en_US.twitter.txt files provided for the Data Science Capstone project.