Summary

The goal of this project is an app that suggests the next word as someone types, like a phone keyboard. This report shows that the training data is loaded, summarizes what it contains, and describes the plan for the prediction model and app.

Main findings so far:

  • The data is three English text files (blogs, news, and Twitter) totaling about 556 MB. That is too large to use in full inside a responsive app, so the model will be built from a sample.
  • The three sources write very differently. Tweets are short, while blog and news lines are much longer. Mixing all three should help the app handle both casual and formal writing.
  • A small number of common words make up most of the text (details in Section 4). This means a compact model can still cover most of what people type.

(Click Code on any section to see the R code behind it.)

1. Loading the data

if (!all(file.exists(paths))) {
  message("Data files not found in ", getwd(), " -- downloading (about 550 MB).")
  zip <- "Coursera-SwiftKey.zip"
  if (!file.exists(zip)) {
    options(timeout = 3600)   # R's default 60-second timeout is too short for 550 MB
    tmp <- paste0(zip, ".part")
    download.file("https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip",
                  tmp, mode = "wb")
    file.rename(tmp, zip)     # only a complete download gets the real name
  }
  unzip(zip)
}

# Read in binary mode: the news file contains a stray control character that
# truncates text-mode reads on Windows.
read_safe <- function(p) {
  con <- file(p, "rb"); on.exit(close(con))
  readLines(con, encoding = "UTF-8", skipNul = TRUE)
}
txt <- lapply(paths, read_safe)
names(txt) <- names(files)

2. Basic summary of the three files

wpl <- lapply(txt, stri_count_words)   # words per line

summ <- data.frame(
  Source              = names(txt),
  `Size (MB)`         = round(file.size(paths) / 1024^2, 1),
  Lines               = sapply(txt, length),
  Words               = sapply(wpl, sum),
  `Avg words / line`  = round(sapply(wpl, mean), 1),
  `Longest line (chars)` = sapply(txt, function(x) max(stri_length(x))),
  check.names = FALSE, row.names = NULL
)
kable(summ, format.args = list(big.mark = ","),
      caption = "Size, line count, and word count of each source file")
Size, line count, and word count of each source file
Source Size (MB) Lines Words Avg words / line Longest line (chars)
blogs 200.4 899,288 37,546,250 41.8 40,833
news 196.3 1,010,242 34,762,395 34.4 11,384
twitter 159.4 2,360,148 30,093,413 12.8 140

Each file has a similar total number of words, but they are spread out very differently. Twitter has by far the most lines, and each line is short. Blogs and news have fewer lines, and each one is much longer.

3. How long is a typical line?

wpl_df <- bind_rows(lapply(names(wpl), function(s) data.frame(source = s, words = wpl[[s]])))

ggplot(wpl_df, aes(words, fill = source)) +
  geom_histogram(binwidth = 2, show.legend = FALSE) +
  facet_wrap(~ source, ncol = 1, scales = "free_y") +
  coord_cartesian(xlim = c(0, 120)) +
  scale_y_continuous(labels = comma) +
  labs(x = "Words per line", y = "Number of lines",
       title = "Distribution of line length by source") +
  theme_minimal()

Twitter lines bunch up at the short end because of the character limit. Blog and news lines spread across a much wider range, and a few blog lines are very long.

4. Most common words and phrases

To keep processing fast, the rest of this analysis uses a random 1% sample of each file. Before counting, the text is cleaned: it is converted to lowercase, and numbers, URLs, punctuation, and non-English symbols are removed.

set.seed(2026)
sample_rate <- 0.01
samp <- bind_rows(lapply(names(txt), function(s) {
  x <- txt[[s]]
  data.frame(source = s, text = sample(x, ceiling(length(x) * sample_rate)),
             stringsAsFactors = FALSE)
}))

samp$text <- samp$text |>
  stri_replace_all_fixed("\u2019", "'") |>                     # curly apostrophes -> straight
  iconv("UTF-8", "ASCII", sub = "") |>                         # drop emoji / non-ASCII
  stri_replace_all_regex("https?://\\S+|www\\.\\S+", " ") |>   # URLs
  stri_replace_all_regex("[0-9]+", " ")                        # numbers

unigrams <- samp |> unnest_tokens(term, text)
bigrams  <- samp |> unnest_tokens(term, text, token = "ngrams", n = 2) |> filter(!is.na(term))
trigrams <- samp |> unnest_tokens(term, text, token = "ngrams", n = 3) |> filter(!is.na(term))

plot_top <- function(df, title, n = 20) {
  df |> count(term, sort = TRUE) |> slice_head(n = n) |>
    ggplot(aes(reorder(term, n), n)) +
    geom_col(fill = "steelblue") + coord_flip() +
    scale_y_continuous(labels = comma) +
    labs(x = NULL, y = "Count in sample", title = title) +
    theme_minimal()
}
plot_top(unigrams, "Top 20 single words")

plot_top(bigrams, "Top 20 two-word phrases")

plot_top(trigrams, "Top 20 three-word phrases")

Common words like “the”, “to”, and “and” are at the top of every list. Text-mining projects often remove these, but they are kept here on purpose: they are exactly the words people type most often, so the app needs to predict them.

How many words do we need to know?

freq <- unigrams |> count(term, sort = TRUE) |>
  mutate(rank = row_number(), coverage = cumsum(n) / sum(n))
n50 <- which(freq$coverage >= 0.5)[1]
n90 <- which(freq$coverage >= 0.9)[1]

ggplot(freq, aes(rank, coverage)) +
  geom_line(color = "steelblue", linewidth = 1) +
  geom_hline(yintercept = c(0.5, 0.9), linetype = "dashed", color = "grey50") +
  scale_x_log10(labels = comma) + scale_y_continuous(labels = percent) +
  labs(x = "Number of distinct words (most common first, log scale)",
       y = "Share of all words in the text",
       title = "A small vocabulary covers most of the text") +
  theme_minimal()

The sample contains 52,125 distinct words. Even so, the most common 142 words account for half of everything written, and 7,014 words cover 90%. The rest is a long tail of rare words, typos, and names. This matters for the app: it can drop most rare words and stay small and fast while losing very little accuracy.

5. Plan for the prediction algorithm and app

How the prediction will work. The model will count how often words follow each other in the text: pairs, triples, and four-word sequences. When the user types “thanks for the”, the app looks up which words most often came after “thanks for the” in the training data. If that exact phrase was never seen, it shortens the context to “for the”, then “the”. The method is called backoff; the version planned here is “Stupid Backoff”, a simple scheme that is known to work well on large text.

Steps:

  1. Build word-sequence frequency tables (1 to 4 words long) from a larger sample of the data, around 10–20%.
  2. Drop sequences that appear only once. This shrinks the model a lot with little loss in accuracy, as Section 4 suggests.
  3. Hold out a portion of the data the model never sees, and measure how often the correct next word is in the top 3 suggestions. Also measure how fast each prediction runs.
  4. Tune the sample size and pruning so the model is both accurate and small enough to load quickly.

The Shiny app. A simple web page with one text box. As the user types, the top 3 predicted next words appear as buttons, and clicking one adds it to the text. The goal is a response that feels instant (well under a second) and a model small enough to run on the free Shiny hosting tier.

Open questions (feedback welcome):

  • Should profanity be filtered out of suggestions?
  • Should the app weight predictions differently depending on whether the user is writing casually (Twitter-style) or formally (news-style)?