Summary

This report is an early checkpoint on the way to building a text prediction app (the kind that suggests the next word as you type, similar to the predictive keyboard on a smartphone). The goal here is not the finished app — it’s to show three things:

  1. I can download, load, and work with the raw text data.
  2. I understand the basic shape and size of the data (how much there is, how it’s structured).
  3. I have a clear, practical plan for turning this data into a working word-prediction model and a simple app.

Everything below uses three text files — collected from blogs, news articles, and Twitter posts — all in US English, provided as the source data for this project.

1. Loading the Data

# Data is downloaded once and cached locally — subsequent knits reuse the local copy
url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
zip_path <- "Coursera-SwiftKey.zip"

if (!file.exists("final/en_US/en_US.blogs.txt")) {
  if (!file.exists(zip_path)) {
    download.file(url, destfile = zip_path, mode = "wb")
  }
  unzip(zip_path)
}
blogs_path   <- "final/en_US/en_US.blogs.txt"
news_path    <- "final/en_US/en_US.news.txt"
twitter_path <- "final/en_US/en_US.twitter.txt"

# Safety check with a clear error message if paths still aren't found
stopifnot(
  "blogs file not found - check your working directory" = file.exists(blogs_path),
  "news file not found - check your working directory" = file.exists(news_path),
  "twitter file not found - check your working directory" = file.exists(twitter_path)
)

con <- file(blogs_path, "r"); blogs <- readLines(con, skipNul = TRUE); close(con)
con <- file(news_path, "r", encoding = "UTF-8"); news <- readLines(con, skipNul = TRUE); close(con)
con <- file(twitter_path, "r"); twitter <- readLines(con, skipNul = TRUE); close(con)

The three files are read in as plain text, one line per entry (one blog post, one news snippet, or one tweet per line).

2. Basic Summary Statistics

The first thing worth checking, before any modeling, is simply: how big is this data, and how is it distributed across the three sources?

file_size_mb <- function(path) round(file.info(path)$size / 1024^2, 1)

word_count <- function(x) sum(stri_count_words(x))

summary_df <- data.frame(
  Source        = c("Blogs", "News", "Twitter"),
  `File Size (MB)` = c(file_size_mb(blogs_path), file_size_mb(news_path), file_size_mb(twitter_path)),
  `Lines`         = c(length(blogs), length(news), length(twitter)),
  `Total Words`   = c(word_count(blogs), word_count(news), word_count(twitter)),
  `Avg Words / Line` = round(c(word_count(blogs)/length(blogs),
                                word_count(news)/length(news),
                                word_count(twitter)/length(twitter)), 1),
  `Longest Line (chars)` = c(max(nchar(blogs)), max(nchar(news)), max(nchar(twitter))),
  check.names = FALSE
)

kable(summary_df, caption = "Table 1: Basic statistics for the three raw text sources")
Table 1: Basic statistics for the three raw text sources
Source File Size (MB) Lines Total Words Avg Words / Line Longest Line (chars)
Blogs 200.4 899288 37546806 41.8 40833
News 196.3 1010206 34761151 34.4 11384
Twitter 159.4 2360148 30096690 12.8 144

What this tells us, in plain terms:

  • Twitter has by far the most lines, but each line is short (tweets are capped in length), so its average words-per-line is the lowest.
  • Blogs have the fewest, longest lines — closer to normal written paragraphs.
  • News sits in between — structured, edited sentences, but longer than tweets.

This matters for the app: it means the model will need to learn from very different “styles” of writing — casual short bursts (Twitter), polished long-form writing (blogs), and formal reporting (news).

3. Sampling the Data

The full files are large (hundreds of MB combined), which is too slow to work with directly while exploring. So for this report, a random 1% sample of each file is used — enough to see reliable patterns without the processing time.

set.seed(1234)
sample_pct <- 0.01

sample_blogs   <- sample(blogs,   length(blogs)   * sample_pct)
sample_news    <- sample(news,    length(news)    * sample_pct)
sample_twitter <- sample(twitter, length(twitter) * sample_pct)

sample_all <- c(sample_blogs, sample_news, sample_twitter)

4. Word Frequency — What Are People Actually Saying?

After basic cleaning (lowercasing, stripping punctuation and numbers), the most common words were counted across the combined sample.

clean_text <- function(x) {
  x <- tolower(x)
  x <- gsub("[^a-z' ]", " ", x)
  x <- gsub("\\s+", " ", x)
  trimws(x)
}

tokens <- unlist(stri_split_boundaries(clean_text(sample_all), type = "word"))
tokens <- tokens[grepl("^[a-z']+$", tokens) & nchar(tokens) > 0]

word_freq <- as.data.frame(table(tokens), stringsAsFactors = FALSE)
names(word_freq) <- c("word", "count")
word_freq <- word_freq[order(-word_freq$count), ]

kable(head(word_freq, 10), row.names = FALSE,
      caption = "Table 2: 10 most frequent words in the sample")
Table 2: 10 most frequent words in the sample
word count
the 47676
to 27748
and 24073
a 24015
of 20191
i 17314
in 16568
for 11047
is 10808
that 10641 
top20 <- head(word_freq, 20)

ggplot(top20, aes(x = reorder(word, count), y = count)) +
  geom_col(fill = "#4285F4") +
  coord_flip() +
  labs(title = "Top 20 Most Frequent Words",
       x = NULL, y = "Frequency") +
  theme_minimal(base_size = 12)

Finding: As expected, the most frequent words are short, common “function” words (like the, and, to, of). This is normal for any natural-language dataset — these words carry grammatical structure rather than meaning, and they will be very easy for a prediction model to get right. The interesting prediction challenge lies in the less frequent words.

5. Two- and Three-Word Sequences (n-grams)

A predictive keyboard doesn’t just look at word frequency — it looks at which words tend to follow each other. These pairs and triplets are called bigrams and trigrams, and they are the real foundation of the prediction algorithm.

get_ngrams <- function(tokens, n) {
  ngrams <- vector("character", length(tokens) - n + 1)
  for (i in 1:(length(tokens) - n + 1)) {
    ngrams[i] <- paste(tokens[i:(i + n - 1)], collapse = " ")
  }
  ngrams
}

bigrams  <- get_ngrams(tokens, 2)
trigrams <- get_ngrams(tokens, 3)

bigram_freq  <- as.data.frame(table(bigrams),  stringsAsFactors = FALSE)
trigram_freq <- as.data.frame(table(trigrams), stringsAsFactors = FALSE)
names(bigram_freq)  <- c("bigram", "count")
names(trigram_freq) <- c("trigram", "count")

bigram_freq  <- bigram_freq[order(-bigram_freq$count), ]
trigram_freq <- trigram_freq[order(-trigram_freq$count), ]

kable(head(bigram_freq, 10), row.names = FALSE, caption = "Table 3: Top 10 two-word sequences")
Table 3: Top 10 two-word sequences
bigram count
of the 4240
in the 4121
for the 2081
to the 2068
on the 1993
to be 1630
at the 1431
and the 1266
in a 1161
with the 1098 
ggplot(head(bigram_freq, 15), aes(x = reorder(bigram, count), y = count)) +
  geom_col(fill = "#34A853") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Word Pairs (Bigrams)",
       x = NULL, y = "Frequency") +
  theme_minimal(base_size = 12)

kable(head(trigram_freq, 10), row.names = FALSE, caption = "Table 4: Top 10 three-word sequences")
Table 4: Top 10 three-word sequences
trigram count
one of the 377
a lot of 321
thanks for the 249
going to be 194
to be a 190
out of the 157
the end of 153
the u s 151
as well as 148
be able to 147

Finding: Common bigrams and trigrams are dominated by everyday phrases (e.g. “of the”, “in the”, “one of the”). This confirms the data is well-suited to n-gram based prediction — the same handful of short phrases account for a disproportionate share of all word sequences, which is exactly the pattern a prediction model can exploit.

6. How Much Vocabulary Do We Actually Need?

One practical question for building a fast, lightweight app: how many unique words are needed to cover most of what people actually type?

word_freq_sorted <- word_freq[order(-word_freq$count), ]
word_freq_sorted$cum_pct <- cumsum(word_freq_sorted$count) / sum(word_freq_sorted$count)

words_for_50 <- which(word_freq_sorted$cum_pct >= 0.5)[1]
words_for_90 <- which(word_freq_sorted$cum_pct >= 0.9)[1]

coverage_df <- data.frame(
  `Coverage Target` = c("50% of all word instances", "90% of all word instances"),
  `Unique Words Needed` = c(words_for_50, words_for_90),
  check.names = FALSE
)
kable(coverage_df, caption = "Table 5: Vocabulary size needed for coverage")
Table 5: Vocabulary size needed for coverage
Coverage Target Unique Words Needed
50% of all word instances 140
90% of all word instances 6822

Finding: A relatively small “core vocabulary” covers half of everything written, and a modest dictionary (far smaller than the full set of unique words in the data) covers 90%. This is good news for app performance — the model doesn’t need to store every rare word to be useful; it can prioritize a manageable core dictionary and still predict well most of the time.

7. Plan for the Prediction Algorithm

In plain terms: the app will work like this —

  • Step 1 — Build a phrase “memory.” Using the bigrams/trigrams above (and longer 4-word sequences), the model will build a large table of “if the last 1–3 words were X, the next word was usually Y.”
  • Step 2 — Predict from context. When a user types a partial sentence, the app looks up the last few words typed and checks the phrase table for the most likely next word(s).
  • Step 3 — Handle the unexpected. If the typed phrase was never seen before (which will happen often), the model falls back to shorter phrase matches, and ultimately to overall word frequency, so it always has something reasonable to suggest — this is a standard technique called “backoff.”
  • Step 4 — Keep it fast and light. Since the app needs to respond instantly as someone types, the phrase table will be trimmed to the most useful, frequent entries (using the coverage findings above) rather than storing the entire dataset — trading a small amount of accuracy for a big gain in speed.

8. Plan for the Shiny App

The final deliverable will be a simple, interactive web app (built with R’s Shiny framework) with:

  • A single text box where a user types a sentence.
  • A live-updating suggestion of the most likely next word(s), shown as the user types — the same experience as a phone keyboard’s predictive text.
  • A minimal, uncluttered interface, since the focus is on prediction quality and speed rather than visual complexity.

Conclusion

The data has been successfully downloaded, loaded, and explored. It’s large, messy, and drawn from three very different writing styles, but the basic patterns — common words, common phrases, and a coverage “sweet spot” for vocabulary size — all point toward a workable, standard n-gram approach with backoff. The next phase of the project will build and tune this model, then wrap it in the Shiny app described above.