Purpose

The goal is an app that suggests the next word as a user types, much like a phone keyboard does. This report shows that the data is loaded, says its main features, and sets out the plan for the model and the app.

The data

The data contains three English text files as downloaded.

Table 1. The three full source files
Source Size (MB) Lines Words Mean words per line Longest line (chars)
blogs 200.42 899,288 37,546,251 41.8 40,833
news 196.28 1,010,242 34,762,395 34.4 11,384
twitter 159.36 2,360,148 30,093,418 12.8 140

Together, the files hold about 102 million words. Twitter has the most lines but the shortest ones. Blogs have the fewest lines, but each one is much longer. All further analysis uses a random 1% sample of each file (42,696 lines).

Line length by source

The three sources differ in style. Tweets are short and informal. News is edited and formal. The model should learn from all three so that it works for both casual and formal typing.

Most common words and phrases

The text was lowercased, and numbers, web links and punctuation were removed (apostrophes were kept).

The most common words are short function words such as “the”, “to” and “and”. These are usually removed in text analysis, but here they must stay, because people type them constantly and the app has to predict them.

The Number of Words we need.

The sample contains 50,917 distinct words. The 141 most common words account for half of all words typed. The 6,854 most common words account for 90%. This means the app can drop rare words and stay small and fast while losing little accuracy.

Foreign and misspelled words

Of the distinct words, 44% are not in a standard English dictionary. Most of these are rare, so they make up only 6.4% of all words typed. This group includes foreign words, typos, slang and names, so it overstates how much foreign text there is. Because the model will drop rare words anyway, most of these will be removed automatically.

Plan for the prediction model

  1. Build lookup tables of common one-, two-, three- and four-word phrases from a larger sample, keeping only phrases that appear more than a few times.
  2. Predict from the last few words typed. The model first looks for the last three words in the four-word table. If there’s no match, it falls back to the last two words.
  3. Test on held-out text that the model has not seen, and measure how often the correct next word appears in the top three suggestions. Speed and memory use will also be measured, since the app must respond instantly.

Plan for the Shiny app

The app will have a text box. As the user types, it will show the three most likely next words, which update with each word typed. The lookup tables will be precomputed and saved, so the app only reads them and does not rebuild them.

R Code

knitr::opts_chunk$set(echo = FALSE)
library(tidyverse)
library(tidytext)
library(stringi)
library(hunspell)
library(knitr)
set.seed(1234)
read_corpus <- function(path) {
        con <- file(path, open = "rb")
        readLines(con, encoding = "UTF-8", skipNul = TRUE)
}

files <- c(blogs   = "data/en_US/en_US.blogs.txt",
           news    = "data/en_US/en_US.news.txt",
           twitter = "data/en_US/en_US.twitter.txt")

raw <- map(files, read_corpus)

summary_tbl <- tibble(
        Source = names(raw),
        `Size (MB)` = round(file.size(files) / 1024^2, 2),
        Lines = map_int(raw, length),
        Words = map_dbl(raw, ~ sum(stri_count_words(.x))),
        `Mean words per line`   = round(Words / Lines, 1),
        `Longest line (chars)`  = map_int(raw, ~ max(nchar(.x)))
)

# Work from a 1% random sample of each file; the full files are too large to process quickly
sample_df <- imap_dfr(raw, 
                      ~ tibble(source = .y,
                                    text   = sample(.x, round(length(.x) * 0.01)))) |> 
        mutate(n_words = stri_count_words(text))

rm(raw); invisible(gc())
kable(summary_tbl, format.args = list(big.mark = ","),
      caption = "Table 1. The three full source files")
ggplot(sample_df, aes(n_words, fill = source)) +
        geom_histogram(binwidth = 5, show.legend = FALSE) +
        facet_wrap(~ source, scales = "free_y") +
        coord_cartesian(xlim = c(0, 150)) +
        labs(x = "Words per line", 
             y = "Number of lines",
             title = "Figure 1. Twitter lines are short; blog and news lines vary widely") +
        theme_classic()
clean_df <- sample_df |> 
        mutate(text = text |> 
                       stri_replace_all_regex("https?://\\S+|www\\.\\S+", " ") |> 
                       stri_replace_all_regex("[^A-Za-z' ]", " "))

count_ngrams <- function(df, n) {
        df |> 
                unnest_tokens(term, text, token = "ngrams", n = n) |> 
                filter(!is.na(term)) |> 
                count(term, sort = TRUE)
}

uni_counts <- count_ngrams(clean_df, 1)
bi_counts  <- count_ngrams(clean_df, 2)
tri_counts <- count_ngrams(clean_df, 3)
plot_top <- function(counts, title, n = 15) {
        counts |> 
                slice_head(n = n) |> 
                ggplot(aes(n, reorder(term, n))) +
                geom_col(fill = "steelblue") +
                labs(x = "Count", y = NULL, title = title) +
                theme_classic()
}

plot_top(uni_counts, "Figure 2. Top 15 single words")
plot_top(bi_counts,  "Figure 3. Top 15 word pairs")
plot_top(tri_counts, "Figure 4. Top 15 three-word phrases")
coverage <- uni_counts |> 
        mutate(rank = row_number(), coverage = cumsum(n) / sum(n))

count_50 <- which(coverage$coverage >= 0.50)[1]
count_90 <- which(coverage$coverage >= 0.90)[1]

ggplot(coverage, aes(rank, coverage)) +
        geom_line(colour = "steelblue", linewidth = 1) +
        geom_vline(xintercept = c(count_50, count_90), linetype = "dashed") +
        scale_x_log10(labels = scales::comma) +
        scale_y_continuous(labels = scales::percent) +
        labs(x = "Number of distinct words (log scale)", y = "Share of all words in text",
             title = "Figure 5. A small vocabulary covers most of the text") +
        theme_classic()
spell <- uni_counts |> 
        mutate(in_dict = hunspell_check(term))
pct_types  <- round(100 * mean(!spell$in_dict), 1)
pct_tokens <- round(100 * sum(spell$n[!spell$in_dict]) / sum(spell$n), 1)