Typing is one of the most common ways people interact with digital products. A useful next-word predictor can reduce the number of keystrokes required to complete a message and make text entry faster and more convenient. The same underlying idea can support several practical applications, including:

  • predictive keyboards for smartphones and tablets;
  • autocomplete in messaging, email, and document-writing tools;
  • faster entry of repeated phrases in customer-service or support environments;
  • writing assistance interfaces where likely continuations are suggested as the user types; and
  • accessibility-oriented interfaces where reducing repetitive typing may improve ease of use.

For this project, the immediate product is a Shiny application that accepts a phrase and returns a small set of likely next words.

Data Loading

The SwiftKey corpus is stored in a compressed ZIP file. The selected language is read directly from the archive so that the workflow remains reproducible and does not require additional extracted copies of the source files.

sources <- tibble(
  source = c("Blogs", "News", "Twitter"),
  type = c("blogs", "news", "twitter")
)

files <- sources %>%
  mutate(
    file = paste0(
      "final/",
      language,
      "/",
      language,
      ".",
      type,
      ".txt"
    )
  )

files
## # A tibble: 3 × 3
##   source  type    file                         
##   <chr>   <chr>   <chr>                        
## 1 Blogs   blogs   final/en_US/en_US.blogs.txt  
## 2 News    news    final/en_US/en_US.news.txt   
## 3 Twitter twitter final/en_US/en_US.twitter.txt

The three datasets are then combined into one corpus, with one row per line of text.

read_swiftkey_file <- function(zip_file, file_path, source_name, language_name) {

  text <- readLines(
    unz(zip_file, file_path),
    encoding = "UTF-8",
    skipNul = TRUE
  )

  tibble(
    language = language_name,
    source = source_name,
    text = text
  )
}
swiftkey_data <- map2_dfr(
  files$file,
  files$source,
  ~ read_swiftkey_file(
    zip_file = zip_file,
    file_path = .x,
    source_name = .y,
    language_name = language
  )
)

Dataset Summary

Basic descriptive statistics were calculated to compare the size and structure of the three sources. The summary includes file size, number of lines, total words and characters, and measures of line length.

zip_contents <- unzip(
  zip_file,
  list = TRUE
)

file_sizes <- files %>%
  left_join(
    zip_contents %>%
      transmute(
        file = Name,
        file_size_mb = Length / 1024^2
      ),
    by = "file"
  ) %>%
  select(
    source,
    file_size_mb
  )

corpus_summary <- swiftkey_data %>%
  mutate(
    words = str_count(str_squish(text), "\\S+"),
    characters = nchar(text)
  ) %>%
  group_by(source) %>%
  summarise(
    lines = n(),
    total_words = sum(words),
    total_characters = sum(characters),
    average_words_per_line = mean(words),
    median_words_per_line = median(words),
    min_words_per_line = min(words),
    max_words_per_line = max(words),
    average_characters_per_line = mean(characters),
    median_characters_per_line = median(characters),
    min_characters_per_line = min(characters),
    max_characters_per_line = max(characters),
    .groups = "drop"
  ) %>%
  left_join(
    file_sizes,
    by = "source"
  )
Summary Statistics of the English SwiftKey Corpus
Source File Size (MB) Lines Total Words Total Characters Avg. Words/Line Median Words/Line Max Words/Line Avg. Characters/Line
Blogs 200.4 899,288 37,334,116 206,824,505 41.52 28 6,630 229.99
News 196.3 1,010,242 34,372,529 203,223,159 34.02 31 1,792 201.16
Twitter 159.4 2,360,148 30,373,583 162,096,241 12.87 12 47 68.68

For the English corpus, Twitter contributes the largest number of individual observations, but those observations are much shorter than blog and news entries. Blogs contain fewer lines but much longer text. News sits between the two.

The three sources are not simply duplicates of the same type of writing. They provide complementary forms of English: short conversational text from Twitter and longer-form language from blogs and news. That variety is useful because a prediction model must work across different writing contexts.

Comparing the Three Sources

Number of Lines by Source

The number of lines shows how many separate observations each source contributes.

Twitter dominates when the corpus is measured by number of lines. This is useful for exposing the model to many short messages, but line counts alone do not show how much actual language each source contributes.

Total Words by Source

Although Twitter has far more lines, blogs contribute the largest total number of words because individual blog entries are considerably longer.

Lines Versus Words: A Direct Comparison

The next plot compares each source’s share of total lines with its share of total words. This makes the difference between number of observations and amount of language directly visible.

contribution_summary <- corpus_summary %>%
  mutate(
    line_share = lines / sum(lines),
    word_share = total_words / sum(total_words)
  ) %>%
  select(
    source,
    line_share,
    word_share
  ) %>%
  pivot_longer(
    cols = c(line_share, word_share),
    names_to = "metric",
    values_to = "share"
  ) %>%
  mutate(
    metric = recode(
      metric,
      line_share = "Share of lines",
      word_share = "Share of words"
    )
  )

Why this matters: Twitter contributes many short examples, while blogs contribute proportionally more words and longer context. A model trained across the sources can therefore learn both conversational patterns and longer sequences.

Average and Median Words per Line

The mean and median are compared because very long lines can pull the average upward. The gap between these measures helps identify skewness and outliers.

mean_median_words <- corpus_summary %>%
  select(
    source,
    average_words_per_line,
    median_words_per_line
  ) %>%
  pivot_longer(
    cols = c(
      average_words_per_line,
      median_words_per_line
    ),
    names_to = "measure",
    values_to = "words_per_line"
  ) %>%
  mutate(
    measure = recode(
      measure,
      average_words_per_line = "Average",
      median_words_per_line = "Median"
    )
  )

A noticeably higher mean than median indicates that a relatively small number of long entries are increasing the average. For model development, this means summary statistics should not be interpreted without considering the full distribution.

Exploratory Visualizations of Text Length

Distribution of Words per Line

Because a small number of unusually long entries can dominate the scale, the visualization focuses on lines containing 100 words or fewer.

line_lengths <- swiftkey_data %>%
  mutate(
    words_per_line = str_count(str_squish(text), "\\S+"),
    characters_per_line = nchar(text)
  )

Twitter is concentrated at short line lengths, while blogs and news contain longer text sequences. This confirms that the sources represent meaningfully different writing environments.

Distribution of Characters per Line

The character distributions reinforce the same pattern. From a product perspective, this suggests that the prediction system should be able to respond effectively to short conversational inputs rather than relying only on long context.

Sampling for Text Analysis

The full corpus is large. For exploratory word-frequency and n-gram analysis, a reproducible random sample of 50,000 lines from each source is used.

Equal sampling serves two purposes. First, it reduces processing time and memory requirements. Second, it prevents Twitter, which has many more lines, from automatically dominating the exploratory language patterns.

set.seed(123)

sample_size <- 50000

corpus_sample <- swiftkey_data %>%
  group_by(source) %>%
  slice_sample(n = sample_size) %>%
  ungroup()

corpus_sample %>%
  count(source) %>%
  knitr::kable(
    col.names = c("Source", "Sampled Lines"),
    caption = "Exploratory Analysis Sample"
  )
Exploratory Analysis Sample
Source Sampled Lines
Blogs 50000
News 50000
Twitter 50000

The sampled text is converted into a corpus so that source information is retained as document metadata. Punctuation, numbers, symbols, and URLs are removed, and words are converted to lowercase.

Common words such as the, to, of, and you are intentionally retained. In many text-mining tasks these words are removed, but in next-word prediction they carry important information because they are often exactly the words a user may type next.

qcorpus_sample <- quanteda::corpus(
  corpus_sample,
  text_field = "text"
)

tokens_sample <- qcorpus_sample %>%
  tokens(
    remove_punct = TRUE,
    remove_numbers = TRUE,
    remove_symbols = TRUE,
    remove_url = TRUE
  ) %>%
  tokens_tolower()

What Does the Model See Most Often?

Most Frequent Words Overall

word_dfm <- dfm(tokens_sample)

top_words_vector <- topfeatures(
  word_dfm,
  n = 20
)

top_words <- tibble(
  feature = names(top_words_vector),
  frequency = as.numeric(top_words_vector)
)

The word cloud makes the concentration of common vocabulary immediately visible: larger words are those that appear more frequently in the balanced sample.

Frequent words matter because a practical predictor should detect patterns that occur often than to combinations seen only once.

Comparing Frequent Words Across Sources

The overall ranking can hide differences between writing environments. The next analysis identifies the ten most frequent words separately within blogs, news, and Twitter.

word_dfm_by_source <- dfm_group(
  word_dfm,
  groups = docvars(word_dfm, "source")
)

source_word_matrix <- as.matrix(word_dfm_by_source)

source_top_words <- purrr::map_dfr(
  seq_len(nrow(source_word_matrix)),
  function(i) {
    values <- sort(
      source_word_matrix[i, ],
      decreasing = TRUE
    )

    n_keep <- min(10, length(values))

    tibble(
      source = rownames(source_word_matrix)[i],
      feature = names(values)[seq_len(n_keep)],
      frequency = as.numeric(values[seq_len(n_keep)])
    )
  }
)

The three word clouds provide a quick visual comparison of the vocabulary that dominates each writing environment. Words shown larger occur more frequently within that source’s balanced sample.

This comparison helps distinguish language patterns shared across the corpus from patterns that are stronger in one type of writing. For the final model, it supports combining sources while recognizing that source style can influence observed frequencies.

How Concentrated Is the Vocabulary?

A prediction system must balance coverage with model size. If a relatively small number of words account for a large share of all observed tokens, the application can focus storage and computation on the patterns that appear most often.

word_counts <- sort(
  colSums(word_dfm),
  decreasing = TRUE
)

word_rank <- tibble(
  rank = seq_along(word_counts),
  frequency = as.numeric(word_counts),
  cumulative_share = cumsum(frequency) / sum(frequency)
)

The curve shows how quickly common vocabulary accounts for observed language. This helps motivate pruning: extremely rare patterns can potentially be excluded from the production model to reduce memory use while retaining the combinations most likely to matter.

Common Word Sequences

Individual word counts describe vocabulary, but next-word prediction depends on context. The central modeling question is whether words repeatedly occur in recognizable sequences.

Bigrams: Two-Word Patterns

bigram_dfm <- tokens_sample %>%
  tokens_ngrams(
    n = 2,
    concatenator = " "
  ) %>%
  dfm()

top_bigrams_vector <- topfeatures(
  bigram_dfm,
  n = 15
)

top_bigrams <- tibble(
  feature = names(top_bigrams_vector),
  frequency = as.numeric(top_bigrams_vector)
)

The larger phrases in the word cloud are the bigrams that occur most frequently in the exploratory sample. The bigram results show that words do not occur independently. Some pairs occur far more often than others.

Once the user types one word, the set of reasonable next words becomes smaller.

Trigrams: Adding More Context

trigram_dfm <- tokens_sample %>%
  tokens_ngrams(
    n = 3,
    concatenator = " "
  ) %>%
  dfm()

top_trigrams_vector <- topfeatures(
  trigram_dfm,
  n = 15
)

top_trigrams <- tibble(
  feature = names(top_trigrams_vector),
  frequency = as.numeric(top_trigrams_vector)
)

The trigram word cloud highlights the longer phrases that recur most often. These sequences provide more context than individual words or bigrams.The words already typed contain useful information about the word that is likely to follow.

The next word is not random. Repeated bigrams and trigrams show that recent context can be converted into ranked next-word candidates.

Comparing One-, Two-, and Three-Word Patterns

The frequency of the most common unigram, bigram, and trigram is placed on a common scale below. The purpose is not to compare the linguistic meaning of different n-gram orders, but to show how exact sequences naturally become less frequent as more context is required.

ngram_comparison <- tibble(
  ngram = c(
    "Most frequent unigram",
    "Most frequent bigram",
    "Most frequent trigram"
  ),
  frequency = c(
    max(top_words$frequency),
    max(top_bigrams$frequency),
    max(top_trigrams$frequency)
  )
)

This illustrates the central trade-off behind a backoff model. Longer sequences provide more context and can produce more specific predictions, but they are less likely to have been observed. Shorter sequences are less specific but provide broader coverage.

Key Findings

The exploratory analysis leads to four practical conclusions.

1. There is enough text to learn useful patterns.
The corpus contains millions of observations and a large volume of words. This gives the model many examples from which to estimate common language patterns.

2. The sources represent different ways of writing.
Twitter contributes many short, conversational messages, while blogs and news provide longer passages. Combining them exposes the model to a wider range of English.

3. Common vocabulary accounts for a meaningful share of observed language.
Word usage is not evenly distributed. Some words appear far more often than others. This creates an opportunity to make the final model smaller by prioritizing frequent patterns and pruning extremely rare ones.

4. Previous words provide information about the next word.
The repeated bigrams and trigrams are the most important finding for the final application. They show that recent context can be used to rank possible next words.

Together, these findings create a clear path from exploratory analysis to a usable product: learn repeated language patterns, keep the most useful ones, use the longest reliable context available, and return the highest-ranked next-word candidates quickly.

Product Plan: Shiny Next-Word Predictor

The planned model will use n-grams of up to four words together with a backoff strategy.

When the user enters text:

  1. The application will clean and normalize the recent input using preprocessing consistent with the training data.
  2. If three preceding words are available, the model will first search the four-gram table for likely continuations.
  3. If that context has not been observed or does not provide a reliable candidate, the model will back off to a trigram context.
  4. If necessary, it will continue to a bigram and finally to unigram frequencies.
  5. Candidate next words will be ranked by their observed frequencies or estimated probabilities.
  6. The highest-ranked suggestions will be returned to the user.

Conceptually:

4-gram context → trigram context → bigram context → unigram fallback

The Shiny app will contain:

  • a text box where the user enters a phrase;
  • a prediction action or automatic update;
  • a small set of likely next-word suggestions; and
  • a clean results area that makes the best candidates easy to identify.

Conclusion

This exploratory analysis shows that the SwiftKey corpus has the characteristics needed to support a next-word prediction application.

The corpus is large and diverse. Twitter supplies many short conversational examples, while blogs and news provide longer context. Word frequencies show that some vocabulary occurs much more often than the rest, creating an opportunity for efficient pruning. Most importantly, the bigram and trigram analyses demonstrate that words repeatedly occur in recognizable sequences.

The main conclusion can therefore be stated simply:

The next word is not random. The words that come before it provide information that can be learned and transformed into a useful prediction.

The next phase will convert these observations into an optimized n-gram backoff model, test its accuracy and efficiency on held-out text, and integrate the model into an interactive Shiny application.