Introduction

This report presents an exploratory analysis of a large text corpus used to build a next-word prediction model. The data consist of three sources: blog posts, news articles, and Twitter messages. After combining and sampling 60,000 lines, the text was tokenized into words and analyzed using unigrams, bigrams, and trigrams.

The goals of this analysis are to:

Corpus Summary

The table below summarizes the number of lines and total words in the 60,000-line working sample.

# Load required packages
library(tidyverse)
library(SnowballC)


# Load the three datasets using correct paths and file names
blogs  <- readLines("../data/en_US.blogs.txt",   encoding = "UTF-8")
news   <- readLines("../data/en_US.news.txt",    encoding = "UTF-8")
twitter <- readLines("../data/en_US.twitter.txt", encoding = "UTF-8")



# Create a 60,000-line sample
set.seed(123)
sample_blogs   <- sample(blogs,   20000)
sample_news    <- sample(news,    20000)
sample_twitter <- sample(twitter, 20000)
sample_data <- c(sample_blogs, sample_news, sample_twitter)

# Tokenize into words
tokens <- tolower(sample_data) %>%
  str_extract_all("[[:alpha:]]+") %>%
  unlist()

# Create a data frame
df <- tibble(word = tokens)

# Summary statistics
corpus_summary <- tibble(
  metric = c("Total lines", "Total words", "Unique words"),
  value = c(
    length(sample_data),
    nrow(df),
    n_distinct(df$word)
  )
)

knitr::kable(corpus_summary,
             col.names = c("Metric", "Value"),
             caption = "Corpus summary for the 60,000-line sample")
Corpus summary for the 60,000-line sample
Metric Value
Total lines 60000
Total words 1771367
Unique words 64403

Top Unigrams

The most frequent individual words in the corpus are shown below. Function words such as “the”, “to”, and “and” dominate, which is typical for natural language text.

# Count word frequencies
unigram_counts <- df %>%
  count(word, sort = TRUE)

# Show top 10
top_unigrams <- unigram_counts %>%
  head(10)

knitr::kable(top_unigrams,
             col.names = c("Word", "Frequency"),
             caption = "Top 10 most frequent words (unigrams)")
Top 10 most frequent words (unigrams)
Word Frequency
the 87030
to 47986
and 45160
a 42321
of 37193
i 31533
in 30036
that 20165
it 19666
s 18917

Top Bigrams

The most frequent two-word sequences reveal common phrases and collocations in the data.

# Create bigrams
bigrams <- df %>%
  mutate(lead_word = lead(word)) %>%
  filter(!is.na(lead_word)) %>%
  unite(bigram, word, lead_word, sep = " ")

# Count bigram frequencies
bigram_counts <- bigrams %>%
  count(bigram, sort = TRUE)

# Show top 10
top_bigrams <- bigram_counts %>%
  head(10)

knitr::kable(top_bigrams,
             col.names = c("Bigram", "Frequency"),
             caption = "Top 10 most frequent bigrams")
Top 10 most frequent bigrams
Bigram Frequency
of the 8033
in the 7659
to the 3882
it s 3831
on the 3533
for the 3384
i m 2923
to be 2863
at the 2456
and the 2446

Top Trigrams

The most frequent three-word sequences capture short phrases that often appear together in the corpus.

# Create trigrams
trigrams <- df %>%
  mutate(lead1 = lead(word),
         lead2 = lead(word, 2)) %>%
  filter(!is.na(lead1), !is.na(lead2)) %>%
  unite(trigram, word, lead1, lead2, sep = " ")

# Count trigram frequencies
trigram_counts <- trigrams %>%
  count(trigram, sort = TRUE)

# Show top 10
top_trigrams <- trigram_counts %>%
  head(10)

knitr::kable(top_trigrams,
             col.names = c("Trigram", "Frequency"),
             caption = "Top 10 most frequent trigrams")
Top 10 most frequent trigrams
Trigram Frequency
i don t 854
one of the 611
a lot of 491
it s a 445
i m not 385
i can t 370
i didn t 326
you don t 302
as well as 293
it s not 291

Vocabulary Coverage

This section examines how much of the corpus is covered by the most frequent words. We calculate how many unique words are needed to account for 50% and 90% of all word tokens.

# Compute cumulative coverage
unigram_cumulative <- unigram_counts %>%
  mutate(cum_n = cumsum(n),
         cum_prop = cum_n / sum(n))

# Find words needed for 50% and 90% coverage
coverage_50 <- unigram_cumulative %>%
  filter(cum_prop >= 0.50) %>%
  slice(1) %>%
  pull(word) %>%
  {which(unigram_counts$word == .)}  # get rank

coverage_90 <- unigram_cumulative %>%
  filter(cum_prop >= 0.90) %>%
  slice(1) %>%
  pull(word) %>%
  {which(unigram_counts$word == .)}  # get rank

# Create coverage summary table
coverage_summary <- tibble(
  coverage_target = c("50%", "90%"),
  unique_words_needed = c(coverage_50, coverage_90)
)

knitr::kable(coverage_summary,
             col.names = c("Coverage Target", "Unique Words Needed"),
             caption = "Number of most frequent words needed to reach coverage targets")
Number of most frequent words needed to reach coverage targets
Coverage Target Unique Words Needed
50% 126
90% 6787

The results show that a relatively small number of high-frequency words account for a large proportion of all tokens, consistent with Zipf’s law in natural language. ## Prediction Model

The next-word predictor uses a backoff n‑gram model with three levels:

  1. Trigram model: Given the last two words, predict the most likely next word based on three-word sequences in the training data.
  2. Bigram fallback: If the trigram is not found, use the last word to predict the next word based on two-word sequences.
  3. Unigram fallback: If no bigram is found, return the most frequent words in the corpus.

This approach ensures the model can always make a prediction, even for rare or unseen word sequences.

Example Predictions

The table below shows example predictions for the phrase “i want” using the trigram model.

# Example: predictions for "i want"
example_context <- c("i", "want")

# Find trigrams starting with "i want"
example_trigrams <- trigram_counts %>%
  filter(str_starts(trigram, "i want ")) %>%
  mutate(next_word = str_remove(trigram, "i want ")) %>%
  select(next_word, n) %>%
  head(3)

# Add model used column
example_trigrams$model_used <- "trigram"

knitr::kable(example_trigrams %>% select(next_word, n, model_used),
             col.names = c("Next Word", "Frequency", "Model Used"),
             caption = "Example trigram predictions for the phrase 'i want'")
Example trigram predictions for the phrase ‘i want’
Next Word Frequency Model Used
to 226 trigram
you 22 trigram
a 17 trigram