Next-Word Prediction: Exploratory Analysis
POOJA AJITH KUMAR
Introduction
This report presents an exploratory analysis of a large text corpus used to build a next-word prediction model. The data consist of three sources: blog posts, news articles, and Twitter messages. After combining and sampling 60,000 lines, the text was tokenized into words and analyzed using unigrams, bigrams, and trigrams.
The goals of this analysis are to:
- Describe the structure and size of the corpus.
- Examine the most frequent words and word sequences.
- Evaluate how much of the vocabulary is covered by the most common terms.
- Demonstrate a trigram-based next-word predictor with bigram and unigram fallback.
Corpus Summary
The table below summarizes the number of lines and total words in the 60,000-line working sample.
# Load required packages
library(tidyverse)
library(SnowballC)
# Load the three datasets using correct paths and file names
blogs <- readLines("../data/en_US.blogs.txt", encoding = "UTF-8")
news <- readLines("../data/en_US.news.txt", encoding = "UTF-8")
twitter <- readLines("../data/en_US.twitter.txt", encoding = "UTF-8")
# Create a 60,000-line sample
set.seed(123)
sample_blogs <- sample(blogs, 20000)
sample_news <- sample(news, 20000)
sample_twitter <- sample(twitter, 20000)
sample_data <- c(sample_blogs, sample_news, sample_twitter)
# Tokenize into words
tokens <- tolower(sample_data) %>%
str_extract_all("[[:alpha:]]+") %>%
unlist()
# Create a data frame
df <- tibble(word = tokens)
# Summary statistics
corpus_summary <- tibble(
metric = c("Total lines", "Total words", "Unique words"),
value = c(
length(sample_data),
nrow(df),
n_distinct(df$word)
)
)
knitr::kable(corpus_summary,
col.names = c("Metric", "Value"),
caption = "Corpus summary for the 60,000-line sample")| Metric | Value |
|---|---|
| Total lines | 60000 |
| Total words | 1771367 |
| Unique words | 64403 |
Top Unigrams
The most frequent individual words in the corpus are shown below. Function words such as “the”, “to”, and “and” dominate, which is typical for natural language text.
# Count word frequencies
unigram_counts <- df %>%
count(word, sort = TRUE)
# Show top 10
top_unigrams <- unigram_counts %>%
head(10)
knitr::kable(top_unigrams,
col.names = c("Word", "Frequency"),
caption = "Top 10 most frequent words (unigrams)")| Word | Frequency |
|---|---|
| the | 87030 |
| to | 47986 |
| and | 45160 |
| a | 42321 |
| of | 37193 |
| i | 31533 |
| in | 30036 |
| that | 20165 |
| it | 19666 |
| s | 18917 |
Word Frequency Distribution
To understand how word usage is distributed in the corpus, we examine how many words appear very frequently versus rarely. The histogram below shows the distribution of word frequencies on a log scale.
library(ggplot2)
# Use unigram_counts from the earlier chunk
word_freq_plot_data <- unigram_counts %>%
mutate(log_n = log10(n))
ggplot(word_freq_plot_data, aes(x = log_n)) +
geom_histogram(binwidth = 0.2, fill = "steelblue", color = "white") +
labs(
title = "Distribution of Word Frequencies (log scale)",
x = "log10(Frequency)",
y = "Number of unique words"
) +
theme_minimal()
The plot shows that most words occur relatively few times, while a small number of words (such as “the”, “to”, “and”) occur very frequently. This pattern is typical in natural language and supports the use of a backoff model that relies heavily on common words and phrases. ## Top Bigrams
The most frequent two-word sequences reveal common phrases and collocations in the data.
# Create bigrams
bigrams <- df %>%
mutate(lead_word = lead(word)) %>%
filter(!is.na(lead_word)) %>%
unite(bigram, word, lead_word, sep = " ")
# Count bigram frequencies
bigram_counts <- bigrams %>%
count(bigram, sort = TRUE)
# Show top 10
top_bigrams <- bigram_counts %>%
head(10)
knitr::kable(top_bigrams,
col.names = c("Bigram", "Frequency"),
caption = "Top 10 most frequent bigrams")| Bigram | Frequency |
|---|---|
| of the | 8033 |
| in the | 7659 |
| to the | 3882 |
| it s | 3831 |
| on the | 3533 |
| for the | 3384 |
| i m | 2923 |
| to be | 2863 |
| at the | 2456 |
| and the | 2446 |
Top Trigrams
The most frequent three-word sequences capture short phrases that often appear together in the corpus.
# Create trigrams
trigrams <- df %>%
mutate(lead1 = lead(word),
lead2 = lead(word, 2)) %>%
filter(!is.na(lead1), !is.na(lead2)) %>%
unite(trigram, word, lead1, lead2, sep = " ")
# Count trigram frequencies
trigram_counts <- trigrams %>%
count(trigram, sort = TRUE)
# Show top 10
top_trigrams <- trigram_counts %>%
head(10)
knitr::kable(top_trigrams,
col.names = c("Trigram", "Frequency"),
caption = "Top 10 most frequent trigrams")| Trigram | Frequency |
|---|---|
| i don t | 854 |
| one of the | 611 |
| a lot of | 491 |
| it s a | 445 |
| i m not | 385 |
| i can t | 370 |
| i didn t | 326 |
| you don t | 302 |
| as well as | 293 |
| it s not | 291 |
Vocabulary Coverage
This section examines how much of the corpus is covered by the most frequent words. We calculate how many unique words are needed to account for 50% and 90% of all word tokens.
# Compute cumulative coverage
unigram_cumulative <- unigram_counts %>%
mutate(cum_n = cumsum(n),
cum_prop = cum_n / sum(n))
# Find words needed for 50% and 90% coverage
coverage_50 <- unigram_cumulative %>%
filter(cum_prop >= 0.50) %>%
slice(1) %>%
pull(word) %>%
{which(unigram_counts$word == .)} # get rank
coverage_90 <- unigram_cumulative %>%
filter(cum_prop >= 0.90) %>%
slice(1) %>%
pull(word) %>%
{which(unigram_counts$word == .)} # get rank
# Create coverage summary table
coverage_summary <- tibble(
coverage_target = c("50%", "90%"),
unique_words_needed = c(coverage_50, coverage_90)
)
knitr::kable(coverage_summary,
col.names = c("Coverage Target", "Unique Words Needed"),
caption = "Number of most frequent words needed to reach coverage targets")| Coverage Target | Unique Words Needed |
|---|---|
| 50% | 126 |
| 90% | 6787 |
The results show that a relatively small number of high-frequency words account for a large proportion of all tokens, consistent with Zipf’s law in natural language. ## Prediction Model
The next-word predictor uses a backoff n‑gram model with three levels:
- Trigram model: Given the last two words, predict the most likely next word based on three-word sequences in the training data.
- Bigram fallback: If the trigram is not found, use the last word to predict the next word based on two-word sequences.
- Unigram fallback: If no bigram is found, return the most frequent words in the corpus.
This approach ensures the model can always make a prediction, even for rare or unseen word sequences.
Example Predictions
The table below shows example predictions for the phrase “i want” using the trigram model.
# Example: predictions for "i want"
example_context <- c("i", "want")
# Find trigrams starting with "i want"
example_trigrams <- trigram_counts %>%
filter(str_starts(trigram, "i want ")) %>%
mutate(next_word = str_remove(trigram, "i want ")) %>%
select(next_word, n) %>%
head(3)
# Add model used column
example_trigrams$model_used <- "trigram"
knitr::kable(example_trigrams %>% select(next_word, n, model_used),
col.names = c("Next Word", "Frequency", "Model Used"),
caption = "Example trigram predictions for the phrase 'i want'")| Next Word | Frequency | Model Used |
|---|---|---|
| to | 226 | trigram |
| you | 22 | trigram |
| a | 17 | trigram |
Project Goals and Plans
The goal of this project is to build a simple, fast next-word prediction tool that can suggest likely words as a user types. The eventual Shiny app will allow a user to enter a short phrase and see the top suggested next words, along with an indication of which model (trigram, bigram, or unigram) was used.
Planned prediction algorithm
- Use the cleaned and sampled corpus to build:
- A trigram model for detailed context (last two words).
- A bigram model as a fallback when trigrams are rare or unseen.
- A unigram model as a final fallback for completely new phrases.
- For any input, the algorithm will:
- Try to predict using the trigram model.
- If no match is found, fall back to the bigram model.
- If still no match, return the most common words from the unigram model.
Planned Shiny app features
- A text input box where users can type a word or short phrase.
- A “Predict Next Word” button that displays:
- The top 3–5 suggested next words.
- The frequency of each suggestion.
- Which model level (trigram, bigram, or unigram) was used.
- A simple, clean layout suitable for demonstration to non-technical stakeholders.
This approach balances accuracy and speed, and ensures the app always returns sensible suggestions even for unusual inputs.