Text Prediction — Exploratory Analysis
Data Science Capstone, Milestone Report
Habiba Chekaik Chaila
13 septembre 2026
Executive summary
Modern smartphone keyboards predict the next word as users type. This project builds a lightweight prediction algorithm trained on a large English-language corpus of blogs, news articles and tweets provided by SwiftKey. This report summarises the exploratory analysis performed on the raw data, highlights key statistical features, and outlines the plan for the prediction model and its accompanying Shiny application.
Loading the data
The corpus is distributed as a single zip file containing text in four languages. Only the English files are used here.
Basic summary statistics
The table below reports file size, line count, word count and
vocabulary size for each source. Word counts are computed with
stringi::stri_count_words.
file_paths <- c("final/en_US/en_US.blogs.txt",
"final/en_US/en_US.news.txt",
"final/en_US/en_US.twitter.txt")
make_summary <- function(lines, path) {
wc <- stri_count_words(lines)
data.frame(
Source = gsub("final/en_US/en_US\\.|.txt", "", path),
Size_MB = round(file.info(path)$size / 1024^2, 1),
Lines = format(length(lines), big.mark = ","),
Total_words = format(sum(wc, na.rm = TRUE), big.mark = ","),
Mean_words_per_line = round(mean(wc, na.rm = TRUE), 1),
Max_words_in_a_line = max(wc, na.rm = TRUE)
)
}
summary_df <- rbind(
make_summary(blogs, file_paths[1]),
make_summary(news, file_paths[2]),
make_summary(twitter, file_paths[3])
)
knitr::kable(summary_df, caption = "Table 1. Basic statistics for the three English corpus files.")| Source | Size_MB | Lines | Total_words | Mean_words_per_line | Max_words_in_a_line |
|---|---|---|---|---|---|
| blogs | 200.4 | 899,288 | 37,546,806 | 41.8 | 6726 |
| news | 196.3 | 1,010,206 | 34,761,151 | 34.4 | 1796 |
| 159.4 | 2,360,148 | 30,096,690 | 12.8 | 47 |
Key observations:
- The corpus contains over 100 million words across three very different registers: long-form blog posts, edited news prose, and short tweets.
- Blog entries are the longest on average, while tweets are capped by Twitter’s character limit and average only about 13 words.
- News articles sit between the two in length and tend to use more formal vocabulary.
Sampling for detailed analysis
The full corpus is too large to tokenise interactively, so all further analysis uses a 5% random sample from each source, combined into a single vector. This gives roughly five million words — enough for stable frequency estimates while keeping computation under a few minutes.
set.seed(42)
sample_frac <- 0.05
samp <- c(
sample(blogs, length(blogs) * sample_frac),
sample(news, length(news) * sample_frac),
sample(twitter, length(twitter) * sample_frac)
)
cat("Sampled lines:", format(length(samp), big.mark = ","), "\n")## Sampled lines: 213,481Text cleaning
Before counting word frequencies, the sample is lowercased, stripped of numbers, punctuation, and excess whitespace. No stemming is applied at this stage because the prediction algorithm will need to distinguish between word forms (e.g. “run” vs “running”).
Word frequency analysis (unigrams)
words <- unlist(strsplit(clean, " "))
words <- words[nchar(words) > 0]
cat("Total word tokens:", format(length(words), big.mark = ","), "\n")## Total word tokens: 5,146,916## Unique words (types): 113,714freq <- sort(table(words), decreasing = TRUE)
top30 <- data.frame(Word = names(freq)[1:30],
Count = as.integer(freq[1:30]),
stringsAsFactors = FALSE)ggplot(top30, aes(x = reorder(Word, Count), y = Count / 1000)) +
geom_col(fill = "#2c3e50") +
coord_flip() +
labs(x = NULL, y = "Frequency (thousands)") +
theme_minimal(base_size = 12)
Figure 1. The 30 most frequent words. Function words (the, to, and, a) dominate, as expected in any English corpus.
Coverage analysis
A small fraction of unique words accounts for the vast majority of text.
cum_pct <- cumsum(as.numeric(freq)) / sum(as.numeric(freq)) * 100
cover50 <- which(cum_pct >= 50)[1]
cover90 <- which(cum_pct >= 90)[1]
cat("Words needed for 50% coverage:", format(cover50, big.mark = ","), "\n")## Words needed for 50% coverage: 127## Words needed for 90% coverage: 6,775cov_df <- data.frame(Rank = seq_along(cum_pct), Coverage = cum_pct)
ggplot(cov_df[1:min(30000, nrow(cov_df)), ],
aes(x = Rank, y = Coverage)) +
geom_line(colour = "#2980b9", linewidth = 0.8) +
geom_hline(yintercept = c(50, 90), linetype = "dashed", colour = "grey50") +
annotate("text", x = cover50 + 2000, y = 52, label = "50%", colour = "grey40") +
annotate("text", x = cover90 + 2000, y = 92, label = "90%", colour = "grey40") +
scale_x_continuous(labels = scales::comma) +
labs(x = "Vocabulary size (ranked by frequency)",
y = "Cumulative coverage (%)") +
theme_minimal(base_size = 12)
Figure 2. Cumulative word coverage. Fewer than 200 words cover 50% of all text; roughly 10,000 cover 90%.
This is encouraging for the prediction model: a dictionary of roughly 10,000 words should suffice for the vast majority of predictions.
Bigram and trigram analysis
For next-word prediction, the model needs to know which words tend to follow which. The helper function below extracts n-grams by shifting the token vector.
make_ngrams <- function(tokens, n = 2) {
if (length(tokens) < n) return(character(0))
ngrams <- character(length(tokens) - n + 1)
for (i in seq_along(ngrams)) {
ngrams[i] <- paste(tokens[i:(i + n - 1)], collapse = " ")
}
ngrams
}
# Build n-grams on each line separately to avoid cross-line artifacts
bigrams <- unlist(lapply(strsplit(clean, " "), make_ngrams, n = 2))
trigrams <- unlist(lapply(strsplit(clean, " "), make_ngrams, n = 3))
freq_bi <- sort(table(bigrams), decreasing = TRUE)
freq_tri <- sort(table(trigrams), decreasing = TRUE)
top_bi <- data.frame(Bigram = names(freq_bi)[1:15],
Count = as.integer(freq_bi[1:15]))
top_tri <- data.frame(Trigram = names(freq_tri)[1:15],
Count = as.integer(freq_tri[1:15]))p1 <- ggplot(top_bi, aes(x = reorder(Bigram, Count), y = Count / 1000)) +
geom_col(fill = "#27ae60") + coord_flip() +
labs(x = NULL, y = "Freq (k)", title = "Bigrams") +
theme_minimal(base_size = 10)
p2 <- ggplot(top_tri, aes(x = reorder(Trigram, Count), y = Count / 1000)) +
geom_col(fill = "#e67e22") + coord_flip() +
labs(x = NULL, y = "Freq (k)", title = "Trigrams") +
theme_minimal(base_size = 10)
gridExtra::grid.arrange(p1, p2, ncol = 2)
Figure 3. Top 15 bigrams and trigrams in the sample.
knitr::kable(
cbind(top_bi[1:10, ], top_tri[1:10, ]),
caption = "Table 2. Ten most frequent bigrams and trigrams."
)| Bigram | Count | Trigram | Count |
|---|---|---|---|
| of the | 21140 | i don t | 2778 |
| in the | 20654 | one of the | 1665 |
| it s | 11643 | a lot of | 1505 |
| to the | 10870 | it s a | 1452 |
| i m | 10764 | i can t | 1282 |
| for the | 9938 | thanks for the | 1212 |
| on the | 9786 | i m not | 1097 |
| to be | 8256 | i didn t | 984 |
| don t | 7931 | going to be | 914 |
| at the | 7337 | to be a | 910 |
The bigram table is dominated by function-word pairs (“of the”, “in the”), but content-bearing patterns like “the united”, “new york” also appear in the top ranks. For trigrams, phrases like “one of the” and “a lot of” are very common, confirming that even three-word context carries strong predictive signal.
Line-length distributions
len_df <- data.frame(
Source = rep(c("Blogs", "News", "Twitter"),
c(length(blogs), length(news), length(twitter))),
Words = c(stri_count_words(blogs),
stri_count_words(news),
stri_count_words(twitter))
)
# Cap at 200 for readability
len_df$Words <- pmin(len_df$Words, 200, na.rm = TRUE)
ggplot(len_df, aes(x = Words, fill = Source)) +
geom_histogram(binwidth = 5, alpha = 0.6, position = "identity") +
facet_wrap(~ Source, scales = "free_y") +
labs(x = "Words per line", y = "Count") +
theme_minimal(base_size = 11) +
theme(legend.position = "none")
Figure 4. Distribution of line length (in words) by source. Blogs span the widest range; tweets cluster tightly.
Plans for the prediction algorithm and Shiny app
Algorithm. The plan is to build an n-gram backoff model:
- Tokenise and clean the full corpus (not just the 5% sample).
- Count all bigrams, trigrams, and 4-grams.
- For a given input phrase, look up the longest matching n-gram and return the most likely next word. If no match is found, fall back to shorter n-grams (backoff), and ultimately to the most frequent unigrams.
- Apply Kneser-Kney or stupid backoff smoothing to handle unseen combinations.
- Prune low-frequency n-grams to keep the model small enough to deploy in a Shiny app.
The coverage analysis above suggests a vocabulary of 10,000 to 20,000 words should be sufficient. The pruned n-gram table will be stored as a compact data frame or data.table for fast lookup.
Shiny app. The app will have a text input field where the user types a phrase. As they type, the app will display the top three predicted next words, which the user can click to accept. The interface will be kept minimal to mimic a real keyboard suggestion bar.
Session info
## R version 4.6.1 (2026-06-24 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
##
## Matrix products: default
## LAPACK version 3.12.1
##
## locale:
## [1] LC_COLLATE=French_Algeria.utf8 LC_CTYPE=French_Algeria.utf8
## [3] LC_MONETARY=French_Algeria.utf8 LC_NUMERIC=C
## [5] LC_TIME=French_Algeria.utf8
##
## time zone: Asia/Riyadh
## tzcode source: internal
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] stringi_1.8.9 ggplot2_4.0.3
##
## loaded via a namespace (and not attached):
## [1] vctrs_0.7.3 cli_3.6.6 knitr_1.51 rlang_1.3.0
## [5] xfun_0.60 otel_0.2.0 generics_0.1.4 S7_0.2.2
## [9] jsonlite_2.0.0 labeling_0.4.3 glue_1.8.1 htmltools_0.5.9
## [13] gridExtra_2.3.1 sass_0.4.10 scales_1.4.0 rmarkdown_2.31
## [17] grid_4.6.1 tibble_3.3.1 evaluate_1.0.5 jquerylib_0.1.4
## [21] fastmap_1.2.0 yaml_2.3.12 lifecycle_1.0.5 compiler_4.6.1
## [25] codetools_0.2-20 dplyr_1.2.1 RColorBrewer_1.1-3 pkgconfig_2.0.3
## [29] rstudioapi_0.19.0 farver_2.1.2 digest_0.6.39 R6_2.6.1
## [33] tidyselect_1.2.1 pillar_1.11.1 magrittr_2.0.5 bslib_0.12.0
## [37] withr_3.0.3 tools_4.6.1 gtable_0.3.6 cachem_1.1.0