Executive Summary

This milestone confirms that the SwiftKey capstone data have been downloaded, loaded, sampled, and explored successfully. The analysis focuses on the English (en_US) blogs, news, and Twitter files because the first version of the prediction app will target English next-word prediction.

The project is on track for the final Shiny app. The exploratory analysis shows the expected natural-language pattern: a small number of words are extremely common, while most phrases are rare. Based on that finding, the planned app will use a compact n-gram model with backoff, so it can return predictions quickly without storing every possible phrase.

Data Loaded

The English corpus contains three large plain-text files. Each row is one text entry.

Source	File	Size (MB)	Lines
blogs	en_US.blogs.txt	200.42	899,288
news	en_US.news.txt	196.28	1,010,242
twitter	en_US.twitter.txt	159.36	2,360,148

Together, these files contain 4,269,678 lines of text and occupy about 556.06 MB on disk. The files were read in chunks rather than loaded all at once, which is important for keeping memory use manageable.

Exploratory Sample

For development, I used a reproducible 1% sample. This is large enough to reveal the main frequency patterns while remaining fast enough for iteration.

Source	Sampled lines	Word tokens	Mean tokens per line
blogs	8,955	370,062	41.3
news	10,177	350,937	34.5
twitter	23,612	304,395	12.9

The sample contains 42,744 lines and 1,025,394 word tokens.

Token volume by source in the 1% English sample.

Major Findings

Word Frequencies Are Highly Concentrated

A small dictionary covers a large share of the language. In the sample, only 159 words cover 50% of all word instances, and 7,661 words cover 90%.

Rank	Word	Count	Cumulative share
1	the	47,277	4.6%
2	to	27,389	7.3%
3	a	24,354	9.7%
4	and	23,773	12.0%
5	of	19,839	13.9%
6	i	16,608	15.5%
7	in	16,556	17.1%
8	for	11,106	18.2%
9	is	10,487	19.3%
10	that	10,269	20.3%

The most frequent words in the English sample.

Word Pairs And Triples Are Sparse

Word pairs and triples are much harder to cover than individual words. This matters because longer phrases provide better context, but they also create many rare combinations.

N-gram	Total instances	Unique terms	Terms appearing once	Singleton share	Terms for 50% coverage	Terms for 90% coverage
1-gram	1,025,394	56,974	29,935	52.5%	159	7,661
2-gram	982,650	460,519	368,008	79.9%	36,949	362,254
3-gram	940,043	795,571	738,899	92.9%	325,550	701,567

The key implication is that the app should not rely only on exact long phrases. It needs a fallback strategy for cases where a typed phrase was not seen in training.

Cumulative coverage by dictionary size for words, 2-grams, and 3-grams.

Common Phrases Are Useful But Limited

The most common 2-grams and 3-grams are normal English phrase fragments. They are useful for prediction, but even the most common phrase covers only a small fraction of all phrase instances.

Most frequent 2-grams
Rank	2-gram	Count
1	in the	4,198
2	of the	4,127
3	to the	2,085
4	for the	2,041
5	on the	1,914

Most frequent 3-grams
Rank	3-gram	Count
1	one of the	358
2	a lot of	303
3	thanks for the	233
4	to be a	195
5	some of the	155

Text Contains Noise

The corpus includes punctuation, numbers, encoding-sensitive characters, URLs, abbreviations, names, and occasional non-English content. In the English sample, most word instances were plain ASCII words, but there were still non-ASCII tokens and numeric tokens.

Category	Unique terms	Token instances	Share of instances
ascii_word	50,568	994,725	97.0%
contains_non_ascii	3,677	11,114	1.1%
number_or_contains_digit	2,729	19,555	1.9%

This means the final app should normalize text before prediction and filter words that should not be suggested.

Prediction Algorithm Plan

The planned model is a compact n-gram predictor:

Clean and tokenize user input.
Use the last three words as context when possible.
Search the 4-gram table for likely next words.
If the phrase is unseen, back off to the last two words, then one word.
If no context is found, use the most frequent words as a final fallback.
Remove blocked or profane words before returning suggestions.

This approach balances accuracy, memory use, and speed. It avoids storing impossible dense tables of all word combinations and instead stores only useful observed transitions.

The current prototype already builds this kind of model.

Context size	Stored transitions	Contexts
0 previous words	5	1
1 previous words	23,538	9,482
2 previous words	36,270	22,355
3 previous words	13,363	10,905

The prototype model uses a 20,000-word vocabulary, stores the top 5 next-word candidates per context, and occupies about 64.15 MB in memory. The saved model file is much smaller because it is compressed on disk.

Prototype model size, predictions, and held-out accuracy.

Initial held-out evaluation:

Evaluated tokens	Top-1 accuracy	Top-3 accuracy	Top-5 accuracy	Unknown target share
10,000	12.84%	22.63%	27.88%	4.91%

Example predictions:

Input	Top prediction	Context used
i love	you	i love
thanks for the	follow	thanks for the
one of the	most	one of the
this is a	good	this is a
data science	and	science

Shiny App Goal

The final Shiny app will be a simple next-word prediction tool. A user will type a phrase, and the app will return the top predicted next words. The app should feel fast, so the prediction model will be built offline and loaded as a compact .rds file when the app starts.

The planned app will include:

A text input box for the user phrase.
A small list of top next-word predictions.
Basic explanation of which context length was used.
A profanity filter for displayed predictions.
A compact model file suitable for deployment on shinyapps.io.

Next Steps

The remaining work is to improve the model quality and prepare the Shiny product:

Tune vocabulary size, minimum n-gram count, and number of stored predictions per context.
Add a real profanity list and final output filter.
Test runtime with system.time() and memory with object.size().
Improve smoothing and backoff, possibly with interpolation.
Build the Shiny interface around the saved model.
Validate the app with unseen phrases and realistic user inputs.

SwiftKey Capstone Milestone Report

Data Science Capstone

August 06, 2026