Draft for student review · 2026-10-07
The three official English SwiftKey files were downloaded and scanned in full. Together they contain 4,269,678 lines and approximately 101.22 million cleaned word tokens. Blogs have the most tokens, while Twitter has the most and shortest lines. A deterministic 1% sample supports exploratory plots and a separate 0.8%/0.2% split supports model development and validation. The final product uses a compact 1–4 gram backoff model.
| Source | Bytes | Lines | Cleaned tokens |
|---|---|---|---|
| Blogs | 210,160,014 | 899,288 | 37,509,522 |
| News | 205,811,889 | 1,010,242 | 34,042,817 |
| 167,105,338 | 2,360,148 | 29,663,285 |
URLs and @mentions are removed; text is lowercased; tokens contain letters and internal apostrophes. A small explicit profanity blocklist prevents common offensive terms from entering predictions.
The model counts unigrams through four-grams. It first searches the last three words, then backs off to two words, one word, and global frequency. Tables keep five candidates per context for a small and responsive deployment.
On 193,477 held-out tokens, top-1 accuracy is 12.44% and top-3 accuracy is 20.07%. Improvements could use more training data, interpolated probabilities, broader profanity filtering, and validation-tuned pruning.
The Shiny app accepts a phrase, applies the same tokenizer, and returns one predicted next word plus optional alternatives and the n-gram order used.