Executive summary
This report explores three English-language text collections supplied
for the SwiftKey next-word prediction project. The aim
is to understand the language data and define a practical plan for a
simple prediction app. The analysis covers 4,269,642 text
entries across blogs, news, and Twitter. The full datasets were
successfully read by R; a reproducible 1% random sample of lines
from each source was used for the visualizations.
1. Data overview
The files differ in both size and writing style. The table below
summarizes all lines in each file (not just the
sample).
Full-dataset summary; word counts use ICU word
boundaries.
| Blogs |
200.4 |
899,288 |
37,546,806 |
41.8 |
40,833 |
| News |
196.3 |
1,010,206 |
34,761,151 |
34.4 |
11,384 |
| Twitter |
159.4 |
2,360,148 |
30,096,690 |
12.8 |
140 |
Business implication: A typical Twitter entry has
12.8 words, whereas a typical blog entry has
41.8 words. The predictor therefore needs to handle
both short, conversational input and longer, more formal sentences.
2. Main exploratory findings
Entry lengths vary substantially
The histogram shows the distribution of words per text entry. To make
the typical entries easy to compare, entries longer than 100 words are
excluded from this chart only; they remain included in the full
statistics above.
Finding: Tweets are concentrated around shorter
lengths; blogs and news cover a broader range. A useful model should
learn from all three sources rather than optimizing for only one style
of writing.
Common words drive many predictions
The most common words in the combined sample are shown below. These
everyday words are important because the algorithm will often predict
short, frequent connecting words.
Finding: High-frequency words dominate the
vocabulary, but a useful predictor must also recognize word
combinations. For this reason, word sequences (n-grams)
are more informative than isolated word counts alone.
3. Plan for the prediction model and Shiny app
The next phase will be deliberately lightweight and focused on
reliable user experience:
- Prepare the text: Clean encoding and unnecessary
symbols, keep sentence boundaries, and build frequency tables for one-,
two-, three-, and four-word sequences.
- Predict the next word: Match the words the user has
just typed to the most likely continuation. Use shorter sequences as a
fallback when a longer match is unavailable.
- Validate performance: Hold out unseen text and
measure top-one and top-three accuracy, response speed, and model size.
Improve the balance between accuracy and speed before deployment.
- Build the Shiny interface: Provide a text input
box, three suggested next words, and a clear/reset
option. Predictions should update as the user types.
Success means delivering fast, understandable
predictions in a simple browser-based app, without requiring users to
know anything about data science. We will also assess how to manage rare
terms, informal spelling, and potentially inappropriate suggestions.
Data source: Coursera
/ SwiftKey Capstone dataset. All reported totals are calculated from
the three local English-language source files when this document is
rendered; the charts use a random 1% line sample per file. The English
datasets are the scope of this milestone.
