Executive Summary

This report is the first milestone of a project to build a smart keyboard: an app that predicts the next word you are about to type, similar to the suggestions on a phone keyboard. To teach the app how people actually write, we use a large collection of real English text from three sources — blogs, news articles, and Twitter posts.

In this report we:

  1. Load the three data files and confirm they were read in successfully.
  2. Summarize their size and basic characteristics.
  3. Explore which words and short phrases appear most often.
  4. Outline the plan for the prediction algorithm and the final Shiny app.

Key takeaway for the busy reader: the data is very large (over 4 million lines and ~100 million words), but a surprisingly small vocabulary does most of the work — roughly 303 unique words cover 50% of all text, and about 8,500 words cover 90%. This means we can build a fast, accurate predictor from a modest sample of the data without needing the entire corpus.

1. The Data

The data comes from the HC Corpora dataset provided by SwiftKey. We use the English (en_US) files.

Table 1: Basic summary of the three source files
File Size (MB) Lines Words Longest line (chars) Avg words per line
Blogs 200.4 899,288 37,581,907 40,833 41.8
News 196.3 1,010,242 34,858,293 11,384 34.5
Twitter 159.4 2,360,148 30,162,832 140 12.8

Each source has a distinct character: tweets are short (capped at 140 characters at the time of collection), blogs are the longest and most conversational, and news sits in between with more formal language. This diversity is useful — a good keyboard must handle both casual and formal writing.

How long are the lines?

As expected, Twitter lines cluster under ~30 words, while blog and news entries have long tails of much lengthier passages.

2. Sampling and Cleaning

The full corpus is too large to process comfortably (and unnecessary, as we will see). We take a random 1% sample of each file (~40,000+ lines total), which is plenty to reveal the structure of the language. Before analysis, the text is cleaned:

  • converted to lowercase
  • URLs, Twitter handles, emails, and numbers removed
  • punctuation and extra whitespace stripped
  • profanity removed using a public bad-words list

3. What Words and Phrases Are Most Common?

The building blocks of our predictor are n-grams: sequences of 1, 2, or 3 consecutive words. If “thanks for the” is almost always followed by “follow” or “support” in real text, that is exactly what the keyboard should suggest.

Unsurprisingly, common function words (“the”, “to”, “and”) and everyday phrases (“of the”, “thanks for the”, “a lot of”) dominate. These frequency tables are, in essence, a first draft of the prediction engine.

How many words do we really need?

Interesting finding: just 303 unique words account for half of everything people write, and about 8,500 words cover 90%. The remaining hundreds of thousands of rare words contribute very little. This is great news: it means the final app can be small and fast without sacrificing much accuracy.

4. Interesting Findings So Far

  • The three sources differ in style. Twitter is short and informal (“thanks for the follow” is a top trigram); news uses more formal phrasing. Blending all three gives balanced predictions.
  • Language is highly repetitive. A tiny core vocabulary dominates, so a compact lookup table can power most predictions.
  • A 1% sample is representative. Frequency rankings stabilize quickly, so we do not need 100M words to train a useful model.
  • Cleaning matters. Raw text contains URLs, emojis, foreign characters, and profanity that must be stripped before modeling.

5. Plan for the Prediction Algorithm and Shiny App

The algorithm (in plain terms): when the user types a few words, the app looks up the last two or three words in pre-computed phrase tables and suggests the word that most often comes next in real text.

  1. Build frequency tables of 2-, 3-, and 4-word phrases from a larger sample.
  2. Use a “backoff” strategy (Stupid Backoff / Katz backoff): try to match the last 3 typed words; if that phrase is unseen, fall back to the last 2 words, then 1. Smoothing handles phrases never seen in training.
  3. Prune rare phrases so the tables stay small enough to run instantly in a web app.
  4. Measure accuracy and speed on held-out test text, and tune the trade-off.

The Shiny app: a simple web page with a text box. As the user types, the top 3 predicted next words appear as clickable buttons — just like a phone keyboard. It will be lightweight enough to respond in a fraction of a second.

Appendix: Reproducibility

Code is hidden above for readability; the full R Markdown source (including all code) is available on request / GitHub. Analysis used R with the tm, RWeka, stringi, and ggplot2 packages, with set.seed(1234) for reproducible sampling.