Executive summary

This report covers the first stage of building a predictive text application — software that guesses the next word you are about to type, the way a phone keyboard does.

The raw material is three large collections of English text scraped from blogs, news sites and Twitter. Before any prediction can happen, we need to know what is actually in that text: how big it is, how words are distributed, and which word sequences are common enough to be worth remembering.

The three headline findings are:

  1. The data is large but very repetitive. Roughly 4.3 million lines and 102 million words, yet a few thousand distinct words account for most of the text.
  2. A small vocabulary goes a long way. Just 140 words cover half of all word occurrences, and about 6,900 words cover 90%. The remaining hundreds of thousands of words are rare typos, names and slang — nearly half of them appear exactly once.
  3. The three sources read very differently. Tweets are short and informal, news is long and formal, blogs sit in between. Mixing all three gives the model the broadest coverage of how people really write.

These facts are good news: they mean the prediction model can be made small and fast enough to run in a web app without storing the whole corpus.

1. The data

Table 1. Basic summary of the three en_US files.
Source Size (MB) Lines Words Characters Longest line (chars) Words per line (mean)
blogs 200.4 899,288 37,334,131 206,824,505 40,833 41.5
news 196.3 1,010,242 34,372,530 203,223,159 11,384 34.0
twitter 159.4 2,360,148 30,373,583 162,096,241 140 12.9

Three sources, three very different shapes. Twitter has by far the most lines but the shortest ones — the old 140-character limit is clearly visible in the “longest line” column. Blogs contain a single entry over 40,000 characters long.

Reading Figure 1: tweets cluster around 10–20 words and stop hard. News and blog entries have a long tail of much longer passages. A prediction model trained only on news would be poorly suited to short, informal typing — which is exactly what users of a phone keyboard do — so we keep all three.

2. Working with a sample

The full corpus does not need to be processed to understand it. We draw a 2% random sample of lines from each file, which keeps the analysis representative while letting the whole report run in a couple of minutes.

Table 2. Size of the working sample (2% of the corpus).
Sampled lines Sampled words
85,705 2,066,319

Before counting anything we clean the text: strip URLs, Twitter handles and hashtags, drop numbers and punctuation, and lower-case everything. Profanity is removed so the finished app never suggests an offensive word.

Note: common words such as “the”, “and” and “to” are kept. In most text analysis they are discarded as noise, but here they are the point — a next-word predictor must be able to suggest them.

3. Which words and phrases dominate?

The patterns are exactly what everyday English looks like: the, to, and, a at the top; of the, in the, for the as pairs; thanks for the, one of the, a lot of as triples. Common conversational phrases such as “thanks for the follow” come straight from the Twitter portion.

4. How much vocabulary do we actually need?

This is the single most important question for keeping the app small.

Table 3. Vocabulary needed to cover a given share of the text.
Coverage of all word occurrences Distinct words required
50% 140
75% 1,410
90% 6,858
95% 15,358
99% 50,444

The curve rises almost vertically and then flattens. In plain terms: we can throw away the overwhelming majority of the vocabulary and still understand almost every sentence a user types. Roughly 6,858 words cover 90% of everything written. That single fact is what makes a phone-sized model possible.

A related finding: about 48% of distinct words appear only once in the sample — misspellings, usernames, one-off slang. They carry almost no predictive value and will be dropped.

5. Interesting findings

  • Repetition is the model’s friend. Fewer than 200 words make up half of all English text here, so most predictions come from a very small set of candidates.
  • Source matters. Twitter contributes short, conversational patterns (“thanks for the”, “looking forward to”), news contributes formal constructions (“according to the”, “said in a statement”). Combining them broadens coverage.
  • The long tail is mostly noise. Dropping words seen only once removes the bulk of the vocabulary while costing very little accuracy.
  • Longer phrases get rare fast. Three-word phrases are far less frequent than two-word ones, which tells us the model must have a fallback plan when it has never seen a particular phrase — covered below.

6. Plan for the prediction algorithm and the app

The idea in one sentence: given the last few words you typed, look up which word most often came next in the corpus, and offer that.

How it will be built:

  1. Store counts, not text. Build tables of the most frequent 2-, 3- and 4-word sequences with their next-word counts. The original 550 MB of text is never shipped — only these compact tables.
  2. Shrink aggressively. Drop sequences seen only once, keep only the top few candidate next-words per phrase, and store everything as integer-indexed data.table objects. Target: under ~100 MB in memory, well within the free shinyapps.io limit.
  3. Back off when the phrase is unfamiliar. If the last three words have never been seen together, try the last two; if that fails, the last one; if all else fails, fall back to the most common words overall. A stupid backoff scheme (penalising each fallback step by a fixed factor) is used because it is accurate enough and far cheaper to compute than full smoothing.
  4. Never return nothing. The fallback chain guarantees a suggestion for any input, including words the model has never encountered.

How we will know it works: the corpus is split into training and held-out test portions. On the held-out text we measure top-1 accuracy (was the actual next word our first guess?) and top-3 accuracy (was it in our top three?), plus the average response time, which must stay comfortably under a tenth of a second.

The finished app will be a single Shiny page: a text box where the user types, three suggested next words shown as clickable buttons, and a small panel showing how confident the model is. Simple enough that no explanation is needed to use it.

Code for this report is available on request; all analysis was performed in R using the quanteda text-analysis package.