Introduction

The goal of this project is to develop a next-word prediction application using English-language text from the SwiftKey corpus. This milestone report summarizes the structure of the training data, explores common words and multi-word sequences, and identifies patterns that will inform the eventual prediction algorithm.

The analysis focuses on three sources of English text: blogs, news articles, and Twitter posts. Because the full corpus contains more than four million lines and over 100 million word instances, a representative sample was used for more computationally intensive text analyses.

Data and Sampling

The English corpus consists of three text sources with substantially different numbers and lengths of documents.

Number of lines included in the 10% sample
Source Sampled Lines
Blogs 89,928
News 101,024
Twitter 236,014

Twitter contains the largest number of text lines, whereas blogs contain the largest approximate number of words. This reflects substantial differences in document length across the three sources: Twitter posts tend to be short, while blog entries are considerably longer.

Sampling

To reduce computational demands while maintaining representation from all three sources, a 10% random sample was selected independently from the blogs, news, and Twitter datasets. A fixed random seed was used to ensure that the sample is reproducible.

Number of lines included in the 10% sample
Source Sampled Lines
Blogs 89,928
News 101,024
Twitter 236,014

Before constructing word-frequency and n-gram measures, the sampled text was standardized to reduce noise. Text was converted to lowercase, web links and Twitter usernames were removed, numbers and most punctuation were excluded, and excess whitespace was removed. Apostrophes were retained so that common contractions such as “don’t” and “I’m” remained intact.

Stop words were intentionally retained because function words such as “the,” “to,” and “of” provide important contextual information for next-word prediction. Stemming and lemmatization were also avoided so that the model preserves the word forms that users are likely to type.

Exploratory Analysis

Word Frequency

After preprocessing and tokenization, the 10% sample contained 10,074,221 word tokens representing 168,609 unique word types.

A small number of common function words dominate the corpus. The most frequent words include “the,” “to,” “and,” “a,” and “of,” which is expected in general English text and reinforces the importance of retaining stop words for prediction.

Vocabulary Coverage

The vocabulary shows a strong long-tail pattern. The 143 most frequent words account for 50% of all word tokens, while 7,197 words account for 90%. Although the sample contains more than 168,000 unique word types, a relatively small subset therefore accounts for most observed word usage.

Frequent Word Sequences

Common bigrams such as “of the,” “in the,” and “to the” demonstrate strong dependencies between adjacent words, which can be used to generate candidate next-word predictions.

Frequent three-word sequences include “one of the,” “a lot of,” “going to be,” and “I want to.” These patterns provide more context than individual words or bigrams and are therefore potentially useful for higher-order prediction.

N-gram Sparsity

Summary of unigram, bigram, and trigram distributions
N-gram Total Instances Unique Types Average Frequency
Unigram 10,074,221 168,609 59.75
Bigram 9,647,476 2,605,017 3.70
Trigram 9,223,275 6,234,747 1.48

As n-gram order increases, the number of unique sequences rises sharply while the average frequency of each sequence decreases. Unigrams occurred an average of 59.75 times, compared with 3.70 for bigrams and only 1.48 for trigrams. In addition, approximately 88% of unique trigrams occurred only once in the sampled corpus.

This sparsity means that many valid word sequences will not be observed during training. A prediction algorithm therefore cannot rely exclusively on high-order n-grams.

Prediction Algorithm and Shiny App Plan

The final prediction algorithm will use an n-gram-based backoff strategy. When a user enters text, the model will first attempt to identify predictions based on the longest available context. For example, predictions based on the preceding two or three words will be preferred when the corresponding higher-order n-grams have been observed. If no suitable match exists, the model will back off to shorter word sequences and ultimately to unigram frequencies.

Because the exploratory analysis shows substantial sparsity among higher-order n-grams, the final model will also consider pruning very rare sequences to reduce memory requirements and improve prediction speed. The model will balance predictive usefulness, storage size, and runtime.

The final Shiny application will allow a user to enter a phrase and will return one or more likely next-word predictions. The interface will be designed to remain simple and responsive while the prediction algorithm operates in the background.