Overview

This report summarises the exploratory analysis of the HC Corpora dataset — a large collection of English text from three sources: Twitter, News, and Blogs. The goal is to understand the structure of the data and lay the groundwork for building a next-word prediction algorithm.

1. The Data

The corpus was downloaded from the Coursera / SwiftKey dataset. It contains three plain-text files in English.

Summary statistics for each corpus file
Source Size (MB) Lines Avg Line Length
Twitter Twitter 167.1 2,360,148 71
News News 205.8 1,010,242 204
Blogs Blogs 210.2 899,288 234

Key observations:

  • Twitter has the most lines but the shortest entries (character limit per tweet).
  • Blogs have the longest average line length — more detailed, longer-form writing.
  • Together, the three sources provide over 100 million words of training data.

2. Word Frequency

After cleaning the text (lowercasing, removing punctuation and numbers), we built a frequency table of all unique words across a 5% sample of the corpus.

Top 20 Most Frequent Words

The most common words are stop words (the, and, to, a, of). These are kept in the model because they are essential for natural language prediction — “I want to”, “thanks a lot”.

Word Frequency Distribution

This follows Zipf’s Law: a small number of words appear very frequently, while most words are rare. The top 100 words cover roughly 50% of all word occurrences in the corpus.

3. N-gram Analysis

N-grams are sequences of consecutive words. They form the backbone of our prediction model — if someone types “happy new”, the most common next word in “happy new ___” trigrams is the prediction.

Top 15 Bigrams (2-word sequences)

Top 15 Trigrams (3-word sequences)

Common trigrams like “one of the” and “a lot of” reflect natural patterns in English that the model will learn to predict reliably.

4. Vocabulary Coverage

How many unique words are needed to cover most of the text?

Coverage Target Unique Words Needed
50% of all text 139
90% of all text 6,954

Only 6,954 unique words are needed to cover 90% of all text. This allows us to build an efficient, compact model that doesn’t require every word in the dictionary.

5. Plan for Prediction Algorithm & App

The Algorithm: N-gram with Stupid Backoff

The model predicts the next word by searching pre-built frequency tables:

Step What it does
4-gram Look up the last 3 words typed
3-gram If no match, look up the last 2 words
2-gram If no match, look up the last 1 word
Fallback Return the most common word in the corpus

This is called Stupid Backoff — it is fast, memory-efficient, and works well in practice for common English phrases.

The Shiny App

  • User types a phrase (e.g. “I would like to”)
  • Presses Predict
  • The app returns the predicted next word instantly (< 5ms)

Live app: https://rameez-captone-project.shinyapps.io/NextWordPrediction/

Next Steps

  • Increase training sample from 5% → 10% for better coverage of rare phrases
  • Return the top 3 candidate words, not just one
  • Evaluate accuracy on held-out test sentences