Executive Summary

This report presents an exploratory data analysis of the English text corpora (Blogs, News, and Twitter) provided by SwiftKey. The goal is to outline data dimensions, examine word frequency patterns, and establish the technical roadmap for the upcoming next-word prediction algorithm and interactive Shiny web application.

1. Corpus Summary Statistics

To evaluate the scope of the raw corpora, basic structural metrics were extracted across all three data sources: ### Key Findings * Twitter entries are significantly shorter due to character constraints, yielding fragmented syntax. * Blogs and News exhibit richer multi-word combinations, making them suitable baselines for sentence structure modeling.

2. Word Frequency Analysis

Natural language follows Zipf’s Law, where a small set of words dominates total occurrences. ### Key Findings * Twitter entries are significantly shorter due to character constraints, yielding fragmented syntax. * Blogs and News exhibit richer multi-word combinations, making them suitable baselines for sentence structure modeling.

Common stop words (e.g., the, to, and) account for the bulk of occurrences. While frequently filtered in classification tasks, they are retained here because valid text autocompletion requires predicting prepositions, conjunctions, and articles.

3. Architecture for Prediction Algorithm & Shiny App

The Algorithm (N-gram Backoff Model)

  1. Lookup Structure: Build indexed 4-gram, 3-gram, and 2-gram frequency tables using prefix-keyed structures.
  2. Backoff Mechanics:
    • Attempt to match the preceding 3 words in the 4-gram table.
    • If no match is found, back off to the 3-gram table (\(\times 0.4\) penalty multiplier), then to the 2-gram table.
    • Default to the unigram baseline ("the") for out-of-vocabulary inputs.
  3. Memory Optimization: Filter out singletons (phrases appearing only once). This slashes memory footprint by over 70%, ensuring low latency on cloud containers.

Shiny Interface Design

  • Clean, single-field text box that evaluates inputs in real time.
  • Displays the primary predicted next word alongside the top 2 alternative suggestions.