1. Introduction

This report presents an exploratory data analysis of the text datasets provided for the SwiftKey text prediction project. The goal is to understand the basic characteristics of the data to inform the development of a predictive text algorithm and a Shiny application. The analysis focuses on three datasets: US blogs, US news, and US Twitter feeds.

2. Data Summary

A sample of 5,000 lines from each dataset was used for this analysis. The following table summarizes the key statistics.

Table 1: Sample Statistics (5,000 lines each)
Dataset Lines Total_Words Total_Characters Avg_Words_Per_Line Max_Line_Length
Blogs 5000 205555 1139725 41.11 2521
News 5000 170940 1013388 34.19 1373
Twitter 5000 63747 340150 12.75 140
Table 2: Full Dataset Statistics
Dataset File_Size_MB Total_Lines Max_Line_Length
Blogs 200 899288 40833
News 200 1010242 11385
Twitter 150 2360148 140

3. Data Visualizations

Visualizations help illustrate the structure and patterns within the text data.

3.1 Line Length Distribution

The density plot and boxplot below show the distribution of line lengths across the three datasets. Twitter lines are predictably much shorter, while blogs and news have a wider range.

3.2 Word Frequency

The most frequent words provide insight into common topics and language use.

3.3 Word Cloud

A word cloud provides a visual summary of the most prominent terms.

4. Interesting Findings

Several interesting patterns emerged from the analysis:

  1. Sentiment on Twitter: The word “love” appears approximately 4.32 times more frequently than the word “hate” in the Twitter sample.

  2. Biostats Reference: One tweet in the sample mentions “biostats”: “Not found in sample”.

  3. Exact Match: The famous quote “A computer once beat me at chess, but it was no match for me at kickboxing” appears exactly 0 time(s) in the Twitter data.

  4. Longest Line: The longest line in the sample comes from the Blogs dataset, with a length of 2521 characters.

  5. Twitter’s Limit: In the Twitter sample, approximately 100% of lines are within the 140-character limit.

5. Algorithm and Application Plan

5.1 Prediction Algorithm

The prediction algorithm will be based on an N-gram Language Model with a Stupid Backoff strategy. The development will proceed as follows:

  1. Data Preprocessing: Text will be cleaned by converting to lowercase, removing punctuation, numbers, and extra whitespace.

  2. N-gram Generation: We will build 1-gram (unigram) through 4-gram (fourgram) models from the training corpus.

  3. Prediction Strategy:

    • Given an input phrase, the model will take the last three words
    • It will first search in the 4-gram model
    • If no match is found, it will “back off” to the 3-gram model, then to the 2-gram model
    • As a final fallback, it will return the most frequent unigram

  4. Output: The system will return the top 3-5 most likely candidate words for the next word.

5.2 Shiny Application

A Shiny application will be developed to provide a user-friendly interface for the prediction model. The planned features include:

  • Input: A text box for users to type a phrase
  • Real-time Prediction: The app will automatically predict the next word as the user types
  • Output: Display the top 3-5 predicted words with confidence scores
  • User Feedback: A simple mechanism for users to indicate if predictions are correct

The interface will be clean and intuitive, designed for a non-technical audience.

6. Conclusion

This exploratory analysis has provided a solid understanding of the text datasets. The findings on data size, structure, and key patterns will directly inform the development of a robust and efficient text prediction algorithm. The subsequent Shiny application will make this model accessible to end-users. The next steps involve building the N-gram models and prototyping the Shiny app.

Report generated on 2026-09-04