Exploratory Analysis of the SwiftKey Dataset
Vishal Prasad
2026-08-15
Introduction
This report presents an exploratory analysis of the SwiftKey text datasets used for the prediction project. The goal of this milestone is to understand the structure and characteristics of the data and identify patterns that will be useful when developing a text prediction algorithm.
The analysis focuses on three sources of English text: blogs, news articles, and Twitter messages. The datasets differ substantially in size and writing style, providing a useful foundation for developing a model that can predict the next word entered by a user.
Data Overview
The three datasets were successfully downloaded and loaded into R. The following table summarizes their size in terms of number of lines, words, and characters.
| Dataset | Lines | Words | Characters |
|---|---|---|---|
| Blogs | 899288 | 79780901 | 206824505 |
| News | 1010206 | 74313644 | 203214543 |
| 2360148 | 65271462 | 162096241 |
Dataset Size
The three sources contain a substantial amount of text. Blogs contain approximately 79.8 million words, news contains approximately 74.3 million words, and Twitter contains approximately 65.3 million words.
Average Text Length
The average number of characters per line differs considerably between the three sources. Blog entries are longest on average, followed by news articles, while Twitter messages are substantially shorter.
| Dataset | Lines | Average_Characters |
|---|---|---|
| Blogs | 899288 | 229.99 |
| News | 1010206 | 201.16 |
| 2360148 | 68.68 |
Most Frequent Words
A sample of 50,000 lines from each source was combined and cleaned to examine the most common vocabulary. Common English stop words and obvious contraction fragments were removed.
| Word | Frequency |
|---|---|
| said | 14825 |
| will | 13648 |
| one | 13369 |
| can | 13119 |
| just | 11347 |
| like | 10749 |
| time | 9544 |
| get | 8561 |
| new | 8017 |
| now | 6917 |
| people | 6903 |
| year | 6680 |
| day | 6509 |
| first | 6437 |
| also | 6432 |
| good | 6355 |
| know | 6132 |
| two | 5853 |
| back | 5817 |
| love | 5364 |
Distribution of Text Length
The following histogram uses a sample of 10,000 lines from each source. It illustrates the differences in text length across blogs, news and Twitter.
Key Findings
Several useful characteristics were identified during the exploratory analysis:
- The three sources provide a large and diverse collection of English text.
- Blogs contain the largest number of words, while Twitter contains the largest number of individual lines.
- Blog entries have the greatest average text length at approximately 230 characters per line.
- News entries average approximately 201 characters per line.
- Twitter messages are much shorter, averaging approximately 69 characters per line.
- The differences in text length indicate that the eventual prediction system will need to account for different writing styles and text formats.
Plans for the Prediction Algorithm
The next stage of the project will focus on developing a statistical language model that predicts the next word based on words already entered by the user.
The cleaned text will be used to identify common word sequences. The model will initially consider relatively short sequences of words and estimate which words are most likely to follow a given sequence. The model will then be evaluated using held-out text to determine how accurately it predicts unseen examples.
Because the source data is large, sampling and efficient data structures will be important for keeping the model fast enough for practical use.
Plans for the Shiny Application
The final Shiny application will provide a simple interface where a user can enter a phrase and receive suggested next words.
The application will use the prediction model created from the training data and display a small number of likely suggestions. The main objective will be to provide useful predictions while keeping response time short and the interface easy to understand.
Conclusion
The exploratory analysis confirms that the SwiftKey datasets contain sufficient text and substantial variation in writing style to support development of a word-prediction system. The next stage will focus on building and evaluating the prediction model and integrating it into a Shiny application.