Exploratory Data Analysis of the SwiftKey Dataset
Akshat Sinha
November 2026
1. Introduction
The objective of this project is to explore a large collection of text data obtained from three different sources: blogs, news articles and Twitter. This analysis is the initial stage of developing a predictive text algorithm and an interactive Shiny application.
The purpose of this exploratory analysis is to understand the structure, size and characteristics of the datasets before developing the prediction model.
2. Dataset Description
The dataset consists of three English-language text files:
- Blogs: Text collected from blog posts.
- News: Text collected from news articles.
- Twitter: Short messages collected from Twitter.
Together, these datasets provide different styles of everyday language. Blogs and news generally contain longer passages, whereas Twitter contains shorter, conversational text.
3. Data Loading and Analysis
The three datasets were downloaded and loaded successfully into R.
The analysis examined: - Total number of lines. - Total number of words. - Total number of characters. - Average words per line. - Average characters per line. - Maximum number of words per line.
The files were processed in chunks to reduce memory consumption.
4. Dataset Summary Statistics
The following table presents the overall characteristics of the three datasets.
| Dataset | Lines | Words | Characters | Average_Words_Per_Line | Average_Characters_Per_Line | Maximum_Words_Per_Line | |
|---|---|---|---|---|---|---|---|
| Blogs | Blogs | 899288 | 37334131 | 206824505 | 41.52 | 229.99 | 6630 |
| News | News | 1010242 | 34372530 | 203223159 | 34.02 | 201.16 | 1792 |
| 2360148 | 30373543 | 162096031 | 12.87 | 68.68 | 47 |
Dataset File Sizes
| Dataset | File_Size_MB |
|---|---|
| Blogs | 200.42 |
| News | 196.28 |
| 159.36 |
5. Exploratory Visualisation
Histograms were generated to understand the distribution of words and characters across individual lines in each dataset.
5.1 Distribution of Words per Line
5.2 Distribution of Characters per Line
6. Key Findings
The exploratory analysis highlights several differences between the three sources of text.
- Dataset size: Twitter contains the largest number of lines, followed by news and blogs.
- Writing patterns: Blogs and news contain longer passages, while Twitter generally contains shorter messages.
- Variation in text: The distributions of words and characters per line demonstrate differences in writing length across the three sources.
- Language diversity: The three datasets provide different writing styles and vocabulary patterns, making them useful for building a general-purpose predictive text model.
These findings provide an initial understanding of the data and will help guide the next stages of the project.
7. Future Project Development
The next stage will focus on building a predictive text algorithm.
The proposed approach includes:
- Cleaning and preprocessing the text.
- Removing unnecessary characters and handling punctuation.
- Tokenising the text into individual words.
- Developing unigram, bigram and trigram models.
- Identifying frequently occurring word combinations.
- Using these patterns to predict the next word based on user input.
A Shiny application will subsequently be developed to provide an interactive interface. Users will be able to enter a word or phrase and receive suggestions for the next likely word.
The model will be evaluated based on prediction accuracy, response time and usability.
8. Conclusion
This exploratory data analysis successfully loaded and examined the three SwiftKey text datasets.
Basic statistical summaries and visualisations helped identify differences in dataset size, writing length and text characteristics.
The findings establish a foundation for developing a predictive text algorithm and an interactive Shiny application in the next phase of the Johns Hopkins Data Science Capstone.