title: “Exploratory Analysis of SwiftKey Dataset” author: “Vighnesh Pawar” date: “2026-09-02” output: html_document ———————

Introduction

The SwiftKey dataset contains text collected from blogs, news articles, and Twitter posts. The aim of this exploratory analysis is to understand the size and characteristics of the English-language text that will be used to develop a predictive text application.

Data Description

The English portion of the SwiftKey dataset contains text from three sources:

Each source is stored in a separate text file:

For this exploratory analysis, the complete English-language files were used.

Data Loading and Cleaning

The three files were loaded into R using their original text lines. For word-frequency analysis, the text was converted to lowercase and punctuation and other non-letter characters were removed. This allows words to be counted consistently.

blogs <- readLines(
  "final/en_US/en_US.blogs.txt",
  encoding = "UTF-8"
)

news <- readLines(
  "final/en_US/en_US.news.txt",
  encoding = "UTF-8"
)

twitter <- readLines(
  "final/en_US/en_US.twitter.txt",
  encoding = "UTF-8"
)

Basic Summary

The first step is to compare the number of lines in each source.

data_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  )
)

data_summary
##   Dataset   Lines
## 1   Blogs  899288
## 2    News   77259
## 3 Twitter 2360148

The three sources contain different amounts of text. Twitter contains a large number of short posts, while blogs and news contain longer pieces of text.

File Sizes

The file sizes provide another measure of the amount of text contained in each source.

file_sizes <- file.info(
  c(
    "final/en_US/en_US.blogs.txt",
    "final/en_US/en_US.news.txt",
    "final/en_US/en_US.twitter.txt"
  )
)$size

file_size_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Size_MB = round(file_sizes / (1024^2), 2)
)

file_size_summary
##   Dataset Size_MB
## 1   Blogs  200.42
## 2    News  196.28
## 3 Twitter  159.36

Longest Line

The length of the longest line in the three files was also calculated.

longest_line <- max(
  nchar(c(blogs, news, twitter))
)

longest_line
## [1] 40833

The longest line contains 40833 characters.

Number of Words

The following analysis calculates the total number of words and the number of unique words in each source.

First, the text is cleaned and converted into individual words.

blogs_clean <- tolower(blogs)
blogs_clean <- gsub("[^a-z']", " ", blogs_clean)
blog_words <- unlist(strsplit(blogs_clean, "\\s+"))
blog_words <- blog_words[blog_words != ""]

blog_freq <- sort(
  table(blog_words),
  decreasing = TRUE
)

news_clean <- tolower(news)
news_clean <- gsub("[^a-z']", " ", news_clean)
news_words <- unlist(strsplit(news_clean, "\\s+"))
news_words <- news_words[news_words != ""]

news_freq <- sort(
  table(news_words),
  decreasing = TRUE
)

twitter_clean <- tolower(twitter)
twitter_clean <- gsub("[^a-z']", " ", twitter_clean)
twitter_words <- unlist(strsplit(twitter_clean, "\\s+"))
twitter_words <- twitter_words[twitter_words != ""]

twitter_freq <- sort(
  table(twitter_words),
  decreasing = TRUE
)

The results can now be summarized in one table.

word_summary <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Total_Words = c(
    sum(blog_freq),
    sum(news_freq),
    sum(twitter_freq)
  ),
  Unique_Words = c(
    length(blog_freq),
    length(news_freq),
    length(twitter_freq)
  )
)

word_summary
##   Dataset Total_Words Unique_Words
## 1   Blogs    37537188       277558
## 2    News     2620534        78659
## 3 Twitter    29757976       331133

This table shows both the overall size of each text source and the variety of vocabulary found in it.

Average Words Per Line

The average number of words per line gives an indication of the typical length of the text in each source.

count_words <- function(x) {
  x <- trimws(x)
  
  if (x == "") {
    return(0)
  }
  
  length(strsplit(x, "\\s+")[[1]])
}

blog_words_per_line <- sapply(blogs, count_words)
news_words_per_line <- sapply(news, count_words)
twitter_words_per_line <- sapply(twitter, count_words)

average_words <- data.frame(
  Dataset = c("Blogs", "News", "Twitter"),
  Average_Words_Per_Line = round(
    c(
      mean(blog_words_per_line),
      mean(news_words_per_line),
      mean(twitter_words_per_line)
    ),
    2
  )
)

average_words
##   Dataset Average_Words_Per_Line
## 1   Blogs                  41.52
## 2    News                  34.22
## 3 Twitter                  12.87

Basic Plots

Number of Lines

The following plot compares the number of lines in the three sources.

barplot(
  data_summary$Lines,
  names.arg = data_summary$Dataset,
  main = "Number of Lines in Each Dataset",
  ylab = "Number of Lines"
)

Most Frequent Words in Blogs

The 20 most frequent words in the blog corpus are shown below.

blog_freq_df <- data.frame(
  word = names(blog_freq),
  frequency = as.numeric(blog_freq),
  row.names = NULL
)

top_blog_words <- head(blog_freq_df, 20)

barplot(
  top_blog_words$frequency,
  names.arg = top_blog_words$word,
  las = 2,
  main = "20 Most Frequent Words in the Blog Corpus",
  ylab = "Frequency"
)

Most Frequent Words in News

news_freq_df <- data.frame(
  word = names(news_freq),
  frequency = as.numeric(news_freq),
  row.names = NULL
)

top_news_words <- head(news_freq_df, 20)

barplot(
  top_news_words$frequency,
  names.arg = top_news_words$word,
  las = 2,
  main = "20 Most Frequent Words in the News Corpus",
  ylab = "Frequency"
)

Most Frequent Words in Twitter

twitter_freq_df <- data.frame(
  word = names(twitter_freq),
  frequency = as.numeric(twitter_freq),
  row.names = NULL
)

top_twitter_words <- head(twitter_freq_df, 20)

barplot(
  top_twitter_words$frequency,
  names.arg = top_twitter_words$word,
  las = 2,
  main = "20 Most Frequent Words in the Twitter Corpus",
  ylab = "Frequency"
)

Words Per Line

The distribution of words per line is shown below for each source.

hist(
  blog_words_per_line,
  breaks = 50,
  main = "Words Per Line in the Blog Corpus",
  xlab = "Number of Words",
  ylab = "Number of Lines"
)

hist(
  news_words_per_line,
  breaks = 50,
  main = "Words Per Line in the News Corpus",
  xlab = "Number of Words",
  ylab = "Number of Lines"
)

hist(
  twitter_words_per_line,
  breaks = 50,
  main = "Words Per Line in the Twitter Corpus",
  xlab = "Number of Words",
  ylab = "Number of Lines"
)

Exploratory Analysis

The three sources have different characteristics. Blogs and news articles contain longer pieces of text, while Twitter consists of many shorter individual posts. The number of unique words also shows that the three sources contain a large variety of vocabulary.

The word-frequency plots show that a small number of common words occur very frequently, while many other words occur less often. This is a typical feature of natural language and will be important when building a predictive text model.

The differences between the three sources are also important because people use language differently in blogs, news articles, and social media. A predictive text system should therefore be able to learn from all three types of text.

Interesting Findings

The exploratory analysis shows that the SwiftKey training data is large and contains a wide range of English vocabulary. Twitter provides a large number of short and informal messages, while blogs and news provide longer and more structured text.

The frequency analysis also shows that some words occur much more often than others. This suggests that word frequency can be useful when predicting the next word in a sentence.

Another important finding is that the text sources have different writing styles. Combining the three sources should therefore provide a more varied training set for the predictive text application.

Prediction Algorithm

The next stage of the project will use the cleaned text to develop a predictive text algorithm. The algorithm will use the words typed by a user to estimate the most likely next word.

Word frequencies and combinations of words will be used to make these predictions. The model will also need to handle words or combinations that were not observed in the training data.

The main goal will be to create a model that provides useful predictions while keeping memory usage and response time low enough for an interactive application.

Shiny App

The final project will include a Shiny application that allows a user to enter text and receive a prediction for the next word.

The application will provide a simple interface so that the prediction can be generated quickly without requiring the user to understand the underlying model.

Conclusion

The exploratory analysis provides an overview of the English SwiftKey dataset and shows clear differences between blogs, news articles, and Twitter posts. The datasets contain a large number of words and a wide range of vocabulary.

The word-frequency analysis shows that common words occur much more frequently than less common words. The differences in writing style between the sources will also be considered when developing the predictive text model.

The next step is to use these findings to build and evaluate an efficient predictive text algorithm.