Introduction

This report presents an exploratory analysis of the SwiftKey dataset used in the Johns Hopkins Data Science Capstone Project. The dataset contains English text collected from blogs, news articles, and Twitter posts. The objective of this report is to understand the dataset before developing a predictive text model and a Shiny application for next-word prediction.

Loading Required Libraries

library(tm)
library(stringi)
library(ggplot2)
library(dplyr)
library(wordcloud)
library(SnowballC)
library(knitr)
library(RColorBrewer)
library(slam)

Loading the Dataset

blogs <- readLines("final/en_US/en_US.blogs.txt",
                   encoding="UTF-8",
                   skipNul=TRUE)

news <- readLines("final/en_US/en_US.news.txt",
                  encoding="UTF-8",
                  skipNul=TRUE)

twitter <- readLines("final/en_US/en_US.twitter.txt",
                     encoding="UTF-8",
                     skipNul=TRUE)

Summary Statistics

summary_data <- data.frame(
  File = c("Blogs","News","Twitter"),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  ),
  Words = c(
    sum(stri_count_words(blogs)),
    sum(stri_count_words(news)),
    sum(stri_count_words(twitter))
  ),
  Size_MB = c(
    round(file.info("final/en_US/en_US.blogs.txt")$size/1024^2,2),
    round(file.info("final/en_US/en_US.news.txt")$size/1024^2,2),
    round(file.info("final/en_US/en_US.twitter.txt")$size/1024^2,2)
  )
)

knitr::kable(summary_data,
             caption="Summary Statistics of the SwiftKey Dataset")
Summary Statistics of the SwiftKey Dataset
File Lines Words Size_MB
Blogs 899288 37546806 200.42
News 1010206 34761151 196.28
Twitter 2360148 30096690 159.36

Dataset Visualization

ggplot(summary_data,
       aes(x = File,
           y = Words,
           fill = File)) +
  geom_col() +
  labs(title = "Total Words in Each Dataset",
       x = "Dataset",
       y = "Word Count") +
  theme_minimal()

Observations

The Blogs dataset contains approximately 899 thousand lines and more than 37 million words, making it the largest source of text by word count.

The News dataset contains approximately 1.01 million lines and over 34 million words.

The Twitter dataset contains the highest number of lines (over 2.3 million), but fewer total words because tweets are much shorter than blog posts or news articles.

These three datasets provide a large and diverse collection of English text that is suitable for building a predictive language model.

Future Work

The next phase of the project will involve preprocessing the text by converting it to lowercase, removing punctuation, numbers, extra spaces, and stop words.

After preprocessing, unigram, bigram, and trigram language models will be created to predict the next word based on the previously typed words.

A backoff strategy will be used whenever an exact word sequence is unavailable.

Finally, the prediction model will be deployed using a Shiny web application where users can enter text and receive predicted next words.

Conclusion

This exploratory analysis provided an overview of the SwiftKey dataset and confirmed that it contains sufficient text for building a predictive text model. The summary statistics and visualization demonstrate the size and diversity of the data. The next step is to clean the text, develop n-gram language models, and build an interactive prediction application.