Executive Summary

The goal of this project is to develop a text prediction algorithm that predicts the next word based on words already entered by a user. This milestone report provides an exploratory analysis of the English SwiftKey text data from blogs, news, and Twitter. Basic characteristics of the three corpora are summarized, including file size, number of lines, and approximate word counts. Several simple visualizations are also presented. Finally, a plan for developing the prediction algorithm and Shiny application is described.

Data Loading

The SwiftKey dataset contains text collected from blogs, news articles, and Twitter.

options(stringsAsFactors = FALSE)

zip_url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
zip_file <- "Coursera-SwiftKey.zip"

if (!file.exists(zip_file)) {
  download.file(zip_url, zip_file, mode = "wb")
}

if (!dir.exists("final")) {
  unzip(zip_file)
}

blogs_file <- "final/en_US/en_US.blogs.txt"
news_file <- "final/en_US/en_US.news.txt"
twitter_file <- "final/en_US/en_US.twitter.txt"

Basic Summary of the Three Files

blogs <- readLines(
  blogs_file,
  encoding = "UTF-8",
  skipNul = TRUE,
  warn = FALSE
)

news <- readLines(
  news_file,
  encoding = "UTF-8",
  skipNul = TRUE,
  warn = FALSE
)

twitter <- readLines(
  twitter_file,
  encoding = "UTF-8",
  skipNul = TRUE,
  warn = FALSE
)

count_words <- function(x) {
  sum(lengths(strsplit(trimws(x), "\\s+")))
}

summary_table <- data.frame(
  Source = c("Blogs", "News", "Twitter"),
  File_Size_MB = round(
    c(
      file.info(blogs_file)$size,
      file.info(news_file)$size,
      file.info(twitter_file)$size
    ) / 1024^2,
    2
  ),
  Lines = c(
    length(blogs),
    length(news),
    length(twitter)
  ),
  Words = c(
    count_words(blogs),
    count_words(news),
    count_words(twitter)
  )
)

summary_table
##    Source File_Size_MB   Lines    Words
## 1   Blogs       200.42  899288 37334131
## 2    News       196.28 1010242 34372530
## 3 Twitter       159.36 2360148 30373583

The three sources differ substantially in both file size and the number and length of text entries. Twitter contains many short lines, whereas blogs and news articles generally contain longer text passages.

Exploratory Analysis

To illustrate differences in text length, a random sample of lines was taken from each source.

set.seed(12345)

sample_lines <- function(x, n = 10000) {
  sample(x, min(n, length(x)))
}

blogs_sample <- sample_lines(blogs)
news_sample <- sample_lines(news)
twitter_sample <- sample_lines(twitter)

words_per_line <- function(x) {
  lengths(strsplit(trimws(x), "\\s+"))
}

blog_lengths <- words_per_line(blogs_sample)
news_lengths <- words_per_line(news_sample)
twitter_lengths <- words_per_line(twitter_sample)

Distribution of Words per Line

boxplot(
  blog_lengths,
  news_lengths,
  twitter_lengths,
  names = c("Blogs", "News", "Twitter"),
  outline = FALSE,
  ylab = "Words per line",
  main = "Text Length Across the Three Data Sources"
)

The plot shows clear differences in text structure. Twitter entries tend to be shorter, while blog and news entries generally contain more words per line.

Most Frequent Words

For a basic examination of vocabulary, common words were identified from a sample of the combined corpus.

combined_sample <- c(
  blogs_sample,
  news_sample,
  twitter_sample
)

clean_text <- tolower(combined_sample)

clean_text <- gsub(
  "[^a-z' ]",
  " ",
  clean_text
)

words <- unlist(
  strsplit(clean_text, "\\s+")
)

words <- words[nchar(words) > 0]

word_freq <- sort(
  table(words),
  decreasing = TRUE
)

head(word_freq, 20)
## words
##   the    to   and     a    of    in     i  that    is   for    it   you    on 
## 44173 24098 22496 21425 18887 15035 14019  9793  9214  9084  8452  7038  6835 
##  with   was  this    at    as  have    my 
##  6319  5842  4789  4753  4681  4656  4623
top_words <- head(word_freq, 20)

barplot(
  rev(top_words),
  names.arg = rev(names(top_words)),
  horiz = TRUE,
  las = 1,
  main = "Twenty Most Frequent Words in a Sample of the Corpus",
  xlab = "Frequency"
)

Many of the most common terms are ordinary English function words such as articles, conjunctions, and pronouns. This suggests that preprocessing decisions, including treatment of punctuation, capitalization, profanity, and very common words, will be important when constructing the prediction model.

Interesting Findings

Several features of the dataset are relevant for the final prediction model.

First, the three data sources have different text-length distributions. Twitter consists primarily of short conversational messages, whereas blogs and news contain longer text segments.

Second, the distribution of word frequencies is highly uneven. A relatively small number of words occur very frequently, while many words appear only rarely.

Third, because the final application must make predictions quickly, it will not be practical to search the complete corpus every time the user enters text. The corpus will therefore need to be converted into compact frequency tables.

Plan for the Prediction Algorithm

The final prediction algorithm will use an n-gram language model. The text will first be cleaned and tokenized. Frequency tables for unigrams, bigrams, trigrams, and possibly four-grams will then be constructed.

When a user enters text, the algorithm will:

  1. identify the most recent words entered by the user;
  2. search the highest-order available n-gram table for matching sequences;
  3. return the most frequent possible next word;
  4. use a backoff strategy when a higher-order match is unavailable.

The model will be optimized to reduce memory usage and prediction time while retaining reasonable predictive accuracy.

Plan for the Shiny Application

The final Shiny application will contain a text input field in which the user can type a phrase. The application will process the phrase using the prediction model and display one or more likely next words. The interface will be kept simple so that predictions can be returned quickly and clearly.

Conclusion

The exploratory analysis confirms that the SwiftKey dataset contains a large and diverse collection of English text. The three sources differ in their structure and text length, but together they provide sufficient material for constructing a next-word prediction model. The next stage of the project will focus on text preprocessing, n-gram construction, model evaluation, and implementation of the prediction algorithm in a Shiny application.