1. Introduction and Objective

This report is part of the Data Science Capstone project. The final goal of this capstone is to build a predictive text model (similar to SwiftKey’s keyboard prediction) and deploy it as a Shiny web application.

This milestone report demonstrates that:

  1. The data has been successfully downloaded and loaded.
  2. Basic summary statistics have been generated for the three English data sets (blogs, news, twitter).
  3. Interesting exploratory findings have been identified.
  4. A clear plan for the prediction algorithm and Shiny app is outlined.

2. Loading the Data

The dataset consists of text collected from three sources: blogs, news articles, and Twitter. We load the English (en_US) files.

library(stringi)
library(ggplot2)
library(dplyr)
library(tm)
library(tokenizers)
library(wordcloud)
library(knitr)
# Download and unzip data (run once)
url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
download.file(url, destfile = "Coursera-SwiftKey.zip")
unzip("Coursera-SwiftKey.zip")
blogs_path   <- "final/en_US/en_US.blogs.txt"
news_path    <- "final/en_US/en_US.news.txt"
twitter_path <- "final/en_US/en_US.twitter.txt"

con <- file(blogs_path, "r")
blogs <- readLines(con, encoding = "UTF-8", skipNul = TRUE)
close(con)

con <- file(news_path, "r")
news <- readLines(con, encoding = "UTF-8", skipNul = TRUE)
close(con)

con <- file(twitter_path, "r")
twitter <- readLines(con, encoding = "UTF-8", skipNul = TRUE)
close(con)

3. Basic Summary Statistics

We compute file size (MB), number of lines, number of words, and the length of the longest line for each of the three files.

file_size_mb <- function(path) round(file.info(path)$size / (1024^2), 2)

summary_df <- data.frame(
  File = c("en_US.blogs.txt", "en_US.news.txt", "en_US.twitter.txt"),
  Size_MB = c(file_size_mb(blogs_path), file_size_mb(news_path), file_size_mb(twitter_path)),
  Lines = c(length(blogs), length(news), length(twitter)),
  Words = c(sum(stri_count_words(blogs)),
            sum(stri_count_words(news)),
            sum(stri_count_words(twitter))),
  Longest_Line_Chars = c(max(nchar(blogs)), max(nchar(news)), max(nchar(twitter)))
)

kable(summary_df, caption = "Summary Statistics of the Three Data Sets")
Summary Statistics of the Three Data Sets
File Size_MB Lines Words Longest_Line_Chars
en_US.blogs.txt 200.42 899288 37546806 40833
en_US.news.txt 196.28 1010206 34761151 11384
en_US.twitter.txt 159.36 2360148 30096690 140 
ggplot(summary_df, aes(x = File, y = Lines, fill = File)) +
  geom_bar(stat = "identity") +
  labs(title = "Number of Lines per Data Set", y = "Number of Lines", x = "") +
  theme_minimal() +
  theme(legend.position = "none")

ggplot(summary_df, aes(x = File, y = Words, fill = File)) +
  geom_bar(stat = "identity") +
  labs(title = "Number of Words per Data Set", y = "Number of Words", x = "") +
  theme_minimal() +
  theme(legend.position = "none")

Observation: The Twitter file has the most lines (short tweets), while the Blogs file has the longest individual lines (long-form writing), and the News file sits in between with more structured sentence lengths.

4. Sampling the Data

Since the full data sets are large (hundreds of MB), we work with a random sample (e.g., 1% of lines from each source) to keep processing fast during exploratory analysis. The final model will be trained on a larger, appropriately sized sample.

set.seed(1234)
sample_pct <- 0.01

sample_blogs   <- sample(blogs,   length(blogs)   * sample_pct)
sample_news    <- sample(news,    length(news)    * sample_pct)
sample_twitter <- sample(twitter, length(twitter) * sample_pct)

sample_data <- c(sample_blogs, sample_news, sample_twitter)
length(sample_data)
## [1] 42695

5. Cleaning the Data

Basic text cleaning: lowercasing, removing numbers, punctuation, extra whitespace, and profanity filtering (a simple bad-words list can be used with tm::removeWords).

corpus <- VCorpus(VectorSource(sample_data))
corpus <- tm_map(corpus, content_transformer(tolower))
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, stripWhitespace)

6. Exploratory Analysis: N-Grams

To understand word patterns, we build unigram, bigram, and trigram frequency tables using RWeka.

# Extract cleaned text back out of the corpus
clean_text <- sapply(corpus, as.character)

get_ngram_freq <- function(text, n, top_n = 15) {
  ngrams <- tokenize_ngrams(text, n = n, simplify = TRUE)
  ngrams <- unlist(ngrams)
  freq_table <- sort(table(ngrams), decreasing = TRUE)
  data.frame(term = names(freq_table)[1:top_n],
             frequency = as.integer(freq_table)[1:top_n])
}

unigram_freq <- get_ngram_freq(clean_text, n = 1)
bigram_freq  <- get_ngram_freq(clean_text, n = 2)
trigram_freq <- get_ngram_freq(clean_text, n = 3)

6.1 Top Unigrams (Single Words)

ggplot(unigram_freq, aes(x = reorder(term, frequency), y = frequency)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Words", x = "Word", y = "Frequency") +
  theme_minimal()

6.2 Top Bigrams (Two-Word Phrases)

ggplot(bigram_freq, aes(x = reorder(term, frequency), y = frequency)) +
  geom_bar(stat = "identity", fill = "darkorange") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Bigrams", x = "Bigram", y = "Frequency") +
  theme_minimal()

6.3 Top Trigrams (Three-Word Phrases)

ggplot(trigram_freq, aes(x = reorder(term, frequency), y = frequency)) +
  geom_bar(stat = "identity", fill = "darkgreen") +
  coord_flip() +
  labs(title = "Top 15 Most Frequent Trigrams", x = "Trigram", y = "Frequency") +
  theme_minimal()

7. Key Findings

  • The three data sources differ meaningfully in style and length: Twitter text is short and informal, Blogs contain long-form personal writing, and News is more formally structured.
  • A small number of very common words (stop words like “the”, “to”, “and”) dominate the unigram frequency distribution, as expected in natural language.
  • Common bigrams and trigrams reveal everyday phrases (e.g. “thanks for”, “happy new year”, “can not wait”) that will be useful as building blocks for next-word prediction.
  • Word frequency follows a long-tail distribution: a small set of words covers a large fraction of all word instances, while a very large number of rare words make up the tail — this is important for deciding vocabulary size in the final model.

8. Plan for the Prediction Algorithm and Shiny App

(Written for a non-technical audience.)

The final product will be a simple web app where a user types a few words, and the app suggests the most likely next word — similar to predictive text on a smartphone keyboard.

How we will build it:

  1. N-gram language model: Using the word-pattern tables built above (bigrams, trigrams, and beyond), we will estimate the probability of each possible next word given the previous 1–3 words the user has typed.
  2. Smoothing/back-off strategy: When an exact phrase hasn’t been seen before, the model will “back off” to shorter phrases (e.g., from trigram to bigram to unigram) so it can still make a reasonable prediction.
  3. Efficiency: Since the raw model built from all the data could be very large, we will prune rare n-grams to keep the app fast and lightweight enough to run in a browser.
  4. Shiny App: A simple interface where the user types text into a box and sees the top 3 predicted next words update in real time.

Next steps: - Finalize n-gram tables using a larger sample of the data. - Build and test the back-off prediction model. - Evaluate accuracy and response speed. - Deploy the final model inside a Shiny app with a clean, simple UI.

9. Conclusion

The data has been successfully downloaded, loaded, sampled, and explored. Basic summary statistics and n-gram frequency analysis have revealed useful patterns in the text that will guide the development of the predictive text algorithm and Shiny application in the next phase of this project.