Milestone Report

Introduction

This report represents the Milestone Report for the SwiftKey Project, which is the Capstone Project for the Data Science Specialization on Coursera. This Milestone Report gives an overview of the Loading, Cleaning and descriptive/ explorative Data Analysis of the Data.

Loading the data

The following Code will load the required Packages and the data into R. The dataset is available in different languages. In this Report we will take a look at the English dataset. There are three files: Twitter, News, Blogs.

library(tm)
## Loading required package: NLP
library(stringi)
library(ngram)
library(stringr)

twitter_con <- file("final/en_US/en_US.twitter.txt", "r")
twitter <- readLines(twitter_con, skipNul = TRUE, warn = FALSE)
close(twitter_con)

blogs_con <- file("final/en_US/en_US.blogs.txt", "r")
blogs <- readLines(blogs_con, skipNul = TRUE, warn = FALSE)
close(blogs_con)

news_con <- file("final/en_US/en_US.news.txt", "r")
news <- readLines(news_con, skipNul = TRUE, warn = FALSE)
close(news_con)

Descriptive Data Analysis

In the following we will take a first look at the datasets. The following code outputs basic summaries about each of the three datasets:

library(knitr)
library(stringi)

twitter_words <- stri_count_words(twitter)
blogs_words <- stri_count_words(blogs)
news_words <- stri_count_words(news)

summary_table <- data.frame(
  Dataset = c("Twitter", "Blogs", "News"),
  Lines = c(length(twitter), length(blogs), length(news)),
  Total_Words = c(sum(twitter_words),
                  sum(blogs_words),
                  sum(news_words)),
  Average_Words_Per_Line = c(mean(twitter_words),
                             mean(blogs_words),
                             mean(news_words))
)

kable(summary_table, digits = 2)
Dataset Lines Total_Words Average_Words_Per_Line
Twitter 2360148 30096690 12.75
Blogs 899288 37546806 41.75
News 1010206 34761151 34.41

Data Cleaning

In the next step we are going to combine the data into one dataset and only take a few samples from the three datasets because they are so big and we might run into memory or performance issues. Then we are going to clean the data, e.g. by removing unneccesary spaces or symbols.

library(tm)
## Loading required package: NLP
twitter_sample <- sample(twitter, 10000)
blogs_sample <- sample(blogs, 10000)
news_sample <- sample(news, 10000)
dataset <- c(twitter_sample, blogs_sample, news_sample)
head(dataset)
## [1] "That sounds...stinky? Really though, what are those things stuffed with, anyway? THAT stuff probably smells weird."             
## [2] "Help our students and our schools. Pass the #AJA now OFA.BO/ipwXw2"                                                             
## [3] "Pics from the Mayday Parade on Polish Hill"                                                                                     
## [4] "Nick Symmonds from Eugene, may have had too much Peking duck and is currently leading with OSU Beaver colored Nike's."          
## [5] "\"How do you do them making love for the first time?\" Director Bill Condon says of the making of \"Breaking Dawn\" ..........."
## [6] "saved it for OT?"
length(dataset)
## [1] 30000
dataset_corpus <- VCorpus(VectorSource(dataset))
dataset_corpus <- tm_map(dataset_corpus, content_transformer(tolower))
dataset_corpus <- tm_map(dataset_corpus, removeWords, stopwords("en"))
dataset_corpus <- tm_map(dataset_corpus, removePunctuation)
dataset_corpus <- tm_map(dataset_corpus, removeNumbers)
dataset_corpus <- tm_map(dataset_corpus, stripWhitespace)
dataset_corpus <- tm_map(dataset_corpus, PlainTextDocument)

Modeling

In this chapter we are going to look at 1-, 2 and 3-gram models. With these models we will look at the most frequently occurring words or words combinations.

library(ngram)
library(stringr)

dataset_ngram <- unlist(sapply(dataset_corpus, `[`, "content"))
dataset_ngram <- dataset_ngram[dataset_ngram != " "]
dataset_ngram <- dataset_ngram[dataset_ngram != ""]
dataset_ngram <- dataset_ngram[sapply(dataset_ngram, wordcount) > 0]
dataset_ngram <- str_trim(dataset_ngram, "both")

par(mar=c(10,4,4,1)+.1)

ngram_1 <- ngram(dataset_ngram, n = 1)
ngram_1_table <- get.phrasetable(ngram_1)
ngram_1_table <- ngram_1_table[ngram_1_table[[1]] != "", ]
head(ngram_1_table)
##   ngrams freq        prop
## 1  said  3001 0.006290455
## 2  will  2550 0.005345105
## 3   one  2533 0.005309471
## 4  just  2238 0.004691116
## 5  like  2077 0.004353641
## 6   can  2032 0.004259315
barplot(ngram_1_table[1:10, 2], names.arg = ngram_1_table[1:10, 1], main = "Top 10 Unigram", las = 2)

dataset_ngram <- dataset_ngram[sapply(dataset_ngram, wordcount) > 1]
ngram_2 <- ngram(dataset_ngram, n = 2)
ngram_2_table <- get.phrasetable(ngram_2)
ngram_2_table <- ngram_2_table[ngram_2_table[[1]] != " ", ]
head(ngram_2_table)
##         ngrams freq         prop
## 1    new york   182 0.0004070168
## 2   last year   173 0.0003868896
## 3   years ago   151 0.0003376898
## 4   right now   138 0.0003086171
## 5 high school   133 0.0002974353
## 6 even though   116 0.0002594173
barplot(ngram_2_table[1:10, 2], names.arg = ngram_2_table[1:10, 1], main = "Top 10 Bigram", las = 2)

dataset_ngram <- dataset_ngram[sapply(dataset_ngram, wordcount) > 2]
ngram_3 <- ngram(dataset_ngram, n = 3)
ngram_3_table <- get.phrasetable(ngram_3)
ngram_3_table <- ngram_3_table[ngram_3_table[[1]] != "", ]
head(ngram_3_table)
##                   ngrams freq         prop
## 1         two years ago    20 4.784220e-05
## 2           omg omg omg    17 4.066587e-05
## 3   little italy boston    17 4.066587e-05
## 4  austin austin austin    17 4.066587e-05
## 5 magianos little italy    17 4.066587e-05
## 6     happy mothers day    15 3.588165e-05
barplot(ngram_3_table[1:10, 2], names.arg = ngram_3_table[1:10, 1], main = "Top 10 Trigram", las = 2)

Summary Findings

The dataset contains over 4 million text entries across Twitter, blogs, and news sources. Blogs contain the longest entries on average, with approximately 42 words per line, while Twitter messages are shorter with approximately 13 words per line. News articles fall between the two sources with approximately 34 words per line. These differences show that the dataset contains a variety of writing styles that can help build a more general predictive text model.

The following histograms show the distribution of words per line for each text source.

hist(twitter_words,
     breaks = 50,
     main = "Distribution of Words per Twitter Line",
     xlab = "Number of Words",
     xlim = c(0, 100))

hist(blogs_words,
     breaks = 50,
     main = "Distribution of Words per Blog Line",
     xlab = "Number of Words",
     xlim = c(0, 150))

hist(news_words,
     breaks = 50,
     main = "Distribution of Words per News Line",
     xlab = "Number of Words",
     xlim = c(0, 150))

Conclusion

In this report we showed the Loading and Cleaning of the dataset as well as some descriptive and explorative data analysis. We finished the report with some first and simple modeling. In the next step we will build the final prediction model and finally deploy the model on a Shiny app. The final model will consist of some n-gram model. It will take all prior known words up to a maximum number (like the mean word-count of a sentence) and use that number of words as the n for the n-gram-model. This way the algorithm would take into account the given number of words but doesn’t use too many words (if the user already typed more sentences). The App will need an input field so the user can input the text as well as a few label fields to show the predictions.