Introduction

This project capstone milestone introduces the basics of natural language processing by analyzing a large corpus of text documents that includes blogs, news articles and twitter data. The goal of the milestone phase is to discover the structure of the data by performing Data Exploration and Data Cleanin concepts. The initial phase of the project also includes building word cloud of common terms used in each data set corpus. Later phases focus on building and sampling from a predictive text model to predict word using ngram techniques.

The final goal of the project is to predict a word after a user types the word in smart keyboard. It will use Natural Processing Language concepts (NLP) N-grams. The idea is to train a model with data from news, blogs and tweets and the let model suggests a word following a Markov probalistic model.

Data

fileURL <- "http://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
fileZip <- "Coursera-SwiftKey.zip"
download.file(fileURL, fileZip, method = "auto")

# Extract the datasets 
unzip(fileZip)

Libraries

# Libraries
suppressWarnings(suppressMessages(library(stringi)))
suppressWarnings(suppressMessages(library(ggplot2)))
suppressWarnings(suppressMessages(library(tm)))
suppressWarnings(suppressMessages(library(SnowballC)))
suppressWarnings(suppressMessages(library(RColorBrewer)))
suppressWarnings(suppressMessages(library(wordcloud)))

Read data

# Blogs dataset
con <- file("final/en_US/en_US.blogs.txt", open="rb")
blogs <- readLines(con, encoding = "UTF-8",skipNul=TRUE)
close(con)

# News dataset
con <- file("final/en_US/en_US.news.txt", open="rb")
news <- readLines(con, encoding = "UTF-8",skipNul=TRUE)
close(con)

# Twitter dataset
con <- file("final/en_US/en_US.twitter.txt", open="rb")
twitter <- readLines(con, encoding = "UTF-8",skipNul=TRUE)
close(con)

rm(con)

Exploratory Data Analysis

Explore file sizes

# File sizes for each dataset
mb <- 1024^2
blogs.size <- paste(round(file.info("final/en_US/en_US.blogs.txt")$size / mb, digits=2),'Mb')
news.size <- paste(round(file.info("final/en_US/en_US.news.txt")$size / mb, digits=2),'Mb')
twitter.size <- paste(round(file.info("final/en_US/en_US.twitter.txt")$size / mb, digits=2),'Mb')
summary_size <- data.frame(blogs.size, news.size, twitter.size)
print (summary_size)

Data summary using stringi library

stri_stats_general(blogs)
##       Lines LinesNEmpty       Chars CharsNWhite 
##      899288      899288   206824382   170389539
stri_stats_general(news)
##       Lines LinesNEmpty       Chars CharsNWhite 
##     1010242     1010242   203223154   169860866
stri_stats_general(twitter)
##       Lines LinesNEmpty       Chars CharsNWhite 
##     2360148     2360148   162096241   134082806

Summary Statistics of each data set

blogs.words <- stri_count_words(blogs)
summary(blogs.words)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.00    9.00   28.00   41.75   60.00 6726.00
news.words <- stri_count_words(news)
summary(news.words)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00   19.00   32.00   34.41   46.00 1796.00
twitter.words <- stri_count_words(twitter)
summary(twitter.words)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    7.00   12.00   12.75   18.00   47.00

Frequencies tables and proportion of words

We are going to analyse the frequencies and proportions of words within each dataset separatly.

Blogs
table(blogs.words > mean(blogs.words)) / length(blogs.words)
## 
##     FALSE      TRUE 
## 0.6177076 0.3822924

We can see that only 38 % of words in blogs contain more than average number of words in a given blog entry. An average blog entry contains 41 words.

News
table(news.words > mean(news.words)) / length(news.words)
## 
##     FALSE      TRUE 
## 0.5603261 0.4396739

For news, we only have 44% of words that are present more than the average number of words in a news article. An average news article has 34 words.

Twitter
table(twitter.words > mean(twitter.words)) / length(twitter.words)
## 
##     FALSE      TRUE 
## 0.5259687 0.4740313

For twitter, we only have 47% of words that are more than the average number of words in a tweets. An average news article has 13 words. *** #### Data Pre-Processing

We first takes sample of take 10,000 records from each data set
# set a seed number so the same sample can be reproduced
set.seed(100)
# Create samble subset for blogs data
blogs.sample <- blogs[sample(1:length(blogs),10000)]

Data Pre-Processing

Building Corpuses

One of the concepts introduced by the tm package is that of a corpus which is a collection of documents. We will convert our datasets into corpus for pre-processing. Here, we construct the corpus from the samples, and then perform the following data cleaning steps:

  1. Create corpus
  2. Convert to ASCII
  3. Convert all characters to lower case
  4. Remove the punctuation
  5. Remove numbers
  6. Strip all white space
  7. Remove English words
  8. Stemming (for the final combined corpus only)

Blog Corpus

# Create a corpus as a vector source for blogs
corpus.blogs <- Corpus(VectorSource(blogs.sample))

# inspect first record
corpus.blogs[[1]]$content
## [1] "So. Jeff has a talk with the monkey and tries to explain to him that he needed to have courage to eat the zucchini. He needed to look at it like Super Man would look at kryptonite and ATTACK the zucchini! The boy took a couple of quick breaths and ran back into the kitchen, determined to beat the dreaded green yuck. A few minutes later he came out, triumphant! Good job, monkey! You did it!"
# convert documents into ASCII format
corpus.blogs <- tm_map(corpus.blogs,content_transformer(function(x)  {iconv(x, to="ASCII", sub="") }))

## Convert to lower case
corpus.blogs <- tm_map(corpus.blogs, content_transformer(tolower), lazy = TRUE)

# Remove punctuation
corpus.blogs  <- tm_map(corpus.blogs, content_transformer(removePunctuation))

# Remove digits
corpus.blogs  <- tm_map(corpus.blogs, content_transformer(removeNumbers))

# Remove Whitespace
corpus.blogs<- tm_map(corpus.blogs, stripWhitespace)

# Remove English words
corpus.blogs <- tm_map(corpus.blogs, removeWords, stopwords("english"))

# Remove Whitespace
corpus.blogs <- tm_map(corpus.blogs, stripWhitespace)

# verify the final preprocessed corpus
corpus.blogs[[1]]$content
## [1] " jeff talk monkey tries explain needed courage eat zucchini needed look like super man look kryptonite attack zucchini boy took couple quick breaths ran back kitchen determined beat dreaded green yuck minutes later came triumphant good job monkey "

Saving the preprocessed blog corpus

saveRDS(corpus.blogs, file = "blog_corpus.RDS")

News corpus

# Create a sample subset for news data (10,000 records)
news.sample <- news[sample(1:length(news),10000)]

Creating a corpus as a vector source for news

# Create a corpus as a vector source for news
corpus.news <- Corpus(VectorSource(news.sample))

# inspect first record
corpus.news[[1]]$content
## [1] "Doug Neville, a spokesman for the public safety department, said the panel decided to take two tracks:"
# convert documents into ASCII format
corpus.news <- tm_map(corpus.news,content_transformer(function(x)  {iconv(x, to="ASCII", sub="") }))

## Convert to lower case
corpus.news <- tm_map(corpus.news, content_transformer(tolower), lazy = TRUE)

# Remove puntuaction
corpus.news  <- tm_map(corpus.news, content_transformer(removePunctuation))

# Remove digits
corpus.news  <- tm_map(corpus.news, content_transformer(removeNumbers))

# Remove Whitespace
corpus.news<- tm_map(corpus.news, stripWhitespace)

# Remove English words
corpus.news <- tm_map(corpus.news, removeWords, stopwords("english"))

# Remove Whitespace
corpus.news <- tm_map(corpus.news, stripWhitespace)

# Verify the final preprocessed corpus
corpus.news[[1]]$content
## [1] "doug neville spokesman public safety department said panel decided take two tracks"

Saving the preprocessed news corpus

saveRDS(corpus.news, file = "news_corpus.RDS")

Twitter corpus

# Create a sample subset for twitter data
twitter.sample <- twitter[sample(1:length(twitter),10000)]
Creating a corpus as a vector source for newss
# Create a corpus as a vector source for twitter
corpus.twitter <- Corpus(VectorSource(twitter.sample))

# inspect a sample record[10]
corpus.twitter[[10]]$content
## [1] "“: Today is Monday and that means its time for SMASH...! Who will be watching!? #greatepisodecomingup” love this show!"
# convert documents into ASCII format
corpus.twitter <- tm_map(corpus.twitter,content_transformer(function(x)  {iconv(x, to="ASCII", sub="") }))

## Convert to lower case
corpus.twitter <- tm_map(corpus.twitter, content_transformer(tolower), lazy = TRUE)

# Remove punctuaction
corpus.twitter  <- tm_map(corpus.twitter, content_transformer(removePunctuation))

# Remove Numbers
corpus.twitter  <- tm_map(corpus.twitter, content_transformer(removeNumbers))

# Remove Whitespace
corpus.twitter <- tm_map(corpus.twitter, stripWhitespace)

# Remove English stop words
corpus.twitter <- tm_map(corpus.twitter, removeWords, stopwords("english"))

# Remove Whitespace
corpus.twitter <- tm_map(corpus.twitter, stripWhitespace)

# Verify the final document after pre-processing
corpus.twitter[[10]]$content
## [1] " today monday means time smash will watching greatepisodecomingup love show"

Saving the preprocessed twitter corpus

saveRDS(corpus.twitter, file = "twitter_corpus.RDS")

Document Term Matrix

Document Term Matrix (DTM) reflects the number of times each word in the corpus is found in the document. DTM generates a matrix where the rows corresponds to documents (blog entries, news stories or tweets) and the columns correspond to words in those data sets. The values in the matrix are the number of times that a word would appears in each document.

# Take small sample of 100 documents to create a Document Term Matrix in order to display its wordcloud 
blogs.dtm <- TermDocumentMatrix(corpus.blogs[sample(1:length(corpus.blogs),100)])
news.dtm <- TermDocumentMatrix(corpus.news[sample(1:length(corpus.news),100)])
twitter.dtm <- TermDocumentMatrix(corpus.twitter[sample(1:length(corpus.twitter),100)])

Analysis of Document Term Frequencies on corpus

We can take a look at our matrices for each corpus.

frequencies.blogs <- DocumentTermMatrix(corpus.blogs)
frequencies.news <- DocumentTermMatrix(corpus.news)
frequencies.twitter <- DocumentTermMatrix(corpus.twitter)

Let’s see what the blog matrix looks like by using inspect(). We can see that the world ‘blog’ appears 3 times in document 1036 but the word ‘blocker’ does not. We see also lots of zero in this matrix.This sample blog data is sparse which means we have many zeros in our matrix.

inspect(frequencies.blogs[1035:1037,2000:2010])
## <<DocumentTermMatrix (documents: 3, terms: 11)>>
## Non-/sparse entries: 0/33
## Sparsity           : 100%
## Maximal term length: 9
## Weighting          : term frequency (tf)
## 
##       Terms
## Docs   baines baird bait bake baked baker bakerella bakeries bakers baking
##   1035      0     0    0    0     0     0         0        0      0      0
##   1036      0     0    0    0     0     0         0        0      0      0
##   1037      0     0    0    0     0     0         0        0      0      0
##       Terms
## Docs   bakiyev
##   1035       0
##   1036       0
##   1037       0

We can now look at what the most popular terms or words are with the function findFreqTerms(). We want to call this on our matrix frequencies ie. frequencies.blogs and then we want to give an argument lowFreq, which is equal to the minimum number of times a term must appear to be displayed. We see that out of 21506 terms, only 78 of them appear at least 300 times in sample blog corpus. We can do similar analysis for news and twitter corpus.We could have a lot of terms that will be useless to our ngram prediction model.

# terms that appears at least 300 times in the sample blog corpus of 10,000 records
findFreqTerms(frequencies.blogs, lowfreq=300)
##  [1] "also"      "around"    "back"      "book"      "can"      
##  [6] "come"      "day"       "didnt"     "dont"      "even"     
## [11] "every"     "find"      "first"     "get"       "going"    
## [16] "good"      "got"       "great"     "ive"       "just"     
## [21] "know"      "last"      "life"      "like"      "little"   
## [26] "love"      "made"      "make"      "many"      "may"      
## [31] "much"      "need"      "never"     "new"       "now"      
## [36] "one"       "people"    "really"    "right"     "said"     
## [41] "say"       "see"       "something" "still"     "take"     
## [46] "things"    "think"     "time"      "two"       "want"     
## [51] "way"       "well"      "will"      "work"      "year"     
## [56] "years"

Creating word cloud for blogs, news and twitter

# Set layout
par(mfrow=c(1,3))
# Word Cloud for blogs
blogs.wcloud <- as.matrix(blogs.dtm)
rowsAgg <- sort(rowSums(blogs.wcloud),decreasing=TRUE)
content <- data.frame(word = names(rowsAgg),freq=rowsAgg)
wordcloud(content$word,content$freq,
          c(3,.2),max.words=20,
          random.order=FALSE,
          colors=brewer.pal(8, "Dark2"),)
title("20 words TagCloud for blogs")
# Word Cloud for news
news.wcloud <- as.matrix(news.dtm)
rowsAgg <- sort(rowSums(news.wcloud),decreasing=TRUE)
content <- data.frame(word = names(rowsAgg),freq=rowsAgg)
wordcloud(content$word,content$freq,
          c(3,.2),max.words=20,
          random.order=FALSE,
          colors=brewer.pal(8, "Dark2"),)
title("20 words TagCloud for News")

# Word Cloud for twitter
twitter.wcloud <- as.matrix(twitter.dtm)
rowsAgg <- sort(rowSums(twitter.wcloud),decreasing=TRUE)
content <- data.frame(word = names(rowsAgg),freq=rowsAgg)
wordcloud(content$word,content$freq,
          c(3,.2),max.words=20,
          random.order=FALSE,
          colors=brewer.pal(8, "Dark2"),)
title("20 words TagCloud Twitter")

Plan for the next steps

Conclusion

We analyzed three datasets written in US English from Social Medias (blogs, news and twitter). The file sizes are around 200 Megabytes (MBs) per file.

We find that the blogs and news corpora consist of about 1 million items each, and the twitter* corpus consist of over 2 million items. Twitter messages have a character limit of 140 (with exceptions for links), this explains why there are some many more items for a corpus of about the same size.

This result is further supported by the fact that the number of characters is similar for all three corpora (around 200 million each).

Finally we find that the frequency distributions of the blogs and news corpora are similar. The frequency distribution of the twitter corpus is again different, as a result of the character limit.

When using Document Term Matrix on our corpuses, we may have lots of zeroes, we have essentially a sparse matrix. This means that we probably have lots of terms that could be pretty useless for our prediction model. The number of term is an issue for two main reasons. One is computational. More terms means more independent variables, which usually means it takes longer to build a predictive model. The other is in building models, the ratio of independent variables to observations will affect how good the model will generalize. So we usually remove terms that not very frequent in the corpus. We can select a sparsity threshold works as follows. If we say 0.98, this means to only keep terms/words that appear in 2% or more the preprocessed corpus. In the final combined corpus, we will attempt to use a sparsity threshold of 0.995 to remove any unneeded terms.