Milestone Report

Executive Summary

This is the milestone report for the Capstone Project of the Data Science Specialization through Coursera and Johns Hopkins. The goal of the capstone project is to create a text prediction application based on previous typed ing text. This report will explore the three text files provided for the class in order to become more familiar with Natural Language Processing. These files include a blog, a news, and a twitter text files. The first task till be to load in the data successfully. Next summary statistics on the text files will be made along with graphs to show interesting findings. Finally, thoughts about how the findings in this report will contribute to a prediction model.

Important Elements

• A line of text refers to an single entry of text like one blog submission or one twitter entry.

• The three files than will be combined in order to do an analysis on n-grams. N-grams are a consecutive combination of words (unigram = one word, bigram = two consecutive words, trigram = three consecutive words, etc.). N-grams may be useful in prediction model by using a common consecutive combination of words.

• A corpus of text will be created in order to explore the three files at once. A corpus is a collection of text documents. By combining the three, a more vast dataset can be used to train a prediction model.

library(stringi)
library(ggplot2)
library(NLP)
library(tm)
library(RColorBrewer)
library(wordcloud)
library(RWeka)
library(SnowballC)
library(knitr)

setwd("~/Desktop/Coursera/Capstone/final/en_US")

Reading Files

The three text files are read in separetly. This allows for data exploration to be done on each file.

blogs <- readLines("en_US.blogs.txt", encoding = "UTF-8", skipNul = TRUE)
news <- readLines("en_US.news.txt", encoding = "UTF-8", skipNul = TRUE)
twitter <- readLines("en_US.twitter.txt", encoding = "UTF-8", skipNul = TRUE)

#blogs stats
blogs.stats <- stri_stats_general(blogs)

words_blogs <-stri_count_words(blogs)
blogs.total.words <- sum(words_blogs)
blogs.words.stats <- summary(words_blogs)

#news stats
news.stats <- stri_stats_general(news)

words_news <-stri_count_words(news)
news.total.words <- sum(words_news)
news.words.stats <- summary(words_news)

#twitter stats
twitter.stats <- stri_stats_general(twitter)

words_twitter <-stri_count_words(twitter)
twitter.total.words <- sum(words_twitter)
twitter.words.stats <- summary(words_twitter)

#Summary dataframe of three text files
stats <- data.frame(File = c("blogs", "news", "twitter"),
                    Lines = c(blogs.stats[1], news.stats[1], twitter.stats[1]),
                    LinesNEmpty = c(blogs.stats[2], news.stats[2], twitter.stats[2]),
                    Chars = c(blogs.stats[3], news.stats[3], twitter.stats[3]),
                    CharsNWhite = c(blogs.stats[4], news.stats[4], twitter.stats[4]),
                    MinWords = c(blogs.words.stats[1], news.words.stats[1], twitter.words.stats[1]),
                    MedianWords = c(blogs.words.stats[3], news.words.stats[3], twitter.words.stats[3]),
                    MeanWords = c(blogs.words.stats[4], news.words.stats[4], twitter.words.stats[4]),
                    MaxWords = c(blogs.words.stats[6], news.words.stats[6], twitter.words.stats[6]),
                    TotalWords = c(blogs.total.words, news.total.words, twitter.total.words))

Summary Statistics

The following are summary statistics on the three text files. Lines refers to the number of lines in the files. LinesNEmpty refers to the number of lines with at least one non-white space character. Chars is the total number of characters detected in files. CharsNWhite is total number of characters detected that are not white space. MinWords is the minimum number of words in a line. MedianWords is the median number of words in a line. MeanWords is mean per line. Maxwords is max words per line. TotalWords is total words in file.

kable(stats)

File	Lines	LinesNEmpty	Chars	CharsNWhite	MinWords	MedianWords	MeanWords	MaxWords	TotalWords
blogs	899288	899288	206824382	170389539	0	28	41.75	6726	37546246
news	1010242	1010242	203223154	169860866	1	32	34.41	1796	34762395
twitter	2360148	2360148	162096241	134082806	1	12	12.75	47	30093410

Histograms

The following histograms show the distribution of the number of words per line of text. The x-axis is the number of words per line. Note that the scales have been adjusted for each file.

qplot(words_blogs, binwidth = 5, xlim = c(0,450)) + xlab("Number of Words per line") + ggtitle("Blog File")

qplot(words_news, binwidth = 5, xlim = c(0,300)) + xlab("Number of Words per line") + ggtitle("News File")

qplot(words_twitter, binwidth = 2, xlim = c(0,50)) + xlab("Number of Words per line") + ggtitle("Twitter File")

Corpus Creation

The creation of a corpus will allow for analysis to be done on all three files at once. Once the corpus is created text mining functions will be used. Text mining functions will clean up the corpus by removing things liek punctuation, number, stopwords, and make everything lowercase. Text mining makes text analysis easier.

# 2106 + 2366 + 5528 = 10,000 samples
# sample proportionally taken based on lines of text file
sample.blogs <- sample(blogs, 2106)
sample.news <- sample(news, 2366)
sample.twitter <- sample(twitter, 5528) 
# combine into one file
sample.text <- paste(sample.blogs, sample.news, sample.twitter)
# create corpus
doc.vec <- VectorSource(sample.text)
corpus <- VCorpus(doc.vec)
# text mining
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, content_transformer(tolower)) 
corpus <- tm_map(corpus, removeWords, stopwords('english'))
corpus <- tm_map(corpus, stemDocument)
# creating corpus dataframe for NGram Tokenizer
corpus.df <- data.frame(text = sapply(corpus, as.character), stringsAsFactors = FALSE)

Unigram

Unigrams are single words. The NGramTokenizer() from the RWeka facilitates the process of creating N-gram tokens.

#N-gram tokenizer
one.word <- NGramTokenizer(corpus.df, Weka_control(min=1,max=1))
#Preparing N-gram data to be graphed
oneword.df <- data.frame(table(one.word))
one.sorted <- oneword.df[order(oneword.df$Freq,decreasing=TRUE),]
colnames(one.sorted) <- c("word", "freq")
one.top20 <- head(one.sorted,20)
#Graphing N-Gram data
ggplot(one.top20, aes(x=reorder(word,freq), y=freq)) +
  geom_bar(stat="Identity", fill="steelblue") +
  ggtitle("Unigrams Frequency") +
  coord_flip() +
  ylab("Frequency") +
  xlab("Unigram")

Unigram Wordcloud

A wordcloud is a nice visualization. This will only be done for unigrams. The more frequently words are used in the corpus the larger their respective size in the visualization.

wordcloud(corpus, max.words = 100, random.order = FALSE,colors=brewer.pal(5, "Dark2"))

Bigram

Bigrams two word phrases that are commonly used.

two.word <- NGramTokenizer(corpus.df, Weka_control(min=2,max=2))
twoword.df <- data.frame(table(two.word))
two.sorted <- twoword.df[order(twoword.df$Freq,decreasing=TRUE),]
colnames(two.sorted) <- c("word", "freq")
two.top20 <- head(two.sorted,20)
ggplot(two.top20, aes(x=reorder(word,freq), y=freq)) +
  geom_bar(stat="Identity", fill="firebrick") +
  ggtitle("Bigrams Frequency") +
  coord_flip() + 
  ylab("Frequency") +
  xlab("Bigram")

Trigram

Trigams are three word phrases that are commonly used. Here the removal of strong words and text mining can be seen since some of the phrases seem incomplete.

tritoken <- NGramTokenizer(corpus.df, Weka_control(min=3,max=3))
three_word <- data.frame(table(tritoken))
sort_three <- three_word[order(three_word$Freq,decreasing=TRUE),]
colnames(sort_three) <- c("Word", "Freq")
three.top20 <- head(sort_three,20)
ggplot(three.top20, aes(x=reorder(Word,Freq), y=Freq)) +
  geom_bar(stat="Identity", fill="darkorange") +
  ggtitle("Trigram frequency") +
  coord_flip() +
  ylab("Frequency") +
  xlab("Trigram")

Word Coverage

One interesting question that can now be answered about this text data easily is how many unique words would you need to cover 50% of all the words used in the text files. This turns out to be only about 3.38% percent.

total.freq <- sum(one.sorted$freq)
break.freq <- 0.5*total.freq
running.freq <- 0
for(i in 1:length(one.sorted$freq)) {
  running.freq <- running.freq + one.sorted$freq[i]
  if(running.freq >= break.freq) { 
          print(i/length(one.sorted$freq)) 
          break
  }
}

## [1] 0.03406002

Now 90% of of all words used in text files. This turns out to 34.72%

total.freq <- sum(one.sorted$freq)
break.freq <- 0.9*total.freq
running.freq <- 0
for(i in 1:length(one.sorted$freq)) {
  running.freq <- running.freq + one.sorted$freq[i]
  if(running.freq >= break.freq) { 
          print(i/length(one.sorted$freq)) 
          break
  }
}

## [1] 0.3456482

Preparation for Model Creation

This has been good practice in order to prepare for the creation of a text mining prediction model. Many of the methods used in this report will be used moving forward.

References

• https://www.r-bloggers.com/joy-to-the-world-and-also-anticipation-disgust-surprise/
• https://www.youtube.com/watch?v=lRTerj8fdY0
• http://stackoverflow.com/questions/33193152/unable-to-convert-a-corpus-to-data-frame-in-r
• https://gist.github.com/benmarwick/5370329