Milestone report

Burhan Khan

February 19, 2018

Summary

This report explains basic steps for text mining analysis consist of:

1. Load training datasets
2. Basic analysis for datasets such as word count, line count, size of the file
3. Data manipulation including limit our sample size, aggregate our dataset, cleaning dataset, corpus creation and build our n-gram model
4. Data Exploration
5. Insights
6. Way forward

Loading libraries

library(stringi)
library(tm)
library(RWeka)
library(ggplot2)
library(wordcloud)

Loading our data set for Twitter, blog and news.

ds_us_twitter <- file("D:\\Data\\final\\en_US\\en_US.twitter.txt")
ds_us_blog <- file("D:\\Data\\final\\en_US\\en_US.blogs.txt")
ds_us_news <- file("D:\\Data\\final\\en_US\\en_US.news.txt")
 
us_twitter <- readLines(ds_us_twitter)
us_blog <- readLines(ds_us_blog)
us_news <- readLines(ds_us_news)

Line Counts

Twitter data line count

twitter_linecount <- length(us_twitter)
twitter_linecount

[1] 2360148

Blog data line count

blog_linecount <- length(us_blog)
blog_linecount

[1] 899288

News data line count

news_linecount <- length(us_news)
news_linecount

[1] 1010242

Words count

twitter_wordcount <- sum(stri_count_words(us_twitter))
blog_wordcount <- sum(stri_count_words(us_blog))
news_wordcount <- sum(stri_count_words(us_news))
cat('Twitter data count: ', twitter_wordcount)

Twitter data count:  30218125

cat('\nBlog data word count: ',blog_wordcount)

Blog data word count:  38154238

cat('\nNews data word count: ',news_wordcount)

News data word count:  35010786

Calculating file size in MB

twitter_size <- file.info("D:\\Data\\final\\en_US\\en_US.twitter.txt")$size/1024000
blog_size <- file.info("D:\\Data\\final\\en_US\\en_US.blogs.txt")$size/1024000
news_size <- file.info("D:\\Data\\final\\en_US\\en_US.news.txt")$size/1024000

Summarizing

basic_summary_table <- data.frame(Dataset.Name=c("Twitter","Blogs","News"),Dataset.Size.MB=format(c(twitter_size,blog_size,news_size),digits = 5),Number.of.Lines=format(c(twitter_linecount,blog_linecount,news_linecount),big.mark = ","),Word.Count=format(c(twitter_wordcount,blog_wordcount,news_wordcount),big.mark = ","))
knitr::kable(basic_summary_table)

Dataset.Name	Dataset.Size.MB	Number.of.Lines	Word.Count
Twitter	163.19	2,360,148	30,218,125
Blogs	205.23	899,288	38,154,238
News	200.99	1,010,242	35,010,786

Segregating data into test and train, taking 70% data for train and 20% for test

Also, we are taking only Blog data for illustration purpose.

train_blog <- sample(us_blog, round(length(us_blog)*.7))

Cleaning the data and setting the default encoding

train_blog <- iconv(train_blog,"UTF-8","ASCII","byte")

Creating our corpus

master_corpus <- Corpus(VectorSource(sample(train_blog)))

We will be using only 5000 lines of data from out train for illustration purpose

main_corpus<-sample(master_corpus,5000)

Standardizing our Corpus to lower case, removing puntuatuations, numbers and white spaces.

main_corpus <- tm_map(main_corpus, tolower)
main_corpus <- tm_map(main_corpus, removePunctuation)
main_corpus <- tm_map(main_corpus, removeWords, stopwords("en"))
main_corpus <- tm_map(main_corpus, removeNumbers)
main_corpus <- tm_map(main_corpus, stripWhitespace)

Create tokenizer function for uni-gram, bi-gram and tri-gram

unigram_token <- NGramTokenizer(main_corpus, Weka_control(min=1, max=1))
bigram_token <- NGramTokenizer(main_corpus, Weka_control(min=2, max=2))
trigram_token <- NGramTokenizer(main_corpus, Weka_control(min=3, max=3))
# Construct data frame for each n-gram
unigram_df <- data.frame(table(unigram_token))
bigram_df <- data.frame(table(bigram_token))
trigram_df <- data.frame(table(trigram_token))

Generating Word Cloud for out train data

wordcloud(main_corpus, max.words=100, random.order=FALSE,colors=brewer.pal(8, "Dark2"))

Get top 20 most frequent words

hplot <- unigram_df[order(-unigram_df$Freq),][1:20,]
ggplot(hplot, aes(reorder(unigram_token, -Freq),Freq)) + geom_bar(stat = "Identity", fill = "Blue") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + labs(x="1-gram", y="Frequency", title="Top Most 20 Unigram Word")

Get top 20 most frequent words

hplot <- bigram_df[order(-bigram_df$Freq),][1:20,]
ggplot(hplot, aes(reorder(bigram_token, -Freq),Freq)) + geom_bar(stat = "Identity", fill = "Blue") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + labs(x="2-gram", y="Frequency", title="Top Most 20 Bigram Word")

Get top 20 most frequent words

hplot <- trigram_df[order(-trigram_df$Freq),][1:20,]
ggplot(hplot, aes(reorder(trigram_token, -Freq),Freq)) + geom_bar(stat = "Identity", fill = "Blue") + theme(axis.text.x = element_text(angle = 45, hjust = 1)) + labs(x="3-gram", y="Frequency", title="Top Most 20 Trigram Word")

Findings

Most of unigram words used commonly in everyday communication There are few places name appear in bigram model and all of it resides in US as our sample dataset comes from US tweet, blogs and news

Way Forward

As we already understand some basic nature of our dataset, next we have to identify how to improve our analysis performance and minimize the footprint when we run our predective text model online with shiny server. Final goal is to take 3 words from the user and predict the next word.