This assignment consists of a basic data analysis report for the coursera Data Science Capstone project. The purpose of this report is understanding the data which we will be working with.
Around the world, people are spending an increasing amount of time on their mobile devices for email, social networking, banking and a whole range of other activities. But typing on mobile devices can be a serious pain. SwiftKey, the corporate partner in this capstone, builds a smart keyboard that makes it easier for people to type on their mobile devices. One cornerstone of their smart keyboard is predictive text models. When someone types:
“I went to the”
The keyboard presents three options for what the next word might be. For example, the three words might be gym, store, restaurant. In this capstone we will work on understanding and building predictive text models like those used by SwiftKey.
R Libraries which we will be working with
library(RWeka)
library(tm)
library(stringi)
library(slam)
library(ggplot2)
library(gridExtra)
The data which we will be working with consist of three text files
dataBlogs <- readLines("en_US.blogs.txt")
dataTwitter <- readLines("en_US.twitter.txt")
dataNews <- readLines("en_US.news.txt")
The first step in building a predictive model for text is understanding the distribution and relationship between the words, tokens, and phrases in the text. The goal of this task is to understand the basic relationships we observe in the data and prepare to build our first linguistic models. Just to get an idea we will count the number of lines and words in each of the files.
data.frame(
File=c("Blogs","Twitter","News"),
Lines=c(length(dataBlogs),length(dataTwitter),length(dataNews)),
Words=c(sum(stri_count_words(dataBlogs)),sum(stri_count_words(dataTwitter)),sum(stri_count_words(dataNews)))
)
## File Lines Words
## 1 Blogs 899288 38154238
## 2 Twitter 2360148 30218125
## 3 News 77259 2693898
Because we will calculate multiple n-grams we defined functions so we can easily call the function multiple times with different parameters.
Create Corpus object and clean the data
cleanCorpus<-function(x) {
x <- VCorpus(VectorSource(x))
x <- tm_map(x, removeNumbers)
x <- tm_map(x, removePunctuation)
x <- tm_map(x, stripWhitespace)
x <- tm_map(x, removePunctuation,preserve_intra_word_dashes = TRUE)
x <- tm_map(x, tolower)
x <- tm_map(x, PlainTextDocument)
}
Calculate n-grams
createNgrams<-function(corpus, n, N=NA){
corpus<-cleanCorpus(corpus)
token<-function(corpus){
NGramTokenizer(corpus, Weka_control(min = n, max = n))
}
tdm <- TermDocumentMatrix(corpus, control = list(tokenize = token))
fm <- rowSums(as.matrix(tdm))
ngram<-data.frame(ngram=names(fm),freq=fm)
ngram<-ngram[order(-ngram$freq),]
if(!is.na(N)){
ngram<-head(ngram,N)
}
return(ngram)
}
combinedData = rbind(subsetBlogs,subsetTwitter,subsetNews)
oneGramBlogs <- createNgrams(subsetBlogs,1,10)
oneGramTwitter <- createNgrams(subsetTwitter,1,10)
oneGramNews <- createNgrams(subsetNews,1,10)
oneGramCombined <- createNgrams(combinedData,1,10)
twoGramBlogs <- createNgrams(subsetBlogs,2,10)
twoGramTwitter <- createNgrams(subsetTwitter,2,10)
twoGramNews <- createNgrams(subsetNews,2,10)
twoGramCombined <- createNgrams(combinedData,2,10)
threeGramBlogs <- createNgrams(subsetBlogs,3,10)
threeGramTwitter <- createNgrams(subsetTwitter,3,10)
threeGramNews <- createNgrams(subsetNews,3,10)
threeGramCombined <- createNgrams(combinedData,3,10)
Because we will plot multiple n-grams we defined a function.
Function for plotting n-grams
plot.ngram<-function(ngram_df, title){
g<-ggplot(ngram_df, aes(x=reorder(ngram, freq),y=freq)) + geom_bar(stat="Identity", fill="#FF9999") +
theme(axis.text.y = element_text(size=8))+
labs(title= title, y = "",x = "") +
theme(axis.title.y = element_text(size=8))+
theme(axis.title.x = element_text(size=8))+
guides(color=FALSE)+
coord_flip()
return(g)
}
Plot 1-grams
t1<-plot.ngram(oneGramBlogs, "Twitter: 1")
n1<-plot.ngram(oneGramTwitter, "News: 1")
b1<-plot.ngram(oneGramNews, "Blogs: 1")
c1<-plot.ngram(oneGramCombined, "Combined: 1")
grid.arrange(t1,n1,b1,c1 , ncol=4)
Plot 2-grams
t2<-plot.ngram(twoGramBlogs, "Twitter: 2")
n2<-plot.ngram(twoGramTwitter, "News: 2")
b2<-plot.ngram(twoGramNews, "Blogs: 2")
c2<-plot.ngram(twoGramCombined, "Combined: 2")
grid.arrange(t2,n2,b2,c2 , ncol=4)
Plot 3-grams
t3<-plot.ngram(threeGramBlogs, "Twitter: 3")
n3<-plot.ngram(threeGramTwitter, "News: 3")
b3<-plot.ngram(threeGramNews, "Blogs: 3")
c3<-plot.ngram(threeGramCombined, "Combined: 3")
grid.arrange(t3,n3,b3,c3 , ncol=4)