This milestone report is part of the coursera Data science specialization capstone project. The goal of the project is to develop a predictive algorithm using a large text data set provided by swifty. During this report I will perform an exploratory data analysis using specific R packages created for text processing. The output of the analysis will be frequency plots showing the most frequent Unigrams, Bigrams and Trigrams in the provided training dataset. I will also conclude the report by expressing some ideas about the final shiny app and the predictive algorithm.
The text data set was provided by Swifty under the following link: https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip. I will now load some of the packages that I will need for the analysis and I will also create three variables (blogs, news, twitter) in which I will store the text lines provided in the dataset.
library(plyr)
library(dplyr)
library(knitr)
library(stringi)
library(RWeka)
library(ggplot2)
library(tm)
library(NLP)
library(rJava)
library(stringi)
library(stringr)
library(plotly)
library(wesanderson)
setwd("~/Santiago/Data science/final/en_US")
twitter<-readLines("en_US.twitter.txt",warn=FALSE,encoding="UTF-8")
blogs<-readLines("en_US.blogs.txt",warn=FALSE,encoding="UTF-8")
news<-readLines("en_US.news.txt",warn=FALSE,encoding="UTF-8")
Given the huge size of the dataset, I will take a sample of 1% that will help me improve the performance during the analysis. I will also clean the data on each variable by removing special characters, white spaces, upper cases, numbers and URLs and stopwords. This cleaning process will then allow me to identify properly the most common combinations of words on the data sets.
set.seed(12)
blogs_1<-iconv(blogs,"latin1","ASCII",sub="")
news_1<-iconv(news,"latin1","ASCII",sub="")
twitter_1<-iconv(twitter,"latin1","ASCII",sub="")
head(news_1)
## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."
## [4] "The Alaimo Group of Mount Holly was up for a contract last fall to evaluate and suggest improvements to Trenton Water Works. But campaign finance records released this week show the two employees donated a total of $4,500 to the political action committee (PAC) Partners for Progress in early June. Partners for Progress reported it gave more than $10,000 in both direct and in-kind contributions to Mayor Tony Mack in the two weeks leading up to his victory in the mayoral runoff election June 15."
## [5] "And when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. Is it absolutely necessary? Is it an issue serious enough to merit their attention? Will it definitely not make the situation worse?"
## [6] "There was a certain amount of scoffing going around a few years ago when the NFL decided to move the draft from the weekend to prime time -- eventually splitting off the first round to a separate day."
sampledata<-c(sample(twitter_1,length(twitter_1)*0.01),
sample(blogs_1,length(blogs_1)*0.01),
sample(news_1,length(news_1)*0.01))
corpus <- VCorpus(VectorSource(sampledata))
toSpace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
corpus <- tm_map(corpus, toSpace, "(f|ht)tp(s?)://(.*)[.][a-z]+")
corpus <- tm_map(corpus, toSpace, "@[^\\s]+")
corpus <- tm_map(corpus, tolower)
corpus <- tm_map(corpus, removeWords, stopwords("en"))
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, PlainTextDocument)
corpusresult<-data.frame(text=unlist(sapply(corpus,'[',"content")),stringsAsFactors = FALSE)
head(corpusresult)
## text
## 1 follow yea
## 2 begin see living world luminous eyes perception world changes
## 3 best food door county
## 4 roses red violets blue hate people
## 5 lol just matter trying make finish line man
## 6 check great place hangout houston tx
Now that I cleaned the data I will use the package tm and ggplot 2 to process the data and to create the corresponding graphs. I will create a corpus containing structured data from the three initial variables that I created from the dataset. I will also use the ggplotlty package to turn the graphs into a plotly element. That should help the readers extract more information from the graph.
unigram<-function(x) NGramTokenizer(x,Weka_control(min=1,max=1))
unigramtab<-TermDocumentMatrix(corpus,control=list(tokenize=unigram))
unigramcorpus<-findFreqTerms(unigramtab,lowfreq=1000)
unigramcorpusnum<-rowSums(as.matrix(unigramtab[unigramcorpus,]))
unigramcorpustab<-data.frame(Word=names(unigramcorpusnum),frequency=unigramcorpusnum)
unigramcorpussort<-unigramcorpustab[order(-unigramcorpustab$frequency),]
A<-ggplot(unigramcorpussort[1:15,],aes(x=reorder(Word,-frequency),y=frequency))+
geom_bar(stat="identity",fill = I("Orange"))+
labs(title="Unigrams",x="Most Words",y="Frequency")+
theme(axis.text.x=element_text(angle=60))
ggplotly(A)
bigram<-function(x) NGramTokenizer(x,Weka_control(min=2,max=2))
bigramtab<-TermDocumentMatrix(corpus,control=list(tokenize=bigram))
bigramcorpus<-findFreqTerms(bigramtab,lowfreq=80)
bigramcorpusnum<-rowSums(as.matrix(bigramtab[bigramcorpus,]))
bigramcorpustab<-data.frame(Word=names(bigramcorpusnum),frequency=bigramcorpusnum)
bigramcorpussort<-bigramcorpustab[order(-bigramcorpustab$frequency),]
B<-ggplot(bigramcorpussort[1:12,],aes(x=reorder(Word,-frequency),y=frequency))+
geom_bar(stat="identity",fill = I("orange"))+
labs(title="Bigrams",x="Most Words",y="Frequency")+
theme(axis.text.x=element_text(angle=60))
ggplotly(B)
trigram<-function(x) NGramTokenizer(x,Weka_control(min=3,max=3))
trigramtab<-TermDocumentMatrix(corpus,control=list(tokenize=trigram))
trigramcorpus<-findFreqTerms(trigramtab,lowfreq=10)
trigramcorpusnum<-rowSums(as.matrix(trigramtab[trigramcorpus,]))
trigramcorpustab<-data.frame(Word=names(trigramcorpusnum),frequency=trigramcorpusnum)
trigramcorpussort<-trigramcorpustab[order(-trigramcorpustab$frequency),]
C<-ggplot(trigramcorpussort[1:10,],aes(x=reorder(Word,-frequency),y=frequency,fill=frequency()))+
scale_fill_brewer(palette="Dark2")+
geom_bar(stat="identity",fill = I("orange"))+
labs(title="Trigrams",x="Most Words",y="Frequency")+
theme(axis.text.x=element_text(angle=60))
ggplotly(C)
Now that I have created the graphs and looked at the data I can conclude the exploratory analysis and move forward to creating the predictive algorithm and the Shiny app. The algorithm should take the most frequent trigrams to try to come up with a prediction. If this model is unable to come up with a match the algorithm would then look on the bigrams and finally on the unigrams. The Shiny app should have a simple text box in the left panel where the user could introduce a simple phrase and an empty box on the main panel where the prediction will be displayed.