Week2

Data Science Capstone Week 2

The goal of this project is just to display that you’ve gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs that explains your exploratory analysis and your goals for the eventual app and algorithm. This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set. The motivation for this project is to: 1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.2. Create a basic report of summary statistics about the data sets.3. Report any interesting findings that you amassed so far.4. Get feedback on your plans for creating a prediction

Read data

Read all data for all files.

library("stringi")
library("NLP")
library("tm")
library("ngram")
library("slam")
library("xtable")
library("RWeka")
library("ggplot2")

## Registered S3 methods overwritten by 'tibble':
##   method     from  
##   format.tbl pillar
##   print.tbl  pillar

## 
## Attaching package: 'ggplot2'

## The following object is masked from 'package:NLP':
## 
##     annotate

setwd("/home/ron/Documentos/DataScience/capstone/final/en_US/")

twitterLines <- readLines("en_US.twitter.txt", encoding = "UTF-8", skipNul=TRUE)
blogslines <- readLines("en_US.blogs.txt", encoding = "UTF-8", skipNul=TRUE)
newslines <- readLines("en_US.news.txt", encoding = "UTF-8", skipNul=TRUE)

Summary for files.

stri_stats_general(twitterLines)

##       Lines LinesNEmpty       Chars CharsNWhite 
##     2360148     2360148   162096241   134082806

stri_stats_general(blogslines)

##       Lines LinesNEmpty       Chars CharsNWhite 
##      899288      899288   206824382   170389539

stri_stats_general(newslines)

##       Lines LinesNEmpty       Chars CharsNWhite 
##     1010242     1010242   203223154   169860866

Get a piece of data

Getting 10% of data and put in one vector.

sample_data<-c(sample(blogslines,length(blogslines)*0.05),
               sample(newslines,length(newslines)*0.05),
               sample(twitterLines,length(twitterLines)*0.05))

Clean Data

Remove all expression signals.

corpus<-VCorpus(VectorSource(sample_data))
corpus<-tm_map(corpus,removePunctuation)
corpus<-tm_map(corpus,stripWhitespace)
corpus<-tm_map(corpus,tolower)
corpus<-tm_map(corpus,removeNumbers)
corpus<-tm_map(corpus,PlainTextDocument)
corpus<-tm_map(corpus,removeWords,stopwords("english"))

Analyzing the information by grammarian and looking for frequency

We need to get the frequency of occurrence for a single word, a two-word expression, and a three-word expression.

one_w<-function(x) NGramTokenizer(x,Weka_control(min=1,max=1))
two_w<-function(x) NGramTokenizer(x,Weka_control(min=2,max=2))
three_w<-function(x) NGramTokenizer(x,Weka_control(min=3,max=3))

wt1 <- TermDocumentMatrix(corpus, control = list(tokenize = one_w))
wt2 <- TermDocumentMatrix(corpus, control = list(tokenize = two_w))
wt3 <- TermDocumentMatrix(corpus, control = list(tokenize = three_w))

one_corpus<-findFreqTerms(wt1,lowfreq=1000)
two_corpus<-findFreqTerms(wt2,lowfreq=80)
three_corpus<-findFreqTerms(wt3,lowfreq=10)

Plotting

For one word.

freq1<-rowSums(as.matrix(wt1[one_corpus,]))
freq1_table <- data.frame(Word=names(freq1),frequency=freq1)
freq1_table_sort <-freq1_table[order(-freq1_table$frequency),]

p1 <- ggplot(data=freq1_table_sort[1:10,], aes(x=reorder(Word, -frequency), y=frequency, fill=frequency)) + 
  geom_bar(stat = "identity")+
  labs(title="One word",x="Words",y="Frequency")+
  coord_flip()

p1

Two word expression.

freq2<-rowSums(as.matrix(wt2[two_corpus,]))
freq2_table <- data.frame(Word=names(freq2),frequency=freq2)
freq2_table_sort <-freq2_table[order(-freq2_table$frequency),]

p2 <- ggplot(data=freq2_table_sort[1:10,], aes(x=reorder(Word, -frequency), y=frequency, fill=frequency)) + 
  geom_bar(stat = "identity")+
  labs(title="Two word",x="Words",y="Frequency")+
  coord_flip()

p2

Three word expression.

freq3<-rowSums(as.matrix(wt3[three_corpus,]))
freq3_table <- data.frame(Word=names(freq3),frequency=freq3)
freq3_table_sort <-freq3_table[order(-freq3_table$frequency),]

p3 <- ggplot(data=freq3_table_sort[1:10,], aes(x=reorder(Word, -frequency), y=frequency, fill=frequency)) + 
  geom_bar(stat = "identity")+
  labs(title="Three word",x="Words",y="Frequency")+
  coord_flip()

p3

Next step

Generate a predictive algorithm based on this information, being able to verify the semantics of the sentences.