Instruction

The goal of this project is just to display that you’ve gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs (http://rpubs.com/) that explains your exploratory analysis and your goals for the eventual app and algorithm. This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set. The motivation for this project is to:

  1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.

  2. Create a basic report of summary statistics about the data sets.

  3. Report any interesting findings that you amassed so far.

  4. Get feedback on your plans for creating a prediction algorithm and Shiny app.

library(stringi) 
library(tm) 
## Loading required package: NLP
library(rJava)
library(RWeka)
library(RWekajars)
library(NLP)
library(openNLP)
library(RColorBrewer)
library(ggplot2)
## 
## Attaching package: 'ggplot2'
## The following object is masked from 'package:NLP':
## 
##     annotate
library(SnowballC)
library(qdap)
## Loading required package: qdapDictionaries
## Loading required package: qdapRegex
## 
## Attaching package: 'qdapRegex'
## The following object is masked from 'package:ggplot2':
## 
##     %+%
## Loading required package: qdapTools
## 
## Attaching package: 'qdap'
## The following objects are masked from 'package:tm':
## 
##     as.DocumentTermMatrix, as.TermDocumentMatrix
## The following object is masked from 'package:NLP':
## 
##     ngrams
## The following objects are masked from 'package:base':
## 
##     Filter, proportions
library(kableExtra)
library(simEd)
## Loading required package: rstream
## 
## Attaching package: 'simEd'
## The following objects are masked from 'package:base':
## 
##     sample, set.seed
library(ngram)
library(slam)
library(xtable)
library(wordcloud)

Reading the data and storing it with UTF-8 codification in variables. The readLines function reads the file line by line (on RStudio Cloud).

blogs <- readLines("./final/en_US/en_US.blogs.txt",  encoding = "UTF-8", skipNul = TRUE)
news <- readLines("./final/en_US/en_US.news.txt",  encoding = "UTF-8", skipNul = TRUE)
# twitter <- readLines("./final/en_US/en_US.twitter.txt",  encoding = "UTF-8", skipNul = TRUE)
# Get file sizes
blogs.size <- file.info("final/en_US/en_US.blogs.txt")$size / 1024 ^ 2
news.size <- file.info("final/en_US/en_US.news.txt")$size / 1024 ^ 2
#twitter.size <- file.info("final/en_US/en_US.twitter.txt")$size / 1024 ^ 2

lines_blogs <- length(blogs) 
lines_news <- length(news) 

library(stringr)
library(stringi)
words_blog <- sum(stri_count_words(blogs))
words_news <- sum(stri_count_words(news))

max_blogs <- max(nchar(blogs)) 
max_news <- max(nchar(news))

Summary table

Show information in table

library(kableExtra)
summary<-data.frame(c("Blog","News"), c(blogs.size, news.size), c(words_blog, words_news), c(lines_blogs,lines_news),
c(max_blogs,max_news))

kable(summary, col.names=c('File','Size (MB)', 'Words', 'Lines','Lenght of longest line'))%>%kable_styling(full_width = F)%>%column_spec(1, width="10em")%>%column_spec(2, width = "10em")%>%column_spec(3,width="10em")%>%column_spec(4,width="10em")%>%column_spec(5,width="10em")
File Size (MB) Words Lines Lenght of longest line
Blog 200.4242 37546250 899288 40833
News 196.2775 34762395 1010242 11384

Basic plots to illustrate features of the data.

1. Sampling the data and building a corpus. Since the files are very large, I will choose a subset 1% lines of each, and then I will make a corpus with the subsets.

2. Cleaning the data. The data will be cleaned by performing the following steps:

  • Remove Punctuation
  • Remove Numbers
  • Strip Whitespaces
  • Remove non english words
  • Convert everything to lower cases
  • Convert in a Plain Text Document

### 3. Tokenization of data. Three categories will be used: - Unigram (1-Gram) - Bigram (2-Gram) - Trigram (3-Gram)

An n-gram is a contiguous sequence of n items from a given sample of text or speech. In this case the items are words.

4. Some n-grams plots.

4a. Barplot of the most common words.

drawing

4b. Barplot of the most common pairs of words.

drawing

4c. Barplot of the most trios of words.

drawing