Introduction

This report aims to explain my exploratory analysis and the major features of the data and briefly summarizes my plans for creating the prediction algorithm and Shiny app in an understandaable way.

The first step is to download the data with which, in the future, the model for the Shiny app is going to be trained.

The second step is to clean the data, construct word corpora (plural of corpus which is a large and structured set of texts that are used for linguistic analysis and natural language processing).

The third step is to perform exploratory data analysis.

Dependencies

First and foremost we have to load the packages that we need.

library(dplyr)
library(stringi) 
library(tm)
library(RWeka)
library(SnowballC)

Downloading Data

The data that we are going to be using can be downloaded from this link.

Considering I don’t know Greman, Finnish, or Russian I’ll only work with the following files, which correspond to the en_US directory’s content:

## [1] "en_US.blogs.txt"           "en_US.news.txt"           
## [3] "en_US.news_cleaned.txt"    "en_US.twitter.txt"        
## [5] "en_US.twitter_cleaned.txt"

Parsing Data

(READ THE COMMENTS IN THE CODE)

# Reads Lines of blogs file and saves the text
blogs <- readLines("final/en_US/en_US.blogs.txt", encoding = "UTF-8", skipNul=TRUE)

# Reads Lines of news file using binary mode becasue there are encoded characters in text and saves said text
con1 <- file("final/en_US/en_US.news.txt", open = "rb")
news <- readLines(con1, encoding = "UTF-8")
close(con1)

# Removes all special characters that aren't letters, digits or a space character or any of the following:
# ,.!%-=+()?"^~*:\\@#$;><
news_cleaned <- gsub('[^a-zA-Z0-9 ,.!%-=+()?"^~*:\\@#$;><]', "", news)

# Writes the cleaned text into a new file (to have for later use)
writeLines(news_cleaned, "final/en_US/en_US.news_cleaned.txt")


# Same process with twitter file
con2 <- file("final/en_US/en_US.twitter.txt", open = "rb")
twitter <- readLines(con2, encoding = "UTF-8", skipNul=TRUE)

twitter_cleaned <- gsub('[^a-zA-Z0-9 ,.!%-=+()?"^~*:\\@#$;><]', "", twitter)
writeLines(twitter_cleaned, "final/en_US/en_US.twitter_cleaned.txt")
close(con2)

# To make my life simpler I'm going to pass the values of the cleaned data to the old variables:
news <- news_cleaned
twitter <- twitter_cleaned

Summarizing Data

Firstly, let’s peek at the files’ properties.

##      File Size_MB Line_Count Word_Count
## 1   Blogs  200.42     899288   37510767
## 2 Twitter  158.92    2360148   30086940
## 3    News  195.44    1010242   34749439

Secondly, let’s see some General Statistics about the different files with: stri_stats_general() and the summary of stri_count_words() which are functions of the package ‘stringi’. The first function returns an integer vector with the following named elements:

Blogs:

##       Lines LinesNEmpty       Chars CharsNWhite 
##      899288      899165   206043906   169609063
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.00    9.00   28.00   41.71   60.00 6725.00

News:

##       Lines LinesNEmpty       Chars CharsNWhite 
##     1010242     1010228   202911237   169548949
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##     0.0    19.0    31.0    34.4    46.0  1796.0

Twitter:

##       Lines LinesNEmpty       Chars CharsNWhite 
##     2360148     2360148   161921775   133908340
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00    7.00   12.00   12.75   18.00   47.00

Data Sampling

Because these files are so large, we are going to sample 10,000 lines from the Blogs, News and Twitter files using sample() to improve data processing speeds. The file containing the samples is called ‘samples_file’.

These are the properties of the samples file:

##    Source Size_MB Line_Count Word_Count
## 1 Samples    4.78      30000     882579

and the General Statistics:

##       Lines LinesNEmpty       Chars CharsNWhite 
##       30000       29998     4953423     4105291
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    0.00    9.00   20.00   29.42   39.00  602.00

Corpora Creation

We create a corpus from the samples file and use the text mining library tm to perform the following transformations:

Exploratory Data Analysis

N-Gram-Model Building

We search for 1-grams, 2-grams and 3-grams to calculate word frequencies and identify relationships between words.

With the following histograms we can see the frequencies of the most common n-grams in our corpus.

Histogram Creation

[1] "1-grams - Top 10 Most Frequent"
     word freq
said said 2953
will will 2791
one   one 2657
like like 2377
just just 2257
get   get 2237
time time 2067
can   can 2059
year year 1995
day   day 1720

[1] "2-grams - Top 10 Most Frequent"
                   word freq
last year     last year  204
new york       new york  200
dont know     dont know  177
right now     right now  145
look like     look like  141
year ago       year ago  141
feel like     feel like  129
last week     last week  127
high school high school  125
im go             im go  113

[1] "3-grams - 10 Most Frequent"
                                   word freq
new york citi             new york citi   37
caprera hotel venic caprera hotel venic   28
hotel venic itali     hotel venic itali   28
rain rain rain           rain rain rain   26
happi mother day       happi mother day   24
cant wait see             cant wait see   19
st loui counti           st loui counti   18
look forward see       look forward see   16
new york time             new york time   14
beat la beat               beat la beat   13

Plans for Shiny App