Text Mining

Overview

The goal of this project is just to display that you have gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs (http://rpubs.com/) that explains your exploratory analysis and your goals for the eventual app and algorithm. This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set.

The motivation for this project is to:

• Demonstrate that you have downloaded the data and have successfully loaded it in.
• Create a basic report of summary statistics about the data sets.
• Report any interesting findings that you amassed so far.
• Get feedback on your plans for creating a prediction algorithm and Shiny app.

Load Data

Let’s read the news, blogs and twitter data.

library(tidyverse)

blogs <- read_lines('../final/en_US/en_US.blogs.txt')
news <- read_lines('../final/en_US/en_US.news.txt')
twitter <- read_lines('../final/en_US/en_US.twitter.txt')

Summarise the Data

Let’s create a function that gives us some basic information about a certain set of generic text data.

summary.text <- function(name, data) {
  nchars <- lapply(data, nchar)
  result <- tibble(Data = name,
                   `Size(Mb)` = round(as.numeric(object.size(data))/1024^2, 2),
                   `#Lines` = length(data),
                   `Min. Char` = nchars[[which.min(nchars)]],
                   `Max. Char` = nchars[[which.max(nchars)]],
                   `Word Count` = sum(sapply(strsplit(data, "\\s+"), length)))
  
  return(result)
}

Let’s get some basic information and statistics from the data sets read.

library(knitr)

kable(bind_rows(summary.text('Blogs', blogs),
                summary.text('News', news),
                summary.text('Twitter', twitter)), 
      align = 'lrrrrr', format.args = list(big.mark = ","))

Data	Size(Mb)	#Lines	Min. Char	Max. Char	Word Count
Blogs	255.35	899,288	1	40,833	37,334,131
News	257.34	1,010,242	1	11,384	34,372,530
Twitter	318.99	2,360,148	2	140	30,373,543

Preprocessing Data

Let’s transform the data into a document and create a corpus. After, we will perform a series of techniques for cleaning text, such as removing spaces, stopwords, etc.

library(tm)

preprocess <- function(data) {
  result <- VCorpus(VectorSource(sample(data, 1000)))
  result <- result %>%
    tm_map(content_transformer(function(x) {
      gsub("/|@|//|$|:|:)|*|&|!|?|_|-|#|", " ", x)})) %>%
    tm_map(content_transformer(tolower)) %>%
    tm_map(removeNumbers) %>%
    tm_map(removePunctuation) %>%
    tm_map(removeWords, stopwords()) %>%
    tm_map(stemDocument) %>%
    tm_map(stripWhitespace)
  return(result)
}

corpus <- preprocess(c(blogs, news, twitter))

Exploratory Charts

In exploratory analysis we will visualize the top 10 most frequent n-grams, with \(n \in \{1,2,3\}\).

High Frequency 1-Gram

For the creation of the graph we will convert it into a matrix and add the lines to have the frequency.

matrix_1gram <- corpus %>%
    TermDocumentMatrix() %>%
    as.matrix() %>%
    rowSums()

df_1gram <- tibble(word = names(matrix_1gram), freq = matrix_1gram)

df_1gram %>%
  arrange(desc(freq)) %>%
  head(10) %>% 
  ggplot(aes(word, freq)) +
  geom_bar(stat="identity") +
  coord_flip() + theme_bw() +
  theme(axis.text.x=element_text(angle=45, hjust=1)) + 
  ylab('Frequency') + xlab('1-Gram') + ggtitle("Top 10 Most Frequent 1-Gram") 

High Frequency 2-Gram

For the creation of the graph we will apply the function to tokenize and take 2-gram and after converting it into a matrix and make the sum of the lines to have the frequency.

library(RWeka)

as_2gram <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))

matrix_2gram <- TermDocumentMatrix(corpus, control = list(tokenize = as_2gram))

matrix_2gram <- matrix_2gram %>%
    as.matrix() %>%
    rowSums()

df_2gram <- tibble(word = names(matrix_2gram), freq = matrix_2gram)

df_2gram %>%
  arrange(desc(freq)) %>%
  head(10) %>% 
  ggplot(aes(word, freq)) +
  geom_bar(stat="identity") +
  coord_flip() + theme_bw() +
  theme(axis.text.x=element_text(angle=45, hjust=1)) + 
  ylab('Frequency') + xlab('1-Gram') + ggtitle("Top 10 Most Frequent 2-Gram") 

High Frequency 3-Gram

For the creation of the graph we will apply the function to tokenize and take 3-gram and after converting it into a matrix and make the sum of the lines to have the frequency.

as_3gram <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))

matrix_3gram <- TermDocumentMatrix(corpus, control = list(tokenize = as_3gram))

matrix_3gram <- matrix_3gram %>%
    as.matrix() %>%
    rowSums()

df_3gram <- tibble(word = names(matrix_3gram), freq = matrix_3gram)

df_3gram %>%
  arrange(desc(freq)) %>%
  head(10) %>% 
  ggplot(aes(word, freq)) +
  geom_bar(stat="identity") +
  coord_flip() + theme_bw() +
  theme(axis.text.x=element_text(angle=45, hjust=1)) + 
  ylab('Frequency') + xlab('3-Gram') + ggtitle("Top 10 Most Frequent 3-Gram") 

Feedback

As the bag of words algorithm and in general the techniques require a lot of processing and storage power, we use a small sample… As future work I intend to use multiple n-grams in prediction model and apply some technique that allows more data to be used without compromising the computational efficiency for the shiny app.