Executive Summary

This is an Milestone Report for the Capstone Project of the Data Science Specialization. In this report, we will document the initial stage of works towards developing an app for predicting the next words based on words typed previously. We will perform exploratory analysis and data cleaning on the data set, and we will summarize our observations and outline future works.

The Data Set, Exploratory Analysis and Data Cleaning

The data set is called the HC Corpora, which was collected from the web by a web crawler. For this app we will be using 3 files containing entries from English language blogs, twitter entries and news. For more information please see the web archive for “About the Corpora”.

For the analysis we downloaded the English corpora and save the files locally. There are 3 files: en_US.blogs.txt, en_US.news.txt and en_US.twitter.txt. As a start, we shows basic information of the filee: file size, number of lines and the number of words contained in the file, to give a general understanding of the source data.

library(tidyr)
library(readr)
library(stringi)
library(ngram)
library(ggplot2)
library(wordcloud)
library(SnowballC)

setwd("~/Documents/R Course/Capstone Project/final/en_US/")

print_file_detail <- function(input){
  {fileconnect <- file(inputfile)
  linesinfile <- readLines(fileconnect, n=-1)
  filenooflines <- length(linesinfile)
  filesize <- format (object.size(linesinfile), units="Mb")
  filewordcount <-sum(stri_count_words(linesinfile))  
  
  close(fileconnect)
  
  cat("File Name",inputfile, "File Size", filesize,
      "No of Lines", filenooflines,"Words Count", filewordcount, "\n")}
}

inputfile <- "en_US.blogs.txt"
print_file_detail(inputfile)
## File Name en_US.blogs.txt File Size 255.4 Mb No of Lines 899288 Words Count 37546250
inputfile <- "en_US.news.txt"
print_file_detail(inputfile)
## File Name en_US.news.txt File Size 257.3 Mb No of Lines 1010242 Words Count 34762395
inputfile <- "en_US.twitter.txt"
print_file_detail(inputfile)
## File Name en_US.twitter.txt File Size 319 Mb No of Lines 2360148 Words Count 30093372

In total, we have around 830 Mb of data, 4.3 million lines of text and about 102 million words.

The R package ngram provides a quick way to load several files using the multiread() function and join them together as a single string for further process:

#using multiread in ngram library to read file

xstring <- multiread("~/Documents/R Course/Capstone Project/final/en_US/", 
                     extension = "txt")

#merge the 3 string into single one for ngram's processing
xstring <- concatenate(xstring)

A good way to visualize the content is to produce a word cloud, to highlight words that most frequently appear in the text. But first we want to perform at least some pre-processing and tidy up the data. The ngram package provides a useful function preprocess() for this purpose. Here, we change all words to lower case, remove punctuation and numbers, and also remove extra spaces between words. This is a pretty minimum set of pre-processing, and we will discuss other processing that could be necessary in later section.

#preprocess to lower case, remove numbers, punctuation and duplicating spcace

xstring <- preprocess(xstring, case="lower", remove.punct = TRUE, 
                      remove.numbers = TRUE, fix.spacing = TRUE)

#summary to check preprocessing is working

string.summary(xstring)
## Chars:       548140600
## Letters:     448307449
## Whitespace:  99651217
## Punctuation: 0
## Digits:      0
## Words:       99651218
## Sentences:   0
## Lines:       1 
## Wordlens:    3170889 4102089 4351856 4756900 7431114 8628280 11773216 16432122 18973670 20031082 
##              1 1 1 1 1 1 1 1 1 1 
## Senlens:     0 
##              10 
## Syllens:     4126 5366 10172 24892 109472 706075 3162150 9381919 25886099 59424593 
##              1 1 1 1 1 1 1 1 1 1
#create list of words
xwords <- unlist(strsplit(xstring, " "))

#compute word frequency
word_freq <- table(xwords)

#create a word cloud
wordcloud(names(word_freq), freq = word_freq, min.freq = 10, max.words = 200, 
          random.order = FALSE, colors = brewer.pal(8, "Dark2"))

As expected we see the most common words are the articles like “the”, “a”, or “i”, “you”, which gives some assurance that our data are not exceptional samples, but align to ordinary texts.

From here we would want to generate some n-grams and plot the frequency charts, to see whether the frequency plots show any abnormal pattern:

#create 2- & 3-gram
bigrams <- ngram(xstring, n=2)

After generating the 2-grams and 3-grams, we can sort thm by descending order of frequency and obtain frequency plots:

temp <- get.phrasetable(bigrams)
data <- data.frame(order=seq_along(temp$freq), freq=temp$freq)
    
    ggplot(data, aes(x=order, y=freq)) +
      geom_line() + scale_x_log10() + scale_y_log10() +
      labs(title="Plot of 2-gram Frequency", 
           x="Order (Log-scale)", y="Frequecny (Log-Scale)")

#remove the relevant variables after plot
rm(bigrams, temp, data)

And here is the similar frequency chart for 3-gram:

#create 2- & 3-gram
trigrams <- ngram(xstring, n=3)
temp <- get.phrasetable(trigrams)
data <- data.frame(order=seq_along(temp$freq), freq=temp$freq)
    
    ggplot(data, aes(x=order, y=freq)) +
      geom_line() + scale_x_log10() + scale_y_log10() +
      labs(title="Plot of 3-gram Frequency", 
           x="Order (Log-scale)", y="Frequecny (Log-Scale)")

#remove the relevant variables after plot
rm(trigrams, temp, data)

While no attempt is made to fit a trend line for the frequency plots, it is quite clear that both the 2-gram and 3-gram can be approximated by a power law distribution, consistent with the Zipf’s law.

Discussions and Future Works

The exploratory analysis gives us some confidence that this is a good set of data as a foundation to build our app.

There are a few decisions to be made down the road: - whether the data need to be further processed; for instance, in this report we had not done any stemming, which is probably something we need to do in next steps - choosing a appropriate modeling approach, and the rationale behind. An n-gram model could be sufficient for the purpose, we should also develop criteria to test for the appropriate value of n. - as n increase the computation resources grows exponentially; there are also other considerations e.g., the sparsity issue for large n; experience also showed that the result may not improve significantly beyond certain value of n.