Executive Summary:

This milestone report is part of the Johns Hopkins - Coursera Data Science Specialization Capstone project. The course objective is to apply data science in the area of natural language processing. The final result of the course is to build a Shiny application with a text predictive algorithm that accepts some text inputed by the user and try to predict what the next word will be.

In this milestone report we show the exploratory analysis we have performed on the data sets we have used, and summarize our plans for building the predictive model and the Shiny app.

Libraries:

library(knitr)
library(ggplot2)
library(stringi)
library(tm)
library(RWeka)
library(NLP)
library(base)

Getting Data:

The dataset are provided in 4 different languages: Finnish, English, Russian and Germal. Also, the data sets consist of text from 3 different sources: News, Blogs and Twitter. In this report, we will only focus on the English data sets. The data can be downloaded from the following URL:

https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip

setwd('/Users/GyanGM/Downloads/Coursera/Final Capstone Project/Coursera-SwiftKey/en_US')

twitter_location <- './en_US.twitter.txt'
news_location <- './en_US.news.txt'
blogs_location <- './en_US.blogs.txt'

data_blogs <- readLines(blogs_location, encoding="UTF-8", skipNul = TRUE)
data_news <- readLines(news_location, encoding="UTF-8", skipNul = TRUE)
data_twitter <- readLines(twitter_location, encoding="UTF-8", skipNul = TRUE)

```

After saving the data in a local directory, it is time to start the analysis as it is shown in the following sections.

Data Summary:

In the table below we can see some properties of the data.

bsummary <- data.frame(Dataset = c("Blogs", "News", "Tweets"),
                       Filesize_bytes = c(file.size("en_US.blogs.txt"),
                                      file.size("en_US.news.txt"),
                                      file.size("en_US.twitter.txt")),
                       Lines = c(length(data_blogs),
                                       length(data_news),
                                       length(data_twitter)),
                       Words = c(sum(sapply(strsplit(data_blogs, " "), FUN=length, simplify = TRUE)),
                                       sum(sapply(strsplit(data_news, " "), FUN=length, simplify = TRUE)),
                                       sum(sapply(strsplit(data_twitter, " "), FUN=length, simplify = TRUE))))

bsummary
##   Dataset Filesize_bytes   Lines    Words
## 1   Blogs             NA  899288 37334131
## 2    News             NA 1010242 34372530
## 3  Tweets             NA 2360148 30373583

Data Cleaning:

Before procedding to create models of given datasets, we need to clean the data set. We will remove special characters, whitespaces, URLS and other non-import characters from the dataset. For this report, we will take sample data from our data set. We will be using whole date set for latter project.

# Sets the default number of threads to use
  options(mc.cores=1)

# Sample the data
set.seed(420)
data.sample <- c(sample(data_blogs, length(data_blogs) * 0.01),
                 sample(data_news, length(data_news) * 0.01),
                 sample(data_twitter, length(data_twitter) * 0.01))


# Create corpus and clean the data
corpus <- VCorpus(VectorSource(data.sample))
toSpace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
corpus <- tm_map(corpus, toSpace, "(f|ht)tp(s?)://(.*)[.][a-z]+")
corpus <- tm_map(corpus, toSpace, "@[^\\s]+")
corpus <- tm_map(corpus, tolower)
corpus <- tm_map(corpus, removeWords, stopwords("en"))
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, PlainTextDocument)

N-grams:

# prepare the word n-gram data
my_corpus <-data.frame(text = unlist(sapply(corpus, `[`, "content")), 
                        stringsAsFactors = FALSE)

findNGrams <- function(corp, grams, top) {
        ngram <- NGramTokenizer(corp, Weka_control(min = grams, max = grams,
                                                   delimiters = " \\r\\n\\t.,;:\"()?!"))
        ngram <- data.frame(table(ngram))
        ngram <- ngram[order(ngram$Freq, decreasing = TRUE),][1:top,]
        colnames(ngram) <- c("Words","Count")
        ngram
}

mono_grams   <- findNGrams(my_corpus, 1, 10)
bi_grams     <- findNGrams(my_corpus, 2, 10)
tri_grams    <- findNGrams(my_corpus, 3, 10)
quad_grams   <- findNGrams(my_corpus, 4, 10)

Exploratory Data Analysis:

As we have developed four different n-gram model, it would be very helpful to see the most occuring grams in those models.

makePlot <- function(data, label) {
  ggplot(data[1:10,], aes(reorder(Words, -Count), Count)) +
         labs(x = label, y = "Frequency") +
         theme(axis.text.x = element_text(angle = 60, size = 12, hjust = 1)) +
         geom_bar(stat = "identity", fill = I("brown"))
}

makePlot(mono_grams, "10 Most Common Mono-grams")

makePlot(bi_grams, "10 Most Common Bi-grams")

makePlot(tri_grams, "10 Most Common Tri-grams")

makePlot(quad_grams, "10 Most Common Quad-grams")

Next Steps

So far we performed exploratory data analysis for given data set. The next challenge would be build a predictive model, evaluate that model and build a friendly UI in shiny.

Our predictive algorithm will be using n-gram model with frequency lookup combined with logistic regression. One possible strategy would be to use the trigram model to predict the next word. If no matching trigram can be found, then the algorithm would back off to logistic regression to predict the word. We will try to drop words and n-grams with low frequency (too many and too little to add the value). We will not try to predict numbers but try to predict words that may come after a number in general.

The user interface of the Shiny app will consist of a text input box that will allow a user to enter a phrase. Then the app will use our algorithm to suggest the most likely next word after a short delay. Our plan is also to allow the user to configure how many words our app would suggest.