title: “Data ScienceMilestone Report” author: “Michelle Tan” date: “2/21/2018” output: html_document —
Introduction
This is the Milestone Report for the Coursera Data Science Capstone project. The goal of the capstone project is to create a predictive text model using a large text corpus of documents as training data. Natural language processing techniques will be used to perform the analysis and build the predictive model.
This milestone report describes the major features of the training data with our exploratory data analysis and summarizes our plans for creating the predictive model.
- Getting the Data
##setup derection and read file .
setwd("~/final/en_US")The data sets consist of text from 3 different sources: News, Blogs and Twitter feeds. The text data is provided in 4 different languages: English(US), German, Finnish and Russian. In this project we will focus only on the English(US) data sets.
library(NLP)
library(tm)## Warning: package 'tm' was built under R version 3.4.3blogs <- readLines("en_US.blogs.txt")
news <- readLines("en_US.news.txt")
twitter <- readLines("en_US.twitter.txt")## Warning in readLines("en_US.twitter.txt"): line 167155 appears to contain
## an embedded nul## Warning in readLines("en_US.twitter.txt"): line 268547 appears to contain
## an embedded nul## Warning in readLines("en_US.twitter.txt"): line 1274086 appears to contain
## an embedded nul## Warning in readLines("en_US.twitter.txt"): line 1759032 appears to contain
## an embedded nul# Get file sizes
blogs.size <- file.info("~/final/en_US/en_US.blogs.txt")$size/1024 ^ 2
news.size <- file.info("~/final/en_US/en_US.news.txt")$size/1024 ^ 2
twitter.size <- file.info("~/final/en_US/en_US.twitter.txt")$size/1024 ^ 2
# Get words in files
library(stringi)
blogs.words <- stri_count_words(blogs)
news.words <- stri_count_words(news)
twitter.words <- stri_count_words(twitter)
# Summary of the data sets
data.frame(source = c("blogs", "news", "twitter"),
file.size.MB = c(blogs.size, news.size, twitter.size),
num.lines = c(length(blogs), length(news), length(twitter)),
num.words = c(sum(blogs.words), sum(news.words), sum(twitter.words)),
mean.num.words = c(mean(blogs.words), mean(news.words), mean(twitter.words)))## source file.size.MB num.lines num.words mean.num.words
## 1 blogs 200.4242 899288 37546246 41.75108
## 2 news 196.2775 1010242 34762395 34.40997
## 3 twitter 159.3641 2360148 30093369 12.75063Cleaning the Data
Before performing exploratory analysis, we must clean the data first. This involves removing URLs, special characters, punctuations, numbers, excess whitespace, stopwords, and changing the text to lower case. Since the data sets are quite large, we will randomly choose 1% of the data to demonstrate the data cleaning and exploratory analysis.
library(tm)# Sample the data
set.seed(679)
data.sample <- c(sample(blogs, length(blogs) * 0.01),
sample(news, length(news) * 0.01),
sample(twitter, length(twitter) * 0.01))
# Create corpus and clean the data
corpus <- VCorpus(VectorSource(data.sample))## Warning in as.POSIXlt.POSIXct(Sys.time(), tz = "GMT"): unknown timezone
## 'zone/tz/2018c.1.0/zoneinfo/America/New_York'toSpace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
corpus <- tm_map(corpus, toSpace, "(f|ht)tp(s?)://(.*)[.][a-z]+")
corpus <- tm_map(corpus, toSpace, "@[^\\s]+")
corpus <- tm_map(corpus, tolower)
corpus <- tm_map(corpus, removeWords, stopwords("en"))
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, stripWhitespace)
corpus <- tm_map(corpus, PlainTextDocument)Exploratory Analysis We are now ready to perform exploratory analysis on the data. It would be interesting and helpful to find the most frequently occurring words in the data. Here we list the most common unigrams, bigrams, and trigrams.
library(RWeka)## Warning: package 'RWeka' was built under R version 3.4.3options(mc.cores=1)
getFreq <- function(tdm) {
freq <- sort(rowSums(as.matrix(tdm)), decreasing = TRUE)
return(data.frame(word = names(freq), freq = freq))
}
bigram <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
trigram <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
# Get frequencies of most common n-grams in data sample
freq1 <- getFreq(removeSparseTerms(TermDocumentMatrix(corpus), 0.9999))
freq2 <- getFreq(removeSparseTerms(
TermDocumentMatrix(corpus, control = list(tokenize = bigram)), 0.9999))
freq3 <- getFreq(removeSparseTerms(
TermDocumentMatrix(corpus, control = list(tokenize = trigram)), 0.9999))Here is a histogram of the 20 most common unigrams in the data sample.
library(ggplot2)##
## Attaching package: 'ggplot2'## The following object is masked from 'package:NLP':
##
## annotateggplot(freq1[1:20,], aes(reorder(word, -freq), freq)) +
labs(x = "20 Most common Unigrams", y = "Frequency") +
theme(axis.text.x = element_text(angle = 60, size = 12, hjust = 1)) +
geom_bar(stat = "identity", fill = I("blue"))
## Here is a histogram of the 20 most common bigrams in the data sample.
ggplot(freq2[1:20,], aes(reorder(word, -freq), freq)) +
labs(x = "20 Most common Bigrams", y = "Frequency") +
theme(axis.text.x = element_text(angle = 60, size = 12, hjust = 1)) +
geom_bar(stat = "identity", fill = I("blue"))
## Here is a histogram of the 20 most common trigrams in the data sample.
ggplot(freq3[1:20,], aes(reorder(word, -freq), freq)) +
labs(x = "20 Most common Trigrams", y = "Frequency") +
theme(axis.text.x = element_text(angle = 60, size = 12, hjust = 1)) +
geom_bar(stat = "identity", fill = I("blue"))
Next Steps For Prediction Algorithm And Shiny App This concludes our exploratory analysis. The next steps of this capstone project would be to finalize our predictive algorithm, and deploy our algorithm as a Shiny app.
Our predictive algorithm will be using n-gram model with frequency lookup similar to our exploratory analysis above. One possible strategy would be to use the trigram model to predict the next word. If no matching trigram can be found, then the algorithm would back off to the bigram model, and then to the unigram model if needed.