This is the Milestone Report for week 2 of the Coursera Data Science Capstone project. In this document, we will look at the major features and properties of the data that will become useful later on to produce a predictive language algorithm.

The model will be trained using English text data from three sources: blogs, news, and Twitter.

Loading packages

library(R.utils)
library(dplyr)
library(stringi)
library(ggpubr)
library(RWeka)
library(tm)

Loading in the Data

twitter <- readLines(con <- file("en_US.twitter.txt"), encoding = "UTF-8", skipNul = TRUE) %>% as_tibble() %>% mutate(num_char = nchar(value)) %>% mutate(num_words = stri_count_words(value))
close(con)
blogs <- readLines(con <- file("en_US.blogs.txt"), encoding = "UTF-8", skipNul = TRUE) %>% as_tibble() %>% mutate(num_char = nchar(value)) %>% mutate(num_words = stri_count_words(value))
close(con)
news <- readLines(con <- file("en_US.news.txt"), encoding = "UTF-8", skipNul = TRUE) %>% as_tibble() %>% mutate(num_char = nchar(value)) %>% mutate(num_words = stri_count_words(value))
## Warning in readLines(con <- file("en_US.news.txt"), encoding = "UTF-8", :
## incomplete final line found on 'en_US.news.txt'
close(con)

Data summary

First, we will perform a simple and general exploration of the features of the data, looking at the size of the files and the lines, characters and words that compose them.

file <- c("blogs","news","twitter")
megabytes <- c(object.size(blogs), object.size(news), object.size(twitter))/1000000
num_lines <- c( dim(blogs)[1], dim(news)[1], dim(twitter)[1])
total_characters <- c(sum(blogs$num_char), sum(news$num_char), sum(twitter$num_char))
mean_characters <- c(mean(blogs$num_char), mean(news$num_char), mean(twitter$num_char))
total_words <- c(sum(blogs$num_words), sum(news$num_words), sum(twitter$num_words))
mean_words <- c(mean(blogs$num_words), mean(news$num_words), mean(twitter$num_words))

mean_words <- 
table <- data.frame(file, megabytes, num_lines, total_characters, mean_characters, total_words, mean_words)

print(table)
##      file megabytes num_lines total_characters mean_characters total_words
## 1   blogs 274.95402    899288        206824505       229.98695    37546250
## 2    news  21.34864     77259         15639408       202.42830     2674536
## 3 twitter 353.36726   2360148        162096241        68.68054    30093413
##   mean_words
## 1   41.75109
## 2   34.61779
## 3   12.75065

We can see that the mean words and characters per line for the three sources are relatively low, with blogs being the highest and twitter being the lowest (which makes sense due to character limits for tweets). The twitter file has the highest number of lines (due to them being individual tweets), making it the greatest size. Nonetheless, the blogs file has the greatest total words and characters.

twt_hist <- ggplot(data=twitter, aes(x = num_words)) + geom_histogram()  + labs(x = "Number of words per line", y= "Frequency", title="Twitter words per line")
blg_hist <- ggplot(data=blogs, aes(x = num_words)) + geom_histogram()  + labs(x = "Number of words per line", y= "Frequency", title="Blog words per line")
nws_hist <- ggplot(data=news, aes(x = num_words)) + geom_histogram()  + labs(x = "Number of words per line", y= "Frequency", title="News words per line")

ggarrange(twt_hist,blg_hist, nws_hist, ncol=3)
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

In these histograms, we can see that the distribution of number of words per line tends to be skewed towards smaller values for the three files. In the case of twitter, it’s more evenly distributed, due to the small possible range of values. For blogs and news, there are few lines with high values.

#Sampling of the data

Since the data sets are so large, we will proceed to selecting a sample from each of them (15% of the total) to perform more exploration of the data’s properties. From these samples, we will remove all strange (non-latin) symbols. Finally, we will join the three samples to generate our data set. Afterwards, we will write out the words in lowercase letters, remove punctuation and numbers, and remove white spaces.

set.seed(898)
sampleb <- sample_n(blogs, dim(blogs)[1]*0.15)
samplen <- sample_n(news, dim(news)[1] * 0.15)
samplet <- sample_n(twitter, dim(twitter)[1] * 0.15)

sampleb <- iconv(sampleb, "latin1", "ASCII", sub = "")
samplen <- iconv(samplen, "latin1", "ASCII", sub = "")
samplet <- iconv(samplet, "latin1", "ASCII", sub = "")


total_data <- rbind(sampleb, samplen, samplet)
total_data <- Corpus(VectorSource(total_data))

total_data<- tm_map(total_data, tolower)
total_data<- tm_map(total_data, removePunctuation)
total_data<- tm_map(total_data, removeNumbers)
total_data<- tm_map(total_data, stripWhitespace)

Tokenizing

Now that we have our corpus, we can tokenize it using the RWeka package to turn it into a numerical data structure suitable for machine learning. We will generate tokenizing functions for unigrams, bigrams and trigrams.

unigrams <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
bigrams <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
trigrams <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))

Most common unigrams, bigrams and trigrams

Here, we will graphically show the most common unigrams, bigrams and trigrams in our corpus. First, we must apply the tokenizing functions generated above, get the frequencies of the most common n-grams and reorder them from greatest to smallest.

unigram_m <- TermDocumentMatrix(total_data, control = list(tokenize = unigrams))

uni_freq <- sort(rowSums(as.matrix(removeSparseTerms(unigram_m, 0.99))), decreasing = TRUE)
uni_freq <- data.frame(uni = names(uni_freq), freq = uni_freq)
uni_freq <- uni_freq[order(uni_freq$freq, decreasing=TRUE),]

bigram_m <- TermDocumentMatrix(total_data, control = list(tokenize = bigrams))

bi_freq <- sort(rowSums(as.matrix(removeSparseTerms(bigram_m, 0.99))), decreasing = TRUE)
bi_freq <- data.frame(bi = names(bi_freq), freq = bi_freq)
bi_freq <- bi_freq[order(bi_freq$freq, decreasing=TRUE),]

trigram_m <- TermDocumentMatrix(total_data, control = list(tokenize = trigrams))

tri_freq <- sort(rowSums(as.matrix(removeSparseTerms(trigram_m, 0.99))), decreasing = TRUE)
tri_freq <- data.frame(tri = names(tri_freq), freq = tri_freq)
tri_freq <- tri_freq[order(tri_freq$freq, decreasing=TRUE),]

Now we can finally plot these! We will only grab the 15 most common instances of each type so the graphs don’t look oversaturated.

uni <- ggplot(data=uni_freq[1:15,], aes(x=reorder(uni, -freq), y=freq)) + geom_bar(stat="identity", fill="darkgoldenrod") + labs(x="Unigrams", y="Frequency", title="15 most common unigrams in the corpus")

bi <- ggplot(data=bi_freq[1:15,], aes(x=reorder(bi, -freq), y=freq)) + geom_bar(stat="identity", fill="violet") + labs(x="Bigrams", y="Frequency", title="15 most common Bigrams in the corpus")

tri <- ggplot(data=tri_freq[1:15,], aes(x=reorder(tri, -freq), y=freq)) + geom_bar(stat="identity", fill="seagreen") + labs(x="Trigrams", y="Frequency", title="15 most common trigrams in the corpus")

ggarrange(uni,bi,tri, nrow=3, heights=)

What comes next??

Now that we have performed this general exploratory analysis of our data sets, we will be trying to develop a predictive algorithm that will work as a Shiny app, taking a sentence as input and giving the next word as an output.

The predictive model shall use an n-gram algorithm with a frequency lookup in the data of the corpus, trying to determine the probability of following appareance of each word. Since we have more data for unigrams than bigrams and trigrams, it might be ideal to predict the next word using the unigrams and then the other two.