Executive summary

This report is realized under the Data Science Capstone project and contains exploratory analysis of the data with the following goal:

For this capstone project the data is from a corpus called HC Corpora. Datasets are provided in four different languages: English (en_US), German (de_DE), Russian (de_DE) and Finnish (fi_FI), but the analysis will be performed on the English data, which consists of three files collected from news sites, blogs, and twitter, as follow:

Download and unzip file

The source dataset is downloaded, stored and unzip

datasetUrl <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"

if(!file.exists("./swiftkey.zip")){
  download.file(datasetUrl,destfile="swiftkey.zip")
}

if(!file.exists("./data/final")){
  unzip(zipfile="./swiftkey.zip",exdir="./data")
}

Summary of datasets

The summary of the English data that is used for the exploratory anlysis is presented below:

library(stringi)

## Read lines of the English files
twitter.file <- "./data/final/en_US/en_US.twitter.txt"
blogs.file <- "./data/final/en_US/en_US.blogs.txt"
news.file <- "./data/final/en_US/en_US.news.txt"
twitter <- readLines(twitter.file, skipNul = TRUE, encoding = "UTF-8")
blogs <- readLines(blogs.file, skipNul = TRUE, encoding = "UTF-8")
news <- readLines(news.file, skipNul = TRUE, encoding = "UTF-8")

## File size
blogs.size <- file.info(blogs.file)$size / 1024 ^ 2
news.size <- file.info(news.file)$size / 1024 ^ 2
twitter.size <- file.info(twitter.file)$size / 1024 ^ 2

## Summary table
summaryTable <- function(twitter,blog,news){
  stats <- data.frame(source = c("Twitter","Blog","News"),
            fileSize_mb = c(twitter.size, blogs.size, news.size),
            lineCount = c(length(twitter),length(blogs),length(news)),
            wordCount = c(sum(stri_count_words(twitter)),sum(stri_count_words(blogs)),sum(stri_count_words(news))),
            charCount = c(stri_stats_general(twitter)[3],stri_stats_general(blog)[3],stri_stats_general(news)[3]),
            meanWordsPerLine = c(mean(stri_count_words(twitter)),mean(stri_count_words(blogs)),mean(stri_count_words(news)))
  )
  print(stats)
}
fileStats <- summaryTable(twitter, blogs, news)
##    source fileSize_mb lineCount wordCount charCount meanWordsPerLine
## 1 Twitter    159.3641   2360148  30093413 162096241         12.75065
## 2    Blog    200.4242    899288  37546250 206824382         41.75109
## 3    News    196.2775     77259   2674536  15639408         34.61779

Sample and cleaning the data

Since the dataset is very large will use a sample of 10%. As a preparatory step will do a data cleaning as remove the punctuation, the numbers, the white spaces, etc.

## Sample data
set.seed(333)
percent_sampling_lines <- 0.1
sampledData <- c(sample(twitter, length(twitter) * percent_sampling_lines),
                 sample(blogs, length(blogs) * percent_sampling_lines),
                 sample(news, length(news) * percent_sampling_lines))

## Prepare the data
library(tm)
## Loading required package: NLP
text_corpus <- Corpus(VectorSource(sampledData))
text_corpus <- tm_map(text_corpus, tolower)
text_corpus <- tm_map(text_corpus, removePunctuation)
text_corpus <- tm_map(text_corpus, removeNumbers)
text_corpus <- tm_map(text_corpus, removeWords, stopwords())
text_corpus <- tm_map(text_corpus, stemDocument)
text_corpus <- tm_map(text_corpus, stripWhitespace)
text_corpus <- tm_map(text_corpus, PlainTextDocument)
text_corpus <- Corpus(VectorSource(text_corpus))

Tokenize sample into n-grams

Next step is to create n-grams: an n-gram is a continuous sequence of n items from the data sample.

library(RWeka)

uniGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
biGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
triGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))

uniGramMatrix <- DocumentTermMatrix(text_corpus, control = list(tokenize = uniGramTokenizer))
biGramMatrix <- TermDocumentMatrix(text_corpus, control = list(tokenize = biGramTokenizer))
triGramMatrix <- TermDocumentMatrix(text_corpus, control = list(tokenize = triGramTokenizer))

Creating a n-gram stats with word cloud and bar plot.

Unigrams stats

Word cloud

library(wordcloud)
## Loading required package: RColorBrewer
library(RColorBrewer)
library(tm)
library(NLP)

freqTerms <- findFreqTerms(uniGramMatrix, lowfreq = 1000)
unigrams_frequency <- sort(colSums(as.matrix(uniGramMatrix[,freqTerms])),decreasing = TRUE)
unigrams_freq_df <- data.frame(word = names(unigrams_frequency), frequency = unigrams_frequency)
wordcloud(unigrams_freq_df$word,unigrams_freq_df$frequency,scale=c(4,.1), colors = brewer.pal(7, "Dark2"))

Word bar plot

library(ggplot2)
## 
## Attaching package: 'ggplot2'
## The following object is masked from 'package:NLP':
## 
##     annotate
library(NLP)

ggplot(unigrams_freq_df[1:30,],aes(x=reorder(word,-frequency),y=frequency))+
  geom_bar(stat="identity", fill = I("blue")) +
  labs(title="Top 30 Most Frequent Unigrams", x="Unigrams", y="Frequency") +
  theme(axis.text.x=element_text(angle=55))

Next Steps: Create Prediction Algorithm and Shiny app

The n-gram tables will provide the basis for the prediction algorithm. When the user inputs a word, the algorithm will use the n-gram starting with that word with the greatest relative frequency.

Next steps planned are the following:

  • Further cleaning the data: I intend to implement profanity filter.
  • Train a model: I will need to train a model on the data. One approach is to use a language model based on maximum likelihood estimation (MLE). I can use the lm function from the ngram package to train a n-gram MLE language model on the provided data.
  • Predict the next word: Once the model is trained, I can use it to predict the next word in a given phrase. I can use the predict function from the ngram package to generate the next word in a phrase.
  • Refine the model: Finally, it’s important to evaluate the performance of the model and refine it as needed. I can use metrics such as perplexity and accuracy to evaluate the model’s performance, and I can try different methods of feature engineering, model selection, and hyperparameter tuning to improve the model’s performance.
  • Create the Shiny app: When the prediction algorithm is complete, this will be used into a Shiny app for evaluation. For this I will use the Shiny package to create an interactive web app that allows users to input a phrase and receive a prediction of the next word.