Data Science Capstone Project - Milestone 1

Executive Summary

The purpose of this report is to demonstrate the first steps of building a predictive text model for the Coursera Data Science Certificate program. The steps this report will cover are obtaining the data, cleaning it, and finally an exploratory analysis.

Getting and Loading Data

First step in the project is to get the data for the project, and import it. Also required, is a list of words that would be described as profanity. I used a list of banned words by Google. The list can be found here.

Once I downloaded the datasets, I unzipped them, and loaded into my R session.

library(tidytext); library(dplyr); library(caret); library(quanteda);library(stringr)
library(tidyr);library(stringi);library(igraph);library(ggraph);library(widyr)
setwd("D:/Personal/Coursea/10.  Capstone Project")
con <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
#Create Connections to load data
con1 <- file("en_US.blogs.txt", "r")
con2 <- file("en_US.news.txt", "r")
con3 <- file("en_US.twitter.txt", "r")
con4 <- file("profane_list.txt","r")
blogdata <- readLines(con1)
newsdata <- readLines(con2)
twitterdata <- readLines(con3)
profanedata <- readLines(con4)
profanedata <- as.data.frame(profanedata)

Corpora Summaries

Before I get to cleaning the data so I can perform some modelling, let’s take a look at some basic summary statistics of the 3 corpora used in the project.

Blog data has 38,601,176 words and 899,288 lines.

News data has 2,755,797 words and 77,259 lines.

Twitter data has 31,130,580 words and 2,360,148 lines.

Cleaning the Data

Before cleaning the data, I decided that since there were so many lines and words between the 3 corpora, I would combine them then take a random sample of 5% of the lines, to be used further. Here is the code:

#Combine the 3 datasets into 1
textdata <- Reduce(function(...) merge(..., all=TRUE), list(blogdata, newsdata, twitterdata))
textdata <- textdata %>%
    gather(-corpus, key = "test", value = "text") %>%
    select(-test)

textdata <- filter(textdata,!is.na(text))

#Randomly sample 5% of rows (166835 of 3336695 rows), then write the data to a csv
set.seed(123987)
sample_data <- textdata %>%
    sample_n(166835)

#remove non ascii characters
sample_data$text <- gsub("[^\x20-\x7E]", "", sample_data$text)

write.csv(sample_data,"sample_textdata.csv")

#Load sample data
sample_data <- read.csv("sample_textdata.csv")

Now that we have a smaller sample of the text, I used the tidytext R package to tokenize the dataset. A token is a meaninful unit of text, most often a word that can then be used for further analysis. Tokenization is then splitting the text into a one token per document per row format. The unnest_tokens function of the tidytext package accomplishes the tokenization task for me, as well, it strips punctuation, converts all words to lower case, and retains the line number from which the word came from. Here is a sample few lines from this process:

head(toktextdata)

Exploratory Data Analysis

Basic Exploration

Now that we have tidied up the data, let’s explore it. First, I want to explore the word frequency. We then could measure the importance of the word by it’s frequency, which is called term frequency(tf). A common problem of this is that common English words like “the”, “of”, “and” would then be classified as the most important. There are various approaches to dealing with this, but we will use a technique called inverse document frequency (idf). Using inverse document frequency reduces the importance (weight) of the most frequent words, and increases the weight of words that appear less frequently. Finally, if you multiply the term frequency and inverse document frequency together, you get a statistic (called tf-idf) that is intended to measure how important a word is to a document.

To start, lets count the occurence of each word in the corpus, and the total number of words. Below are the first few results.

head(toktextdata_words)

As expected, “the” is the most common word, with 146,809 occurences out of 3,550,923 possible words. We can then calculate the term frequency (tf) by dividing n/total. After calculating the term frequency, we calculate the inverse document frequeny, then the “term frequence - inverse document frequency statistic” and plot the results.

plot_tf

For the three different corpora, the results were not unexpected. For example, with the twitter data, I was expecting to see things like abbreviations as the imporant words, based on their frequency. For the news corpora, I was expecting to see a bit more nouns (people, places) as news would be reporting on something. For the blogs data, I was expecting a bit more random information since anyone can write a blog, and that’s what it looks like happened.

N-Grams

Moving on, we will take a look at relationships between words, starting with n-grams. N-grams are a tokenization of adjacent words. In this analysis, we will look at bi-grams (two adjacent words) and tri-grams (three adjacent words). Taking a first look at our bigrams, yields the below:

bigrams

The most frequent bigrams include the most frequent words, which is not very interesting. A method of finding the more interesting pairs of words is to remove stop words. Stop words are common words that do not really add any information about the text. So, next we can remove the stop words.

bigrams

The bigrams are far more interesting than the common ones. Now, let’s look at the tri-grams. Following the same process as above and removing the stop words, we get the following:

trigrams

Tri-grams are a little less interesting because most of the information in the top 10 are either numbers or a special occassion.

As a next step, let’s build a network map of our bigrams so we can visualize all the relationships between words.

markovchain

The visualization above is an example of a Markov Chain in text analysis. A Markov Chain in our context, is a word that depends solely on the previous word. What it shows is the links between two words. In this case, only bigrams that appear greater than 50 times are visualized, otherwise the visualization would be unreadable. The other thing to note is there are several clusters of words. For example, “minutes” is in the middle, and is surrounded by numbers. This symbolizes that combinations of phrases, which as “10 minutes” or “30 minutes” were very popular in the corpora.

Next Steps

Next steps of this project are to start building the first predictive model, so, stay tuned!