Exploratory Data Analysis - Data Science Capstone
Ian Skinner
Wednesday, July 15, 2015
Background
This is an exploratory data analysis to understand the dynamics of the supplied data which will be the source of information to create a predictive text application.
The data includes English entries across three feeds; News, Twitter and Blogs.
The purpose of this EDA is to to show that:
- The data has been downloaded and cleaned appropriately including removal of spurious data and profanity
- Provide some key statistics around the supplied data
- Identify some key insights into the data
- Propose an approach to how this data will be used to build a prediction model and app
Prerequisites
Load the Libraries, set seed for reproducibility and a sample size for the corpus creation
library(ggplot2)
library(tm)
library(stringi)
library(wordcloud)
library(RWeka)
library(qdap)
library(plyr)
set.seed(12345)
sampleSize <-25000Exploratory Data Analysis
Load the raw data
con <- file("final/en_us/en_US.twitter.txt", "r", blocking = FALSE)
twitter <-readLines(con)
close (con)
con <- file("final/en_us/en_US.news.txt", "r", blocking = FALSE)
news <-readLines(con)
close (con)
con <- file("final/en_us/en_US.blogs.txt", "r", blocking = FALSE)
blogs<-readLines(con)
close (con)Appendix A, fig 1,2,3 contains sample output of this data
Key Insight: On inspecting the data samples there appears to be a lot of spurious characters (and non words) in the data that will need to be cleaned
File Volumes are as follows:
- The Twitter feed has 2360148 lines
- The News feed has 77258 lines
- The Blogs feed has 899288 lines
Word (and non word sequence) Volumes are as follows:
- The Twitter feed has 30373792 words
- The News feed has 2643958 words
- The Blogs feed has 37334441 words
Lets now look at the summary and histograms for number of Words per line
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.0 7.0 12.0 12.9 18.0 47.0
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.0 19.0 31.0 34.2 45.0 1030.0
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1 9 28 42 59 6630Key Insight: Unsuprisingly the twitter feed consists of short lines with a mean word count of 12.8, both the News and Blogs articles have a higher mean count at 34.2 and 42 respectively. The News and blogs data feed is most noticeable from its histogram in that it tends to have the highest density of lines at the lower end of the scale with then just a handful of lines with much higher word counts
Appendix B, fig 1,2,3 contains wordcloud output of this data to get a feel for the content
Data Prep Exploration
To understand the preparation of the data for analysis and modelling I go through the following steps:
- Create a corpus which is a random 2.5 × 104 rows of each of the original text sources (will resample later for test and train in modelling)
- Remove all characters that are not alpha, space or apostrophe (I want to leave apostrophes in my words for prediction)
- Remove any numbers
- Convert the corpus to all lower case
- Remove any unnecessary whitespace
- Remove profanity, I have sourced a list from the following location https://code.google.com/p/badwordslist/downloads/detail?name=badwords.txt
corp<- VCorpus(VectorSource(
c(sample(twitter,sampleSize),
sample(news,sampleSize),
sample(blogs,sampleSize))))
corp <- tm_map(corp, content_transformer(function(x,pattern)gsub("[^[:alpha:][:space:]']", "",x)))
corp <- tm_map(corp, content_transformer(tolower))
corp <- tm_map(corp, stripWhitespace)
con <- file("badwords.txt", "r", blocking = FALSE)
badwords <-readLines(con)
badwords<-gsub("[^[:alnum:][:space:]']", "",badwords)
close (con)
corp <- tm_map(corp, removeWords,c(badwords))
corpDF<-data.frame(corp)Now I have a clean dataset I generate n-grams to understand permutations of 1 2 and 3 term subsets of the corpus. These word combinations (n-grams) could potentially form a basis for predicting next words by understanding common phrases.
rm(corp)
uniGram <- NGramTokenizer(corpDF$text,Weka_control(min = 1, max = 1, delimiters = " "))
biGram <- NGramTokenizer(corpDF$text,Weka_control(min = 2, max = 2, delimiters = " "))
triGram <- NGramTokenizer(corpDF$text,Weka_control(min = 3, max = 3, delimiters = " "))Lets now look at the top 25 permutations of terms for each of the n-grams
Key Insight: The n-grams all appear to be producing reasonable output and the terms returned are all unsuprising, using n-grams to predict next words seems to be a robust approach to this project
Next Steps for Application
I will be undergoing the following steps with this project:
- Build a test and train dataset
- Create and test models that use the n-grams to predict next word. For example if a user enters two words we can predict third word options based on commonly occuring tri-grams that start with the same two words. Consider what magnitude of n-grams is appropriate to go to e.g. 4-grams, 5-grams etc.
- Build an approach to populate most appropriate word options where ngram does not exist (no model coverage)
- Refine Models based on any limitations of shiny apps
- Integrate model with a User Interface into shiny
- Test Shiny app against against previous examples to ensure stability
- Document and go live with app on shiny website
Appendix A - Raw Data
fig 1. Twitter (sample)
head(twitter)## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."
## [4] "So Tired D; Played Lazer Tag & Ran A LOT D; Ughh Going To Sleep Like In 5 Minutes ;)"
## [5] "Words from a complete stranger! Made my birthday even better :)"
## [6] "First Cubs game ever! Wrigley field is gorgeous. This is perfect. Go Cubs Go!"fig 2. News (sample)
head(news)## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."
## [4] "The Alaimo Group of Mount Holly was up for a contract last fall to evaluate and suggest improvements to Trenton Water Works. But campaign finance records released this week show the two employees donated a total of $4,500 to the political action committee (PAC) Partners for Progress in early June. Partners for Progress reported it gave more than $10,000 in both direct and in-kind contributions to Mayor Tony Mack in the two weeks leading up to his victory in the mayoral runoff election June 15."
## [5] "And when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. Is it absolutely necessary? Is it an issue serious enough to merit their attention? Will it definitely not make the situation worse?"
## [6] "There was a certain amount of scoffing going around a few years ago when the NFL decided to move the draft from the weekend to prime time -- eventually splitting off the first round to a separate day."fig 3. Blogs (sample)
head(blogs)## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan âgodsâ."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him."
## [4] "so anyways, i am going to share some home decor inspiration that i have been storing in my folder on the puter. i have all these amazing images stored away ready to come to life when we get our home."
## [5] "With graduation season right around the corner, Nancy has whipped up a fun set to help you out with not only your graduation cards and gifts, but any occasion that brings on a change in one's life. I stamped the images in Memento Tuxedo Black and cut them out with circle Nestabilities. I embossed the kraft and red cardstock with TE's new Stars Impressions Plate, which is double sided and gives you 2 fantastic patterns. You can see how to use the Impressions Plates in this tutorial Taylor created. Just one pass through your die cut machine using the Embossing Pad Kit is all you need to do - super easy!"
## [6] "If you have an alternative argument, let's hear it! :)"Appendix B - Word Clouds
fig 3. Twitter (Top 100 Words)
wordcloud(paste(twitter, collapse= ""), max.words=100,colors=brewer.pal(9,"BuGn"))
fig 4. News (Top 100 Words)
wordcloud(paste(news, collapse= ""), max.words=100,colors=brewer.pal(9,"BuGn"))
fig 5. Blogs (Top 100 Words)
wordcloud(paste(blogs, collapse= ""), max.words=100,colors=brewer.pal(9,"BuGn"))