Data Science Capstone Milestone Report

Preliminary Exploration.

The Data

Coursesa provided from SwitKey a large data set which consisted of entries from web blogs, twitter, and news items. There were a number of languages, however this report focuses only on the English data set. A summary of the sources is below (after being scrubbed and tokenized—see the section on Pre-Processing below).

Source	Total Docs	Distinct Tokens
Blogs	899,288	259113
Twitter	2,360,148	330963
News	1,010,242	232635

Each source has it’s unique issues.

Blogs: Most entries were encoded in utf-8, however, there were numerous entries in utf-16 and non-English words and phrases. A few were completely in a foreign language. There are numerous situations where intentional spelling errors such as “ooooooh” or “aaaaaaaand.”
Twitter: Twits have there own unique language and many abbreviation: example: “r u ready for this ….”
News: Has a higher fraction of words covered when plotted against minimum word frequency, suggesting that there are fewer low frequency tokens.

A sample of 500 scrubbed entries from the blog, news and twitter documents shows many words are repeated only a few times. Shown below is plot of the token fraction vs. the minimum token frequency.

As can be seen the three sources have similar high frequency tokens.

Common Tokens

The following table shows the number of tokens that are in common between sources.

Source	in News	in Twitter	in Blogs	in Both Others
in News	232635	105387	110834	85572
in Twitter	105387	330963	113466	85572
in Blogs	110834	113466	259113	85572

Unique Tokens

The following table shows the number of unique tokens within a source compared to another source

Source	not in News	not in Twitter	not in Blogs	not in Either
in News	0	127248	121801	101986
in Twitter	225576	0	217497	197682
in Blogs	148279	145647	0	120385

Most Frequent Tokens in a Random Sample

The 45 most common tokens and their counts (the numbers) in a random sample of 500 Blog documents:

##   @num@     the   @com@     and      to       a      of       i      in 
##    1035     848     763     518     475     429     399     377     247 
##    that      it      is     for     you    with    this      my     was 
##     216     194     181     168     147     142     138     129     125 
## @quodl@   @per@    have      on      be     but      as     are      we 
##     123     111     109     109     103      96      92      91      89 
##     not      me      so     all   about     can    they      at   @exc@ 
##      85      83      83      74      72      71      70      68      68 
##      by    from      he      or @quodr@      up     had     his     one 
##      65      63      63      63      61      59      58      57      56

The 45 most common tokens and their counts in a random sample of 500 News documents:

##   @num@     the   @com@     and      to       a      of      in       i 
##    2423    1743    1600     933     925     835     729     523     467 
## @quodl@    that     for      it      is     was    with      on    this 
##     407     369     317     303     291     235     230     219     201 
##     you @quodr@      at      be      as      he    have     but     are 
##     186     179     169     165     163     161     160     158     157 
##      my   @per@    from      by      we    said     not     his   about 
##     152     143     139     136     134     128     127     124     118 
##    they     all   @col@      so      or     one  @parc@    will     can 
##     117     106     105     105     104     103     103     103     101

The 45 most common tokens and their counts in a random sample of 500 Twitter documents:

##   @num@     the   @com@      to     and       a      of       i      in 
##    2904    1955    1742    1093    1030     958     814     633     607 
## @quodl@    that     for      is      it   @exc@     you      on   @per@ 
##     450     412     397     370     364     340     280     279     278 
##    with     was    this      my      at      be @quodr@   @col@   @sym@ 
##     270     254     231     226     214     212     199     196     195 
##    have     are     but      as      he    from      we  @parc@      by 
##     194     185     176     172     169     164     159     151     150 
##     not   @que@     all     his    said   about      so      me    they 
##     147     142     137     136     136     135     134     132     131

Pre-proccesing.

The text of each entry was pre-processed using the following function. Reliance on the built in function of tm were avoided since the scrub function will need to be used in the predictive algorithm.

scrub <- function (txt){
    # modify encoding if needed and move to lower case to prevent introduction
    # of equivalent n-grams
    txt <-enc2utf8(txt)
    txt <- tolower(txt)
    
    # @ is used as a special symbol to indicate markers -- it is transformed first     
    txt <- gsub("@",   " @at@ ", txt)
    
    # ellipse as set to be no more than 3 periods in a row    
    txt <- gsub("\\.{3,}",' @per@ @per@ @per@ ',txt)
    
    # number modifiers    
    txt <- gsub("\\$", " @dol@ ", txt) 
    txt <- gsub("%",   " @per@ ", txt)
    
    # '-' are removed if is a single - it is assumed part of a compound 
    # adjective, e.g., at-the-park vendor, If muliple dashes it is assume
    # a pause similar to a comma
    txt <- gsub("-{2,}",   " @com@", txt) 
    txt <- gsub("-{1}",   " ", txt) 
    
    # '_' have no reason to be in english text    
    txt <- gsub("_",   " ", txt) 
    
    # Open and closing parathesis and brackets often indicate 
    # a new n-gram is starting.
    txt <- gsub("\\(", " @paro@ ", txt)
    txt <- gsub("\\[", " @brao@ ", txt)
    txt <- gsub("\\)", " @parc@ ", txt)
    txt <- gsub("\\]", " @brac@ ", txt)
    
    # numbers are replaced with a number indicator
    txt <- gsub("[\\.0-9]([0-9]*[:,\\.]{0,1})*[0-9]*('s|%)*", " @num@ ", txt) 
    
    # Single ' or apostrophe in contractions is replaced by '_' while at the
    # beginning or end by a single quote marker
    txt <- gsub("(’|')(\\s|$)", " @quosr@ ",txt) # ' at end of word
    txt <- gsub("(’|')", "_", txt)               # ' -- an apostrophe, 
    txt <- gsub("‘", " @quosl@ ",txt)            #single apostrophe, single left quote
    
    #double quotes -- order on closing punctuation is changed for conistency
    txt <- gsub("(!|\\?|\\.|,)(”|\")", " @quodr@\1 ", txt)  #right " preceeded by ,
    txt <- gsub("(^|\\s)(“|\"|”)",     " @quodl@ ", txt)    #left "
    txt <- gsub("(”|\"|“)(\\s|$)",     " @quodr@ ", txt)    #right "
    
    # various pause/stop charaters
    txt <- gsub("(\\.)", " @per@ ", txt) 
    txt <- gsub("(\\?)"," @que@ ", txt)
    txt <- gsub("(!)", " @exc@ ", txt)   
    txt <- gsub("(:)", " @col@ ", txt)   
    txt <- gsub("(;)", " @sem@ ", txt)   
    txt <- gsub("(,)", " @com@ ", txt)   
    
    #miscellenous symbols -- begining of a new n-gram??
    txt <- gsub("\\||\\*|/|<|>|#|\\^|&|\\+|=", " @sym@ ", txt)  
    
    #some stray utf-8 code, that mess things up, are removed
    txt <- gsub("\u0095", "", txt)  #message waiting
    txt <- gsub("\u0096", "", txt)  #start of guarded area
    txt <- gsub("\u00f8", "", txt)  #latin small letter o with stroke
    
    # extra whitespace elininated
    txt <- gsub("\\s+", " ", txt)    #Extra Space
    txt <- trimws(txt) #Extra Space
    
    # remove lines with foriegn charater words (i.e. those with 
    # non-ascii characters)
    w2 <- unlist(strsplit(txt, split=" "))
    ne <- grep("w2", iconv(w2, "latin1", "ASCII", sub="w2"))
    if(length(ne)>0){ txt<-''} 
    
    return(txt)
}

Data Science Capstone Milestone Report

Gary De Young

March 17, 2016

The Project