{r setup, include=FALSE}` knitr::opts_chunk$set(cache = TRUE) set.seed(1234)

Overview

The purpose of this exploratory data analysis is to understand:

Loading the Data

The corpora are collected from publicly available sources by a web crawler. The crawler checks for language, so as to mainly get texts consisting of the desired language. (The data was downloaded from the Coursera course site for the purposes of this project.)

library(quanteda)
## Package version: 1.5.1
## Parallel computing: 2 of 8 threads used.
## See https://quanteda.io for tutorials and examples.
## 
## Attaching package: 'quanteda'
## The following object is masked from 'package:utils':
## 
##     View
library(readtext)
library(ggplot2)
library(gridExtra)
library(stringi)
library(knitr)
library(quanteda)
library(readtext)

url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
filename <- "Coursera-SwiftKey.zip"
download.file(url,filename)

unzip("Coursera-SwiftKey.zip")

twitter <- readLines("final/en_US/en_US.twitter.txt", skipNul = TRUE)
blog <- readLines("final/en_US/en_US.blogs.txt", skipNul = TRUE)
news <- readLines("final/en_US/en_US.news.txt", skipNul = TRUE)

# adding names for later tracability

docnum <- function(x,docnm) {paste(rep(docnm,
                                       length(x)),seq(1,length(x)),sep = "")}

names(twitter) <- docnum(twitter,"twit")
names(blog) <- docnum(blog,"blog")
names(news) <- docnum(news,"news")

combo <- c(twitter,blog,news)

Text File Statistics

The following basic statistics help understand the content of the text files.

source <- c("Blog", "News", "Twitter")
fsize <- c(file.size("final/en_US/en_US.blogs.txt")/1000000,
           file.size("final/en_US/en_US.news.txt")/1000000,
           file.size("final/en_US/en_US.twitter.txt")/1000000)
blogwordcnt <- stri_count_words(blog)
newswordcnt <- stri_count_words(news)
twitwordcnt <- stri_count_words(twitter)
linecount <- c(length(blog), length(news), length(twitter))
wordcount <- c(sum(blogwordcnt), sum(newswordcnt), sum(twitwordcnt))
maxwordcount <- c(max(blogwordcnt), max(newswordcnt), max(twitwordcnt))
avgwordcount <- c(mean(blogwordcnt), mean(newswordcnt), mean(twitwordcnt))
summary <- data.frame(fsize, source, linecount, wordcount, maxwordcount, 
                           avgwordcount)
headers <- c("File Size (MB)", "Source", "Line Count", "Word Count", 
             "Max Word Count", "Avg Word Count")
kable(summary, col.names = headers, align = 'c', digits = 1, format.args = list(big.mark = ","))
File Size (MB) Source Line Count Word Count Max Word Count Avg Word Count
210.2 Blog 899,288 37,546,239 6,726 41.8
205.8 News 1,010,242 34,762,395 1,796 34.4
167.1 Twitter 2,360,148 30,093,413 47 12.8

Not surprisingly the max and average word counts per line are larger for Blogs than News, and larger for News than twitter.

Removing profanity and non-English text

Profanity

I created a list of swear words by editing the list found on the English Swear Words page in Wiktionary.

badwords <- c("asshole","bitch","cunt","damn","fuck","goddamn","motherfucker","nigga",
              "nigger","prick","shit")

badcount <- sapply(badwords,grepl,combo)
combo1 <- combo[rowSums(badcount)==0]

The swear words appears in 1.5% of entries. These entries were removed prior to additional processing.

non-English

non-ASCII Character Based Approach

I found that there were 11.2% of lines that contained non-English characters; including 101 with exclusively non-English characters. I removed lines that contained exclusively non-ascii characters and substituted spaces for non-English characters in other lines.

combo1 <- combo[grepl("[\x01-\x7F]+",combo)] # removing elements that are all non-ascii
combo1 <- gsub("[^\x01-\x7F]+"," ",combo1) # replacing non-ascii characters with space

Search of other languages using ASCII characters

I also tried to ascertain if there were any other languages (besides English) in the data that use ASCII characters. I did this by searching for the most common words in Spanish, French and English. (Using the lists from the 1000mostcommonwords.com site.) In doing so I uncovered a small number of lines using one of those languages. Often the lines were a mix of English and the other language. Given that less than 100 were found in total, I left them assuming they would be immaterial to the analysis.

lang <- c("Spanish","French","German")
num1word <- c("como","comme","wie")
wordcnt <- c(sum(grepl(" como ",combo,ignore.case=FALSE)),
             sum(grepl(" comme ",combo,ignore.case=FALSE)),
             sum(grepl(" wie ",combo,ignore.case=FALSE)))
samptxt <- c(combo[sample(grep(" como ",combo,ignore.case = FALSE),1)],
             combo[sample(grep(" comme ",combo,ignore.case = FALSE),1)],
             combo[sample(grep(" wie ",combo,ignore.case = FALSE),1)])
summary <- data.frame(lang,num1word,wordcnt,samptxt)
headers <- c("Language","Most Common Word","# of Entries Containing","Sample Entry")
kable(summary, col.names = headers, align = 'c', digits = 1, row.names = FALSE)
Language Most Common Word # of Entries Containing Sample Entry
Spanish como 42 To bem e vs como ta amora?
French comme 15 En art comme en amour, l’instinct suffit (whether it is art or love, your instinct will suffice)
German wie 12 Nun, Urlaub muß ja nicht immer mit einer weiten Reise verbunden sein. Darum habe ich den runden Text “mal die Seele baumeln lassen” am PC erstellt, dazu noch die Worte “schöne Ferien” und ihn dann zusammen mit der süßen Fee von Belles´n´Whistles “Fairy Fond Thoughts” kombiniert. Für mich sieht sie aus als ob sie einen freien Tag im Wald genießen und einfach in den Tag träumen würde. Auch eine Art Urlaub, wie ich finde.

Sampling

I took a sample of the remaining elements.

sampledata <- c(sample(combo1, length(blog)*0.05))

Tokenization

I created three document feature matrixes for unigram, bi-grams and tri-grams. In the process I also removed punctuation, numbers, symbols, etc.

I decided not to stem words or remove stop words for this application. If the intent of the application is to predict the next word in a phrase, stop words should be included as part of natural phrases. Also, stemming would lead to truncated words and incorrect grammar (e.g., “tailgat” not “tailgate”.

myDFM <- function(x,n = 1) {dfm(x,
      ngram = n,
      tolower = TRUE,
      remove_punct = TRUE,
      remove_twitter = TRUE,
      remove_numbers = TRUE,
      remove_hyphens = TRUE,
      remove_symbols = TRUE,
      remove_url = TRUE
)}

uniDFM <- myDFM(sampledata)
biDFM <- myDFM(sampledata,2)
triDFM <- myDFM(sampledata,3)

Word Counts and Coverage

Word Coverage

The plot below shows the number of unique words it takes (based on a frequency sorted word list) for a given percentage of coverage of the overall sample.

uniFreq <- textstat_frequency(uniDFM)

uniFreq$runsum <- cumsum(uniFreq$frequency)
uniFreq$coverage <- uniFreq$runsum/max(uniFreq$runsum)
uniFreq$count <- as.numeric(row.names(uniFreq))
plot(uniFreq$count,uniFreq$coverage, main = "Word Coverage",xlab = "# of Unique Words",
     ylab = "% Coverage")

The 142 most common words will cover 50% of all of the text and the 7149 most common words will cover 90% of the text.

Word Frequency

Below are graphs of the 20 most common words (uni-grams), bi-grams and tri-grams.

ggplot(uniFreq[1:20], aes(x=reorder(feature,frequency), y=frequency)) +
  geom_bar(stat='identity') +
  coord_flip() +
  xlab("Word (uni-gram)")

biFreq <- textstat_frequency(biDFM)
triFreq <- textstat_frequency(triDFM)

ggplot(biFreq[1:20], aes(x=reorder(feature,frequency), y=frequency)) +
  geom_bar(stat='identity') +
  coord_flip() + 
  xlab("bi-gram")

ggplot(triFreq[1:20], aes(x=reorder(feature,frequency), y=frequency)) +
  geom_bar(stat='identity') +
  coord_flip() + 
  xlab ("tri-gram")

Application and Algorithm approach

I intend to build a prediction algorithm by first finding the highest probability words by matching the first two words of the tri-grams in the sample, failing that matching the first word of a bi-gram and then failing that randomly selecting from the most common unigram words.

The shiney application will take test as input and display the 3 highest probability options as output.

References