Capstone Project - Progress Report

This is the progress report for John Hopkins University Data Science Specialization Capstone course. The project is focused on demonstration of the use of Natural Language Processing tools to build a model to predict the next word typed in a sentence. The project uses several language processing packages:

• tm: used to read the corpus of documents in a folder and create a vCorpus
• quanteda: used in this excercise to show summary statistics of the corpus
• rweka: used to create a tokenizer and n-grams from a TermDocumentMatrix
• dplyr and grid.Extra: used to identify and plot most used terms and n-grams
  

The tm package was the primary package utilized in this project

The first step in the project is to prepare the environment there are three steps: 1. Load packages needed 2. Download the data if needed 3. Create a sample directory to hold the sample corpus

Prepare Environment Code

packages<-function(x){    #function to detemine if needed packages are installed
  x<-as.character(match.call()[[2]])
  if (!require(x,character.only=TRUE)){
    install.packages(pkgs=x,repos="http://cran.r-project.org")
    require(x,character.only=TRUE)
  }
}

packages(dplyr); packages (downloader); packages(NLP); packages(openNLP); packages(RWeka)
packages(stringr); packages(stringi); packages(tm); packages(quanteda); packages(ggplot2)
packages(gridExtra); remove(packages)

#Step 2. Download the data for the project.  Test to see if it already exist first since this a 510MB file
if (!file.exists("dataset.zip")) {
    fileurl <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
    download(fileurl, dest="dataset.zip", mode="wb") 
    unzip ("dataset.zip", exdir = ".")  # this command creates a directory "????" that contains the data and extracts the files 
}
#Step 3. Test to see if the sample directory already exist if the code is run more than once 

if (!dir.exists("./sample")) {
    dir.create("./sample")
}    

The data set from SwiftKey allows for analysis of multiple languages English, German, Russian and Finnish This project will focus on the English version. The entire corpus of three documents is 510Mb which is too large to manipulate. A sample of each document is selected and then written out in a sample directory.

Read Data Code

#sample the twitter file
twitter <- readLines("./final/en_US/en_US.twitter.txt", encoding="UTF-8", skipNul=T) 
twitter_s <- sample(twitter, length(twitter) * 0.001); remove(twitter)

#sample the blogs file
blogs <- readLines("./final/en_US/en_US.blogs.txt", encoding="UTF-8", skipNul=T) 
blogs_s <- sample(blogs, length(blogs) * 0.001); remove(blogs)

#sample the news file
news <- readLines("./final/en_US/en_US.news.txt", encoding="UTF-8", skipNul=TRUE) 
news_s <- sample(news, length(news) * 0.001); remove(news)

#write the files to a folder
writeLines(blogs_s, con = "./sample/en_US.blogs.txt", sep = "\n", useBytes = F)
writeLines(twitter_s, con = "./sample/en_US.twitter.txt", sep = "\n", useBytes = F)
writeLines(news_s, con = "./sample/en_US.news.txt", sep = "\n", useBytes = F)

Analyze Corpus: The corpus has three files. A US News file, blog file extracts and a file of tweets. A sample of all the files was chosen for analysis to build or broader range of phrases and terms upon which to build a aplication dictionary. In cleaning the data, stemming and removing stop words produced somewhat meaninless words and phrases.

Create and Analyze Corpus Code

#================================
#The code uses tm to create a corpus of the documents in a directory and then it uses quanteda command to create a corpus
##================================
cname <- file.path("./sample/"); docs <- Corpus(DirSource(cname)); myCorpus <- corpus(docs) 

#=======Create document feature matrix
myDfm <- dfm(myCorpus, ignoredFeatures = stopwords("english"), stem = FALSE)  #remove punctuation, make lowercase, index
top10 <- topfeatures(myDfm, 10); top10df <- data.frame(ngram=names(top10), occurrences=top10); top10df <- arrange(top10df, desc(occurrences)) 

#======Cleaning the tm Vcorpus before creating ngrams 
toEmpty <- content_transformer(function(x, pattern) gsub(pattern, "", x))
docs <- tm_map(docs, toEmpty, "#\\w+"); docs <- tm_map(docs, removePunctuation); docs <- tm_map(docs, content_transformer(tolower))
docs <- tm_map(docs, removeNumbers); docs <- tm_map(docs, stripWhitespace)

summary(myCorpus); topfeatures(myDfm, 10)

Corpus consisting of 3 documents.

  Text Types Tokens Sentences author       datetimestamp description
 text1  8162  42449      2029   <NA> 2016-08-29 02:28:37        <NA>
 text2  1305   2827       120   <NA> 2016-08-29 02:28:37        <NA>
 text3  7801  37650      2555   <NA> 2016-08-29 02:28:37        <NA>
 heading                id language origin
    <NA>   en_US.blogs.txt       en   <NA>
    <NA>    en_US.news.txt       en   <NA>
    <NA> en_US.twitter.txt       en   <NA>

Source:  Converted from tm VCorpus 'docs'
Created: Sun Aug 28 19:28:37 2016
Notes:   

just  one    u will like  can  get time  day good 
 248  229  200  197  192  189  176  176  170  162