Text Mining in R with the tm Package

1. This presentation is an offshoot of the Jonhs Hopkins University's Data Science Specialization's final Capstone project. The project consisted in creating a predictive text app in Shiny that would estimate and display the likeliest word to be typed in next by the user, given the previous input text.

2. CRAN Natural Language Processing Task View.

3. Text Mining Infrastructure in R <- Comprehensive Journal of Statistical Software article.

4. Intro to the tm package <- Getting started pdf.

1. Developed by Feinerer et al at the Wirtschaftsuniversitaet Wien (Vienna University of Economics and Business).
2. Capabilities include
   • Preprocessing
   • Association
   • Clustering
   • Summarization
   • Categorization
   • API availibility

This presentation will focus on preprocessing.

1. The dataset is the HC Corpora
2. This corpus contains 2,360,148 blog posts so we'll work with a 1% random sample.
3. The blogs object is a character vector where each element from 1 to n is a blog entry.The number of blog entries is equivalent to the length of the vector.

blogs <- readLines("C:/Users/marino.re/Box Sync/Capstone/data/en_US/en_US.blogs.txt", 
    encoding = "utf-8", skipNul = TRUE)
set.seed(50) #Reproducibility seed
blogsSample <- sample(blogs, size= length(blogs)*0.01) #Sampling 1%
rm(blogs)

Corpora can be created using the VCorpus() function (V stands for volatile. The corpus is an R object fully held in memory; when deleted the corpus is gone). Then a source has to be specified, in this case the source is a character vector, so VectorSource() can be used.

library(tm)
corpus <- VCorpus(VectorSource(blogsSample))

as.character(corpus[[6835]])

“The charity inspired by the encounter has raised $60m and in 2009 said it was supporting 54 schools in across Homeward serving 28,475 students. Obama donated $100,000 to the group from the proceeds of his Nobel prize. The book has become required reading in the US of A.”

1. This blog post contains numbers, punctuation, capitalized words.

transformations <- function(corpus) {
  corpus <- tm_map(corpus, content_transformer(tolower))
  corpus <- tm_map(corpus, removePunctuation)
  corpus <- tm_map(corpus, removeNumbers)
  corpus <- tm_map(corpus, removeWords, stopwords("english"))
  #Removing words creates white spaces, so stripWhitespace has to come later
  corpus <- tm_map(corpus, stripWhitespace)
}

1. The order of execution matters. e.g. stripWhitespace() function has to come after removeNumbers()
2. The transformations applied depend strictly on objective of the analysis. e.g. profane words would be kept for sentiment analysis and removed for prediction.

“The charity inspired by the encounter has raised $60m and in 2009 said it was supporting 54 schools in across Homeward serving 28,475 students. Obama donated $100,000 to the group from the proceeds of his Nobel prize. The book has become required reading in the US of A.”

cleanCorpus <- transformations(corpus)
as.character(cleanCorpus[[6835]])

[1] “ charity inspired encounter raised m said supporting schools across homeward serving students obama donated group proceeds nobel prize book become required reading us ”

1. N-grams. Ann-gram is a contiguous sequence of n items from a given sequence of text or speech

2. Document Term Matrix (dtm). A dtm is a matrix where each document is placed in the rows and each unique word (in the whole corpus) constitutes a feature or a column. This is a very convenient way to carry out frequency counts.

3. One problem with dtms is that the matrix can be very sparse, difficulting calculations. The slam package (Sparse Lightweight Arrays and Matrices) is recommended to deal with sparse matrices.

Each element in the matrix is a term frequency counter for a specific term in a specific document.

Capstone Link