Introduction

This report presents an exploratory analysis of the English language corpus and outlines plans for a next-word prediction model.

Load Data

blogs <- readLines("final/en_US/en_US.blogs.txt", encoding="UTF-8", skipNul=TRUE)
news <- readLines("final/en_US/en_US.news.txt", encoding="UTF-8", skipNul=TRUE)
twitter <- readLines("final/en_US/en_US.twitter.txt", encoding="UTF-8", skipNul=TRUE)

Summary Statistics

summaryTable <- data.frame(
  Dataset=c("Blogs","News","Twitter"),
  Lines=c(length(blogs),length(news),length(twitter)),
  Words=c(sum(stri_count_words(blogs)),
          sum(stri_count_words(news)),
          sum(stri_count_words(twitter))),
  Characters=c(sum(nchar(blogs)),
               sum(nchar(news)),
               sum(nchar(twitter)))
)
kable(summaryTable)
Dataset Lines Words Characters
Blogs 5000 115728 764313
News 5000 90701 745973
Twitter 5000 33585 249258

Sample Data

set.seed(123)
sampleData <- c(sample(blogs,5000),
                sample(news,5000),
                sample(twitter,5000))

Data Cleaning

corpus <- Corpus(VectorSource(sampleData))
corpus <- tm_map(corpus, content_transformer(tolower))
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, removeWords, stopwords("english"))
corpus <- tm_map(corpus, stripWhitespace)

Word Frequency

tdm <- TermDocumentMatrix(corpus)
m <- as.matrix(tdm)
freq <- sort(rowSums(m), decreasing=TRUE)
freqData <- data.frame(Word=names(freq), Frequency=freq)

Top 20 Words

top20 <- head(freqData,20)
ggplot(top20,aes(reorder(Word,Frequency),Frequency))+
geom_col()+coord_flip()+
labs(title="Top 20 Most Frequent Words",x="Word",y="Frequency")

Sentence Length Distribution

sentenceLength <- stri_count_words(sampleData)
ggplot(data.frame(sentenceLength),aes(sentenceLength))+
geom_histogram(binwidth=2)+
labs(title="Sentence Length Distribution",x="Words",y="Count")

Word Cloud

wordcloud(words=freqData$Word,
          freq=freqData$Frequency,
          max.words=100,
          colors=brewer.pal(8,"Dark2"))

Findings

The exploratory analysis shows differences in writing style among blogs, news articles, and tweets. Blogs tend to contain longer passages, tweets are shorter and more conversational, and news articles use a more formal style. Word frequencies indicate that a relatively small set of words appears very frequently.

Future Work

The next phase will develop an n-gram based next-word prediction model and an interactive Shiny application for text prediction.