Executive Summary

The goal of this project is to build a predictive text model, using the HC Corpora data set. The data are in English and come from three different types of websites: blogs, news, or Twitter. In this milestone report, the data were loaded into R version 3.3.2 (2016-10-31) and an exploratory analysis was performed.

A few of the interesting findings from the exploratory analysis include:

The most common word in all three data sets is the, followed by to, and, a, of
The most common two-word pairs are of the and in the
The top five two-word pairs all contain the
The most common three-word phrases in the Blogs and News sets are one of the and a lot of, while thanks for the is the most common phrase in the Twitter set

Overall, there appears to be a lot of similarity in the word phrases contained in the Blogs and News sets, while the Twitter set has more variation. This is not entirely surprising, given the 140 character limit on tweets.

The next step in the project will be to build an initial predictive model. This will be done by using the word pairs and word phrases to predict the three most likely options for the next word.

Introduction

This is the final project for the Data Science Capstone course. The goal of this project is to build a predictive text model, such as those used by smart keyboards on mobile devices. The model will be built using the HC Corpora data set11 HC Corpora. The data contained in the corpus were obtained from publicly available websites using a web crawler. The data were then divided into three categories based on the type of website: blogs, news, or Twitter (also referred to as tweets). The project will focus on building a predictive text model for the English language. The goal of this milestone report is to explore the three data sets in the corpus.

The data do contain some words which are considered profanity and should not be included in the prediction model. A list of “bad words” compiled by Google22 List of profane words was used to identify words for exclusion from the data set.

NOTE: The source R code to generate the tables and figures in this report can be found here.

Exploratory Analysis

The exploratory analysis utilized the R package tidytext to perform text mining, along with the tidyverse packages for data manipulation. The data were processed using R version 3.3.2 (2016-10-31).

Summary of Data Sets

The following table provides a basic overview of the three data sets contained in the HC Corpora, including the number of lines of data in each set, and summary statistics for the number of characters per line in each set. Unsurprising, the blogs and news data sets have a much larger mean number of characters per line compared with the Twitter set, where tweets are limited to a maximum of 140 characters.

Number of lines and characters in each data set

Data Set	Lines	Min Chars	Median Chars	Mean Chars	Max Chars
blogs	899,288	1	156	230.00	40,830
news	1,010,242	1	185	201.20	11,380
tweets	2,360,148	2	64	68.68	140

Exploring Word Counts

The next table provides a summary of the number of word instances contained in each set, as well as the number of unique words. It should be noted that prior to calculating the number of word instances, words considered profanity as well as words containing numeric digits were removed.

Number of word instances and unique words in each data set

Data Set	Word Instances	Unique Words
blogs	37,050,078	295,664
news	33,817,677	248,496
tweets	29,404,728	338,038

In each set, the number of words which make up 50% and 90% of the total word instances was calculated. This demonstrates that a small number of words make up the majority of word instances.

Number of words in 50% and 90% of word instances

Data Set	Words in 50% of Instances	Words in 90% of Instances
blogs	107	6,421
news	194	7,883
tweets	120	5,157

Visualizing the Data

⊕Given the large size of the data sets, a random sample of 20% of the total data was used to perform most of the exploratory analysis.

Using a sample of the data, a word cloud can be used to visualize the the top 250 unique words in the three data sets combined. Words in the word cloud are sized based on their frequency, with more frequent terms appearing in a larger font size.

Word cloud for the combined data sets

Further evaluation of the top 25 words in the combined data set reveals that the most common word, the, is almost twice as common as the second-most frequent word, to.

Most frequent words in combined data set

There appears to be a lot of similarity between the three data sets, with words such as the, to, and, and a among the most frequent words in each set.

Comparison of top words in each data set

Word Pairs and Phrases

Using the data sample for each set, two-word pairs and three-word phrases were created using the n-gram method. These will be used in the prediction model to predict what the next word will be based upon the current word being typed.

A summary of the number of the word-pairs and word-phrases contained in the data sample is presented in the table below.

Number of two-word pairs and three-word phrases in each data set

Data Set	Word Pairs	Unique Pairs	Word Phrases	Unique Phrases
blogs	7,154,401	1,901,676	6,922,269	4,628,035
news	6,422,882	1,899,303	6,106,787	4,307,736
tweets	5,314,872	1,502,704	4,770,276	3,184,858

The frequency of the top 10 two-word pairs in each data set are displayed below. Interestingly, the top 4 word pairs are the same in the blogs and news sets, but differ in the Twitter set.

Comparison of top word pairs in each data set

When the three data sets are combined, the top 25 word pairs are depicted in the figure below. This reveals that the most common word pairs are of the and in the, while the third most common pair, to the, is about half as frequent. Additionally, the top five pairs all contain the most common word, the.

Top word pairs in combined data set

The top three-word phrases in each set are presented below. The top three phrases are the same in the blogs and news sets, however, the top phrases in the Twitter set vary considerably.

Comparison of top word phrases in each data set

When the three data sets are combined, the top 25 three-word phrases are depicted in the figure below.

Top word phrases in combined data set

Next Steps

The next setp will be to build a predictive text model using these data sets. The two-word pairs and three-word phrases will be utilized to predict the next most-likely word based upon the word currently entered.

Other considerations for the model will include the use of synsets from the WordNet lexical database. These groups of synonms may allow the model to predict additional word possibilities, even for words which are not in the HC Corpora.

The goal will be to create a Shiny app which will take a word and return the three most likely options for the next word.

Capstone Project

Milestone Report

Brian Gulbis

November 25, 2016