Introduction

The Johns Hopkins Coursera Data Science Capstone Project is the development of a shiny app that will predict the next word given some user-input text. This milestone report for the project will show some basic information about the three data sets provided for analysis. These data sets (corpora) will be used to develop a predictive model. This report will also propose the additional steps for completing the project.


Getting the Data

The data for this project consists of three text files harvested from twitter, news, and blog posts. Basic information about the three raw files is shown in the table below:


Twitter News Blogs
file size 167.1 MB 205.8 MB 210.2 MB
line count 2,360,148 1,010,242 899,288
word count 30,374,206 34,372,720 37,334,690


Sampling and Exploring the Data

A sample size of one percent for each corpus was extracted. This size was determined to be large enough for each source file to generate results at the 99% confidence level with a margin of error less than one-half of one percent (http://www.raosoft.com/samplesize.html).

Each sample was cleaned (punctuation, numbers, and excess whitespace removed) and converted to lower case. After brief deliberation, stemming and stop word removal were not done. The term “stop word” is given to a word that appears very frequently in a given language. Stop words are integral to the corpora and become candidates for prediction in the planned model. In fact most of the terms in the “Top 15” plots below could be considered to be stop words. Stemming, which involves converting all forms of a word to its root, would also interfere with accurate ‘next word’ predictions.


A document term matrix was created for each sample. The 15 most frequent terms for each sample are plotted below:




Term Distribution - word counts

Twitter News Blogs
500 times or more 55 47 69
100 times or more 329 318 347
20 times or more 1398 1984 1820


Next Steps

  1. Combine the three sample corpora into one corpus
  2. Split the combined corpus into training and test data
  3. Create n-grams tables (planning to build unigrams, bigrams, trigrams, and 4-grams)
  4. Build a basic predictive model and test it
  5. Build Shiny template using model
  6. Work to improve model and test again (repeat until done)