Milestone Report

Milestone Report

This is the Milestone Report of Module 2 of the Data Science Capstone. Here we present the exploratory data analysis of the text data set that will help to create a predictive text algorithm and we present a proposal for such algorithm

Summary Statistics

Below we present summary statistics. The twitter and blog corpora have over 30 million words, while the news corpora has over 2 million. Naturally the twitter corpora has the most lines. We can also see the distribution of words by number of letters.

##                              en_US.twitter.txt en_US.blogs.txt en_US.news.txt
## Word Count                   30096649          37546806        2674561       
## Line Count                   2360148           899288          77259         
## One-letter Words             2484804           2487782         142821        
## Two-letter Words             5287223           6448459         393590        
## Three-letter Words           6144493           7714361         509911        
## Four-letter Words            6205605           7102686         456151        
## Five-letter Words            3484433           4306276         308660        
## Six-letter Words             2353839           3082562         248409        
## Seven-letter Words           1943422           2583998         221944        
## Eight-letter Words           1120629           1635118         151045        
## Nine-letter Words            646075            1106254         102769        
## Ten-letter Words             362328            668704          61583         
## At least Eleven-Letter Words 170525            352515          31850         
## Number of Unique Words       525678            435579          98387

Below we can see in a plot the distribution of words by number of letters for each corpora. We can see there is relative homogeneity across corpora. The total distribution can be seen below Below we can see the Type-token ratio (TTR) And words per line

News and blogs seem to be the richest corpora

Prediction Algorithm Proposal

We propose a text prediction algorithm that consists of the following elements

• The 1-gram, 2-gram and 3-gram prediction will be the most repeated word in the corpora for the given n-gram input.
• We will always choose for prediction as input the largest n-gram input that has an exact equivalent in the corpora. The largest n-gram that will be acceptable is the 3-gram.
• In case there is no equivalence we will predict based on the 1-gram most similar to a word in the corpora.
• We will validate this prediction scheme using k-fold cross-validation using the maximum number of n-grams possible.