Getting the Data

The data for this project consists of three text files harvested from twitter, news, and blog posts. Basic information about the three raw files (shown in Table A) was collected using a Vi Editor:

Table A
twitter twitter sample news news sample blogs blogs sample
file size 167.1 MB 41.1 MB 205.8 MB 46.4 MB 210.2 MB 52.1 MB
line count 2,360,148 160,114 1,010,242 184,893 899,288 205,352
word count 30,374,206 1,222,540 34,372,720 1,408,541 37,334,690 1,566,864

Most frequently appearing words in each data sample:

Each of the three source files was loaded separately, and three separate corpora were created from a sample of the lines comprising each file.

Each corpus sample was then cleansed of whitespace, punctuation, and numbers. Letters were all converted to lower case.

The ten most frequent words are shown below for each source data sample. The most frequent words for the news and blog samples are ranked almost identically!

Top 10 twitter words: 

##      word count
## 1     you  1678
## 2   thank   832
## 3    good   643
## 4  thanks   526
## 5    love   504
## 6     lol   322
## 7    what   321
## 8     yes   247
## 9  follow   228
## 10   that   210

Top 10 news words: 

##    word count
## 1   the  7570
## 2   and  3443
## 3  that  1385
## 4   for  1328
## 5  with   924
## 6   was   829
## 7  said   808
## 8   his   635
## 9  from   587
## 10 have   562

Top 10 blog post words: 

##    word count
## 1   the  9678
## 2   and  5799
## 3  that  2556
## 4   for  1893
## 5   was  1569
## 6  with  1449
## 7   you  1426
## 8  this  1219
## 9   but  1090
## 10 have  1037

Histograms:

Most of the words do not appear more than a few times. In the Twitter sample containing 3948 distinct words, only 334 appear 10 or more times. In the News sample there are 18,404 distinct words, with only 1677 appearing 10 or more times. And in the Blogs sample there are 20,954 distinct words, with 2036 appearing 10 or more times. The histograms below show the frequency of occurence for distinct words in each data set.

Frequency of word occurence in Twitter data sample:

Frequency of word occurence in News data sample:

Frequency of word occurence in Blogs data sample:

Next Steps:

I need to repeat these steps with a larger sample. I thought the code: subTweets <- sample(tweets, length(tweets)*.5) would give me a 50% sample but clearly that is not the case. Then I will split the data into training and testing subsets, create n-grams (planning to do 1-grams to 4-grams), and begin to build and test predictive models.