Introduction

This milestone report show some basic data exploration of the datasets we’re using for this capstone project to develop a natual language procesing word prediction engine.

Data Sets

To develop our model we will be using 3 data sets:

Sample data from each file

Data from blogs:

[1] "in the years thereafter most of the oil fields and platforms were named after pagan gods"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
[2] "we love you mr brown"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
[3] "chad has been awesome with the kids and holding down the fort while i work later than usual the kids have been busy together playing skylander on the xbox together after kyan cashed in his  from his piggy bank he wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it he never taps into that thing either that is how we know he wanted it so bad we made him count all of his money to make sure that he had enough it was very cute to watch his reaction when he realized he did he also does a very good job of letting lola feel like she is playing too by letting her switch out the characters she loves it almost as much as him"
[4] "so anyways i am going to share some home decor inspiration that i have been storing in my folder on the puter i have all these amazing images stored away ready to come to life when we get our home"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
[5] "with graduation season right around the corner nancy has whipped up a fun set to help you out with not only your graduation cards and gifts but any occasion that brings on a change in ones life i stamped the images in memento tuxedo black and cut them out with circle nestabilities i embossed the kraft and red cardstock with tes new stars impressions plate which is double sided and gives you  fantastic patterns you can see how to use the impressions plates in this tutorial taylor created just one pass through your die cut machine using the embossing pad kit is all you need to do  super easy"                                                                                 
[6] "if you have an alternative argument lets hear it "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

Data from news:

[1] "he wasnt home alone apparently"                                                                                                                                                                                                                                                                                                                                                                                                                                                               
[2] "the st louis plant had to close it would die of old age workers had been making cars there since the onset of mass automotive production in the s"                                                                                                                                                                                                                                                                                                                                            
[3] "wsus plans quickly became a hot topic on local online sites though most people applauded plans for the new biomedical center many deplored the potential loss of the building"                                                                                                                                                                                                                                                                                                                
[4] "the alaimo group of mount holly was up for a contract last fall to evaluate and suggest improvements to trenton water works but campaign finance records released this week show the two employees donated a total of  to the political action committee pac partners for progress in early june partners for progress reported it gave more than  in both direct and inkind contributions to mayor tony mack in the two weeks leading up to his victory in the mayoral runoff election june "
[5] "and when its often difficult to predict a laws impact legislators should think twice before carrying any bill is it absolutely necessary is it an issue serious enough to merit their attention will it definitely not make the situation worse"                                                                                                                                                                                                                                              
[6] "there was a certain amount of scoffing going around a few years ago when the nfl decided to move the draft from the weekend to prime time  eventually splitting off the first round to a separate day"                                                                                                                                                                                                                                                                                        

Data from twitter:

[1] "how are you btw thanks for the rt you gonna be in dc anytime soon love to see you been way way too long" 
[2] "when you meet someone special youll know your heart will beat more rapidly and youll smile for no reason"
[3] "theyve decided its more fun if i dont"                                                                   
[4] "so tired d played lazer tag  ran a lot d ughh going to sleep like in  minutes "                          
[5] "words from a complete stranger made my birthday even better "                                            
[6] "first cubs game ever wrigley field is gorgeous this is perfect go cubs go"                               

Line and word counts

         data_files     lines      words
1  en_US.blogs.txt    899,288 37,292,095
2  en_US.news.txt   1,010,242 34,306,494
3 en_US.twitter.txt 2,360,148 30,107,874

Summary

We have completed basic data exploration on the 3 data sets. Initial inspection shows that the dasta sets are quite large for a our computing resources and we will need to be careful with how we store and manipulate the data to control both memory consumption and calculation times.