Capstone Project Initial Report
Samer Mestom
8/14/2019
Report Descritpion
This is an initial report for the capstone project. It is a high level report that will illustrate the following:
1-Capture the data and load it into vectors. 2-Create a basic report for summary statistics. 3-Report any interesting finding. 4-Make a summary how to create a prediction algorithm and use Shiny app.
1-Data Capture
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(stringr)
setwd("D://Box Sync//Data Science//Capstone Project//final//en_US")
con1 <- file("en_US.twitter.txt", "r")
con2 <- file("en_US.news.txt", "r")
con3 <- file("en_US.blogs.txt", "r")
twitter=gsub("[^[:alnum:] ,\\.']","",readLines(con1,-1))
news=gsub("[^[:alnum:] ,\\.']","",readLines(con2,-1))
blogs=gsub("[^[:alnum:] ,\\.']","",readLines(con3,-1))
close(con1)
close(con2)
close(con3)
head(twitter)## [1] "How are you Btw thanks for the RT. You gonna be in DC anytime soon Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."
## [4] "So Tired D Played Lazer Tag Ran A LOT D Ughh Going To Sleep Like In 5 Minutes "
## [5] "Words from a complete stranger Made my birthday even better "
## [6] "First Cubs game ever Wrigley field is gorgeous. This is perfect. Go Cubs Go"head(news)## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."
## [4] "The Alaimo Group of Mount Holly was up for a contract last fall to evaluate and suggest improvements to Trenton Water Works. But campaign finance records released this week show the two employees donated a total of 4,500 to the political action committee PAC Partners for Progress in early June. Partners for Progress reported it gave more than 10,000 in both direct and inkind contributions to Mayor Tony Mack in the two weeks leading up to his victory in the mayoral runoff election June 15."
## [5] "And when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. Is it absolutely necessary Is it an issue serious enough to merit their attention Will it definitely not make the situation worse"
## [6] "There was a certain amount of scoffing going around a few years ago when the NFL decided to move the draft from the weekend to prime time eventually splitting off the first round to a separate day."head(blogs)## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan âgodsâ."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it he never taps into that thing either, that is how we know he wanted it so bad. We made him count all of his money to make sure that he had enough It was very cute to watch his reaction when he realized he did He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters She loves it almost as much as him."
## [4] "so anyways, i am going to share some home decor inspiration that i have been storing in my folder on the puter. i have all these amazing images stored away ready to come to life when we get our home."
## [5] "With graduation season right around the corner, Nancy has whipped up a fun set to help you out with not only your graduation cards and gifts, but any occasion that brings on a change in one's life. I stamped the images in Memento Tuxedo Black and cut them out with circle Nestabilities. I embossed the kraft and red cardstock with TE's new Stars Impressions Plate, which is double sided and gives you 2 fantastic patterns. You can see how to use the Impressions Plates in this tutorial Taylor created. Just one pass through your die cut machine using the Embossing Pad Kit is all you need to do super easy"
## [6] "If you have an alternative argument, let's hear it "en_us.twitter.txt
## [1] "Number of lines in en_us.twitter.txt is 2360148"## [1] "Number of words in en_us.twitter.txt is 32040305"## [1] "Most used 10 words"## word rep
## 2 the 935637
## 3 to 787239
## 4 i 724161
## 5 a 612489
## 6 you 546845
## 7 and 437363
## 8 for 385098
## 9 in 377786
## 10 of 359172
## 11 is 358562
en_us.news.txt
## [1] "Number of lines in en_us.news.txt is 77259"## [1] "Number of words in en_us.news.txt is 2894312"## [1] "Most used 10 words"## word rep
## 2 the 151502
## 3 to 69350
## 4 a 68288
## 5 and 68219
## 6 of 59090
## 7 in 51462
## 8 for 27109
## 9 that 26353
## 10 is 21952
## 11 on 20573
en_us.blog.txt
## [1] "Number of lines in en_us.blogs.txt is 899288"## [1] "Number of words in en_us.blogs.txt is 40533477"## [1] "Most used 10 words"## word rep
## 2 the 1856794
## 3 and 1088814
## 4 to 1065938
## 5 a 899878
## 6 of 875181
## 7 i 774429
## 8 in 593948
## 9 that 459937
## 10 is 432084
## 11 it 402503
2-Intersting Finding:
It is clear that the basic used words in English are “the”, “to”, “a”, etc. as in the above three lists. The difference in sequence depends on the context of the sentence. For example “I” and “you” are commonly used in twitter while they are of minimal use in news or blogs.
3-Creating prediction algorithm:
The basis of my work is to make 2-gram, 3-gram and 4-gram statistics. That would be used to build my statistical models. Naive bayes prediction will be using those models to predict the next word.
4-Project Interface:
The interface would be a basic web application made by Shiny App. It would be simple enough to illustrate the basic needed functions of the project. Just a simple text input field that is continulusly read the previous word inputs as suggesting the new word. You can either select the suggested word by the app or input your own word.