Peer-graded Assignment: Milestone Report
Usama Fouad
3/8/2020
Instructions
The goal of this project is just to display that you’ve gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs (http://rpubs.com/) that explains your exploratory analysis and your goals for the eventual app and algorithm. This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set. The motivation for this project is to: 1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.2. Create a basic report of summary statistics about the data sets.3. Report any interesting findings that you amassed so far.4. Get feedback on your plans for creating a prediction algorithm and Shiny app.
Downloading and Loading Data
require('tm')## Loading required package: tm## Loading required package: NLPrequire('ggplot2')## Loading required package: ggplot2##
## Attaching package: 'ggplot2'## The following object is masked from 'package:NLP':
##
## annotaterequire('stringi')## Loading required package: stringisetwd("/media/usama/F/NTL/Data Science/10_Capstone/final/en_US/")
blogFile <- "en_US.blogs.txt"
twittFile <- "en_US.twitter.txt"
blogLines <- readLines(file(blogFile))
twittLines <- readLines(file(twittFile))## Warning in readLines(file(twittFile)): line 167155 appears to contain an
## embedded nul## Warning in readLines(file(twittFile)): line 268547 appears to contain an
## embedded nul## Warning in readLines(file(twittFile)): line 1274086 appears to contain an
## embedded nul## Warning in readLines(file(twittFile)): line 1759032 appears to contain an
## embedded nulThe first 6 tweets in the en_US.twitter.txt file
head(twittLines)## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."
## [4] "So Tired D; Played Lazer Tag & Ran A LOT D; Ughh Going To Sleep Like In 5 Minutes ;)"
## [5] "Words from a complete stranger! Made my birthday even better :)"
## [6] "First Cubs game ever! Wrigley field is gorgeous. This is perfect. Go Cubs Go!"The first 6 sentences in the en_US.blogs.txt file
head(blogLines)## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him."
## [4] "so anyways, i am going to share some home decor inspiration that i have been storing in my folder on the puter. i have all these amazing images stored away ready to come to life when we get our home."
## [5] "With graduation season right around the corner, Nancy has whipped up a fun set to help you out with not only your graduation cards and gifts, but any occasion that brings on a change in one's life. I stamped the images in Memento Tuxedo Black and cut them out with circle Nestabilities. I embossed the kraft and red cardstock with TE's new Stars Impressions Plate, which is double sided and gives you 2 fantastic patterns. You can see how to use the Impressions Plates in this tutorial Taylor created. Just one pass through your die cut machine using the Embossing Pad Kit is all you need to do - super easy!"
## [6] "If you have an alternative argument, let's hear it! :)"Twitter File Sentences number
length(twittLines)## [1] 2360148Twitter File Sentences number is 2360148
Blog File Sentences number
length(blogLines)## [1] 899288Blog File Sentences number is 899288
For each word in twitter and blog fies get the frequency for each word
twittWordsNo <- stri_count_words(twittLines)
blogWordsNo <- stri_count_words(blogLines)head(twittWordsNo)## [1] 24 19 8 18 10 14head(blogWordsNo)## [1] 16 5 139 40 110 9Summary of Twitter
Get the summary for all words frequencies in the twitter file
summary(twittWordsNo)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.00 7.00 12.00 12.75 18.00 47.00Each word repeated at least 1 times and at most 47 times, and the average are repeated about 12 times … thhe 1st quantile is 7 and the 3rd quantile is 18.
Summary of Blog
Get the summary for all words frequencies in the blog file
summary(blogWordsNo)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 9.00 28.00 41.75 60.00 6726.00Each word repeated at least 0 times and at most 6726 times, and the average are repeated about 41 times … the 1st quantile is 9 and the 3rd quantile is 60.
Get sample sentences from each file
sblogLines <- sample(blogLines, 0.1*length(blogLines))
stwittLines <- sample(twittLines, 0.1*length(twittLines))
blogWordsNo <- stri_count_words(sblogLines)
stwittWordsNo <- stri_count_words(stwittLines)
allWorsNo.DF <- data.frame(n = c(blogWordsNo, stwittWordsNo),
tp = c(rep("blog", length(blogWordsNo)), rep("twitter",length(stwittWordsNo))))
head(sblogLines)## [1] "people to plan/prepare for themselves."
## [2] "They serve it with Creme Friache, I used whipped cream."
## [3] "Senior Sabah Umno lawmaker Datuk Bung Mokhtar Radin, who is also the BN backbenchers’ deputy chief, added that the ruling party is “ready enough” to follow Najib who will lead BN into polls for the first time."
## [4] "I figured I’d drop this here as an addendum to my prior post on dystopia. Like the definition there, this is one my students and I co-wrote based on my lecture comments on genre characteristics and reading an excerpt from More’s Utopia (Book 2, obviously). I wanted us to come up with a definition because those online and in lit. terms books were not particularly good (to put it kindly). However, it could still use some revision. Also, one of the most useful discussions of the problems of defining utopia is found in the introduction to Lyman Tower Sargent’s British and American Utopian Literature: 1516-1975."
## [5] "I found I did get creasing each time I used the shadows, but given the humidity and my insane sweatiness lately, that's happening with most eye looks I do, regardless of the products used."
## [6] "Karl: you lost in 1976"Density probability Plot for frequency of number of words in the line
ggplot(data = allWorsNo.DF) + geom_density(aes(n)) + facet_wrap(~tp, nrow = 2) + xlim(0,200) ## Warning: Removed 866 rows containing non-finite values (stat_density).