Assingnment capstone project, week two: Milestone Report

Importing and cleaning data

Files imported

Name	Length	Date
final/	0	2014-07-22 10:10:00
final/de_DE/	0	2014-07-22 10:10:00
final/de_DE/de_DE.twitter.txt	75578341	2014-07-22 10:11:00
final/de_DE/de_DE.blogs.txt	85459666	2014-07-22 10:11:00
final/de_DE/de_DE.news.txt	95591959	2014-07-22 10:11:00
final/ru_RU/	0	2014-07-22 10:10:00
final/ru_RU/ru_RU.blogs.txt	116855835	2014-07-22 10:12:00
final/ru_RU/ru_RU.news.txt	118996424	2014-07-22 10:12:00
final/ru_RU/ru_RU.twitter.txt	105182346	2014-07-22 10:12:00
final/en_US/	0	2014-07-22 10:10:00
final/en_US/en_US.twitter.txt	167105338	2014-07-22 10:12:00
final/en_US/en_US.news.txt	205811889	2014-07-22 10:13:00
final/en_US/en_US.blogs.txt	210160014	2014-07-22 10:13:00
final/fi_FI/	0	2014-07-22 10:10:00
final/fi_FI/fi_FI.news.txt	94234350	2014-07-22 10:11:00
final/fi_FI/fi_FI.blogs.txt	108503595	2014-07-22 10:12:00
final/fi_FI/fi_FI.twitter.txt	25331142	2014-07-22 10:10:00

Cleaning and size of files and samples

Length, characters and words in each dataset

Dataset	Length_Datasets	Length_Samples	MaxLineCharacters	AvLineCharacters
blogs	899288	899	2013	233.81090
news	1010242	1010	1593	199.60990
twitter	2360148	2360	140	69.22034

	Dataset	Words_Datasets	Words_Samples
blogs_US.words	blogs	37546246	38121
news_US.words	news	34762395	34354
tweets_US.words	twitter	30093369	30299

N-grams

N-gram 1 word (kind and number of more frequent words in the sample)

## # A tibble: 15,981 x 2
##    word1     n
##    <chr> <int>
##  1 the    4683
##  2 to     2801
##  3 and    2450
##  4 a      2392
##  5 of     1962
##  6 in     1599
##  7 i      1578
##  8 for    1144
##  9 is     1078
## 10 you    1021
## # ... with 15,971 more rows

N-gram 2 words (kind and number of more frequent pairs of words in the sample)

## # A tibble: 69,535 x 3
##    word1 word2     n
##    <chr> <chr> <int>
##  1 of    the     405
##  2 in    the     397
##  3 on    the     212
##  4 for   the     201
##  5 to    the     195
##  6 at    the     174
##  7 to    be      155
##  8 and   the     135
##  9 is    a       117
## 10 for   a       113
## # ... with 69,525 more rows

N-gram 3 words (kind and number of more frequent triplets of words in the sample)

## # A tibble: 95,291 x 4
##    word1  word2 word3     n
##    <chr>  <chr> <chr> <int>
##  1 one    of    the      37
##  2 a      lot   of       32
##  3 thanks for   the      25
##  4 you    have  to       19
##  5 i      want  to       18
##  6 be     able  to       17
##  7 some   of    the      16
##  8 this   is    a        15
##  9 to     be    a        15
## 10 of     the   day      13
## # ... with 95,281 more rows

N-gram 4 words (kind and number of more frequent quadruplets words in the sample)

## # A tibble: 99,626 x 5
##    word1  word2      word3  word4      n
##    <chr>  <chr>      <chr>  <chr>  <int>
##  1 thanks for        the    follow    10
##  2 at     the        same   time       8
##  3 when   it         comes  to         8
##  4 in     the        middle of         7
##  5 the    end        of     the        7
##  6 get    permission want   to         6
##  7 to     be         able   to         6
##  8 at     the        end    of         5
##  9 num    million    or     num        5
## 10 the    rest       of     the        5
## # ... with 99,616 more rows

N-gram 5 words (kind and number of more frequent quintuplets words in the sample)

## # A tibble: 100,135 x 6
##    word1   word2    word3    word4     word5         n
##    <chr>   <chr>    <chr>    <chr>     <chr>     <int>
##  1 a       balanced and      healthful ecology       4
##  2 in      the      middle   of        a             4
##  3 million or       num      percent   for           4
##  4 num     million  or       num       percent       4
##  5 right   to       a        balanced  and           4
##  6 to      a        balanced and       healthful     4
##  7 allow   it       we       could     have          3
##  8 at      the      end      of        the           3
##  9 for     those    of       you       who           3
## 10 i       step     on       the       court         3
## # ... with 100,125 more rows

Histograms of more frequent words

One Word

Two Words

Three Words

The Word Cloud of more frequent words

The Next step

With this databases I will create an interactive predictive n-gram model in shiny.

Acknowledgments

Basic codes were taken from tidytext and corpus packages at CRAN server. This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

Appendix : Codes to develop the assignment

Importing libraries

# make this an external chunk that can be included in any file
library(knitr)
 library(dplyr)
library(ggplot2)
library(tidyr)
library(tidytext)
library(rJava)
 ## install.packages("RWeka")  
 library(RWeka)
        ## install.packages("NLP")
library(NLP)
      ##  install.packages("qdap")
library(qdap)
library(tm)
        

options(width = 100)
opts_chunk$set(message = F, error = F, warning = F, comment = NA, fig.align = 'center', dpi = 100, tidy = F, cache.path = '.cache/', fig.path = 'fig/')

options(xtable.type = 'html')
knit_hooks$set(inline = function(x) {
  if(is.numeric(x)) {
    round(x, getOption('digits'))
  } else {
    paste(as.character(x), collapse = ', ')
  }
})
knit_hooks$set(plot = knitr:::hook_plot_html)

Importing and cleaning data

setwd("/Volumes/Javier_SD/Learning2/capstone")
        
        library(curl)

        if (!file.exists("./Coursera-SwiftKey.zip")) {
        fileUrl <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
        download.file(fileUrl, destfile = "./Coursera-SwiftKey.zip", method = "curl")
}

        file.exists("./Coursera-SwiftKey.zip")
        file.info("./Coursera-SwiftKey.zip")

     list <- unzip("./Coursera-SwiftKey.zip", files = NULL, list = TRUE, overwrite = TRUE, junkpaths = FALSE, exdir = ".", unzip = "internal", setTimes = FALSE)
    
     file.info("./Coursera-SwiftKey/final/en_US/en_US.twitter.txt")$size
        
        news_US <- readLines(unzip("./Coursera-SwiftKey.zip", files = "final/en_US/en_US.news.txt"), n = -1L, ok = TRUE, warn = TRUE, encoding = "unknown", skipNul = FALSE)

        blogs_US <- readLines(unzip("./Coursera-SwiftKey.zip", files = "final/en_US/en_US.blogs.txt"), n = -1L, ok = TRUE, warn = TRUE, encoding = "unknown", skipNul = FALSE)
        
        tweets_US <- readLines(unzip("./Coursera-SwiftKey.zip", files = "final/en_US/en_US.twitter.txt"), n = -1L, ok = TRUE, warn = TRUE, encoding = "unknown", skipNul = FALSE)
        
         set.seed(10-3-2018)
        
        sample.blogs <- sample(blogs_US, length(blogs_US) * 0.001)
        sample.news <- sample(news_US, length(news_US) * 0.001)
        sample.tweets <- sample(tweets_US, length(tweets_US) * 0.001)
        
         sample <- as.data.frame(c(sample.blogs, sample.news, sample.tweets))
         library(tm)
         corpora_US <- VCorpus(VectorSource(sample))

Files imported

knitr::kable(list)

Cleaning and size of files and samples

        Length_sample.blogs <- nchar(sample.blogs)
        Length_sample.news <- nchar(sample.news)
        Length_sample.tweets <- nchar(sample.tweets)
        table1 <-data.frame(Dataset = c("blogs", "news", "twitter"),
                   Length_Datasets = c(length(blogs_US), length(news_US), length(tweets_US)),
                   Length_Samples  = c(length(sample.blogs), length(sample.news), length(sample.tweets)),
                   MaxLineCharacters = c(max(Length_sample.blogs),max(Length_sample.news),max(Length_sample.tweets)),
                   AvLineCharacters = c(mean(Length_sample.blogs),mean(Length_sample.news),mean(Length_sample.tweets))
        )

Length, characters and words in each dataset

knitr::kable(table1)

N-grams (Words)

        corpora_US <- tolower(corpora_US)
        corpora_US<- gsub("\\d+[[:punct:]]*\\s*\\d*","NUM",corpora_US)
        corpora_US <- gsub("-","",corpora_US)
        corpora_US <- stripWhitespace(corpora_US)
        corpora_US <- VCorpus(VectorSource(corpora_US)) 
       corpora_US <- tm_map(corpora_US, removePunctuation, preserve_intra_word_contractions = TRUE, preserve_intra_word_dashes = TRUE)
        library(tidyr)
        library(tidytext)
        library(tidyverse)

N-gram 1 word (kind and number of more frequent words in the sample)

        onegram_data <- tidy(corpora_US) %>%
                unnest_tokens(onegrams_data, text, token = "ngrams",to_lower = TRUE, n= 1) %>% separate(onegrams_data, c("word1"), sep = " ") %>%
                count(word1, sort = TRUE)
        
        
        bigrams_data <- tidy(corpora_US) %>%
                unnest_tokens(bigrams_data, text, token = "ngrams",to_lower = TRUE, n= 2) %>% separate(bigrams_data, c("word1", "word2"), sep = " ") %>%
                count(word1, word2, sort = TRUE)
        
        threegrams_data <- tidy(corpora_US) %>%
                unnest_tokens(threegrams_data, text, token = "ngrams",to_lower = TRUE, n= 3) %>% separate(threegrams_data, c("word1", "word2", "word3"), sep = " ") %>%
                count(word1, word2, word3, sort = TRUE)
        
        fourgrams_data <- tidy(corpora_US) %>%
                unnest_tokens(fourgrams_data, text, token = "ngrams",to_lower = TRUE, n= 4) %>% separate(fourgrams_data, c("word1", "word2", "word3", "word4"), sep = " ") %>%
                count(word1, word2, word3, word4, sort = TRUE) 
        
        fivegrams_data <- tidy(corpora_US) %>%
                unnest_tokens(fivegrams_data, text, token = "ngrams",to_lower = TRUE, n= 5) %>% separate(fivegrams_data, c("word1", "word2", "word3", "word4", "word5"), sep = " ") %>% count(word1, word2, word3, word4, word5, sort = TRUE)



onegram_data
## knitr::kable(onegram_data)[1:10]

N-gram 2 words (kind and number of more frequent pairs of words in the sample)

bigrams_data
## knitr::kable(bigrams_data)[1:10]

N-gram 3 words (kind and number of more frequent triplets of words in the sample)

threegrams_data
##knitr::kable(threegrams_data)[1:10]

N-gram 4 words (kind and number of more frequent quadruplets words in the sample)

fourgrams_data
##knitr::kable(fourgrams_data)[1:10]

N-gram 5 words (kind and number of more frequent quintuplets words in the sample)

fivegrams_data
##knitr::kable(fivegrams_data)[1:10]

Histograms of more frequent words

One word

library(data.table)

onegram_data.sort <- setorder(onegram_data, -n)
subset(onegram_data.sort, onegram_data.sort$n>250)    %>% ##sort(onegram_data, decreasing = TRUE) %>%
        ggplot(aes(reorder(word1, -n), n), onegram_data.sort) +
        geom_bar(stat="identity", fill="blue", colour="darkblue") +
        theme(axis.text.x=element_text(angle=90, hjust=1)) +
        labs(x="Word", title = "One word frequency")

Two words

library(data.table)
bigrams <- as.data.frame(cbind(Words =paste(bigrams_data$word1, bigrams_data$word2), n=bigrams_data$n))
bigrams$n <- as.numeric(bigrams$n)
bigrams_data.sort <- setorder(bigrams, -n)
subset(bigrams_data.sort)[1:30,]    %>%
        ggplot(aes(reorder(Words, -n), n), bigrams_data.sort) +
        geom_bar(stat="identity", fill="blue", colour="darkblue") +
        theme(axis.text.x=element_text(angle=90, hjust=1)) +
        labs(x="Words", title = "Two-words frequency")

Three words

library(data.table)
threegrams <- as.data.frame(cbind(Words =paste(threegrams_data$word1, threegrams_data$word2, threegrams_data$word3), n=threegrams_data$n))
threegrams$n <- as.numeric(threegrams$n)
threegrams_data.sort <- setorder(threegrams, -n)
subset(threegrams_data.sort)[1:30,]    %>%
        ggplot(aes(reorder(Words, -n), n), threegrams_data.sort) +
        geom_bar(stat="identity", fill="blue", colour="darkblue") +
        theme(axis.text.x=element_text(angle=90, hjust=1)) +
        labs(x="Words", title = "Three-words frequency")

The Word cloud of more frequent words

library(wordcloud)
wordcloud(bigrams$Words,bigrams$n,scale=c(4,.5), min.freq=3,max.words=Inf,random.order = TRUE, colors = blues9, ordered.colors = F, rot.per = .1)

Assingnment capstone project, week two: Milestone Report

Javier Eslava-Schmalbach

10/16/2018

Importing and cleaning data

Files imported

Cleaning and size of files and samples

Length, characters and words in each dataset

N-grams

N-gram 1 word (kind and number of more frequent words in the sample)

N-gram 2 words (kind and number of more frequent pairs of words in the sample)

N-gram 3 words (kind and number of more frequent triplets of words in the sample)

N-gram 4 words (kind and number of more frequent quadruplets words in the sample)

N-gram 5 words (kind and number of more frequent quintuplets words in the sample)

Histograms of more frequent words

One Word

Two Words

Three Words

The Word Cloud of more frequent words

The Next step

Acknowledgments

Appendix : Codes to develop the assignment

Importing libraries

Importing and cleaning data

Files imported

Cleaning and size of files and samples

Length, characters and words in each dataset

N-grams (Words)

N-gram 1 word (kind and number of more frequent words in the sample)

N-gram 2 words (kind and number of more frequent pairs of words in the sample)

N-gram 3 words (kind and number of more frequent triplets of words in the sample)

N-gram 4 words (kind and number of more frequent quadruplets words in the sample)

N-gram 5 words (kind and number of more frequent quintuplets words in the sample)

Histograms of more frequent words

One word

Two words

Three words

The Word cloud of more frequent words