Download the data and open them:

Data are already provided. The dataset contains 3 sources of text: blogs, news and tweets in four languages including English, Finnish, German and Russian. For this project the English texts have been selected to be analyzed.

blogs <- readLines("/Users/asudeberber/Documents/Coursera § Mycourses/DataScienceSpecializationJohnHopkins/Final Capstone Project/final/en_US/en_US.blogs.txt", skipNul = TRUE, warn = FALSE)
news <- readLines("/Users/asudeberber/Documents/Coursera § Mycourses/DataScienceSpecializationJohnHopkins/Final Capstone Project/final/en_US/en_US.news.txt", skipNul = TRUE, warn = FALSE)
twitter <- readLines("/Users/asudeberber/Documents/Coursera § Mycourses/DataScienceSpecializationJohnHopkins/Final Capstone Project/final/en_US/en_US.twitter.txt", skipNul = TRUE, warn=FALSE)

Introduction

This project has been done for Coursera JHU Data Science Specialization Capstone Final Project Report. In this task, called milestone report, aim is to develop word predictor tool, which based on commonly used phrases in a collection of texts will predict the next word. The outputs of this project will be presented as exploratory data analysis (bi-gram, tri-gram). Later, the project aims to develop shiny application.

Summary of the data

Summary statistics of the three English texts is presented in Table. data are so big so it is enough just to readLines function

library(readr)
library(readxl)
library(ggplot2)
library(NLP)
## 
## Attaching package: 'NLP'
## The following object is masked from 'package:ggplot2':
## 
##     annotate
library(tm)
## Warning: package 'tm' was built under R version 4.3.2
library(RWeka)
require(devtools)
## Loading required package: devtools
## Loading required package: usethis

#Data information Data are huge. Therefore we need to do data sampling

set.seed(100)
sample_size = 1000

sample_blog <- blogs[sample(1:length(blogs),sample_size)]
sample_news <- news[sample(1:length(news),sample_size)]
sample_twitter <- twitter[sample(1:length(twitter),sample_size)]
head(sample_blog)
## [1] "Note: Apparently this was a sequel to another TV-movie made earlier the same year (titled Killer Hair), and both were based on the “Crimes of Fashion” book series by Ellen Byerrum. Feh."                                                                                        
## [2] "Knead all crust ingredients and spread them on the bottom of a pie / tart form. Prick with a fork and blind-bake at 160 degrees C for ten minutes. Set aside."                                                                                                                    
## [3] "Corruption and cronyism are endemic in Greece. But no politician or senior businessman had been convicted in recent years, fuelling popular frustration with mainstream parties that pledge to uphold the debt-laden country’s international bailout and remain in the euro zone."
## [4] "Completed: March 2012"                                                                                                                                                                                                                                                            
## [5] "Here are the fabulous Design Team creations for this week! Hope they inspire you to have a play yourselves. Do pay them all a visit and let them know what you think!"                                                                                                            
## [6] "As performance art goes, I say well done! However, as far as efficiency, customer service and general awareness of how germs are spread goes, I say holy crap this is UNBELIEVABLE!"
head(sample_twitter)
## [1] "Good point. We were both too cheap to get a \"Razr\" too. Damn those cool kids. RT Correction: I was every day of 28."
## [2] "Paul Pierce just passed Larry Bird on Celtics All-Time Scoring List, that one makes you stop and think!"              
## [3] "he said I only have 1 minute and 60 seconds left to get to class, whaa? okay so basically 2 minutes?"                 
## [4] "DOWNLOAD MY NEW TRACK \"LEAN WIT IT\" RETWEET"                                                                        
## [5] "Dijon vu - the same mustard as before."                                                                               
## [6] "You're very welcome - it was great!"
head(sample_news)
## [1] "Tall and lithe, this Blanche is less fading flower than resilient reed. You can tell she’s endured many storms by bending to the winds. Her sharp wits must have helped — she describes a trip to Miami as “an investment, thinking I’d meet someone with a million dollars” — and Parker earns her laughs."      
## [2] "\u0095 13800 block of Red Hill, shoplifter"                                                                                                                                                                                                                                                                       
## [3] "St. Louisans Jane Ellen Ibur and Sondra Seiler offer counsel to a late-blooming lesbian in the documentary \"Jan's Coming Out,\" screening at 7 p.m. April 24."                                                                                                                                                   
## [4] "There were conflicting accounts on the number of dead in Wednesday's attack, ranging from at least six to at least 10."                                                                                                                                                                                           
## [5] "Bergenfield native Peter Negron had been there before. At age 13, two years after the death of his father, also named Peter, Negron read a poem during the second anniversary of the attack. Yesterday, that young boy, now a 21-year-old man, spoke again."                                                      
## [6] "Trailing 35-22 at halftime, Monmouth rallied in the second half and led, 72-70, on a follow by George Barbour with 11.1 seconds left in regulation, but Eugene Harvey (27 points) hit two foul shots with 5.5 seconds left to force overtime. The Pirates dominated the extra session to win their season opener."
sample_data<-rbind(sample_blog,sample_news,sample_twitter)
rm(blogs,news,twitter)

Clean the data

mycorpus <- VCorpus(VectorSource(sample_data))
mycorpus <- tm_map(mycorpus, content_transformer(tolower)) # convert to lowercase
mycorpus <- tm_map(mycorpus, removePunctuation) # remove punctuation
mycorpus <- tm_map(mycorpus, removeNumbers) # remove numbers
mycorpus <- tm_map(mycorpus, stripWhitespace) # remove multiple whitespace
changetospace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
mycorpus <- tm_map(mycorpus, changetospace, "/|@|\\|")

signing the sentences (tokenizing) and analyzing of one two three gram, oneGM - twoGM - threeGM respectively.

It refers to n-gram matrix method = a collection of n successive items in a text document that may include words, numbers, symbols, and punctuation

uniGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
biGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
triGramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
OneT <- NGramTokenizer(mycorpus, Weka_control(min = 1, max = 1))
oneGM <- TermDocumentMatrix(mycorpus, control = list(tokenize = uniGramTokenizer))
twoGM <- TermDocumentMatrix(mycorpus, control = list(tokenize = biGramTokenizer))
threeGM <- TermDocumentMatrix(mycorpus, control = list(tokenize = triGramTokenizer))

Generation of histogram for text freuqency

Unigram frequency = assumed that the occurrence of each word is independent of its previous word.

freqTerms <- findFreqTerms(oneGM, lowfreq = 200)
termFreq <- rowSums(as.matrix(oneGM[freqTerms,]))
termFreq <- data.frame(unigram=names(termFreq), frequency=termFreq)

g1 <- ggplot(termFreq, aes(x=reorder(unigram, frequency), y=frequency)) +
    geom_bar(stat = "identity") +  coord_flip() +
    theme(legend.title=element_blank()) +
    xlab("Unigram") + ylab("Frequency") +
    labs(title = "Top unigrams by frequency")
print(g1)

#"the" the most repeated one

bigram frequency

Measures how often a pair of letters occurs / as two words

freqTerms <- findFreqTerms(twoGM, lowfreq = 70)
termFreq <- rowSums(as.matrix(twoGM[freqTerms,]))
termFreq <- data.frame(bigram=names(termFreq), frequency=termFreq)

g2 <- ggplot(termFreq, aes(x=reorder(bigram, frequency), y=frequency)) +
    geom_bar(stat = "identity") +  coord_flip() +
    theme(legend.title=element_blank()) +
    xlab("Bigram") + ylab("Frequency") +
    labs(title = "Top bigrams by frequency")
print(g2)

# "in the" the most repeated one

Trigram frequency

a sequence of three adjacent letters or symbols. As three words

freqTerms <- findFreqTerms(threeGM, lowfreq = 10)
termFreq <- rowSums(as.matrix(threeGM[freqTerms,]))
termFreq <- data.frame(trigram=names(termFreq), frequency=termFreq)

g3 <- ggplot(termFreq, aes(x=reorder(trigram, frequency), y=frequency)) +
    geom_bar(stat = "identity") +  coord_flip() +
    theme(legend.title=element_blank()) +
    xlab("Trigram") + ylab("Frequency") +
    labs(title = "Top trigrams by frequency")
print(g3)

# "one of the" the most repeated one

Next plan

Developing a natural language processing (NLP) algorithm capable of predicting the most likely word a user is to type next. Development of Shiny app.