This capstone project introduces the basics of natural language processing, analyzing a large corpus of text documents to discover the structure in the data and how words are put together. The initail phase of the project covers cleaning and analyzing text data. Later phases focus on building and sampling from a predictive text model.
The goal of the project is to predict a word after a user types the word in smart keyboard. It will use NPL N-grams to predict the natuarl word to suggest after a user types in a given word. The idea is train a model with data from news, blogs and tweets and the let model suggests a ward based on propability.
The projects focuses on data set written in US english. This analysis can be expanded to the same dataset written in different langiages with mininal reqords. Project files are: en_US, US_blogs and US.twitter The data is from a corpus called HC Corpora
Libraries:
library(stringi)
library(ggplot2)
Blogs data:
con <- file("~/Downloads/final/en_US/en_US.blogs.txt", open="rb")
blogs <- readLines(con, encoding = "UTF-8")
close(con)
rm(con)
blogs.words <- sum(sapply(gregexpr("\\W+", blogs), length))
Twitter data:
con <- file("~/Downloads/final/en_US/en_US.twitter.txt", open="rb")
tweets <- readLines(con, encoding = "UTF-8")
## Warning in readLines(con, encoding = "UTF-8"): line 167155 appears to
## contain an embedded nul
## Warning in readLines(con, encoding = "UTF-8"): line 268547 appears to
## contain an embedded nul
## Warning in readLines(con, encoding = "UTF-8"): line 1274086 appears to
## contain an embedded nul
## Warning in readLines(con, encoding = "UTF-8"): line 1759032 appears to
## contain an embedded nul
tweets <- iconv(tweets, from = "latin1", to = "UTF-8", sub="")
close(con)
rm(con)
tweets.words <- sum(sapply(gregexpr("\\W+", tweets), length))
News data:
con <- file("~/Downloads/final/en_US/en_US.news.txt", open="rb")
news <- readLines(con, encoding="UTF-8")
close(con)
rm(con)
news.words <- sum(sapply(gregexpr("\\W+", news), length))
Summary statistics: Blogs
summary(blogs)
## Length Class Mode
## 899288 character character
head(blogs, 2)
## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”."
## [2] "We love you Mr. Brown."
Number of words in blogs:
blogs.words
## [1] 38221261
Summary statistics: News
summary(news)
## Length Class Mode
## 1010242 character character
head(news, 2)
## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
Number of words in news:
news.words
## [1] 35710845
Summary statistics: Tweets
summary(tweets)
## Length Class Mode
## 2360148 character character
head(tweets, 2)
## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
Number of words in news:
tweets.words
## [1] 30514034
Summary of word counts for each dataset of each line
counts.en.blogs <- unlist(stri_count_words(blogs))
counts.en.news <- unlist(stri_count_words(news))
counts.en.twitter <- unlist(stri_count_words(tweets))
Distribution summary of each datasets:
summary(counts.en.blogs)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 9.00 28.00 41.75 60.00 6726.00
summary(counts.en.news)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.00 19.00 32.00 34.41 46.00 1796.00
summary(counts.en.twitter)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.00 7.00 12.00 12.79 18.00 61.00
Frequencies table for blogs wth more than the average number of words in a blog.
table(counts.en.blogs > mean(counts.en.blogs))
##
## FALSE TRUE
## 555497 343791
Frequencies table for news articles with more than the average number of words.
table(counts.en.news > mean(counts.en.news))
##
## FALSE TRUE
## 566065 444177
Frequencies table for twitts with more than the average word in a tweet.
table(counts.en.twitter > mean(counts.en.twitter))
##
## FALSE TRUE
## 1236535 1123613
Ploting data distribution for each data set:
Blogs data:
ggplot(as.data.frame(counts.en.blogs), aes(x=counts.en.blogs)) +
geom_histogram(binwidth=300, fill="blue", color="black") +
labs(x="Word counts per entry", y="Frequency (log10)", title="Histogram of word counts for blogs") +
scale_y_log10()
## Warning: Stacking not well defined when ymin != 0
News data:
ggplot(as.data.frame(counts.en.news), aes(x=counts.en.news)) +
geom_histogram(binwidth = 80, fill="green", color="black") +
labs(x="Word counts per entry", y="Frequency (log10)", title="Histogram of word counts for new") +
scale_y_log10()
## Warning: Stacking not well defined when ymin != 0
Tweets data:
ggplot(as.data.frame(counts.en.twitter), aes(x=counts.en.twitter)) +
geom_histogram(binwidth = 4, fill="cyan", color="black") +
labs(x="Word counts per entry", y="Frequency (log10)", title="Histogram of word counts for tweets") +
scale_y_log10()
## Warning: Stacking not well defined when ymin != 0
The above figures shows the frequency distribution of each dataset: blogs, news and twitter. The histograms of news category is a left skewed distributon which means that we have lots of words that get used more often than others. The blogs data is also a left skewed histogram for the same reason. However, the tweets data are not skewed to the right not the left, the distribution is almost uniform which means that tweets often appears with slightly same frequency or because we ony have 140 characaters maximum in a given tweet.