NLP project - Explorator analysis
Tan Vu
2024-06-21
1. Loading the datasets
url <- "https://d396qusza40orc.cloudfront.net/dsscapstone/dataset/Coursera-SwiftKey.zip"
download.file(url,destfile = "./datasets.zip")
dir.create("./datasets")
unzip("./datasets.zip",exdir = "./datasets")
dataset_dir <- "./datasets/final/en_US"blogCorpus <- readLines("./datasets/final/en_US/en_US.blogs.txt")
newsCorpus <- readLines("./datasets/final/en_US/en_US.news.txt")
twtCorpus <- readLines("./datasets/final/en_US/en_US.twitter.txt")2. Exploratory analysis
a. Basic exploration
- Text content
head(blogCorpus)## [1] "In the years thereafter, most of the Oil fields and platforms were named after pagan “gods”."
## [2] "We love you Mr. Brown."
## [3] "Chad has been awesome with the kids and holding down the fort while I work later than usual! The kids have been busy together playing Skylander on the XBox together, after Kyan cashed in his $$$ from his piggy bank. He wanted that game so bad and used his gift card from his birthday he has been saving and the money to get it (he never taps into that thing either, that is how we know he wanted it so bad). We made him count all of his money to make sure that he had enough! It was very cute to watch his reaction when he realized he did! He also does a very good job of letting Lola feel like she is playing too, by letting her switch out the characters! She loves it almost as much as him."
## [4] "so anyways, i am going to share some home decor inspiration that i have been storing in my folder on the puter. i have all these amazing images stored away ready to come to life when we get our home."
## [5] "With graduation season right around the corner, Nancy has whipped up a fun set to help you out with not only your graduation cards and gifts, but any occasion that brings on a change in one's life. I stamped the images in Memento Tuxedo Black and cut them out with circle Nestabilities. I embossed the kraft and red cardstock with TE's new Stars Impressions Plate, which is double sided and gives you 2 fantastic patterns. You can see how to use the Impressions Plates in this tutorial Taylor created. Just one pass through your die cut machine using the Embossing Pad Kit is all you need to do - super easy!"
## [6] "If you have an alternative argument, let's hear it! :)"head(newsCorpus)## [1] "He wasn't home alone, apparently."
## [2] "The St. Louis plant had to close. It would die of old age. Workers had been making cars there since the onset of mass automotive production in the 1920s."
## [3] "WSU's plans quickly became a hot topic on local online sites. Though most people applauded plans for the new biomedical center, many deplored the potential loss of the building."
## [4] "The Alaimo Group of Mount Holly was up for a contract last fall to evaluate and suggest improvements to Trenton Water Works. But campaign finance records released this week show the two employees donated a total of $4,500 to the political action committee (PAC) Partners for Progress in early June. Partners for Progress reported it gave more than $10,000 in both direct and in-kind contributions to Mayor Tony Mack in the two weeks leading up to his victory in the mayoral runoff election June 15."
## [5] "And when it's often difficult to predict a law's impact, legislators should think twice before carrying any bill. Is it absolutely necessary? Is it an issue serious enough to merit their attention? Will it definitely not make the situation worse?"
## [6] "There was a certain amount of scoffing going around a few years ago when the NFL decided to move the draft from the weekend to prime time -- eventually splitting off the first round to a separate day."head(twtCorpus)## [1] "How are you? Btw thanks for the RT. You gonna be in DC anytime soon? Love to see you. Been way, way too long."
## [2] "When you meet someone special... you'll know. Your heart will beat more rapidly and you'll smile for no reason."
## [3] "they've decided its more fun if I don't."
## [4] "So Tired D; Played Lazer Tag & Ran A LOT D; Ughh Going To Sleep Like In 5 Minutes ;)"
## [5] "Words from a complete stranger! Made my birthday even better :)"
## [6] "First Cubs game ever! Wrigley field is gorgeous. This is perfect. Go Cubs Go!"- Number of lines of each corpus & average length of lines
length(blogCorpus)## [1] 899288length(newsCorpus)## [1] 77259length(twtCorpus)## [1] 2360148mean(nchar(blogCorpus))## [1] 229.987mean(nchar(newsCorpus))## [1] 202.4283mean(nchar(twtCorpus))## [1] 68.69173b. Word frequency
install.packages("tm")
install.packages("tidytext")
install.packages("readr")library(tm)
library(tidytext)
library(ggplot2)
library(readr)
library(dplyr)
library(stringr)A quick surfing through top 50 most common words within each corpus shows, as expected, the most common words in English.
blog_df <- tibble(text = blogCorpus)
blog_freq <- blog_df %>%
unnest_tokens(word, text) %>%
count(word, sort = TRUE)
blog_freq %>%
top_n(50) %>%
ggplot(aes(x = reorder(word, n), y = n)) +
geom_bar(stat = "identity") +
coord_flip() +
labs(title = "Top 50 Words in Blog Corpus", x = "Words", y = "Frequency")## Selecting by n
news_df <- tibble(text = newsCorpus)
news_freq <- news_df %>%
unnest_tokens(word, text) %>%
count(word, sort = TRUE)
news_freq %>%
top_n(50) %>%
ggplot(aes(x = reorder(word, n), y = n)) +
geom_bar(stat = "identity") +
coord_flip() +
labs(title = "Top 50 Words in News Corpus", x = "Words", y = "Frequency")## Selecting by n
twt_df <- tibble(text = twtCorpus)
twt_freq <- twt_df %>%
unnest_tokens(word, text) %>%
count(word, sort = TRUE)
twt_freq %>%
top_n(50) %>%
ggplot(aes(x = reorder(word, n), y = n)) +
geom_bar(stat = "identity") +
coord_flip() +
labs(title = "Top 50 Words in Twitter Corpus", x = "Words", y = "Frequency")## Selecting by n
c. Bi-gram & Tri-gram check
The bi-gram check of blog corpus unsurprisingl shows the most common pair of words in English, therefore no need to check it for news & twitter corpus, since this task is resource-consuming.
blog_bigram <- blog_df %>%
unnest_tokens(bigram, text, token = "ngrams", n = 2) %>%
count(bigram, sort = TRUE)
print(blog_bigram %>% top_n(20))## Selecting by n## # A tibble: 20 × 2
## bigram n
## <chr> <int>
## 1 of the 187228
## 2 in the 153336
## 3 to the 85990
## 4 on the 75421
## 5 to be 68395
## 6 and the 58656
## 7 for the 58093
## 8 i was 49595
## 9 and i 49465
## 10 i have 48109
## 11 it is 48029
## 12 it was 47815
## 13 at the 47321
## 14 is a 45580
## 15 in a 45393
## 16 with the 43949
## 17 i am 42510
## 18 from the 38098
## 19 that i 37974
## 20 of a 33980The same observation can be drawn with tri-gram check, which I run for news corpus, the shorter one among the 3, since this is very resource-consuming check
news_trigram <- news_df %>%
unnest_tokens(trigram, text, token = "ngrams", n = 3) %>%
count(trigram, sort = TRUE)
print(news_trigram %>% top_n(20))## Selecting by n## # A tibble: 20 × 2
## trigram n
## <chr> <int>
## 1 <NA> 1127
## 2 one of the 1080
## 3 a lot of 877
## 4 as well as 479
## 5 according to the 435
## 6 going to be 420
## 7 in the first 419
## 8 part of the 410
## 9 the end of 408
## 10 out of the 395
## 11 some of the 390
## 12 to be a 382
## 13 the united states 345
## 14 the first time 340
## 15 it was a 332
## 16 be able to 320
## 17 for the first 296
## 18 said in a 293
## 19 of the year 275
## 20 end of the 246d. Unique & common words of each corpus
The list of unique words of each corpus reveal mainly typo, jargons, names or places that should be removed from the buildup of text prediction model.
Interesting enough, the list of common words of 3 corpora but with the least frequency also contains jargons, names & places, which reasonably should also be removed from text prediction model.
blog_unique <- setdiff(blog_freq$word, union(news_freq$word, twt_freq$word))
news_unique <- setdiff(news_freq$word, union(blog_freq$word, twt_freq$word))
twt_unique <- setdiff(twt_freq$word, union(blog_freq$word, news_freq$word))
common_words <- intersect(intersect(blog_freq$word, news_freq$word), twt_freq$word)
head(blog_unique,50)## [1] "stampin" "copics" "sooo" "bersih"
## [5] "stickles" "amazon.de" "amazon.es" "amazon.fr"
## [9] "amazon.it" "brother’s" "cuttlebug" "sooooo"
## [13] "kṛṣṇa" "promarkers" "nestabilities" "bazzill"
## [17] "random.org" "afrikaner" "guan" "unschoolers"
## [21] "malema" "yoochun" "sarawak" "afrikaners"
## [25] "ummm" "justrite" "pakatan" "sponged"
## [29] "hmmmm" "rakeback" "dcwv" "singaporeans"
## [33] "church’s" "papertrey" "sizzix" "blogland"
## [37] "digis" "datuk" "ctmh" "pretoria"
## [41] "qaradhawi" "versamark" "thе" "anwaar"
## [45] "malays" "mft" "namics" "hermel"
## [49] "ukip" "dimensionals"head(news_unique,50)## [1] "øthe" "freeholder" "metrohealth" "dewine" "ladue"
## [6] "sneiderman" "babeu" "drewniak" "bridgeton" "øbut"
## [11] "freeholders" "kyrillos" "psal" "tribune.com" "creve"
## [16] "lefthanded" "righthander" "schurick" "decroce" "hoynsie"
## [21] "m.p.g" "neiheiser" "petre" "yallop" "øa"
## [26] "amtrust" "bement" "kleem" "lioi" "modot"
## [31] "nj.com" "shuhandler" "theisen" "township’s" "vanecko"
## [36] "wisniewski" "beldini" "habra" "husted" "jerramiah"
## [41] "kucinich's" "mehlville" "pattonville" "renacci" "slay's"
## [46] "spagnuolo" "twinsburg" "appling" "ausby" "bbif"head(twt_unique,50)## [1] "tryna" "oomf" "2nite"
## [4] "finna" "bruh" "mke"
## [7] "nowplaying" "mahomies" "wats"
## [10] "tmrw" "bball" "followfriday"
## [13] "followback" "lmaoo" "mahomie"
## [16] "thevoice" "hmu" "teamfollowback"
## [19] "20thingsaboutme" "fridayreads" "dats"
## [22] "socialmedia" "idgaf" "some1"
## [25] "saysomethin" "ctfu" "lmaooo"
## [28] "forreal" "wbu" "yhu"
## [31] "rva" "shawty" "bcuz"
## [34] "thts" "thatawkwardmoment" "lmfaoo"
## [37] "directioners" "iloveyou" "wld"
## [40] "cuse" "hungergames" "nat'l"
## [43] "bouta" "atcha" "directioner"
## [46] "3wordsforyou" "pgh" "str8"
## [49] "sportingkc" "viggle"head(common_words,50)## [1] "the" "and" "to" "a" "of" "i" "in" "that" "is"
## [10] "it" "for" "you" "with" "was" "on" "my" "this" "as"
## [19] "have" "be" "but" "are" "we" "not" "at" "so" "from"
## [28] "all" "he" "or" "me" "they" "one" "by" "about" "will"
## [37] "up" "out" "his" "what" "an" "if" "had" "her" "when"
## [46] "just" "like" "your" "can" "there"tail(common_words,50)## [1] "wwe's" "www.redcross.org" "wxyt" "wyss"
## [5] "xalapa" "xan" "xavier's" "xchange"
## [9] "xdrive" "xers" "xiamen" "xlt"
## [13] "xm's" "yahoo's" "yair" "yannick"
## [17] "yarmulkes" "yasmina" "yauch's" "yellowthroat"
## [21] "yelm" "yelper" "yim" "yisroel"
## [25] "yoakum" "yola" "yonamine" "yost's"
## [29] "ypo" "yuhas" "yuni" "yunque"
## [33] "yurts" "zagat" "zagg" "zandi"
## [37] "zanny" "zant" "zante" "zayed"
## [41] "zellerbach" "zeni" "zeoli" "zico"
## [45] "zidlicky" "zion's" "zubrus" "zug"
## [49] "zygmunt" "éclair"