via GIPHY

Dickens wrote fourteen and a half novels. I will start from analyzing five of them - “A Tale of Two Cities,”, “Great Expectations”, “A Christmas Carol in Prose; Being a Ghost Story of Christmas”, “Oliver Twist”, “Hard Times”.

Project Gutenberg offers over 53,000 free books. I will download Dickens’ novels in UTF-8 encoded texts from there using gutenbergr package developed by David Robinson. Besides, I will be using the following packages for this project.

Download Dickens’ five novels by Project Gutenberg ID numbers.

The unnest_tokens package is used to split each row so that there is one token (word) in each row of the new data frame (tidy_dickens). Then remove stop words with an anti_join function.

## # A tibble: 19,634 × 2
##      word     n
##     <chr> <int>
## 1    time  1218
## 2    hand   918
## 3   night   835
## 4  looked   814
## 5    head   813
## 6  oliver   766
## 7    dear   751
## 8     joe   718
## 9    miss   702
## 10    sir   697
## # ... with 19,624 more rows

After remove the stop words, here is a list of words starts from the most frequent.

tidytext package contains several sentiment lexicons, I am using “bing” for the following tasks.

## # A tibble: 3,145 × 3
##      word sentiment     n
##     <chr>     <chr> <int>
## 1    miss  negative   702
## 2    poor  negative   350
## 3    dark  negative   299
## 4    hard  negative   223
## 5    dead  negative   218
## 6  strong  positive   203
## 7    love  positive   202
## 8    fell  negative   198
## 9   death  negative   194
## 10   cold  negative   192
## # ... with 3,135 more rows

Her I got the sentiment categories of Dickens’ words.

The word “miss” is the most frequent negative word here, but it is used to describe unmarried women in Dickens’ works. In particualr, Miss Havisham is a significant character in the Charles Dickens novel Great Expectations. Dickens describes her as looking like “the witch of the place”. In this case, probably “miss”" should be a negative word.

via GIPHY

World cloud is a good idea to identify trends and patterns that would otherwise be unclear or difficult to see in a tabular format.

And compare most frequent positive and negative words in word cloud.

Relationships between words

## # A tibble: 616,994 × 2
##    gutenberg_id          bigram
##           <int>           <chr>
## 1            46     a christmas
## 2            46 christmas carol
## 3            46        carol in
## 4            46        in prose
## 5            46     prose being
## 6            46         being a
## 7            46         a ghost
## 8            46     ghost story
## 9            46        story of
## 10           46    of christmas
## # ... with 616,984 more rows

Each token now represents a bigram (two words paired). if one of the words in the bigram is a stop word, this word will be removed. After filtering out stop words, what are the most frequent bigrams?

## Source: local data frame [41,584 x 3]
## Groups: word1 [10,206]
## 
##     word1      word2     n
##     <chr>      <chr> <int>
## 1    miss   havisham   236
## 2    miss      pross   144
## 3  madame    defarge   113
## 4    dear        boy    77
## 5      ha         ha    77
## 6  doctor    manette    75
## 7    miss havisham's    74
## 8  oliver      twist    55
## 9     sir    replied    54
## 10   wine       shop    52
## # ... with 41,574 more rows

Names are the most common paired words in Dickens’ novels.

## # A tibble: 51,379 × 2
##    gutenberg_id            bigram
## *         <int>             <chr>
## 1            46   christmas carol
## 2            46       ghost story
## 3            46   charles dickens
## 4            46   dickens preface
## 5            46 houses pleasantly
## 6            46   faithful friend
## 7            46     december 1843
## 8            46     1843 contents
## 9            46    contents stave
## 10           46    marley's ghost
## # ... with 51,369 more rows

Before visualize it, I have to make these bigram back to united.

Yes, the most frequent bigrams in Dickens’ works are names. I also notice some pairings of a common verb such as “wine shop” from “A Tale of Two Cities” and “oliver twist”.

At last, visualizing a network of bigrams of Dickens’ five novels.

## IGRAPH DN-- 61 37 -- 
## + attr: name (v/c), n (e/n)
## + edges (vertex names):
##  [1] miss    ->havisham   miss    ->pross      madame  ->defarge   
##  [4] dear    ->boy        ha      ->ha         doctor  ->manette   
##  [7] miss    ->havisham's oliver  ->twist      sir     ->replied   
## [10] wine    ->shop       charles ->darnay     replied ->oliver    
## [13] master  ->bates      miss    ->manette    saint   ->antoine   
## [16] james   ->harthouse  sir     ->returned   charley ->bates     
## [19] low     ->voice      stephen ->blackpool  sydney  ->carton    
## [22] god     ->bless      monsieur->defarge    public  ->house     
## + ... omitted several edges

Wow, that was so much fun! I don’t want to finish as yet. I want to look into one of these five novels - “A Tale of Two Cities”.

This time I will download the plain text file for “A Tale of Two Cities” only, leave out the Project Gutenberg header and footer information. Then concatenate these lines into paragraphs as following.

## [1] "I. The Period   It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light,"                                                                                                                                                                                                        
## [2] "it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way-- in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only."

Apply NRC sentiment dictionary to this novel.

Create a data frame combine the line number of the book with the sentiment score, then extract positive and negative scores for visualization.