Write text and code here.

Executive summary

Q. What is (are) your main question(s)? What is your story? What does the final graphic show?

The main questions guiding this analysis are:

  1. What are the most frequently used words in “The Adventures of Sherlock Holmes”?
  2. What is the overall sentiment distribution in the text? Are there more positive or negative sentiments?
  3. Which specific words contribute most to positive and negative sentiments?
  4. How are the words in the text connected to each other? What are the common bigrams (word pairs)?

[What is your story?]

The story unfolds through an exploration of the text “The Adventures of Sherlock Holmes” by Arthur Conan Doyle. By utilizing text mining techniques, I aim to uncover patterns and insights hidden within the text. I start with a basic analysis of word frequency, proceed to a sentiment analysis to comprehend the emotional tone of the text, and then visualize the connections between words through a bigram network graph. This sequence enables me to create a comprehensive depiction of the text’s linguistic and emotional landscape.

[What does the final graphic show?]

The final graphic is a bigram network graph that illustrates the relationships between word pairs (bigrams) in “The Adventures of Sherlock Holmes”

Specifically, it shows:

  1. Nodes: Each node represents a word from the text.
  2. Edges: Each edge represents a bigram, which is a pair of words that appear consecutively in the text. The opacity (alpha) of the edges indicates the frequency of the bigram, darker edges representing more frequent bigrams.
  3. Node Labels: The labels on the nodes indicate the actual words.
  4. Graph Layout: The layout is visually structured to highlight clusters of words that frequently occur together, revealing common phrases and word associations in the text.

This graph provides a visual representation of how words are interconnected within the narrative, highlighting significant word pairings and potentially revealing thematic or stylistic elements of Conan Doyle’s writing. It showcases the most common word pairs that occur together more than ten times, offering insights into the structure and flow of the language used in the book.

This comprehensive analysis enables me to extract meaningful insights from the text, focusing on individual word usage and the relationships between words. This process enriches the understanding of Conan Doyle’s literary work.

Data background

Q. Explain where the data came from, what agency or company made it, how it is structured, what it shows, etc.

[Data Source]

The data used in this analysis comes from Project Gutenberg, a digital library offering over 70,000 free eBooks. Specifically, the text analyzed is “The Adventures of Sherlock Holmes” by Arthur Conan Doyle, available under the Project Gutenberg ID 48320.

[Agency or Company]

Project Gutenberg is a volunteer effort to digitize, archive, and distribute cultural works. Founded by Michael S. Hart in 1971, it is the oldest digital library. The texts are mostly in the public domain, and the collection includes works for which copyright has expired.

[Data Structure]

The dataset is structured as follows:

  1. Download: The text was downloaded using the gutenbergr package in R, which provides easy access to the full collection of Project Gutenberg texts.
  2. Data Format: The data is initially in a plain text format with the entire content of the book in a single column named text. Each row represents a line from the book.

[Data Cleaning and Preprocessing]

The text data undergoes several preprocessing steps to make it suitable for analysis:

  1. Tokenization: The text is split into individual words using the unnest_tokens function from the tidytext package.
  2. Stop Words Removal: Common stop words (e.g., “the”, “and”, “of”) that do not contribute meaningful information are removed.
  3. Bigram Creation: Bigrams (pairs of consecutive words) are extracted to analyze word pair relationships.

[What the Data Shows]

  1. Word Frequency: The dataset allows me to identify the most frequently used words in “The Adventures of Sherlock Holmes”. This can highlight key themes and topics within the text.
  2. Sentiment Analysis: By matching words to sentiment lexicons, I can gauge the overall emotional tone of the text and identify words that contribute most to positive or negative sentiments.
  3. Bigram Network Graph: The final output, a bigram network graph, visualizes how words are interconnected within the narrative. This graph helps to see common phrases and word associations, revealing patterns in the language and structure used by Arthur Conan Doyle.

Data loading, cleaning and preprocessing

library(readr)
#install.packages("gutenbergr")
library(gutenbergr)
sherlock <- gutenberg_download(48320)
## Determining mirror for Project Gutenberg from https://www.gutenberg.org/robot/harvest
## Using mirror http://aleph.gutenberg.org
sherlock
## # A tibble: 12,350 × 2
##    gutenberg_id text                                                  
##           <int> <chr>                                                 
##  1        48320 "ADVENTURES OF SHERLOCK HOLMES"                       
##  2        48320 ""                                                    
##  3        48320 ""                                                    
##  4        48320 ""                                                    
##  5        48320 ""                                                    
##  6        48320 "[Illustration:"                                      
##  7        48320 ""                                                    
##  8        48320 "  “THE GENTLEMAN IN THE PEW HANDED IT UP TO HER”"    
##  9        48320 "                                           [Page 238"
## 10        48320 "]"                                                   
## # ℹ 12,340 more rows
tidy_sherlock <- sherlock %>%
  unnest_tokens(word, text)
tidy_sherlock
## # A tibble: 107,466 × 2
##    gutenberg_id word        
##           <int> <chr>       
##  1        48320 adventures  
##  2        48320 of          
##  3        48320 sherlock    
##  4        48320 holmes      
##  5        48320 illustration
##  6        48320 the         
##  7        48320 gentleman   
##  8        48320 in          
##  9        48320 the         
## 10        48320 pew         
## # ℹ 107,456 more rows
tidy_sherlock <- tidy_sherlock %>%
  anti_join(stop_words)
## Joining with `by = join_by(word)`
tidy_sherlock
## # A tibble: 33,391 × 2
##    gutenberg_id word        
##           <int> <chr>       
##  1        48320 adventures  
##  2        48320 sherlock    
##  3        48320 holmes      
##  4        48320 illustration
##  5        48320 gentleman   
##  6        48320 pew         
##  7        48320 handed      
##  8        48320 page        
##  9        48320 238         
## 10        48320 adventures  
## # ℹ 33,381 more rows

Describe and show how you cleaned and reshaped the data

Text data analysis

tidy_sherlock %>%
  count(word, sort = T) %>%
  head(10)
## # A tibble: 10 × 2
##    word       n
##    <chr>  <int>
##  1 holmes   452
##  2 time     156
##  3 door     147
##  4 house    127
##  5 matter   125
##  6 hand     121
##  7 night    116
##  8 heard    113
##  9 day      111
## 10 found    110
bing <- get_sentiments("bing")
bing
## # A tibble: 6,786 × 2
##    word        sentiment
##    <chr>       <chr>    
##  1 2-faces     negative 
##  2 abnormal    negative 
##  3 abolish     negative 
##  4 abominable  negative 
##  5 abominably  negative 
##  6 abominate   negative 
##  7 abomination negative 
##  8 abort       negative 
##  9 aborted     negative 
## 10 aborts      negative 
## # ℹ 6,776 more rows
bing_sherlock_wordcount <- tidy_sherlock %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = T) %>%
  ungroup()
## Joining with `by = join_by(word)`
bing_sherlock_wordcount
## # A tibble: 1,411 × 3
##    word    sentiment     n
##    <chr>   <chr>     <int>
##  1 miss    negative     88
##  2 doubt   negative     64
##  3 strange negative     46
##  4 dark    negative     40
##  5 strong  positive     36
##  6 crime   negative     32
##  7 death   negative     31
##  8 hard    negative     29
##  9 cry     negative     28
## 10 fear    negative     28
## # ℹ 1,401 more rows
bing_sherlock_wordcount %>%
  filter(sentiment %in% c("positive", "negative")) %>%
  count(sentiment)
## # A tibble: 2 × 2
##   sentiment     n
##   <chr>     <int>
## 1 negative    882
## 2 positive    529
bing_sherlock_wordcount %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup()
## # A tibble: 21 × 3
##    word    sentiment     n
##    <chr>   <chr>     <int>
##  1 miss    negative     88
##  2 doubt   negative     64
##  3 strange negative     46
##  4 dark    negative     40
##  5 crime   negative     32
##  6 death   negative     31
##  7 hard    negative     29
##  8 cry     negative     28
##  9 fear    negative     28
## 10 lost    negative     28
## # ℹ 11 more rows
library(wordcloud)
## Loading required package: RColorBrewer
library(RColorBrewer)
tidy_sherlock %>%
  anti_join(stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 100))
## Joining with `by = join_by(word)`

tidy_sherlock %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("bing") %>% 
               filter(sentiment == "positive")) %>% 
  count(word) %>%
  with(wordcloud(word, n, max.words = 50))
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`
## Warning in wordcloud(word, n, max.words = 50): instantly could not be fit on
## page. It will not be plotted.
## Warning in wordcloud(word, n, max.words = 50): silent could not be fit on page.
## It will not be plotted.
## Warning in wordcloud(word, n, max.words = 50): quiet could not be fit on page.
## It will not be plotted.

tidy_sherlock %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("bing") %>% 
               filter(sentiment == "negative")) %>% 
  count(word) %>%
  with(wordcloud(word, n, max.words = 50))
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`

library(gutenbergr)
library(tidytext)
library(dplyr)
library(igraph)
## 
## Attaching package: 'igraph'
## 
## The following objects are masked from 'package:lubridate':
## 
##     %--%, union
## 
## The following objects are masked from 'package:dplyr':
## 
##     as_data_frame, groups, union
## 
## The following objects are masked from 'package:purrr':
## 
##     compose, simplify
## 
## The following object is masked from 'package:tidyr':
## 
##     crossing
## 
## The following object is masked from 'package:tibble':
## 
##     as_data_frame
## 
## The following objects are masked from 'package:stats':
## 
##     decompose, spectrum
## 
## The following object is masked from 'package:base':
## 
##     union
sherlock_bigrams <- sherlock %>%
  unnest_tokens(bigram, text, token = "ngrams", n = 2)

bigrams_separated <- sherlock_bigrams %>%
  separate(bigram, into = c("word1", "word2"), sep = " ")

data("stop_words")
bigrams_filtered <- bigrams_separated %>%
  filter(!word1 %in% stop_words$word,
         !word2 %in% stop_words$word)

bigram_counts <- bigrams_filtered %>%
  count(word1, word2, sort = TRUE)

bigram_graph <- bigram_counts %>%
  filter(n > 10) %>%
  graph_from_data_frame()
## Warning in graph_from_data_frame(.): In `d' `NA' elements were replaced with
## string "NA"

Individual analysis and figures

Anaysis and Figure 1

bing <- get_sentiments("bing")
bing
## # A tibble: 6,786 × 2
##    word        sentiment
##    <chr>       <chr>    
##  1 2-faces     negative 
##  2 abnormal    negative 
##  3 abolish     negative 
##  4 abominable  negative 
##  5 abominably  negative 
##  6 abominate   negative 
##  7 abomination negative 
##  8 abort       negative 
##  9 aborted     negative 
## 10 aborts      negative 
## # ℹ 6,776 more rows
bing_sherlock_wordcount <- tidy_sherlock %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = T) %>%
  ungroup()
## Joining with `by = join_by(word)`
bing_sherlock_wordcount
## # A tibble: 1,411 × 3
##    word    sentiment     n
##    <chr>   <chr>     <int>
##  1 miss    negative     88
##  2 doubt   negative     64
##  3 strange negative     46
##  4 dark    negative     40
##  5 strong  positive     36
##  6 crime   negative     32
##  7 death   negative     31
##  8 hard    negative     29
##  9 cry     negative     28
## 10 fear    negative     28
## # ℹ 1,401 more rows
bing_sherlock_wordcount %>%
  filter(sentiment %in% c("positive", "negative")) %>%
  count(sentiment)
## # A tibble: 2 × 2
##   sentiment     n
##   <chr>     <int>
## 1 negative    882
## 2 positive    529
bing_sherlock_wordcount %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup()
## # A tibble: 21 × 3
##    word    sentiment     n
##    <chr>   <chr>     <int>
##  1 miss    negative     88
##  2 doubt   negative     64
##  3 strange negative     46
##  4 dark    negative     40
##  5 crime   negative     32
##  6 death   negative     31
##  7 hard    negative     29
##  8 cry     negative     28
##  9 fear    negative     28
## 10 lost    negative     28
## # ℹ 11 more rows
bing_sherlock_wordcount %>%
  group_by(sentiment) %>%
  slice_max(n, n = 10) %>% 
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(n, word, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(x = "Contribution to sentiment",
       y = NULL)

Q. Describe and show how you created the first figure. Why did you choose this figure type?

Creating the First Figure: Sentiment Distribution Visualization

[Description of the Figure]

The first figure we created is a bar plot showing the contribution of words to positive and negative sentiments in “The Adventures of Sherlock Holmes.” This figure visualizes the top 10 words contributing to each sentiment, enabling to identify the words, most closely linked to positive or negative emotions in the text.

[Steps to Create the Figure]

  1. Download and Prepare the Text Data

I used the gutenbergr package to download “The Adventures of Sherlock Holmes” from Project Gutenberg. The text data was then tokenized into individual words using the tidytext package. Common stop words were removed to focus on meaningful words.

  1. Perform Sentiment Analysis

I used the Bing sentiment lexicon to categorize words as positive or negative. The inner_join function was used to match the words in our text with their sentiment scores. Then I counted the occurrences of each word by sentiment.

  1. Create the Bar Plot

I filtered the data to include only the top 10 positive and top 10 negative words. The ggplot2 package was used to create a bar plot, with words reordered by their contribution counts. The plot was faceted by sentiment to distinguish positive and negative contributions separately.

[Why This Figure Type Was Chosen]

  1. Clarity and Simplicity: Bar plots are straightforward and easy to interpret. They allow to clearly see the contribution of each word to the overall sentiment, making it easy to compare the top positive and negative words.
  2. Faceting by Sentiment: By segmenting the plot into positive and negative sentiments, I can directly compare the words influencing each sentiment category. This aids in comprehending the emotional equilibrium in the text.
  3. Focus on Key Words: By limiting the plot to the top 10 words for each sentiment, I focus on the most important contributors, avoiding clutter and making the plot more readable.
  4. Visual Appeal: The use of color to differentiate between positive and negative sentiments enhances the visual appeal and helps quickly identify the sentiment associated with each word.

Anaysis and Figure 2

options(repos = c(CRAN = "https://cran.r-project.org"))
install.packages("wordcloud")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
install.packages("RColorBrewer")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
library(wordcloud)
library(RColorBrewer)
library(ggplot2)

tidy_sherlock %>%
  anti_join(stop_words) %>%
  count(word) %>%
  with(wordcloud(word, n, max.words = 100))
## Joining with `by = join_by(word)`

tidy_sherlock %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("bing") %>% 
               filter(sentiment == "positive")) %>% 
  count(word) %>%
  with(wordcloud(word, n, max.words = 50))
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`
## Warning in wordcloud(word, n, max.words = 50): strong could not be fit on page.
## It will not be plotted.
## Warning in wordcloud(word, n, max.words = 50): bright could not be fit on page.
## It will not be plotted.

tidy_sherlock %>%
  anti_join(stop_words) %>%
  inner_join(get_sentiments("bing") %>% 
               filter(sentiment == "negative")) %>% 
  count(word) %>%
  with(wordcloud(word, n, max.words = 50))
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`

library(reshape2)
## 
## Attaching package: 'reshape2'
## 
## The following object is masked from 'package:tidyr':
## 
##     smiths
tidy_sherlock %>%
  anti_join(stop_words) %>%
  inner_join(bing) %>%
  dplyr::count(word, sentiment, sort = TRUE) %>%
  acast(word ~ sentiment, value.var = "n", fill = 0) %>%
  comparison.cloud(colors = c("blue", "red"),
                   max.words = 100)
## Joining with `by = join_by(word)`
## Joining with `by = join_by(word)`
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## astonishment could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## beauty could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## correct could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## immense could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## secure could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## success could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## gained could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## goodness could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## keenly could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## reward could not be fit on page. It will not be plotted.
## Warning in comparison.cloud(., colors = c("blue", "red"), max.words = 100):
## stable could not be fit on page. It will not be plotted.

Anaysis and Figure 3

# Install and load necessary packages
install.packages("gutenbergr")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
install.packages("tidytext")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
install.packages("dplyr")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
install.packages("igraph")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
install.packages("ggraph")
## 
## The downloaded binary packages are in
##  /var/folders/y9/689nq1j96z3d7c31nznf3v300000gn/T//RtmpI9PBma/downloaded_packages
library(gutenbergr)
library(tidytext)
library(dplyr)
library(igraph)
library(ggraph)


sherlock_bigrams <- sherlock %>%
  unnest_tokens(bigram, text, token = "ngrams", n = 2)

bigrams_separated <- sherlock_bigrams %>%
  separate(bigram, into = c("word1", "word2"), sep = " ")

data("stop_words")
bigrams_filtered <- bigrams_separated %>%
  filter(!word1 %in% stop_words$word,
         !word2 %in% stop_words$word)

bigram_counts <- bigrams_filtered %>%
  count(word1, word2, sort = TRUE)

bigram_graph <- bigram_counts %>%
  filter(n > 10) %>%
  graph_from_data_frame()
## Warning in graph_from_data_frame(.): In `d' `NA' elements were replaced with
## string "NA"
set.seed(1234)
ggraph(bigram_graph, layout = "fr") +
  geom_edge_link(aes(edge_alpha = n), show.legend = FALSE) +
  geom_node_point(color = "lightblue", size = 5) +
  geom_node_text(aes(label = name), vjust = 1, hjust = 1) +
  theme_void() +
  labs(title = "Bigram Network Graph for 'The Adventures of Sherlock Holmes'",
       subtitle = "Bigrams with more than 10 occurrences")

Q. In showing the figures that you created, describe why you designed it the way you did. Why did you choose those colors, fonts, and other design elements? Does it convey truth?

[[Comparative Wordclouds]]

[Description and Design Choices]

  1. Comparison Cloud Colors

Blue and Red: The choice of blue for positive sentiment and red for negative sentiment was deliberate. Blue is often associated with calm and positive feelings, while red can denote alertness or negativity. These colors are universally recognizable for their respective connotations.

Contrast: Using contrasting colors for different sentiments helps to immediately distinguish between positive and negative words, making the visualization intuitive.

[Fonts and Layout]

  1. Font Size: The size of each word in the wordcloud represents its frequency, making it easy to see which words are most common at a glance.

Legibility: A simple, sans-serif font was chosen for readability. Wordclouds can sometimes be cluttered, so a clear font ensures that words are easy to read.

  1. Conveying Truth

Accuracy in Representation: The wordcloud accurately represents the frequency of words by their size, ensuring an honest portrayal of word significance in each sentiment category.

Visual Balance: The balance in word sizes within each cloud prevents the visualization from being misleading, focusing on the most relevant words without overwhelming the viewer with less significant data.

[[Bigram Network Graph]]

[Description and Design Choices]

  1. Node and Edge Colors

Light blue Nodes: The nodes representing words were colored light blue to make them distinct against the white background, enhancing visibility.

Edge Alpha: The edges connecting the nodes were given varying levels of transparency (edge_alpha), reflecting the frequency of the bigrams. This highlights the more significant connections while still showing the less frequent ones.

  1. Font and Labels

Node Labels: Node labels were added to ensure that each word is easily identifiable, aiding in understanding the relationships depicted in the graph.

Font Size: The font size for the node labels was chosen to ensure readability without making the graph appear cluttered.

  1. Layout and Structure

Fruchterman-Reingold Layout: This layout algorithm was chosen for its effectiveness in presenting complex networks in an aesthetically pleasing way. It ensures that nodes evenly spaced to minimize edge crossings and overlapping.

Theme Void: Using a minimalistic theme (theme_void()) removes unnecessary background elements, focusing attention on the network structure.

  1. Conveying Truth

Transparency and Size: By using transparency and varying node sizes, the graph accurately conveys the relative importance of different bigrams, providing a truthful representation of the text’s structure.

Balanced Visualization: The chosen design elements ensure that the visualization is both informative and visually appealing, without introducing bias or misinterpretation.

You can also include images like this: