The Project Gutenberg eBook Analysis

# CRAN 미러 설정
options(repos = c(CRAN = "https://cran.r-project.org"))

# 필요한 패키지 설치
install.packages("rmarkdown")

## Warning in download.file(url, destfile, method, mode = "wb", ...): downloaded
## length 0 != reported length 0

## Warning in download.file(url, destfile, method, mode = "wb", ...): URL
## 'https://cran.r-project.org/bin/windows/contrib/4.3/rmarkdown_2.27.zip':
## Timeout of 60 seconds was reached

## Error in download.file(url, destfile, method, mode = "wb", ...) : 
##   download from 'https://cran.r-project.org/bin/windows/contrib/4.3/rmarkdown_2.27.zip' failed

## Warning in download.packages(pkgs, destdir = tmpd, available = available, :
## 패키지 'rmarkdown'를 다운로드 하는데 실패했습니다

install.packages("dplyr")

## 패키지 'dplyr'를 성공적으로 압축해제하였고 MD5 sums 이 확인되었습니다
## 
## 다운로드된 바이너리 패키지들은 다음의 위치에 있습니다
##  C:\Users\USER\AppData\Local\Temp\Rtmp4GnwH3\downloaded_packages

install.packages("ggplot2")

## 패키지 'ggplot2'를 성공적으로 압축해제하였고 MD5 sums 이 확인되었습니다
## 
## 다운로드된 바이너리 패키지들은 다음의 위치에 있습니다
##  C:\Users\USER\AppData\Local\Temp\Rtmp4GnwH3\downloaded_packages

install.packages("tidytext")

## 패키지 'tidytext'를 성공적으로 압축해제하였고 MD5 sums 이 확인되었습니다
## 
## 다운로드된 바이너리 패키지들은 다음의 위치에 있습니다
##  C:\Users\USER\AppData\Local\Temp\Rtmp4GnwH3\downloaded_packages

install.packages("topicmodels")

## 패키지 'topicmodels'를 성공적으로 압축해제하였고 MD5 sums 이 확인되었습니다

## Warning: 패키지 'topicmodels'의 이전설치를 삭제할 수 없습니다

## Warning in file.copy(savedcopy, lib, recursive = TRUE): C:\Riot
## Games\R-4.3.3\library\00LOCK\topicmodels\libs\x64\topicmodels.dll를 C:\Riot
## Games\R-4.3.3\library\topicmodels\libs\x64\topicmodels.dll로 복사하는데 문제가
## 발생했습니다: Permission denied

## Warning: 'topicmodels'를 복구하였습니다

## 
## 다운로드된 바이너리 패키지들은 다음의 위치에 있습니다
##  C:\Users\USER\AppData\Local\Temp\Rtmp4GnwH3\downloaded_packages

install.packages("rvest")

## 패키지 'rvest'를 성공적으로 압축해제하였고 MD5 sums 이 확인되었습니다
## 
## 다운로드된 바이너리 패키지들은 다음의 위치에 있습니다
##  C:\Users\USER\AppData\Local\Temp\Rtmp4GnwH3\downloaded_packages

library(ggplot2)
library(topicmodels)
library(rvest)

## 
## 다음의 패키지를 부착합니다: 'rvest'

## The following object is masked from 'package:readr':
## 
##     guess_encoding

Executive summary

What is (are) your main question(s)? What is your story? What does the final graphic show? - Main question(s) The main questions for this analysis are centered around understanding the most prominent themes in the text data, determining whether the text is positive or negative, and identifying the emotional trends throughout the text. Additionally, the analysis seeks to uncover the overall tone of the text and gain insights into the author’s intent.

• Story This analysis focuses on exploring the key themes and emotional changes within a book’s text provided by Project Gutenberg. Using LDA (Latent Dirichlet Allocation) topic modeling, three major themes were identified from the text, and the important words within each theme were visualized. Sentiment analysis was also conducted to understand the emotional trends throughout the text, and word frequency analysis was performed to extract the most frequently used words. Through these methods, a deeper understanding of the structure and content of the text was achieved, providing insights into the narrative and the author’s intent.

• Final Graphic The final graphic visualizes the three major themes extracted from the text data. Each theme is represented in separate graphs and distinguished by different colors. This visualization highlights the key topics in the text and illustrates their relative importance and context within the overall narrative.

Data background

Explain where the data came from, what agency or company made it, how it is structured, what it shows, etc.

1. Where the Data Came From and Who Made It The data was obtained from Project Gutenberg, a well-known digital library that provides free digital copies of public domain books. Project Gutenberg is a volunteer-driven non-profit organization dedicated to the digitization and dissemination of literature. Since its inception in 1971, it has been a valuable resource for accessing a diverse collection of books without cost. The primary goal of Project Gutenberg is to encourage the free distribution and access to books in digital formats, making them accessible to anyone, anywhere.

2. Agency or Company -Organization: Project Gutenberg -Non-Profit Nature: Project Gutenberg operates as a non-profit organization, relying on the contributions of volunteers who help digitize and manage a vast number of books. Its mission is to provide free, unrestricted access to literature in the public domain.

3. Data Structure The data consists of the text of a book provided by Project Gutenberg, downloaded in HTML format. The extracted text is structured as follows:

• Text Data: The primary content of the book is available as text. This includes the main body of the book, typically divided into chapters or sections.
• Format: The text data is organized into a single-column data frame, with each row representing a line of the book. This format allows for easy manipulation and analysis of the text content.

4. What the Data Shows The data encompasses the entire text of the book and covers a wide array of literary elements and themes. Key components of the data include: Introduction and Chapters: The data includes the introduction and the full text of each chapter. Narrative and Dialogue: The text provides insights into the narrative progression and includes dialogues between characters. Descriptions and Expositions: It offers detailed descriptions and expositions of settings, characters, and events, contributing to a comprehensive understanding of the book’s literary style and content.

Overall, the data from Project Gutenberg serves as a rich resource for literary analysis, allowing for an exploration of the themes, structures, and stylistic elements present in public domain books.

Data loading, cleaning and preprocessing

Describe and show how you cleaned and reshaped the data

# HTML 페이지에서 텍스트 추출
url <- "https://www.gutenberg.org/cache/epub/73811/pg73811-images.html"
web_page <- read_html(url)
book_text <- web_page %>% html_nodes("p") %>% html_text()

#'book' 객체 정의하기
book <- book_text

# rmarkdown 패키지 로드
library(rmarkdown)

# book이 character 벡터인 경우 데이터프레임으로 변환
if (is.character(book)) {
  book <- data.frame(text = book, stringsAsFactors = FALSE)
}

# 데이터 청소
cleaned_text <- book %>%
  mutate(text = gsub("[^[:alnum:] ]", "", text)) %>%  # 특수문자 제거
  mutate(text = tolower(text))  # 모든 텍스트를 소문자로 변환

# 청소된 텍스트 데이터의 첫 몇 줄 확인
head(cleaned_text)

##                                                                                            text
## 1                                                     title american painting and its tradition
## 2         as represented by inness wyant martin homer la farge whistler chase alexander sargent
## 3                                                                        author john c van dyke
## 4                                                         release date june 11 2024 ebook 73811
## 5                                                                              language english
## 6                                     original publication new york charles scribners sons 1919

# 텍스트 토큰화
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 토큰화된 데이터 확인
head(tokens)

##        word
## 1     title
## 2  american
## 3  painting
## 4       and
## 5       its
## 6 tradition

#1. Text data analysis 1. 단어 빈도 분석

# 불용어 데이터 로드
data("stop_words")

# 불용어 제거 후 단어 빈도 계산
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

word_counts <- filtered_tokens %>%
  count(word, sort = TRUE)

# 상위 10개 단어 시각화
top_words <- word_counts %>% head(10)
ggplot(top_words, aes(x = reorder(word, n), y = n)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = "Top 10 Words in the Book (After Removing Stop Words)", x = "Words", y = "Frequency")

2. 감정 분석

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# 감정 사전 로드 및 텍스트에 감정 레이블 추가
sentiments <- get_sentiments("bing")

sentiment_analysis <- filtered_tokens %>%
  inner_join(sentiments, by = "word")

# 감정 분포 시각화
sentiment_counts <- sentiment_analysis %>%
  count(sentiment, sort = TRUE)

ggplot(sentiment_counts, aes(x = sentiment, y = n, fill = sentiment)) +
  geom_bar(stat = "identity") +
  labs(title = "Sentiment Analysis of the Book (After Removing Stop Words)", x = "Sentiment", y = "Count") +
  theme_minimal()

3. 핵심 주제 추출

# 불용어 제거 후 토큰화
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# DTM(문서-용어 행렬) 생성
dtm <- filtered_tokens %>%
  count(document = row_number(), word) %>%
  cast_dtm(document, word, n)

# LDA 모델 적용
lda_model <- LDA(dtm, k = 3, control = list(seed = 1234))

# 각 주제별 상위 단어 확인
topics <- tidy(lda_model, matrix = "beta")
top_terms <- topics %>%
  group_by(topic) %>%
  slice_max(beta, n = 10) %>%
  ungroup() %>%
  arrange(topic, -beta)

# 각 주제별 상위 단어 시각화
ggplot(top_terms, aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free_y") +
  coord_flip() +
  labs(title = "각 주제의 상위 단어",
       x = "단어", y = "베타 값") +
  theme_minimal()

#2. Individual analysis and figures

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# 단어 빈도 계산
word_counts <- filtered_tokens %>%
  count(word, sort = TRUE)

# 상위 10개 단어 시각화
top_words <- word_counts %>% head(10)
ggplot(top_words, aes(x = reorder(word, n), y = n)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = "Top 10 Words in the Book (After Removing Stop Words)", x = "Words", y = "Frequency")

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# 감정 사전 로드 및 텍스트에 감정 레이블 추가
sentiments <- get_sentiments("bing")

sentiment_analysis <- filtered_tokens %>%
  inner_join(sentiments, by = "word")

# 감정 분포 시각화
sentiment_counts <- sentiment_analysis %>%
  count(sentiment, sort = TRUE)

ggplot(sentiment_counts, aes(x = sentiment, y = n, fill = sentiment)) +
  geom_bar(stat = "identity") +
  labs(title = "Sentiment Analysis of the Book (After Removing Stop Words)", x = "Sentiment", y = "Count") +
  theme_minimal()

# 불용어 제거 후 텍스트 데이터 토큰화
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# DTM(문서-용어 행렬) 생성
dtm <- filtered_tokens %>%
  count(document = row_number(), word) %>%
  cast_dtm(document, word, n)

# LDA 모델 적용
lda_model <- LDA(dtm, k = 3, control = list(seed = 1234))

# 각 주제별 상위 단어 확인
topics <- tidy(lda_model, matrix = "beta")
top_terms <- topics %>%
  group_by(topic) %>%
  slice_max(beta, n = 10) %>%
  ungroup() %>%
  arrange(topic, -beta)

# 각 주제별 상위 단어 시각화
ggplot(top_terms, aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free_y") +
  coord_flip() +
  labs(title = "각 주제의 상위 단어", x = "단어", y = "베타 값") +
  theme_minimal()

#3. Anaysis and Figure 1

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# 단어 빈도 계산
word_counts <- filtered_tokens %>%
  count(word, sort = TRUE)

# 상위 10개 단어 시각화
top_words <- word_counts %>% head(10)
ggplot(top_words, aes(x = reorder(word, n), y = n)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = "Top 10 Words in the Book (After Removing Stop Words)", x = "Words", y = "Frequency") +
  theme_minimal()

# JPEG 파일로 그래프 저장
jpeg("Anaysis and Figure 1.jpg", width = 800, height = 600)
top_words <- word_counts %>% head(10)
ggplot(top_words, aes(x = reorder(word, n), y = n)) +
  geom_bar(stat = "identity") +
  coord_flip() +
  labs(title = "Top 10 Words in the Book (After Removing Stop Words)", x = "Words", y = "Frequency") +
  theme_minimal()
dev.off()

## png 
##   2

Describe and show how you created the first figure. Why did you choose this figure type? - Reasons for Choosing a Bar Chart Intuitive Comparison: Bar charts are convenient for understanding the frequency of each word at a glance. They allow for an easy and quick comparison of how frequently each word appears in the text, providing a clear and immediate visualization of word prevalence.

• Simplicity and Effectiveness: The method of arranging words based on their frequency and displaying them in a bar chart is simple yet highly effective in conveying information. By highlighting the top 10 words, the chart focuses on presenting only the most significant data, thus ensuring that important information is communicated effectively and efficiently.

#4. Anaysis and Figure 2

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# 감정 사전 로드
sentiments <- get_sentiments("bing")

# 문서의 인덱스 추가 (순서 정보 포함)
filtered_tokens <- filtered_tokens %>%
  mutate(document = ceiling(row_number() / 100))  # 예: 100단어마다 하나의 문서로 간주

# 감정 분석 수행
sentiment_analysis <- filtered_tokens %>%
  inner_join(sentiments, by = "word")

# 각 문서별 긍정/부정 감정 점수 계산
sentiment_timeline <- sentiment_analysis %>%
  group_by(document, sentiment) %>%
  summarize(n = n()) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment_score = positive - negative)

## `summarise()` has grouped output by 'document'. You can override using the
## `.groups` argument.

# 감정 변화를 시계열 그래프로 시각화
ggplot(sentiment_timeline, aes(x = document, y = sentiment_score, color = sentiment_score > 0)) +
  geom_line() +
  labs(title = "Sentiment Timeline of the Book",
       x = "Document Segment",
       y = "Sentiment Score") +
  scale_color_manual(values = c("TRUE" = "blue", "FALSE" = "red")) +
  theme_minimal()

# JPEG 파일로 그래프 저장
jpeg("Anaysis and Figure 2.jpg", width = 800, height = 600)
ggplot(sentiment_timeline, aes(x = document, y = sentiment_score, color = sentiment_score > 0)) +
  geom_line() +
  labs(title = "Sentiment Timeline of the Book",
       x = "Document Segment",
       y = "Sentiment Score") +
  scale_color_manual(values = c("TRUE" = "blue", "FALSE" = "red")) +
  theme_minimal()
dev.off()

## png 
##   2

The reasons for choosing a graph to visualize the sentiment changes over time are as follows:

1. Identifying Patterns Over Time This graph visually represents the flow of sentiment in the text, allowing us to identify changes in sentiment at specific points in time. As the text progresses, we can visually observe how sentiments evolve, such as whether there are significant changes in sentiment at the story’s climax or any emotional transitions towards the end. By identifying how sentiment shifts over time, we can detect periods of emotional fluctuation within the text.

2. Enhancing Visual Intuition The graph enhances visual intuition, making it easier to understand the data. It intuitively shows the rise and fall of emotions. Positive values represent positive sentiment, while negative values indicate negative sentiment, making it easy to discern the overall sentiment flow. By visualizing sentiment scores along the timeline, we can clearly identify critical points of emotional change in the text.

3. Analyzing Sentiment Changes to Understand the Text By demonstrating how the sentiment changes emotionally, the graph helps us better understand the structure and progression of the narrative. Identifying significant emotional turning points allows us to pinpoint key events or transitions within specific sections of the text.

#5. Anaysis and Figure 3

# 텍스트 데이터를 데이터프레임으로 변환
tokens <- cleaned_text %>%
  unnest_tokens(word, text)

# 불용어 제거
filtered_tokens <- tokens %>%
  anti_join(stop_words, by = "word")

# DTM(문서-용어 행렬) 생성
dtm <- filtered_tokens %>%
  count(document = row_number(), word) %>%
  cast_dtm(document, word, n)

# LDA 모델 적용
lda_model <- LDA(dtm, k = 3, control = list(seed = 1234))

# 각 주제별 상위 단어 확인
topics <- tidy(lda_model, matrix = "beta")
top_terms <- topics %>%
  group_by(topic) %>%
  slice_max(beta, n = 10) %>%
  ungroup() %>%
  arrange(topic, -beta)

# 각 주제별 상위 단어 시각화
ggplot(top_terms, aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free_y") +
  coord_flip() +
  labs(title = "각 주제의 상위 단어", x = "단어", y = "베타 값") +
  theme_minimal()

# JPEG 파일로 그래프 저장
jpeg("Anaysis and Figure 3.jpg", width = 800, height = 600)
ggplot(top_terms, aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free_y") +
  coord_flip() +
  labs(title = "각 주제의 상위 단어", x = "단어", y = "베타 값") +
  theme_minimal()
dev.off()

## png 
##   2

• The use of bar charts to visualize topics extracted through LDA (Latent Dirichlet Allocation) is chosen for several key reasons:

Bar charts allow us to visually emphasize the important words within each topic, making it easy to intuitively understand the relative significance and relationships of the data. The beta values for each word indicate how crucial that word is within a specific topic. This enables us to compare and understand the key terms that define each topic. By comparing the top words across different topics, we can clearly see which words are central to each topic and how they contribute to the overall composition of the topic.

#6. In showing the figures that you created, describe why you designed it the way you did. Why did you choose those colors, fonts, and other design elements? Does it convey truth?

1. Why you designed it the way you did. I chose a bar chart because it is effective for comparing the important words within each topic. It allows for a clear comparison of the significance of each word across different topics, making it easier to understand the data visually.

2. Colors I used different colors for each topic to help visually distinguish between them. The colors make it easier to identify and emphasize the key words for each topic, providing a clear visual separation.

3. Fonts I opted for the default ggplot2 theme with theme_minimal because it provides a clean and neat appearance, focusing attention on the data. It uses a sans-serif font which is easy to read, enhancing the overall clarity of the graph.

4. Other design elements: Titles and labels I chose clear and concise titles and axis labels. The title “Top Words in Each Topic” clearly describes what the graph represents, and the axis labels, “Words” and “Beta Value,” specify the content of each axis, making it easier for the viewer to understand the context of the data.

5. Facets I used facet_wrap for each topic to create small, separated graphs for each topic. This method allows for an individual comparison of the important words within each topic, making the differences between topics more apparent and enhancing the clarity of the analysis.