=Write text and code here.

Executive summary

What is (are) your main question(s)? What is your story? What does the final graphic show? The topic originated from something one professor said during a lecture I took this semester. He said that the lyrics of popular music in the past were lyrical and beautiful, but since the 2010s, they have rapidly become repetitive and sensational, making it difficult to understand the content. Also, there is a saying that “music is a mirror that reflects society.” Through lyrics of songs, people passed on knowledge from generation to generation. Therefore, I wanted to determine how society has changed over time by analyzing the lyrics of popular music. After searching several papers, I developed the following research hypothesis: 1. Compared to the past, the lyrics of modern popular music will use informative words more often. 2. Compared to the past, the lyrics of modern popular music will be negative.

However, this study has the following limitations. First of all, the sample is small. For research, I downloaded a cvs file from the Internet that summarized the basic information and lyrics of songs nominated for the Billboard Hot 100 from 1959 to 2020. However, there are limitations in using all the data, so for simplicity, only the top 30 songs on the charts in 1960, 1970, 2010, and 2020 were used. Therefore, it is difficult to generalize the research results.

Data background

Explain where the data came from, what agency or company made it, how it is structured, what it shows, etc. The data was created and published by a blogger. He stated that he collected the data using a web scraping technique using Python, and appears to have extracted it from the Billboard official site. It would be good if it was a material made by a reliable agency or company, but it was difficult to find an appropriate one, and I also thought about using Python directly to do web scraping in R studio, but it would be difficult for beginners to do it because it was not learned in class. As I mentioned before, the data consists of basic information (title, artist, ranking, lyrics) of songs nominated on the Billboard Hot 100 chart from 1959 to 2020.

Data loading, cleaning and preprocessing

Describe and show how you cleaned and reshaped the data It was downloaded as an Excel file and imported in CVS format.

billboard_data <- read_csv("billboard_year_end_hot_1959_2020.csv")
## New names:
## • `` -> `...1`
## Warning: One or more parsing issues, call `problems()` on your data frame for details,
## e.g.:
##   dat <- vroom(...)
##   problems(dat)
## Rows: 6201 Columns: 8
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (5): song, song_url, artist, artist_url, lyrics
## dbl (3): ...1, rank, year
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
glimpse(billboard_data)
## Rows: 6,201
## Columns: 8
## $ ...1       <dbl> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 1…
## $ rank       <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, …
## $ song       <chr> "The Battle of New Orleans", "Mack the Knife", "Personality…
## $ song_url   <chr> "/wiki/The_Battle_of_New_Orleans", "/wiki/Mack_the_Knife", …
## $ artist     <chr> "Johnny Horton", "Bobby Darin", "Lloyd Price", "Frankie Ava…
## $ artist_url <chr> "/wiki/Johnny_Horton", "/wiki/Bobby_Darin", "/wiki/Lloyd_Pr…
## $ year       <dbl> 1959, 1959, 1959, 1959, 1959, 1959, 1959, 1959, 1959, 1959,…
## $ lyrics     <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…

Using filtered_data function like following,

filtered_data <- billboard_data %>%
  filter(year %in% c(1960, 1970, 2010, 2020) & rank <= 30)
filtered_data
## # A tibble: 120 × 8
##     ...1  rank song                      song_url artist artist_url  year lyrics
##    <dbl> <dbl> <chr>                     <chr>    <chr>  <chr>      <dbl> <chr> 
##  1     0     1 Theme from A Summer Place /wiki/T… Percy… /wiki/Per…  1960 "Ther…
##  2     1     2 He'll Have to Go          /wiki/H… Jim R… /wiki/Jim…  1960 "Put …
##  3     2     3 Cathy's Clown             /wiki/C… The E… /wiki/The…  1960 "Don'…
##  4     3     4 Running Bear              /wiki/R… Johnn… /wiki/Joh…  1960 "On t…
##  5     4     5 Teen Angel                /wiki/T… Mark … /wiki/Mar…  1960 "Teen…
##  6     5     6 I'm Sorry                 /wiki/I… Brend… /wiki/Bre…  1960 "I'm …
##  7     6     7 It's Now or Never         /wiki/I… Elvis… /wiki/Elv…  1960 "It's…
##  8     7     8 Handy Man                 /wiki/H… Jimmy… /wiki/Jim…  1960 "Hey …
##  9     8     9 Stuck on You              /wiki/S… Elvis… /wiki/Elv…  1960 "You …
## 10     9    10 The Twist                 /wiki/T… Chubb… /wiki/Chu…  1960 "Come…
## # ℹ 110 more rows

I extract the necessary information only, and filtered_data1 and filtered_data2 were created and used for detailed analysis.

Text data analysis

Then, a preprocessing process for text analysis was performed, including removal of stop words.

word_counts <- filtered_data %>%
  mutate(year = as.character(year)) %>%
  unnest_tokens(word, lyrics) %>%
  anti_join(stop_words) %>% 
  count(year, word, sort = TRUE)
## Joining with `by = join_by(word)`
top_words <- word_counts %>%
  group_by(year) %>%
  top_n(10, n) %>%
  arrange(year, desc(n))
top_words
## # A tibble: 57 × 3
## # Groups:   year [4]
##    year  word      n
##    <chr> <chr> <int>
##  1 1960  love     27
##  2 1960  oop      21
##  3 1960  baby     14
##  4 1960  ah       11
##  5 1960  heart     7
##  6 1960  sweet     7
##  7 1960  hear      6
##  8 1960  angel     5
##  9 1960  gonna     5
## 10 1960  hand      5
## # ℹ 47 more rows

Individual analysis and figures

Anaysis and Figure 1

Describe and show how you created the first figure. Why did you choose this figure type? Afterwards, the top 10 words for each year were visualized. I used ggplot and geom_bar to vizualize the data. The bar graph was chosen because it allows people to easily identify data patterns and intuitively check the ranking and freqnency of words.

ggplot(top_words, aes(x = reorder_within(word, n, year), y = n, fill = year)) +
  geom_bar(stat = "identity") +
  labs(title = "Top 10 Most Frequently Used Words in Lyrics by Year",
       x = "Word",
       y = "Frequency") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  facet_wrap(~ year, scales = "free") +
  coord_flip() +
  scale_x_reordered() + 
  guides(fill = "none")

By looking at the generated bar graph, I was able to identify the most used words by year. In the 1960s and 1970s, love commonly ranked first, and lyrical words such as sweet, baby, lady, and angel stood out. On the other hand, it was confirmed that expressions such as yeah, ah, and woo were used relatively frequently in the 2010s and 2020s. In addition, swear words such as bit, fu, and sh** were also used without hesitation.

For the next step, I decided to find out the frequency of positive and negative expressions in lyrics by year. First, I extracted the neccessary datas from the huge dataset.

filtered_data1 <- billboard_data %>%
  filter(year %in% c(1960, 1970) & rank <= 30)
word_counts <- filtered_data1 %>%
  unnest_tokens(word, lyrics) %>%
  anti_join(stop_words)
## Joining with `by = join_by(word)`

I installed “textdata” packages to use “affin”

library(textdata)

I entered the code to load the affin lexion and calculate the emotion score.

afinn <- get_sentiments("afinn")
sentiment_words <- word_counts %>%
  inner_join(afinn, by = "word")

positive_words <- sentiment_words %>%
  filter(value > 0) %>%
  count(word, sort = TRUE)
negative_words <- sentiment_words %>%
  filter(value < 0) %>%
  count(word, sort = TRUE)
positive_words
## # A tibble: 40 × 2
##    word       n
##    <chr>  <int>
##  1 love      44
##  2 yeah       8
##  3 sweet      7
##  4 kiss       5
##  5 matter     5
##  6 smile      5
##  7 care       4
##  8 lovely     4
##  9 pretty     4
## 10 dreams     3
## # ℹ 30 more rows
negative_words
## # A tibble: 42 × 2
##    word       n
##    <chr>  <int>
##  1 die        7
##  2 lonely     7
##  3 lost       6
##  4 miss       6
##  5 mess       5
##  6 war        5
##  7 afraid     3
##  8 broken     3
##  9 cry        3
## 10 fool       3
## # ℹ 32 more rows

Anaysis and Figure 2

Then, I visualized the data. I generated a word cloud, and beyond trying various methods, I chose to see at a glance how negative and positive expressions differ by year. For this analyzation, the years 1960 and 1970 were grouped into one group, and the years 2010 and 2020 were grouped into one group. This is because I thought that since the time period was similar, there would be a lot in common.

par(mfrow = c(1, 2), mar = c(4, 4, 2, 2)) 
wordcloud(words = positive_words$word, freq = positive_words$n,
          scale = c(3, 1), min.freq = 1, colors = brewer.pal(4, "Blues"),
          random.order = FALSE)
title(main = "Positive Words - 1960 and 1970")

wordcloud(words = negative_words$word, freq = negative_words$n,
          scale = c(3, 1), min.freq = 1, colors = brewer.pal(4, "Reds"),
          random.order = FALSE)
title(main = "Negative Words - 1960 and 1970")

Same steps were repeated for 2010 and 2020 data.

filtered_data2 <- billboard_data %>%
  filter(year %in% c(2010, 2020) & rank <= 30)
word_counts <- filtered_data2 %>%
  unnest_tokens(word, lyrics) %>%
  anti_join(stop_words)
## Joining with `by = join_by(word)`
sentiment_words <- word_counts %>%
  inner_join(afinn, by = "word")
positive_words <- sentiment_words %>%
  filter(value > 0) %>%
  count(word, sort = TRUE)

negative_words <- sentiment_words %>%
  filter(value < 0) %>%
  count(word, sort = TRUE)

positive_words
## # A tibble: 42 × 2
##    word          n
##    <chr>     <int>
##  1 yeah         32
##  2 love         21
##  3 god           4
##  4 hope          4
##  5 woo           4
##  6 dream         3
##  7 romance       3
##  8 smile         3
##  9 beautiful     2
## 10 easy          2
## # ℹ 32 more rows
negative_words
## # A tibble: 57 × 2
##    word       n
##    <chr>  <int>
##  1 bitch     10
##  2 bad        8
##  3 cried      5
##  4 hate       5
##  5 leave      5
##  6 fuck       4
##  7 niggas     4
##  8 shit       4
##  9 ass        3
## 10 fire       3
## # ℹ 47 more rows
par(mfrow = c(1, 2), mar = c(4, 4, 2, 2)) 
wordcloud(words = positive_words$word, freq = positive_words$n,
          scale = c(3, 1), min.freq = 1, colors = brewer.pal(4, "Blues"),
          random.order = FALSE)
title(main = "Positive Words - 2010 and 2020")

wordcloud(words = negative_words$word, freq = negative_words$n,
          scale = c(3, 1), min.freq = 1, colors = brewer.pal(4, "Reds"),
          random.order = FALSE)
title(main = "Negative Words - 2010 and 2020")

Using affin lexion, expressions frequently used in lyrics by year were divided into two parts, positive and negative, and visualized through word clouds. As a result, it was confirmed that more violent profanity was used in music in 2010 and 2020. I used color blue for negative words and red for positive data, because blue color commonly talking about the negatives, blue presents aloofness or unfriendliness. and red usually highlights courage, strength and excitement.

Anaysis and Figure 3

For my last figure, I decided to decided to analyze emotional changes over time. I was able to confirm the change in positive and negative emotions over time. I used the filtered_data set again for this, and similar steps were repeated.

years <- c(1960, 1970, 2010, 2020)
filtered_data <- billboard_data %>%
  filter(year %in% years & rank <= 30)
word_counts <- filtered_data %>%
  unnest_tokens(word, lyrics) %>%
  anti_join(stop_words) 
## Joining with `by = join_by(word)`
afinn <- get_sentiments("afinn")
sentiment_words <- word_counts %>%
  inner_join(afinn, by = "word")

Using the following code, the ratio of positive and negative words by year was calculated.

sentiment_summary <- sentiment_words %>%
  group_by(year) %>%
  summarise(positive = sum(value > 0),
            negative = sum(value < 0),
            total = n()) %>%
  mutate(positive_ratio = positive / total,
         negative_ratio = negative / total)

And visualized it as a bar graph. It would have been nice to try a different way of expressing it, but I decided that a bar graph was best suited to show how the rate of use of positive and negative words changes over time.

ggplot(sentiment_summary, aes(x = as.factor(year), y = positive_ratio, fill = "Positive")) +
  geom_bar(stat = "identity", position = "dodge", width = 0.3) +
  geom_bar(aes(y = -negative_ratio, fill = "Negative"), stat = "identity", position = "dodge", width = 0.3) +
  labs(title = "Positive and Negative Sentiment Over Time",
       x = "Year",
       y = "Ratio",
       fill = "Sentiment") +
  scale_fill_manual(values = c("Positive" = "blue", "Negative" = "red")) +
  theme_minimal() +
  coord_flip()

The second hypothesis, “Compared to the past, the lyrics of modern popular music will be negative,” was confirmed to have some credibility. In the 1960s and 1970s, positive words were used overwhelmingly. However, in 2020, the proportion of negative words was much higher compared to positive words. On the other hand, the ratio of positive and negative lyrics in music in 2010 was somewhat similar to that in the 1960s.

Furthermore, I also wanted to identify music with social issues as the theme. Music has been a powerful medium for expressing social and political views throughout history. In fact, until the 1970s, many protest songs such as “Blowin’ in the Wind” and “A Hard Rain’s A-Gonna Fall,” were released. However, I could not find a keyword dataset related to social issues to analyze this. I tried to construct my own keywords by referring to various data and dictionaries.

filtered_data <- billboard_data %>%
  filter(year %in% years & rank <= 30) %>%
  select(year, lyrics)

social_issue_words <- c("protest", "equality", "freedom", "environment", "war", "ealth", "poverty", "education", "justice, equalit","human rights", "discrimination","community")

word_data <- filtered_data %>%
  unnest_tokens(word, lyrics) %>%
  filter(word %in% social_issue_words) %>%
  count(year, word) %>%
  group_by(year) %>%
  summarise(total_mentions = sum(n))

word_data
## # A tibble: 3 × 2
##    year total_mentions
##   <dbl>          <int>
## 1  1960              1
## 2  1970              4
## 3  2010              1

Nevertheless, I could not come up with any meaningful results. If given the opportunity, I would like to use a larger sample to analyze how music lyrics have changed in the way they reflect social reality or convey messages.

Through this study, it was confirmed that the lyrics of modern popular music use informative words more often than in the past. Although I coulnd’t come up with meaningful conclusions proving that the second research hypothesis, “The lyrics of modern popular music will be negative compared to the past,” is true, my last graph confirms that music released recently (2020) mainly contains negative emotions. Recently, I read an article saying that the number of ‘digital cocoon’ people is increasing. Due to rapid social change and increased crime, people tend to pursue a stable life. I would like to find a correlation to see if this uncertain society is the reason why the younger generation is more enthusiastic about music centered on negative emotions. As previously mentioned in the introduction, music expresses the contemporary society. There is a suspicion that the increasing negative tendencies indicate that the current generation is in crisis.