lyrics analysis

library(geniusr)                         # This package gets lyrics
library(tidyverse)
library(tidytext)
library(wordcloud2)

Using Genius to get lyrics

Genius.com is a site that has music lyrics. It is used by music services like Apple Music and Spotify. The package geniusr is an R interface to the Genius site, and brings lyrics into R. Let’s work through the geniusr API, starting with signing up for access:

1. Go here and create a login and app https://genius.com/api-clients/new
2. Go here and generate a client acces token https://genius.com/api-clients
3. Call genius_token() and then enter the client access token in the console when prompted

genius_token()

[1] "a4tjUC6-o2Q2woojDmmId0UtiQNH6kLYA-S6yNW2SHNIcyxLZWkxvMzXp5-3yJxY"

Once you get that working, let’s work through how to get information from Genius.

1. get album id. search for song with unique name on the album. find the song_id that matches the correct correct song

search_song("like a rolling stone")

2. Put the song_id number into the parentheses below to get more info about it.

get_song_meta(54784)

3. One of the columns above is album_id. Enter that number into the parentheses below to get all of the other tracks on the album. this one we have to save as a file.

highway_tracks <- scrape_tracklist(13573)

argument is not an atomic vector; coercing

highway_tracks

4. get the lyrics. map_df() repeatedly applies, or maps, one function (scrape_lyrics_url) to each line of highway_tracks$song_lyrics_url, and puts the results into highway_lyrics.

highway_lyrics <- map_df(highway_tracks$song_lyrics_url, scrape_lyrics_url)
highway_lyrics

NA

Preparing lyrics for analysis

Now we need to use unnest_tokens() to separate the words one to a row, make them lowercase, and remove the punctuation. Put that into a new data frame called highway_words. Note that the column with the lyrics is called ‘line.’

highway_words <- highway_lyrics %>%
  unnest_tokens(word, line) %>% 
  select(song_name, word)

highway_words

Word counts and word cloud

To get a sense of the common words on the album, pipe highway_words into count(word, sort = T)

highway_words %>% 
  count( word, sort = T)

But that has a lot of simple and unimportant words at the top of the list. Let’s get rid of stop words first. This uses anti_join() to remove the stopwords from our words:

highway_words %>% 
  anti_join(get_stopwords()) %>% 
  count(word, sort = T)

Joining, by = "word"

Create a word cloud by copy-pasting the code above and adding the following:

1. Add the line top_n(200) to limit the number of words in the cloud to 200
2. add the line wordcloud2(size = .5)

highway_words %>% 
  anti_join(get_stopwords()) %>% 
  count(word, sort = T) %>% 
  top_n(200) %>% 
  wordcloud2(size = .5)

Joining, by = "word"
Selecting by n

Sentiment analysis

Sentiment is another word for opinion or emotion. Sentiment analysis is the analysis of text for the opinions or emotions it contains. It is often used in marketing research to see customers’ opinions in the reviews they leave on sites like Amazon.

Sentiment analysis uses dictionaries that contain words related to sentiments and then compares the dictionary to the text being analyzed. Let’s look at one of the sentiment dictionaries, called bing after one of the researchers who developed it.

bing <- get_sentiments("bing")
bing

We need to join the sentiment dictionary with our lyrics. inner_join() will retain all of the words that are present in both.

highway_words %>% 
  inner_join(bing) %>% 
  count(word, sentiment, sort = TRUE)

Joining, by = "word"

Take the above code and pipe it into the following to create a graph of the table above.

highway_words %>% 
  inner_join(bing) %>% 
  count(word, sentiment, sort = TRUE) %>% 
  group_by(sentiment) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(vars(sentiment), scales = "free") +
  labs(y = "Bob Dylan's Highway 61 album: Words that contribute the most to each sentiment",
       x = NULL) +
  scale_fill_viridis_d() +
  coord_flip() +
  theme_minimal()

Joining, by = "word"
Selecting by n

word clouds with each sentiment:

highway_words %>% 
  inner_join(bing) %>% 
  count(word, sentiment, sort = TRUE) %>% 
  filter(sentiment == "positive") %>%
  select(word, n) %>% 
  wordcloud2()

Joining, by = "word"

highway_words %>% 
  inner_join(bing) %>% 
  count(word, sentiment, sort = TRUE) %>% 
  filter(sentiment == "negative") %>%
  select(word, n) %>% 
  wordcloud2()

Joining, by = "word"

NA

Other sentiment dictionaries

Another sentiment dictionary is nrc, which stands for National Research Council, a government agency in Canada similar to the National Science Foundation in the US.

nrc <- get_sentiments("nrc")
nrc

Notice that this dictionary has several different sentiments, not just positive and negative. You can view all the different sentiments by piping nrc into distinct(sentiment). Do that below:

nrc %>% 
  distinct(sentiment)

Conduct the same analysis as above - the tables and graphs of words that contribute to each sentiment, but do it for this new lexicon. Hint: When you create the graph, try a smaller value in top_n(), like 3 or 5.

highway_words %>% 
  inner_join(nrc) %>% 
  count(word, sentiment, sort = TRUE) %>% 
  group_by(sentiment) %>%
  top_n(3) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(vars(sentiment), scales = "free") +
  labs(y = "Bob Dylan's Highway 61 album: Words that contribute the most to each sentiment",
       x = NULL) +
  scale_fill_viridis_d() +
  coord_flip() +
  theme_minimal()

Joining, by = "word"
Selecting by n

Bigrams

So far we have been examining individual words, but it is often useful to examine words that appear together. To prepare the data we use unnest_tokens() again, but now we add token = “ngrams”, n = 2 to look at word pairs or bigrams. Here is how to do that:

highway_lyrics %>%
  unnest_tokens(bigram, line, token = "ngrams", n = 2) %>% 
  select(bigram)

NA

Notice that the 2nd word of the first bigram is the 1st word of the second bigram. In this way, every possible two-word combination is retained.

Once you are satisfied that it is working, make a dataframe of this new set of words called highway_bigrams. Here’s a new way to do that. We have used this as an assignment operator: <- but you can also use it the other direction -> .

Copy-paste the above code chunk, but then put the following line at the end of the last line: -> highway_bigrams

Then view the new data frame by typing in its name.

highway_lyrics %>%
  unnest_tokens(bigram, line, token = "ngrams", n = 2) %>% 
  select(bigram) -> highway_bigrams

To count the most common bigrams, pipe highway_bigrams into count(bigram, sort = T) below:

highway_bigrams %>% 
  count(bigram, sort = T)

This is the same problem as before, that many of the bigrams contain common and uninteresting words. It’s a little more complicated to remove stop words from bigrams, but here’s how to do it. The following separates the bigrams, checks each word against the list of stop words, filters them out if they are, and then puts them back together with unite().

highway_bigrams %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>% 
  filter(!word1 %in% stop_words$word) %>%
  filter(!word2 %in% stop_words$word) %>% 
  unite(bigram, word1, word2, sep = " ")

NA

Now pipe the above code into count(bigram, sort = T) again to see the difference in the top bigrams when stopwords are removed:

highway_bigrams %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>% 
  filter(!word1 %in% stop_words$word) %>%
  filter(!word2 %in% stop_words$word) %>% 
  unite(bigram, word1, word2, sep = " ") %>% 
  count(bigram, sort = T)

NA

Now take the above chunk, including the count() line, and pipe it into the following to create a word cloud of all the bigrams that occur more than once in the album:

highway_bigrams %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>% 
  filter(!word1 %in% stop_words$word) %>%
  filter(!word2 %in% stop_words$word) %>% 
  unite(bigram, word1, word2, sep = " ") %>% 
  count(bigram, sort = T) %>% 
  filter(n > 1) %>% 
  wordcloud2(size = .5)

Identifying specific word pairs

Using bigrams, we can also see what words typically follow a given word. For example, which words follow I and you? This can tell us something about the psychology of how someone sees themselves vs. other people. Heres how to do that:

first_word <- c("i", "you")                                  # these need to be lowercase

highway_bigrams %>% 
  count(bigram, sort = T) %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>%       # separate the two words
  filter(word1 %in% first_word) %>%                          # find first words from our list
  count(word1, word2, wt = n, sort = TRUE) %>% 
  rename(total = nn)

NA

Take all of the above chunk that creates the I/you counts, put it on top of the following chunk (don’t forget the pipe to connect them), and run it to create a graph.

first_word <- c("i", "you")                                  # these need to be lowercase

highway_bigrams %>% 
  count(bigram, sort = T) %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>%       # separate the two words
  filter(word1 %in% first_word) %>%                          # find first words from our list
  count(word1, word2, wt = n, sort = TRUE) %>% 
  rename(total = nn) %>% 


  mutate(word2 = factor(word2, levels = rev(unique(word2)))) %>%     # put the words in order
  group_by(word1) %>% 
  top_n(5) %>% 
  ggplot(aes(word2, total, fill = word1)) +                          #
  scale_fill_viridis_d() +                                           # set the color palette
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = NULL, title = "Word following:") +
  facet_wrap(~word1, scales = "free") +
  coord_flip() +
  theme_minimal()

Selecting by total

Copy and paste the chunk above, but replace (“i”, “you”) with (“he”, “she”) to see what Bob has men and women doing in his songs.

first_word <- c("he", "she")                                  # these need to be lowercase

highway_bigrams %>% 
  count(bigram, sort = T) %>% 
  separate(bigram, c("word1", "word2"), sep = " ") %>%       # separate the two words
  filter(word1 %in% first_word) %>%                          # find first words from our list
  count(word1, word2, wt = n, sort = TRUE) %>% 
  rename(total = nn) %>% 


  mutate(word2 = factor(word2, levels = rev(unique(word2)))) %>%     # put the words in order
  group_by(word1) %>% 
  top_n(3) %>% 
  ggplot(aes(word2, total, fill = word1)) +                          #
  scale_fill_viridis_d() +                                           # set the color palette
  geom_col(show.legend = FALSE) +
  labs(x = NULL, y = NULL, title = "Word following:") +
  facet_wrap(~word1, scales = "free") +
  coord_flip() +
  theme_minimal()

Selecting by total

One final example: You can set a higher ngram. Let’s look at 4-word patterns:

highway_lyrics %>%
  unnest_tokens(words, line, token = "ngrams", n = 4) %>%        # get ngrams of 4
  count(words, sort = T)

NA

How often does the line “like a rolling stone” appear on the album? Take the above chunk and copy-paste it below, and add the following line: filter(words == “like a rolling stone”)

highway_lyrics %>%
  unnest_tokens(words, line, token = "ngrams", n = 4) %>%        # get ngrams of 4
  count(words, sort = T) %>% 
  filter(words =="like a rolling stone")

NA