Main goal
This project aims to show which music genres are “particularly liked” by a couple of cities in the UK, and compare them. EveryNoise.com is a website that visualize Spotify’s streaming data on multiple dimensions. Every Place At Once looks at individual cities, for instance.
How does the data looks like online
The end result in R
The net end result for ths project (after playing with it in Illustrator)
So let’s get going !
Loading the required packages…
library(httr)
library(httpuv)
library(tidyverse)
library(showtext)
library(rlist)
library(rvest)
library(RColorBrewer)
library(colorRamps)
library(grDevices)
library(scales)For an easier understanding, I’ll start here with one city only. Birmingham sounds like a good one.
Grabbing the html page from EveryNoise. This stores the html page’s code in R.
BirminghamReaded <- read_html("http://everynoise.com/everyplace.cgi?root=Birmingham%20GB&scope=all")Grabbing the genres specifically, CSS information was found by inspecting the items on the webpage (which can be done, in your web browser by right-clicking the text or data you want to grab and choose “Inspect” and then “Copy Selector” or something similar while looking at it selected in the page source code).
BirminghamList <- html_text(html_node(BirminghamReaded, css = "div.note > div"))
print(BirminghamList) ### How our data looks like## [1] "\nuk hip hop\ngrime\nuk drill\nbirmingham grime\nuk garage\nmanchester hip hop\nuk alternative hip hop\nbirmingham indie\ndesi pop\nhouse\npunjabi pop\nbassline\nbhangra\ndance pop\ndeep groove house\nindian folk\npunjabi\nspeed garage\nuk funky\nbass house\nenglish indie rock\ninstrumental grime\nliquid funk\nlovers rock\nroots reggae\ntropical house\nuk dance\n"A wild set of “/n” appears! So we’ll get rid of them, but first we’ll actually use them to separate our elements. In the strsplit function below, the argument “/n” is the separator - which characters are to be used to separate and distinguish our actual data.
BirminghamGenresL <- strsplit(BirminghamList, "\n")
print(BirminghamGenresL)## [[1]]
## [1] "" "uk hip hop"
## [3] "grime" "uk drill"
## [5] "birmingham grime" "uk garage"
## [7] "manchester hip hop" "uk alternative hip hop"
## [9] "birmingham indie" "desi pop"
## [11] "house" "punjabi pop"
## [13] "bassline" "bhangra"
## [15] "dance pop" "deep groove house"
## [17] "indian folk" "punjabi"
## [19] "speed garage" "uk funky"
## [21] "bass house" "english indie rock"
## [23] "instrumental grime" "liquid funk"
## [25] "lovers rock" "roots reggae"
## [27] "tropical house" "uk dance"Now it’s a list, but it will probably be more useful as a vector
BirminghamGenresV <- unlist(BirminghamGenresL)
print(BirminghamGenresV)## [1] "" "uk hip hop"
## [3] "grime" "uk drill"
## [5] "birmingham grime" "uk garage"
## [7] "manchester hip hop" "uk alternative hip hop"
## [9] "birmingham indie" "desi pop"
## [11] "house" "punjabi pop"
## [13] "bassline" "bhangra"
## [15] "dance pop" "deep groove house"
## [17] "indian folk" "punjabi"
## [19] "speed garage" "uk funky"
## [21] "bass house" "english indie rock"
## [23] "instrumental grime" "liquid funk"
## [25] "lovers rock" "roots reggae"
## [27] "tropical house" "uk dance"Because our first few characters were “/n”, our first value is now an empty string, we can see it easily in the console so now we can just remove it by telling R to remove the first value in BirminghamGenresV
BirminghamGenresV <- BirminghamGenresV[-1]
print(BirminghamGenresV)## [1] "uk hip hop" "grime"
## [3] "uk drill" "birmingham grime"
## [5] "uk garage" "manchester hip hop"
## [7] "uk alternative hip hop" "birmingham indie"
## [9] "desi pop" "house"
## [11] "punjabi pop" "bassline"
## [13] "bhangra" "dance pop"
## [15] "deep groove house" "indian folk"
## [17] "punjabi" "speed garage"
## [19] "uk funky" "bass house"
## [21] "english indie rock" "instrumental grime"
## [23] "liquid funk" "lovers rock"
## [25] "roots reggae" "tropical house"
## [27] "uk dance"It works! We got ourselves a vector of music genres that are especially popular in Birmingham. Now we’ll create a function to do this for all of our big or medium cities of interest.
In what follows, we put everything we did above in one single function that does all the job.
Note that to do this for every city of interest, we need the URL of every city’s unique page. In this case, it is easy, because the URL acts like a set of parameters. In our URL above with Birmingham, we can simply replace it with another city name like “London” or “Leeds”. The city name is the only difference between our URLs.
Cities <- c("London", "Birmingham", "Manchester", "Leeds", "Sheffield", "Glasgow", "Edinburgh", "Bristol", "Liverpool", "Bradford", "Cardiff", "Belfast")The function below will, for a given city, generate the URL, read and store the html page, grab the data it needs, clean it, convert it into a vector that makes sense. It also limits the number of values to twenty, because I don’t want more than that. Finally, it converts the values into factors (discrete).
ParticularLikesFUN <- function(x) {
City <- x # The function's main input is one single city, not all 12 of them
City_EN_URL <- str_c("http://everynoise.com/everyplace.cgi?root=", City, "%20GB&scope=all", sep = "", collapse = NULL)
City_Readed <- read_html(City_EN_URL)
City_Genres <- html_text(html_node(City_Readed, css = "div.note > div"))
City_GenreL <- strsplit(City_Genres, "\n")
City_GenreV <- unlist(City_GenreL)
City_GenreV <- City_GenreV[-1]
City_GenreV <- City_GenreV[1:20]
factor(City_GenreV)
print(City_GenreV)
}Now we haven’t done the scraping just yet, we just defined the function that will do it.
The function alone as designed can only do it to one single city, not all of them.
To do it on all of them, we’ll of course use the apply function family.
ParticularLikesAllCities <- lapply(Cities, ParticularLikesFUN)## [1] "uk hip hop" "grime"
## [3] "uk garage" "uk drill"
## [5] "speed garage" "dance pop"
## [7] "uk alternative hip hop" "uk funky"
## [9] "afro dancehall" "pop"
## [11] "r&b" "uk dancehall"
## [13] "urban contemporary" "bassline"
## [15] "hip pop" "house"
## [17] "neo soul" "new jack swing"
## [19] "australian pop" "bass house"
## [1] "uk hip hop" "grime"
## [3] "uk drill" "birmingham grime"
## [5] "uk garage" "manchester hip hop"
## [7] "uk alternative hip hop" "birmingham indie"
## [9] "desi pop" "house"
## [11] "punjabi pop" "bassline"
## [13] "bhangra" "dance pop"
## [15] "deep groove house" "indian folk"
## [17] "punjabi" "speed garage"
## [19] "uk funky" "bass house"
## [1] "uk hip hop" "grime" "indie rock"
## [4] "britpop" "english indie rock" "modern rock"
## [7] "rock" "uk drill" "madchester"
## [10] "garage rock" "dance pop" "manchester hip hop"
## [13] "australian pop" "disco house" "electronic"
## [16] "europop" "new wave" "bubblegum dance"
## [19] "hip house" "house"
## [1] "indie rock" "modern rock"
## [3] "uk hip hop" "grime"
## [5] "english indie rock" "rock"
## [7] "garage rock" "britpop"
## [9] "dance pop" "uk drill"
## [11] "europop" "house"
## [13] "disco house" "uk garage"
## [15] "british indie rock" "deep groove house"
## [17] "deep house" "new rave"
## [19] "tropical house" "uk alternative hip hop"
## [1] "indie rock" "modern rock" "rock"
## [4] "english indie rock" "garage rock" "britpop"
## [7] "grime" "uk hip hop" "uk garage"
## [10] "europop" "dance pop" "british indie rock"
## [13] "house" "bass house" "deep groove house"
## [16] "new rave" "uk drill" "dance-punk"
## [19] "new wave" "pop"
## [1] "hardcore techno" "scottish folk"
## [3] "scottish indie rock" "irish folk"
## [5] "traditional scottish folk" "bouncy house"
## [7] "celtic" "celtic rock"
## [9] "football" "glasgow indie"
## [11] "britpop" "eurodance"
## [13] "indie rock" "scottish rock"
## [15] "bubble trance" "happy hardcore"
## [17] "hip house" "english indie rock"
## [19] "rock" "bagpipe"
## [1] "britpop" "indie rock" "hardcore techno"
## [4] "rock" "bouncy house" "modern rock"
## [7] "new wave" "scottish rock" "dance rock"
## [10] "celtic rock" "new wave pop" "scottish indie"
## [13] "celtic" "chamber psych" "disco house"
## [16] "europop" "folk-pop" "garage rock"
## [19] "scottish indie rock" "chamber pop"
## [1] "uk hip hop" "grime"
## [3] "uk garage" "drum and bass"
## [5] "jump up" "liquid funk"
## [7] "jungle" "uk alternative hip hop"
## [9] "uk drill" "speed garage"
## [11] "bass house" "neurofunk"
## [13] "house" "bassline"
## [15] "dance pop" "dancehall"
## [17] "dub" "lovers rock"
## [19] "pop" "roots reggae"
## [1] "house" "disco house" "deep groove house"
## [4] "tropical house" "britpop" "deep house"
## [7] "dance rock" "new wave" "uk dance"
## [10] "new wave pop" "rock" "edm"
## [13] "grime" "uk hip hop" "vocal house"
## [16] "liverpool indie" "new romantic" "pop"
## [19] "bouncy house" "indie rock"
## [1] "uk hip hop" "grime"
## [3] "uk drill" "indie rock"
## [5] "britpop" "modern rock"
## [7] "rock" "uk garage"
## [9] "dance pop" "europop"
## [11] "deep groove house" "garage rock"
## [13] "house" "manchester hip hop"
## [15] "tropical house" "uk alternative hip hop"
## [17] "birmingham grime" "post-teen pop"
## [19] "australian pop" "bass house"
## [1] "grime" "uk hip hop" "uk garage"
## [4] "indie rock" "modern rock" "house"
## [7] "dance pop" "jump up" "pop"
## [10] "drum and bass" "liquid funk" "bass house"
## [13] "english indie rock" "rock" "uk drill"
## [16] "europop" "jungle" "speed garage"
## [19] "deep groove house" "post-teen pop"
## [1] "irish folk" "bouncy house"
## [3] "hardcore techno" "irish country"
## [5] "trance" "bubble trance"
## [7] "celtic" "uplifting trance"
## [9] "progressive house" "progressive trance"
## [11] "house" "disco house"
## [13] "deep house" "tech house"
## [15] "eurodance" "progressive uplifting trance"
## [17] "deep groove house" "german techno"
## [19] "europop" "float house"Now, we got all our music genres, and at first they look as if they were all part of the same vector, but no. The sum of them is a list because we used lapply, which generates lists, but our list items are vectors.
Each item within our list represents a city, in the same order. So let’s identify them as such.
names(ParticularLikesAllCities) <- Cities
print(ParticularLikesAllCities)## $London
## [1] "uk hip hop" "grime"
## [3] "uk garage" "uk drill"
## [5] "speed garage" "dance pop"
## [7] "uk alternative hip hop" "uk funky"
## [9] "afro dancehall" "pop"
## [11] "r&b" "uk dancehall"
## [13] "urban contemporary" "bassline"
## [15] "hip pop" "house"
## [17] "neo soul" "new jack swing"
## [19] "australian pop" "bass house"
##
## $Birmingham
## [1] "uk hip hop" "grime"
## [3] "uk drill" "birmingham grime"
## [5] "uk garage" "manchester hip hop"
## [7] "uk alternative hip hop" "birmingham indie"
## [9] "desi pop" "house"
## [11] "punjabi pop" "bassline"
## [13] "bhangra" "dance pop"
## [15] "deep groove house" "indian folk"
## [17] "punjabi" "speed garage"
## [19] "uk funky" "bass house"
##
## $Manchester
## [1] "uk hip hop" "grime" "indie rock"
## [4] "britpop" "english indie rock" "modern rock"
## [7] "rock" "uk drill" "madchester"
## [10] "garage rock" "dance pop" "manchester hip hop"
## [13] "australian pop" "disco house" "electronic"
## [16] "europop" "new wave" "bubblegum dance"
## [19] "hip house" "house"
##
## $Leeds
## [1] "indie rock" "modern rock"
## [3] "uk hip hop" "grime"
## [5] "english indie rock" "rock"
## [7] "garage rock" "britpop"
## [9] "dance pop" "uk drill"
## [11] "europop" "house"
## [13] "disco house" "uk garage"
## [15] "british indie rock" "deep groove house"
## [17] "deep house" "new rave"
## [19] "tropical house" "uk alternative hip hop"
##
## $Sheffield
## [1] "indie rock" "modern rock" "rock"
## [4] "english indie rock" "garage rock" "britpop"
## [7] "grime" "uk hip hop" "uk garage"
## [10] "europop" "dance pop" "british indie rock"
## [13] "house" "bass house" "deep groove house"
## [16] "new rave" "uk drill" "dance-punk"
## [19] "new wave" "pop"
##
## $Glasgow
## [1] "hardcore techno" "scottish folk"
## [3] "scottish indie rock" "irish folk"
## [5] "traditional scottish folk" "bouncy house"
## [7] "celtic" "celtic rock"
## [9] "football" "glasgow indie"
## [11] "britpop" "eurodance"
## [13] "indie rock" "scottish rock"
## [15] "bubble trance" "happy hardcore"
## [17] "hip house" "english indie rock"
## [19] "rock" "bagpipe"
##
## $Edinburgh
## [1] "britpop" "indie rock" "hardcore techno"
## [4] "rock" "bouncy house" "modern rock"
## [7] "new wave" "scottish rock" "dance rock"
## [10] "celtic rock" "new wave pop" "scottish indie"
## [13] "celtic" "chamber psych" "disco house"
## [16] "europop" "folk-pop" "garage rock"
## [19] "scottish indie rock" "chamber pop"
##
## $Bristol
## [1] "uk hip hop" "grime"
## [3] "uk garage" "drum and bass"
## [5] "jump up" "liquid funk"
## [7] "jungle" "uk alternative hip hop"
## [9] "uk drill" "speed garage"
## [11] "bass house" "neurofunk"
## [13] "house" "bassline"
## [15] "dance pop" "dancehall"
## [17] "dub" "lovers rock"
## [19] "pop" "roots reggae"
##
## $Liverpool
## [1] "house" "disco house" "deep groove house"
## [4] "tropical house" "britpop" "deep house"
## [7] "dance rock" "new wave" "uk dance"
## [10] "new wave pop" "rock" "edm"
## [13] "grime" "uk hip hop" "vocal house"
## [16] "liverpool indie" "new romantic" "pop"
## [19] "bouncy house" "indie rock"
##
## $Bradford
## [1] "uk hip hop" "grime"
## [3] "uk drill" "indie rock"
## [5] "britpop" "modern rock"
## [7] "rock" "uk garage"
## [9] "dance pop" "europop"
## [11] "deep groove house" "garage rock"
## [13] "house" "manchester hip hop"
## [15] "tropical house" "uk alternative hip hop"
## [17] "birmingham grime" "post-teen pop"
## [19] "australian pop" "bass house"
##
## $Cardiff
## [1] "grime" "uk hip hop" "uk garage"
## [4] "indie rock" "modern rock" "house"
## [7] "dance pop" "jump up" "pop"
## [10] "drum and bass" "liquid funk" "bass house"
## [13] "english indie rock" "rock" "uk drill"
## [16] "europop" "jungle" "speed garage"
## [19] "deep groove house" "post-teen pop"
##
## $Belfast
## [1] "irish folk" "bouncy house"
## [3] "hardcore techno" "irish country"
## [5] "trance" "bubble trance"
## [7] "celtic" "uplifting trance"
## [9] "progressive house" "progressive trance"
## [11] "house" "disco house"
## [13] "deep house" "tech house"
## [15] "eurodance" "progressive uplifting trance"
## [17] "deep groove house" "german techno"
## [19] "europop" "float house"CitiesLikes <- ParticularLikesAllCities # The variable was getting long to type every time so let's shorten its nameMight be easier to make it a data.frame from now on.
CitiesLikesDF <- as.data.frame(CitiesLikes)
print(CitiesLikesDF)## London Birmingham Manchester
## 1 uk hip hop uk hip hop uk hip hop
## 2 grime grime grime
## 3 uk garage uk drill indie rock
## 4 uk drill birmingham grime britpop
## 5 speed garage uk garage english indie rock
## 6 dance pop manchester hip hop modern rock
## 7 uk alternative hip hop uk alternative hip hop rock
## 8 uk funky birmingham indie uk drill
## 9 afro dancehall desi pop madchester
## 10 pop house garage rock
## 11 r&b punjabi pop dance pop
## 12 uk dancehall bassline manchester hip hop
## 13 urban contemporary bhangra australian pop
## 14 bassline dance pop disco house
## 15 hip pop deep groove house electronic
## 16 house indian folk europop
## 17 neo soul punjabi new wave
## 18 new jack swing speed garage bubblegum dance
## 19 australian pop uk funky hip house
## 20 bass house bass house house
## Leeds Sheffield Glasgow
## 1 indie rock indie rock hardcore techno
## 2 modern rock modern rock scottish folk
## 3 uk hip hop rock scottish indie rock
## 4 grime english indie rock irish folk
## 5 english indie rock garage rock traditional scottish folk
## 6 rock britpop bouncy house
## 7 garage rock grime celtic
## 8 britpop uk hip hop celtic rock
## 9 dance pop uk garage football
## 10 uk drill europop glasgow indie
## 11 europop dance pop britpop
## 12 house british indie rock eurodance
## 13 disco house house indie rock
## 14 uk garage bass house scottish rock
## 15 british indie rock deep groove house bubble trance
## 16 deep groove house new rave happy hardcore
## 17 deep house uk drill hip house
## 18 new rave dance-punk english indie rock
## 19 tropical house new wave rock
## 20 uk alternative hip hop pop bagpipe
## Edinburgh Bristol Liverpool
## 1 britpop uk hip hop house
## 2 indie rock grime disco house
## 3 hardcore techno uk garage deep groove house
## 4 rock drum and bass tropical house
## 5 bouncy house jump up britpop
## 6 modern rock liquid funk deep house
## 7 new wave jungle dance rock
## 8 scottish rock uk alternative hip hop new wave
## 9 dance rock uk drill uk dance
## 10 celtic rock speed garage new wave pop
## 11 new wave pop bass house rock
## 12 scottish indie neurofunk edm
## 13 celtic house grime
## 14 chamber psych bassline uk hip hop
## 15 disco house dance pop vocal house
## 16 europop dancehall liverpool indie
## 17 folk-pop dub new romantic
## 18 garage rock lovers rock pop
## 19 scottish indie rock pop bouncy house
## 20 chamber pop roots reggae indie rock
## Bradford Cardiff Belfast
## 1 uk hip hop grime irish folk
## 2 grime uk hip hop bouncy house
## 3 uk drill uk garage hardcore techno
## 4 indie rock indie rock irish country
## 5 britpop modern rock trance
## 6 modern rock house bubble trance
## 7 rock dance pop celtic
## 8 uk garage jump up uplifting trance
## 9 dance pop pop progressive house
## 10 europop drum and bass progressive trance
## 11 deep groove house liquid funk house
## 12 garage rock bass house disco house
## 13 house english indie rock deep house
## 14 manchester hip hop rock tech house
## 15 tropical house uk drill eurodance
## 16 uk alternative hip hop europop progressive uplifting trance
## 17 birmingham grime jungle deep groove house
## 18 post-teen pop speed garage german techno
## 19 australian pop deep groove house europop
## 20 bass house post-teen pop float houseNow here comes the dplyr part. We manipulate the data so now each city is a column, in a wide format.
CitiesLikesTidy <- CitiesLikesDF %>% gather(City, Genre)## Warning: attributes are not identical across measure variables;
## they will be droppedCitiesLikesTidyRank <- cbind(CitiesLikesTidy, Rank = 1:20) # Keeping track of their initial order (top = most particularly liked)Almost done. Visualizing the whole thing.
CitiesLikesTidyRank %>%
ggplot(aes(x = City, y = Rank)) +
geom_text(aes(label = Genre, col = Genre), size = 3.5) +
theme(
legend.position = "none",
axis.title.x = element_blank(),
panel.background = element_blank(),
axis.text.y = element_blank(),
axis.title.y=element_blank(),
axis.ticks.y=element_blank(),
text = element_text(size = 18)
) +
scale_y_reverse()