A Data Analysis and Visualization of Music Chart History

STAT 545A - Homework 6
Christian Birch Okkels
October 21, 2013

Introduction

Here, we perform a data analysis and visualization of the music chart history. The dataset to be investigated comes from the so-called Whitburn Project - a huge undertaking by music enthusiasts to preserve and share high-quality recordings of popular songs since the 1890s. The project has spawned a vast spreadsheet with data about almost 40,000 songs (as of 2013) that have been hits on the Billboard Chart since 1890. The dataset contains more than a hundred columns of raw data, with everything from song lengths, artists, songwriters, albums, labels, peak positions, number of weeks on the charts, and even week-by-week chart position, and so much more. This presents a ton of different options for data analysis and visualization; thus, with the relatively small time allotted for this assignment, we will probably just scratch the surface of the true range of possibilities. Although various versions of the dataset exist, none of them are easy to come by. However, we have been lucky to dig one up from the mound of information that is the Internet. Of course, the dataset does come with plenty of shortcomings related to missing data for certain variables. As a result, it needs cleaning and preparation before we can go toe-to-toe with the fun stuff. This cleaning procedure will be described more thoroughly later. For now, let us have a look at some of what the dataset has to offer.

Description of Dataset

As mentioned above, the dataset contains countless columns of raw data for each of the many observations. Some of the more interesting columns (for our case) are described in the table below. As some of the original variable names can be difficult to decipher, the variable names below are those specified in the cleaned version of the data.

Data Cleaning and Preparation

A separate R script, data_cleanPrepare.R has been coded to perform the initial data cleaning and preparation.

First, the script loads some necessary R packages: lattice, plyr, and xtable:

library(lattice)
library(plyr)
library(xtable)

It then proceeds to load, or source, two functions from separate scripts:

source("func_timeToSec.R")
source("func_htmlPrint.R")

The first script, func_timeToSec.R, contains a function that converts time formats of “hh:mm:ss” into seconds. This conversion is critical, since we need numeric or integer values to compute certain statistical properties, etc. The second script, func_htmlPrint.R, holds a function to print data.frames as HTML tables.
Now, the data cleaning script then loads the raw data from the text file charts.txt:

charts_orig = read.delim("charts.txt")
# str(charts_orig) # basic sanity check.

The second line performs a basic sanity check. It has been out-commented since the output is very space-consuming.
Next, we start to cut to the bone by keeping only the most interesting columns:

charts <- subset(charts_orig, select = c(Year, Yearly.Rank, Prefix, CH, X40, 
    X10, PK, High, Artist, Featured, Album, Track, Time, Genre, Temp.1, Date.Entered, 
    Date.Peaked))

All of the many other columns are thereby excluded in the new data.frame charts. Now, as seen in the code above, some variables have rather odd names; e.g. CH, X40, etc. We therefore rename them to something more meaningful:

charts <- rename(charts, c(CH = "nWeeksChart", X40 = "nWeeksChartTop40", X10 = "nWeeksChartTop10", 
    PK = "nWeeksChartPeak", Temp.1 = "ScorePoints"))

Many observations are missing data for certain variables. This is particularly so for older songs from the 1940s and earlier. Therefore, we keep only the data for the years 1950-2013:

charts <- subset(charts, Year > 1949)

Now, the current column Time contains song lengths in the format “mm:ss”. For many purposes, we would like to work with a more manageable format. We thus create a new column, Time.num, which contains the song length in seconds, and then add it to our data.frame. This is where we use the aforementioned function in the script func_timeToSec.R.

charts$Time.num <- sapply(charts$Time, func_timeToSec)

As mentioned, some observations are missing data for certain variables–even after we cut away the older data. The main variable that we consider here is the new Time.num. Therefore, we find all blank entries and replace them with NAs, after which we eliminate all observations with NA:

is.na(charts$Time) <- which(charts$Time == "")
charts <- subset(charts, !is.na(charts$Time))

The nWeeksChart column has also presented plenty of problems. It contains both blank elements and strange n/a (not even “NA”). Moreover, we would like it to be numeric. These cleaning procedures took a long time to figure out for this particular case, but we finally found the solution:

# Remove all observations with blanks or 'n/a''s in nWeeksChart column.
charts <- subset(charts, !nWeeksChart == "")
charts <- subset(charts, !nWeeksChart == "n/a")
charts$nWeeksChart <- factor(charts$nWeeksChart)  # update factor levels.
# Convert nWeeksChart from Factor to Numeric:
charts$nWeeksChart <- as.numeric(levels(charts$nWeeksChart))[as.integer(charts$nWeeksChart)]

Finally, we write the cleaned data to the file charts_clean.tsv:

write.table(charts, "charts_clean.tsv", quote = FALSE, sep = "\t", row.names = FALSE)

Data Aggregation and Plotting

This part is the central one; here, we perform a variety of data aggregation and plotting tasks and write the data tables and figures to file. The R script to perform all of this is data_aggregatePlot.R. It starts out by loading the necessary libraries and sourcing the two functions. Moreover, it reads the cleaned data saved by the script described above.

charts <- read.delim("charts_clean.tsv")
str(charts)  # basic sanity check.

## 'data.frame':    28142 obs. of  18 variables:
##  $ Year            : int  2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##  $ Yearly.Rank     : Factor w/ 890 levels "1","10","100",..: 127 206 230 244 271 295 314 330 339 352 ...
##  $ Prefix          : Factor w/ 28142 levels "1950_001","1950_002",..: 27812 27813 27814 27815 27816 27817 27818 27819 27820 27821 ...
##  $ nWeeksChart     : int  48 40 47 24 33 55 25 49 22 24 ...
##  $ nWeeksChartTop40: int  17 33 29 19 17 35 15 41 6 18 ...
##  $ nWeeksChartTop10: int  0 13 0 10 0 20 0 21 0 1 ...
##  $ nWeeksChartPeak : int  1 1 2 2 1 4 1 6 1 1 ...
##  $ High            : Factor w/ 103 levels "","--","0","1",..: 17 60 11 49 12 27 22 4 26 93 ...
##  $ Artist          : Factor w/ 7290 levels "'Til Tuesday",..: 5075 5357 3116 3761 2108 3116 6033 4397 4160 5148 ...
##  $ Featured        : Factor w/ 1856 levels ""," Feat. Master P, Destiny's Child, O'Dell, Mo B. Dick",..: 1 1 1 1088 1 1 1417 189 647 1 ...
##  $ Album           : Factor w/ 6668 levels "","#1's","#1s\311and Then Some",..: 3723 6656 1224 675 17 1224 6656 6656 6656 6116 ...
##  $ Track           : Factor w/ 22807 levels "#1","#1 Dee Jay",..: 11329 7476 10001 1905 18377 14935 6755 19766 13542 20213 ...
##  $ Time            : Factor w/ 357 levels "1:02","1:16",..: 185 128 158 146 177 106 121 154 141 166 ...
##  $ Genre           : Factor w/ 39 levels "","Adult Contemporary",..: 1 32 1 1 1 1 1 1 1 1 ...
##  $ ScorePoints     : Factor w/ 2686 levels "","#REF!","0",..: 1346 1688 1555 911 935 1855 475 1918 2681 810 ...
##  $ Date.Entered    : Factor w/ 3285 levels "02/20/2010","04/21/2012",..: 1755 2448 2322 2709 2637 2818 3268 3069 3141 513 ...
##  $ Date.Peaked     : Factor w/ 3284 levels "","1/1/00","1/1/05",..: 1155 91 162 232 27 2698 1232 1197 27 1155 ...
##  $ Time.num        : int  266 209 239 227 258 187 202 235 222 247 ...

The output above gives us a good overview of the cleaned, prepared data. For instance, we see the new Time.num column; it has integer elements which are exactly the song lengths in seconds (this can be confirmed by comparing the song lengths to those in the “mm:ss” format in the Time column). The nWeeksChart column, which was earlier a factor, has also successfully been converted; it was converted to numeric, but since it only contained integers, it is now saved as such.

In the following, we consider an array of different data aggregation and plotting tasks.

Distribution of Song Lengths

We start by the simple task of visualizing the distribution of song durations.

histogram(~Time.num, charts, nint = 50, col = "blue", main = "Song length distribution", 
    xlab = "Song length (seconds)")

We see two peaks at roughly 150 and 225 seconds, corresponding to 2:30 and 3:45 minutes, respectively. It is interesting to see that the left dropoff is significantly steeper than the one on the right; apparently, for these more unusual song lengths, it is more common for songs to be longer.

Song Length vs. Year

In this part we consider how song lengths have evolved through time. This is done by making a scatterplot of duration against year. In order to avoid overplotting, we specify the alpha argument. Moreover, a line is drawn through the average song length for each year.

startYear <- min(charts$Year) # starting year (used for plotting x-range).
endYear <- max(charts$Year)   # final year (used for plotting x-range).
songLengthVsYear_xyPlot <- xyplot(Time.num ~ Year, charts, 
       main = "Song length vs. year", xlab = "Year", ylab = "Song length (seconds)", grid = TRUE,
       alpha = 0.5, # combat overplotting via alpha.
       type = c("p", "a"), col.line = "darkorange", lwd = 3,  # draw averages.
       scales = list(y = list(at = seq(0, 600, 30)),
                     x = list(at = seq(startYear, endYear, 5))))
print(songLengthVsYear_xyPlot)

This shows something very interesting; having increased through the 1960s and up to the beginning of 1990s, the average song length peaked just before 1995 and has slowly decreased since then. (A similar plot can be made using stripplot, as exemplified in the data aggregation script.)
Now, we can also combat overplotting via smoothScatter in the panel argument, as done below:

songLengthVsYear_xyPlot2 <- xyplot(Time.num ~ Year, charts, main = "Song length vs. year", 
    xlab = "Year", ylab = "Song length (seconds)", grid = TRUE, scales = list(y = list(at = seq(0, 
        600, 30)), x = list(at = seq(startYear, endYear, 5))), panel = panel.smoothScatter)  # combat overplotting via smoothScatter.
print(songLengthVsYear_xyPlot2)

## KernSmooth 2.23 loaded Copyright M. P. Wand 1997-2009 (loaded the
## KernSmooth namespace)

This gives a nice view of where the majority of the data points are located. From the blue brush-like stroke, we see the same behaviour as described above.

Song Length vs. Decade

This part is much like the previous, except we consider decades instead of years. This is mainly used as an exercise to learn more about R. In this case, considering decades instead of years can lead to some very interesting possibilities; for example, it allows us to more easily condition on the decade variable as a factor (conditioning on the year variable would simply be too much, as there are too many levels), whereby we can obtain a lot of interesting plots.
First, we use cut() to make a new factor called Decade. The factor levels are renamed directly in the function call. Then we use the decades in the plot:

Decade <- cut(charts$Year, 6, labels = c("1950s", "1960s", "1970s", "1980s", 
    "1990s", "2000s"))
songLengthVsDecade_stripplot <- stripplot(Time.num ~ Decade, charts, grid = TRUE, 
    main = "Song length vs. decade", xlab = "Decade", ylab = "Song length (seconds)", 
    jitter.data = TRUE, panel = panel.smoothScatter, scales = list(y = list(at = seq(0, 
        600, 30), rot = c(0, 0), cex = 0.8), x = list(rot = c(0, 0), cex = 0.8)))
print(songLengthVsDecade_stripplot)

The same behaviour described above can also be hinted at here. Another interesting thing is the data for the 1960s; the big, blue, faded dot appears darker than those for the other decades. This means that there are more points gathered there. Consequently, it was more common in the 1960s for songs to be of roughly similar durations. Especially later on, in the past two decades, there seems to be a larger spread.

Shortest and Longest Songs Each Year

Let us find the shortest and longest songs for each year. We not only want just the minimum and maximum song lengths, but we would also like to see some info about these songs; e.g. song title, artist, etc. This data aggregation is performed via the plyr library. In the ddply() call we have custumized our own function:

minSongLengthInfoEachYear <- ddply(charts, ~Year, function(x) {
    theMin <- which.min(x$Time.num)
    shortestSongInfo <- x[theMin, c("Year", "Track", "Artist", "Time.num", "Time")]
    shortestSongInfo <- rename(shortestSongInfo, c(Time.num = "TimeInSeconds"))
})
maxSongLengthInfoEachYear <- ddply(charts, ~Year, function(x) {
    theMax <- which.max(x$Time.num)
    longestSongInfo <- x[theMax, c("Year", "Track", "Artist", "Time.num", "Time")]
    longestSongInfo <- rename(longestSongInfo, c(Time.num = "TimeInSeconds"))
})
write.table(minSongLengthInfoEachYear, "table_minSongLengthInfoEachYear.txt", 
    quote = FALSE, sep = "\t", row.names = FALSE)  # write to file.
write.table(maxSongLengthInfoEachYear, "table_maxSongLengthInfoEachYear.txt", 
    quote = FALSE, sep = "\t", row.names = FALSE)  # write to file.

The above code only defines the data.frames and writes them to file.
NOTE: For the sake of exercise, let us try to read in data from these files and then show it in HTML tables. The tables are rather long, so we will just do it for the minimum song lengths. (To see the longest songs and their info, uncomment the second line.)

htmlPrint(read.delim("table_minSongLengthInfoEachYear.txt"))

# htmlPrint(read.delim('table_maxSongLengthInfoEachYear.txt'))

We can easily find the shortest and longest songs for ALL the years considered:

htmlPrint(minSongLengthInfoEachYear[which.min(minSongLengthInfoEachYear$TimeInSeconds), 
    c("Year", "Track", "Artist", "TimeInSeconds", "Time")])

htmlPrint(maxSongLengthInfoEachYear[which.max(maxSongLengthInfoEachYear$TimeInSeconds), 
    c("Year", "Track", "Artist", "TimeInSeconds", "Time")])

On a related note, we can also find the average song length for ALL years:

sprintf("Average song length for years %d-%d = %4.2f seconds.", min(charts$Year), 
    max(charts$Year), mean(charts$Time.num, na.rm = TRUE))

## [1] "Average song length for years 1950-2013 = 200.79 seconds."

Average, Minimum, and Maximum Song Length Each Year:

Above, we looked at tables and song info for the shortest and longest songs. We now take a step forward and try to visualize the minimum and maximum song lengths through time. We also include the average. In the function in the ddply() call below, these statistics are included as the levels of a factor in the resulting data.frame:

songLengthStatsEachYear2 <- ddply(charts, ~Year, function(x) {
    cLevels <- c("min", "max", "avg")
    data.frame(stat = factor(cLevels, levels = cLevels), songLength = c(range(x$Time.num, 
        na.rm = TRUE), mean(x$Time.num, na.rm = TRUE)))
})
write.table(songLengthStatsEachYear2, "table_songLengthStatsEachYear2.txt", 
    quote = FALSE, sep = "\t", row.names = FALSE)

NOTE: Above, the data.frame is only saved and written it to file, because we once again want to toy with reading it back in and showing it;

htmlPrint(read.delim("table_songLengthStatsEachYear2.txt"))

The data in this data.frame can be used for plotting:

minMaxAvgSongLengthVsYear <- xyplot(songLength ~ Year, songLengthStatsEachYear2, 
    main = "min, max, and average song length vs. year", ylab = "Song length (seconds)", 
    group = stat, type = "b", grid = "h", as.table = TRUE, auto.key = list(columns = 3))
# print(minMaxAvgSongLengthVsYear)
png("plot_minMaxAvgSongLengthVsYear.png")
print(minMaxAvgSongLengthVsYear)
dev.off()

## pdf 
##   2

It is on purpose that we don't immediately print the plot in the above code chunk; we just save it to file.
NOTE: We now want to embed the pre-made plot in this document:

From the green graph we see that the average song length increased from the 1960s to the beginning of the 1990s, and from there on is decreasing a little bit. This is just like what we discussed in some of the earlier sections. Moreover, it is interesting to notice how the minimum song length appears to vary much less from year to year than the maximum song length. (It should be noted that the point–somewhere in the 1960s–where the max, min, and avg are the same, is likely the only data point for that year and should thus be taken with a grain of salt–or more likely just be disregarded entirely.)

Number of Songs Per Year

We now consider how the number of songs listed on the charts has evolved from year to year. We skip years 1955 and older, since these contain so few data points and so many missing values to really cause havoc now that we look at the number of songs.

nSongsEachYear <- ddply(subset(charts, Year > 1955), ~Year, summarize, nSongs = length(Prefix))
# the variable Prefix is unique for each song, which is optimal for this
# case.
nSongsVsYear <- xyplot(nSongs ~ Year, nSongsEachYear, main = "No. of songs on the charts vs. year", 
    ylab = "No. of songs", type = "b", grid = "h")
# print(nSongsVsYear)
png("plot_nSongsVsYear.png")
print(nSongsVsYear)
dev.off()

## pdf 
##   2

NOTE: Again, we only define the plot and print it to file; we want to bring it back in just like before by embedding the pre-made plot in this document:

This plot shows something very interesting, telling us a lot about the diversity of songs on the chart through time. We see a clear peak in the end of the 1960s; more than 700 songs were on the chart for each year in this period. From the 1970s and all the way to the beginning of the 2000s, the number of songs decreased overall; this observed behaviour indicates a smaller amount of diversity on the charts in this period, and thus that the same songs seemed to dominate. From about 2004, the number of songs takes a sudden increase, peaking at about 500 in 2011. The sudden downfall after this could be true, or it could be attributed to the dataset not having been updated with all the new songs for the most recent years of 2012 and 2013.

Proportion of Songs with a Duration Longer than the Total Average Duration

Here, we consider the number and the proportion of songs that have a duration longer than a certain threshold. This threshold is set to be the average song length for the entire time period, but it can be changed at will. The resulting data.frame is shown in the table below.

threshold <- mean(charts$Time.num, na.rm = TRUE)
nSongsLongerThanAvgEachYear <- ddply(subset(charts, Year > 1955), ~Year, function(x) {
    count <- sum(x$Time.num >= threshold, na.rm = TRUE)
    total <- nrow(x)
    prop <- count/total
    data.frame(Count = count, Total = total, Proportion = prop)
})
htmlPrint(nSongsLongerThanAvgEachYear, digits = 2)

We then plot the proportion of songs vs. year:

propSongsLongerThanAvgVsYear <- xyplot(Proportion ~ Year, nSongsLongerThanAvgEachYear, 
    main = paste("Proportion of songs with length >= ", threshold, "(= avg. over all years)"), 
    ylab = "Proportion of songs", type = "b", grid = "h")
print(propSongsLongerThanAvgVsYear)

So, we are comparing the song lengths to the average duration (over all years), which is about 3:20 minutes. Evidently, there are very few older songs longer than this threshold. But there actually is a reason for this: In the 1960s and earlier, the songs were recorded in a so-called 45 RPM format, which had a capacity of about 3 minutes. It is thus no wonder why the left end of graph looks the way it does, with very low proportions at each year. Now, in the end of the 1960s, these recording constraints were removed. And this is exactly what we can see in the plot; from the end of the 1960s, the proportion of songs with a duration longer than 3:20 increases. The peak seemed to have been reached in the 1990s, where almost all songs on the chart were longer than 3:20 minutes. Since then, the trend has been decreasing.

Longest Charting Songs

In this part we will investigate which songs that charted the longest in the Top 100, Top 40, and Top 10, as well as which song charted longest at its highest position. We begin by considering the entire time period. We also extract some additional information belonging to the longest charting songs, e.g. artist, song title, etc.

# longest charting song in top 100 and its info:
htmlPrint(charts[which.max(charts$nWeeksChart), c("Track", "Artist", "Time", 
    "Date.Entered", "High", "Date.Peaked", "nWeeksChart")])

# in top 40:
htmlPrint(charts[which.max(charts$nWeeksChartTop40), c("Track", "Artist", "Time", 
    "Date.Entered", "High", "Date.Peaked", "nWeeksChartTop40")])

# in top 10:
htmlPrint(charts[which.max(charts$nWeeksChartTop10), c("Track", "Artist", "Time", 
    "Date.Entered", "High", "Date.Peaked", "nWeeksChartTop10")])

# song that charted longest at its peak/highest position:
htmlPrint(charts[which.max(charts$nWeeksChartPeak), c("Track", "Artist", "Time", 
    "Date.Entered", "High", "Date.Peaked", "nWeeksChartPeak")])

It is quite impressive what the outputs above tell us. First of all, “I'm Yours” by Jason Mraz was in Top 100 for 76 weeks, and in Top 40 for as long as 62 weeks! And this even though its highest position was only 6. More impressive, perhaps, is that “One Sweet Day” with Mariah Carey held the no. 1 position for 16 weeks–that's 4 months without being pushed off the pole position!
One can also look at earlier years, as is done in the data aggregation and plotting script file. This poses the question whether songs are charting longer nowadays?
We can investigate this:

nWeeksInTop100vsYear <- xyplot(nWeeksChart ~ Year, subset(charts, Year > 1955), 
    grid = "h", main = "No. of weeks a song has charted in Top 100 vs. Year", 
    ylab = "No. of weeks", type = c("p", "a"), col.line = "darkorange", lwd = 3, 
    alpha = 0.5)
print(nWeeksInTop100vsYear)

Some songs in newer time do seem to be charting much longer (as seen by the scattered points in the top right), but the average (orange line) is decreasing.

Chart Position, Weeks on Chart, etc. vs. Song Length

In this part we investigate whether/how song length is related to success on the charts; i.e. should we make our new song long or short? For this purpose we create a large data.frame through a customized function in a ddply() call:

# longest charting songs (in Top 100, 40, 10) and their lengths for each
# year:
longestChartingSongsEachYear <- ddply(charts, ~Year, function(x) {
    max_nWeeksChart <- max(x$nWeeksChart)
    theMax_nWeeksChart <- which.max(x$nWeeksChart)
    max_nWeeksChartTop40 <- max(x$nWeeksChartTop40)
    theMax_nWeeksChartTop40 <- which.max(x$nWeeksChartTop40)
    max_nWeeksChartTop10 <- max(x$nWeeksChartTop10)
    theMax_nWeeksChartTop10 <- which.max(x$nWeeksChartTop10)
    cLevels <- c("weeksInTop100", "weeksInTop40", "weeksInTop10")
    data.frame(successMeasure = factor(cLevels, levels = cLevels), nWeeks = c(max_nWeeksChart, 
        max_nWeeksChartTop40, max_nWeeksChartTop10), Track = c(as.character(x$Track[theMax_nWeeksChart]), 
        as.character(x$Track[theMax_nWeeksChartTop40]), as.character(x$Track[theMax_nWeeksChartTop10])), 
        Artist = c(as.character(x$Artist[theMax_nWeeksChart]), as.character(x$Artist[theMax_nWeeksChartTop40]), 
            as.character(x$Artist[theMax_nWeeksChartTop10])), songLength = c(x$Time.num[theMax_nWeeksChart], 
            x$Time.num[theMax_nWeeksChartTop40], x$Time.num[theMax_nWeeksChartTop10]))
})
write.table(longestChartingSongsEachYear, "table_longestChartingSongsEachYear.txt", 
    quote = FALSE, sep = "\t", row.names = FALSE)

The data.frame is written to file in the last two lines of code above. NOTE: Let's read the table back in and print it as HTML (though it is quite large):

htmlPrint(read.delim("table_longestChartingSongsEachYear.txt"))

What's more interesting, perhaps, is to make a plot:

nWeeksOnChartsVsSongLength <- xyplot(nWeeks ~ songLength, longestChartingSongsEachYear, 
    group = successMeasure, main = "Longest charting songs (3 different measures) for each year vs. song length", 
    xlab = "Song length (seconds)", ylab = "No. of weeks", grid = "h", auto.key = list(columns = 3))
print(nWeeksOnChartsVsSongLength)

pdf("plot_nWeeksOnChartsVsSongLength.pdf")
print(nWeeksOnChartsVsSongLength)
dev.off()

## pdf 
##   2

Finally, we can calculate the average song length based on the longest charting songs; this would give us the “best” song length for success:

bestSongLengthForSucess <- mean(longestChartingSongsEachYear$songLength)
sprintf("Best song length for staying on charts = %4.2f seconds.", bestSongLengthForSucess)

## [1] "Best song length for staying on charts = 213.27 seconds."

So, if we were to make a song and want it to last long on the charts, a good starting point might be to have it last 213.2656 seconds.

Number and Proportion of Songs Charting Longer than 10 Weeks in Top 100 vs. Year

Here, we look at the number and the proportion of songs that have charted longer than a certain amount of weeks (set to 10 by default) in Top 100. We do this for every year.

benchmark <- 10
nSongsChartLongerEachYear <- ddply(subset(charts, Year > 1955), ~Year, function(x) {
    count <- sum(x$nWeeksChart >= benchmark, na.rm = TRUE)
    total <- nrow(x)
    prop <- count/total
    data.frame(Count = count, Total = total, Proportion = prop)
})
# write table to file:
write.table(nSongsChartLongerEachYear, "table_nSongsChartLongerEachYear.txt", 
    quote = FALSE, sep = "\t", row.names = FALSE)
# print HTML table:
htmlPrint(nSongsChartLongerEachYear, digits = 2)

# plot:
propSongsChartLongerVsYear <- xyplot(Proportion ~ Year, nSongsChartLongerEachYear, 
    main = paste("Proportion of songs charting in Top 100 \n                                                  longer than", 
        benchmark, "weeks vs. Year"), ylab = "Proportion", type = "b", grid = "h")
print(propSongsChartLongerVsYear)

# write plot to file:
pdf("plot_propSongsChartLongerVsYear.pdf")
print(propSongsChartLongerVsYear)
dev.off()

pdf 2

This is quite interesting to see; fewer and fewer songs charted in Top 100 for more than 10 weeks in the time period from th 1950s to the end of the 1960s. Afterwards, we observe an overall increase lasting all the way into the 2000s; here, an increasing number, or proportion rather, of songs charted longer than 10 weeks in top 100. The trend for the last 10 or so years is decreasing, however, indicating that songs are on the top 100 for fewer and fewer weeks.

In the data aggregation and plotting script, data_aggregatePlot.R, we have also considered the variable ScorePoints that contains a certain score (see the beginning sections for details) for each song. We leave it out here for the sake of brevity (though it seems we are a far cry from brevity with this report…) and leave it at the mention.

Final Notes

As mentioned in the beginning, the dataset considered here provides an almost endless array of opportunities for data aggregation and visualization. The limiting factors are only the finite amount of time we have available in our lives as well as the dataset's shortcomings in regards to e.g. missing observations.
One thing we would have liked to look at was the Genre variable/column. This would be a great factor on which we could condition, thus making e.g. nice multi-panel plots out of some of the plots already made in the above. However, there are only very few observations in the dataset for which the Genre is available.

There are other cool analyses and visualizations of the data in the Whitburn Project:

• The Top 100 most clichéd clichés in pop song titles: mentions that the many tens of thousands of songs have titles that use a vocabulary of just 9,000 words. Moreover, they show in a nice tag cloud the top 100 most frequently used words in song titles. It can't come as a surprise that “love” is the most common word.
• Climb the Charts, Schmimb the Charts shows that songs do not so much “climb the charts” as they hit the charts and then fade out.
• The Whitburn Project: One-hit Wonders and Pop Longevity gives a nice data visualization of e.g. one-hit wonders for the different decades.
• The Billboard Wayback Machine provides an interactive tool for exploring the Whitburn Project data.
• Six Decades of the Billboard Hot 100 Singles Chart is another interactive data visualization tool by the same person as above.

Regarding code externalization, I have tried to read in code chunks from my R scripts to this R Markdown document–but without success so far. It seems most of the Internet resources on this topic deals with .Rnw files with a slightly different syntax for code chunks than what is used here for .Rmd. My failed attempt is below. The referenced code chunk with the label my-label is at the bottom of the data_aggregatePlot.R script.

read_chunk('data_aggregatePlot.R')
<<my-label>>=
@

However, in the analysis and visualizations above, I have embedded pre-existing figures into this R Markdown document as well as imported pre-existing data from files and worked with it.

As of Monday, Oct 21, just before the deadline, I am also working on a Git repository.
UPDATE on Monday, Oct 21: The Git repository is up and running!

Christian Birch Okkels
October 21, 2013