Two songs drop this week. One enters the Hot 100 at #62, the other at #95.

Four months from now, which one is still on the chart?

My guess was the one that opened strong. A big debut means people are already listening, so it should hang around longer.

R ships with the data to check that. The billboard dataset in tidyr has every song that entered the Hot 100 in 2000, all 317 of them, tracked week by week for up to 76 weeks.

chart <- billboard %>%
  pivot_longer(wk1:wk76, names_to = "week", values_to = "rank",
               values_drop_na = TRUE) %>%
  mutate(week = as.integer(gsub("wk", "", week)))

songs <- chart %>%
  group_by(artist, track) %>%
  summarise(debut = first(rank), peak = min(rank), weeks = n(),
            .groups = "drop")

The raw data has one row per song and 76 separate week columns, which is hard to work with. pivot_longer() turns it into one row per song per week. Then group_by() and summarise() boil each song down to three numbers: where it started, the highest it ever climbed, and how many weeks it lasted.

These five lasted the longest of any songs that year. They climb for a few months, sit near the top, then fade slowly. Look at the left edge, though. All five entered the chart down around #81, which is the first sign my guess was off.

Call \(x\) the debut position and \(Y\) the number of weeks on the chart. A straight line is the simplest thing that could describe the pattern:

\[ Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad \varepsilon_i \sim N(0, \sigma^2) \]

R picks the line by finding the slope and intercept that make the vertical misses as small as possible:

\[ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} \]

If my guess holds up, \(\beta_1\) should come out solidly negative. A worse debut number would mean fewer weeks.

fit_debut <- lm(weeks ~ debut, data = songs)

round(summary(fit_debut)$coefficients, 4)

##             Estimate Std. Error t value Pr(>|t|)
## (Intercept)  27.6712     2.7629 10.0154    0e+00
## debut        -0.1367     0.0340 -4.0220    1e-04

round(summary(fit_debut)$r.squared, 4)

## [1] 0.0488

The slope is \(-0.137\) weeks per chart position, with \(t = -4.02\) and \(p = 0.0000723\).

That p-value is extremely small, so we can rule out coincidence. There really is a downward slope here.

That is a blob. Songs that debuted at #81 lasted anywhere from 1 week to 57 weeks. Hand me a debut position and ask me to guess the run, and I would still be guessing, even with a significant line on the plot.

The p-value answers one narrow question: is the slope something other than zero? How much of the outcome the slope accounts for is a separate measurement:

\[ R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} \]

Debut position scores \(R^2 = 0.049\). It accounts for under 5% of why some songs last and others disappear.

With \(n = 317\) songs, even a very weak pattern clears the significance bar. So the significance test confirmed a slope existed, and the \(R^2\) showed that slope was too weak to predict anything with.

fit_peak <- lm(weeks ~ peak, data = songs)

round(summary(fit_peak)$coefficients, 4)

##             Estimate Std. Error  t value Pr(>|t|)
## (Intercept)  27.3592     0.5864  46.6564        0
## peak         -0.2406     0.0111 -21.7079        0

round(summary(fit_peak)$r.squared, 4)

## [1] 0.5994

Same model, one swap. Instead of where a song started, use the highest position it ever reached.

\(R^2\) goes from \(0.049\) to \(0.599\). Every position closer to #1 is worth about a quarter of an extra week on the chart, and we are 95% confident the real value sits in \([-0.262, -0.219]\).

The points collapse onto the line. These are the same 317 songs as two slides ago, with one variable swapped, and now the relationship is tight enough to actually predict with.

Drag to rotate and hover over a point to see which song it is. Rotate until you are looking down the peak axis and the points line up along a clear diagonal. Looking down the debut axis they stay scattered.

• Where a song enters the chart has a real effect on how long it lasts (\(p = 7.2 \times 10^{-5}\)), but it only accounts for about 5% of the variation (\(R^2 = 0.049\)).
• How high it climbs accounts for 60% on its own (\(R^2 = 0.599\)), and 64% when both variables are used together.
• The big chart runs of 2000 all started near the bottom and climbed for months, so the entry position says very little about what follows.

The thing I would take into the next analysis: report \(R^2\) next to every p-value. On this data the p-value alone would have convinced me to trust a variable that explains 5% of the outcome.