About this case. This is a hypothetical scenario. Spotify is used only because it is a familiar name. The offer, the costs and every row of data are simulated for class; nothing here reflects Spotify’s actual offers, data or results, and the course is not affiliated with Spotify.

1 How to use this document

This walkthrough follows the same five sections as the live-session slides:

  1. Explore & prepare — load the data, set reference levels, hold out a test set.
  2. Build the model — fit a logistic regression with continuous and categorical predictors.
  3. Check & refine — test every assumption, fix what fails, and test one interaction.
  4. Interpret the results — overall tests, odds ratios, the quadratic’s peak, and probabilities.
  5. Validate & test — goodness of fit and diagnostics on the training data, then ROC, cutoffs and the confusion matrix on the test data.

Each step has three parts: what the code does, the code, and how to read the output. Numbers in the text are computed by R as the document knits, so the explanations always match the output.

Before knitting, install the packages once:

install.packages(c("tidyverse", "broom", "car", "lmtest", "pROC", "caret",
                   "ResourceSelection", "knitr"))
library(tidyverse)          # data wrangling (dplyr) and plots (ggplot2)
library(broom)              # tidy(), augment(): model output as data frames
library(car)                # vif(), Anova(), influencePlot()
library(lmtest)             # lrtest(): likelihood-ratio tests
library(pROC)               # roc(), auc(), coords(): ROC curves and cutoffs
library(caret)              # createDataPartition(), confusionMatrix()
library(ResourceSelection)  # hoslem.test(): Hosmer-Lemeshow goodness of fit
library(knitr)              # kable(): tables in this document

# small helpers for writing numbers in the text
f   <- function(x, d = 2) formatC(x, digits = d, format = "f", big.mark = ",")
pct <- function(x, d = 1) paste0(formatC(100 * x, digits = d, format = "f"), "%")
p_txt <- function(p) if (p < .001) "p < .001" else paste0("p = ", sub("^0", "", formatC(p, digits = 3, format = "f")))

2 The business problem

In the scenario, Spotify shows Free listeners an in-app offer: “Try Premium free for one month.” The growth team wants to answer three questions:

  • Who should see the offer?
  • Which format works best — a banner, a video, a full-screen pop-up or an audio ad?
  • How often can the offer be shown before listeners tune it out?

The outcome is binary — the listener either tapped “Try Premium” (1) or did not (0) — so ordinary least-squares regression is the wrong tool: it can predict “probabilities” below 0 or above 1, and its errors cannot be normal. Logistic regression models the log-odds of a click as a linear function of the predictors, which keeps every predicted probability between 0 and 1.

The two kinds of mistakes do not cost the same:

  • A missed clicker (a listener who would have clicked but was not shown the offer) costs about $6 in expected subscription profit, net of the ad slot the offer would have used.
  • Showing the offer to someone who never clicks uses ad slots Spotify could have sold — about $1.

These costs are illustrative assumptions; they matter in Validate and test, when we choose the cutoff.

2.1 The data dictionary

Variable Type Meaning
clicked outcome 1 = tapped “Try Premium”, 0 = did not
age continuous listener’s age in years (18–64)
daily_minutes continuous average minutes of listening per day
weekly_minutes continuous total minutes of listening last week
liked_songs continuous songs in the listener’s Liked Songs library
prior_clicks_90d count ads (any advertiser) the listener clicked in the past 90 days
offer_views_7d count times the listener saw the Premium offer in the past 7 days
device categorical Mobile, Desktop, Tablet
ad_format categorical Banner, Video, Popup, Audio
personalized categorical was the offer personalized to the listener (No, Yes)?

Because the data are simulated, every assumption check has a “right answer” for you to discover. Two are built in on purpose: one pair of predictors that measure the same thing, and two predictors whose effects are curved.

3 Explore and prepare

3.1 Load the data and set reference levels

What the code does. read_csv() reads the file. mutate() then turns the text columns into factors (R’s categorical type) and fixes the order of their levels. The first level of each factor becomes the reference group: every coefficient for that factor compares a level with the reference.

Why set the levels yourself? By default R sorts levels alphabetically, which would make “Audio” the reference for ad_format. Choosing the most common or most natural group (Mobile, Banner, No) makes each odds ratio answer a sensible question, such as “how does a video compare with a banner?”

clicked must also be a factor with levels 0, 1, so that glm() models the probability of 1 (a click) and caret treats 1 as the “positive” class.

clicks <- read_csv("PremiumOffers.csv") |>
  mutate(
    clicked      = factor(clicked, levels = c(0, 1)),
    device       = factor(device,       levels = c("Mobile", "Desktop", "Tablet")),
    ad_format    = factor(ad_format,    levels = c("Banner", "Video", "Popup", "Audio")),
    personalized = factor(personalized, levels = c("No", "Yes"))
  )

glimpse(clicks)
#> Rows: 4,000
#> Columns: 11
#> $ user_id          <chr> "U35057", "U18542", "U91857", "U24844", "U34576", "U4~
#> $ clicked          <fct> 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0,~
#> $ age              <dbl> 33, 28, 39, 18, 18, 42, 39, 21, 43, 30, 35, 30, 35, 2~
#> $ daily_minutes    <dbl> 55.0, 88.7, 27.8, 35.5, 113.9, 35.9, 73.2, 80.2, 115.~
#> $ weekly_minutes   <dbl> 411, 563, 204, 252, 738, 223, 498, 546, 842, 529, 167~
#> $ liked_songs      <dbl> 1124, 187, 968, 137, 27, 731, 758, 642, 55, 601, 539,~
#> $ prior_clicks_90d <dbl> 2, 1, 0, 2, 2, 2, 2, 1, 3, 1, 1, 1, 0, 2, 3, 0, 0, 3,~
#> $ offer_views_7d   <dbl> 4, 3, 0, 8, 3, 6, 3, 2, 10, 8, 2, 0, 4, 2, 5, 4, 6, 0~
#> $ device           <fct> Mobile, Tablet, Desktop, Mobile, Mobile, Desktop, Mob~
#> $ ad_format        <fct> Banner, Banner, Video, Audio, Video, Banner, Video, V~
#> $ personalized     <fct> No, Yes, No, No, No, No, Yes, No, No, No, Yes, Yes, N~
base <- clicks |> count(clicked) |> mutate(pct = n / sum(n))
base
#> # A tibble: 2 x 3
#>   clicked     n    pct
#>   <fct>   <int>  <dbl>
#> 1 0        3133 0.7832
#> 2 1         867 0.2168

How to read it. 867 of 4,000 listeners clicked — a base rate of 21.7%. A “model” that predicts no one clicks would therefore be right 78.3% of the time. Remember this number: in Validate and test it is the bar that raw accuracy must clear (the no-information rate), and it is why accuracy alone can mislead.

summary(select(clicks, age:offer_views_7d))
#>       age        daily_minutes    weekly_minutes    liked_songs    
#>  Min.   :18.00   Min.   :  4.60   Min.   :  28.0   Min.   :  12.0  
#>  1st Qu.:26.00   1st Qu.: 38.20   1st Qu.: 266.0   1st Qu.: 136.0  
#>  Median :34.00   Median : 55.30   Median : 382.0   Median : 254.0  
#>  Mean   :34.32   Mean   : 59.67   Mean   : 417.9   Mean   : 382.4  
#>  3rd Qu.:41.00   3rd Qu.: 75.60   3rd Qu.: 532.0   3rd Qu.: 473.2  
#>  Max.   :64.00   Max.   :240.00   Max.   :1601.0   Max.   :5000.0  
#>  prior_clicks_90d offer_views_7d 
#>  Min.   : 0.000   Min.   : 0.00  
#>  1st Qu.: 1.000   1st Qu.: 2.00  
#>  Median : 2.000   Median : 3.00  
#>  Mean   : 1.806   Mean   : 4.02  
#>  3rd Qu.: 3.000   3rd Qu.: 6.00  
#>  Max.   :10.000   Max.   :15.00

Notice that liked_songs is strongly right-skewed (the mean is well above the median) and that prior_clicks_90d and offer_views_7d are counts with many zeros. Both facts matter when we check linearity.

3.2 Split before you tune

What the code does. createDataPartition() randomly selects 70% of the rows for training and leaves 30% for testing. It stratifies on clicked, so both sets keep the same click rate. list = FALSE returns plain row numbers that we can use to index the data. set.seed() makes the random split reproducible.

Why split first? Every decision we make — which variable to drop, which to transform, where to put the cutoff — is a choice fitted to the data. If we make those choices and judge the model on the same rows, the performance numbers are optimistic. The test set is set aside now and used once, at the very end.

set.seed(2026)
idx   <- createDataPartition(clicks$clicked, p = 0.70, list = FALSE)
train <- clicks[idx, ]
test  <- clicks[-idx, ]

c(train = nrow(train), test = nrow(test))
#> train  test 
#>  2801  1199
c(train_rate = mean(train$clicked == "1"), test_rate = mean(test$clicked == "1"))
#> train_rate  test_rate 
#>  0.2167083  0.2168474

How do you choose the percentages? There is no single correct split; 70/30 and 80/20 are conventions. Think in counts, not percentages:

  • The training set needs enough events to estimate the model — at least 10 events per estimated parameter. Here it has 607 clicks.
  • The test set needs enough events to estimate performance precisely; for validating a logistic model, Collins et al. (2016) recommend at least 100 events, ideally 200 or more. Here it has 260 clicks.

With very large data a small percentage is enough for testing; with small data, a single split wastes information and cross-validation or the bootstrap is better (Steyerberg et al., 2001).

4 Build the model

4.1 Fit the full model

What the code does. glm() fits a generalized linear model. family = binomial(link = "logit") makes it a logistic regression: the model is

\[\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k,\]

where \(p\) is the probability of a click. We start with every candidate predictor: six continuous (or count) predictors and three categorical ones. R turns each factor into dummy variables automatically — one for every level except the reference.

m_full <- glm(clicked ~ age + daily_minutes + weekly_minutes +
                liked_songs + prior_clicks_90d + offer_views_7d +
                device + ad_format + personalized,
              data = train, family = binomial(link = "logit"))
summary(m_full)
#> 
#> Call:
#> glm(formula = clicked ~ age + daily_minutes + weekly_minutes + 
#>     liked_songs + prior_clicks_90d + offer_views_7d + device + 
#>     ad_format + personalized, family = binomial(link = "logit"), 
#>     data = train)
#> 
#> Coefficients:
#>                    Estimate Std. Error z value Pr(>|z|)    
#> (Intercept)      -2.6661794  0.2402893 -11.096  < 2e-16 ***
#> age              -0.0227319  0.0048941  -4.645 3.41e-06 ***
#> daily_minutes    -0.0008004  0.0114048  -0.070 0.944048    
#> weekly_minutes    0.0017079  0.0016066   1.063 0.287758    
#> liked_songs       0.0009731  0.0001104   8.815  < 2e-16 ***
#> prior_clicks_90d  0.1706092  0.0350109   4.873 1.10e-06 ***
#> offer_views_7d    0.0954210  0.0152833   6.243 4.28e-10 ***
#> deviceDesktop    -0.5480778  0.1284189  -4.268 1.97e-05 ***
#> deviceTablet     -0.1952428  0.1627807  -1.199 0.230364    
#> ad_formatVideo    0.5871662  0.1191238   4.929 8.26e-07 ***
#> ad_formatPopup    0.2920276  0.1374857   2.124 0.033665 *  
#> ad_formatAudio   -0.5854725  0.1726263  -3.392 0.000695 ***
#> personalizedYes   0.5862886  0.0985796   5.947 2.73e-09 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> (Dispersion parameter for binomial family taken to be 1)
#> 
#>     Null deviance: 2928.2  on 2800  degrees of freedom
#> Residual deviance: 2610.1  on 2788  degrees of freedom
#> AIC: 2636.1
#> 
#> Number of Fisher Scoring iterations: 5

How to read it.

  • Each Estimate is on the log-odds scale. For a continuous predictor it is the change in the log-odds of a click for a one-unit increase, holding the other predictors constant. For a dummy (for example ad_formatVideo) it is the difference in log-odds between that level and the reference (Banner).
  • z value and Pr(>|z|) are the Wald test of whether that coefficient is zero.
  • There are 12 coefficients plus the intercept: 6 continuous terms, 2 device dummies, 3 ad_format dummies and 1 personalized dummy.

Something is wrong. Listening time should matter, yet daily_minutes (p = .944) and weekly_minutes (p = .288) are both non-significant, and daily_minutes even has a negative sign. Note the standard error of daily_minutes, 0.0114; we will see it shrink once the problem is fixed. That is the classic symptom of multicollinearity: two predictors that measure the same thing hide each other.

Also note that offer_views_7d has a positive slope (0.095): read literally, “every extra view of the offer raises the odds of a click.” Hold that thought — the linearity check shows it is wrong.

5 Check and refine the model

Each check ends in one of three words: pass, fix, or limitation.

5.1 Assumption 1: adequate sample size

What the code does. It counts the events (clicks) in the training data and the number of estimated parameters (coefficients, not counting the intercept), then divides. It also cross-tabulates every categorical predictor against the outcome.

events <- sum(train$clicked == "1")
k      <- length(coef(m_full)) - 1          # parameters, not variables
c(events = events, parameters = k, EPV = round(events / k, 1))
#>     events parameters        EPV 
#>      607.0       12.0       50.6
table(train$device,       train$clicked)
#>          
#>              0    1
#>   Mobile  1418  444
#>   Desktop  542  101
#>   Tablet   234   62
table(train$ad_format,    train$clicked)
#>         
#>            0   1
#>   Banner 775 187
#>   Video  609 235
#>   Popup  415 129
#>   Audio  395  56
table(train$personalized, train$clicked)
#>      
#>          0    1
#>   No  1292  286
#>   Yes  902  321

How to read it.

  • Events per variable (EPV) = 607 / 12 = 50.6, far above the rule of thumb of 10. Count parameters, not variables: ad_format is one variable but uses 3 parameters.
  • Cell counts: every outcome × category cell should exceed 5. The smallest is 56 (audio ads that were clicked). Pass.

5.2 Assumption 2: no multicollinearity

What the code does. vif() computes a variance inflation factor for each predictor: how much the variance of its coefficient is inflated because it overlaps with the other predictors. When a model contains factors, car reports a generalized VIF (GVIF). For 1-df terms GVIF equals VIF; for a factor, square the last column, GVIF^(1/(2*Df)), before comparing it with the usual rubric. The correlation matrix shows which predictors overlap.

The rubric. About 1: none. 1–5: usually fine. 5–10: concerning. Above 10: severe.

vif(m_full)
#>                       GVIF Df GVIF^(1/(2*Df))
#> age               1.010005  1        1.004990
#> daily_minutes    51.451696  1        7.172984
#> weekly_minutes   51.425073  1        7.171128
#> liked_songs       1.020608  1        1.010252
#> prior_clicks_90d  1.042406  1        1.020983
#> offer_views_7d    1.014522  1        1.007235
#> device            1.010466  2        1.002606
#> ad_format         1.025460  3        1.004199
#> personalized      1.017413  1        1.008669
train |>
  select(age, daily_minutes, weekly_minutes, liked_songs,
         prior_clicks_90d, offer_views_7d) |>
  cor() |>
  round(2)
#>                    age daily_minutes weekly_minutes liked_songs
#> age               1.00         -0.02          -0.02        0.01
#> daily_minutes    -0.02          1.00           0.99        0.00
#> weekly_minutes   -0.02          0.99           1.00        0.00
#> liked_songs       0.01          0.00           0.00        1.00
#> prior_clicks_90d  0.01          0.19           0.19       -0.01
#> offer_views_7d    0.02          0.01           0.01       -0.02
#>                  prior_clicks_90d offer_views_7d
#> age                          0.01           0.02
#> daily_minutes                0.19           0.01
#> weekly_minutes               0.19           0.01
#> liked_songs                 -0.01          -0.02
#> prior_clicks_90d             1.00          -0.02
#> offer_views_7d              -0.02           1.00

How to read it. daily_minutes (GVIF = 51.45) and weekly_minutes (GVIF = 51.43) are severe. Their correlation is r = 0.99 — weekly minutes is essentially 7 × daily minutes. Every other term is close to 1.

Multicollinearity does not bias coefficients; it inflates their variance. The square root of the VIF is roughly how much the standard error is inflated: √51 ≈ 7.2.

5.2.1 Fix: drop the redundant predictor

What the code does. update() refits an existing model without retyping the formula. In . ~ ., the left dot means “the same outcome” and the right dot means “the same predictors”; - weekly_minutes removes one term. We keep daily_minutes because a manager can picture “minutes per day.”

m2 <- update(m_full, . ~ . - weekly_minutes)
vif(m2)
#>                      GVIF Df GVIF^(1/(2*Df))
#> age              1.009929  1        1.004952
#> daily_minutes    1.038679  1        1.019156
#> liked_songs      1.020505  1        1.010200
#> prior_clicks_90d 1.042443  1        1.021001
#> offer_views_7d   1.014506  1        1.007227
#> device           1.009743  2        1.002427
#> ad_format        1.023207  3        1.003831
#> personalized     1.017574  1        1.008749
summary(m2)$coefficients["daily_minutes", ]
#>     Estimate   Std. Error      z value     Pr(>|z|) 
#> 1.119530e-02 1.621340e-03 6.904969e+00 5.021424e-12

How to read it. Every VIF is now close to 1. Pass. daily_minutes now has a positive coefficient (0.0112), its standard error fell from 0.0114 to 0.0016 — about 7 times smaller, matching the √VIF above — and it is clearly significant (p < .001). The variable mattered all along; its twin was hiding it.

Lesson. Never drop a variable just because it is non-significant in a model with collinear predictors. Other fixes include combining the two into one measure or using regularization.

5.3 Assumption 3: linearity in the logit

Logistic regression assumes that each continuous predictor has a straight-line relationship with the log-odds of the outcome (not with the probability). Categorical predictors meet this automatically because each level gets its own dummy.

5.3.1 Look first: the empirical log-odds plot

What the code does. It splits listeners into 10 equal-sized groups by the predictor (ntile()), finds the middle value of the predictor and the share who clicked in each group, converts that share to log-odds with \(\log(p / (1 - p))\), and plots the result. If the relationship is linear in the logit, the points follow a straight line.

binned <- train |>
  mutate(group = ntile(liked_songs, 10)) |>        # 10 equal-sized groups
  group_by(group) |>
  summarise(typical_x  = median(liked_songs),       # middle value of each group
            click_rate = mean(clicked == "1")) |>   # share who clicked
  mutate(log_odds = log(click_rate / (1 - click_rate)))
binned
#> # A tibble: 10 x 4
#>    group typical_x click_rate log_odds
#>    <int>     <dbl>      <dbl>    <dbl>
#>  1     1      59      0.09253  -2.283 
#>  2     2      99      0.1607   -1.653 
#>  3     3     137.5    0.1643   -1.627 
#>  4     4     178      0.175    -1.551 
#>  5     5     227      0.2107   -1.321 
#>  6     6     284      0.2036   -1.364 
#>  7     7     360.5    0.2071   -1.342 
#>  8     8     478      0.275    -0.9694
#>  9     9     662.5    0.3143   -0.7802
#> 10    10    1064      0.3643   -0.5568
ggplot(binned, aes(typical_x, log_odds)) +
  geom_point(size = 3) +
  geom_smooth(method = "lm", se = FALSE) +
  labs(x = "Liked songs (middle value of each group)", y = "Log-odds of a click",
       title = "liked_songs: rises fast, then flattens")

How to read it. The log-odds climb quickly across the first few groups and then rise more slowly. The points form a concave curve — the shape of a logarithm — and the straight line misses both ends. This is a warning sign.

The same steps for age and daily_minutes (change the variable name in ntile() and median()):

binned_age <- train |>
  mutate(group = ntile(age, 10)) |>
  group_by(group) |>
  summarise(typical_x = median(age), click_rate = mean(clicked == "1")) |>
  mutate(log_odds = log(click_rate / (1 - click_rate)))

binned_min <- train |>
  mutate(group = ntile(daily_minutes, 10)) |>
  group_by(group) |>
  summarise(typical_x = median(daily_minutes), click_rate = mean(clicked == "1")) |>
  mutate(log_odds = log(click_rate / (1 - click_rate)))

bind_rows(age = binned_age, daily_minutes = binned_min, .id = "predictor") |>
  ggplot(aes(typical_x, log_odds)) +
  geom_point(size = 2.5) +
  geom_smooth(method = "lm", se = FALSE) +
  facet_wrap(~ predictor, scales = "free_x") +
  labs(x = "Middle value of each group", y = "Log-odds of a click")

Both are noisy but roughly straight: age gently falling, daily_minutes gently rising.

5.3.2 Confirm it: the Box–Tidwell test

What the code does. For each strictly positive continuous predictor \(X\), it adds the term \(X \cdot \ln(X)\) to the model. If the relationship is linear in the logit, this extra term adds nothing; a significant \(X \cdot \ln(X)\) term means the predictor is not linear. I() tells R to do the arithmetic — inside a formula, * would otherwise mean “interaction.”

prior_clicks_90d and offer_views_7d contain zeros, and \(\ln(0)\) is undefined, so they are checked visually instead (below).

m_bt <- update(m2, . ~ . + I(age * log(age)) +
                 I(daily_minutes * log(daily_minutes)) +
                 I(liked_songs * log(liked_songs)))

bt_rows <- c("I(age * log(age))",
             "I(daily_minutes * log(daily_minutes))",
             "I(liked_songs * log(liked_songs))")
summary(m_bt)$coefficients[bt_rows, ]
#>                                            Estimate   Std. Error   z value
#> I(age * log(age))                     -0.0311623906 0.0289344057 -1.077001
#> I(daily_minutes * log(daily_minutes)) -0.0082691745 0.0052061328 -1.588353
#> I(liked_songs * log(liked_songs))     -0.0008280462 0.0001714362 -4.830055
#>                                           Pr(>|z|)
#> I(age * log(age))                     2.814797e-01
#> I(daily_minutes * log(daily_minutes)) 1.122066e-01
#> I(liked_songs * log(liked_songs))     1.364952e-06

How to read it. Only the three \(X \cdot \ln(X)\) rows matter.

Term p-value Verdict
age × ln(age) p = .281 linear — pass
daily_minutes × ln(daily_minutes) p = .112 linear — pass
liked_songs × ln(liked_songs) p < .001 not linear — fix

This agrees with the plot. The negative coefficient on the \(X \cdot \ln(X)\) term (-0.00083) is consistent with a curve that flattens as \(X\) grows, which is why a log transformation is the natural fix.

5.3.3 Fix 1: transform the predictor

What the code does.

  1. Swap liked_songs for log(liked_songs); everything else stays the same.
  2. Re-run Box–Tidwell on the new predictor. Box–Tidwell adds \(X \cdot \ln(X)\), and the new \(X\) is log(liked_songs), so the term is log(liked_songs) * log(log(liked_songs)).
  3. Read only that one row: p > .05 means there is no evidence of a remaining bend.
  4. Compare fit with AIC. Both models have the same number of parameters, so the lower AIC wins.
m3 <- update(m2, . ~ . - liked_songs + log(liked_songs))

m3_bt <- update(m3, . ~ . + I(log(liked_songs) * log(log(liked_songs))))
summary(m3_bt)$coefficients["I(log(liked_songs) * log(log(liked_songs)))", ]
#>    Estimate  Std. Error     z value    Pr(>|z|) 
#> -0.03088137  0.52081072 -0.05929481  0.95271729
AIC(m2, m3)
#>    df      AIC
#> m2 12 2635.225
#> m3 12 2604.495

How to read it. The new Box–Tidwell term is not significant (p = .953): pass. AIC fell from 2,635.2 to 2,604.5 — 30.7 points — with the same number of parameters. The log scale also makes business sense: going from 50 to 100 liked songs says much more about a listener than going from 2,000 to 2,050. The coefficient is now read per doubling of liked songs (see Odds ratios).

5.3.4 A curve that rises, then falls

Counts with many zeros produce ties when cut into deciles, so for offer_views_7d we compute the empirical log-odds at each value instead. The few listeners with 12 or more views are pooled into one group (“12 or more”).

by_views <- train |>
  mutate(views = pmin(offer_views_7d, 12)) |>       # 12 = "12 or more"
  group_by(views) |>
  summarise(listeners  = n(),
            click_rate = mean(clicked == "1")) |>
  mutate(log_odds = log(click_rate / (1 - click_rate)))
by_views
#> # A tibble: 13 x 4
#>    views listeners click_rate log_odds
#>    <dbl>     <int>      <dbl>    <dbl>
#>  1     0       199    0.06030  -2.746 
#>  2     1       396    0.09343  -2.272 
#>  3     2       445    0.1753   -1.549 
#>  4     3       394    0.2360   -1.175 
#>  5     4       372    0.2608   -1.042 
#>  6     5       263    0.3384   -0.6704
#>  7     6       210    0.3476   -0.6295
#>  8     7       162    0.2654   -1.018 
#>  9     8       111    0.3514   -0.6131
#> 10     9        83    0.3012   -0.8416
#> 11    10        56    0.1786   -1.526 
#> 12    11        39    0.2308   -1.204 
#> 13    12        71    0.02817  -3.541
ggplot(by_views, aes(views, log_odds)) +
  geom_point(aes(size = listeners)) +               # bigger dot = more listeners
  geom_smooth(method = "lm", se = FALSE) +
  scale_x_continuous(breaks = 0:12) +
  labs(x = "Offer views in the past 7 days (12 = 12 or more)",
       y = "Log-odds of a click", size = "Listeners")

How to read it. The chance of a click rises from about 6% with no prior views to about 35% in the middle, then falls back to about 3% at 12 or more. This is an inverted U: the first few views build interest, and after that listeners tire of seeing the same offer (offer fatigue). The straight fitted line goes through the middle and misses both ends — and it is why the full model reported a misleading positive slope.

A log cannot fix this: a log is still monotone (it only rises). A curve that changes direction needs a squared (quadratic) term.

prior_clicks_90d, checked the same way, rises steadily and is close enough to straight:

train |>
  mutate(prior = pmin(prior_clicks_90d, 6)) |>
  group_by(prior) |>
  summarise(listeners = n(), click_rate = mean(clicked == "1")) |>
  mutate(log_odds = log(click_rate / (1 - click_rate)))
#> # A tibble: 7 x 4
#>   prior listeners click_rate log_odds
#>   <dbl>     <int>      <dbl>    <dbl>
#> 1     0       479     0.1461  -1.765 
#> 2     1       851     0.1904  -1.448 
#> 3     2       701     0.2282  -1.218 
#> 4     3       457     0.2713  -0.9879
#> 5     4       204     0.2843  -0.9232
#> 6     5        72     0.3056  -0.8210
#> 7     6        37     0.2973  -0.8602

5.3.5 Fix 2: add a centered quadratic term

What the code does.

  1. Centers the predictor: views_c = offer_views_7d - 4. Four is a round number close to the training mean (4.01). The new column is added to both train and test — otherwise predict() on the test set will fail later.
  2. Shows why centering matters by comparing the VIFs of the raw and centered versions.
  3. Fits m4, replacing offer_views_7d with views_c + I(views_c^2). I() is required: inside a formula, ^ has a different meaning, and I() tells R to square the numbers.
  4. Tests whether the squared term earns its extra parameter with a nested likelihood-ratio test and AIC.
train <- train |> mutate(views_c = offer_views_7d - 4)
test  <- test  |> mutate(views_c = offer_views_7d - 4)

vif(update(m3, . ~ . + I(offer_views_7d^2)))     # raw X and X^2
#>                          GVIF Df GVIF^(1/(2*Df))
#> age                  1.014388  1        1.007168
#> daily_minutes        1.038956  1        1.019292
#> prior_clicks_90d     1.053408  1        1.026357
#> offer_views_7d      11.267780  1        3.356751
#> device               1.013481  2        1.003353
#> ad_format            1.030272  3        1.004983
#> personalized         1.021515  1        1.010700
#> log(liked_songs)     1.030306  1        1.015040
#> I(offer_views_7d^2) 11.164595  1        3.341346
m4 <- update(m3, . ~ . - offer_views_7d + views_c + I(views_c^2))
vif(m4)                                          # centered
#>                      GVIF Df GVIF^(1/(2*Df))
#> age              1.014388  1        1.007168
#> daily_minutes    1.038956  1        1.019292
#> prior_clicks_90d 1.053408  1        1.026357
#> device           1.013481  2        1.003353
#> ad_format        1.030272  3        1.004983
#> personalized     1.021515  1        1.010700
#> log(liked_songs) 1.030306  1        1.015040
#> views_c          1.593725  1        1.262428
#> I(views_c^2)     1.573437  1        1.254367
lrtest(m3, m4)
#> Likelihood ratio test
#> 
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + offer_views_7d + 
#>     device + ad_format + personalized + log(liked_songs)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2)
#>   #Df  LogLik Df  Chisq Pr(>Chisq)    
#> 1  12 -1290.2                         
#> 2  13 -1216.0  1 148.44  < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
AIC(m3, m4)
#>    df      AIC
#> m3 12 2604.495
#> m4 13 2458.053
summary(m4)$coefficients[c("views_c", "I(views_c^2)"), ]
#>                 Estimate  Std. Error   z value     Pr(>|z|)
#> views_c       0.28501523 0.024836696 11.475569 1.750217e-30
#> I(views_c^2) -0.05893736 0.006006997 -9.811452 1.005114e-22

How to read it.

  • Why center? When \(X\) is always positive, \(X\) and \(X^2\) rise together, which creates built-in multicollinearity: the raw versions have VIFs of 11.3 and 11.2. After centering they fall to 1.59 and 1.57. Centering does not change the fit, the fitted values or the deviance. Only the intercept and the linear term change (what they mean and how precisely they are estimated); the squared term’s coefficient and standard error are identical in both versions.
  • Did it work? The LR test gives χ²(1) = 148.4, p < .001, and AIC fell by 146.4 points. The squared term is clearly needed.
  • What shape? The squared term is negative (-0.0589), so the curve opens downward: an inverted U, exactly what the plot showed.

Keep the linear term whenever you include its square (the hierarchy principle). Without it, the model forces the peak to sit exactly at the centering value.

5.4 Assumption 4: independence — plus a separation check

What the code does. It checks that each listener appears only once, that the maximum-likelihood fit converged, and that there are no fitted probabilities of exactly 0 or 1 or runaway coefficients (the signs of perfect separation).

n_distinct(clicks$user_id) == nrow(clicks)   # one row per listener?
#> [1] TRUE
m4$converged                                  # did the estimation converge?
#> [1] TRUE
range(fitted(m4))                             # any fitted p of exactly 0 or 1?
#> [1] 0.002042521 0.918038718
max(abs(coef(m4)))                            # any runaway coefficient?
#> [1] 5.051972

How to read it. Every listener appears once, the model converged, the fitted probabilities range from 0.002 to 0.918, and the largest coefficient (the intercept) is modest with a small standard error. Pass on both checks.

A limitation to name. If the same ad creative was shown to many listeners, listeners are clustered within creatives, and the standard errors would be too small. The file has no creative ID, so we cannot check this. The fixes would be mixed-effects models (lme4::glmer) or cluster-robust standard errors.

5.5 Refine: test an interaction before you add it

What the code does. It asks whether personalization works equally well for every ad format by adding the personalized × ad_format interaction and comparing the two models with a nested likelihood-ratio test and AIC.

m_int <- update(m4, . ~ . + personalized:ad_format)
lrtest(m4, m_int)                    # nested LR test: 3 extra parameters
#> Likelihood ratio test
#> 
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2) + 
#>     ad_format:personalized
#>   #Df  LogLik Df  Chisq Pr(>Chisq)
#> 1  13 -1216.0                     
#> 2  16 -1215.5  3 0.9889     0.8039
AIC(m4, m_int)
#>       df      AIC
#> m4    13 2458.053
#> m_int 16 2463.064
summary(m_int)$coefficients[grep(":", names(coef(m_int))), ]
#>                                    Estimate Std. Error     z value  Pr(>|z|)
#> ad_formatVideo:personalizedYes -0.003743884  0.2460457 -0.01521621 0.9878597
#> ad_formatPopup:personalizedYes  0.115976712  0.2850875  0.40681098 0.6841468
#> ad_formatAudio:personalizedYes  0.315099397  0.3589492  0.87783837 0.3800314

How to read it. An interaction between a 2-level and a 4-level factor costs (2 − 1) × (4 − 1) = 3 parameters, so the test has 3 degrees of freedom. The result is χ²(3) = 0.99, p = .804, and AIC rises by 5.0 points. There is no evidence that the effect of personalization depends on the format, so we keep the simpler model.

Compare this with the quadratic: both were judged by the same nested LR test, with opposite verdicts. Test interactions you have a business reason to expect; do not search every possible pair.

5.6 Summary: how the model was built

tibble(
  Model    = c("m_full", "m2", "m3", "m4", "m_int"),
  Change   = c("All candidate predictors", "Drop weekly_minutes", "Use log(liked_songs)",
               "Add centered offer views squared", "Add personalized x ad_format"),
  df       = c(attr(logLik(m_full), "df"), attr(logLik(m2), "df"), attr(logLik(m3), "df"),
               attr(logLik(m4), "df"), attr(logLik(m_int), "df")),
  AIC      = round(c(AIC(m_full), AIC(m2), AIC(m3), AIC(m4), AIC(m_int)), 1),
  Decision = c("Fix", "Keep", "Keep", "Final", "Reject")
) |>
  kable(align = "llrrl")
Model Change df AIC Decision
m_full All candidate predictors 13 2636.1 Fix
m2 Drop weekly_minutes 12 2635.2 Keep
m3 Use log(liked_songs) 12 2604.5 Keep
m4 Add centered offer views squared 13 2458.1 Final
m_int Add personalized x ad_format 16 2463.1 Reject

df is the number of estimated parameters including the intercept. Dropping weekly_minutes barely changes AIC — it fixes the standard errors, not the fit. The large gains come from modeling the two curves correctly. m4 is the final model.

final <- m4
formula(final)
#> clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2)

6 Interpret the results

6.1 Is the model useful?

What the code does.

  • lrtest(final) compares the final model with an intercept-only model (no predictors). This is the global likelihood-ratio test.
  • Anova(..., type = "III", test.statistic = "LR") tests each predictor as a whole — all of its levels together — given every other predictor.
  • Offer views is in the model as two terms, so it must be tested by dropping both together (a 2-df likelihood-ratio test). The Type III row for views_c alone drops the linear term but keeps the square; that only tests whether the curve is flat at 4 views, and its size depends on where we centered. Do not read it as evidence about offer views.
  • McFadden’s pseudo-R² is \(1 - \text{deviance}_{\text{model}} / \text{deviance}_{\text{null}}\).
lrtest(final)
#> Likelihood ratio test
#> 
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2)
#> Model 2: clicked ~ 1
#>   #Df  LogLik  Df  Chisq Pr(>Chisq)    
#> 1  13 -1216.0                          
#> 2   1 -1464.1 -12 496.17  < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Anova(final, type = "III", test.statistic = "LR")
#> Analysis of Deviance Table (Type III tests)
#> 
#> Response: clicked
#>                  LR Chisq Df Pr(>Chisq)    
#> age                24.858  1  6.171e-07 ***
#> daily_minutes      40.568  1  1.899e-10 ***
#> prior_clicks_90d   30.517  1  3.309e-08 ***
#> device             18.459  2  9.812e-05 ***
#> ad_format          66.253  3  2.706e-14 ***
#> personalized       39.167  1  3.891e-10 ***
#> log(liked_songs)  125.000  1  < 2.2e-16 ***
#> views_c           153.478  1  < 2.2e-16 ***
#> I(views_c^2)      148.442  1  < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
lrtest(update(final, . ~ . - views_c - I(views_c^2)), final)   # offer views: both terms
#> Likelihood ratio test
#> 
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format + 
#>     personalized + log(liked_songs) + views_c + I(views_c^2)
#>   #Df  LogLik Df  Chisq Pr(>Chisq)    
#> 1  11 -1308.5                         
#> 2  13 -1216.0  2 184.88  < 2.2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
1 - final$deviance / final$null.deviance          # McFadden pseudo-R2
#> [1] 0.1694438

How to read it.

  • The global test gives χ²(12) = 496.2, p < .001: the model is far better than no predictors.
  • Offer views as a whole: χ²(2) = 184.9, p < .001. Its curvature row, I(views_c^2), is meaningful on its own (is the curve needed?) and repeats the m3-versus-m4 test above.
  • Every other predictor is significant in the Type III tests. device is significant as a whole even though its Tablet dummy alone is not.
  • McFadden’s R² = 0.169. Pseudo-R² values run much lower than OLS R²; values of .2 to .4 are already considered excellent, so never judge them against 0.7-style benchmarks.

“Significant” means “adds information beyond the other predictors” — not “has a large effect.” The odds ratios show the size of each effect.

6.2 Odds ratios

What the code does. tidy(..., exponentiate = TRUE, conf.int = TRUE) converts each coefficient to an odds ratio (OR = \(e^{\beta}\)) with a 95% confidence interval. Because a one-unit change is too small to matter for some predictors, three are rescaled:

  • daily_minutes and age: per 10 units, \(e^{10\beta}\).
  • log(liked_songs): per doubling of liked songs, \(2^{\beta}\).
tidy(final, exponentiate = TRUE, conf.int = TRUE) |>
  select(term, odds_ratio = estimate, conf.low, conf.high, p.value) |>
  mutate(across(where(is.numeric), \(x) round(x, 3)))
#> # A tibble: 13 x 5
#>    term             odds_ratio conf.low conf.high p.value
#>    <chr>                 <dbl>    <dbl>     <dbl>   <dbl>
#>  1 (Intercept)           0.006    0.003     0.014   0    
#>  2 age                   0.975    0.966     0.985   0    
#>  3 daily_minutes         1.011    1.008     1.014   0    
#>  4 prior_clicks_90d      1.225    1.14      1.316   0    
#>  5 deviceDesktop         0.577    0.444     0.745   0    
#>  6 deviceTablet          0.825    0.589     1.143   0.256
#>  7 ad_formatVideo        1.841    1.446     2.348   0    
#>  8 ad_formatPopup        1.347    1.018     1.779   0.036
#>  9 ad_formatAudio        0.507    0.356     0.713   0    
#> 10 personalizedYes       1.89     1.547     2.312   0    
#> 11 log(liked_songs)      1.891    1.685     2.127   0    
#> 12 views_c               1.33     1.267     1.397   0    
#> 13 I(views_c^2)          0.943    0.931     0.954   0
exp(10 * coef(final)["daily_minutes"])           # per 10 more minutes a day
#> daily_minutes 
#>      1.114408
exp(10 * coef(final)["age"])                     # per 10 years older
#>       age 
#> 0.7787868
2 ^ coef(final)["log(liked_songs)"]              # per doubling of liked songs
#> log(liked_songs) 
#>          1.55526

How to read it. An OR above 1 raises the odds of a click; below 1 lowers them; a confidence interval that includes 1 means the difference is not statistically significant at the 5% level (the data are consistent with no difference).

Effect Odds ratio Reading
personalized: Yes vs. No 1.89 about 1.9 times the odds
Video vs. Banner 1.84 video ads raise the odds most among formats
Popup vs. Banner 1.35 a smaller gain
Audio vs. Banner 0.51 about half the odds
Desktop vs. Mobile 0.58 lower odds on desktop
Tablet vs. Mobile 0.83 CI 0.59–1.14 includes 1
+1 prior ad click 1.23 past clickers click more
+10 minutes a day 1.11 heavier listeners click more
+10 years of age 0.78 odds fall about 22% per decade
Doubling of liked songs 1.56 bigger libraries, more clicks

Two cautions:

  • Odds are not probabilities. With a base rate near 22%, “1.9 times the odds” is not “1.9 times as likely.” From odds ratios to probabilities makes this concrete.
  • Association, not cause. These are observational data. Listeners who receive personalized offers may differ in ways we did not measure; an A/B test would establish the causal effect.

views_c and I(views_c^2) also appear in the tidy() output, but do not read either one alone: the effect of one more view depends on how many views the listener has already had. Read them together, as a curve.

6.3 The quadratic: find the peak

What the code does. For a curve \(\beta_1 X + \beta_2 X^2\), the slope is \(\beta_1 + 2\beta_2 X\). Setting it to zero gives the turning point \(X = -\beta_1 / (2\beta_2)\). Because we centered at 4, we add 4 back to express the peak in actual views. The code then predicts the probability of a click at every number of views for a typical listener (age 30, 60 minutes a day, 250 liked songs, 2 prior ad clicks, Mobile, banner, not personalized), changing only the number of views.

b    <- coef(final)
peak <- 4 + (-b["views_c"] / (2 * b["I(views_c^2)"]))
peak
#> views_c 
#> 6.41795
curve <- tibble(offer_views_7d = 0:15, views_c = offer_views_7d - 4,
                age = 30, daily_minutes = 60, liked_songs = 250, prior_clicks_90d = 2,
                device       = factor("Mobile", levels = levels(clicks$device)),
                ad_format    = factor("Banner", levels = levels(clicks$ad_format)),
                personalized = factor("No",     levels = levels(clicks$personalized)))
curve$p_click <- predict(final, newdata = curve, type = "response")

ggplot(curve, aes(offer_views_7d, p_click)) +
  geom_line(linewidth = 1) + geom_point(size = 2.5) +
  geom_vline(xintercept = peak, linetype = "dashed") +
  scale_y_continuous(labels = scales::percent) +
  scale_x_continuous(breaks = 0:15) +
  labs(x = "Offer views in the past 7 days", y = "Predicted P(click)",
       title = "The chance of a click peaks, then falls")

How to read it. The peak is at 6.4 views per week, where this listener’s chance of clicking is about 29%. It starts at 3.5% with no views and falls to 6% by 12 views.

Note that offer_views_7d counts views before the current one, so “6 prior views” means this is the 7th.

Business reading: after about 6 prior views, each extra view lowers the chance of a click. That is not the same as “stop at 6”: by the cost rule (see Choose the cutoff), a view still pays as long as the predicted probability stays above the cost-based cutoff (about 0.12, computed in Minimize expected cost), which for this listener holds through about 10 prior views. The peak and the cutoff therefore bracket a sensible weekly cap — somewhere between about 6 and 10 views — that a frequency-cap experiment should settle. Listeners with 0–2 prior views also score below the cutoff, but nobody reaches a third view without a first, and the curve is an association, not a dose-response experiment.

Two further cautions: few training listeners had 12 or more views (71), so the right end of the curve rests on little data; and a quadratic forces the curve to be symmetric around its peak (a spline, mgcv::gam, could check that). A frequency-cap experiment would confirm the cap before Spotify relies on it.

6.4 From odds ratios to probabilities

What the code does. expand_grid() builds every combination of three ad formats and the two personalization settings for the typical listener (4 views, so views_c = 0). predict(..., type = "response") returns probabilities; without type = "response", predict() returns log-odds. Every variable in newdata must match the model, including views_c.

The confidence intervals are built on the log-odds scale and then converted with plogis(), which keeps them between 0 and 1.

profiles <- expand_grid(
  age = 30, daily_minutes = 60, liked_songs = 250, prior_clicks_90d = 2,
  views_c = 0,                                   # 4 offer views
  device       = factor("Mobile", levels = levels(clicks$device)),
  ad_format    = factor(c("Banner", "Video", "Audio"), levels = levels(clicks$ad_format)),
  personalized = factor(c("No", "Yes"), levels = levels(clicks$personalized)))

pr <- predict(final, newdata = profiles, type = "link", se.fit = TRUE)
profiles <- profiles |>
  mutate(p_click = plogis(pr$fit),
         lower   = plogis(pr$fit - 1.96 * pr$se.fit),
         upper   = plogis(pr$fit + 1.96 * pr$se.fit))

profiles |>
  select(ad_format, personalized, p_click, lower, upper) |>
  mutate(across(where(is.numeric), \(x) round(x, 3))) |>
  kable()
ad_format personalized p_click lower upper
Banner No 0.227 0.190 0.268
Banner Yes 0.356 0.307 0.409
Video No 0.350 0.302 0.402
Video Yes 0.505 0.447 0.562
Audio No 0.129 0.096 0.172
Audio Yes 0.219 0.168 0.280

How to read it. Personalization multiplies the odds by the same 1.89 for every format (there is no interaction in the model), yet it adds different amounts of probability:

  • Audio: 12.9% → 21.9% (+9.0 points)
  • Banner: 22.7% → 35.6% (+13.0 points)
  • Video: 35.0% → 50.5% (+15.4 points)

The logistic curve is steepest near 50%, so the same change in odds moves the probability more when the starting probability is closer to the middle. This is the practical meaning of “the model is linear in the log-odds, not in the probability.”

7 Validate and test

Validation happens in two stages. In-sample: does the model fit the data it was built on, and does any single listener drive it? Out-of-sample: does it still work on listeners it has never seen? The cutoff is chosen on the training data and scored once on the test data.

7.1 Goodness of fit: the Hosmer–Lemeshow test

What the code does. hoslem.test() sorts listeners by predicted probability, cuts them into 10 groups, and compares the observed number of clicks in each group with the number the model expected (the sum of the predicted probabilities). The test statistic is a Pearson χ² with \(g - 2 = 8\) degrees of freedom.

hl <- hoslem.test(as.numeric(train$clicked == "1"), fitted(final), g = 10)
hl
#> 
#>  Hosmer and Lemeshow goodness of fit (GOF) test
#> 
#> data:  as.numeric(train$clicked == "1"), fitted(final)
#> X-squared = 5.9216, df = 8, p-value = 0.656
hl_tab <- tibble(group    = 1:10,
                 observed = as.numeric(hl$observed[, "y1"]),
                 expected = round(as.numeric(hl$expected[, "yhat1"]), 1))
kable(hl_tab)
group observed expected
1 4 6.3
2 11 14.4
3 28 22.5
4 38 32.1
5 37 42.3
6 57 52.9
7 64 66.9
8 89 87.4
9 111 114.4
10 168 167.9
hl_tab |>
  pivot_longer(c(observed, expected), names_to = "type", values_to = "clicks") |>
  mutate(type = factor(type, levels = c("observed", "expected"))) |>
  ggplot(aes(factor(group), clicks, fill = type)) +
  geom_col(position = position_dodge(width = 0.8), width = 0.75) +
  scale_fill_manual(values = c(expected = "#D9772B", observed = "#0E7FA8")) +
  labs(x = "Risk group (1 = lowest predicted probability)", y = "Clicks", fill = NULL)

How to read it. Here, unusually, we want a non-significant result: p > .05 means there is no evidence of misfit (not proof that the model is right). The test gives χ²(8) = 5.92, p = .656: no evidence of lack of fit. The table shows observed and expected clicks track closely in every group, from the lowest-risk listeners to the highest.

This is a calibration check — when the model says 30%, about 30% click. It matters for the 1/7 theory check in Choose the cutoff, which assumes calibrated probabilities. Note that this is calibration on the training data; calibration on new data is not tested here. The HL test has low power and depends on the number of groups, so a pass is necessary but not sufficient; read the residuals too.

7.2 Residuals and influence

What the code does. augment() adds each listener’s deviance residual and fitted value to a data frame; cooks.distance() measures how much the whole fit would change without that listener; dfbetas() measures how much each coefficient would change, in standard-error units.

diag <- augment(final, type.residuals = "deviance") |>
  mutate(cooks = cooks.distance(final))

mean(abs(diag$.resid) > 2)                        # share of |deviance residual| > 2
#> [1] 0.02963227
c(max_cooks = max(diag$cooks), cutoff_4n = 4 / nrow(train))
#>   max_cooks   cutoff_4n 
#> 0.010988919 0.001428061
max(abs(dfbetas(final)))                          # compare with 1
#> [1] 0.2279674
ggplot(diag, aes(.fitted, .resid)) +
  geom_point(alpha = 0.3) +
  geom_hline(yintercept = c(-2, 0, 2), linetype = c(2, 1, 2)) +
  labs(x = "Fitted log-odds", y = "Deviance residual")

influencePlot(final)

#>         StudRes          Hat        CookD
#> 124  -0.8073037 0.0235435325 0.0007197332
#> 191   2.8827245 0.0007062817 0.0033398312
#> 489  -1.2898599 0.0243018063 0.0024777104
#> 1441  2.1036967 0.0183029397 0.0109889187
#> 1625  2.2470419 0.0068140952 0.0058571591
#> 1744  2.8652336 0.0008160891 0.0036598753

How to read it.

  • Deviance residuals: 3.0% exceed 2 in absolute value. With a 0/1 outcome these residuals are not normal, so there is no fixed share to expect: a clicker’s residual exceeds 2 only when its predicted probability is below about .135 (\(e^{-2}\)), so these are mostly clickers the model rated unlikely. The two bands in the plot are normal for a 0/1 outcome (clickers above, non-clickers below); what matters is that there is no trend or curve.
  • Cook’s distance: the largest is 0.011, far below 1. The 4/n rule (0.0014) flags many harmless points when n is in the thousands, so judge against 1.
  • DFBETAS: the largest is 0.23, far below 1 (a full standard error).
  • Influence plot: no listener combines high leverage with a large residual.

Conclusion: no single listener drives the results; keep everyone.

7.3 ROC curve and AUC: training versus test

What the code does. predict() scores every listener in the training and test sets. roc() builds the ROC curve, which traces sensitivity against 1 − specificity at every possible cutoff. levels and direction are set explicitly so pROC does not have to guess which class is the event. The AUC is the probability that the model ranks a random clicker above a random non-clicker. Comparing training and test AUC shows how optimistic the in-sample numbers were.

p_train <- predict(final, type = "response")
p_test  <- predict(final, newdata = test, type = "response")

roc_train <- roc(train$clicked, p_train, levels = c("0", "1"), direction = "<")
roc_test  <- roc(test$clicked,  p_test,  levels = c("0", "1"), direction = "<")

auc(roc_train)
#> Area under the curve: 0.7757
auc(roc_test)
#> Area under the curve: 0.7749
ci.auc(roc_test)
#> 95% CI: 0.744-0.8058 (DeLong)
plot(roc_test, legacy.axes = TRUE, print.auc = TRUE)

How to read it. Training AUC = 0.776; test AUC = 0.775 (95% CI 0.744–0.806). On the lecture’s rubric (.90+ excellent, .80 good, .70 fair, .60 poor) this is fair, near the top of the band. The two AUCs are almost identical, so there is very little optimism — expected for a model with few parameters and many events per parameter.

For comparison, the model without the squared term (m3) has a test AUC of 0.717: the curve improves predictions on new listeners, not just the fit statistics.

AUC summarizes ranking across all cutoffs. It says nothing about accuracy at any one cutoff — that is what the confusion matrix is for.

7.4 Choose the cutoff — on the training data

A probability becomes a decision only when we choose a cutoff: show the offer to everyone whose predicted probability is at or above it. The default of 0.50 is almost never right when the outcome is unbalanced. We compare two principled choices, both chosen on the training data.

7.4.1 Youden’s J

What the code does. coords(..., "best", best.method = "youden") finds the cutoff that maximizes sensitivity + specificity − 1: the point on the ROC curve farthest above the diagonal.

cut_youden <- coords(roc_train, "best", best.method = "youden",
                     ret = c("threshold", "sensitivity", "specificity"))
cut_youden
#>   threshold sensitivity specificity
#> 1 0.2185061   0.7001647    0.706928

How to read it. Youden’s cutoff is 0.219 — close to the base rate, not 0.50. It treats a missed clicker and a wasted offer as equally costly, which is not true here.

7.4.2 Minimize expected cost

What the code does. coords(..., x = "all") returns the number of false negatives and false positives at every possible cutoff. We put a dollar value on each, add them up, and pick the cutoff with the lowest total cost.

cost_FN <- 6     # $ expected profit lost per missed clicker
cost_FP <- 1     # $ ad-slot revenue lost per listener who never clicks

co <- coords(roc_train, x = "all", ret = c("threshold", "fn", "fp"))
co$cost <- co$fn * cost_FN + co$fp * cost_FP
cut_cost <- co[which.min(co$cost), ]
cut_cost
#>      threshold fn   fp cost
#> 1018 0.1187064 58 1235 1583

How to read it. The cost-minimizing cutoff is 0.119, lower than Youden’s. When a missed clicker costs six times a wasted offer, it pays to show the offer to more listeners: more sensitivity, less specificity. As a check, for well-calibrated probabilities the cost-optimal cutoff is about cost_FP / (cost_FP + cost_FN) = 1/7 = 0.143 — the same neighborhood. They differ because the cost curve is flat near its minimum and the search uses one sample’s counts.

If a missed clicker were worth only $2, the theoretical cutoff would rise to 1/3: Spotify would target fewer, more likely listeners. The cutoff is a business decision, informed by statistics.

7.5 The confusion matrix — on the test set

What the code does. It classifies each test listener as a predicted clicker (1) or not (0) at each cutoff and compares the predictions with what actually happened. confusionMatrix() needs two factors with the same levels and the positive = "1" argument, which sets clicking as the event.

pred_cut <- factor(as.integer(p_test >= cut_cost$threshold), levels = c(0, 1))
confusionMatrix(pred_cut, test$clicked, positive = "1")
#> Confusion Matrix and Statistics
#> 
#>           Reference
#> Prediction   0   1
#>          0 418  28
#>          1 521 232
#>                                           
#>                Accuracy : 0.5421          
#>                  95% CI : (0.5134, 0.5706)
#>     No Information Rate : 0.7832          
#>     P-Value [Acc > NIR] : 1               
#>                                           
#>                   Kappa : 0.2002          
#>                                           
#>  Mcnemar's Test P-Value : <2e-16          
#>                                           
#>             Sensitivity : 0.8923          
#>             Specificity : 0.4452          
#>          Pos Pred Value : 0.3081          
#>          Neg Pred Value : 0.9372          
#>              Prevalence : 0.2168          
#>          Detection Rate : 0.1935          
#>    Detection Prevalence : 0.6280          
#>       Balanced Accuracy : 0.6687          
#>                                           
#>        'Positive' Class : 1               
#> 

The same for Youden’s cutoff and the habitual 0.50, collected into one comparison table:

score <- function(cut, label) {
  pred <- factor(as.integer(p_test >= cut), levels = c(0, 1))
  cm   <- confusionMatrix(pred, test$clicked, positive = "1")
  t    <- cm$table
  tibble(cutoff = label, threshold = round(cut, 3),
         TP = t["1", "1"], FN = t["0", "1"], FP = t["1", "0"], TN = t["0", "0"],
         accuracy    = round(cm$overall[["Accuracy"]], 3),
         sensitivity = round(cm$byClass[["Sensitivity"]], 3),
         specificity = round(cm$byClass[["Specificity"]], 3),
         precision   = round(cm$byClass[["Pos Pred Value"]], 3),
         cost        = t["0", "1"] * cost_FN + t["1", "0"] * cost_FP)
}
results <- bind_rows(score(cut_cost$threshold, "Cost-based"),
                     score(cut_youden$threshold, "Youden"),
                     score(0.50, "Default 0.50"))
kable(results)
cutoff threshold TP FN FP TN accuracy sensitivity specificity precision cost
Cost-based 0.119 232 28 521 418 0.542 0.892 0.445 0.308 689
Youden 0.219 187 73 301 638 0.688 0.719 0.679 0.383 739
Default 0.50 0.500 69 191 46 893 0.802 0.265 0.951 0.600 1192

How to read it.

  • Read down the columns (actual outcome): sensitivity is the share of actual clickers we targeted; specificity is the share of non-clickers we skipped.
  • Read across the rows (prediction): precision is the share of targeted listeners who actually clicked.
  • The cost-based cutoff catches 232 of the 260 clickers (sensitivity 0.892) and costs $689 on the test set — the lowest of the three — even though its accuracy (0.542) is the lowest.
  • The 0.50 cutoff has the highest accuracy (0.802), barely above the no-information rate of 0.783 (the accuracy of predicting “no one clicks”), but it finds only 69 clickers and costs $1,192.

This is the accuracy paradox: the most accurate cutoff is the worst business decision. Report the metrics that match the decision, not the one that looks best. A precision of about 31% means roughly 1 in 3 targeted listeners clicks, against about 22% if Spotify showed the offer to everyone.

8 Recommendations and limitations

What the model says (associations, in this simulated scenario):

  • Format: video ads and pop-ups beat banners; audio ads roughly halve the odds of a click.
  • Targeting: personalized offers nearly double the odds.
  • Listeners: past ad clickers, heavier daily listeners, listeners with bigger libraries and younger listeners are more likely to click; desktop listeners are less likely than mobile listeners.
  • Frequency: the chance of a click peaks at about 6 prior views a week, then falls.

What Spotify should do:

  1. Score every Free listener and show the offer to those above the cost-based cutoff (about 0.12).
  2. Where the format can be chosen, prefer video and pop-up over audio; personalize the offer.
  3. Test a weekly frequency cap between about 6 views (where returns start to fall) and 10 (where the typical listener drops below the cost cutoff).
  4. Run A/B tests of personalization and of the frequency cap to confirm the causal effects, and re-estimate the costs and re-tune the cutoff each campaign.

Limitations:

  • Observational data: personalization and the number of views may be confounded with which listeners the delivery system chose.
  • Possible clustering by ad creative that the file cannot reveal.
  • Few listeners above 12 views, so the right end of the curve rests on little data.
  • AUC of about 0.77: useful for ranking listeners, not for confident individual predictions.
  • The $6 and $1 costs drive the cutoff; they should come from the business, not the analyst.

9 Key takeaways

  1. Split first. Tune on the training data; score once on the test data.
  2. Collinear predictors hide each other. Check VIF, then refit.
  3. Check linearity in the logit. Use a log for a curve that flattens; use a centered squared term for a curve that turns.
  4. Validate in-sample (Hosmer–Lemeshow, residuals, influence), then test on new data (ROC, AUC, confusion matrix).
  5. Choose the cutoff by cost and judge it with the confusion matrix, not raw accuracy.

10 References

Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016). Sample size considerations for the external validation of a multivariable prognostic model: A resampling study. Statistics in Medicine, 35(2), 214–226. https://doi.org/10.1002/sim.6787

Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). Wiley.

Steyerberg, E. W., Harrell, F. E., Jr., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001). Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology, 54(8), 774–781.

11 Session information

sessionInfo()
#> R version 4.3.3 (2024-02-29)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 24.04.5 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0 
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0
#> 
#> locale:
#> [1] C
#> 
#> time zone: America/Chicago
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] knitr_1.45                   ResourceSelection_0.0.0.9000
#>  [3] caret_6.0-94                 lattice_0.22-5              
#>  [5] pROC_1.18.5                  lmtest_0.9-40               
#>  [7] zoo_1.8-12                   car_3.1-2                   
#>  [9] carData_3.0-5                broom_1.0.5                 
#> [11] lubridate_1.9.3              forcats_1.0.0               
#> [13] stringr_1.5.1                dplyr_1.1.4                 
#> [15] purrr_1.0.2                  readr_2.1.5                 
#> [17] tidyr_1.3.1                  tibble_3.2.1                
#> [19] ggplot2_3.4.4                tidyverse_2.0.0             
#> 
#> loaded via a namespace (and not attached):
#>  [1] tidyselect_1.2.0     timeDate_4032.109    farver_2.1.1        
#>  [4] fastmap_1.1.1        digest_0.6.34        rpart_4.1.23        
#>  [7] timechange_0.3.0     lifecycle_1.0.4      ellipsis_0.3.2      
#> [10] survival_3.5-8       magrittr_2.0.3       compiler_4.3.3      
#> [13] rlang_1.1.3          sass_0.4.8           tools_4.3.3         
#> [16] utf8_1.2.4           yaml_2.3.8           data.table_1.14.10  
#> [19] labeling_0.4.3       bit_4.0.5            plyr_1.8.9          
#> [22] abind_1.4-5          withr_2.5.0          nnet_7.3-19         
#> [25] grid_4.3.3           stats4_4.3.3         fansi_1.0.5         
#> [28] e1071_1.7-14         colorspace_2.1-0     future_1.33.1       
#> [31] globals_0.16.2       scales_1.3.0         iterators_1.0.14    
#> [34] MASS_7.3-60.0.1      cli_3.6.2            crayon_1.5.2        
#> [37] rmarkdown_2.25       generics_0.1.3       future.apply_1.11.1 
#> [40] reshape2_1.4.4       tzdb_0.4.0           proxy_0.4-27        
#> [43] cachem_1.0.8         splines_4.3.3        parallel_4.3.3      
#> [46] vctrs_0.6.5          hardhat_1.3.1        Matrix_1.6-5        
#> [49] jsonlite_1.8.8       hms_1.1.3            bit64_4.0.5         
#> [52] listenv_0.9.1        foreach_1.5.2        gower_1.0.1         
#> [55] jquerylib_0.1.4      recipes_1.0.9        glue_1.7.0          
#> [58] parallelly_1.37.1    codetools_0.2-19     stringi_1.8.3       
#> [61] gtable_0.3.4         munsell_0.5.0        pillar_1.9.0        
#> [64] htmltools_0.5.7      ipred_0.9-14         lava_1.7.3          
#> [67] R6_2.5.1             vroom_1.6.5          evaluate_0.23       
#> [70] highr_0.10           backports_1.4.1      bslib_0.6.1         
#> [73] class_7.3-22         Rcpp_1.0.12          nlme_3.1-164        
#> [76] prodlim_2023.08.28   mgcv_1.9-1           xfun_0.41           
#> [79] pkgconfig_2.0.3      ModelMetrics_1.2.2.2