About this case. This is a hypothetical scenario. Spotify is used only because it is a familiar name. The offer, the costs and every row of data are simulated for class; nothing here reflects Spotify’s actual offers, data or results, and the course is not affiliated with Spotify.
This walkthrough follows the same five sections as the live-session slides:
Each step has three parts: what the code does, the code, and how to read the output. Numbers in the text are computed by R as the document knits, so the explanations always match the output.
Before knitting, install the packages once:
install.packages(c("tidyverse", "broom", "car", "lmtest", "pROC", "caret",
"ResourceSelection", "knitr"))library(tidyverse) # data wrangling (dplyr) and plots (ggplot2)
library(broom) # tidy(), augment(): model output as data frames
library(car) # vif(), Anova(), influencePlot()
library(lmtest) # lrtest(): likelihood-ratio tests
library(pROC) # roc(), auc(), coords(): ROC curves and cutoffs
library(caret) # createDataPartition(), confusionMatrix()
library(ResourceSelection) # hoslem.test(): Hosmer-Lemeshow goodness of fit
library(knitr) # kable(): tables in this document
# small helpers for writing numbers in the text
f <- function(x, d = 2) formatC(x, digits = d, format = "f", big.mark = ",")
pct <- function(x, d = 1) paste0(formatC(100 * x, digits = d, format = "f"), "%")
p_txt <- function(p) if (p < .001) "p < .001" else paste0("p = ", sub("^0", "", formatC(p, digits = 3, format = "f")))In the scenario, Spotify shows Free listeners an in-app offer: “Try Premium free for one month.” The growth team wants to answer three questions:
The outcome is binary — the listener either tapped “Try Premium” (1) or did not (0) — so ordinary least-squares regression is the wrong tool: it can predict “probabilities” below 0 or above 1, and its errors cannot be normal. Logistic regression models the log-odds of a click as a linear function of the predictors, which keeps every predicted probability between 0 and 1.
The two kinds of mistakes do not cost the same:
These costs are illustrative assumptions; they matter in Validate and test, when we choose the cutoff.
| Variable | Type | Meaning |
|---|---|---|
clicked |
outcome | 1 = tapped “Try Premium”, 0 = did not |
age |
continuous | listener’s age in years (18–64) |
daily_minutes |
continuous | average minutes of listening per day |
weekly_minutes |
continuous | total minutes of listening last week |
liked_songs |
continuous | songs in the listener’s Liked Songs library |
prior_clicks_90d |
count | ads (any advertiser) the listener clicked in the past 90 days |
offer_views_7d |
count | times the listener saw the Premium offer in the past 7 days |
device |
categorical | Mobile, Desktop, Tablet |
ad_format |
categorical | Banner, Video, Popup, Audio |
personalized |
categorical | was the offer personalized to the listener (No, Yes)? |
Because the data are simulated, every assumption check has a “right answer” for you to discover. Two are built in on purpose: one pair of predictors that measure the same thing, and two predictors whose effects are curved.
What the code does. read_csv() reads
the file. mutate() then turns the text columns into
factors (R’s categorical type) and fixes the order of their
levels. The first level of each factor becomes the
reference group: every coefficient for that factor
compares a level with the reference.
Why set the levels yourself? By default R sorts
levels alphabetically, which would make “Audio” the reference for
ad_format. Choosing the most common or most natural group
(Mobile, Banner, No) makes each odds ratio answer a sensible question,
such as “how does a video compare with a banner?”
clicked must also be a factor with levels
0, 1, so that glm() models the probability of
1 (a click) and caret treats 1 as
the “positive” class.
clicks <- read_csv("PremiumOffers.csv") |>
mutate(
clicked = factor(clicked, levels = c(0, 1)),
device = factor(device, levels = c("Mobile", "Desktop", "Tablet")),
ad_format = factor(ad_format, levels = c("Banner", "Video", "Popup", "Audio")),
personalized = factor(personalized, levels = c("No", "Yes"))
)
glimpse(clicks)#> Rows: 4,000
#> Columns: 11
#> $ user_id <chr> "U35057", "U18542", "U91857", "U24844", "U34576", "U4~
#> $ clicked <fct> 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0,~
#> $ age <dbl> 33, 28, 39, 18, 18, 42, 39, 21, 43, 30, 35, 30, 35, 2~
#> $ daily_minutes <dbl> 55.0, 88.7, 27.8, 35.5, 113.9, 35.9, 73.2, 80.2, 115.~
#> $ weekly_minutes <dbl> 411, 563, 204, 252, 738, 223, 498, 546, 842, 529, 167~
#> $ liked_songs <dbl> 1124, 187, 968, 137, 27, 731, 758, 642, 55, 601, 539,~
#> $ prior_clicks_90d <dbl> 2, 1, 0, 2, 2, 2, 2, 1, 3, 1, 1, 1, 0, 2, 3, 0, 0, 3,~
#> $ offer_views_7d <dbl> 4, 3, 0, 8, 3, 6, 3, 2, 10, 8, 2, 0, 4, 2, 5, 4, 6, 0~
#> $ device <fct> Mobile, Tablet, Desktop, Mobile, Mobile, Desktop, Mob~
#> $ ad_format <fct> Banner, Banner, Video, Audio, Video, Banner, Video, V~
#> $ personalized <fct> No, Yes, No, No, No, No, Yes, No, No, No, Yes, Yes, N~
#> # A tibble: 2 x 3
#> clicked n pct
#> <fct> <int> <dbl>
#> 1 0 3133 0.7832
#> 2 1 867 0.2168
How to read it. 867 of 4,000 listeners clicked — a base rate of 21.7%. A “model” that predicts no one clicks would therefore be right 78.3% of the time. Remember this number: in Validate and test it is the bar that raw accuracy must clear (the no-information rate), and it is why accuracy alone can mislead.
#> age daily_minutes weekly_minutes liked_songs
#> Min. :18.00 Min. : 4.60 Min. : 28.0 Min. : 12.0
#> 1st Qu.:26.00 1st Qu.: 38.20 1st Qu.: 266.0 1st Qu.: 136.0
#> Median :34.00 Median : 55.30 Median : 382.0 Median : 254.0
#> Mean :34.32 Mean : 59.67 Mean : 417.9 Mean : 382.4
#> 3rd Qu.:41.00 3rd Qu.: 75.60 3rd Qu.: 532.0 3rd Qu.: 473.2
#> Max. :64.00 Max. :240.00 Max. :1601.0 Max. :5000.0
#> prior_clicks_90d offer_views_7d
#> Min. : 0.000 Min. : 0.00
#> 1st Qu.: 1.000 1st Qu.: 2.00
#> Median : 2.000 Median : 3.00
#> Mean : 1.806 Mean : 4.02
#> 3rd Qu.: 3.000 3rd Qu.: 6.00
#> Max. :10.000 Max. :15.00
Notice that liked_songs is strongly
right-skewed (the mean is well above the median) and
that prior_clicks_90d and offer_views_7d are
counts with many zeros. Both facts matter when we check
linearity.
What the code does.
createDataPartition() randomly selects 70% of the rows for
training and leaves 30% for testing.
It stratifies on clicked, so both sets keep the
same click rate. list = FALSE returns plain row numbers
that we can use to index the data. set.seed() makes the
random split reproducible.
Why split first? Every decision we make — which variable to drop, which to transform, where to put the cutoff — is a choice fitted to the data. If we make those choices and judge the model on the same rows, the performance numbers are optimistic. The test set is set aside now and used once, at the very end.
set.seed(2026)
idx <- createDataPartition(clicks$clicked, p = 0.70, list = FALSE)
train <- clicks[idx, ]
test <- clicks[-idx, ]
c(train = nrow(train), test = nrow(test))#> train test
#> 2801 1199
#> train_rate test_rate
#> 0.2167083 0.2168474
How do you choose the percentages? There is no single correct split; 70/30 and 80/20 are conventions. Think in counts, not percentages:
With very large data a small percentage is enough for testing; with small data, a single split wastes information and cross-validation or the bootstrap is better (Steyerberg et al., 2001).
What the code does. glm() fits a
generalized linear model.
family = binomial(link = "logit") makes it a logistic
regression: the model is
\[\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k,\]
where \(p\) is the probability of a click. We start with every candidate predictor: six continuous (or count) predictors and three categorical ones. R turns each factor into dummy variables automatically — one for every level except the reference.
m_full <- glm(clicked ~ age + daily_minutes + weekly_minutes +
liked_songs + prior_clicks_90d + offer_views_7d +
device + ad_format + personalized,
data = train, family = binomial(link = "logit"))
summary(m_full)#>
#> Call:
#> glm(formula = clicked ~ age + daily_minutes + weekly_minutes +
#> liked_songs + prior_clicks_90d + offer_views_7d + device +
#> ad_format + personalized, family = binomial(link = "logit"),
#> data = train)
#>
#> Coefficients:
#> Estimate Std. Error z value Pr(>|z|)
#> (Intercept) -2.6661794 0.2402893 -11.096 < 2e-16 ***
#> age -0.0227319 0.0048941 -4.645 3.41e-06 ***
#> daily_minutes -0.0008004 0.0114048 -0.070 0.944048
#> weekly_minutes 0.0017079 0.0016066 1.063 0.287758
#> liked_songs 0.0009731 0.0001104 8.815 < 2e-16 ***
#> prior_clicks_90d 0.1706092 0.0350109 4.873 1.10e-06 ***
#> offer_views_7d 0.0954210 0.0152833 6.243 4.28e-10 ***
#> deviceDesktop -0.5480778 0.1284189 -4.268 1.97e-05 ***
#> deviceTablet -0.1952428 0.1627807 -1.199 0.230364
#> ad_formatVideo 0.5871662 0.1191238 4.929 8.26e-07 ***
#> ad_formatPopup 0.2920276 0.1374857 2.124 0.033665 *
#> ad_formatAudio -0.5854725 0.1726263 -3.392 0.000695 ***
#> personalizedYes 0.5862886 0.0985796 5.947 2.73e-09 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for binomial family taken to be 1)
#>
#> Null deviance: 2928.2 on 2800 degrees of freedom
#> Residual deviance: 2610.1 on 2788 degrees of freedom
#> AIC: 2636.1
#>
#> Number of Fisher Scoring iterations: 5
How to read it.
ad_formatVideo) it is the
difference in log-odds between that level and the reference
(Banner).device dummies, 3 ad_format dummies and 1
personalized dummy.Something is wrong. Listening time should matter,
yet daily_minutes (p = .944) and
weekly_minutes (p = .288) are both
non-significant, and daily_minutes even has a negative
sign. Note the standard error of daily_minutes, 0.0114; we
will see it shrink once the problem is fixed. That is the classic
symptom of multicollinearity: two predictors that
measure the same thing hide each other.
Also note that offer_views_7d has a
positive slope (0.095): read literally, “every extra
view of the offer raises the odds of a click.” Hold that thought — the
linearity check shows it is wrong.
Each check ends in one of three words: pass, fix, or limitation.
What the code does. It counts the events (clicks) in the training data and the number of estimated parameters (coefficients, not counting the intercept), then divides. It also cross-tabulates every categorical predictor against the outcome.
events <- sum(train$clicked == "1")
k <- length(coef(m_full)) - 1 # parameters, not variables
c(events = events, parameters = k, EPV = round(events / k, 1))#> events parameters EPV
#> 607.0 12.0 50.6
#>
#> 0 1
#> Mobile 1418 444
#> Desktop 542 101
#> Tablet 234 62
#>
#> 0 1
#> Banner 775 187
#> Video 609 235
#> Popup 415 129
#> Audio 395 56
#>
#> 0 1
#> No 1292 286
#> Yes 902 321
How to read it.
ad_format is one
variable but uses 3 parameters.What the code does. vif() computes a
variance inflation factor for each predictor: how much
the variance of its coefficient is inflated because it overlaps with the
other predictors. When a model contains factors, car
reports a generalized VIF (GVIF). For 1-df terms GVIF equals
VIF; for a factor, square the last column, GVIF^(1/(2*Df)),
before comparing it with the usual rubric. The correlation matrix shows
which predictors overlap.
The rubric. About 1: none. 1–5: usually fine. 5–10: concerning. Above 10: severe.
#> GVIF Df GVIF^(1/(2*Df))
#> age 1.010005 1 1.004990
#> daily_minutes 51.451696 1 7.172984
#> weekly_minutes 51.425073 1 7.171128
#> liked_songs 1.020608 1 1.010252
#> prior_clicks_90d 1.042406 1 1.020983
#> offer_views_7d 1.014522 1 1.007235
#> device 1.010466 2 1.002606
#> ad_format 1.025460 3 1.004199
#> personalized 1.017413 1 1.008669
train |>
select(age, daily_minutes, weekly_minutes, liked_songs,
prior_clicks_90d, offer_views_7d) |>
cor() |>
round(2)#> age daily_minutes weekly_minutes liked_songs
#> age 1.00 -0.02 -0.02 0.01
#> daily_minutes -0.02 1.00 0.99 0.00
#> weekly_minutes -0.02 0.99 1.00 0.00
#> liked_songs 0.01 0.00 0.00 1.00
#> prior_clicks_90d 0.01 0.19 0.19 -0.01
#> offer_views_7d 0.02 0.01 0.01 -0.02
#> prior_clicks_90d offer_views_7d
#> age 0.01 0.02
#> daily_minutes 0.19 0.01
#> weekly_minutes 0.19 0.01
#> liked_songs -0.01 -0.02
#> prior_clicks_90d 1.00 -0.02
#> offer_views_7d -0.02 1.00
How to read it. daily_minutes (GVIF =
51.45) and weekly_minutes (GVIF = 51.43) are
severe. Their correlation is r = 0.99 — weekly minutes
is essentially 7 × daily minutes. Every other term is close to 1.
Multicollinearity does not bias coefficients; it inflates their variance. The square root of the VIF is roughly how much the standard error is inflated: √51 ≈ 7.2.
What the code does. update() refits an
existing model without retyping the formula. In . ~ ., the
left dot means “the same outcome” and the right dot means “the same
predictors”; - weekly_minutes removes one term. We keep
daily_minutes because a manager can picture “minutes per
day.”
#> GVIF Df GVIF^(1/(2*Df))
#> age 1.009929 1 1.004952
#> daily_minutes 1.038679 1 1.019156
#> liked_songs 1.020505 1 1.010200
#> prior_clicks_90d 1.042443 1 1.021001
#> offer_views_7d 1.014506 1 1.007227
#> device 1.009743 2 1.002427
#> ad_format 1.023207 3 1.003831
#> personalized 1.017574 1 1.008749
#> Estimate Std. Error z value Pr(>|z|)
#> 1.119530e-02 1.621340e-03 6.904969e+00 5.021424e-12
How to read it. Every VIF is now close to 1.
Pass. daily_minutes now has a positive
coefficient (0.0112), its standard error fell from 0.0114 to 0.0016 —
about 7 times smaller, matching the √VIF above — and it is clearly
significant (p < .001). The variable mattered all along; its twin was
hiding it.
Lesson. Never drop a variable just because it is non-significant in a model with collinear predictors. Other fixes include combining the two into one measure or using regularization.
Logistic regression assumes that each continuous predictor has a straight-line relationship with the log-odds of the outcome (not with the probability). Categorical predictors meet this automatically because each level gets its own dummy.
What the code does. It splits listeners into 10
equal-sized groups by the predictor (ntile()), finds the
middle value of the predictor and the share who clicked in each group,
converts that share to log-odds with \(\log(p
/ (1 - p))\), and plots the result. If the relationship is linear
in the logit, the points follow a straight line.
binned <- train |>
mutate(group = ntile(liked_songs, 10)) |> # 10 equal-sized groups
group_by(group) |>
summarise(typical_x = median(liked_songs), # middle value of each group
click_rate = mean(clicked == "1")) |> # share who clicked
mutate(log_odds = log(click_rate / (1 - click_rate)))
binned#> # A tibble: 10 x 4
#> group typical_x click_rate log_odds
#> <int> <dbl> <dbl> <dbl>
#> 1 1 59 0.09253 -2.283
#> 2 2 99 0.1607 -1.653
#> 3 3 137.5 0.1643 -1.627
#> 4 4 178 0.175 -1.551
#> 5 5 227 0.2107 -1.321
#> 6 6 284 0.2036 -1.364
#> 7 7 360.5 0.2071 -1.342
#> 8 8 478 0.275 -0.9694
#> 9 9 662.5 0.3143 -0.7802
#> 10 10 1064 0.3643 -0.5568
ggplot(binned, aes(typical_x, log_odds)) +
geom_point(size = 3) +
geom_smooth(method = "lm", se = FALSE) +
labs(x = "Liked songs (middle value of each group)", y = "Log-odds of a click",
title = "liked_songs: rises fast, then flattens")How to read it. The log-odds climb quickly across the first few groups and then rise more slowly. The points form a concave curve — the shape of a logarithm — and the straight line misses both ends. This is a warning sign.
The same steps for age and daily_minutes
(change the variable name in ntile() and
median()):
binned_age <- train |>
mutate(group = ntile(age, 10)) |>
group_by(group) |>
summarise(typical_x = median(age), click_rate = mean(clicked == "1")) |>
mutate(log_odds = log(click_rate / (1 - click_rate)))
binned_min <- train |>
mutate(group = ntile(daily_minutes, 10)) |>
group_by(group) |>
summarise(typical_x = median(daily_minutes), click_rate = mean(clicked == "1")) |>
mutate(log_odds = log(click_rate / (1 - click_rate)))
bind_rows(age = binned_age, daily_minutes = binned_min, .id = "predictor") |>
ggplot(aes(typical_x, log_odds)) +
geom_point(size = 2.5) +
geom_smooth(method = "lm", se = FALSE) +
facet_wrap(~ predictor, scales = "free_x") +
labs(x = "Middle value of each group", y = "Log-odds of a click")Both are noisy but roughly straight: age gently falling,
daily_minutes gently rising.
What the code does. For each strictly positive
continuous predictor \(X\), it adds the
term \(X \cdot \ln(X)\) to the model.
If the relationship is linear in the logit, this extra term adds
nothing; a significant \(X
\cdot \ln(X)\) term means the predictor is not
linear. I() tells R to do the arithmetic — inside a
formula, * would otherwise mean “interaction.”
prior_clicks_90d and offer_views_7d contain
zeros, and \(\ln(0)\) is undefined, so
they are checked visually instead (below).
m_bt <- update(m2, . ~ . + I(age * log(age)) +
I(daily_minutes * log(daily_minutes)) +
I(liked_songs * log(liked_songs)))
bt_rows <- c("I(age * log(age))",
"I(daily_minutes * log(daily_minutes))",
"I(liked_songs * log(liked_songs))")
summary(m_bt)$coefficients[bt_rows, ]#> Estimate Std. Error z value
#> I(age * log(age)) -0.0311623906 0.0289344057 -1.077001
#> I(daily_minutes * log(daily_minutes)) -0.0082691745 0.0052061328 -1.588353
#> I(liked_songs * log(liked_songs)) -0.0008280462 0.0001714362 -4.830055
#> Pr(>|z|)
#> I(age * log(age)) 2.814797e-01
#> I(daily_minutes * log(daily_minutes)) 1.122066e-01
#> I(liked_songs * log(liked_songs)) 1.364952e-06
How to read it. Only the three \(X \cdot \ln(X)\) rows matter.
| Term | p-value | Verdict |
|---|---|---|
| age × ln(age) | p = .281 | linear — pass |
| daily_minutes × ln(daily_minutes) | p = .112 | linear — pass |
| liked_songs × ln(liked_songs) | p < .001 | not linear — fix |
This agrees with the plot. The negative coefficient on the \(X \cdot \ln(X)\) term (-0.00083) is consistent with a curve that flattens as \(X\) grows, which is why a log transformation is the natural fix.
What the code does.
liked_songs for log(liked_songs);
everything else stays the same.log(liked_songs), so
the term is log(liked_songs) * log(log(liked_songs)).m3 <- update(m2, . ~ . - liked_songs + log(liked_songs))
m3_bt <- update(m3, . ~ . + I(log(liked_songs) * log(log(liked_songs))))
summary(m3_bt)$coefficients["I(log(liked_songs) * log(log(liked_songs)))", ]#> Estimate Std. Error z value Pr(>|z|)
#> -0.03088137 0.52081072 -0.05929481 0.95271729
#> df AIC
#> m2 12 2635.225
#> m3 12 2604.495
How to read it. The new Box–Tidwell term is not significant (p = .953): pass. AIC fell from 2,635.2 to 2,604.5 — 30.7 points — with the same number of parameters. The log scale also makes business sense: going from 50 to 100 liked songs says much more about a listener than going from 2,000 to 2,050. The coefficient is now read per doubling of liked songs (see Odds ratios).
Counts with many zeros produce ties when cut into deciles, so for
offer_views_7d we compute the empirical log-odds at
each value instead. The few listeners with 12 or more views are
pooled into one group (“12 or more”).
by_views <- train |>
mutate(views = pmin(offer_views_7d, 12)) |> # 12 = "12 or more"
group_by(views) |>
summarise(listeners = n(),
click_rate = mean(clicked == "1")) |>
mutate(log_odds = log(click_rate / (1 - click_rate)))
by_views#> # A tibble: 13 x 4
#> views listeners click_rate log_odds
#> <dbl> <int> <dbl> <dbl>
#> 1 0 199 0.06030 -2.746
#> 2 1 396 0.09343 -2.272
#> 3 2 445 0.1753 -1.549
#> 4 3 394 0.2360 -1.175
#> 5 4 372 0.2608 -1.042
#> 6 5 263 0.3384 -0.6704
#> 7 6 210 0.3476 -0.6295
#> 8 7 162 0.2654 -1.018
#> 9 8 111 0.3514 -0.6131
#> 10 9 83 0.3012 -0.8416
#> 11 10 56 0.1786 -1.526
#> 12 11 39 0.2308 -1.204
#> 13 12 71 0.02817 -3.541
ggplot(by_views, aes(views, log_odds)) +
geom_point(aes(size = listeners)) + # bigger dot = more listeners
geom_smooth(method = "lm", se = FALSE) +
scale_x_continuous(breaks = 0:12) +
labs(x = "Offer views in the past 7 days (12 = 12 or more)",
y = "Log-odds of a click", size = "Listeners")How to read it. The chance of a click rises from about 6% with no prior views to about 35% in the middle, then falls back to about 3% at 12 or more. This is an inverted U: the first few views build interest, and after that listeners tire of seeing the same offer (offer fatigue). The straight fitted line goes through the middle and misses both ends — and it is why the full model reported a misleading positive slope.
A log cannot fix this: a log is still monotone (it only rises). A curve that changes direction needs a squared (quadratic) term.
prior_clicks_90d, checked the same way, rises steadily
and is close enough to straight:
train |>
mutate(prior = pmin(prior_clicks_90d, 6)) |>
group_by(prior) |>
summarise(listeners = n(), click_rate = mean(clicked == "1")) |>
mutate(log_odds = log(click_rate / (1 - click_rate)))#> # A tibble: 7 x 4
#> prior listeners click_rate log_odds
#> <dbl> <int> <dbl> <dbl>
#> 1 0 479 0.1461 -1.765
#> 2 1 851 0.1904 -1.448
#> 3 2 701 0.2282 -1.218
#> 4 3 457 0.2713 -0.9879
#> 5 4 204 0.2843 -0.9232
#> 6 5 72 0.3056 -0.8210
#> 7 6 37 0.2973 -0.8602
What the code does.
views_c = offer_views_7d - 4. Four is a round number close
to the training mean (4.01). The new column is added to
both train and test —
otherwise predict() on the test set will fail later.m4, replacing offer_views_7d with
views_c + I(views_c^2). I() is required:
inside a formula, ^ has a different meaning, and
I() tells R to square the numbers.train <- train |> mutate(views_c = offer_views_7d - 4)
test <- test |> mutate(views_c = offer_views_7d - 4)
vif(update(m3, . ~ . + I(offer_views_7d^2))) # raw X and X^2#> GVIF Df GVIF^(1/(2*Df))
#> age 1.014388 1 1.007168
#> daily_minutes 1.038956 1 1.019292
#> prior_clicks_90d 1.053408 1 1.026357
#> offer_views_7d 11.267780 1 3.356751
#> device 1.013481 2 1.003353
#> ad_format 1.030272 3 1.004983
#> personalized 1.021515 1 1.010700
#> log(liked_songs) 1.030306 1 1.015040
#> I(offer_views_7d^2) 11.164595 1 3.341346
#> GVIF Df GVIF^(1/(2*Df))
#> age 1.014388 1 1.007168
#> daily_minutes 1.038956 1 1.019292
#> prior_clicks_90d 1.053408 1 1.026357
#> device 1.013481 2 1.003353
#> ad_format 1.030272 3 1.004983
#> personalized 1.021515 1 1.010700
#> log(liked_songs) 1.030306 1 1.015040
#> views_c 1.593725 1 1.262428
#> I(views_c^2) 1.573437 1 1.254367
#> Likelihood ratio test
#>
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + offer_views_7d +
#> device + ad_format + personalized + log(liked_songs)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2)
#> #Df LogLik Df Chisq Pr(>Chisq)
#> 1 12 -1290.2
#> 2 13 -1216.0 1 148.44 < 2.2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> df AIC
#> m3 12 2604.495
#> m4 13 2458.053
#> Estimate Std. Error z value Pr(>|z|)
#> views_c 0.28501523 0.024836696 11.475569 1.750217e-30
#> I(views_c^2) -0.05893736 0.006006997 -9.811452 1.005114e-22
How to read it.
Keep the linear term whenever you include its square (the hierarchy principle). Without it, the model forces the peak to sit exactly at the centering value.
What the code does. It checks that each listener appears only once, that the maximum-likelihood fit converged, and that there are no fitted probabilities of exactly 0 or 1 or runaway coefficients (the signs of perfect separation).
#> [1] TRUE
#> [1] TRUE
#> [1] 0.002042521 0.918038718
#> [1] 5.051972
How to read it. Every listener appears once, the model converged, the fitted probabilities range from 0.002 to 0.918, and the largest coefficient (the intercept) is modest with a small standard error. Pass on both checks.
A limitation to name. If the same ad creative was
shown to many listeners, listeners are clustered within creatives, and
the standard errors would be too small. The file has no creative ID, so
we cannot check this. The fixes would be mixed-effects models
(lme4::glmer) or cluster-robust standard errors.
What the code does. It asks whether personalization
works equally well for every ad format by adding the
personalized × ad_format interaction and comparing the two
models with a nested likelihood-ratio test and AIC.
m_int <- update(m4, . ~ . + personalized:ad_format)
lrtest(m4, m_int) # nested LR test: 3 extra parameters#> Likelihood ratio test
#>
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2) +
#> ad_format:personalized
#> #Df LogLik Df Chisq Pr(>Chisq)
#> 1 13 -1216.0
#> 2 16 -1215.5 3 0.9889 0.8039
#> df AIC
#> m4 13 2458.053
#> m_int 16 2463.064
#> Estimate Std. Error z value Pr(>|z|)
#> ad_formatVideo:personalizedYes -0.003743884 0.2460457 -0.01521621 0.9878597
#> ad_formatPopup:personalizedYes 0.115976712 0.2850875 0.40681098 0.6841468
#> ad_formatAudio:personalizedYes 0.315099397 0.3589492 0.87783837 0.3800314
How to read it. An interaction between a 2-level and a 4-level factor costs (2 − 1) × (4 − 1) = 3 parameters, so the test has 3 degrees of freedom. The result is χ²(3) = 0.99, p = .804, and AIC rises by 5.0 points. There is no evidence that the effect of personalization depends on the format, so we keep the simpler model.
Compare this with the quadratic: both were judged by the same nested LR test, with opposite verdicts. Test interactions you have a business reason to expect; do not search every possible pair.
tibble(
Model = c("m_full", "m2", "m3", "m4", "m_int"),
Change = c("All candidate predictors", "Drop weekly_minutes", "Use log(liked_songs)",
"Add centered offer views squared", "Add personalized x ad_format"),
df = c(attr(logLik(m_full), "df"), attr(logLik(m2), "df"), attr(logLik(m3), "df"),
attr(logLik(m4), "df"), attr(logLik(m_int), "df")),
AIC = round(c(AIC(m_full), AIC(m2), AIC(m3), AIC(m4), AIC(m_int)), 1),
Decision = c("Fix", "Keep", "Keep", "Final", "Reject")
) |>
kable(align = "llrrl")| Model | Change | df | AIC | Decision |
|---|---|---|---|---|
| m_full | All candidate predictors | 13 | 2636.1 | Fix |
| m2 | Drop weekly_minutes | 12 | 2635.2 | Keep |
| m3 | Use log(liked_songs) | 12 | 2604.5 | Keep |
| m4 | Add centered offer views squared | 13 | 2458.1 | Final |
| m_int | Add personalized x ad_format | 16 | 2463.1 | Reject |
df is the number of estimated parameters including the
intercept. Dropping weekly_minutes barely changes AIC — it
fixes the standard errors, not the fit. The large gains come from
modeling the two curves correctly. m4 is the final
model.
#> clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2)
What the code does.
lrtest(final) compares the final model with an
intercept-only model (no predictors). This is the
global likelihood-ratio test.Anova(..., type = "III", test.statistic = "LR") tests
each predictor as a whole — all of its levels together
— given every other predictor.views_c alone
drops the linear term but keeps the square; that only tests whether the
curve is flat at 4 views, and its size depends on where we centered. Do
not read it as evidence about offer views.#> Likelihood ratio test
#>
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2)
#> Model 2: clicked ~ 1
#> #Df LogLik Df Chisq Pr(>Chisq)
#> 1 13 -1216.0
#> 2 1 -1464.1 -12 496.17 < 2.2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> Analysis of Deviance Table (Type III tests)
#>
#> Response: clicked
#> LR Chisq Df Pr(>Chisq)
#> age 24.858 1 6.171e-07 ***
#> daily_minutes 40.568 1 1.899e-10 ***
#> prior_clicks_90d 30.517 1 3.309e-08 ***
#> device 18.459 2 9.812e-05 ***
#> ad_format 66.253 3 2.706e-14 ***
#> personalized 39.167 1 3.891e-10 ***
#> log(liked_songs) 125.000 1 < 2.2e-16 ***
#> views_c 153.478 1 < 2.2e-16 ***
#> I(views_c^2) 148.442 1 < 2.2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> Likelihood ratio test
#>
#> Model 1: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs)
#> Model 2: clicked ~ age + daily_minutes + prior_clicks_90d + device + ad_format +
#> personalized + log(liked_songs) + views_c + I(views_c^2)
#> #Df LogLik Df Chisq Pr(>Chisq)
#> 1 11 -1308.5
#> 2 13 -1216.0 2 184.88 < 2.2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> [1] 0.1694438
How to read it.
I(views_c^2), is meaningful on its own (is the curve
needed?) and repeats the m3-versus-m4 test above.device is significant as a whole even
though its Tablet dummy alone is not.“Significant” means “adds information beyond the other predictors” — not “has a large effect.” The odds ratios show the size of each effect.
What the code does.
tidy(..., exponentiate = TRUE, conf.int = TRUE) converts
each coefficient to an odds ratio (OR = \(e^{\beta}\)) with a 95% confidence
interval. Because a one-unit change is too small to matter for some
predictors, three are rescaled:
daily_minutes and age: per 10 units, \(e^{10\beta}\).log(liked_songs): per doubling of
liked songs, \(2^{\beta}\).tidy(final, exponentiate = TRUE, conf.int = TRUE) |>
select(term, odds_ratio = estimate, conf.low, conf.high, p.value) |>
mutate(across(where(is.numeric), \(x) round(x, 3)))#> # A tibble: 13 x 5
#> term odds_ratio conf.low conf.high p.value
#> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 (Intercept) 0.006 0.003 0.014 0
#> 2 age 0.975 0.966 0.985 0
#> 3 daily_minutes 1.011 1.008 1.014 0
#> 4 prior_clicks_90d 1.225 1.14 1.316 0
#> 5 deviceDesktop 0.577 0.444 0.745 0
#> 6 deviceTablet 0.825 0.589 1.143 0.256
#> 7 ad_formatVideo 1.841 1.446 2.348 0
#> 8 ad_formatPopup 1.347 1.018 1.779 0.036
#> 9 ad_formatAudio 0.507 0.356 0.713 0
#> 10 personalizedYes 1.89 1.547 2.312 0
#> 11 log(liked_songs) 1.891 1.685 2.127 0
#> 12 views_c 1.33 1.267 1.397 0
#> 13 I(views_c^2) 0.943 0.931 0.954 0
#> daily_minutes
#> 1.114408
#> age
#> 0.7787868
#> log(liked_songs)
#> 1.55526
How to read it. An OR above 1 raises the odds of a click; below 1 lowers them; a confidence interval that includes 1 means the difference is not statistically significant at the 5% level (the data are consistent with no difference).
| Effect | Odds ratio | Reading |
|---|---|---|
| personalized: Yes vs. No | 1.89 | about 1.9 times the odds |
| Video vs. Banner | 1.84 | video ads raise the odds most among formats |
| Popup vs. Banner | 1.35 | a smaller gain |
| Audio vs. Banner | 0.51 | about half the odds |
| Desktop vs. Mobile | 0.58 | lower odds on desktop |
| Tablet vs. Mobile | 0.83 | CI 0.59–1.14 includes 1 |
| +1 prior ad click | 1.23 | past clickers click more |
| +10 minutes a day | 1.11 | heavier listeners click more |
| +10 years of age | 0.78 | odds fall about 22% per decade |
| Doubling of liked songs | 1.56 | bigger libraries, more clicks |
Two cautions:
views_c and I(views_c^2) also appear in the
tidy() output, but do not read either one
alone: the effect of one more view depends on how many views
the listener has already had. Read them together, as a curve.
What the code does. For a curve \(\beta_1 X + \beta_2 X^2\), the slope is \(\beta_1 + 2\beta_2 X\). Setting it to zero gives the turning point \(X = -\beta_1 / (2\beta_2)\). Because we centered at 4, we add 4 back to express the peak in actual views. The code then predicts the probability of a click at every number of views for a typical listener (age 30, 60 minutes a day, 250 liked songs, 2 prior ad clicks, Mobile, banner, not personalized), changing only the number of views.
#> views_c
#> 6.41795
curve <- tibble(offer_views_7d = 0:15, views_c = offer_views_7d - 4,
age = 30, daily_minutes = 60, liked_songs = 250, prior_clicks_90d = 2,
device = factor("Mobile", levels = levels(clicks$device)),
ad_format = factor("Banner", levels = levels(clicks$ad_format)),
personalized = factor("No", levels = levels(clicks$personalized)))
curve$p_click <- predict(final, newdata = curve, type = "response")
ggplot(curve, aes(offer_views_7d, p_click)) +
geom_line(linewidth = 1) + geom_point(size = 2.5) +
geom_vline(xintercept = peak, linetype = "dashed") +
scale_y_continuous(labels = scales::percent) +
scale_x_continuous(breaks = 0:15) +
labs(x = "Offer views in the past 7 days", y = "Predicted P(click)",
title = "The chance of a click peaks, then falls")How to read it. The peak is at 6.4 views per week, where this listener’s chance of clicking is about 29%. It starts at 3.5% with no views and falls to 6% by 12 views.
Note that offer_views_7d counts views
before the current one, so “6 prior views” means this
is the 7th.
Business reading: after about 6 prior views, each extra view lowers the chance of a click. That is not the same as “stop at 6”: by the cost rule (see Choose the cutoff), a view still pays as long as the predicted probability stays above the cost-based cutoff (about 0.12, computed in Minimize expected cost), which for this listener holds through about 10 prior views. The peak and the cutoff therefore bracket a sensible weekly cap — somewhere between about 6 and 10 views — that a frequency-cap experiment should settle. Listeners with 0–2 prior views also score below the cutoff, but nobody reaches a third view without a first, and the curve is an association, not a dose-response experiment.
Two further cautions: few training listeners had 12 or more views
(71), so the right end of the curve rests on little data; and a
quadratic forces the curve to be symmetric around its peak (a spline,
mgcv::gam, could check that). A frequency-cap experiment
would confirm the cap before Spotify relies on it.
What the code does. expand_grid()
builds every combination of three ad formats and the two personalization
settings for the typical listener (4 views, so
views_c = 0). predict(..., type = "response")
returns probabilities; without
type = "response", predict() returns log-odds.
Every variable in newdata must match the model, including
views_c.
The confidence intervals are built on the log-odds
scale and then converted with plogis(), which
keeps them between 0 and 1.
profiles <- expand_grid(
age = 30, daily_minutes = 60, liked_songs = 250, prior_clicks_90d = 2,
views_c = 0, # 4 offer views
device = factor("Mobile", levels = levels(clicks$device)),
ad_format = factor(c("Banner", "Video", "Audio"), levels = levels(clicks$ad_format)),
personalized = factor(c("No", "Yes"), levels = levels(clicks$personalized)))
pr <- predict(final, newdata = profiles, type = "link", se.fit = TRUE)
profiles <- profiles |>
mutate(p_click = plogis(pr$fit),
lower = plogis(pr$fit - 1.96 * pr$se.fit),
upper = plogis(pr$fit + 1.96 * pr$se.fit))
profiles |>
select(ad_format, personalized, p_click, lower, upper) |>
mutate(across(where(is.numeric), \(x) round(x, 3))) |>
kable()| ad_format | personalized | p_click | lower | upper |
|---|---|---|---|---|
| Banner | No | 0.227 | 0.190 | 0.268 |
| Banner | Yes | 0.356 | 0.307 | 0.409 |
| Video | No | 0.350 | 0.302 | 0.402 |
| Video | Yes | 0.505 | 0.447 | 0.562 |
| Audio | No | 0.129 | 0.096 | 0.172 |
| Audio | Yes | 0.219 | 0.168 | 0.280 |
How to read it. Personalization multiplies the odds by the same 1.89 for every format (there is no interaction in the model), yet it adds different amounts of probability:
The logistic curve is steepest near 50%, so the same change in odds moves the probability more when the starting probability is closer to the middle. This is the practical meaning of “the model is linear in the log-odds, not in the probability.”
Validation happens in two stages. In-sample: does the model fit the data it was built on, and does any single listener drive it? Out-of-sample: does it still work on listeners it has never seen? The cutoff is chosen on the training data and scored once on the test data.
What the code does. hoslem.test() sorts
listeners by predicted probability, cuts them into 10 groups, and
compares the observed number of clicks in each group
with the number the model expected (the sum of the
predicted probabilities). The test statistic is a Pearson χ² with \(g - 2 = 8\) degrees of freedom.
#>
#> Hosmer and Lemeshow goodness of fit (GOF) test
#>
#> data: as.numeric(train$clicked == "1"), fitted(final)
#> X-squared = 5.9216, df = 8, p-value = 0.656
hl_tab <- tibble(group = 1:10,
observed = as.numeric(hl$observed[, "y1"]),
expected = round(as.numeric(hl$expected[, "yhat1"]), 1))
kable(hl_tab)| group | observed | expected |
|---|---|---|
| 1 | 4 | 6.3 |
| 2 | 11 | 14.4 |
| 3 | 28 | 22.5 |
| 4 | 38 | 32.1 |
| 5 | 37 | 42.3 |
| 6 | 57 | 52.9 |
| 7 | 64 | 66.9 |
| 8 | 89 | 87.4 |
| 9 | 111 | 114.4 |
| 10 | 168 | 167.9 |
hl_tab |>
pivot_longer(c(observed, expected), names_to = "type", values_to = "clicks") |>
mutate(type = factor(type, levels = c("observed", "expected"))) |>
ggplot(aes(factor(group), clicks, fill = type)) +
geom_col(position = position_dodge(width = 0.8), width = 0.75) +
scale_fill_manual(values = c(expected = "#D9772B", observed = "#0E7FA8")) +
labs(x = "Risk group (1 = lowest predicted probability)", y = "Clicks", fill = NULL)How to read it. Here, unusually, we want a non-significant result: p > .05 means there is no evidence of misfit (not proof that the model is right). The test gives χ²(8) = 5.92, p = .656: no evidence of lack of fit. The table shows observed and expected clicks track closely in every group, from the lowest-risk listeners to the highest.
This is a calibration check — when the model says 30%, about 30% click. It matters for the 1/7 theory check in Choose the cutoff, which assumes calibrated probabilities. Note that this is calibration on the training data; calibration on new data is not tested here. The HL test has low power and depends on the number of groups, so a pass is necessary but not sufficient; read the residuals too.
What the code does. augment() adds each
listener’s deviance residual and fitted value to a data
frame; cooks.distance() measures how much the whole fit
would change without that listener; dfbetas() measures how
much each coefficient would change, in standard-error units.
diag <- augment(final, type.residuals = "deviance") |>
mutate(cooks = cooks.distance(final))
mean(abs(diag$.resid) > 2) # share of |deviance residual| > 2#> [1] 0.02963227
#> max_cooks cutoff_4n
#> 0.010988919 0.001428061
#> [1] 0.2279674
ggplot(diag, aes(.fitted, .resid)) +
geom_point(alpha = 0.3) +
geom_hline(yintercept = c(-2, 0, 2), linetype = c(2, 1, 2)) +
labs(x = "Fitted log-odds", y = "Deviance residual")#> StudRes Hat CookD
#> 124 -0.8073037 0.0235435325 0.0007197332
#> 191 2.8827245 0.0007062817 0.0033398312
#> 489 -1.2898599 0.0243018063 0.0024777104
#> 1441 2.1036967 0.0183029397 0.0109889187
#> 1625 2.2470419 0.0068140952 0.0058571591
#> 1744 2.8652336 0.0008160891 0.0036598753
How to read it.
Conclusion: no single listener drives the results; keep everyone.
What the code does. predict() scores
every listener in the training and test sets. roc() builds
the ROC curve, which traces sensitivity against 1 −
specificity at every possible cutoff. levels and
direction are set explicitly so pROC does not
have to guess which class is the event. The AUC is the
probability that the model ranks a random clicker above a random
non-clicker. Comparing training and test AUC shows how optimistic the
in-sample numbers were.
p_train <- predict(final, type = "response")
p_test <- predict(final, newdata = test, type = "response")
roc_train <- roc(train$clicked, p_train, levels = c("0", "1"), direction = "<")
roc_test <- roc(test$clicked, p_test, levels = c("0", "1"), direction = "<")
auc(roc_train)#> Area under the curve: 0.7757
#> Area under the curve: 0.7749
#> 95% CI: 0.744-0.8058 (DeLong)
How to read it. Training AUC = 0.776; test AUC = 0.775 (95% CI 0.744–0.806). On the lecture’s rubric (.90+ excellent, .80 good, .70 fair, .60 poor) this is fair, near the top of the band. The two AUCs are almost identical, so there is very little optimism — expected for a model with few parameters and many events per parameter.
For comparison, the model without the squared term
(m3) has a test AUC of 0.717: the curve improves
predictions on new listeners, not just the fit statistics.
AUC summarizes ranking across all cutoffs. It says nothing about accuracy at any one cutoff — that is what the confusion matrix is for.
A probability becomes a decision only when we choose a cutoff: show the offer to everyone whose predicted probability is at or above it. The default of 0.50 is almost never right when the outcome is unbalanced. We compare two principled choices, both chosen on the training data.
What the code does.
coords(..., "best", best.method = "youden") finds the
cutoff that maximizes sensitivity + specificity − 1: the point on the
ROC curve farthest above the diagonal.
cut_youden <- coords(roc_train, "best", best.method = "youden",
ret = c("threshold", "sensitivity", "specificity"))
cut_youden#> threshold sensitivity specificity
#> 1 0.2185061 0.7001647 0.706928
How to read it. Youden’s cutoff is 0.219 — close to the base rate, not 0.50. It treats a missed clicker and a wasted offer as equally costly, which is not true here.
What the code does.
coords(..., x = "all") returns the number of false
negatives and false positives at every possible cutoff.
We put a dollar value on each, add them up, and pick the cutoff with the
lowest total cost.
cost_FN <- 6 # $ expected profit lost per missed clicker
cost_FP <- 1 # $ ad-slot revenue lost per listener who never clicks
co <- coords(roc_train, x = "all", ret = c("threshold", "fn", "fp"))
co$cost <- co$fn * cost_FN + co$fp * cost_FP
cut_cost <- co[which.min(co$cost), ]
cut_cost#> threshold fn fp cost
#> 1018 0.1187064 58 1235 1583
How to read it. The cost-minimizing cutoff is 0.119, lower than Youden’s. When a missed clicker costs six times a wasted offer, it pays to show the offer to more listeners: more sensitivity, less specificity. As a check, for well-calibrated probabilities the cost-optimal cutoff is about cost_FP / (cost_FP + cost_FN) = 1/7 = 0.143 — the same neighborhood. They differ because the cost curve is flat near its minimum and the search uses one sample’s counts.
If a missed clicker were worth only $2, the theoretical cutoff would rise to 1/3: Spotify would target fewer, more likely listeners. The cutoff is a business decision, informed by statistics.
What the code does. It classifies each
test listener as a predicted clicker (1) or not (0) at
each cutoff and compares the predictions with what actually happened.
confusionMatrix() needs two factors with the same levels
and the positive = "1" argument, which sets clicking as the
event.
pred_cut <- factor(as.integer(p_test >= cut_cost$threshold), levels = c(0, 1))
confusionMatrix(pred_cut, test$clicked, positive = "1")#> Confusion Matrix and Statistics
#>
#> Reference
#> Prediction 0 1
#> 0 418 28
#> 1 521 232
#>
#> Accuracy : 0.5421
#> 95% CI : (0.5134, 0.5706)
#> No Information Rate : 0.7832
#> P-Value [Acc > NIR] : 1
#>
#> Kappa : 0.2002
#>
#> Mcnemar's Test P-Value : <2e-16
#>
#> Sensitivity : 0.8923
#> Specificity : 0.4452
#> Pos Pred Value : 0.3081
#> Neg Pred Value : 0.9372
#> Prevalence : 0.2168
#> Detection Rate : 0.1935
#> Detection Prevalence : 0.6280
#> Balanced Accuracy : 0.6687
#>
#> 'Positive' Class : 1
#>
The same for Youden’s cutoff and the habitual 0.50, collected into one comparison table:
score <- function(cut, label) {
pred <- factor(as.integer(p_test >= cut), levels = c(0, 1))
cm <- confusionMatrix(pred, test$clicked, positive = "1")
t <- cm$table
tibble(cutoff = label, threshold = round(cut, 3),
TP = t["1", "1"], FN = t["0", "1"], FP = t["1", "0"], TN = t["0", "0"],
accuracy = round(cm$overall[["Accuracy"]], 3),
sensitivity = round(cm$byClass[["Sensitivity"]], 3),
specificity = round(cm$byClass[["Specificity"]], 3),
precision = round(cm$byClass[["Pos Pred Value"]], 3),
cost = t["0", "1"] * cost_FN + t["1", "0"] * cost_FP)
}
results <- bind_rows(score(cut_cost$threshold, "Cost-based"),
score(cut_youden$threshold, "Youden"),
score(0.50, "Default 0.50"))
kable(results)| cutoff | threshold | TP | FN | FP | TN | accuracy | sensitivity | specificity | precision | cost |
|---|---|---|---|---|---|---|---|---|---|---|
| Cost-based | 0.119 | 232 | 28 | 521 | 418 | 0.542 | 0.892 | 0.445 | 0.308 | 689 |
| Youden | 0.219 | 187 | 73 | 301 | 638 | 0.688 | 0.719 | 0.679 | 0.383 | 739 |
| Default 0.50 | 0.500 | 69 | 191 | 46 | 893 | 0.802 | 0.265 | 0.951 | 0.600 | 1192 |
How to read it.
This is the accuracy paradox: the most accurate cutoff is the worst business decision. Report the metrics that match the decision, not the one that looks best. A precision of about 31% means roughly 1 in 3 targeted listeners clicks, against about 22% if Spotify showed the offer to everyone.
What the model says (associations, in this simulated scenario):
What Spotify should do:
Limitations:
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016). Sample size considerations for the external validation of a multivariable prognostic model: A resampling study. Statistics in Medicine, 35(2), 214–226. https://doi.org/10.1002/sim.6787
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). Wiley.
Steyerberg, E. W., Harrell, F. E., Jr., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001). Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology, 54(8), 774–781.
#> R version 4.3.3 (2024-02-29)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 24.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0
#>
#> locale:
#> [1] C
#>
#> time zone: America/Chicago
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] knitr_1.45 ResourceSelection_0.0.0.9000
#> [3] caret_6.0-94 lattice_0.22-5
#> [5] pROC_1.18.5 lmtest_0.9-40
#> [7] zoo_1.8-12 car_3.1-2
#> [9] carData_3.0-5 broom_1.0.5
#> [11] lubridate_1.9.3 forcats_1.0.0
#> [13] stringr_1.5.1 dplyr_1.1.4
#> [15] purrr_1.0.2 readr_2.1.5
#> [17] tidyr_1.3.1 tibble_3.2.1
#> [19] ggplot2_3.4.4 tidyverse_2.0.0
#>
#> loaded via a namespace (and not attached):
#> [1] tidyselect_1.2.0 timeDate_4032.109 farver_2.1.1
#> [4] fastmap_1.1.1 digest_0.6.34 rpart_4.1.23
#> [7] timechange_0.3.0 lifecycle_1.0.4 ellipsis_0.3.2
#> [10] survival_3.5-8 magrittr_2.0.3 compiler_4.3.3
#> [13] rlang_1.1.3 sass_0.4.8 tools_4.3.3
#> [16] utf8_1.2.4 yaml_2.3.8 data.table_1.14.10
#> [19] labeling_0.4.3 bit_4.0.5 plyr_1.8.9
#> [22] abind_1.4-5 withr_2.5.0 nnet_7.3-19
#> [25] grid_4.3.3 stats4_4.3.3 fansi_1.0.5
#> [28] e1071_1.7-14 colorspace_2.1-0 future_1.33.1
#> [31] globals_0.16.2 scales_1.3.0 iterators_1.0.14
#> [34] MASS_7.3-60.0.1 cli_3.6.2 crayon_1.5.2
#> [37] rmarkdown_2.25 generics_0.1.3 future.apply_1.11.1
#> [40] reshape2_1.4.4 tzdb_0.4.0 proxy_0.4-27
#> [43] cachem_1.0.8 splines_4.3.3 parallel_4.3.3
#> [46] vctrs_0.6.5 hardhat_1.3.1 Matrix_1.6-5
#> [49] jsonlite_1.8.8 hms_1.1.3 bit64_4.0.5
#> [52] listenv_0.9.1 foreach_1.5.2 gower_1.0.1
#> [55] jquerylib_0.1.4 recipes_1.0.9 glue_1.7.0
#> [58] parallelly_1.37.1 codetools_0.2-19 stringi_1.8.3
#> [61] gtable_0.3.4 munsell_0.5.0 pillar_1.9.0
#> [64] htmltools_0.5.7 ipred_0.9-14 lava_1.7.3
#> [67] R6_2.5.1 vroom_1.6.5 evaluate_0.23
#> [70] highr_0.10 backports_1.4.1 bslib_0.6.1
#> [73] class_7.3-22 Rcpp_1.0.12 nlme_3.1-164
#> [76] prodlim_2023.08.28 mgcv_1.9-1 xfun_0.41
#> [79] pkgconfig_2.0.3 ModelMetrics_1.2.2.2