The L7 live-session deck fits one logistic regression to
Insurance_Calls.csv. This file reports the main results for
that model, in the order an analyst reads them:
Each section shows the R code, the result, and a written interpretation that is generated from the numbers, so the text stays correct if the data change.
library(readr)
calls <- read_csv("C:/Users/erl222/OneDrive - University of Texas at San Antonio/Desktop/BAN 3073/3. BAN 3073 Course notes/L7. Logistic regression/L7. Live session (Part 1)/Insurance_Calls.csv")
#View(calls)
# The outcome must be 0/1 or a factor whose second level is the event.
if (is.character(calls$purchased) || is.factor(calls$purchased)) {
calls$purchased <- factor(tolower(as.character(calls$purchased)), levels = c("no", "yes"))
}
fit <- glm(purchased ~ age + income + married + dependents + has_mortgage + smoker +
existing_customer + lead_source,
data = calls, family = binomial)
n <- nrow(calls)
y <- as.numeric(calls$purchased) - if (is.factor(calls$purchased)) 1 else 0
n1 <- sum(y == 1); n0 <- sum(y == 0)
k <- length(coef(fit)) # parameters including the intercept
ev <- if (is.factor(calls$purchased)) "yes" else "1" # the event level
The file has 5,000 prospects and 0 missing values.
Every category of every predictor has hundreds of prospects, so no
coefficient rests on a handful of cases. Every predictor is something
the agency knows before it calls: age (years),
income (household income in thousands of dollars),
married, dependents (children under 18),
has_mortgage, smoker,
existing_customer (already holds another policy with the
agency) and lead_source (how the prospect reached the
agency).
R uses the first level of each categorical predictor, in alphabetical order, as its reference level. In this model those are:
| Predictor | Reference level |
|---|---|
| married | No |
| has_mortgage | No |
| smoker | No |
| existing_customer | No |
| lead_source | Purchased list |
| Ordered value | purchased | Total frequency |
|---|---|---|
| 1 | 0 | 3490 |
| 2 | 1 | 1510 |
R models the probability that purchased = 1 (bought a
policy). 30.2% of prospects bought. A model that predicts the larger
class for everyone would be right 69.8% of the time, the benchmark
accuracy must beat.
m0 <- glm(purchased ~ 1, data = calls, family = binomial) # intercept only
m2LL <- c(intercept_only = -2 * as.numeric(logLik(m0)), with_covariates = -2 * as.numeric(logLik(fit)))
AICs <- c(AIC(m0), AIC(fit))
SCs <- m2LL + c(1, k) * log(n) # BIC
| Criterion | Intercept only | Intercept and covariates |
|---|---|---|
| AIC | 6,127.49 | 4,729.05 |
| BIC | 6,134.01 | 4,794.22 |
| −2 Log L | 6,125.49 | 4,709.05 |
Interpretation. −2 Log L is the deviance: lower means less misfit. Adding the 9 predictor terms lowers −2 Log L from 6,125.49 to 4,709.05. AIC and BIC add a penalty for the number of parameters. Both fall, so the improvement in fit is more than the extra parameters cost. These statistics are for comparing models fitted to the same data; their size on its own has no benchmark.
LR <- m2LL[1] - m2LL[2] # likelihood ratio
score <- anova(m0, fit, test = "Rao")$Rao[2] # score (Rao) test
b <- coef(fit)[-1]; V <- vcov(fit)[-1, -1]
wald <- as.numeric(t(b) %*% solve(V) %*% b) # joint Wald test
df_g <- k - 1
global <- data.frame(Test = c("Likelihood Ratio", "Score", "Wald"),
`Chi-Square` = c(LR, score, wald), DF = df_g,
`Pr > ChiSq` = pchisq(c(LR, score, wald), df_g, lower.tail = FALSE), check.names = FALSE)
| Test | Chi-Square | DF | Pr > ChiSq |
|---|---|---|---|
| Likelihood Ratio | 1,416.44 | 9 | <.0001 |
| Score | 1,271.16 | 9 | <.0001 |
| Wald | 941.32 | 9 | <.0001 |
Interpretation. All three tests ask the same
question: do the predictors, taken together, improve on a model with no
predictors? The likelihood-ratio test is the one taught in L7
(lrtest()): χ²(9) = 1,416.44, p < .001. The predictors
jointly improve the model significantly, so the model is better than
predicting the base rate for everyone. The score and Wald versions
agree. A significant joint test does not show that every predictor
matters; that is what the Type 3 table answers.
r2_cs <- 1 - exp((m2LL[2] - m2LL[1]) / n) # Cox–Snell R-square
r2_max <- r2_cs / (1 - exp(-m2LL[1] / n)) # Nagelkerke (max-rescaled) R-square
r2_mcf <- 1 - m2LL[2] / m2LL[1] # McFadden, as in the L7 lecture
round(c(cox_snell = unname(r2_cs), max_rescaled = unname(r2_max), mcfadden = unname(r2_mcf)), 4)
cox_snell max_rescaled mcfadden
0.2467 0.3493 0.2312
Interpretation. McFadden’s pseudo-R² is 0.231. On the lecture’s scale (0 to about 0.4, with 0.2–0.4 already excellent), that is a strong fit. Two other versions, Cox–Snell (0.247) and Nagelkerke (0.349), are also common. None of these is the share of variance explained, so they should not be judged against OLS R² values.
t3w <- Anova(fit, type = 3, test.statistic = "Wald") # Wald version
t3lr <- Anova(fit, type = 3, test.statistic = "LR") # the L7 likelihood-ratio version
t3w <- t3w[rownames(t3w) != "(Intercept)", ]
t3 <- data.frame(Effect = rownames(t3w), DF = t3w$Df, `Wald Chi-Square` = t3w$Chisq,
`Pr > ChiSq` = t3w$`Pr(>Chisq)`, `LR Chi-Square` = t3lr[rownames(t3w), "LR Chisq"],
`LR p` = t3lr[rownames(t3w), "Pr(>Chisq)"], check.names = FALSE)
| Effect | DF | Wald Chi-Square | Pr > ChiSq | LR Chi-Square | LR p |
|---|---|---|---|---|---|
| age | 1 | 72.86 | <.0001 | 75.17 | <.0001 |
| income | 1 | 248.50 | <.0001 | 271.61 | <.0001 |
| married | 1 | 41.57 | <.0001 | 41.90 | <.0001 |
| dependents | 1 | 244.84 | <.0001 | 259.58 | <.0001 |
| has_mortgage | 1 | 61.51 | <.0001 | 61.69 | <.0001 |
| smoker | 1 | 93.35 | <.0001 | 103.65 | <.0001 |
| existing_customer | 1 | 178.56 | <.0001 | 181.17 | <.0001 |
| lead_source | 2 | 398.27 | <.0001 | 440.46 | <.0001 |
Interpretation. Each row tests one predictor as a whole, holding the others constant. A categorical predictor with several levels gets one test on several degrees of freedom. The table shows both the Wald and the likelihood-ratio versions; the L7 lecture uses the likelihood-ratio version, and the two lead to the same decision when they agree on which side of .05 each p-value falls.
age, income,
married, dependents,
has_mortgage, smoker,
existing_customer, lead_source.The Type 3 test for lead_source compares all three
sources at once. The coefficient table compares each source only with
the reference level (Purchased list). The remaining comparison, Quote
request versus Referral, comes from the same model:
b_m <- coef(fit); V_m <- vcov(fit)
r1 <- "lead_sourceQuote request"; r2 <- "lead_sourceReferral"
est <- unname(b_m[r1] - b_m[r2])
se <- unname(sqrt(V_m[r1, r1] + V_m[r2, r2] - 2 * V_m[r1, r2]))
sm <- c(OR = exp(est), lower = exp(est - 1.96 * se), upper = exp(est + 1.96 * se), p = 2 * pnorm(-abs(est / se)))
linearHypothesis(fit, "lead_sourceQuote request = lead_sourceReferral") # same test as a Wald chi-square
Linear hypothesis test
Hypothesis:
lead_sourceQuote request - lead_sourceReferral = 0
Model 1: restricted model
Model 2: purchased ~ age + income + married + dependents + has_mortgage +
smoker + existing_customer + lead_source
Res.Df Df Chisq Pr(>Chisq)
1 4991
2 4990 1 59.618 0.00000000000001152 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Interpretation. Prospects who requested a quote have about 2.090 times the odds of buying that referred prospects have (95% CI 1.733 to 2.520, p < .001), holding the other predictors constant. So the three sources differ from one another, not only from the purchased list: prospects who asked for a quote are the strongest leads, which makes sense because they have already shown interest.
s <- summary(fit)$coefficients
mle <- data.frame(Parameter = rownames(s), Estimate = s[, 1], `Standard Error` = s[, 2],
`Wald Chi-Square` = s[, 3]^2, `Pr > ChiSq` = s[, 4], check.names = FALSE)
| Parameter | Estimate | Standard Error | Wald Chi-Square | Pr > ChiSq |
|---|---|---|---|---|
| (Intercept) | -2.7096 | 0.1968 | 189.48 | <.0001 |
| age | -0.0354 | 0.0041 | 72.86 | <.0001 |
| income | 0.0158 | 0.0010 | 248.50 | <.0001 |
| marriedYes | 0.5575 | 0.0865 | 41.57 | <.0001 |
| dependents | 0.5506 | 0.0352 | 244.84 | <.0001 |
| has_mortgageYes | 0.5866 | 0.0748 | 61.51 | <.0001 |
| smokerYes | -0.9998 | 0.1035 | 93.35 | <.0001 |
| existing_customerYes | 1.0842 | 0.0811 | 178.56 | <.0001 |
| lead_sourceQuote request | 1.7763 | 0.0902 | 387.65 | <.0001 |
| lead_sourceReferral | 1.0391 | 0.0899 | 133.59 | <.0001 |
Interpretation. Each estimate is the change in the
log-odds of buying for a one-unit increase in a numeric
predictor, or for a category compared with its reference level, holding
the other predictors constant. The Wald chi-square is the square of the
z statistic in summary(), so the p-values are
identical. The sign gives the direction; the size is easier to read as
an odds ratio, below.
or <- data.frame(Effect = rownames(s)[-1], `Point Estimate` = exp(coef(fit))[-1],
`95% Wald Lower` = exp(confint.default(fit))[-1, 1],
`95% Wald Upper` = exp(confint.default(fit))[-1, 2], check.names = FALSE)
# Per-unit odds ratios for wide-ranging numeric predictors are close to 1, so also report a
# meaningful change
units <- c(age = 10, income = 10, dependents = 1)
or_units <- data.frame(Effect = names(units), Change = units,
`Odds ratio` = exp(coef(fit)[names(units)] * units),
Lower = exp(confint.default(fit)[names(units), 1] * units),
Upper = exp(confint.default(fit)[names(units), 2] * units), check.names = FALSE)
| Effect | Point Estimate | 95% Wald Lower | 95% Wald Upper |
|---|---|---|---|
| age | 0.9652 | 0.9574 | 0.9731 |
| income | 1.0160 | 1.0140 | 1.0180 |
| marriedYes | 1.7463 | 1.4741 | 2.0687 |
| dependents | 1.7342 | 1.6186 | 1.8580 |
| has_mortgageYes | 1.7978 | 1.5527 | 2.0816 |
| smokerYes | 0.3679 | 0.3004 | 0.4507 |
| existing_customerYes | 2.9571 | 2.5223 | 3.4669 |
| lead_sourceQuote request | 5.9077 | 4.9502 | 7.0503 |
| lead_sourceReferral | 2.8268 | 2.3701 | 3.3715 |
| Effect | Change | Odds ratio | Lower | Upper |
|---|---|---|---|---|
| age | 10 | 0.702 | 0.647 | 0.761 |
| income | 10 | 1.172 | 1.149 | 1.195 |
| dependents | 1 | 1.734 | 1.619 | 1.858 |
Interpretation. An odds ratio above 1 raises the odds of buying and one below 1 lowers them, holding the other predictors constant. A confidence interval that includes 1 is not significant at the 5% level. Odds ratios describe odds, not probabilities, and they describe association in observational data, not cause.
Do the directions make business sense? Yes. Prospects with more to protect (a spouse, children, a mortgage) and more to spend (higher income) have higher odds of buying, and so do warm leads: existing customers, referrals and, above all, prospects who asked for a quote. Smokers and older prospects have lower odds, which fits term life insurance, where their premiums are higher.
p <- fitted(fit)
p1 <- p[y == 1]; p0 <- sort(p[y == 0])
lower <- findInterval(p1, p0, left.open = TRUE) # non-buyers with a lower probability
le <- findInterval(p1, p0) # ... lower or equal
conc <- sum(lower); tied <- sum(le - lower); pairs <- as.numeric(n1) * n0
disc <- pairs - conc - tied
assoc <- c(concordant = conc / pairs, discordant = disc / pairs, tied = tied / pairs, pairs = pairs,
somers_d = (conc - disc) / pairs, gamma = (conc - disc) / (conc + disc),
tau_a = (conc - disc) / (n * (n - 1) / 2), c = (conc + 0.5 * tied) / pairs)
| Statistic | Value | Value | |
|---|---|---|---|
| Percent Concordant | 81.1 | Somers’ D | 0.622 |
| Percent Discordant | 18.9 | Gamma | 0.622 |
| Percent Tied | 0.0 | Tau-a | 0.262 |
| Pairs | 5,269,900 | c | 0.811 |
Interpretation. Concordance pairs every buyer with every non-buyer (5,269,900 pairs). A pair is concordant when the buyer has the higher predicted probability. The model ranks 81.1% of pairs correctly and 18.9% the wrong way. The c statistic, 0.811, counts ties as half and is the AUC. Somers’ D (0.622) equals 2c − 1, so it rescales the same information to run from −1 to 1.
calls$p_hat <- predict(fit, type = "response") # predicted probability of buying
roc_fit <- roc(calls$purchased, calls$p_hat,
levels = levels(factor(calls$purchased)), # non-event first, event second
direction = "<", quiet = TRUE) # higher probability = more likely to buy
auc(roc_fit) # area under the curve
Area under the curve: 0.8109
auc_ci <- ci.auc(roc_fit); auc_ci # 95% CI (DeLong)
95% CI: 0.798-0.8238 (DeLong)
auc_val <- as.numeric(auc(roc_fit))
plot(roc_fit, legacy.axes = TRUE, print.auc = TRUE, # x-axis = 1 - specificity
main = "ROC curve: insurance call model")
Interpretation. AUC = 0.811 (95% CI 0.798 to 0.824). A randomly chosen buyer receives a higher predicted probability than a randomly chosen non-buyer about 81% of the time. On the course scale (.90–1.00 excellent, .80–.90 good, .70–.80 fair, .60–.70 poor, .50–.60 fail) the model’s discrimination is good. It equals the concordance statistic c (0.811) because both count tied pairs as half. This AUC is measured on the same prospects used to fit the model, so it is an in-sample value and is usually somewhat optimistic for new prospects.
hl <- performance_hosmer(fit, n_bins = 10)
hl
# Hosmer-Lemeshow Goodness-of-Fit Test
Chi-squared: 2.560
df: 8
p-value: 0.959
Interpretation. Hosmer–Lemeshow sorts prospects into 10 groups by predicted probability and compares predicted with observed buyers in each group. The null hypothesis is that they agree. χ²(8) = 2.56, p = .959. There is no evidence of lack of fit: predicted and observed purchase rates agree across the risk groups. That is not proof the model is correct, because the test has limited power.
library(ResourceSelection)
hlg <- hoslem.test(calls$purchased, calls$p_hat, g = 10) # same statistic, with the group table
hlg
Hosmer and Lemeshow goodness of fit (GOF) test
data: calls$purchased, calls$p_hat
X-squared = 2.5596, df = 8, p-value = 0.9589
grp <- data.frame(Group = 1:10, Prospects = rowSums(hlg$observed),
Observed = hlg$observed[, 2], Expected = hlg$expected[, 2])
grp$Gap <- grp$Observed - grp$Expected # positive = model under-predicts
barplot(t(cbind(Observed = grp$Observed, Expected = grp$Expected)), beside = TRUE,
names.arg = grp$Group, legend.text = TRUE, args.legend = list(x = "topleft"),
xlab = "Risk group (1 = lowest predicted probability)", ylab = "Buyers",
main = "Observed vs. expected buyers")
| Group | Prospects | Observed | Expected | Gap |
|---|---|---|---|---|
| 1 | 500 | 20 | 16.6 | 3.4 |
| 2 | 500 | 34 | 34.0 | 0.0 |
| 3 | 500 | 52 | 52.4 | -0.4 |
| 4 | 500 | 76 | 75.0 | 1.0 |
| 5 | 500 | 99 | 101.6 | -2.6 |
| 6 | 500 | 133 | 133.9 | -0.9 |
| 7 | 500 | 174 | 173.4 | 0.6 |
| 8 | 500 | 214 | 225.4 | -11.4 |
| 9 | 500 | 305 | 296.2 | 8.8 |
| 10 | 500 | 403 | 401.5 | 1.5 |
Interpretation. The largest under-prediction is in group 9 (305 observed vs. 296 expected) and the largest over-prediction in group 8 (214 observed vs. 225 expected). The biggest gap is 11.4 buyers in a group of about 500 prospects. The gaps are small and change sign 6 times across the ten groups rather than following a pattern, which is what a well-calibrated model looks like: when it predicts a 30% chance, about 30% of those prospects buy.
cls <- function(cut) {
pred <- as.integer(p >= cut)
tp <- sum(pred == 1 & y == 1); tn <- sum(pred == 0 & y == 0); fp <- sum(pred == 1 & y == 0); fn <- sum(pred == 0 & y == 1)
c(cutoff = cut, correct_events = tp, correct_nonevents = tn, incorrect_events = fp, incorrect_nonevents = fn,
accuracy = (tp + tn) / n, sensitivity = tp / (tp + fn), specificity = tn / (tn + fp))
}
best <- coords(roc_fit, "best", best.method = "youden", ret = "threshold", transpose = TRUE)[1] # exact cutoff; shown rounded
grid0 <- seq(.1, .9, by = .1); grid0 <- grid0[abs(grid0 - best) > .005] # skip round cutoffs that duplicate the Youden cutoff
ct <- as.data.frame(t(sapply(sort(c(grid0, best)), cls)))
ct$youden_J <- ct$sensitivity + ct$specificity - 1 # Youden's J at each cutoff
| Prob level | Correct: event | Correct: non-event | Incorrect: event | Incorrect: non-event | Correct % | Sensitivity % | Specificity % | Youden’s J |
|---|---|---|---|---|---|---|---|---|
| 0.100 | 1437 | 1115 | 2375 | 73 | 51.0 | 95.2 | 31.9 | 0.271 |
| 0.200 | 1282 | 1998 | 1492 | 228 | 65.6 | 84.9 | 57.2 | 0.421 |
| 0.300 | 1109 | 2577 | 913 | 401 | 73.7 | 73.4 | 73.8 | 0.473 |
| 0.400 | 910 | 2930 | 560 | 600 | 76.8 | 60.3 | 84.0 | 0.442 |
| 0.500 | 725 | 3169 | 321 | 785 | 77.9 | 48.0 | 90.8 | 0.388 |
| 0.600 | 541 | 3319 | 171 | 969 | 77.2 | 35.8 | 95.1 | 0.309 |
| 0.700 | 378 | 3407 | 83 | 1132 | 75.7 | 25.0 | 97.6 | 0.227 |
| 0.800 | 221 | 3462 | 28 | 1289 | 73.7 | 14.6 | 99.2 | 0.138 |
| 0.900 | 62 | 3486 | 4 | 1448 | 71.0 | 4.1 | 99.9 | 0.040 |
Interpretation. At the default 0.50 cutoff the model classifies 77.9% of prospects correctly, finding 48.0% of buyers (sensitivity) and correctly clearing 90.8% of non-buyers (specificity). Because the agency values finding buyers and screening out non-buyers equally, the cutoff that maximizes sensitivity + specificity (Youden’s J) is a natural choice: 0.300, giving accuracy 73.7%, sensitivity 73.4% and specificity 73.8%. Judge accuracy against the 69.8% a model that predicts “no” for everyone achieves; because only 30.2% of prospects bought, 0.50 is not a neutral cutoff.
Youden’s J = sensitivity + specificity − 1. It runs from 0 (no better than guessing) to 1 (perfect), and the cutoff with the largest J gives the best balance between finding buyers and screening out non-buyers, with the two error rates weighted equally.
best_row <- coords(roc_fit, "best", best.method = "youden",
ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)
best_row$J <- best_row$sensitivity + best_row$specificity - 1 # Youden's J
best_row
threshold sensitivity specificity J
1 0.2999379 0.7344371 0.7383954 0.4728325
# J at a few cutoffs (pROC's ret = "youden" reports sensitivity + specificity, so subtract 1)
cuts <- c(0.30, 0.40, 0.50, 0.60, 0.70)
cuts <- sort(c(cuts[abs(cuts - best_row$threshold[1]) > .005], best_row$threshold[1])) # add the Youden cutoff
yd <- coords(roc_fit, x = cuts, input = "threshold",
ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)
yd$J <- yd$sensitivity + yd$specificity - 1
round(yd, 3)
threshold sensitivity specificity J
1 0.3 0.734 0.738 0.473
2 0.4 0.603 0.840 0.442
3 0.5 0.480 0.908 0.388
4 0.6 0.358 0.951 0.309
5 0.7 0.250 0.976 0.227
plot(roc_fit, legacy.axes = TRUE, print.thres = "best",
print.thres.best.method = "youden", main = "ROC curve with the Youden cutoff")
Interpretation. Youden’s J is largest, 0.473, at a cutoff of 0.300. There the model finds 73.4% of buyers and clears 73.8% of non-buyers. At the default 0.50 cutoff J is 0.388, noticeably lower, so the default cutoff gives up some balance. On the plot, the marked point is the spot on the ROC curve farthest above the diagonal. J weights the miss rate and the false-alarm rate equally. Because buyers are the smaller group, that treats each missed buyer as costing about 2.3 wasted calls (non-buyers ÷ buyers = 3,490 ÷ 1,510). If the agency’s real costs differ, it should choose the cutoff from those costs instead.
Think of the cutoff as deciding who goes on the agency’s call list. Youden’s J asks one question: how much more likely is a prospect who would buy to make the list than a prospect who would not?
Reasons it fits this case:
coords().When J is the wrong tool:
Each method below picks one cutoff from the same model. “Called” is the share of all prospects the agency would phone at that cutoff.
grid <- coords(roc_fit, "all", ret = "threshold", transpose = FALSE)$threshold
grid <- grid[is.finite(grid)] # every distinct cutoff the data allow
m <- function(t) {
pred <- p >= t
tp <- sum(pred & y == 1); fp <- sum(pred & y == 0); fn <- sum(!pred & y == 1); tn <- sum(!pred & y == 0)
sens <- tp / (tp + fn); spec <- tn / (tn + fp); prec <- tp / (tp + fp)
c(cutoff = t, called = mean(pred), sensitivity = sens, specificity = spec, accuracy = (tp + tn) / n,
J = sens + spec - 1, precision = prec, F1 = 2 * prec * sens / (prec + sens), fn = fn, fp = fp)
}
G <- as.data.frame(t(sapply(grid, m)))
cost_fn <- 5; cost_fp <- 1 # ILLUSTRATIVE ONLY: a missed sale costs 5 times a wasted call
G$cost <- cost_fn * G$fn + cost_fp * G$fp
topleft <- coords(roc_fit, "best", best.method = "closest.topleft", ret = "threshold", transpose = FALSE)$threshold[1]
pick <- c("Default 0.50" = 0.5,
"Youden's J (largest J)" = G$cutoff[which.max(G$J)],
"Closest to the top-left corner" = topleft,
"Highest accuracy" = G$cutoff[which.max(G$accuracy)],
"Sensitivity = specificity" = G$cutoff[which.min(abs(G$sensitivity - G$specificity))],
"Catch at least 80% of buyers" = max(G$cutoff[G$sensitivity >= .80]),
"Highest F1 (precision and recall)" = G$cutoff[which.max(G$F1)],
"Lowest cost (missed = 5 x a wasted call)" = G$cutoff[which.min(G$cost)])
cmp <- as.data.frame(t(sapply(pick, m)))
| Method | Cutoff | Called | Sensitivity | Specificity | Accuracy | Youden’s J | Precision |
|---|---|---|---|---|---|---|---|
| Default 0.50 | 0.500 | 20.9% | 48.0% | 90.8% | 77.9% | 0.388 | 69.3% |
| Youden’s J (largest J) | 0.300 | 40.4% | 73.4% | 73.8% | 73.7% | 0.473 | 54.8% |
| Closest to the top-left corner | 0.300 | 40.4% | 73.4% | 73.8% | 73.7% | 0.473 | 54.8% |
| Highest accuracy | 0.536 | 18.5% | 44.6% | 92.8% | 78.2% | 0.374 | 72.8% |
| Sensitivity = specificity | 0.298 | 40.7% | 73.5% | 73.5% | 73.5% | 0.470 | 54.5% |
| Catch at least 80% of buyers | 0.247 | 48.1% | 80.0% | 65.7% | 70.0% | 0.457 | 50.2% |
| Highest F1 (precision and recall) | 0.300 | 40.4% | 73.4% | 73.8% | 73.7% | 0.473 | 54.8% |
| Lowest cost (missed = 5 x a wasted call) | 0.157 | 63.3% | 90.2% | 48.3% | 60.9% | 0.385 | 43.0% |
| Method | What it optimizes | Use it when |
|---|---|---|
| Default 0.50 | Nothing; a convention | Only as a starting point; it ignores costs and the base rate |
| Youden’s J | Sensitivity + specificity, equally weighted | Finding buyers and screening out non-buyers matter equally, as in this brief |
| Closest to the top-left corner | Distance to the perfect corner of the ROC plot | Rarely; J is usually preferable (Perkins & Schisterman, 2006) |
| Highest accuracy | Share of all prospects classified correctly | Classes are balanced and errors cost the same |
| Sensitivity = specificity | Equal catch rate and clear rate | You want a symmetric, easy-to-explain rule |
| Minimum sensitivity (here 80%) | A required catch rate | Missing likely buyers is unacceptable beyond a set level |
| Highest F1 | Balance of precision and recall | The positive class is rare and true negatives matter little |
| Lowest expected cost | Dollar cost of errors | You can put a value on each kind of error (the best choice when you can) |
Interpretation.
Youden’s J, the top-left rule and highest accuracy choose cutoffs of 0.300, 0.300 and 0.536. Accuracy picks a higher cutoff and calls only 18.5% of prospects, against 40.4% for J: because most prospects do not buy, accuracy rewards predicting “no” and finds only 44.6% of buyers.
Requiring an 80% catch rate lowers the cutoff to 0.247: the agency would call 48.1% of prospects and reach 80.0% of buyers, but specificity falls to 65.7%.
If a missed sale cost five times a wasted call (an illustrative assumption, not a figure from the agency), the cost-minimizing cutoff is 0.157, reaching 90.2% of buyers. Compared with Youden’s J (0.300), the agency would lower the cutoff and call 63.3% of prospects instead of 40.4%.
Highest F1 picks 0.300. F1 ignores prospects correctly left off the list, so it suits problems where the event is rare and true negatives matter little.
Youden’s J is itself a cost rule. Maximizing J gives the same cutoff as minimizing cost when a missed buyer costs 2.3 times a wasted call, the ratio of non-buyers to buyers. A cost ratio above that lowers the cutoff (more calls); one below it raises the cutoff.
Bottom line. For the brief as stated, with finding buyers and screening out non-buyers valued equally, Youden’s J is a sound, defensible choice. Once the agency can attach a value to a sale and a cost to a call, the cost-based cutoff should replace it.
age, income, married,
dependents, has_mortgage, smoker,
existing_customer, lead_source.Check the deck’s case-study slide against these results before class. In particular, compare its statement about which factors raise or lower the odds, and its AUC rating, with the odds ratios and AUC above.
Perkins, N. J., & Schisterman, E. F. (2006). The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 163(7), 670–675. https://doi.org/10.1093/aje/kwj063
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77. https://doi.org/10.1186/1471-2105-12-77
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3