1 What this file does

The L7 live-session deck fits one logistic regression to Insurance_Calls.csv. This file reports the main results for that model, in the order an analyst reads them:

  1. Response profile
  2. Model fit statistics (−2 Log L, AIC, BIC)
  3. Testing the global null hypothesis (likelihood ratio, score, Wald)
  4. Pseudo-R² (McFadden, from the lecture, plus Cox–Snell and Nagelkerke)
  5. Type 3 analysis of effects
  6. Analysis of maximum likelihood estimates
  7. Odds ratio estimates with 95% Wald confidence limits
  8. Association of predicted probabilities and observed responses (concordance, Somers’ D, Gamma, Tau-a, c)
  9. ROC curve and AUC
  10. Hosmer–Lemeshow goodness of fit
  11. Classification table

Each section shows the R code, the result, and a written interpretation that is generated from the numbers, so the text stays correct if the data change.

2 Data and model

library(readr)
calls <- read_csv("C:/Users/erl222/OneDrive - University of Texas at San Antonio/Desktop/BAN 3073/3. BAN 3073 Course notes/L7. Logistic regression/L7. Live session (Part 1)/Insurance_Calls.csv")
#View(calls)

# The outcome must be 0/1 or a factor whose second level is the event.
if (is.character(calls$purchased) || is.factor(calls$purchased)) {
  calls$purchased <- factor(tolower(as.character(calls$purchased)), levels = c("no", "yes"))
}

fit <- glm(purchased ~ age + income + married + dependents + has_mortgage + smoker +
                       existing_customer + lead_source,
           data = calls, family = binomial)

n   <- nrow(calls)
y   <- as.numeric(calls$purchased) - if (is.factor(calls$purchased)) 1 else 0
n1  <- sum(y == 1); n0 <- sum(y == 0)
k   <- length(coef(fit))                 # parameters including the intercept
ev  <- if (is.factor(calls$purchased)) "yes" else "1"   # the event level

The file has 5,000 prospects and 0 missing values. Every category of every predictor has hundreds of prospects, so no coefficient rests on a handful of cases. Every predictor is something the agency knows before it calls: age (years), income (household income in thousands of dollars), married, dependents (children under 18), has_mortgage, smoker, existing_customer (already holds another policy with the agency) and lead_source (how the prospect reached the agency).

R uses the first level of each categorical predictor, in alphabetical order, as its reference level. In this model those are:

Predictor Reference level
married No
has_mortgage No
smoker No
existing_customer No
lead_source Purchased list

2.1 Response profile

Ordered value purchased Total frequency
1 0 3490
2 1 1510

R models the probability that purchased = 1 (bought a policy). 30.2% of prospects bought. A model that predicts the larger class for everyone would be right 69.8% of the time, the benchmark accuracy must beat.

3 Is the model better than no model?

3.1 Model fit statistics

m0  <- glm(purchased ~ 1, data = calls, family = binomial)   # intercept only
m2LL <- c(intercept_only = -2 * as.numeric(logLik(m0)),  with_covariates = -2 * as.numeric(logLik(fit)))
AICs <- c(AIC(m0), AIC(fit))
SCs  <- m2LL + c(1, k) * log(n)                                  # BIC
Criterion Intercept only Intercept and covariates
AIC 6,127.49 4,729.05
BIC 6,134.01 4,794.22
−2 Log L 6,125.49 4,709.05

Interpretation. −2 Log L is the deviance: lower means less misfit. Adding the 9 predictor terms lowers −2 Log L from 6,125.49 to 4,709.05. AIC and BIC add a penalty for the number of parameters. Both fall, so the improvement in fit is more than the extra parameters cost. These statistics are for comparing models fitted to the same data; their size on its own has no benchmark.

3.2 Testing the global null hypothesis: β = 0

LR    <- m2LL[1] - m2LL[2]                                          # likelihood ratio
score <- anova(m0, fit, test = "Rao")$Rao[2]                         # score (Rao) test
b <- coef(fit)[-1]; V <- vcov(fit)[-1, -1]
wald  <- as.numeric(t(b) %*% solve(V) %*% b)                          # joint Wald test
df_g  <- k - 1
global <- data.frame(Test = c("Likelihood Ratio", "Score", "Wald"),
                     `Chi-Square` = c(LR, score, wald), DF = df_g,
                     `Pr > ChiSq` = pchisq(c(LR, score, wald), df_g, lower.tail = FALSE), check.names = FALSE)
Test Chi-Square DF Pr > ChiSq
Likelihood Ratio 1,416.44 9 <.0001
Score 1,271.16 9 <.0001
Wald 941.32 9 <.0001

Interpretation. All three tests ask the same question: do the predictors, taken together, improve on a model with no predictors? The likelihood-ratio test is the one taught in L7 (lrtest()): χ²(9) = 1,416.44, p < .001. The predictors jointly improve the model significantly, so the model is better than predicting the base rate for everyone. The score and Wald versions agree. A significant joint test does not show that every predictor matters; that is what the Type 3 table answers.

3.3 R-square

r2_cs   <- 1 - exp((m2LL[2] - m2LL[1]) / n)          # Cox&ndash;Snell R-square
r2_max  <- r2_cs / (1 - exp(-m2LL[1] / n))           # Nagelkerke (max-rescaled) R-square
r2_mcf  <- 1 - m2LL[2] / m2LL[1]                     # McFadden, as in the L7 lecture
round(c(cox_snell = unname(r2_cs), max_rescaled = unname(r2_max), mcfadden = unname(r2_mcf)), 4)
   cox_snell max_rescaled     mcfadden 
      0.2467       0.3493       0.2312 

Interpretation. McFadden’s pseudo-R² is 0.231. On the lecture’s scale (0 to about 0.4, with 0.2–0.4 already excellent), that is a strong fit. Two other versions, Cox–Snell (0.247) and Nagelkerke (0.349), are also common. None of these is the share of variance explained, so they should not be judged against OLS R² values.

4 Which predictors matter?

4.1 Type 3 analysis of effects

t3w  <- Anova(fit, type = 3, test.statistic = "Wald")   # Wald version
t3lr <- Anova(fit, type = 3, test.statistic = "LR")     # the L7 likelihood-ratio version
t3w  <- t3w[rownames(t3w) != "(Intercept)", ]
t3 <- data.frame(Effect = rownames(t3w), DF = t3w$Df, `Wald Chi-Square` = t3w$Chisq,
                 `Pr > ChiSq` = t3w$`Pr(>Chisq)`, `LR Chi-Square` = t3lr[rownames(t3w), "LR Chisq"],
                 `LR p` = t3lr[rownames(t3w), "Pr(>Chisq)"], check.names = FALSE)
Effect DF Wald Chi-Square Pr > ChiSq LR Chi-Square LR p
age 1 72.86 <.0001 75.17 <.0001
income 1 248.50 <.0001 271.61 <.0001
married 1 41.57 <.0001 41.90 <.0001
dependents 1 244.84 <.0001 259.58 <.0001
has_mortgage 1 61.51 <.0001 61.69 <.0001
smoker 1 93.35 <.0001 103.65 <.0001
existing_customer 1 178.56 <.0001 181.17 <.0001
lead_source 2 398.27 <.0001 440.46 <.0001

Interpretation. Each row tests one predictor as a whole, holding the others constant. A categorical predictor with several levels gets one test on several degrees of freedom. The table shows both the Wald and the likelihood-ratio versions; the L7 lecture uses the likelihood-ratio version, and the two lead to the same decision when they agree on which side of .05 each p-value falls.

  • Significant at the 5% level (likelihood-ratio test): age, income, married, dependents, has_mortgage, smoker, existing_customer, lead_source.
  • Not significant: none.

4.2 Which lead sources differ?

The Type 3 test for lead_source compares all three sources at once. The coefficient table compares each source only with the reference level (Purchased list). The remaining comparison, Quote request versus Referral, comes from the same model:

b_m <- coef(fit); V_m <- vcov(fit)
r1 <- "lead_sourceQuote request"; r2 <- "lead_sourceReferral"
est <- unname(b_m[r1] - b_m[r2])
se  <- unname(sqrt(V_m[r1, r1] + V_m[r2, r2] - 2 * V_m[r1, r2]))
sm  <- c(OR = exp(est), lower = exp(est - 1.96 * se), upper = exp(est + 1.96 * se), p = 2 * pnorm(-abs(est / se)))
linearHypothesis(fit, "lead_sourceQuote request = lead_sourceReferral")   # same test as a Wald chi-square
Linear hypothesis test

Hypothesis:
lead_sourceQuote request - lead_sourceReferral = 0

Model 1: restricted model
Model 2: purchased ~ age + income + married + dependents + has_mortgage + 
    smoker + existing_customer + lead_source

  Res.Df Df  Chisq          Pr(>Chisq)    
1   4991                                  
2   4990  1 59.618 0.00000000000001152 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Interpretation. Prospects who requested a quote have about 2.090 times the odds of buying that referred prospects have (95% CI 1.733 to 2.520, p < .001), holding the other predictors constant. So the three sources differ from one another, not only from the purchased list: prospects who asked for a quote are the strongest leads, which makes sense because they have already shown interest.

4.3 Analysis of maximum likelihood estimates

s   <- summary(fit)$coefficients
mle <- data.frame(Parameter = rownames(s), Estimate = s[, 1], `Standard Error` = s[, 2],
                  `Wald Chi-Square` = s[, 3]^2, `Pr > ChiSq` = s[, 4], check.names = FALSE)
Parameter Estimate Standard Error Wald Chi-Square Pr > ChiSq
(Intercept) -2.7096 0.1968 189.48 <.0001
age -0.0354 0.0041 72.86 <.0001
income 0.0158 0.0010 248.50 <.0001
marriedYes 0.5575 0.0865 41.57 <.0001
dependents 0.5506 0.0352 244.84 <.0001
has_mortgageYes 0.5866 0.0748 61.51 <.0001
smokerYes -0.9998 0.1035 93.35 <.0001
existing_customerYes 1.0842 0.0811 178.56 <.0001
lead_sourceQuote request 1.7763 0.0902 387.65 <.0001
lead_sourceReferral 1.0391 0.0899 133.59 <.0001

Interpretation. Each estimate is the change in the log-odds of buying for a one-unit increase in a numeric predictor, or for a category compared with its reference level, holding the other predictors constant. The Wald chi-square is the square of the z statistic in summary(), so the p-values are identical. The sign gives the direction; the size is easier to read as an odds ratio, below.

5 Odds ratio estimates

or <- data.frame(Effect = rownames(s)[-1], `Point Estimate` = exp(coef(fit))[-1],
                 `95% Wald Lower` = exp(confint.default(fit))[-1, 1],
                 `95% Wald Upper` = exp(confint.default(fit))[-1, 2], check.names = FALSE)

# Per-unit odds ratios for wide-ranging numeric predictors are close to 1, so also report a
# meaningful change
units <- c(age = 10, income = 10, dependents = 1)
or_units <- data.frame(Effect = names(units), Change = units,
                       `Odds ratio` = exp(coef(fit)[names(units)] * units),
                       Lower = exp(confint.default(fit)[names(units), 1] * units),
                       Upper = exp(confint.default(fit)[names(units), 2] * units), check.names = FALSE)
Odds ratio estimates per one-unit change or versus the reference level
Effect Point Estimate 95% Wald Lower 95% Wald Upper
age 0.9652 0.9574 0.9731
income 1.0160 1.0140 1.0180
marriedYes 1.7463 1.4741 2.0687
dependents 1.7342 1.6186 1.8580
has_mortgageYes 1.7978 1.5527 2.0816
smokerYes 0.3679 0.3004 0.4507
existing_customerYes 2.9571 2.5223 3.4669
lead_sourceQuote request 5.9077 4.9502 7.0503
lead_sourceReferral 2.8268 2.3701 3.3715
Odds ratios for a meaningful change in each numeric predictor
Effect Change Odds ratio Lower Upper
age 10 0.702 0.647 0.761
income 10 1.172 1.149 1.195
dependents 1 1.734 1.619 1.858

Interpretation. An odds ratio above 1 raises the odds of buying and one below 1 lowers them, holding the other predictors constant. A confidence interval that includes 1 is not significant at the 5% level. Odds ratios describe odds, not probabilities, and they describe association in observational data, not cause.

Do the directions make business sense? Yes. Prospects with more to protect (a spouse, children, a mortgage) and more to spend (higher income) have higher odds of buying, and so do warm leads: existing customers, referrals and, above all, prospects who asked for a quote. Smokers and older prospects have lower odds, which fits term life insurance, where their premiums are higher.

6 How well does the model rank prospects?

6.1 Association of predicted probabilities and observed responses

p  <- fitted(fit)
p1 <- p[y == 1]; p0 <- sort(p[y == 0])
lower <- findInterval(p1, p0, left.open = TRUE)          # non-buyers with a lower probability
le    <- findInterval(p1, p0)                             # ... lower or equal
conc <- sum(lower); tied <- sum(le - lower); pairs <- as.numeric(n1) * n0
disc <- pairs - conc - tied
assoc <- c(concordant = conc / pairs, discordant = disc / pairs, tied = tied / pairs, pairs = pairs,
           somers_d = (conc - disc) / pairs, gamma = (conc - disc) / (conc + disc),
           tau_a = (conc - disc) / (n * (n - 1) / 2), c = (conc + 0.5 * tied) / pairs)
Statistic Value Value
Percent Concordant 81.1 Somers’ D 0.622
Percent Discordant 18.9 Gamma 0.622
Percent Tied 0.0 Tau-a 0.262
Pairs 5,269,900 c 0.811

Interpretation. Concordance pairs every buyer with every non-buyer (5,269,900 pairs). A pair is concordant when the buyer has the higher predicted probability. The model ranks 81.1% of pairs correctly and 18.9% the wrong way. The c statistic, 0.811, counts ties as half and is the AUC. Somers’ D (0.622) equals 2c − 1, so it rescales the same information to run from −1 to 1.

6.2 ROC curve and AUC

calls$p_hat <- predict(fit, type = "response")            # predicted probability of buying
roc_fit <- roc(calls$purchased, calls$p_hat,
               levels = levels(factor(calls$purchased)),    # non-event first, event second
               direction = "<", quiet = TRUE)               # higher probability = more likely to buy
auc(roc_fit)                                                # area under the curve
Area under the curve: 0.8109
auc_ci  <- ci.auc(roc_fit); auc_ci                          # 95% CI (DeLong)
95% CI: 0.798-0.8238 (DeLong)
auc_val <- as.numeric(auc(roc_fit))
plot(roc_fit, legacy.axes = TRUE, print.auc = TRUE,         # x-axis = 1 - specificity
     main = "ROC curve: insurance call model")

Interpretation. AUC = 0.811 (95% CI 0.798 to 0.824). A randomly chosen buyer receives a higher predicted probability than a randomly chosen non-buyer about 81% of the time. On the course scale (.90–1.00 excellent, .80–.90 good, .70–.80 fair, .60–.70 poor, .50–.60 fail) the model’s discrimination is good. It equals the concordance statistic c (0.811) because both count tied pairs as half. This AUC is measured on the same prospects used to fit the model, so it is an in-sample value and is usually somewhat optimistic for new prospects.

7 Calibration: the Hosmer–Lemeshow test

hl <- performance_hosmer(fit, n_bins = 10)
hl
# Hosmer-Lemeshow Goodness-of-Fit Test

  Chi-squared: 2.560
           df: 8    
      p-value: 0.959

Interpretation. Hosmer–Lemeshow sorts prospects into 10 groups by predicted probability and compares predicted with observed buyers in each group. The null hypothesis is that they agree. χ²(8) = 2.56, p = .959. There is no evidence of lack of fit: predicted and observed purchase rates agree across the risk groups. That is not proof the model is correct, because the test has limited power.

7.1 Where do predicted and observed buyers disagree?

library(ResourceSelection)
hlg <- hoslem.test(calls$purchased, calls$p_hat, g = 10)   # same statistic, with the group table
hlg

    Hosmer and Lemeshow goodness of fit (GOF) test

data:  calls$purchased, calls$p_hat
X-squared = 2.5596, df = 8, p-value = 0.9589
grp <- data.frame(Group = 1:10, Prospects = rowSums(hlg$observed),
                  Observed = hlg$observed[, 2], Expected = hlg$expected[, 2])
grp$Gap <- grp$Observed - grp$Expected               # positive = model under-predicts
barplot(t(cbind(Observed = grp$Observed, Expected = grp$Expected)), beside = TRUE,
        names.arg = grp$Group, legend.text = TRUE, args.legend = list(x = "topleft"),
        xlab = "Risk group (1 = lowest predicted probability)", ylab = "Buyers",
        main = "Observed vs. expected buyers")

Observed and expected buyers in 10 risk groups (1 = lowest predicted probability)
Group Prospects Observed Expected Gap
1 500 20 16.6 3.4
2 500 34 34.0 0.0
3 500 52 52.4 -0.4
4 500 76 75.0 1.0
5 500 99 101.6 -2.6
6 500 133 133.9 -0.9
7 500 174 173.4 0.6
8 500 214 225.4 -11.4
9 500 305 296.2 8.8
10 500 403 401.5 1.5

Interpretation. The largest under-prediction is in group 9 (305 observed vs. 296 expected) and the largest over-prediction in group 8 (214 observed vs. 225 expected). The biggest gap is 11.4 buyers in a group of about 500 prospects. The gaps are small and change sign 6 times across the ten groups rather than following a pattern, which is what a well-calibrated model looks like: when it predicts a 30% chance, about 30% of those prospects buy.

8 Classification

cls <- function(cut) {
  pred <- as.integer(p >= cut)
  tp <- sum(pred == 1 & y == 1); tn <- sum(pred == 0 & y == 0); fp <- sum(pred == 1 & y == 0); fn <- sum(pred == 0 & y == 1)
  c(cutoff = cut, correct_events = tp, correct_nonevents = tn, incorrect_events = fp, incorrect_nonevents = fn,
    accuracy = (tp + tn) / n, sensitivity = tp / (tp + fn), specificity = tn / (tn + fp))
}
best <- coords(roc_fit, "best", best.method = "youden", ret = "threshold", transpose = TRUE)[1]   # exact cutoff; shown rounded
grid0 <- seq(.1, .9, by = .1); grid0 <- grid0[abs(grid0 - best) > .005]   # skip round cutoffs that duplicate the Youden cutoff
ct   <- as.data.frame(t(sapply(sort(c(grid0, best)), cls)))
ct$youden_J <- ct$sensitivity + ct$specificity - 1     # Youden's J at each cutoff
Prob level Correct: event Correct: non-event Incorrect: event Incorrect: non-event Correct % Sensitivity % Specificity % Youden’s J
0.100 1437 1115 2375 73 51.0 95.2 31.9 0.271
0.200 1282 1998 1492 228 65.6 84.9 57.2 0.421
0.300 1109 2577 913 401 73.7 73.4 73.8 0.473
0.400 910 2930 560 600 76.8 60.3 84.0 0.442
0.500 725 3169 321 785 77.9 48.0 90.8 0.388
0.600 541 3319 171 969 77.2 35.8 95.1 0.309
0.700 378 3407 83 1132 75.7 25.0 97.6 0.227
0.800 221 3462 28 1289 73.7 14.6 99.2 0.138
0.900 62 3486 4 1448 71.0 4.1 99.9 0.040

Interpretation. At the default 0.50 cutoff the model classifies 77.9% of prospects correctly, finding 48.0% of buyers (sensitivity) and correctly clearing 90.8% of non-buyers (specificity). Because the agency values finding buyers and screening out non-buyers equally, the cutoff that maximizes sensitivity + specificity (Youden’s J) is a natural choice: 0.300, giving accuracy 73.7%, sensitivity 73.4% and specificity 73.8%. Judge accuracy against the 69.8% a model that predicts “no” for everyone achieves; because only 30.2% of prospects bought, 0.50 is not a neutral cutoff.

8.1 Youden’s J: the balanced cutoff

Youden’s J = sensitivity + specificity − 1. It runs from 0 (no better than guessing) to 1 (perfect), and the cutoff with the largest J gives the best balance between finding buyers and screening out non-buyers, with the two error rates weighted equally.

best_row <- coords(roc_fit, "best", best.method = "youden",
                   ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)
best_row$J <- best_row$sensitivity + best_row$specificity - 1   # Youden's J
best_row
  threshold sensitivity specificity         J
1 0.2999379   0.7344371   0.7383954 0.4728325
# J at a few cutoffs (pROC's ret = "youden" reports sensitivity + specificity, so subtract 1)
cuts <- c(0.30, 0.40, 0.50, 0.60, 0.70)
cuts <- sort(c(cuts[abs(cuts - best_row$threshold[1]) > .005], best_row$threshold[1]))   # add the Youden cutoff
yd <- coords(roc_fit, x = cuts, input = "threshold",
             ret = c("threshold", "sensitivity", "specificity"), transpose = FALSE)
yd$J <- yd$sensitivity + yd$specificity - 1
round(yd, 3)
  threshold sensitivity specificity     J
1       0.3       0.734       0.738 0.473
2       0.4       0.603       0.840 0.442
3       0.5       0.480       0.908 0.388
4       0.6       0.358       0.951 0.309
5       0.7       0.250       0.976 0.227
plot(roc_fit, legacy.axes = TRUE, print.thres = "best",
     print.thres.best.method = "youden", main = "ROC curve with the Youden cutoff")

Interpretation. Youden’s J is largest, 0.473, at a cutoff of 0.300. There the model finds 73.4% of buyers and clears 73.8% of non-buyers. At the default 0.50 cutoff J is 0.388, noticeably lower, so the default cutoff gives up some balance. On the plot, the marked point is the spot on the ROC curve farthest above the diagonal. J weights the miss rate and the false-alarm rate equally. Because buyers are the smaller group, that treats each missed buyer as costing about 2.3 wasted calls (non-buyers ÷ buyers = 3,490 ÷ 1,510). If the agency’s real costs differ, it should choose the cutoff from those costs instead.

8.1.1 What Youden’s J means for the agency

Think of the cutoff as deciding who goes on the agency’s call list. Youden’s J asks one question: how much more likely is a prospect who would buy to make the list than a prospect who would not?

  • At the cutoff of 0.300, the list includes 73.4% of the prospects who buy (sensitivity) and 26.2% of the prospects who do not (1 − specificity, the false-alarm rate).
  • J is the gap between those two rates: 73.4% − 26.2% = 47.3 percentage points, or J = 0.473. A prospect who would buy is 47.3 percentage points more likely to be called than one who would not.
  • The same number is sensitivity + specificity − 1: 73.4% + 73.8% − 100% = 47.3%.

8.1.2 How to interpret J

  • J = 0: the call list catches buyers and non-buyers at the same rate, which is no better than picking prospects at random. On the ROC curve this is the diagonal.
  • J = 1: every buyer is called and no non-buyer is. This is the top-left corner of the ROC plot and does not happen with real data.
  • Between 0 and 1: the larger J is, the better the cutoff separates the two groups. A negative J means the cutoff is working backwards.
  • On the ROC plot, J at any cutoff is the vertical distance from that point on the curve down to the diagonal, so the largest J is the point farthest above the diagonal.
  • There is no fixed “good” J. Use J to compare cutoffs for the same model, or models on the same data, rather than against a universal benchmark.

8.1.3 Why use Youden’s J here, and when not to

Reasons it fits this case:

  1. It matches the brief. The agency wants to find as many buyers as it can and to screen out as many non-buyers as it can, and it values the two goals equally. J gives sensitivity and specificity exactly equal weight. In terms of individual calls, that means one missed buyer counts as much as 2.3 wasted calls, the ratio of non-buyers to buyers.
  2. It is not distorted by class balance. Sensitivity is computed among buyers only and specificity among non-buyers only, so J does not reward a model for guessing the larger group. Accuracy can: in the lecture’s fraud example, a model that flags no one is 98.7% accurate and catches no fraud. Here only 30.2% of prospects bought, so accuracy favors predicting “no”; J does not.
  3. It is better grounded than the closest-to-the-corner rule. Maximizing J is the same as minimizing the sum of the two error rates: the share of buyers missed plus the share of non-buyers called. The popular alternative, the point closest to the top-left corner, adds a squared term with no business meaning and can pick a cutoff with higher error rates (Perkins & Schisterman, 2006).
  4. It is simple and reproducible. One number per cutoff, computed directly from the ROC curve with coords().

When J is the wrong tool:

  • Unequal costs. If a missed buyer is worth far more than the cost of a call (or the reverse), J’s equal weights no longer match the business. Choose the cutoff that minimizes expected cost instead; the cost-based row in the table below shows how much that can change the answer.
  • A capacity limit. If the call center can make only a fixed number of calls, the cutoff is set by capacity: call the top-ranked prospects until the budget runs out.
  • A required catch rate. If the agency must reach, say, 80% of likely buyers, set the cutoff to meet that target and accept the false alarms that come with it.
  • New prospects. The J-optimal cutoff here is chosen on the same prospects used to fit the model, so it is somewhat optimistic. Check it on a holdout sample, or on the next campaign, before relying on it.

8.1.4 Other ways to choose a cutoff

Each method below picks one cutoff from the same model. “Called” is the share of all prospects the agency would phone at that cutoff.

grid <- coords(roc_fit, "all", ret = "threshold", transpose = FALSE)$threshold
grid <- grid[is.finite(grid)]                       # every distinct cutoff the data allow
m <- function(t) {
  pred <- p >= t
  tp <- sum(pred & y == 1); fp <- sum(pred & y == 0); fn <- sum(!pred & y == 1); tn <- sum(!pred & y == 0)
  sens <- tp / (tp + fn); spec <- tn / (tn + fp); prec <- tp / (tp + fp)
  c(cutoff = t, called = mean(pred), sensitivity = sens, specificity = spec, accuracy = (tp + tn) / n,
    J = sens + spec - 1, precision = prec, F1 = 2 * prec * sens / (prec + sens), fn = fn, fp = fp)
}
G <- as.data.frame(t(sapply(grid, m)))
cost_fn <- 5; cost_fp <- 1                          # ILLUSTRATIVE ONLY: a missed sale costs 5 times a wasted call
G$cost <- cost_fn * G$fn + cost_fp * G$fp
topleft <- coords(roc_fit, "best", best.method = "closest.topleft", ret = "threshold", transpose = FALSE)$threshold[1]
pick <- c("Default 0.50"                              = 0.5,
          "Youden's J (largest J)"                    = G$cutoff[which.max(G$J)],
          "Closest to the top-left corner"            = topleft,
          "Highest accuracy"                          = G$cutoff[which.max(G$accuracy)],
          "Sensitivity = specificity"                 = G$cutoff[which.min(abs(G$sensitivity - G$specificity))],
          "Catch at least 80% of buyers"         = max(G$cutoff[G$sensitivity >= .80]),
          "Highest F1 (precision and recall)"         = G$cutoff[which.max(G$F1)],
          "Lowest cost (missed = 5 x a wasted call)"  = G$cutoff[which.min(G$cost)])
cmp <- as.data.frame(t(sapply(pick, m)))
Cutoffs chosen by eight common methods (all prospects; in-sample)
Method Cutoff Called Sensitivity Specificity Accuracy Youden’s J Precision
Default 0.50 0.500 20.9% 48.0% 90.8% 77.9% 0.388 69.3%
Youden’s J (largest J) 0.300 40.4% 73.4% 73.8% 73.7% 0.473 54.8%
Closest to the top-left corner 0.300 40.4% 73.4% 73.8% 73.7% 0.473 54.8%
Highest accuracy 0.536 18.5% 44.6% 92.8% 78.2% 0.374 72.8%
Sensitivity = specificity 0.298 40.7% 73.5% 73.5% 73.5% 0.470 54.5%
Catch at least 80% of buyers 0.247 48.1% 80.0% 65.7% 70.0% 0.457 50.2%
Highest F1 (precision and recall) 0.300 40.4% 73.4% 73.8% 73.7% 0.473 54.8%
Lowest cost (missed = 5 x a wasted call) 0.157 63.3% 90.2% 48.3% 60.9% 0.385 43.0%
Method What it optimizes Use it when
Default 0.50 Nothing; a convention Only as a starting point; it ignores costs and the base rate
Youden’s J Sensitivity + specificity, equally weighted Finding buyers and screening out non-buyers matter equally, as in this brief
Closest to the top-left corner Distance to the perfect corner of the ROC plot Rarely; J is usually preferable (Perkins & Schisterman, 2006)
Highest accuracy Share of all prospects classified correctly Classes are balanced and errors cost the same
Sensitivity = specificity Equal catch rate and clear rate You want a symmetric, easy-to-explain rule
Minimum sensitivity (here 80%) A required catch rate Missing likely buyers is unacceptable beyond a set level
Highest F1 Balance of precision and recall The positive class is rare and true negatives matter little
Lowest expected cost Dollar cost of errors You can put a value on each kind of error (the best choice when you can)

Interpretation.

  • Youden’s J, the top-left rule and highest accuracy choose cutoffs of 0.300, 0.300 and 0.536. Accuracy picks a higher cutoff and calls only 18.5% of prospects, against 40.4% for J: because most prospects do not buy, accuracy rewards predicting “no” and finds only 44.6% of buyers.

  • Requiring an 80% catch rate lowers the cutoff to 0.247: the agency would call 48.1% of prospects and reach 80.0% of buyers, but specificity falls to 65.7%.

  • If a missed sale cost five times a wasted call (an illustrative assumption, not a figure from the agency), the cost-minimizing cutoff is 0.157, reaching 90.2% of buyers. Compared with Youden’s J (0.300), the agency would lower the cutoff and call 63.3% of prospects instead of 40.4%.

  • Highest F1 picks 0.300. F1 ignores prospects correctly left off the list, so it suits problems where the event is rare and true negatives matter little.

  • Youden’s J is itself a cost rule. Maximizing J gives the same cutoff as minimizing cost when a missed buyer costs 2.3 times a wasted call, the ratio of non-buyers to buyers. A cost ratio above that lowers the cutoff (more calls); one below it raises the cutoff.

Bottom line. For the brief as stated, with finding buyers and screening out non-buyers valued equally, Youden’s J is a sound, defensible choice. Once the agency can attach a value to a sale and a cost to a call, the cost-based cutoff should replace it.

9 Summary for the case study

Check the deck’s case-study slide against these results before class. In particular, compare its statement about which factors raise or lower the odds, and its AUC rating, with the odds ratios and AUC above.

10 References

Perkins, N. J., & Schisterman, E. F. (2006). The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 163(7), 670–675. https://doi.org/10.1093/aje/kwj063

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77. https://doi.org/10.1186/1471-2105-12-77

Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3