Exec Summary

  1. There is strong statistical evidence to support the importance of impact interactions on the likelihood of renewal.

  2. The threshold for impact interactions appears to be 4-6.

  3. Events, Research Interviews, K-12 volume popped as both statistically significant and correlated with utilization of other IIs.

Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

  3. We include data from FY21 onwards.

  4. We look at unit decision rates, not $ Renewed.

K-12 Year Analysis

Decision Rate by Type, Overall
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 227 134 93 1.4 59.0%
Opt Out 154 127 27 4.7 82.5%
Total 381 261 120 2.2 68.5%
Renewal by Decision Type & Fiscal Year

We want to understand whether we can combine fiscal years, because they look similar, and therefore get more power for our statistical tests. We also want to understand whether Opt Outs behave differently from NNLOA and we have to separate them when we get further into the analysis.

  1. [NNLOA] FY21 NNLOA stands out as an outlier. There is decent variation amongst the years, with the past couple of FYs being the worst performers.
  2. [Opt Out] Generally, opt outs perform much better than NNLOA, except FY25.
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2021 20 15 5 3.0 75.0%
Need New LOA 2022 55 33 22 1.5 60.0%
Need New LOA 2023 62 39 23 1.7 62.9%
Need New LOA 2024 53 26 27 1.0 49.1%
Need New LOA 2025 37 21 16 1.3 56.8%
Opt Out 2021 27 23 4 5.8 85.2%
Opt Out 2022 40 33 7 4.7 82.5%
Opt Out 2023 36 30 6 5.0 83.3%
Opt Out 2024 34 32 2 16.0 94.1%
Opt Out 2025 17 9 8 1.1 52.9%
Total NA 381 261 120 2.2 68.5%
Impact Interaction Volume Histogram
  1. Peak at 0, with a lot also at 1.
  2. Solid density between 3-6.
  3. Decent tail at 10+

##### Impact Interaction Volume Histogram Split by Decision

  1. Would say that the two distributions - Opt Out & NNLOA - are roughly equivalent.

Impact Interaction Volume Histogram Split by Year
  1. The non-users are composed of nearly all of the FYs - least from 2021.
  2. FY25 has a decent number of the super users.
  3. FY25 also looks to be a bit more spread out.
  4. Really checking to see if we can draw conclusion based on all of these years. Think we can.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA <=2 does not renew. This is a red flag.
  2. [NNLOA] Looks like the threshold could be 4.
  3. [Opt Out] Looks like the threshold could be 2.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 25 6 19 0.3 24.0%
Need New LOA 1 24 8 16 0.5 33.3%
Need New LOA 2 21 6 15 0.4 28.6%
Need New LOA 3 33 16 17 0.9 48.5%
Need New LOA 4 25 17 8 2.1 68.0%
Need New LOA 5 22 16 6 2.7 72.7%
Need New LOA 6 28 20 8 2.5 71.4%
Need New LOA 7 13 12 1 12.0 92.3%
Need New LOA 8 5 5 0 Inf 100.0%
Need New LOA 9 7 5 2 2.5 71.4%
Need New LOA 10 4 4 0 Inf 100.0%
Need New LOA 11 4 4 0 Inf 100.0%
Need New LOA 12 1 1 0 Inf 100.0%
Need New LOA 13 4 4 0 Inf 100.0%
Need New LOA 14 1 0 1 0.0 0.0%
Need New LOA 15 3 3 0 Inf 100.0%
Need New LOA 16 2 2 0 Inf 100.0%
Need New LOA 17 1 1 0 Inf 100.0%
Need New LOA 19 3 3 0 Inf 100.0%
Need New LOA 22 1 1 0 Inf 100.0%
Opt Out 0 30 18 12 1.5 60.0%
Opt Out 1 23 18 5 3.6 78.3%
Opt Out 2 12 10 2 5.0 83.3%
Opt Out 3 8 7 1 7.0 87.5%
Opt Out 4 20 15 5 3.0 75.0%
Opt Out 5 13 13 0 Inf 100.0%
Opt Out 6 7 7 0 Inf 100.0%
Opt Out 7 9 9 0 Inf 100.0%
Opt Out 8 6 6 0 Inf 100.0%
Opt Out 9 10 9 1 9.0 90.0%
Opt Out 10 3 3 0 Inf 100.0%
Opt Out 11 2 2 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 13 3 3 0 Inf 100.0%
Opt Out 14 3 3 0 Inf 100.0%
Opt Out 15 1 0 1 0.0 0.0%
Opt Out 16 1 1 0 Inf 100.0%
Opt Out 19 1 1 0 Inf 100.0%
Opt Out 20 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, Just Fy24 & FY25
  1. [NNLoA] Similarly aligned with the above, FY24 & FY25 could have a threshold of 4, yet there are still some troublesome data points beyond 4.
  2. [Opt Out] Much less evidence here, as there are much fewer drops.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 14 3 11 0.3 21.4%
Need New LOA 1 12 2 10 0.2 16.7%
Need New LOA 2 7 2 5 0.4 28.6%
Need New LOA 3 12 6 6 1.0 50.0%
Need New LOA 4 7 5 2 2.5 71.4%
Need New LOA 5 7 3 4 0.8 42.9%
Need New LOA 6 9 5 4 1.2 55.6%
Need New LOA 7 3 2 1 2.0 66.7%
Need New LOA 8 1 1 0 Inf 100.0%
Need New LOA 9 3 3 0 Inf 100.0%
Need New LOA 10 2 2 0 Inf 100.0%
Need New LOA 11 3 3 0 Inf 100.0%
Need New LOA 13 2 2 0 Inf 100.0%
Need New LOA 15 3 3 0 Inf 100.0%
Need New LOA 16 1 1 0 Inf 100.0%
Need New LOA 19 3 3 0 Inf 100.0%
Need New LOA 22 1 1 0 Inf 100.0%
Opt Out 0 18 13 5 2.6 72.2%
Opt Out 1 3 3 0 Inf 100.0%
Opt Out 3 1 1 0 Inf 100.0%
Opt Out 4 7 3 4 0.8 42.9%
Opt Out 5 6 6 0 Inf 100.0%
Opt Out 6 2 2 0 Inf 100.0%
Opt Out 7 2 2 0 Inf 100.0%
Opt Out 8 2 2 0 Inf 100.0%
Opt Out 9 3 3 0 Inf 100.0%
Opt Out 13 2 2 0 Inf 100.0%
Opt Out 14 3 3 0 Inf 100.0%
Opt Out 15 1 0 1 0.0 0.0%
Opt Out 20 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] 4+ seems to have the most evidence for it.
  2. [Opt Out] 2+ seems to be the point at which you hit the average here.
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 25 6 19 0.3 24.0%
Need New LOA 1 24 8 16 0.5 33.3%
Need New LOA 2-3 54 22 32 0.7 40.7%
Need New LOA 4-6 75 53 22 2.4 70.7%
Need New LOA 7+ 49 45 4 11.2 91.8%
Opt Out 0 30 18 12 1.5 60.0%
Opt Out 1 23 18 5 3.6 78.3%
Opt Out 2-3 20 17 3 5.7 85.0%
Opt Out 4-6 40 35 5 7.0 87.5%
Opt Out 7+ 41 39 2 19.5 95.1%
Renewal by Impact Interaction Volume, (Binned), W AskEAB
  1. [NNLOA] 4+ seems to have the most evidence for it.
  2. [Opt Out] 2+ seems to be the point at which you hit the average here.
Renewal Ratio for Impact Interaction Volume w AskEAB
DecisionType II_Volume_Bins_wAskEAB N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 15 4 11 0.4 26.7%
Need New LOA 1 15 5 10 0.5 33.3%
Need New LOA 2-3 41 14 27 0.5 34.1%
Need New LOA 4-6 40 18 22 0.8 45.0%
Need New LOA 7+ 116 93 23 4.0 80.2%
Opt Out 0 25 15 10 1.5 60.0%
Opt Out 1 15 12 3 4.0 80.0%
Opt Out 2-3 13 10 3 3.3 76.9%
Opt Out 4-6 27 23 4 5.8 85.2%
Opt Out 7+ 74 67 7 9.6 90.5%
Renewal by Impact Interaction Volume, (Binned), No Pres Dev
  1. [NNLOA] 4+ seems to have the most evidence for it.
  2. [Opt Out] 2+ seems to be the point at which you hit the average here.
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins_LessPresDev N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 25 6 19 0.3 24.0%
Need New LOA 1 24 8 16 0.5 33.3%
Need New LOA 2-3 54 22 32 0.7 40.7%
Need New LOA 4-6 75 53 22 2.4 70.7%
Need New LOA 7+ 49 45 4 11.2 91.8%
Opt Out 0 30 18 12 1.5 60.0%
Opt Out 1 23 18 5 3.6 78.3%
Opt Out 2-3 20 17 3 5.7 85.0%
Opt Out 4-6 40 35 5 7.0 87.5%
Opt Out 7+ 41 39 2 19.5 95.1%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] Some consistency here, 4+ II in the previous year seems to be correlated.
  2. [Opt Out] You could say the same for Opt Outs, which is oddly higher than what might be required in the year leading up to the decision.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 20 8 12 0.7 40.0%
Need New LOA 1 22 7 15 0.5 31.8%
Need New LOA 2 29 16 13 1.2 55.2%
Need New LOA 3 26 11 15 0.7 42.3%
Need New LOA 4 24 17 7 2.4 70.8%
Need New LOA 5 18 11 7 1.6 61.1%
Need New LOA 6 14 11 3 3.7 78.6%
Need New LOA 7 7 7 0 Inf 100.0%
Need New LOA 8 8 5 3 1.7 62.5%
Need New LOA 9 9 8 1 8.0 88.9%
Need New LOA 10 6 3 3 1.0 50.0%
Need New LOA 11 3 3 0 Inf 100.0%
Need New LOA 12 1 0 1 0.0 0.0%
Need New LOA 13 3 1 2 0.5 33.3%
Need New LOA 14 2 1 1 1.0 50.0%
Need New LOA 16 1 1 0 Inf 100.0%
Need New LOA 20 1 1 0 Inf 100.0%
Need New LOA NA 33 23 10 2.3 69.7%
Opt Out 0 2 1 1 1.0 50.0%
Opt Out 1 7 6 1 6.0 85.7%
Opt Out 2 6 4 2 2.0 66.7%
Opt Out 3 2 0 2 0.0 0.0%
Opt Out 4 8 8 0 Inf 100.0%
Opt Out 5 7 6 1 6.0 85.7%
Opt Out 6 2 2 0 Inf 100.0%
Opt Out 7 5 3 2 1.5 60.0%
Opt Out 8 3 3 0 Inf 100.0%
Opt Out 9 5 5 0 Inf 100.0%
Opt Out 11 3 3 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 13 1 1 0 Inf 100.0%
Opt Out NA 102 84 18 4.7 82.4%
Renewal by Previous Yr Impact Interaction Volume Binned & Decision

Note: If II_PrevYr_Volume_Bins = NA, then there was not an active contract in the previous year.

  1. Cleaner look of the above confirms the 4+ range for both Opt Outs and NNLOA.
Renewal Ratio by Previous Year Impact Interaction
DecisionType II_PrevYr_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 20 8 12 0.7 40.0%
Need New LOA 1 22 7 15 0.5 31.8%
Need New LOA 2-3 55 27 28 1.0 49.1%
Need New LOA 4-6 56 39 17 2.3 69.6%
Need New LOA 7+ 41 30 11 2.7 73.2%
Need New LOA NA 33 23 10 2.3 69.7%
Opt Out 0 2 1 1 1.0 50.0%
Opt Out 1 7 6 1 6.0 85.7%
Opt Out 2-3 8 4 4 1.0 50.0%
Opt Out 4-6 17 16 1 16.0 94.1%
Opt Out 7+ 18 16 2 8.0 88.9%
Opt Out NA 102 84 18 4.7 82.4%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table.

  1. One thing that is interesting here, is dropoff. If you go from using to not using, you are very unlikely to renew.
  2. Increasing utilization appears to be a positive indicator (i.e. going from 2-3 to 4-6)
Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 0 11 4 7 0.6 36.4%
0 1 2 0 2 0.0 0.0%
0 2-3 4 1 3 0.3 25.0%
0 4-6 1 1 0 Inf 100.0%
0 7+ 2 2 0 Inf 100.0%
1 0 3 0 3 0.0 0.0%
1 1 3 1 2 0.5 33.3%
1 2-3 12 3 9 0.3 25.0%
1 4-6 3 2 1 2.0 66.7%
1 7+ 1 1 0 Inf 100.0%
2-3 0 4 1 3 0.3 25.0%
2-3 1 10 3 7 0.4 30.0%
2-3 2-3 14 5 9 0.6 35.7%
2-3 4-6 20 12 8 1.5 60.0%
2-3 7+ 7 6 1 6.0 85.7%
4-6 0 4 0 4 0.0 0.0%
4-6 1 2 0 2 0.0 0.0%
4-6 2-3 11 7 4 1.8 63.6%
4-6 4-6 26 19 7 2.7 73.1%
4-6 7+ 13 13 0 Inf 100.0%
7+ 0 2 0 2 0.0 0.0%
7+ 1 1 0 1 0.0 0.0%
7+ 2-3 8 5 3 1.7 62.5%
7+ 4-6 13 11 2 5.5 84.6%
7+ 7+ 17 14 3 4.7 82.4%
NA 0 1 1 0 Inf 100.0%
NA 1 6 4 2 2.0 66.7%
NA 2-3 5 1 4 0.2 20.0%
NA 4-6 12 8 4 2.0 66.7%
NA 7+ 9 9 0 Inf 100.0%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. Significant. More impact interactions does translate to a higher percentage renewal rate, with a decent sized impact.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept -1.12 0.270 -4.14 <0.001
Impact Interaction Volume 0.37 0.064 5.83 <0.001
Penetration by Event Grouping
  1. Events is the most heavily penetrated, with expert calls a close second.
  2. Opt Outs and NNLOA penetration roughly equivalent.
  3. Service and PLW/Onsite roughly the same.
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_ExpertCall Perc_K12
Need New LOA 67.8% 21.1% 1.8% 40.1% 22.0% 55.5% 11.0%
Opt Out 60.4% 15.6% 5.8% 39.0% 17.5% 53.2% 14.3%
Renewal by Event Consumption
  1. [NNLOA] About 11% lift from attending, vs. the average, yet not attending is a red flag.
  2. [Opt Out] Less impactful increase for Opt Out, 7% above the mean, still meaningful.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 73 25 48 0.5 34.2%
Need New LOA 1 154 109 45 2.4 70.8%
Opt Out 0 61 43 18 2.4 70.5%
Opt Out 1 93 84 9 9.3 90.3%
Renewal by Service Consumption
  1. [NNLOA] Much tigher around the mean, suggests not as much value from consuming this II.
  2. [Opt out] Yet, on Opt Outs you see something very different insofar as those who consume this II perform better than their counterparts who do not consume it.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 179 103 76 1.4 57.5%
Need New LOA 1 48 31 17 1.8 64.6%
Opt Out 0 130 105 25 4.2 80.8%
Opt Out 1 24 22 2 11.0 91.7%
Renewal by SL Consumption

N too small.

Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 223 131 92 1.4 58.7%
Need New LOA 1 4 3 1 3.0 75.0%
Opt Out 0 145 118 27 4.4 81.4%
Opt Out 1 9 9 0 Inf 100.0%
Renewal by Research Interview Consumption
  1. [NNLOA] Appears to be a somewhat effective renewal instrument for NNLOA, ~21% gap between utilization and not, plus 12% bump above the mean.
  2. [Opt OUt] Similar to the NNLOA, although a bit attenuated.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 136 69 67 1.0 50.7%
Need New LOA 1 91 65 26 2.5 71.4%
Opt Out 0 94 72 22 3.3 76.6%
Opt Out 1 60 55 5 11.0 91.7%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Appears to be a very effective renewal instrument for NNLOA, ~27% gap between utilization and not, plus 21% lift above the mean.
  2. [Opt OUt] Not nearly as much impact delivered for Opt Outs…somewhat tightly grouped around the mean.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 177 94 83 1.1 53.1%
Need New LOA 1 50 40 10 4.0 80.0%
Opt Out 0 127 103 24 4.3 81.1%
Opt Out 1 27 24 3 8.0 88.9%
Renewal by Expert Call Consumption
  1. [NNLOA] Consumption here doesn’t seem to add much lift…unlikely to have statistical significance.
  2. [Opt Out] 7% separation from the mean in either direction, somewhat meaningful, but not the largest effect we’ve seen.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 101 56 45 1.2 55.4%
Need New LOA 1 126 78 48 1.6 61.9%
Opt Out 0 72 53 19 2.8 73.6%
Opt Out 1 82 74 8 9.2 90.2%
Renewal by K12
  1. [NNLOA] Smaller N, although some impact. 17% above the mean when this is consumed.
  2. [Opt Out] 12% lift above the mean, but those who don’t consume are close to the mean. Could be a value add.
Renewal Ratio by K-12 Consumption
DecisionType II_K12_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 202 115 87 1.3 56.9%
Need New LOA 1 25 19 6 3.2 76.0%
Opt Out 0 132 106 26 4.1 80.3%
Opt Out 1 22 21 1 21.0 95.5%
Renewal by Professional Development
  1. [NNLOA] Smaller N, although some impact. 17% above the mean when this is consumed.
  2. [Opt Out] 12% lift above the mean, but those who don’t consume are close to the mean. Could be a value add.
Renewal Ratio by Professional Development consumption
DecisionType II_Professional_Development_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 57 31 26 1.2 54.4%
Need New LOA 1 7 6 1 6.0 85.7%
Opt Out 0 30 22 8 2.8 73.3%
Opt Out 1 3 2 1 2.0 66.7%
Renewal by AskEAB
Renewal Ratio by AskEAB consumption
DecisionType II_AskEAB_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 65 29 36 0.8 44.6%
Need New LOA 1 162 105 57 1.8 64.8%
Opt Out 0 52 37 15 2.5 71.2%
Opt Out 1 102 90 12 7.5 88.2%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Correlation with Renewal: Events have the strongest correlation, followed by Expert Calls and PLW/Onsite.
  2. No negatively correlated impact interactions with renewal rate.
  3. Events are correlated, positively, with many of the other impact interactions, especially PLWs.
  4. Lots of correlation between Research Interviews, PLWs, Expert Calls, and Events.

Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only.

  1. Events and Research Interviews are the two winners.
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) -0.31 0.193 -1.63 0.10
II_Events_Volume 0.26 0.071 3.69 0.00
II_ResearchInterview_Volume 0.60 0.202 2.96 0.00
II_PLW_Onsite_Volume 0.13 0.208 0.65 0.52
II_ExpertCall_Volume 0.01 0.134 0.08 0.94
II_Service_Volume 0.04 0.240 0.18 0.85
II_SLLed_Volume 1.01 1.107 0.91 0.36
II_K12_Volume 0.29 0.192 1.50 0.13
II_AskEAB_Volume 0.09 0.042 2.07 0.04