Exec Summary

  1. The amount of impact interactions remain important, statistically.

  2. The threshold for impact interactions appears to be 4.

  3. No particular impact interaction stands out, as there is a lot of correlation amongst several of them: Events, SL Led, Expert Calls, and Onsites.

Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

  3. We include data from FY21 onwards.

  4. We look at unit decision rates, not $ Renewed.

ITS Analysis

Renewal Rate by Type, All Data
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 125 81 44 1.8 64.8%
Opt Out 79 73 6 12.2 92.4%
Total 204 154 50 3.1 75.5%
Renewal by Decision Type & Fiscal Year

We want to understand whether we can combine fiscal years, because they look similar, and therefore get more power for our statistical tests. We also want to understand whether Opt Outs behave differently from NNLOA and we have to separate them when we get further into the analysis.

  1. FY21 & FY22 looks different from the rest of the years, so going forward we’ll just look at the last 3 years.
  2. [NNLOA] Some variation, but the N isn’t too high, so a unit here or there can swing the #s.
  3. [Opt Out] Performance and volume seems relatively consistent, with FY25 performance a bit higher and volume lower.
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2021 23 10 13 0.8 43.5%
Need New LOA 2022 25 14 11 1.3 56.0%
Need New LOA 2023 22 18 4 4.5 81.8%
Need New LOA 2024 34 25 9 2.8 73.5%
Need New LOA 2025 21 14 7 2.0 66.7%
Opt Out 2021 10 9 1 9.0 90.0%
Opt Out 2022 24 21 3 7.0 87.5%
Opt Out 2023 19 18 1 18.0 94.7%
Opt Out 2024 15 14 1 14.0 93.3%
Opt Out 2025 11 11 0 Inf 100.0%
Total NA 204 154 50 3.1 75.5%
Renewal Rate by Type, Excluding CRP and <=FY22
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 33 24 9 2.7 72.7%
Opt Out 28 26 2 13.0 92.9%
Total 204 154 50 3.1 75.5%
Renewal Rate by Type and Fiscal Year
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2021 23 10 13 0.8 43.5%
Need New LOA 2022 25 14 11 1.3 56.0%
Need New LOA 2023 22 18 4 4.5 81.8%
Need New LOA 2024 34 25 9 2.8 73.5%
Need New LOA 2025 21 14 7 2.0 66.7%
Opt Out 2021 10 9 1 9.0 90.0%
Opt Out 2022 24 21 3 7.0 87.5%
Opt Out 2023 19 18 1 18.0 94.7%
Opt Out 2024 15 14 1 14.0 93.3%
Opt Out 2025 11 11 0 Inf 100.0%
Total NA 204 154 50 3.1 75.5%
Renewal Rate by Account Segment
  1. Some consistency in performance across segments:
  • Large Public & Private and Selective occupy high rates, but most partners are LPPs.
  • Regional Public underperforms the two above, but outperforms Regional Privates.
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
Canada 3 3 0 Inf 100.0%
Large Public & Private 62 51 11 4.6 82.3%
Regional Private 3 2 1 2.0 66.7%
Regional Public 16 11 5 2.2 68.8%
Selective 16 13 3 4.3 81.2%
Renewal by Decision Type & Segment
  1. [Large P&P]
  • Performance on NNLOA is relatively high.
  • Performance on Opt Outs is clearly what driving the entire percentage up.
  1. [Selective]
  • Best NNLOA performance.
  • Perfect Opt Out renewals.
  1. [Regional Publics]
  • NNLOA is a particular problem for Regional Public.
  • Opt Out Performance appears to be pretty solid, so they continue with the service until the NNLOA and then drop.
  1. [Regional Privates]
  • NNLOA better than Publics, but still under LPP and Selectives.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
Canada Need New LOA 3 3 0 Inf 100.0%
Large Public & Private Need New LOA 38 29 9 3.2 76.3%
Large Public & Private Opt Out 24 22 2 11.0 91.7%
Regional Private Need New LOA 3 2 1 2.0 66.7%
Regional Public Need New LOA 9 4 5 0.8 44.4%
Regional Public Opt Out 7 7 0 Inf 100.0%
Selective Need New LOA 11 8 3 2.7 72.7%
Selective Opt Out 5 5 0 Inf 100.0%
GRP 3-4
  1. Unlike AAS, no clear evidence of a bump from GRP 3-4.
  2. GRP 3-4 distribution largely large public & private, correlation here makes sense.
Renewal Ratio by Navigate Bundling
DecisionType GRP3-4 N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 33 24 9 2.7 72.7%
Need New LOA 1 31 22 9 2.4 71.0%
Opt Out 0 28 26 2 13.0 92.9%
Opt Out 1 8 8 0 Inf 100.0%
Impact Interaction Volume Histogram
  1. Lots of ITS partners w/o interactions!
  2. Solid, yet declining density between 1 and 7.
  3. Long tail, some super users.

##### Impact Interaction Volume Histogram Split by Decision

  1. Would say that the two distributions - Opt Out & NNLOA - are roughly equivalent.

Impact Interaction Volume Histogram Split by Year
  1. Fiscal Years 24 and 25 looks more similar to each other than FY23
  2. Some high utilizers in 24 and 25.
  3. Really checking to see if we can draw conclusion based on all of these years.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA 0 still renews!
  2. NNLOA at 4 seems to be the threshold, but could argue even higher to get more gains.
  3. Opt Out performance is invariant to Impact interaction volume.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 12 8 4 2.0 66.7%
Need New LOA 1 9 3 6 0.5 33.3%
Need New LOA 2 8 5 3 1.7 62.5%
Need New LOA 3 9 7 2 3.5 77.8%
Need New LOA 4 9 8 1 8.0 88.9%
Need New LOA 5 4 2 2 1.0 50.0%
Need New LOA 6 4 4 0 Inf 100.0%
Need New LOA 7 4 4 0 Inf 100.0%
Need New LOA 9 2 2 0 Inf 100.0%
Need New LOA 10 1 1 0 Inf 100.0%
Need New LOA 13 1 1 0 Inf 100.0%
Need New LOA 14 1 1 0 Inf 100.0%
Opt Out 0 11 10 1 10.0 90.9%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2 6 5 1 5.0 83.3%
Opt Out 3 3 3 0 Inf 100.0%
Opt Out 5 3 3 0 Inf 100.0%
Opt Out 6 2 2 0 Inf 100.0%
Opt Out 7 2 2 0 Inf 100.0%
Opt Out 9 1 1 0 Inf 100.0%
Opt Out 10 1 1 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 14 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, Just Fy24 & FY25
  1. NNLOA has so much variation to it at the lower numbers because of Low N.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 11 7 4 1.8 63.6%
Need New LOA 1 7 2 5 0.4 28.6%
Need New LOA 2 5 3 2 1.5 60.0%
Need New LOA 3 7 5 2 2.5 71.4%
Need New LOA 4 8 7 1 7.0 87.5%
Need New LOA 5 4 2 2 1.0 50.0%
Need New LOA 6 4 4 0 Inf 100.0%
Need New LOA 7 4 4 0 Inf 100.0%
Need New LOA 9 2 2 0 Inf 100.0%
Need New LOA 10 1 1 0 Inf 100.0%
Need New LOA 13 1 1 0 Inf 100.0%
Need New LOA 14 1 1 0 Inf 100.0%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2 4 3 1 3.0 75.0%
Opt Out 3 3 3 0 Inf 100.0%
Opt Out 5 1 1 0 Inf 100.0%
Opt Out 6 2 2 0 Inf 100.0%
Opt Out 7 2 2 0 Inf 100.0%
Opt Out 9 1 1 0 Inf 100.0%
Opt Out 10 1 1 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] 4+ still looks like the threshold with improving renewal at 7+
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 12 8 4 2.0 66.7%
Need New LOA 1 9 3 6 0.5 33.3%
Need New LOA 2-3 17 12 5 2.4 70.6%
Need New LOA 4-6 17 14 3 4.7 82.4%
Need New LOA 7+ 9 9 0 Inf 100.0%
Opt Out 0 11 10 1 10.0 90.9%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2-3 9 8 1 8.0 88.9%
Opt Out 4-6 5 5 0 Inf 100.0%
Opt Out 7+ 6 6 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned), Excluding CRP
  1. [NNLOA] 4+ still looks like the threshold with improving renewal at 7+
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 2 1 2.0 66.7%
Need New LOA 1 5 2 3 0.7 40.0%
Need New LOA 2-3 7 4 3 1.3 57.1%
Need New LOA 4-6 12 10 2 5.0 83.3%
Need New LOA 7+ 6 6 0 Inf 100.0%
Opt Out 0 10 9 1 9.0 90.0%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2-3 6 5 1 5.0 83.3%
Opt Out 4-6 3 3 0 Inf 100.0%
Opt Out 7+ 4 4 0 Inf 100.0%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] 6+ in the previous year looks pretty good.
  2. [Opt Out] Invariance.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 12 8 4 2.0 66.7%
Need New LOA 1 5 3 2 1.5 60.0%
Need New LOA 2 9 7 2 3.5 77.8%
Need New LOA 3 6 4 2 2.0 66.7%
Need New LOA 4 6 4 2 2.0 66.7%
Need New LOA 5 5 3 2 1.5 60.0%
Need New LOA 6 7 6 1 6.0 85.7%
Need New LOA 7 3 2 1 2.0 66.7%
Need New LOA 8 1 1 0 Inf 100.0%
Need New LOA 10 1 0 1 0.0 0.0%
Need New LOA 14 1 1 0 Inf 100.0%
Need New LOA NA 8 7 1 7.0 87.5%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 2 2 2 0 Inf 100.0%
Opt Out 4 1 1 0 Inf 100.0%
Opt Out 5 1 1 0 Inf 100.0%
Opt Out 6 1 1 0 Inf 100.0%
Opt Out 7 1 1 0 Inf 100.0%
Opt Out 8 1 1 0 Inf 100.0%
Opt Out NA 23 21 2 10.5 91.3%
Renewal by Previous Yr Impact Interaction Volume Binned & Decision

Note: If II_PrevYr_Volume_Bins = NA, then there was not an active contract in the previous year.

  1. Binning makes the previous conclusion a little suspicious. Could be 2+.
Renewal Ratio by Previous Year Impact Interaction
DecisionType II_PrevYr_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 12 8 4 2.0 66.7%
Need New LOA 1 5 3 2 1.5 60.0%
Need New LOA 2-3 15 11 4 2.8 73.3%
Need New LOA 4-6 18 13 5 2.6 72.2%
Need New LOA 7+ 6 4 2 2.0 66.7%
Need New LOA NA 8 7 1 7.0 87.5%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 2-3 2 2 0 Inf 100.0%
Opt Out 4-6 3 3 0 Inf 100.0%
Opt Out 7+ 2 2 0 Inf 100.0%
Opt Out NA 23 21 2 10.5 91.3%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table. Find this representation difficult to understand.

Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 0 6 4 2 2.0 66.7%
0 1 3 1 2 0.5 33.3%
0 2-3 3 3 0 Inf 100.0%
1 0 1 0 1 0.0 0.0%
1 1 2 1 1 1.0 50.0%
1 4-6 1 1 0 Inf 100.0%
1 7+ 1 1 0 Inf 100.0%
2-3 0 1 1 0 Inf 100.0%
2-3 2-3 7 3 4 0.8 42.9%
2-3 4-6 5 5 0 Inf 100.0%
2-3 7+ 2 2 0 Inf 100.0%
4-6 0 3 2 1 2.0 66.7%
4-6 1 2 0 2 0.0 0.0%
4-6 2-3 4 4 0 Inf 100.0%
4-6 4-6 6 4 2 2.0 66.7%
4-6 7+ 3 3 0 Inf 100.0%
7+ 2-3 2 1 1 1.0 50.0%
7+ 4-6 3 2 1 2.0 66.7%
7+ 7+ 1 1 0 Inf 100.0%
NA 0 1 1 0 Inf 100.0%
NA 1 2 1 1 1.0 50.0%
NA 2-3 1 1 0 Inf 100.0%
NA 4-6 2 2 0 Inf 100.0%
NA 7+ 2 2 0 Inf 100.0%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. Significant. More impact interactions does translate to a higher percentage renewal rate.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept -0.02 0.433 -0.04 0.968
Impact Interaction Volume 0.36 0.146 2.44 0.015
Penetration by Event Grouping
  1. More heavily distributed towards events.
  2. The rest of the categories, which are consumed, are penetrated about the same amount.
  3. Opt Outs and NNLOA look distributed similarly across the interactions.
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 64.1% 1.6% 32.8% 28.1% 31.2% 0.0% 28.1%
Opt Out 47.2% 0.0% 27.8% 19.4% 22.2% 0.0% 38.9%
Renewal by Event Consumption
  1. [NNLOA] Small bump from attending, ~66% distributed towards attending.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 23 16 7 2.3 69.6%
Need New LOA 1 41 30 11 2.7 73.2%
Opt Out 0 19 18 1 18.0 94.7%
Opt Out 1 17 16 1 16.0 94.1%
Renewal by Service Consumption
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 63 45 18 2.5 71.4%
Need New LOA 1 1 1 0 Inf 100.0%
Opt Out 0 36 34 2 17.0 94.4%
Renewal by SL Consumption
  1. [NNLOA] Large increase, 16%, if this is consumed vs. not.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 43 28 15 1.9 65.1%
Need New LOA 1 21 18 3 6.0 85.7%
Opt Out 0 26 24 2 12.0 92.3%
Opt Out 1 10 10 0 Inf 100.0%
Renewal by Research Interview Consumption
  1. [NNLOA] Very large lift, nearly 20% if consumed vs. not. Less 33% have consumed this.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 46 30 16 1.9 65.2%
Need New LOA 1 18 16 2 8.0 88.9%
Opt Out 0 29 28 1 28.0 96.6%
Opt Out 1 7 6 1 6.0 85.7%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Another very large increase, 22%, if consumed. Again, less than 30% consumed this.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 44 28 16 1.8 63.6%
Need New LOA 1 20 18 2 9.0 90.0%
Opt Out 0 28 26 2 13.0 92.9%
Opt Out 1 8 8 0 Inf 100.0%
Renewal by Experience Consumption
  1. [NNLOA] No Data.
Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 64 46 18 2.6 71.9%
Opt Out 0 36 34 2 17.0 94.4%
Renewal by Expert Call Consumption
  1. [NNLOA] Small lift, 4%, especially in comparison to other impact interactions. Less than 33% consuming this, again.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 46 33 13 2.5 71.7%
Need New LOA 1 18 13 5 2.6 72.2%
Opt Out 0 22 20 2 10.0 90.9%
Opt Out 1 14 14 0 Inf 100.0%
Renewal by Benchmark Consumption
Renewal Ratio by Benchmark Consumption
DecisionType II_Benchmark_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 64 46 18 2.6 71.9%
Opt Out 0 36 34 2 17.0 94.4%
Renewal by AskEAB Consumption
Renewal Ratio by AskEAB Consumption
DecisionType II_AskEAB_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 64 46 18 2.6 71.9%
Opt Out 0 36 34 2 17.0 94.4%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Correlation with Renewal: Events, Research Interview, and SL Led
  2. Nothing really negatively correlated with renewal rate.
  3. A few interactions are correlated with each other: SL Led, Events, PLW/Onsite, Expert Call.
  4. SL Led seems to be correlated with many of the other interactions, differing from AAS.

Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only.

  1. Too much correlation between everything, so nothing stands out.
  2. Onsite effect is likely the largest.
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) 0.68 0.348 1.95 0.05
II_Events_Volume 0.15 0.183 0.84 0.40
II_ResearchInterview_Volume 0.34 0.662 0.51 0.61
II_PLW_Onsite_Volume 0.94 0.752 1.26 0.21
II_ExpertCall_Volume 0.09 0.370 0.25 0.80
II_Service_Volume 14.24 1455.398 0.01 0.99
II_SLLed_Volume 0.63 0.660 0.95 0.34