Exec Summary

  1. The amount of impact interactions remain important, statistically.

  2. The threshold for impact interactions appears to be 4.

  3. Events and Research Interviews appear to have the most enduring effects, statistically.

Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

  3. We include data from FY21 onwards.

  4. We look at unit decision rates, not $ Renewed.

AAS Analysis

Renewal by Decision Type
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 218 160 58 2.8 73.4%
Opt Out 121 110 11 10.0 90.9%
Total 339 270 69 3.9 79.6%
Renewal by Decision Type & Fiscal Year

We want to understand whether we can combine fiscal years, because they look similar, and therefore get more power for our statistical tests. We also want to understand whether Opt Outs behave differently from NNLOA and we have to separate them when we get further into the analysis.

  1. FY25 looks different from past years.
  2. [NNLOA] Bit of alternating years effect, starting higher, than lower, etc.
  3. [Opt Out] Performance and volume seems relatively consistent, with FY25 performance a bit higher and FY22 volume a bit higher.
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2021 29 23 6 3.8 79.3%
Need New LOA 2022 44 29 15 1.9 65.9%
Need New LOA 2023 48 38 10 3.8 79.2%
Need New LOA 2024 58 39 19 2.1 67.2%
Need New LOA 2025 39 31 8 3.9 79.5%
Opt Out 2021 15 14 1 14.0 93.3%
Opt Out 2022 40 36 4 9.0 90.0%
Opt Out 2023 28 25 3 8.3 89.3%
Opt Out 2024 23 21 2 10.5 91.3%
Opt Out 2025 15 14 1 14.0 93.3%
Total NA 339 270 69 3.9 79.6%
Renewal Rate by Account Segment
  1. Some consistency in performance across segments:
  • Large Public & Private and Selective occupy high rates and most of the partnership
  • Regional Public and Regional Private perform similarly.
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
Canada 20 15 5 3.0 75.0%
Large Public & Private 180 152 28 5.4 84.4%
Other 3 2 1 2.0 66.7%
Regional Private 15 10 5 2.0 66.7%
Regional Public 39 25 14 1.8 64.1%
Selective 80 64 16 4.0 80.0%
Sub-Department 2 2 0 Inf 100.0%
Renewal by Decision Type & Segment
  1. [Large P&P & Selectives]
  • Performance on NNLOA is relatively high.
  • Performance on Opt Outs is clearly what driving the entire percentage up.
  1. [Regional Publics]
  • NNLOA is a particular problem for Regional Public.
  • Opt Out Performance appears to be pretty solid, so they continue with the service until the NNLOA and then drop.
  1. [Regional Privates]
  • NNLOA is the lowest of the pack.
  • Opt Outs are too small of an N.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
Canada Need New LOA 16 12 4 3.0 75.0%
Canada Opt Out 4 3 1 3.0 75.0%
Large Public & Private Need New LOA 112 88 24 3.7 78.6%
Large Public & Private Opt Out 68 64 4 16.0 94.1%
Other Need New LOA 2 1 1 1.0 50.0%
Other Opt Out 1 1 0 Inf 100.0%
Regional Private Need New LOA 12 8 4 2.0 66.7%
Regional Private Opt Out 3 2 1 2.0 66.7%
Regional Public Need New LOA 20 9 11 0.8 45.0%
Regional Public Opt Out 19 16 3 5.3 84.2%
Selective Need New LOA 56 42 14 3.0 75.0%
Selective Opt Out 24 22 2 11.0 91.7%
Sub-Department Opt Out 2 2 0 Inf 100.0%
GRP 3-4
  1. Clear evidence that being GRP 3-4 is a bump regardless of decision type, about 7% in each decision type. Big enough N here.
  2. GRP 3-4 distribution largely large public & private, correlation here makes sense.
Renewal Ratio by Navigate Bundling
DecisionType GRP3-4 N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 176 126 50 2.5 71.6%
Need New LOA 1 42 34 8 4.2 81.0%
Opt Out 0 89 79 10 7.9 88.8%
Opt Out 1 32 31 1 31.0 96.9%
Impact Interaction Volume Histogram
  1. High impact interaction consumption with a peak at 5!
  2. Density between 2 and 5.
  3. Minimal low utilizers.
  4. Long tail.

##### Impact Interaction Volume Histogram Split by Decision

  1. Would say that the two distributions - Opt Out & NNLOA - are roughly equivalent, except NNLOA more represented in the long tail.

Impact Interaction Volume Histogram Split by Year
  1. Fiscal Years 24 and 25 looks more similar to each other than to FY21 and FY22 (which look similar to each other)
  2. Some high utilizers in 24 and 25.
  3. Really checking to see if we can draw conclusion based on all of these years. Might be some evidence to suggest lower volume might be sufficient.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA performance varies a lot due to a lot of buckets!
  2. NNLOA at 4 seems to be the threshold.
  3. Utilization <= 3 for NNLOA is deadly and 17 of the 68 partnerships fall in these buckets.
  4. Opt Out performance is invariant to Impact interaction volume.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 9 6 3 2.0 66.7%
Need New LOA 1 13 9 4 2.2 69.2%
Need New LOA 2 24 12 12 1.0 50.0%
Need New LOA 3 25 16 9 1.8 64.0%
Need New LOA 4 29 22 7 3.1 75.9%
Need New LOA 5 36 26 10 2.6 72.2%
Need New LOA 6 12 8 4 2.0 66.7%
Need New LOA 7 21 19 2 9.5 90.5%
Need New LOA 8 15 11 4 2.8 73.3%
Need New LOA 9 15 13 2 6.5 86.7%
Need New LOA 10 7 6 1 6.0 85.7%
Need New LOA 11 8 8 0 Inf 100.0%
Need New LOA 13 1 1 0 Inf 100.0%
Need New LOA 15 1 1 0 Inf 100.0%
Need New LOA 16 1 1 0 Inf 100.0%
Need New LOA 17 1 1 0 Inf 100.0%
Opt Out 0 6 4 2 2.0 66.7%
Opt Out 1 11 11 0 Inf 100.0%
Opt Out 2 15 14 1 14.0 93.3%
Opt Out 3 23 21 2 10.5 91.3%
Opt Out 4 12 10 2 5.0 83.3%
Opt Out 5 15 13 2 6.5 86.7%
Opt Out 6 11 11 0 Inf 100.0%
Opt Out 7 12 12 0 Inf 100.0%
Opt Out 8 9 8 1 8.0 88.9%
Opt Out 9 3 3 0 Inf 100.0%
Opt Out 10 3 2 1 2.0 66.7%
Opt Out 12 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, Just FY24 & FY25
  1. NNLOA at 4 seems to be the threshold, still.
  2. Opt Outs remain invariant.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 4 2 2 1.0 50.0%
Need New LOA 1 11 7 4 1.8 63.6%
Need New LOA 2 12 5 7 0.7 41.7%
Need New LOA 3 12 7 5 1.4 58.3%
Need New LOA 4 15 11 4 2.8 73.3%
Need New LOA 5 17 13 4 3.2 76.5%
Need New LOA 6 2 2 0 Inf 100.0%
Need New LOA 7 8 8 0 Inf 100.0%
Need New LOA 8 3 2 1 2.0 66.7%
Need New LOA 9 5 5 0 Inf 100.0%
Need New LOA 10 3 3 0 Inf 100.0%
Need New LOA 11 3 3 0 Inf 100.0%
Need New LOA 16 1 1 0 Inf 100.0%
Need New LOA 17 1 1 0 Inf 100.0%
Opt Out 0 2 1 1 1.0 50.0%
Opt Out 1 7 7 0 Inf 100.0%
Opt Out 2 6 6 0 Inf 100.0%
Opt Out 3 5 5 0 Inf 100.0%
Opt Out 4 4 4 0 Inf 100.0%
Opt Out 5 5 3 2 1.5 60.0%
Opt Out 6 1 1 0 Inf 100.0%
Opt Out 7 6 6 0 Inf 100.0%
Opt Out 9 1 1 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] 4+ still looks like the threshold with improving renewal at 7+
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 9 6 3 2.0 66.7%
Need New LOA 1 13 9 4 2.2 69.2%
Need New LOA 2-3 49 28 21 1.3 57.1%
Need New LOA 4-6 77 56 21 2.7 72.7%
Need New LOA 7+ 70 61 9 6.8 87.1%
Opt Out 0 6 4 2 2.0 66.7%
Opt Out 1 11 11 0 Inf 100.0%
Opt Out 2-3 38 35 3 11.7 92.1%
Opt Out 4-6 38 34 4 8.5 89.5%
Opt Out 7+ 28 26 2 13.0 92.9%
Renewal by Impact Interaction Volume, (Binned), wAskEAB
  1. [NNLOA] 7+ still looks like the threshold with improving renewal.
Renewal Ratio for Impact Interaction Volume w AskEAB
DecisionType II_Volume_Bins_wAskEAB N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 8 5 3 1.7 62.5%
Need New LOA 1 7 6 1 6.0 85.7%
Need New LOA 2-3 30 16 14 1.1 53.3%
Need New LOA 4-6 61 41 20 2.0 67.2%
Need New LOA 7+ 112 92 20 4.6 82.1%
Opt Out 0 5 3 2 1.5 60.0%
Opt Out 1 9 9 0 Inf 100.0%
Opt Out 2-3 23 22 1 22.0 95.7%
Opt Out 4-6 36 31 5 6.2 86.1%
Opt Out 7+ 48 45 3 15.0 93.8%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] 3+ in the previous year looks pretty good.
  2. [Opt Out] Invariance.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 11 5 6 0.8 45.5%
Need New LOA 1 14 7 7 1.0 50.0%
Need New LOA 2 17 9 8 1.1 52.9%
Need New LOA 3 21 16 5 3.2 76.2%
Need New LOA 4 21 15 6 2.5 71.4%
Need New LOA 5 22 14 8 1.8 63.6%
Need New LOA 6 27 23 4 5.8 85.2%
Need New LOA 7 15 13 2 6.5 86.7%
Need New LOA 8 18 18 0 Inf 100.0%
Need New LOA 9 10 8 2 4.0 80.0%
Need New LOA 10 5 4 1 4.0 80.0%
Need New LOA 11 6 5 1 5.0 83.3%
Need New LOA 12 1 1 0 Inf 100.0%
Need New LOA 15 2 2 0 Inf 100.0%
Need New LOA 19 2 1 1 1.0 50.0%
Need New LOA NA 26 19 7 2.7 73.1%
Opt Out 0 3 2 1 2.0 66.7%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2 9 8 1 8.0 88.9%
Opt Out 3 8 8 0 Inf 100.0%
Opt Out 4 4 4 0 Inf 100.0%
Opt Out 5 4 4 0 Inf 100.0%
Opt Out 6 7 6 1 6.0 85.7%
Opt Out 7 7 6 1 6.0 85.7%
Opt Out 8 5 4 1 4.0 80.0%
Opt Out 10 3 3 0 Inf 100.0%
Opt Out NA 66 60 6 10.0 90.9%
Renewal by Previous Yr Impact Interaction Volume Binned & Decision

Note: If II_PrevYr_Volume_Bins = NA, then there was not an active contract in the previous year.

  1. Previous Year interaction is a good indicator, likely correlated with continued use, and 2-3 seems to be good.
Renewal Ratio by Previous Year Impact Interaction
DecisionType II_PrevYr_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 11 5 6 0.8 45.5%
Need New LOA 1 14 7 7 1.0 50.0%
Need New LOA 2-3 38 25 13 1.9 65.8%
Need New LOA 4-6 70 52 18 2.9 74.3%
Need New LOA 7+ 59 52 7 7.4 88.1%
Need New LOA NA 26 19 7 2.7 73.1%
Opt Out 0 3 2 1 2.0 66.7%
Opt Out 1 5 5 0 Inf 100.0%
Opt Out 2-3 17 16 1 16.0 94.1%
Opt Out 4-6 15 14 1 14.0 93.3%
Opt Out 7+ 15 13 2 6.5 86.7%
Opt Out NA 66 60 6 10.0 90.9%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table. Find this representation difficult to understand.

Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 0 3 2 1 2.0 66.7%
0 1 2 1 1 1.0 50.0%
0 2-3 3 0 3 0.0 0.0%
0 4-6 2 1 1 1.0 50.0%
0 7+ 1 1 0 Inf 100.0%
1 0 1 1 0 Inf 100.0%
1 1 3 1 2 0.5 33.3%
1 2-3 4 0 4 0.0 0.0%
1 4-6 4 4 0 Inf 100.0%
1 7+ 2 1 1 1.0 50.0%
2-3 0 4 2 2 1.0 50.0%
2-3 1 2 1 1 1.0 50.0%
2-3 2-3 14 9 5 1.8 64.3%
2-3 4-6 12 7 5 1.4 58.3%
2-3 7+ 6 6 0 Inf 100.0%
4-6 1 2 2 0 Inf 100.0%
4-6 2-3 13 7 6 1.2 53.8%
4-6 4-6 34 23 11 2.1 67.6%
4-6 7+ 21 20 1 20.0 95.2%
7+ 0 1 1 0 Inf 100.0%
7+ 1 2 2 0 Inf 100.0%
7+ 2-3 9 8 1 8.0 88.9%
7+ 4-6 17 15 2 7.5 88.2%
7+ 7+ 30 26 4 6.5 86.7%
NA 1 2 2 0 Inf 100.0%
NA 2-3 6 4 2 2.0 66.7%
NA 4-6 8 6 2 3.0 75.0%
NA 7+ 10 7 3 2.3 70.0%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. Significant. More impact interactions does translate to a higher percentage renewal rate.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept 0.11 0.299 0.37 0.712
Impact Interaction Volume 0.19 0.058 3.27 0.001
Penetration by Event Grouping
  1. More heavily distributed towards events and services.
  2. Opt Outs and NNLOA look distributed similarly across the interactions.
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 72.0% 66.1% 42.7% 36.7% 36.2% 0.0% 55.5%
Opt Out 66.9% 61.2% 30.6% 31.4% 30.6% 0.0% 56.2%
Renewal by Event Consumption
  1. [NNLOA] 12% bump on NNLOA when they attend an event. N for not attending is getting small relative to those who attend.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 61 38 23 1.7 62.3%
Need New LOA 1 157 122 35 3.5 77.7%
Opt Out 0 40 35 5 7.0 87.5%
Opt Out 1 81 75 6 12.5 92.6%
Renewal by Service Consumption
  1. [NNLOA] Decent lift, 14%, when a service is consumed vs. not.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 74 45 29 1.6 60.8%
Need New LOA 1 144 115 29 4.0 79.9%
Opt Out 0 47 39 8 4.9 83.0%
Opt Out 1 74 71 3 23.7 95.9%
Renewal by SL Consumption
  1. [NNLOA] Negligible positive increase in rate. More haven’t had this than those who have had it.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 125 93 32 2.9 74.4%
Need New LOA 1 93 67 26 2.6 72.0%
Opt Out 0 84 77 7 11.0 91.7%
Opt Out 1 37 33 4 8.2 89.2%
Renewal by Research Interview Consumption
  1. [NNLOA] Relatively high lift above average NNLOA renewal rate - 12%; this has come down since last time (21%)
  2. [NNLOA] More not consuming this than are consuming it.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 138 94 44 2.1 68.1%
Need New LOA 1 80 66 14 4.7 82.5%
Opt Out 0 83 75 8 9.4 90.4%
Opt Out 1 38 35 3 11.7 92.1%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Close to a 11% bump above not having one; increased from 6% last time.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 139 96 43 2.2 69.1%
Need New LOA 1 79 64 15 4.3 81.0%
Opt Out 0 84 79 5 15.8 94.0%
Opt Out 1 37 31 6 5.2 83.8%
Renewal by Experience Consumption
  1. [NNLOA] No Data.
Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 218 160 58 2.8 73.4%
Opt Out 0 121 110 11 10.0 90.9%
Renewal by Expert Call Consumption
  1. [NNLOA] This used to have a positve lift of 10%, but now this looks to be negative vs. those who have not consumed it.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 97 73 24 3.0 75.3%
Need New LOA 1 121 87 34 2.6 71.9%
Opt Out 0 53 49 4 12.2 92.5%
Opt Out 1 68 61 7 8.7 89.7%
Renewal by AIP
  1. [NNLOA] Some lift, 6% above the mean, 16% difference vs. not.
Renewal Ratio by AIPI
DecisionType II_AIPI_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 80 51 29 1.8 63.7%
Need New LOA 1 138 109 29 3.8 79.0%
Opt Out 0 54 45 9 5.0 83.3%
Opt Out 1 67 65 2 32.5 97.0%
Renewal w AskEAB
  1. [NNLOA] Some lift, 6% above the mean, 16% difference vs. not.
Renewal Ratio by AskEAB
DecisionType II_AskEAB_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 66 44 22 2.0 66.7%
Need New LOA 1 152 116 36 3.2 76.3%
Opt Out 0 45 37 8 4.6 82.2%
Opt Out 1 76 73 3 24.3 96.1%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Correlation with Renewal: Events, Research Interview, and Service.
  2. Expert Calls slightly negatively correlated with renewal…close to 0.
  3. Events and Service correlated with each other and other impact interactions.
  4. SL Led doesn’t seem to be correlated with much else.
  5. AskEAB is weakly associated with renewal, and strongly associated with Service, Expert Calls, and PLW/Onsite

Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only.

  1. Research Interviews have the largest effect, barely significant if you draw the line at .1
  2. Events are signficant at p <= 0.5.
  3. Service has a decent effect size, but is not significant.
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) 0.08 0.310 0.26 0.79
II_Events_Volume 0.31 0.133 2.32 0.02
II_ResearchInterview_Volume 0.50 0.292 1.71 0.09
II_PLW_Onsite_Volume 0.16 0.211 0.78 0.44
II_ExpertCall_Volume -0.01 0.137 -0.04 0.97
II_Service_Volume 0.18 0.176 1.01 0.31
II_SLLed_Volume 0.03 0.219 0.15 0.88
II_AskEAB_Volume 0.03 0.072 0.42 0.68
Multivariate Regression sub in AIPI for Service

Note: Looking at NNLOA, only.

Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) 0.11 0.302 0.37 0.71
II_Events_Volume 0.32 0.133 2.40 0.02
II_ResearchInterview_Volume 0.50 0.293 1.71 0.09
II_PLW_Onsite_Volume 0.18 0.208 0.88 0.38
II_ExpertCall_Volume 0.00 0.134 0.01 0.99
II_SLLed_Volume 0.01 0.220 0.05 0.96
II_AIPI 0.23 0.202 1.16 0.25