Exec Summary

  1. There is strong statistical evidence to support the importance of impact interactions on the likelihood of renewal.

  2. The threshold for impact interactions appears to be 7+ for NNLOA and Opt Outs.

  3. Events and Expert Calls popped as both statistically significant (positive) and positively correlated with utilization of other IIs.

Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

  3. We include data from FY21 onwards.

  4. We look at unit decision rates, not $ Renewed.

2 Year Analysis

Renewal by Decision Type & Fiscal Year

We want to understand whether we can combine fiscal years, because they look similar, and therefore get more power for our statistical tests. We also want to understand whether Opt Outs behave differently from NNLOA and we have to separate them when we get further into the analysis.

  1. [NNLOA] FY22 & FY25 NNLOA are particularly low performing years. The rest of the FYs look closer to each other.
  2. [Opt Out] Increasing trend on renewal rate for Opt Outs, albeit on smaller unit count.
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2021 8 6 2 3.0 75.0%
Need New LOA 2022 35 19 16 1.2 54.3%
Need New LOA 2023 26 21 5 4.2 80.8%
Need New LOA 2024 27 19 8 2.4 70.4%
Need New LOA 2025 20 11 9 1.2 55.0%
Opt Out 2022 16 12 4 3.0 75.0%
Opt Out 2023 9 7 2 3.5 77.8%
Opt Out 2024 6 5 1 5.0 83.3%
Opt Out 2025 3 3 0 Inf 100.0%
Total NA 150 103 47 2.2 68.7%
Impact Interaction Volume Histogram
  1. Peak at 3.
  2. Solid density between 3-8.
  3. Decent tail at 10+
  4. Not many ‘non-users’.

##### Impact Interaction Volume Histogram Split by Decision

  1. Would say that the two distributions - Opt Out & NNLOA - are roughly equivalent.

Impact Interaction Volume Histogram Split by Year
  1. FY 22 looks to have a similar distribution to the other years.
  2. FY 25 has a decent number of the super users.
  3. Most of the non-utilizers from 23 & 24.
  4. Really checking to see if we can draw conclusion based on all of these years.Think we can.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA <=1 does not renew. This is a red flag.
  2. [NNLOA] Looks like the threshold could be 6.
  3. [Opt Out] Looks like the threshold could be 4.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 5 1 4 0.2 20.0%
Need New LOA 1 2 0 2 0.0 0.0%
Need New LOA 2 8 6 2 3.0 75.0%
Need New LOA 3 13 5 8 0.6 38.5%
Need New LOA 4 14 9 5 1.8 64.3%
Need New LOA 5 10 6 4 1.5 60.0%
Need New LOA 6 14 12 2 6.0 85.7%
Need New LOA 7 11 9 2 4.5 81.8%
Need New LOA 8 11 6 5 1.2 54.5%
Need New LOA 9 4 3 1 3.0 75.0%
Need New LOA 10 6 6 0 Inf 100.0%
Need New LOA 11 6 5 1 5.0 83.3%
Need New LOA 12 5 3 2 1.5 60.0%
Need New LOA 13 1 0 1 0.0 0.0%
Need New LOA 14 2 2 0 Inf 100.0%
Need New LOA 15 2 2 0 Inf 100.0%
Need New LOA 21 1 0 1 0.0 0.0%
Need New LOA 24 1 1 0 Inf 100.0%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 1 3 1 2 0.5 33.3%
Opt Out 3 4 3 1 3.0 75.0%
Opt Out 4 1 1 0 Inf 100.0%
Opt Out 5 5 4 1 4.0 80.0%
Opt Out 6 2 1 1 1.0 50.0%
Opt Out 7 2 2 0 Inf 100.0%
Opt Out 8 3 2 1 2.0 66.7%
Opt Out 9 2 1 1 1.0 50.0%
Opt Out 10 2 2 0 Inf 100.0%
Opt Out 11 1 1 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 13 1 1 0 Inf 100.0%
Opt Out 14 3 3 0 Inf 100.0%
Opt Out 16 1 1 0 Inf 100.0%
Opt Out 21 1 1 0 Inf 100.0%
Opt Out 23 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, Just Fy24 & FY25
  1. [NNLoA] Similarly aligned with the above, FY24 & FY25 could have a threshold of 6.
  2. [Opt Out] Much less evidence here, as there are much fewer drops.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 0 3 0.0 0.0%
Need New LOA 1 1 0 1 0.0 0.0%
Need New LOA 2 4 3 1 3.0 75.0%
Need New LOA 3 6 3 3 1.0 50.0%
Need New LOA 4 2 1 1 1.0 50.0%
Need New LOA 5 8 5 3 1.7 62.5%
Need New LOA 6 6 6 0 Inf 100.0%
Need New LOA 7 6 5 1 5.0 83.3%
Need New LOA 8 4 2 2 1.0 50.0%
Need New LOA 9 2 1 1 1.0 50.0%
Need New LOA 14 2 2 0 Inf 100.0%
Need New LOA 15 1 1 0 Inf 100.0%
Need New LOA 21 1 0 1 0.0 0.0%
Need New LOA 24 1 1 0 Inf 100.0%
Opt Out 1 1 1 0 Inf 100.0%
Opt Out 4 1 1 0 Inf 100.0%
Opt Out 5 2 1 1 1.0 50.0%
Opt Out 7 1 1 0 Inf 100.0%
Opt Out 10 1 1 0 Inf 100.0%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 16 1 1 0 Inf 100.0%
Opt Out 21 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] 7+ seems to have the most evidence for it.
  2. [Opt Out] 2+ looks ok, still, to climb above 79%, you’d need 7+, which seems exorbitant.
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 5 1 4 0.2 20.0%
Need New LOA 1 2 0 2 0.0 0.0%
Need New LOA 2-3 21 11 10 1.1 52.4%
Need New LOA 4-6 38 27 11 2.5 71.1%
Need New LOA 7+ 50 37 13 2.8 74.0%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 1 3 1 2 0.5 33.3%
Opt Out 2-3 4 3 1 3.0 75.0%
Opt Out 4-6 8 6 2 3.0 75.0%
Opt Out 7+ 18 16 2 8.0 88.9%
Renewal by Impact Interaction Volume, (Binned), W AskEAB
Renewal Ratio for Impact Interaction Volume with AskEAB
DecisionType II_Volume_Bins_WAskEAB N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 4 0 4 0.0 0.0%
Need New LOA 2-3 17 9 8 1.1 52.9%
Need New LOA 4-6 24 14 10 1.4 58.3%
Need New LOA 7+ 71 53 18 2.9 74.6%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 2-3 5 2 3 0.7 40.0%
Opt Out 4-6 5 4 1 4.0 80.0%
Opt Out 7+ 23 20 3 6.7 87.0%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] Some consistency here, 4+ II in the previous year seems to be correlated.
  2. [Opt Out] You could say the same for Opt Outs.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 6 2 4 0.5 33.3%
Need New LOA 1 11 6 5 1.2 54.5%
Need New LOA 2 11 4 7 0.6 36.4%
Need New LOA 3 8 5 3 1.7 62.5%
Need New LOA 4 9 7 2 3.5 77.8%
Need New LOA 5 19 13 6 2.2 68.4%
Need New LOA 6 11 8 3 2.7 72.7%
Need New LOA 7 9 6 3 2.0 66.7%
Need New LOA 8 4 4 0 Inf 100.0%
Need New LOA 9 1 0 1 0.0 0.0%
Need New LOA 10 5 5 0 Inf 100.0%
Need New LOA 11 2 2 0 Inf 100.0%
Need New LOA 13 3 2 1 2.0 66.7%
Need New LOA 14 2 2 0 Inf 100.0%
Need New LOA 18 1 1 0 Inf 100.0%
Need New LOA 21 1 1 0 Inf 100.0%
Need New LOA NA 13 8 5 1.6 61.5%
Opt Out 1 2 1 1 1.0 50.0%
Opt Out 2 1 0 1 0.0 0.0%
Opt Out 3 4 3 1 3.0 75.0%
Opt Out 4 2 2 0 Inf 100.0%
Opt Out 5 3 3 0 Inf 100.0%
Opt Out 6 1 0 1 0.0 0.0%
Opt Out 7 1 1 0 Inf 100.0%
Opt Out 8 2 2 0 Inf 100.0%
Opt Out 9 2 2 0 Inf 100.0%
Opt Out 10 1 0 1 0.0 0.0%
Opt Out 11 1 1 0 Inf 100.0%
Opt Out 14 1 1 0 Inf 100.0%
Opt Out NA 13 11 2 5.5 84.6%
Renewal by Previous Yr Impact Interaction Volume Binned & Decision

Note: If II_PrevYr_Volume_Bins = NA, then there was not an active contract in the previous year.

  1. Cleaner look of the above confirms the 4+ range for both Opt Outs and NNLOA.
Renewal Ratio by Previous Year Impact Interaction
DecisionType II_PrevYr_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 6 2 4 0.5 33.3%
Need New LOA 1 11 6 5 1.2 54.5%
Need New LOA 2-3 19 9 10 0.9 47.4%
Need New LOA 4-6 39 28 11 2.5 71.8%
Need New LOA 7+ 28 23 5 4.6 82.1%
Need New LOA NA 13 8 5 1.6 61.5%
Opt Out 1 2 1 1 1.0 50.0%
Opt Out 2-3 5 3 2 1.5 60.0%
Opt Out 4-6 6 5 1 5.0 83.3%
Opt Out 7+ 8 7 1 7.0 87.5%
Opt Out NA 13 11 2 5.5 84.6%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table.

  1. One thing that is interesting here, is dropoff. If you go from using to not using, you are very unlikely to renew.
Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 2-3 4 1 3 0.3 25.0%
0 4-6 1 1 0 Inf 100.0%
0 7+ 1 0 1 0.0 0.0%
1 0 1 0 1 0.0 0.0%
1 1 1 0 1 0.0 0.0%
1 2-3 1 1 0 Inf 100.0%
1 4-6 5 2 3 0.7 40.0%
1 7+ 3 3 0 Inf 100.0%
2-3 0 1 0 1 0.0 0.0%
2-3 2-3 3 1 2 0.5 33.3%
2-3 4-6 6 3 3 1.0 50.0%
2-3 7+ 9 5 4 1.2 55.6%
4-6 0 1 0 1 0.0 0.0%
4-6 2-3 9 6 3 2.0 66.7%
4-6 4-6 11 10 1 10.0 90.9%
4-6 7+ 18 12 6 2.0 66.7%
7+ 0 1 1 0 Inf 100.0%
7+ 2-3 2 0 2 0.0 0.0%
7+ 4-6 12 9 3 3.0 75.0%
7+ 7+ 13 13 0 Inf 100.0%
NA 0 1 0 1 0.0 0.0%
NA 1 1 0 1 0.0 0.0%
NA 2-3 2 2 0 Inf 100.0%
NA 4-6 3 2 1 2.0 66.7%
NA 7+ 6 4 2 2.0 66.7%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. Significant. More impact interactions does translate to a higher percentage renewal rate.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept -0.09 0.387 -0.24 0.814
Impact Interaction Volume 0.12 0.057 2.09 0.036
Penetration by Event Grouping
  1. Service is the most heavily penetrated, with expert calls and events roughly even.
  2. While in other products, there hasn’t been much variation between Opt Outs and NNLOAs, here you see SL Led is much greater for opt outs.
  3. Low research interview pen.
  4. PLW and Service roughly equivalent.
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 75.0% 41.4% 54.3% 6.0% 39.7% 0.0% 81.0%
Opt Out 76.5% 41.2% 73.5% 14.7% 35.3% 0.0% 73.5%
Renewal by Event Consumption
  1. [NNLOA] Small bump from attending, >50%+ distributed towards attending.
  2. [Opt Out] similar to NNLOA.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 29 15 14 1.1 51.7%
Need New LOA 1 87 61 26 2.3 70.1%
Opt Out 0 8 5 3 1.7 62.5%
Opt Out 1 26 22 4 5.5 84.6%
Renewal by Service Consumption
  1. [NNLOA] Much tigher around the mean, suggests not as much value from consuming this II.
  2. [Opt out] Yet, on Opt Outs you see something very different insofar as those who consume this II perform remarkably better than their counterparts who do not consume it.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 68 44 24 1.8 64.7%
Need New LOA 1 48 32 16 2.0 66.7%
Opt Out 0 20 14 6 2.3 70.0%
Opt Out 1 14 13 1 13.0 92.9%
Renewal by SL Consumption
  1. [NNLOA] Negative impact! Although since they are both close to the mean, it is unlikely to be statistically significant. 2 [Opt out] Same as NNLOA, except a little bit more of an effect and more of the distribution towards utilization.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 53 36 17 2.1 67.9%
Need New LOA 1 63 40 23 1.7 63.5%
Opt Out 0 9 8 1 8.0 88.9%
Opt Out 1 25 19 6 3.2 76.0%
Renewal by Research Interview Consumption

Not really enough data here.

Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 109 73 36 2.0 67.0%
Need New LOA 1 7 3 4 0.8 42.9%
Opt Out 0 29 24 5 4.8 82.8%
Opt Out 1 5 3 2 1.5 60.0%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Appears to be a very effective renewal instrument for NNLOA, ~27% gap between utilization and not, plus 16% bump above the mean.
  2. [Opt OUt] Negative impact! Tighter around the mean than the last one, though, so less likely to be statistically sig.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 70 38 32 1.2 54.3%
Need New LOA 1 46 38 8 4.8 82.6%
Opt Out 0 22 18 4 4.5 81.8%
Opt Out 1 12 9 3 3.0 75.0%
Renewal by Expert Call Consumption
  1. [NNLOA] Not consuming this is problematic, even though consuming doesn’t put a unit’s chance of renewal much above the mean.
  2. [Opt Out] Same as NNLOA.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 22 11 11 1.0 50.0%
Need New LOA 1 94 65 29 2.2 69.1%
Opt Out 0 9 6 3 2.0 66.7%
Opt Out 1 25 21 4 5.2 84.0%
Renewal by AskEAB
  1. [NNLOA] Not consuming this is problematic, even though consuming doesn’t put a unit’s chance of renewal much above the mean.
  2. [Opt Out] Same as NNLOA.
Renewal Ratio by AskEAB Consumption
DecisionType II_AskEAB_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 31 20 11 1.8 64.5%
Need New LOA 1 85 56 29 1.9 65.9%
Opt Out 0 6 4 2 2.0 66.7%
Opt Out 1 28 23 5 4.6 82.1%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Correlation with Renewal: Events have the strongest correlation, followed by Expert Calls and PLW/Onsite.
  2. SL Led slightly negatively correlated with renewal rate.
  3. SL Led have very little correlation with the rest of the impact interactions.
  4. Events are correlated, positively, with many of the other impact interactions.
  5. Expert Calls have a very strong relationship with Service interactions, and PLWs.

Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only.

  1. Events and Expert Calls are the two winners.
  2. Service, SL Led, and Research Interviews do show negative effects, although are not significant.
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) -0.21 0.376 -0.56 0.58
II_Events_Volume 0.48 0.152 3.16 0.00
II_ResearchInterview_Volume -1.20 0.627 -1.91 0.06
II_PLW_Onsite_Volume 0.45 0.260 1.72 0.08
II_ExpertCall_Volume 0.28 0.136 2.03 0.04
II_Service_Volume -0.30 0.192 -1.54 0.12
II_SLLed_Volume -0.14 0.169 -0.82 0.41
II_AskEAB_Volume -0.06 0.103 -0.58 0.56