Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

EMF Analysis

Key Takeaways

Renewal by Decision Type
  1. NNLOA is exactly at 50%: towards the bottom of all forums.
  2. Opt Out performance and volume is helping overall renewal rate, as opt outs are ~32% of total decisions.
  3. Keep the 1.0 Ratio for NNLOA in mind throughout the analysis.
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 42 21 21 1.0 50.0%
Opt Out 20 18 2 9.0 90.0%
Total 62 39 23 1.7 62.9%
Renewal Rate by Account Segment
  1. Some consistency in performance across segments:
  • Regional Private and Large P&P renew at higher rates.
    • Regional Privates are small N, but still surprising to see a high rate.
    • Half of the partnerships are with Large P&P schools.
    • Regional Public and Selectives perform equivalently and low
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
Large Public & Private 31 22 9 2.4 71.0%
Regional Private 5 4 1 4.0 80.0%
Regional Public 14 7 7 1.0 50.0%
Selective 12 6 6 1.0 50.0%
Renewal by Decision Type & Segment
  1. [Large P&P]
  • Performance on NNLOA is roughly what we have seen in other flagship forums
    • Performance on Opt Outs is superb - perfect score.
  1. [Regional Private]
  • Only NNLOA and performance is pretty strong.
  1. [Regional Public & Selective]
  • NNLOA is a particular problem for Regional Public and Selective
    • Opt Out Performance appears to be OK, even with low N.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
Large Public & Private Need New LOA 21 12 9 1.3 57.1%
Large Public & Private Opt Out 10 10 0 Inf 100.0%
Regional Private Need New LOA 5 4 1 4.0 80.0%
Regional Public Need New LOA 9 3 6 0.5 33.3%
Regional Public Opt Out 5 4 1 4.0 80.0%
Selective Need New LOA 7 2 5 0.4 28.6%
Selective Opt Out 5 4 1 4.0 80.0%
GRP 3-4
  1. The performance for GRP 3-4 does not look very different across Large Public & Private Decision Types.
  2. Perhaps a small boost for Selective opt outs - super low N.
Renewal Ratio by Navigate Bundling
AccountSegment DecisionType GRP3-4 N ATL Dropped Ratio Perc_Renewed
Large Public & Private Need New LOA 0 20 11 9 1.2 55.0%
Large Public & Private Need New LOA 1 1 1 0 Inf 100.0%
Large Public & Private Opt Out 0 4 4 0 Inf 100.0%
Large Public & Private Opt Out 1 6 6 0 Inf 100.0%
Regional Private Need New LOA 0 5 4 1 4.0 80.0%
Regional Public Need New LOA 0 9 3 6 0.5 33.3%
Regional Public Opt Out 0 3 2 1 2.0 66.7%
Regional Public Opt Out 1 2 2 0 Inf 100.0%
Selective Need New LOA 0 7 2 5 0.4 28.6%
Selective Opt Out 0 4 3 1 3.0 75.0%
Selective Opt Out 1 1 1 0 Inf 100.0%
Impact Interaction Volume Histogram
  1. Overall low engagement: a lot of partners in 0, 1, 2 categories, with a peak at 2.
  2. Thick right tail suggests some heavier users in the 5+ categories.

##### Impact Interaction Volume Histogram Split by Decision

  1. Opt Outs are spread out.
  2. Relatively uniform distribution of Opt Outs and NNLOA in the tail.
  3. NNLOAs represent most of the 0-2 utilization bins.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA performance varies a lot due to small N.
  2. No apparent threshold as the variation in higher utilization makes a cutoff difficult without binning.
  3. Utilization at 0-2 for NNLOA is deadly as 10 fo 16 partners dropped.
  4. Opt Out performance is invariant to Impact interaction volume.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 1 0 1 0.0 0.0%
Need New LOA 1 4 3 1 3.0 75.0%
Need New LOA 2 11 3 8 0.4 27.3%
Need New LOA 3 8 5 3 1.7 62.5%
Need New LOA 4 5 3 2 1.5 60.0%
Need New LOA 5 7 3 4 0.8 42.9%
Need New LOA 7 2 1 1 1.0 50.0%
Need New LOA 8 2 2 0 Inf 100.0%
Need New LOA 9 1 1 0 Inf 100.0%
Need New LOA 10 1 0 1 0.0 0.0%
Opt Out 1 1 1 0 Inf 100.0%
Opt Out 2 4 4 0 Inf 100.0%
Opt Out 3 2 1 1 1.0 50.0%
Opt Out 4 4 4 0 Inf 100.0%
Opt Out 5 4 3 1 3.0 75.0%
Opt Out 6 1 1 0 Inf 100.0%
Opt Out 8 2 2 0 Inf 100.0%
Opt Out 9 1 1 0 Inf 100.0%
Opt Out 10 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] When you take the variation out, you might say 7+ is the threshold.
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 1 0 1 0.0 0.0%
Need New LOA 1 4 3 1 3.0 75.0%
Need New LOA 2-3 19 8 11 0.7 42.1%
Need New LOA 4-6 12 6 6 1.0 50.0%
Need New LOA 7+ 6 4 2 2.0 66.7%
Opt Out 1 1 1 0 Inf 100.0%
Opt Out 2-3 6 5 1 5.0 83.3%
Opt Out 4-6 9 8 1 8.0 88.9%
Opt Out 7+ 4 4 0 Inf 100.0%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] Renewal rates vary a lot, but it is important to note the heavy part of the distribution is in the 3-5 range and the renewal rate is below average.
  2. [Opt Out] Invariance.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2 2 0 Inf 100.0%
Need New LOA 1 2 1 1 1.0 50.0%
Need New LOA 2 4 3 1 3.0 75.0%
Need New LOA 3 6 0 6 0.0 0.0%
Need New LOA 4 6 2 4 0.5 33.3%
Need New LOA 5 7 3 4 0.8 42.9%
Need New LOA 6 1 0 1 0.0 0.0%
Need New LOA 7 4 3 1 3.0 75.0%
Need New LOA 8 3 3 0 Inf 100.0%
Need New LOA NA 7 4 3 1.3 57.1%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 1 1 0 1 0.0 0.0%
Opt Out 3 2 2 0 Inf 100.0%
Opt Out 4 1 1 0 Inf 100.0%
Opt Out 7 5 5 0 Inf 100.0%
Opt Out 8 2 1 1 1.0 50.0%
Opt Out NA 8 8 0 Inf 100.0%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table.

  1. Maintenance looks like it is key. 16 fall in the 3+,3+ bucket and renew at a high rate.
  2. Dropping to 0 is undesirable.
Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 1 1 1 0 Inf 100.0%
0 2-3 1 1 0 Inf 100.0%
1 2-3 2 1 1 1.0 50.0%
2-3 1 2 1 1 1.0 50.0%
2-3 2-3 5 1 4 0.2 20.0%
2-3 4-6 2 0 2 0.0 0.0%
2-3 7+ 1 1 0 Inf 100.0%
4-6 1 1 1 0 Inf 100.0%
4-6 2-3 5 1 4 0.2 20.0%
4-6 4-6 6 3 3 1.0 50.0%
4-6 7+ 2 0 2 0.0 0.0%
7+ 2-3 3 2 1 2.0 66.7%
7+ 4-6 2 2 0 Inf 100.0%
7+ 7+ 2 2 0 Inf 100.0%
NA 0 1 0 1 0.0 0.0%
NA 2-3 3 2 1 2.0 66.7%
NA 4-6 2 1 1 1.0 50.0%
NA 7+ 1 1 0 Inf 100.0%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. Increasing Impact Interactions is not associated with changes in renewal probability.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept -0.41 0.594 -0.69 0.489
Impact Interaction Volume 0.11 0.139 0.81 0.420
Penetration by Event Grouping
  1. Opt Outs have higher penetration in most of the Event Groupings:
  • Events (marginally), SL Led, Research Interviews, and Expert Center Calls.
  1. NNLOAs are higher in Service and PLW/Onsites (marginally)
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 50.0% 33.3% 59.5% 42.9% 9.5% 0.0% 42.9%
Opt Out 55.0% 20.0% 70.0% 60.0% 5.0% 0.0% 70.0%
Renewal by Event Consumption
  1. [NNLOA] Small lift, 2.4%, above the average. And roughly a 5% bump in renewal from consuming vs. not.
  2. [Opt Outs] Attending an event is negatively related to renewal.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 21 10 11 0.9 47.6%
Need New LOA 1 21 11 10 1.1 52.4%
Opt Out 0 9 9 0 Inf 100.0%
Opt Out 1 11 9 2 4.5 81.8%
Renewal by Service Consumption
  1. [NNLOA] No difference between consuming this interaction and not. Plus, both are at the average.
  2. [Opt Out] Some evidence of benefit from Service Consumption - Small N.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 28 14 14 1 50.0%
Need New LOA 1 14 7 7 1 50.0%
Opt Out 0 16 14 2 7 87.5%
Opt Out 1 4 4 0 Inf 100.0%
Renewal by SL Consumption
  1. [NNLOA] Decent lift for having an SL Led Interaction - 10% above NNLOA average, 25% bump above not having one.
  2. [Opt Outs] SL consumption is negatively related to renewal. Not sure this is worth reacting to.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 17 6 11 0.5 35.3%
Need New LOA 1 25 15 10 1.5 60.0%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 1 14 12 2 6.0 85.7%
Renewal by Research Interview Consumption
  1. [NNLOA] First instance where Research Interviews are not positively related to renewal. -5.6% below the average.
  2. [Opt Out] Opposite result from NNLOA. We’d conclude service consumption offers a decent lift.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 24 13 11 1.2 54.2%
Need New LOA 1 18 8 10 0.8 44.4%
Opt Out 0 8 6 2 3.0 75.0%
Opt Out 1 12 12 0 Inf 100.0%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Onsite and/or PLW positively associated with renewal! Small N, though. Big lift, nearly 29%.
  2. [Opt Out] Not enough data to conclude anything.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 38 18 20 0.9 47.4%
Need New LOA 1 4 3 1 3.0 75.0%
Opt Out 0 19 17 2 8.5 89.5%
Opt Out 1 1 1 0 Inf 100.0%
Renewal by Experience Consumption
  1. No data.
Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 42 21 21 1 50.0%
Opt Out 0 20 18 2 9 90.0%
Renewal by Expert Call Consumption
  1. [NNLOA] No difference between consuming this interaction and not. Plus, both are at the average.
  2. [Opt Out] Some evidence of benefit from Service Consumption - a bump of nearly 10% vs. not having the interaction.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 24 12 12 1 50.0%
Need New LOA 1 18 9 9 1 50.0%
Opt Out 0 6 5 1 5 83.3%
Opt Out 1 14 13 1 13 92.9%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Correlation with Renewal: Despite what we saw above, volume of impact interaction consumption, across categories, correlates with renewal, positively.
    • If we run this for only NNLOAs, then expert calls are the only negatively related impact interaction. SL Led and Service are highest positive correlations.
  2. Halo Effect: Events highly correlated with the other categories, except Research Interviews.
  3. Onsites and SL Led are the highest correlated event categories.

##### Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only.

  1. Nothing Pops as significant.
  2. If you control for segment, you’d find that SL Led is a significant impact interaction.
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) -0.83 0.774 -1.07 0.28
II_Events_Volume 0.31 0.383 0.81 0.42
II_SLLed_Volume 0.76 0.490 1.55 0.12
II_ResearchInterview_Volume 0.15 0.431 0.35 0.73
II_PLW_Onsite_Volume 0.09 1.364 0.07 0.95
II_ExpertCall_Volume -0.09 0.360 -0.25 0.80
II_Service_Volume 0.56 0.715 0.79 0.43
AccountSegmentRegional Private 1.64 1.357 1.20 0.23
AccountSegmentRegional Public -2.20 1.247 -1.77 0.08
AccountSegmentSelective -0.94 1.153 -0.82 0.41