Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

Aggregate Analysis

Renewal by Decision Type
  1. NNLOA is hovering just above 50%, which is relatively low.
  2. Opt Out performance and volume is buoying overall renewal rate.
  3. Keep the 1.1 Ratio for NNLOA in mind throughout the analysis.
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 498 265 233 1.1 53.2%
Opt Out 293 264 29 9.1 90.1%
Total 791 529 262 2.0 66.9%
Renewal Rate by Account Segment
  1. Regional Public & Selective are performing relatively similar with similar Ns.
  2. Large P&P and International are both substantially above the average. The N for Large is overwhelming.
  3. Regional Private: Struggling to maintain partners in this segment. Pulling the average down. But N is low.
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
International 39 29 10 2.9 74.4%
Large Public & Private 356 267 89 3.0 75.0%
Regional Private 84 41 43 1.0 48.8%
Regional Public 159 95 64 1.5 59.7%
Selective 153 97 56 1.7 63.4%
Renewal by Decision Type & Segment
  1. NNLOA in Large P&P and International are above the average.
  2. NNLOA in Regionals are roughly the same and well below the average.
  3. Large P&P Opt Outs account for 40% plus of all of the opt outs. And perform exceedingly well.
  4. Opt Outs in the Regional space, while lower N, is likely worth monitoring.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
International Need New LOA 27 18 9 2.0 66.7%
International Opt Out 12 11 1 11.0 91.7%
Large Public & Private Need New LOA 227 142 85 1.7 62.6%
Large Public & Private Opt Out 129 125 4 31.2 96.9%
Regional Private Need New LOA 59 23 36 0.6 39.0%
Regional Private Opt Out 25 18 7 2.6 72.0%
Regional Public Need New LOA 87 33 54 0.6 37.9%
Regional Public Opt Out 72 62 10 6.2 86.1%
Selective Need New LOA 98 49 49 1.0 50.0%
Selective Opt Out 55 48 7 6.9 87.3%
Impact Interaction Volume Histogram
  1. Central Tendency is moving much closer to 3 with 2 and 3 having the highest peaks.
  2. Still a decent number of renewals with 0-1 interactions in the year leading up to the renewal.
  3. Beautiful long tail of interactions 4+.

##### Impact Interaction Volume Histogram Split by Decision

  1. Opt Outs occupy an outsized portion long tail.
  2. A lot of NNLOAs in the 2,3,& range.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. At least 4 interactions are required to climb above the average NNLOA rate.
  2. NNLOA climbs above the average around 4 interactions.
  3. Outsized NNLOA performance does not occur until 9+ interactions, and that is likely due to low N.
  4. Opt Outs with Low interactions will still renew at a high rate.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 44 10 34 0.3 22.7%
Need New LOA 1 48 19 29 0.7 39.6%
Need New LOA 2 67 30 37 0.8 44.8%
Need New LOA 3 67 33 34 1.0 49.3%
Need New LOA 4 59 35 24 1.5 59.3%
Need New LOA 5 51 31 20 1.6 60.8%
Need New LOA 6 36 20 16 1.2 55.6%
Need New LOA 7 42 27 15 1.8 64.3%
Need New LOA 8 21 13 8 1.6 61.9%
Need New LOA 9 25 18 7 2.6 72.0%
Need New LOA 10 14 11 3 3.7 78.6%
Need New LOA 11 7 6 1 6.0 85.7%
Need New LOA 12 10 7 3 2.3 70.0%
Need New LOA 13 2 2 0 Inf 100.0%
Need New LOA 14 1 1 0 Inf 100.0%
Need New LOA 16 1 0 1 0.0 0.0%
Need New LOA 18 1 1 0 Inf 100.0%
Need New LOA 21 1 1 0 Inf 100.0%
Need New LOA 24 1 0 1 0.0 0.0%
Opt Out 0 26 24 2 12.0 92.3%
Opt Out 1 24 23 1 23.0 95.8%
Opt Out 2 34 27 7 3.9 79.4%
Opt Out 3 45 42 3 14.0 93.3%
Opt Out 4 31 27 4 6.8 87.1%
Opt Out 5 29 25 4 6.2 86.2%
Opt Out 6 22 20 2 10.0 90.9%
Opt Out 7 23 22 1 22.0 95.7%
Opt Out 8 20 18 2 9.0 90.0%
Opt Out 9 6 6 0 Inf 100.0%
Opt Out 10 11 9 2 4.5 81.8%
Opt Out 11 3 2 1 2.0 66.7%
Opt Out 12 3 3 0 Inf 100.0%
Opt Out 13 6 6 0 Inf 100.0%
Opt Out 14 2 2 0 Inf 100.0%
Opt Out 15 2 2 0 Inf 100.0%
Opt Out 16 1 1 0 Inf 100.0%
Opt Out 18 1 1 0 Inf 100.0%
Opt Out 19 1 1 0 Inf 100.0%
Opt Out 22 1 1 0 Inf 100.0%
Opt Out 24 1 1 0 Inf 100.0%
Opt Out 46 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. Neater table of the above shows the same insight. 4+ to hit average, 7+ to be in good shape for NNLOA.
  2. Notice the large and variable jumps between each of the II Volume Bins, 17%, 8.6%, 11.9%, 10.1%
  3. Odd that Opt Outs dip between 1 and 4 interactions.
Renewal Ratio by Impact Interaction Volume (Binned)
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 44 10 34 0.3 22.7%
Need New LOA 1 48 19 29 0.7 39.6%
Need New LOA 2-3 134 63 71 0.9 47.0%
Need New LOA 4-6 146 86 60 1.4 58.9%
Need New LOA 7+ 126 87 39 2.2 69.0%
Opt Out 0 26 24 2 12.0 92.3%
Opt Out 1 24 23 1 23.0 95.8%
Opt Out 2-3 79 69 10 6.9 87.3%
Opt Out 4-6 82 72 10 7.2 87.8%
Opt Out 7+ 82 76 6 12.7 92.7%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. 3 or fewer impact interactions in the previous year suggests a negative correlation with renewal rate.
  2. 4+ interactions in the previous year looks like a winning ticket and may trump activity in the year leading up to the renewal decision.
  3. Previous Year volume does not have an effect on Opt Outs.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 33 14 19 0.7 42.4%
Need New LOA 1 46 20 26 0.8 43.5%
Need New LOA 2 62 26 36 0.7 41.9%
Need New LOA 3 63 29 34 0.9 46.0%
Need New LOA 4 53 30 23 1.3 56.6%
Need New LOA 5 53 32 21 1.5 60.4%
Need New LOA 6 38 20 18 1.1 52.6%
Need New LOA 7 30 21 9 2.3 70.0%
Need New LOA 8 16 6 10 0.6 37.5%
Need New LOA 9 17 14 3 4.7 82.4%
Need New LOA 10 9 6 3 2.0 66.7%
Need New LOA 11 3 0 3 0.0 0.0%
Need New LOA 12 2 1 1 1.0 50.0%
Need New LOA 13 2 0 2 0.0 0.0%
Need New LOA 14 6 6 0 Inf 100.0%
Need New LOA 15 2 1 1 1.0 50.0%
Need New LOA 17 3 2 1 2.0 66.7%
Need New LOA 20 1 1 0 Inf 100.0%
Need New LOA 23 1 0 1 0.0 0.0%
Need New LOA 24 1 0 1 0.0 0.0%
Need New LOA 32 1 1 0 Inf 100.0%
Need New LOA NA 56 35 21 1.7 62.5%
Opt Out 0 13 12 1 12.0 92.3%
Opt Out 1 10 9 1 9.0 90.0%
Opt Out 2 15 12 3 4.0 80.0%
Opt Out 3 15 13 2 6.5 86.7%
Opt Out 4 13 11 2 5.5 84.6%
Opt Out 5 13 13 0 Inf 100.0%
Opt Out 6 12 10 2 5.0 83.3%
Opt Out 7 16 14 2 7.0 87.5%
Opt Out 8 7 6 1 6.0 85.7%
Opt Out 9 2 2 0 Inf 100.0%
Opt Out 10 3 2 1 2.0 66.7%
Opt Out 12 1 1 0 Inf 100.0%
Opt Out 13 2 2 0 Inf 100.0%
Opt Out 14 3 3 0 Inf 100.0%
Opt Out 15 1 0 1 0.0 0.0%
Opt Out 16 2 2 0 Inf 100.0%
Opt Out 18 1 0 1 0.0 0.0%
Opt Out 24 1 0 1 0.0 0.0%
Opt Out 26 1 1 0 Inf 100.0%
Opt Out 27 1 1 0 Inf 100.0%
Opt Out NA 161 150 11 13.6 93.2%
Renewal by Previous Yr Impact Interaction Volume Binned & Decision

Note: If II_PrevYr_Volume_Bins = NA, then there was not an active contract inthe previous year.

  1. Most of the findings above hold true when looking at the binned version of volume.
Renewal Ratio by Previous Year Impact Interaction
DecisionType II_PrevYr_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 33 14 19 0.7 42.4%
Need New LOA 1 46 20 26 0.8 43.5%
Need New LOA 2-3 125 55 70 0.8 44.0%
Need New LOA 4-6 144 82 62 1.3 56.9%
Need New LOA 7+ 94 59 35 1.7 62.8%
Need New LOA NA 56 35 21 1.7 62.5%
Opt Out 0 13 12 1 12.0 92.3%
Opt Out 1 10 9 1 9.0 90.0%
Opt Out 2-3 30 25 5 5.0 83.3%
Opt Out 4-6 38 34 4 8.5 89.5%
Opt Out 7+ 41 34 7 4.9 82.9%
Opt Out NA 161 150 11 13.6 93.2%
Renewals Previous Yr and Current Yr Interaction Volume

Only looking at NNLOAs in the below table.

  1. The table becomes a bit sparse due to all of the combinations.
  2. Overall, if you are in the 4-6 or 7+ the previous year, dropping down below 4-6 interaction volumes is correlated with lower renewal rates.
  3. Low engagement in the previous year tends to be correlated with low engagement in the subsequent year.
  • Even when you bump a school from 1 to 2-3, the rate does not jump much
Renewal Ratio by Previous and Current Year Impact Interaction
II_PrevYr_Volume_Bins II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
0 0 8 2 6 0.3 25.0%
0 1 9 3 6 0.5 33.3%
0 2-3 7 4 3 1.3 57.1%
0 4-6 5 3 2 1.5 60.0%
0 7+ 4 2 2 1.0 50.0%
1 0 4 1 3 0.3 25.0%
1 1 8 4 4 1.0 50.0%
1 2-3 15 6 9 0.7 40.0%
1 4-6 12 5 7 0.7 41.7%
1 7+ 7 4 3 1.3 57.1%
2-3 0 18 4 14 0.3 22.2%
2-3 1 14 4 10 0.4 28.6%
2-3 2-3 39 17 22 0.8 43.6%
2-3 4-6 35 14 21 0.7 40.0%
2-3 7+ 19 16 3 5.3 84.2%
4-6 0 9 1 8 0.1 11.1%
4-6 1 11 5 6 0.8 45.5%
4-6 2-3 40 19 21 0.9 47.5%
4-6 4-6 46 31 15 2.1 67.4%
4-6 7+ 38 26 12 2.2 68.4%
7+ 0 1 0 1 0.0 0.0%
7+ 1 3 0 3 0.0 0.0%
7+ 2-3 18 9 9 1.0 50.0%
7+ 4-6 25 19 6 3.2 76.0%
7+ 7+ 47 31 16 1.9 66.0%
NA 0 4 2 2 1.0 50.0%
NA 1 3 3 0 Inf 100.0%
NA 2-3 15 8 7 1.1 53.3%
NA 4-6 23 14 9 1.6 60.9%
NA 7+ 11 8 3 2.7 72.7%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. The Estimate for Impact Interaction Volume is Positive -> increased consumption means increased likelihood to renew.
  2. This is statistically significant.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept 0.16 0.127 1.26 0.208
Impact Interaction Volume 0.12 0.025 4.98 <0.001
Penetration by Event Grouping
  1. Unexpected amount of balance between NNLOA and Opt Outs on % of each group consumed.
  2. Very little in Experience, overall. Perhaps only applies to a few Forums with lower N.
  3. Opt Outs tend to get more SL Led Impact interactions. Maybe too much focus is being placed here?
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 60.2% 28.7% 49.6% 23.1% 21.1% 4.6% 56.4%
Opt Out 58.7% 31.4% 58.7% 23.2% 19.1% 4.4% 55.6%
Renewal by Event Consumption
  1. Nearly a 13% bump and the N relatively large in both bins.
  2. NNLOA with an Event are 5% above the average for NNLOA.
  3. One of only two case where we see a decent drop in NNLOA renewal vs. the average, if NOT consumed.
  4. Even Opt Outs see a nice bump from engaging in the Event group.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 198 90 108 0.8 45.5%
Need New LOA 1 300 175 125 1.4 58.3%
Opt Out 0 121 102 19 5.4 84.3%
Opt Out 1 172 162 10 16.2 94.2%
Renewal by Service Consumption
  1. 7% bump in NNLOA.
  2. Nearly no lift in the Opt Outs.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 355 181 174 1.0 51.0%
Need New LOA 1 143 84 59 1.4 58.7%
Opt Out 0 201 182 19 9.6 90.5%
Opt Out 1 92 82 10 8.2 89.1%
Renewal by SL Consumption
  1. Smaller bump for SL Led consumption - 7% - as compared to Events.
  2. Much closer to the average NNLOA renewal rate.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 251 121 130 0.9 48.2%
Need New LOA 1 247 144 103 1.4 58.3%
Opt Out 0 121 109 12 9.1 90.1%
Opt Out 1 172 155 17 9.1 90.1%
Renewal by Research Interview Consumption
  1. Relatively high lift above average NNLOA renewal rate - 10+%
  2. Difference between consuming and not consuming is roughly 15%
  3. Opt Outs also disproportionately benefitting from Research Interviews.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 383 191 192 1.0 49.9%
Need New LOA 1 115 74 41 1.8 64.3%
Opt Out 0 225 198 27 7.3 88.0%
Opt Out 1 68 66 2 33.0 97.1%
Renewal by PLW OR Onsite Consumption
  1. Onsite and/or PLW positively associated with renewal: 10% bump if you do consume and 8% bump above NNLOA average.
  2. Opt Outs see basically no difference.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 393 201 192 1.0 51.1%
Need New LOA 1 105 64 41 1.6 61.0%
Opt Out 0 237 214 23 9.3 90.3%
Opt Out 1 56 50 6 8.3 89.3%
Renewal by Experience Consumption
  1. Very few consuming but a huge life above the NNLOA average - 20%.
  2. If Experience is not consumed, renewal is close to the NNLOA average. No adverse impact from non-consumption.
Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 475 248 227 1.1 52.2%
Need New LOA 1 23 17 6 2.8 73.9%
Opt Out 0 280 252 28 9.0 90.0%
Opt Out 1 13 12 1 12.0 92.3%
Renewal by Expert Call Consumption
  1. Consuming Expert Calls is associated with a 3% bump above the NNLOA average.
  2. One of only two cases where we see a decent drop in NNLOA renewal vs. the average, if NOT consumed.
  3. Potential negative relation on Opt Outs.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 217 106 111 1.0 48.8%
Need New LOA 1 281 159 122 1.3 56.6%
Opt Out 0 130 122 8 15.2 93.8%
Opt Out 1 163 142 21 6.8 87.1%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Highest Correlation with Renewals: Events, SL Led, Research Interviews
  2. Lowest Correlations with Renewals: Service and Expert Calls
  3. Everything is positively related with Renewals -> consumption is a good thing
  4. Events are highly correlated with many of the other consumption

##### Multivariate Regression (still kind of simple)

Note: Looking at NNLOA, only. I also took out ‘Service’ which is not statistically significant but whose large estimate was distracting.

  1. A few of the impact interaction groups appear to be statistically significant:
  • Events
    • SL Led
    • Expert Call
    • Research Interview
Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) -0.68 0.163 -4.17 0.00
II_Events_Volume 0.27 0.076 3.60 0.00
II_SLLed_Volume 0.16 0.081 1.96 0.05
II_ResearchInterview_Volume 0.34 0.140 2.45 0.01
II_PLW_Onsite_Volume 0.20 0.165 1.19 0.23
II_ExpertCall_Volume 0.17 0.074 2.36 0.02
II_Experience_Volume 0.39 0.390 1.00 0.32
II_Service_Volume -0.01 0.068 -0.10 0.92