Methodological

  1. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  2. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

HESF Analysis

Key Takeaways

Renewal by Decision Type
  1. NNLOA is low but N = 3.
  2. Opt Out performance is superb.
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 3 1 2 0.5 33.3%
Opt Out 14 14 0 Inf 100.0%
Total 17 15 2 7.5 88.2%
Renewal Rate by Account Segment
  1. Some differences across the segments:
  • Large P&P all renewed.
    • 1 drop a piece in Regional Public and Selectives -> results in varying renewal rates
  1. Most of the decisions are in Large P&P
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
Large Public & Private 10 10 0 Inf 100.0%
Regional Public 4 3 1 3 75.0%
Selective 3 2 1 2 66.7%
Renewal by Decision Type & Segment
  1. [Large P&P]
  • Solid across the board
  1. [Regional Public & Selective]
  • NNLOA is where we see the drop.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
Large Public & Private Need New LOA 1 1 0 Inf 100.0%
Large Public & Private Opt Out 9 9 0 Inf 100.0%
Regional Public Need New LOA 1 0 1 0 0.0%
Regional Public Opt Out 3 3 0 Inf 100.0%
Selective Need New LOA 1 0 1 0 0.0%
Selective Opt Out 2 2 0 Inf 100.0%
GRP 3-4
  1. Most of the partnerhips for HESF live with GRP 3-4 partners: 11/17.
Renewal Ratio by Navigate Bundling
AccountSegment DecisionType GRP3-4 N ATL Dropped Ratio Perc_Renewed
Large Public & Private Need New LOA 1 1 1 0 Inf 100.0%
Large Public & Private Opt Out 0 2 2 0 Inf 100.0%
Large Public & Private Opt Out 1 7 7 0 Inf 100.0%
Regional Public Need New LOA 0 1 0 1 0 0.0%
Regional Public Opt Out 0 1 1 0 Inf 100.0%
Regional Public Opt Out 1 2 2 0 Inf 100.0%
Selective Need New LOA 0 1 0 1 0 0.0%
Selective Opt Out 0 1 1 0 Inf 100.0%
Selective Opt Out 1 1 1 0 Inf 100.0%
Impact Interaction Volume Histogram
  1. Looks like two populations: low utilizers and high utilizers.

##### Impact Interaction Volume Histogram Split by Decision

  1. [NNLOA] Strictly in the utilization part of the impact interactions.
  2. [Opt Out] Dominate low utilization.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. NNLOA performance looks worse as volume increases. Remember, these partnerships are not part of GRP 3-4 and are for Regional Public and Selective schools.
  2. Opt Out performance is invariant to Impact interaction volume.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 5 1 1 0 Inf 100.0%
Need New LOA 6 1 0 1 0 0.0%
Need New LOA 8 1 0 1 0 0.0%
Opt Out 0 2 2 0 Inf 100.0%
Opt Out 1 3 3 0 Inf 100.0%
Opt Out 2 2 2 0 Inf 100.0%
Opt Out 3 4 4 0 Inf 100.0%
Opt Out 5 2 2 0 Inf 100.0%
Opt Out 7 1 1 0 Inf 100.0%
Renewal by Impact Interaction Volume, (Binned)
  1. [NNLOA] Very low N and issues with other variables confounding analysis.
  2. [Opt Out] Uniformly good results regardless of impact interaction volume.
Renewal Ratio for Impact Interaction Volume
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 4-6 2 1 1 1 50.0%
Need New LOA 7+ 1 0 1 0 0.0%
Opt Out 0 2 2 0 Inf 100.0%
Opt Out 1 3 3 0 Inf 100.0%
Opt Out 2-3 6 6 0 Inf 100.0%
Opt Out 4-6 2 2 0 Inf 100.0%
Opt Out 7+ 1 1 0 Inf 100.0%
Renewal by Previous Yr Impact Interaction Volume & Decision
  1. [NNLOA] Nothing meaningful here.
  2. [Opt Out] Invariance.
Renewal Ratio by Previous Year Impact Interaction
DecisionType PreviousYr_II_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 5 1 0 1 0 0.0%
Need New LOA NA 2 1 1 1 50.0%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 2 1 1 0 Inf 100.0%
Opt Out 5 2 2 0 Inf 100.0%
Opt Out 6 1 1 0 Inf 100.0%
Opt Out NA 9 9 0 Inf 100.0%
Simple Regression for Impact Interaction Volume

Note: Looking at NNLOAs, only.

  1. If we only look at NNLOAs, we conclude there is no positive impact from increased Impact Interaction consumption.
  2. BUT, N is too low and there are other mitigating variables.
Simple Regression Model with Impact Interaction Volume
term estimate std.error statistic p.value
Intercept 259.44 635317.3 0 >0.999
Impact Interaction Volume -47.14 113370.9 0 >0.999
Penetration by Event Grouping
  1. Comparing the % differences is not very useful, since N is so low on NNLOA.
  2. Worth noting that Research interviews are the only category consumed only by Opt Outs and not NNLOA.
Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 100.0% 0.0% 100.0% 0.0% 66.7% 0.0% 33.3%
Opt Out 57.1% 0.0% 57.1% 7.1% 28.6% 0.0% 50.0%
Renewal by Event Consumption
  1. [NNLOA] All attended an event.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 1 3 1 2 0.5 33.3%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 1 8 8 0 Inf 100.0%
Renewal by Service Consumption

No data here.

Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 1 2 0.5 33.3%
Opt Out 0 14 14 0 Inf 100.0%
Renewal by SL Consumption
  1. [NNLOA] All had an SL Led impact interaction and only 33% renewed.
  2. [Opt Out] Invariance.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 1 3 1 2 0.5 33.3%
Opt Out 0 6 6 0 Inf 100.0%
Opt Out 1 8 8 0 Inf 100.0%
Renewal by Research Interview Consumption
  1. [NNLOA] No data.
  2. [Opt Out] N too low.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 1 2 0.5 33.3%
Opt Out 0 13 13 0 Inf 100.0%
Opt Out 1 1 1 0 Inf 100.0%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] Onsite and/or PLW positively associated with renewal! N way too small, so it just happens the renewal landed in the consumption bucket.
  2. [Opt Out] Invariance. Although few partnerships consumed this type of Impact Interaction.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 1 0 1 0 0.0%
Need New LOA 1 2 1 1 1 50.0%
Opt Out 0 10 10 0 Inf 100.0%
Opt Out 1 4 4 0 Inf 100.0%
Renewal by Experience Consumption

No data here.

Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 1 2 0.5 33.3%
Opt Out 0 14 14 0 Inf 100.0%
Renewal by Expert Call Consumption
  1. [NNLOA] N way too small, so it just happens the renewal landed in the no consumption bucket.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2 1 1 1 50.0%
Need New LOA 1 1 0 1 0 0.0%
Opt Out 0 7 7 0 Inf 100.0%
Opt Out 1 7 7 0 Inf 100.0%
Correlation betwen Event Groupings, and Renewal (for fun)
  1. Halo Effect: Most of the categories are positively correlated with the others, with the exception of Research Interviews.

##### Multivariate Regression (still kind of simple)

Note: Looking at NNLOA and Opt Outs.

Too little variation to make any assessments.

Regression Estimates for a Model Fitted on Event Grouping Volumes
term estimate std.error statistic p.value
(Intercept) 50.92 329081.35 0 1
II_Events_Volume -23.85 55033.42 0 1
II_SLLed_Volume 0.19 70437.86 0 1
II_ResearchInterview_Volume -0.70 486876.64 0 1
II_PLW_Onsite_Volume 44.69 364813.54 0 1
II_ExpertCall_Volume -1.63 168318.73 0 1