URF Analysis

The purpose of this analysis is to understand the variable effect of impact interactions on the likelihood of a partner to renew their partnership. Simply, which service levers matter more for partnership renewal.

Executive Summary

Methodological Considerations

  1. The data set consists of decisions from FY19 through March of FY22.

  2. Similar to previous analyses, we are primarily looking at the year leading up to a decision.

  3. We are not looking at ‘integrated’ partnerships. That is, we are focusing on utilization at the account where the renewal decision is made.

  4. I’ve taken CRPs out of this analysis, because they renew at a high rate - obscuring other performance - AND are covered in a separate analysis.

  5. Lastly, we are looking at unit renewal rates instead of $ renewal rates. This allows us to treat impact interactions equitably across partnerships.

Renewal by Decision Type

The below table should not come as a surprise. Opt Outs tend to perform very well while NNLOA can be quite mixed.

  1. Remember the Ratios: 0.7 for NNLOA, 2.8 for Opt Outs.
  2. Most of decisions are opt outs: ~66%.
Renewal Ratio by Decision Type
DecisionType N ATL Dropped Ratio Perc_Renewed
Need New LOA 50 21 29 0.7 42.0%
Opt Out 23 17 6 2.8 73.9%
Total 73 38 35 1.1 52.1%

Renewals by Decision Type and Year

Since we are combining multiple years of data, we will frequently take a look at whether there are differences across the years.

  1. NNLOA performance had been climbing since FY19, FY22 is a reversal, albeit with a lower N.
  2. Opt Out performance improved since FY19 -> the N for FY21/22 is very low.
Renewal Ratio by Decision Type
DecisionType FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 2019 12 4 8 0.5 33.3%
Need New LOA 2020 13 5 8 0.6 38.5%
Need New LOA 2021 18 10 8 1.2 55.6%
Need New LOA 2022 7 2 5 0.4 28.6%
Opt Out 2019 6 4 2 2.0 66.7%
Opt Out 2020 14 10 4 2.5 71.4%
Opt Out 2021 2 2 0 Inf 100.0%
Opt Out 2022 1 1 0 Inf 100.0%
Total NA 73 38 35 1.1 52.1%
Renewal Rate by Account Segment
  1. Most of the decisions are in Large P&P or Selective - performance in both is lower with Large P&P.
  2. Regionals and Canada are lower Ns, but are performing better than Large P&P.
Renewal Ratio by Account Segment
AccountSegment N ATL Dropped Ratio Perc_Renewed
Canada 4 3 1 3.0 75.0%
Large Public & Private 38 17 21 0.8 44.7%
Regional Public 10 5 5 1.0 50.0%
Selective 21 13 8 1.6 61.9%
Renewal Rate by Account Segment & Fiscal Year
  1. Large P&P had a couple of tough years FY19 and FY22.
  2. Selectives have been relatively steady across the years. Number of drops doesn’t vary much across FY19-FY21, just the number of overall units.
Renewal Ratio by Account Segment
AccountSegment FiscalYear N ATL Dropped Ratio Perc_Renewed
Canada 2019 2 2 0 Inf 100.0%
Canada 2021 1 1 0 Inf 100.0%
Canada 2022 1 0 1 0.0 0.0%
Large Public & Private 2019 10 3 7 0.4 30.0%
Large Public & Private 2020 14 8 6 1.3 57.1%
Large Public & Private 2021 8 4 4 1.0 50.0%
Large Public & Private 2022 6 2 4 0.5 33.3%
Regional Public 2019 3 2 1 2.0 66.7%
Regional Public 2020 5 2 3 0.7 40.0%
Regional Public 2021 2 1 1 1.0 50.0%
Selective 2019 3 1 2 0.5 33.3%
Selective 2020 8 5 3 1.7 62.5%
Selective 2021 9 6 3 2.0 66.7%
Selective 2022 1 1 0 Inf 100.0%
Renewal by Decision Type & Segment
  1. NNLOA is <=50% for all segments. Large P&P is the lowest.
  2. Opt Outs pretty low for large P&P. Selective looks more on target.
Renewal Ratio by Account Segment
AccountSegment DecisionType N ATL Dropped Ratio Perc_Renewed
Canada Need New LOA 2 1 1 1.0 50.0%
Canada Opt Out 2 2 0 Inf 100.0%
Large Public & Private Need New LOA 30 12 18 0.7 40.0%
Large Public & Private Opt Out 8 5 3 1.7 62.5%
Regional Public Need New LOA 5 2 3 0.7 40.0%
Regional Public Opt Out 5 3 2 1.5 60.0%
Selective Need New LOA 13 6 7 0.9 46.2%
Selective Opt Out 8 7 1 7.0 87.5%
CRP Inclusion
  1. Taken out of the analysis. Proof of the fact.
Renewal Ratio by CRP Inclusion
GRP3-4 N ATL Dropped Ratio Perc_Renewed
0 73 38 35 1.1 52.1%
Impact Interaction Volume Histogram
  1. Central tendency is at 4.
  2. Lots of low utilizers in the 0-3 bucket.
  3. A lot of partners in the 5-7 area.

Impact Interaction Volume Histogram Split by Decision
  1. Super users definitively renew at a pretty high rate; the cut off could be around 7 for this forum.
  2. You see relatively consistent drops across utilization up to 7. This is not suggestive of strong renewal impact from service levers.

Impact Interaction Volume Histogram Split by Pool Date FY
  1. A lot of the 5-6 utilizers come from FY20/21.
  2. FY22 could be split into those who used the product and those who did not.

Renewal by Interaction Volume and Decision Type, Unbinned
  1. [NNLOA] You start to see much lift until the 7+ region.
  2. [Opt Out] Tough to judge. Could be 2+.
Renewal Ratio by Impact Interaction Volume
DecisionType II_Credited_Volume N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 1 2 0.5 33.3%
Need New LOA 1 6 0 6 0.0 0.0%
Need New LOA 2 5 2 3 0.7 40.0%
Need New LOA 3 7 2 5 0.4 28.6%
Need New LOA 4 7 2 5 0.4 28.6%
Need New LOA 5 6 2 4 0.5 33.3%
Need New LOA 6 8 5 3 1.7 62.5%
Need New LOA 7 4 3 1 3.0 75.0%
Need New LOA 8 1 1 0 Inf 100.0%
Need New LOA 9 1 1 0 Inf 100.0%
Need New LOA 10 1 1 0 Inf 100.0%
Need New LOA 11 1 1 0 Inf 100.0%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 1 3 2 1 2.0 66.7%
Opt Out 2 4 4 0 Inf 100.0%
Opt Out 3 1 1 0 Inf 100.0%
Opt Out 4 3 1 2 0.5 33.3%
Opt Out 5 7 6 1 6.0 85.7%
Opt Out 6 3 2 1 2.0 66.7%
Opt Out 7 1 0 1 0.0 0.0%
Renewal by Impact Interaction Volume and Decision Type, (Binned)

Adjusted the binning to better capture the high volume present in URF.

  1. For NNLOA, increased impact interaction volume does look to help, as you go from 2 to 7+.
  2. For NNLOA, as stated above, 7+ is where superior lift occurs.
  3. For Opt Out, as suggested above, it is all over the place.
Renewal Ratio by Impact Interaction Volume (Binned)
DecisionType II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 3 1 2 0.5 33.3%
Need New LOA 1 6 0 6 0.0 0.0%
Need New LOA 2-3 12 4 8 0.5 33.3%
Need New LOA 4-6 21 9 12 0.8 42.9%
Need New LOA 7+ 8 7 1 7.0 87.5%
Opt Out 0 1 1 0 Inf 100.0%
Opt Out 1 3 2 1 2.0 66.7%
Opt Out 2-3 5 5 0 Inf 100.0%
Opt Out 4-6 13 9 4 2.2 69.2%
Opt Out 7+ 1 0 1 0.0 0.0%

Renewal by Impact Interaction Volume and Account Segment, (Binned)

  1. For Large P&P and Selectives, 4+ cutoff looks solid.
Renewal Ratio by Impact Interaction Volume (Binned)
AccountSegment II_Volume_Bins N ATL Dropped Ratio Perc_Renewed
Canada 2-3 1 1 0 Inf 100.0%
Canada 4-6 2 1 1 1.0 50.0%
Canada 7+ 1 1 0 Inf 100.0%
Large Public & Private 1 5 0 5 0.0 0.0%
Large Public & Private 2-3 12 5 7 0.7 41.7%
Large Public & Private 4-6 16 7 9 0.8 43.8%
Large Public & Private 7+ 5 5 0 Inf 100.0%
Regional Public 0 2 1 1 1.0 50.0%
Regional Public 2-3 1 1 0 Inf 100.0%
Regional Public 4-6 7 3 4 0.8 42.9%
Selective 0 2 1 1 1.0 50.0%
Selective 1 4 2 2 1.0 50.0%
Selective 2-3 3 2 1 2.0 66.7%
Selective 4-6 9 7 2 3.5 77.8%
Selective 7+ 3 1 2 0.5 33.3%

Large P&P NNLOA Renewal by Impact Interaction Volume (Binned) & Decision Type

  1. Parsed a little too finely to draw much of a conclusion here.
Large P&P Renewal Ratio by Decision Type & Impact Interaction Volume (Binned)
II_Volume_Bins FiscalYear N ATL Dropped Ratio Perc_Renewed
1 2019 1 0 1 0.0 0.0%
1 2022 3 0 3 0.0 0.0%
2-3 2019 4 0 4 0.0 0.0%
2-3 2020 3 1 2 0.5 33.3%
2-3 2021 2 1 1 1.0 50.0%
4-6 2019 3 2 1 2.0 66.7%
4-6 2020 3 1 2 0.5 33.3%
4-6 2021 3 0 3 0.0 0.0%
4-6 2022 3 2 1 2.0 66.7%
7+ 2019 1 1 0 Inf 100.0%
7+ 2020 1 1 0 Inf 100.0%
7+ 2021 3 3 0 Inf 100.0%

#####Simple Regression for Impact Interaction Volume

  1. The Estimate for Impact Interaction Volume is Positive -> increased consumption means increased likelihood to renew.
  2. This is statistically significant.
  3. Controlling for account segment shows that Impact Interactions seem to matter more for Large P&P and Selectives.
  Target_Binary
Predictors Odds Ratios CI p
(Intercept) 0.10 0.02 – 0.39 0.003
II Credited Volume 1.57 1.20 – 2.21 0.003
DecisionType [Opt Out] 92.10 6.83 – 2862.93 0.002
II Credited Volume *
DecisionType [Opt Out] 		0.47 0.23 – 0.84 0.018
Observations 73
R2 Tjur 0.249
Penetration by Event Grouping

Reminder: 1. Experience = VIP Days & Collaboratives

  1. Service = Benchmarking, Audits, Secret Shopper, Customer Research Projects, Market Insights, and Survey Walkthroughs

  2. Expert Call = Expert Center, Benchmarking Walkthrough, Tool’s Walkthrough

  3. SL Led = Research SL Impact, Evergreen Action, Exec Update, National Meeting Pre-wire

  4. Research Interview = Research Initial & Scoping

  5. Events = National Meeting, COVID Working Group

  6. PLW/Onsite = Onsite, PLW

  7. Events and Expert Calls are the highest penetration. SL Led and Research Interviews follow suit.

  8. A few of the interactions tend to favor Opt Outs - Events, SL led, and Onsites.

Percent of Renewals w/ At Least 1 Interaction in a Group
DecisionType Perc_Events Perc_Service Perc_SLLed Perc_ResearchInt Perc_PLW_Onsite Perc_Experience Perc_ExpertCall
Need New LOA 68.0% 10.0% 46.0% 34.0% 28.0% 2.0% 68.0%
Opt Out 65.2% 0.0% 47.8% 47.8% 21.7% 0.0% 69.6%
Renewal by Event Consumption
  1. [NNLOA] Compelling evidence for the value of events. Attending is a 25% bump above not attending, and a 8% lift above the average.
  2. [Opt Out] No effect for Opt Outs.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 16 4 12 0.3 25.0%
Need New LOA 1 34 17 17 1.0 50.0%
Opt Out 0 8 6 2 3.0 75.0%
Opt Out 1 15 11 4 2.8 73.3%
Renewal by Event Consumption Volume
  1. 1 Event seems better than 0 and 2-3 combined look strong.

Renewal by Event, OT

  1. [NNLOA] Consistent pattern across the years - attending events is better than not.
  2. [Opt Out] Same as NNLOA.
Renewal Ratio by Event Consumption
DecisionType II_Events_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 3 0 3 0.0 0.0%
Need New LOA 0 2020 5 1 4 0.2 20.0%
Need New LOA 0 2021 4 2 2 1.0 50.0%
Need New LOA 0 2022 4 1 3 0.3 25.0%
Need New LOA 1 2019 9 4 5 0.8 44.4%
Need New LOA 1 2020 8 4 4 1.0 50.0%
Need New LOA 1 2021 14 8 6 1.3 57.1%
Need New LOA 1 2022 3 1 2 0.5 33.3%
Opt Out 0 2019 3 2 1 2.0 66.7%
Opt Out 0 2020 3 2 1 2.0 66.7%
Opt Out 0 2021 1 1 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Opt Out 1 2019 3 2 1 2.0 66.7%
Opt Out 1 2020 11 8 3 2.7 72.7%
Opt Out 1 2021 1 1 0 Inf 100.0%
Renewal by Service Consumption
  1. [NNLOA] Decent lift above the average - 11%. Not consuming is just below the average.
  2. [Opt Out] Not much lift from consumption.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 45 18 27 0.7 40.0%
Need New LOA 1 5 3 2 1.5 60.0%
Opt Out 0 23 17 6 2.8 73.9%
Renewal by Service Consumption Volume
  1. You start to see some marginal benefit from consuming 1 service.
  2. More considerable renewal benefit from 2+.

Renewal by Service Consumption, OT

  1. [NNLOA] Alternating pattern on whether consumption of Services is beneficial.
  2. [NNLOA] Most of the decisions taking place in years where consumption is pretty low.
  3. [Opt Out] Similar to NNLOA.
Renewal Ratio by Service Consumption
DecisionType II_Service_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 12 4 8 0.5 33.3%
Need New LOA 0 2020 13 5 8 0.6 38.5%
Need New LOA 0 2021 16 8 8 1.0 50.0%
Need New LOA 0 2022 4 1 3 0.3 25.0%
Need New LOA 1 2021 2 2 0 Inf 100.0%
Need New LOA 1 2022 3 1 2 0.5 33.3%
Opt Out 0 2019 6 4 2 2.0 66.7%
Opt Out 0 2020 14 10 4 2.5 71.4%
Opt Out 0 2021 2 2 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Renewal by SL Consumption
  1. [NNLOA] Inverse relationship, with a relatively strong negative association.
  2. [Opt Out] Same as NNLOA.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 27 14 13 1.1 51.9%
Need New LOA 1 23 7 16 0.4 30.4%
Opt Out 0 12 10 2 5.0 83.3%
Opt Out 1 11 7 4 1.8 63.6%

SL Consumption Volume & Renewal

  1. Increasing the number of SL interactions does not appear to change the overall negative effect.

Renewal by SL Consumption, OT
  1. [NNLOA] Nearly all of the years show the same negative impact. FY21 is driving a lot of the scale behind the negative impact.
  2. [Opt Out] FY19 and FY20 driving the impact here.
Renewal Ratio by SL-Led Consumption
DecisionType II_SLLed_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 4 2 2 1.0 50.0%
Need New LOA 0 2020 8 3 5 0.6 37.5%
Need New LOA 0 2021 12 8 4 2.0 66.7%
Need New LOA 0 2022 3 1 2 0.5 33.3%
Need New LOA 1 2019 8 2 6 0.3 25.0%
Need New LOA 1 2020 5 2 3 0.7 40.0%
Need New LOA 1 2021 6 2 4 0.5 33.3%
Need New LOA 1 2022 4 1 3 0.3 25.0%
Opt Out 0 2019 2 2 0 Inf 100.0%
Opt Out 0 2020 8 6 2 3.0 75.0%
Opt Out 0 2021 1 1 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Opt Out 1 2019 4 2 2 1.0 50.0%
Opt Out 1 2020 6 4 2 2.0 66.7%
Opt Out 1 2021 1 1 0 Inf 100.0%
Renewal by Research Interview Consumption
  1. [NNLOA] Very small lift from consuming research interviews.
  2. [Opt Out] Inverse relationship - pretty wide margin.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 33 13 20 0.7 39.4%
Need New LOA 1 17 8 9 0.9 47.1%
Opt Out 0 12 10 2 5.0 83.3%
Opt Out 1 11 7 4 1.8 63.6%
Research Interview Volume & Renewal
  1. Increasing consumption in this category does not seem to change the renewal effect much.

Renewal By Research Interview Consumption, OT
  1. [NNLOA] Most do not use this interaction, and FY20/21, where we see a lot of the decisions, has alternating years of this being positive and negative.
  2. [Opt Out] No consumption past FY20.
Renewal Ratio by Research Interview Consumption
DecisionType II_ResearchInterview_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 4 0 4 0.0 0.0%
Need New LOA 0 2020 10 4 6 0.7 40.0%
Need New LOA 0 2021 13 7 6 1.2 53.8%
Need New LOA 0 2022 6 2 4 0.5 33.3%
Need New LOA 1 2019 8 4 4 1.0 50.0%
Need New LOA 1 2020 3 1 2 0.5 33.3%
Need New LOA 1 2021 5 3 2 1.5 60.0%
Need New LOA 1 2022 1 0 1 0.0 0.0%
Opt Out 0 2020 9 7 2 3.5 77.8%
Opt Out 0 2021 2 2 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Opt Out 1 2019 6 4 2 2.0 66.7%
Opt Out 1 2020 5 3 2 1.5 60.0%
Renewal by PLW OR Onsite Consumption
  1. [NNLOA] No lift -= both consuming and not are close to the overall average.
  2. [Opt Out] Inverse relationship with a wide margin, on a small N.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 36 15 21 0.7 41.7%
Need New LOA 1 14 6 8 0.8 42.9%
Opt Out 0 18 14 4 3.5 77.8%
Opt Out 1 5 3 2 1.5 60.0%
PLW / Onsite Volume & Renewal
  1. More than 1 onsite/plw does not increase the renewal effect.

Renewal by PLW/Onsite OT
  1. [NNLOA] FY19 shows some positive effect, but most of the mass in FY20/21 and the negative effect exists there.
  2. [Opt Out] Low N and completely driven by FY20.
Renewal Ratio by PLW OR Onsite Consumption
DecisionType II_PLW_Onsite_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 7 1 6 0.2 14.3%
Need New LOA 0 2020 10 4 6 0.7 40.0%
Need New LOA 0 2021 13 8 5 1.6 61.5%
Need New LOA 0 2022 6 2 4 0.5 33.3%
Need New LOA 1 2019 5 3 2 1.5 60.0%
Need New LOA 1 2020 3 1 2 0.5 33.3%
Need New LOA 1 2021 5 2 3 0.7 40.0%
Need New LOA 1 2022 1 0 1 0.0 0.0%
Opt Out 0 2019 5 3 2 1.5 60.0%
Opt Out 0 2020 10 8 2 4.0 80.0%
Opt Out 0 2021 2 2 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Opt Out 1 2019 1 1 0 Inf 100.0%
Opt Out 1 2020 4 2 2 1.0 50.0%
Renewal by Experience Consumption

N to low. Perfect renewal rate, though.

Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 49 20 29 0.7 40.8%
Need New LOA 1 1 1 0 Inf 100.0%
Opt Out 0 23 17 6 2.8 73.9%
Experience Volume & Renewal

N too low.

Renewal by Experience Consumption OT

N too low.

Renewal Ratio by Experience Consumption
DecisionType II_Experience_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 12 4 8 0.5 33.3%
Need New LOA 0 2020 13 5 8 0.6 38.5%
Need New LOA 0 2021 17 9 8 1.1 52.9%
Need New LOA 0 2022 7 2 5 0.4 28.6%
Need New LOA 1 2021 1 1 0 Inf 100.0%
Opt Out 0 2019 6 4 2 2.0 66.7%
Opt Out 0 2020 14 10 4 2.5 71.4%
Opt Out 0 2021 2 2 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Renewal by Expert Call Consumption
  1. [NNLOA] Largest Lift yet - 10% above the average. Those without this consumption renew at VERY LOW rates.
  2. [Opt Out] Small lift, unlikely to be statistically significant.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 16 3 13 0.2 18.8%
Need New LOA 1 34 18 16 1.1 52.9%
Opt Out 0 7 5 2 2.5 71.4%
Opt Out 1 16 12 4 3.0 75.0%
Expert Call Volume & Renewal
  1. At 2/3+ is where we see greater success.

Renewal by Expert Call Consumption OT
  1. [NNLOA] Relatively consistent overperformance for consumption - FY22 is s very strong.
Renewal Ratio by Expert Call Consumption
DecisionType II_ExpertCall_Binary FiscalYear N ATL Dropped Ratio Perc_Renewed
Need New LOA 0 2019 4 1 3 0.3 25.0%
Need New LOA 0 2020 6 2 4 0.5 33.3%
Need New LOA 0 2021 2 0 2 0.0 0.0%
Need New LOA 0 2022 4 0 4 0.0 0.0%
Need New LOA 1 2019 8 3 5 0.6 37.5%
Need New LOA 1 2020 7 3 4 0.8 42.9%
Need New LOA 1 2021 16 10 6 1.7 62.5%
Need New LOA 1 2022 3 2 1 2.0 66.7%
Opt Out 0 2019 1 0 1 0.0 0.0%
Opt Out 0 2020 4 3 1 3.0 75.0%
Opt Out 0 2021 1 1 0 Inf 100.0%
Opt Out 0 2022 1 1 0 Inf 100.0%
Opt Out 1 2019 5 4 1 4.0 80.0%
Opt Out 1 2020 10 7 3 2.3 70.0%
Opt Out 1 2021 1 1 0 Inf 100.0%

Finer Grain Look at What Was Consumed

Event Category and Event Purpose

  1. A few COVID working groups had solid results
  2. Curious about the web audit…not great but something different.
Renewal Ratio by Event Category and Event Purpose
Event_Category EventPurpose N ATL Dropped Ratio Perc_Renewed
COVID Working Group Session URF COVID-19 Working Group - Learn How Fellow VPRs are Responding 11 8 3 2.7 72.7%
Diagnostic or Audit URF Web Audit 5 3 2 1.5 60.0%
Evergreen Action Evergreen Action 7 2 5 0.4 28.6%
Expert Center Expert Center 28 21 9 2.3 75.0%
Expert Center Expert Center Call 17 7 10 0.7 41.2%
National Meeting 2018 University Research Forum National Meeting, 2 1 1 1.0 50.0%
National Meeting 2018 University Research Forum National Meeting, Chicago, IL 3 2 1 2.0 66.7%
National Meeting 2018 University Research Forum National Meeting, Dana Point, CA 3 1 2 0.5 33.3%
National Meeting 2018 University Research Forum National Meeting, Washington, DC 4 2 2 1.0 50.0%
National Meeting 2019 University Research Forum National Meeting, Chicago, IL 3 1 2 0.5 33.3%
National Meeting 2019 University Research Forum National Meeting, Washington, DC 19 12 7 1.7 63.2%
National Meeting 2020 University Research Forum National Meeting, 12 7 5 1.4 58.3%
National Meeting New Opportunities for Research Partnerships: Leveraging Pandemic Disruptions to Expand International, 2 2 0 Inf 100.0%
National Meeting Resilient Leadership Workshop, 2 2 0 Inf 100.0%
National Meeting Unpacking the Responsibility and Opportunities for Economic Justice Initiatives in Higher Education, 5 3 2 1.5 60.0%
National Meeting Virtual Working Session URF: Responding to COVID19: Learn How Fellow VPRs are Responding to COVID19, 11 8 3 2.7 72.7%
National Meeting Virtual Working Session URF: The State of Industry Partnerships: Research Leaders COVID-19 Virtual R, 5 3 2 1.5 60.0%
National Meeting Pre-Wire National Meeting Pre-Wire 1 0 1 0.0 0.0%
Onsite Onsite | URF | 01/08/2019 1 1 0 Inf 100.0%
Onsite Onsite | URF | 02/19/2019 1 0 1 0.0 0.0%
Onsite Onsite | URF | 05/08/2019 1 0 1 0.0 0.0%
Onsite Onsite | URF | 09/19/2019 1 0 1 0.0 0.0%
Onsite Onsite | URF | 09/27/2018 1 1 0 Inf 100.0%
Onsite Onsite | URF | 12/13/2018 1 1 0 Inf 100.0%
Onsite Onsite | URF | Presentation date: 5/23/18 1 1 0 Inf 100.0%
Onsite Onsite l URF l Presentation date: 12/12/18 1 1 0 Inf 100.0%
Onsite NA 7 3 4 0.8 42.9%
Research Initial 2019 Future of Research Roundtable 1 0 1 0.0 0.0%
Research Initial Big Bets 2.0 2 2 0 Inf 100.0%
Research Initial Confronting the Future of F&A Funding 1 0 1 0.0 0.0%
Research Initial Econ Recovery and Development, Discovery Project 1 1 0 Inf 100.0%
Research Initial Foreign International Partnerships, Discovery Project 1 0 1 0.0 0.0%
Research Initial Helping Faculty Achieve Their Full Research Potential 1 1 0 Inf 100.0%
Research Initial Industry Partnerships 2.0 5 2 3 0.7 40.0%
Research Initial Research Development 21 12 9 1.3 57.1%
Research Initial URF 2019 Agenda Setting Scoping Calls 2 2 0 Inf 100.0%
Research SL Impact Research SL Impact - Expert Call 3 1 2 0.5 33.3%
Research SL Impact Research SL Impact - In Person Visit 1 0 1 0.0 0.0%
Research SL Impact Research SL Impact - Lunch with Tom Piechota 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact - Meeting 2 1 1 1.0 50.0%
Research SL Impact Research SL Impact - Meeting - American - URF - Sarah Belson & Team 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact - Meeting - Baylor - URF - Kevin Chambliss (VP, Research) 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact - Naval Postgraduate School - URF - Welcome Call 1 0 1 0.0 0.0%
Research SL Impact Research SL Impact - Partner Intensive | AAF | BIPOC Faculty Recruitment 1 0 1 0.0 0.0%
Research SL Impact Research SL Impact - Private Label Webinar 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact - Research Forum Networking Call 1 0 1 0.0 0.0%
Research SL Impact Research SL Impact - SDL Presentation to MPOC & Board 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact - Second Exec Roundtable Topic 4 1 3 0.3 25.0%
Research SL Impact Research SL Impact - Tools Walk-through 1 1 0 Inf 100.0%
Research SL Impact Research SL Impact Visit 10 4 6 0.7 40.0%
Scoping Econ Recovery and Development, Discovery Project 5 2 3 0.7 40.0%
Tool’s Walkthrough Tool’s Walk Through 4 3 1 3.0 75.0%
VIP Day NC State University Virtual Impact Workshops 1 1 0 Inf 100.0%
Webinar Virtual Meeting | Expanding the Scope - RF | Presentation Date: 5/28/20 1 0 1 0.0 0.0%
Webinar Virtual Meeting | RF | F&A Task Force | Discussion with EAB | Presentation Date: 04/28/21 1 1 0 Inf 100.0%
Webinar Virtual Meeting | URF | Expanding the Scope of Industry Partnerships | Presentation Date: 03/29/21 1 0 1 0.0 0.0%
Webinar Virtual Meeting | URF | Fine-Tuning Center and Institute Management 1 0 1 0.0 0.0%
Webinar Virtual Meeting | URF | Navigating International Tensions from the Research Office | 1/28/21 1 0 1 0.0 0.0%
Webinar Virtual Meeting | URF | Telling the Story of Research | Presentation Date: 11/13/20 1 0 1 0.0 0.0%
Webinar NA 2 1 1 1.0 50.0%
NA NA 4 2 2 1.0 50.0%
Correlation betwen Event Groupings, and Renewal
  1. Events and Expert Calls are most correlated with renewal.
  2. Positive Correlations: Expert Calls and (Events, PLW Onsites)
  3. Negative Correlations: Research Interviews and (Service, SL Led)

Multivariate Regression (still kind of simple)

Note: Looking at all decision types.

  1. For NNLOAs, Events and Expert Calls pop as significant.
  2. In Total, Events are the only thing that pop as significant.
  Target_Binary
Predictors Odds Ratios CI p
(Intercept) 0.46 0.15 – 1.31 0.156
II Events Volume 1.55 0.92 – 2.72 0.109
II SLLed Volume 0.51 0.20 – 1.18 0.132
II ResearchInterview
Volume 		1.17 0.65 – 2.15 0.602
II PLW Onsite Volume 0.49 0.16 – 1.25 0.171
II ExpertCall Volume 1.91 1.25 – 3.25 0.007
II Service Volume 1.67 0.17 – 20.43 0.662
Observations 73
R2 Tjur 0.216

Combinations of Impact Interactions

Combinations of Impact Interactions

Renewal Ratio by Impact Interaction Combos
II_Combo_Text N ATL Dropped Ratio Perc_Renewed
NA 4 2 2 1.0 50.0%
Events 3 1 2 0.5 33.3%
Expert Call; Events 3 3 0 Inf 100.0%
Expert Call 2 0 2 0.0 0.0%
SL-Led Impact 2 1 1 1.0 50.0%
Events; Events 1 0 1 0.0 0.0%
Events; Events; Expert Call; Expert Call; Events 1 1 0 Inf 100.0%
Events; Events; PLW/Onsite; Expert Call; Expert Call; Expert Call; Expert Call; Expert Call; Research Interview; Events 1 1 0 Inf 100.0%
Events; Events; SL-Led Impact; Research Interview 1 0 1 0.0 0.0%
Events; Expert Call 1 1 0 Inf 100.0%
Events; Expert Call; Expert Call 1 0 1 0.0 0.0%
Events; Expert Call; Expert Call; Research Interview; Events; Experience 1 1 0 Inf 100.0%
Events; Research Interview 1 1 0 Inf 100.0%
Events; Research Interview; Expert Call 1 0 1 0.0 0.0%
Events; Service/Tool; Expert Call; Events 1 1 0 Inf 100.0%
Events; SL-Led Impact; Expert Call; PLW/Onsite; Events; Expert Call; Events; Expert Call 1 1 0 Inf 100.0%
Events; SL-Led Impact; PLW/Onsite; SL-Led Impact; Service/Tool 1 0 1 0.0 0.0%
Events; SL-Led Impact; Research Interview 1 0 1 0.0 0.0%
Events; SL-Led Impact; SL-Led Impact 1 0 1 0.0 0.0%
Expert Call; Events; Events; Events; SL-Led Impact 1 1 0 Inf 100.0%
Expert Call; Events; Events; Expert Call; Expert Call; Events 1 1 0 Inf 100.0%
Expert Call; Events; Expert Call; Events; Expert Call; Events 1 1 0 Inf 100.0%
Expert Call; Events; Expert Call; PLW/Onsite 1 0 1 0.0 0.0%
Expert Call; Events; Research Interview; Research Interview; Expert Call; Expert Call; Research Interview 1 0 1 0.0 0.0%
Expert Call; Events; SL-Led Impact; Expert Call; Events 1 0 1 0.0 0.0%
Expert Call; Expert Call; Events; Expert Call; SL-Led Impact; Events; Events 1 1 0 Inf 100.0%
Expert Call; Expert Call; Events; PLW/Onsite; SL-Led Impact; PLW/Onsite 1 0 1 0.0 0.0%
Expert Call; Expert Call; Expert Call; Expert Call; Expert Call; SL-Led Impact; Expert Call; Expert Call; Expert Call; Expert Call 1 1 0 Inf 100.0%
Expert Call; Expert Call; Expert Call; Expert Call; Research Interview 1 1 0 Inf 100.0%
Expert Call; Expert Call; Research Interview; Research Interview 1 0 1 0.0 0.0%
Expert Call; Expert Call; Research Interview; SL-Led Impact 1 0 1 0.0 0.0%
Expert Call; PLW/Onsite; Events; Research Interview 1 1 0 Inf 100.0%
Expert Call; Research Interview; Events; SL-Led Impact; Research Interview; Expert Call 1 1 0 Inf 100.0%
Expert Call; Research Interview; Expert Call; Research Interview; Events 1 1 0 Inf 100.0%
Expert Call; Research Interview; Research Interview; Expert Call; Events; Research Interview; Events; Research Interview; Expert Call 1 1 0 Inf 100.0%
Expert Call; Research Interview; SL-Led Impact; Events 1 0 1 0.0 0.0%
Expert Call; Service/Tool; Expert Call; Expert Call; Expert Call; Expert Call; SL-Led Impact 1 1 0 Inf 100.0%
Expert Call; SL-Led Impact; SL-Led Impact 1 1 0 Inf 100.0%
PLW/Onsite; Events; Expert Call; SL-Led Impact; Research Interview; Research Interview; Research Interview 1 0 1 0.0 0.0%
PLW/Onsite; Expert Call; Events; Events; Events 1 1 0 Inf 100.0%
PLW/Onsite; Expert Call; Events; Events; SL-Led Impact; Events 1 1 0 Inf 100.0%
PLW/Onsite; Expert Call; Events; Expert Call; Research Interview 1 1 0 Inf 100.0%
PLW/Onsite; Expert Call; SL-Led Impact; Events; Events; Events 1 0 1 0.0 0.0%
PLW/Onsite; Research Interview; Expert Call; Events; SL-Led Impact 1 0 1 0.0 0.0%
PLW/Onsite; SL-Led Impact; Expert Call; Events; Events 1 0 1 0.0 0.0%
Research Interview 1 0 1 0.0 0.0%
Research Interview; PLW/Onsite; Expert Call; Expert Call; Events; Expert Call; Expert Call; Expert Call; PLW/Onsite; SL-Led Impact; Expert Call 1 1 0 Inf 100.0%
Research Interview; PLW/Onsite; Research Interview; PLW/Onsite; Expert Call; PLW/Onsite 1 0 1 0.0 0.0%
Research Interview; Research Interview 1 1 0 Inf 100.0%
Research Interview; Research Interview; Events 1 1 0 Inf 100.0%
Research Interview; Research Interview; Expert Call; SL-Led Impact 1 0 1 0.0 0.0%
Research Interview; Research Interview; PLW/Onsite; Expert Call; Events; Expert Call; Expert Call 1 1 0 Inf 100.0%
Research Interview; SL-Led Impact; Expert Call; SL-Led Impact 1 0 1 0.0 0.0%
Research Interview; SL-Led Impact; Research Interview; Research Interview; Events; Research Interview 1 1 0 Inf 100.0%
Service/Tool 1 0 1 0.0 0.0%
Service/Tool; Expert Call; Expert Call 1 1 0 Inf 100.0%
SL-Led Impact; Events; Events 1 0 1 0.0 0.0%
SL-Led Impact; Events; Events; Events; Expert Call 1 1 0 Inf 100.0%
SL-Led Impact; Events; Expert Call; Expert Call; Events 1 0 1 0.0 0.0%
SL-Led Impact; Expert Call 1 0 1 0.0 0.0%
SL-Led Impact; Expert Call; Research Interview; PLW/Onsite 1 1 0 Inf 100.0%
SL-Led Impact; Expert Call; SL-Led Impact; Expert Call; Events 1 1 0 Inf 100.0%
SL-Led Impact; PLW/Onsite 1 0 1 0.0 0.0%
SL-Led Impact; PLW/Onsite; Events; Events; Events; Events 1 0 1 0.0 0.0%

Looking At Combinations of Different Groupings

  1. You see that Expert Call and Event combo is a good basis for renewals.
Renewal Ratio by Impact Interaction Combos Slim
II_Combo_Text_Slim N ATL Dropped Ratio Perc_Renewed
Events; Expert Call 5 5 0 Inf 100.0%
Events 4 1 3 0.3 25.0%
NA 4 2 2 1.0 50.0%
Expert Call; Events 3 2 1 2.0 66.7%
Expert Call; SL-Led Impact; Events 3 3 0 Inf 100.0%
Events; Research Interview 2 2 0 Inf 100.0%
Events; Research Interview; SL-Led Impact 2 1 1 1.0 50.0%
Expert Call 2 0 2 0.0 0.0%
Expert Call; Research Interview 2 1 1 1.0 50.0%
Expert Call; Research Interview; PLW/Onsite; Events 2 2 0 Inf 100.0%
Research Interview 2 1 1 1.0 50.0%
Research Interview; Expert Call; Events 2 1 1 1.0 50.0%
SL-Led Impact 2 1 1 1.0 50.0%
SL-Led Impact; Events; Expert Call 2 1 1 1.0 50.0%
SL-Led Impact; Expert Call 2 2 0 Inf 100.0%
Events; Expert Call; PLW/Onsite 1 0 1 0.0 0.0%
Events; Expert Call; PLW/Onsite; SL-Led Impact 1 0 1 0.0 0.0%
Events; Expert Call; Research Interview; PLW/Onsite 1 1 0 Inf 100.0%
Events; Expert Call; Research Interview; PLW/Onsite; SL-Led Impact 1 0 1 0.0 0.0%
Events; Expert Call; SL-Led Impact; PLW/Onsite 1 1 0 Inf 100.0%
Events; Expert Call; SL-Led Impact; Research Interview 1 1 0 Inf 100.0%
Events; PLW/Onsite; Expert Call 1 1 0 Inf 100.0%
Events; PLW/Onsite; SL-Led Impact; Expert Call; Research Interview 1 1 0 Inf 100.0%
Events; Research Interview; Expert Call 1 0 1 0.0 0.0%
Events; SL-Led Impact 1 0 1 0.0 0.0%
Events; SL-Led Impact; Expert Call 1 0 1 0.0 0.0%
Expert Call; Events; PLW/Onsite; SL-Led Impact 1 0 1 0.0 0.0%
Expert Call; Events; Research Interview 1 1 0 Inf 100.0%
Expert Call; Research Interview; PLW/Onsite 1 0 1 0.0 0.0%
Expert Call; Research Interview; SL-Led Impact 1 0 1 0.0 0.0%
Expert Call; Research Interview; SL-Led Impact; Events 1 0 1 0.0 0.0%
Expert Call; Service/Tool 1 1 0 Inf 100.0%
Expert Call; Service/Tool; Events 1 1 0 Inf 100.0%
Expert Call; SL-Led Impact 1 0 1 0.0 0.0%
Expert Call; SL-Led Impact; Service/Tool 1 1 0 Inf 100.0%
PLW/Onsite; Events; SL-Led Impact; Service/Tool 1 0 1 0.0 0.0%
PLW/Onsite; Research Interview; Expert Call; Events 1 1 0 Inf 100.0%
PLW/Onsite; SL-Led Impact 1 0 1 0.0 0.0%
PLW/Onsite; SL-Led Impact; Events 1 0 1 0.0 0.0%
PLW/Onsite; SL-Led Impact; Events; Expert Call 1 1 0 Inf 100.0%
PLW/Onsite; SL-Led Impact; Expert Call; Events 1 0 1 0.0 0.0%
Research Interview; Events; PLW/Onsite; SL-Led Impact; Expert Call 1 0 1 0.0 0.0%
Research Interview; Experience; Expert Call; Events 1 1 0 Inf 100.0%
Research Interview; Expert Call; SL-Led Impact; PLW/Onsite 1 1 0 Inf 100.0%
Research Interview; SL-Led Impact; Events 1 0 1 0.0 0.0%
Service/Tool 1 0 1 0.0 0.0%
SL-Led Impact; Events 1 0 1 0.0 0.0%
SL-Led Impact; Expert Call; Research Interview 1 0 1 0.0 0.0%
SL-Led Impact; Research Interview; Expert Call 1 0 1 0.0 0.0%