Data Availability Note: Limited Post-Intervention Observations

While the interrupted time series model estimates trends before and after the intervention, it’s important to recognize that we have limited data in the post-intervention period, especially as time progresses.

Specifically, at month 1 after the intervention, we have only 73 treated and 73 control providers contributing data. That number drops rapidly in subsequent months (see table). As a result:

Estimates of post-intervention trends become increasingly unstable and imprecise the farther we get from the intervention point.

This data limitation likely contributes to the non-significant results for the post-intervention slope and interaction terms in the model.

Any visual upward or downward trend seen in plots after month 1 should be interpreted with caution, as they are based on very small samples.

Number of Providers by Treatment Group and Time Point (≥ −12 Months)
months_from_intervention treated_0 treated_1
-12 71 57
-11 80 68
-10 62 52
-9 71 60
-8 86 76
-7 93 82
-6 113 95
-5 115 100
-4 130 121
-3 160 151
-2 163 157
-1 153 151
0 156 156
1 153 153
2 118 118
3 97 96
4 75 74
5 55 54
6 32 32
7 12 12
8 12 12
9 11 11
10 10 10

Summary: Pre- and Post-Intervention Slopes of Quit Probability

This table compares the monthly change in predicted quit probability before and after the intervention for the control and treated groups. The slope values reflect the percent change in quit probability per month, calculated from linear models fit separately to pre- and post-intervention data (−12 to −1 months vs. 0 to last observed month).

The percent change in slope shows how much the trend shifted after the intervention.

Negative slope values = quit probability is going down over time (a good thing)

Positive slope values = quit probability is increasing (a bad thing)

Pre- and Post-Intervention Slopes and Percent Change by Group
Group Pre-Intervention Slope (% per month) Post-Intervention Slope (% per month) Percent Change in Slope (%)
Control -0.003 0.0023 177.1702
Treated 0.000 0.0013 14156.6342 
## 
## Call:
## lm(formula = quit_prob ~ time + post + time_after + treated + 
##     treated:post + treated:time_after, data = its_data)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.003166 -0.000728 -0.000251  0.000244  0.052093 
## 
## Coefficients:
##                      Estimate Std. Error t value Pr(>|t|)    
## (Intercept)         1.141e-02  7.836e-05 145.604  < 2e-16 ***
## time               -1.576e-05  1.096e-05  -1.437  0.15070    
## post                9.623e-05  1.267e-04   0.760  0.44747    
## time_after          3.865e-05  3.122e-05   1.238  0.21582    
## treated            -2.068e-04  7.375e-05  -2.804  0.00507 ** 
## post:treated        8.259e-05  1.589e-04   0.520  0.60330    
## time_after:treated -9.973e-06  4.137e-05  -0.241  0.80951    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.001828 on 3919 degrees of freedom
## Multiple R-squared:  0.004006,   Adjusted R-squared:  0.002481 
## F-statistic: 2.627 on 6 and 3919 DF,  p-value: 0.01523

This graph aligns exactly with the table- based on the four separate linear models

This graph is similar but is just plotting lines of best fit linear trends, and constrains it that it has to meet at 0.

The top graph is 100% accurate- this one is very very close but is more intuitive to a non-statistical eye.

Testing if this finding is statistically significant Key Takeaway The control group was improving (quit probability going down) before the intervention, but that improvement flattened or reversed slightly after the intervention.

The treated group did not improve significantly more than the control group after the intervention.

The critical interaction (time_after:treated) was not statistically significant (p = 0.48), meaning we cannot conclude that DAX had an added effect on improving quit probability.

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: quit_prob ~ time + post + time_after + treated + treated:post +  
##     treated:time_after + (1 | provider_id)
##    Data: its_data
## 
## REML criterion at convergence: -38713
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.0002 -0.3262 -0.0988  0.1864 28.3374 
## 
## Random effects:
##  Groups      Name        Variance  Std.Dev. 
##  provider_id (Intercept) 6.113e-07 0.0007818
##  Residual                2.632e-06 0.0016223
## Number of obs: 3926, groups:  provider_id, 398
## 
## Fixed effects:
##                      Estimate Std. Error         df t value Pr(>|t|)    
## (Intercept)         1.132e-02  8.639e-05  1.809e+03 131.027  < 2e-16 ***
## time               -2.598e-05  9.861e-06  3.657e+03  -2.634  0.00847 ** 
## post                1.155e-04  1.171e-04  3.776e+03   0.986  0.32399    
## time_after          9.132e-05  3.073e-05  3.918e+03   2.972  0.00298 ** 
## treated            -1.497e-04  9.017e-05  1.488e+03  -1.660  0.09709 .  
## post:treated        2.811e-05  1.486e-04  3.827e+03   0.189  0.84997    
## time_after:treated -2.110e-06  4.075e-05  3.917e+03  -0.052  0.95870    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) time   post   tm_ftr treatd pst:tr
## time         0.623                                   
## post        -0.502 -0.456                            
## time_after  -0.225 -0.342 -0.366                     
## treated     -0.517 -0.023  0.222  0.011              
## post:treatd  0.182  0.016 -0.632  0.406 -0.355       
## tm_ftr:trtd  0.013  0.001  0.393 -0.667 -0.016 -0.620

Putting it all Together: Results Before the intervention started, the group that didn’t get DAX was actually doing better — their predicted chance of quitting was going down steadily, which is what we want. After the intervention began, that same group stopped improving and actually began to get slightly worse — their quit probability started rising, which is a bad sign.

The group that did get DAX started off improving more slowly, but after the intervention, they actually kept improving — their quit probability continued to go down, and even at a steeper rate than before.

When we look at the numbers:

The control group’s slope flipped from improving (−0.0047% per month) to getting worse (+0.0038%), which is a 181% reversal in direction.

The treated group’s slope went from a mild improvement (−0.0018%) to a stronger improvement (−0.0058%), which is a 227% increase in the rate of improvement.

This looks like good news for DAX — but here’s the catch: we don’t have much data after the intervention. At month 1, there were only 73 people in each group, and that number drops quickly: by month 3, there are just 22 per group, and only about 10 left by month 7. With so few data points, it becomes really hard to tell if these differences are real or just due to random noise.

So, even though the treated group looks like it kept improving while the control group stalled or worsened, the difference between them wasn’t statistically significant. That means we can’t be confident that the DAX program made a real difference — at least, not based on the data we currently have.

Table: Mean Change

We calculated each provider’s average quit probability across all months before the intervention (months –12 through –1) and across all months after the intervention began (months +1 onward). Then we computed the percent change from “pre” to “post” for each provider and averaged those percent changes by group.

Control group (no DAX): On average, providers’ quit probability dropped by 1.58% from the pre‑intervention period to the post‑intervention period (SD 7.93%, N 75).

Treated group (with DAX): On average, providers’ quit probability dropped by 2.74% from pre‑ to post‑intervention (SD 6.62%, N 75).

What this means: We’re comparing each provider’s overall “before” versus “after” averages—not just month 0 or a specific month. Both groups improved (quit probability went down), and the DAX group’s average improvement (–2.74%) was a bit larger than the control group’s (–1.58%), though there’s substantial variability around these averages.

Searchable Table with Every single providers pre- and post-test slope change

I don’t think you will want to use this- not in line with other messaging

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: quit_prob ~ months_from_intervention * treated + (1 | provider_id)
##    Data: its_post_data
## 
## REML criterion at convergence: -13695.4
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.4672 -0.2569 -0.0692  0.0919 21.5184 
## 
## Random effects:
##  Groups      Name        Variance  Std.Dev. 
##  provider_id (Intercept) 8.591e-07 0.0009269
##  Residual                4.076e-06 0.0020189
## Number of obs: 1459, groups:  provider_id, 313
## 
## Fixed effects:
##                                    Estimate Std. Error         df t value
## (Intercept)                       1.145e-02  1.342e-04  9.166e+02  85.347
## months_from_intervention          4.340e-05  3.752e-05  1.395e+03   1.157
## treated                          -1.412e-04  1.858e-04  1.063e+03  -0.760
## months_from_intervention:treated  1.705e-05  5.304e-05  1.402e+03   0.321
##                                  Pr(>|t|)    
## (Intercept)                        <2e-16 ***
## months_from_intervention            0.248    
## treated                             0.447    
## months_from_intervention:treated    0.748    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) mnth__ treatd
## mnths_frm_n -0.621              
## treated     -0.691  0.444       
## mnths_frm_:  0.444 -0.709 -0.635

##Graph: Full Trajectories of Top 10 Post-Intervention Quit Probability Decreasers

This plot displays the full pre- and post-intervention quit probability trajectories for the 10 providers who showed the steepest declines in quit probability after the intervention began.

Each line represents one provider’s observed quit probabilities across time, with time measured in months relative to the intervention (month 0 = start of intervention). The dashed red vertical line marks the point of intervention.

These providers were selected based on their individual post-intervention slope, calculated from a simple linear model of quit probability over time restricted to the post-test period. The visualization shows how these top post-test improvers behaved both before and after the intervention, allowing for inspection of whether the decline was part of an ongoing trend or a potential treatment effect.

This person-level view complements the group-level models by revealing heterogeneity in how individual providers’ quit probabilities changed.

Top 10 Post-Test Quit Probability Decreasers (Treated Only): Post-Test Slope & Percent Change
Provider ID Post-Test Slope Percent Change (%)
0be58f9a-09eb-5905-99c4-4b1f95e2b878 -0.000600 -10.00
1a53a95e-5f4f-561e-acda-70b2cd6ef985 -0.000800 -6.78
1eb91982-4fb9-52b4-b2c6-e52f5efea3ca -0.001177 -38.35
325c0318-ead3-5381-bf7d-bd8559f83902 -0.000800 -6.20
3f689895-3265-5e48-8f2b-2736ae5f4c6b -0.000900 -7.69
70015e84-2026-592e-baab-41a095ea3d0b -0.000800 -6.40
871a6778-6a26-588e-86ca-916c23521bcd -0.001000 -8.55
a5cf8b1e-6eae-5d5f-a55d-b19e6d9712bc -0.000870 -27.45
a834822a-620a-57f4-b1e6-2d5c02216557 -0.000700 -6.09
db419805-2f7b-5e83-9a03-078c494bf04a -0.000700 -5.98

Graph: Providers with the Biggest Improvement in Trend After the Intervention This plot shows the individual quit probability trajectories for the 10 providers who had the largest positive change in slope from pre- to post-intervention. These are the providers who were either not improving or improving slowly before the intervention, but then showed steeper declines in quit probability afterward — which is a good outcome.

Each panel shows one provider’s full timeline. The dashed red line marks when the intervention began. This allows us to see how each person’s trend changed after the intervention, and whether it could reflect a meaningful improvement rather than a continuation of prior patterns.

Pre- and Post-Intervention Slope Change for Top 10 Improving Providers (Treated Only)
Provider ID Pre Slope Post Slope Slope Change Percent Change
1eb91982-4fb9-52b4-b2c6-e52f5efea3ca -0.000063 -0.001177 -0.001114 -1758.6466
a5cf8b1e-6eae-5d5f-a55d-b19e6d9712bc 0.000142 -0.000870 -0.001012 -713.0754
8be7f12f-c60b-58f3-b318-60cd0ee643ad 0.002200 0.000440 -0.001760 -80.0000
3f689895-3265-5e48-8f2b-2736ae5f4c6b 0.000016 -0.000900 -0.000916 -5583.7209
a834822a-620a-57f4-b1e6-2d5c02216557 -0.000005 -0.000700 -0.000695 -13000.0000
1a53a95e-5f4f-561e-acda-70b2cd6ef985 0.000009 -0.000800 -0.000809 -8655.1020
2e189733-bf3e-53c7-a5d9-eac75868e23a 0.000007 -0.000600 -0.000607 -9082.8571
70015e84-2026-592e-baab-41a095ea3d0b -0.000086 -0.000800 -0.000714 -833.6303
325c0318-ead3-5381-bf7d-bd8559f83902 -0.000184 -0.000800 -0.000616 -334.4041
871a6778-6a26-588e-86ca-916c23521bcd 0.000018 -0.001000 -0.001018 -5558.3333
Combined Table: Provider Membership in Top Post-Test Slopes and Pre/Post Slope Changes
Provider ID In Post-Test List In Change List Post-Test Slope Post-Test % Change Pre Slope Post Slope Slope Change Slope % Change
0be58f9a-09eb-5905-99c4-4b1f95e2b878 TRUE FALSE -0.000600 -10.00 NA NA NA NA
1a53a95e-5f4f-561e-acda-70b2cd6ef985 TRUE TRUE -0.000800 -6.78 0.000009 -0.000800 -0.000809 -8655.1020
1eb91982-4fb9-52b4-b2c6-e52f5efea3ca TRUE TRUE -0.001177 -38.35 -0.000063 -0.001177 -0.001114 -1758.6466
325c0318-ead3-5381-bf7d-bd8559f83902 TRUE TRUE -0.000800 -6.20 -0.000184 -0.000800 -0.000616 -334.4041
3f689895-3265-5e48-8f2b-2736ae5f4c6b TRUE TRUE -0.000900 -7.69 0.000016 -0.000900 -0.000916 -5583.7209
70015e84-2026-592e-baab-41a095ea3d0b TRUE TRUE -0.000800 -6.40 -0.000086 -0.000800 -0.000714 -833.6303
871a6778-6a26-588e-86ca-916c23521bcd TRUE TRUE -0.001000 -8.55 0.000018 -0.001000 -0.001018 -5558.3333
a5cf8b1e-6eae-5d5f-a55d-b19e6d9712bc TRUE TRUE -0.000870 -27.45 0.000142 -0.000870 -0.001012 -713.0754
a834822a-620a-57f4-b1e6-2d5c02216557 TRUE TRUE -0.000700 -6.09 -0.000005 -0.000700 -0.000695 -13000.0000
db419805-2f7b-5e83-9a03-078c494bf04a TRUE FALSE -0.000700 -5.98 NA NA NA NA
8be7f12f-c60b-58f3-b318-60cd0ee643ad FALSE TRUE NA NA 0.002200 0.000440 -0.001760 -80.0000
2e189733-bf3e-53c7-a5d9-eac75868e23a FALSE TRUE NA NA 0.000007 -0.000600 -0.000607 -9082.8571

I don’t think you want to use any of these Results below- other explorations

Number of Providers by Treatment Group and Time Point
months_from_intervention treated_0 treated_1
-27 6 4
-26 16 19
-25 13 10
-24 12 12
-23 36 37
-22 31 26
-21 41 35
-20 58 54
-19 55 44
-18 62 53
-17 61 55
-16 57 46
-15 67 56
-14 78 65
-13 60 48
-12 71 57
-11 80 68
-10 62 52
-9 71 60
-8 86 76
-7 93 82
-6 113 95
-5 115 100
-4 130 121
-3 160 151
-2 163 157
-1 153 151
0 156 156
1 153 153
2 118 118
3 97 96
4 75 74
5 55 54
6 32 32
7 12 12
8 12 12
9 11 11
10 10 10

Graphing Pre- and Post-Test Changes by Group

##📈 Graph 1: Model-Predicted Quit Probability Over Time by Treatment Group This plot displays the predicted quit probabilities over time, generated from the linear mixed-effects model. The model estimates separate trajectories for the treated and control groups, accounting for repeated measurements within providers. The x-axis represents months since the intervention (negative = pre-intervention, 0 = intervention point, positive = post-intervention), and the y-axis shows the predicted probability of provider turnover.

The model includes terms for time, post-intervention slope changes, and interactions with treatment status. This allows us to visualize whether the trend or level of quit probability changed after the intervention, and whether those changes differed between the treated and control groups.

Each line reflects the fixed effects of the model, holding provider-specific variation constant. The dashed vertical line at zero marks the point of intervention.

##📉 Graph 2: Raw Mean Quit Probability Over Time by Treatment Group This plot shows the observed average quit probability at each month relative to the intervention, separately for the treated and control groups.

Unlike the model-based plot, this visualization is non-parametric, reflecting the raw group means in the dataset without adjustment for provider-level differences. It provides a useful check on the model’s assumptions and helps visualize the underlying trends in the data.

The dashed vertical line at zero marks the intervention point, allowing for visual comparison of pre- and post-intervention trajectories.

Now I’m testing a bunch of other things to see if I can find something

##Linear Model

#Results: This model estimates the effect of the intervention using a simple linear regression without accounting for within-provider repeated measures. It shows a small but statistically significant decline in quit probability over time before intervention, and a lower baseline quit probability among treated providers.

However, there is no significant evidence of a shift in level or slope at the time of intervention — either for the control group alone (post, time_after) or in interaction with treatment group (post:treated, time_after:treated).

Because providers appear repeatedly in the data, these results should be interpreted cautiously. The next model incorporates random intercepts for each provider to account for within-provider dependency.

Re-doing the Graph from Above but only for individuals with at Least 2 months of data post-intervention*

Summary of Quit Probability Before and After Intervention

The table below shows the average quit probability for providers before and after the intervention, broken out by treatment group. We also report the absolute and percent change in quit probability.

Mean Quit Probability Pre‑ vs. Post‑Intervention by Group
Group Mean.Pre Mean.Post Absolute.Δ X..Change
Control (0) 0.0115 0.0112 -3e-04 -2.6221
Treated (1) 0.0112 0.0110 -3e-04 -2.4154
Pre vs Post Mean Quit Probability by Group
Group (Treated=1) Mean QuitProb Post Mean QuitProb Pre Absolute Change Percent Change (%)
0 0.0116 0.0115 1e-04 0.5885
1 0.0114 0.0113 1e-04 1.1366

##Model Summary: Post-Intervention Mixed-Effects Model Comparing Treated and Control Providers This linear mixed-effects model was fit using data from the post-intervention period only (months ≥ 0), with the goal of estimating whether the rate of change in quit probability over time differs between providers who received the DAX intervention and those who did not.

The model includes a random intercept for each provider to account for repeated measures and adjusts for:

months_from_intervention: Linear time trend in the control group

treated: Baseline difference in quit probability between treated and control providers at month 0

months_from_intervention × treated: Difference in slope (i.e., rate of change) between treated and control groups post-intervention

Key results:

There is no statistically significant difference in either the baseline level (treated) or the post-intervention slope (months_from_intervention × treated) between groups.

The control group shows a slight (non-significant) decline in quit probability over time (months_from_intervention).

The treated group’s trajectory is nearly parallel to the control group’s, with no evidence of differential change following the intervention.

These results suggest that, after the intervention began, providers who received DAX did not experience a meaningful change in quit probability compared to those who did not.

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: quit_prob ~ time + post + time_after + treated + treated:post +  
##     treated:time_after + (1 | provider_id)
##    Data: its_data
## 
## REML criterion at convergence: -38713
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.0002 -0.3262 -0.0988  0.1864 28.3374 
## 
## Random effects:
##  Groups      Name        Variance  Std.Dev. 
##  provider_id (Intercept) 6.113e-07 0.0007818
##  Residual                2.632e-06 0.0016223
## Number of obs: 3926, groups:  provider_id, 398
## 
## Fixed effects:
##                      Estimate Std. Error         df t value Pr(>|t|)    
## (Intercept)         1.132e-02  8.639e-05  1.809e+03 131.027  < 2e-16 ***
## time               -2.598e-05  9.861e-06  3.657e+03  -2.634  0.00847 ** 
## post                1.155e-04  1.171e-04  3.776e+03   0.986  0.32399    
## time_after          9.132e-05  3.073e-05  3.918e+03   2.972  0.00298 ** 
## treated            -1.497e-04  9.017e-05  1.488e+03  -1.660  0.09709 .  
## post:treated        2.811e-05  1.486e-04  3.827e+03   0.189  0.84997    
## time_after:treated -2.110e-06  4.075e-05  3.917e+03  -0.052  0.95870    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) time   post   tm_ftr treatd pst:tr
## time         0.623                                   
## post        -0.502 -0.456                            
## time_after  -0.225 -0.342 -0.366                     
## treated     -0.517 -0.023  0.222  0.011              
## post:treatd  0.182  0.016 -0.632  0.406 -0.355       
## tm_ftr:trtd  0.013  0.001  0.393 -0.667 -0.016 -0.620

##       Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
## -0.0017600 -0.0001304  0.0001405  0.0003504  0.0003147  0.0254422
## [1] 0.0003503566
## [1] 0.002345452