Summary of Texas Clubhouse Engagement, by Month



Monthly Summary for Texas



Average Monthly Clubhouse Summary for Texas



Average Monthly Member Summary for Texas



## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
## `geom_smooth()` using formula = 'y ~ x'
## `geom_smooth()` using formula = 'y ~ x'


Program Effectiveness at The State Level

## 
## Call:
## lm(formula = total_engagement_score ~ month, data = incentive_period_data)
## 
## Residuals:
##       1       2       3       4       5       6 
## -246.52  496.62  -98.24 -219.10  -20.95   88.19 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)   
## (Intercept)  2523.95     420.55   6.002  0.00388 **
## month         440.86      73.02   6.037  0.00380 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 305.5 on 4 degrees of freedom
## Multiple R-squared:  0.9011, Adjusted R-squared:  0.8764 
## F-statistic: 36.45 on 1 and 4 DF,  p-value: 0.003796


In analyzing the incentive program for Texas clubhouses, our objective was to determine if there was a significant increase in engagement throughout the incentive period. Initial visualizations indicated an upward trend in engagement, leading us to hypothesize that the total engagement score would increase as the months progressed.
To formally assess this, we fit a linear regression model using month as the predictor for total engagement score. With a p-value of 0.00380 (less than our significance threshold of 0.05) and a positive coefficient for the month variable, we concluded that month had a significant positive impact on the total engagement score. Additionally, the model’s Multiple R-squared value of 0.9252 suggested that 90.11% of the variability in total engagement could be explained by the month, indicating a strong upward trend in engagement over the course of the incentive period.


Post Program Stability at the State Level

## [1] -221
## 
##  One Sample t-test
## 
## data:  post_incentive_period_data$total_engagement_score
## t = -0.47835, df = 1, p-value = 0.358
## alternative hypothesis: true mean is less than 6139
## 95 percent confidence interval:
##      -Inf 8834.953
## sample estimates:
## mean of x 
##      5918


Since one goal of this program was to develop better data collection habits, we aimed to determine if the Total Engagement Score remained stable or increased after the incentive program ended. The calculated difference between the final month of the incentive program and the average of the two months following was -226.5. This means that, on average, in the two months following the incentive program the Total Engagement Score decreased by approximately 226.5pts in the two months post-incentive. Using a one-sample t-test, we compared the average Total Engagement Score from the two post-incentive months to the final Total Engagement Score during the incentive period. With a significance level of 0.05 and a p-value of 0.3534, we determined that there was no statistically significant evidence of a drop in engagement in the first two months following the incentive period.


Limitations: Despite the lack of statistically significant evidence for a drop in total engagement, the 95% confidence interval for the true post-program mean spanned from \(-\infty\) to 8794.728, indicating high variability in Total Engagement Scores during the post-incentive period. This outcome is expected due to the limited observation period of only two months post-incentive. As data collection continues, we expect the variability to decrease, providing a clearer understanding of the program’s long-term impact on data collection habits in Texas clubhouses. For now, while there appears to be no significant drop in engagement, the results are ultimately inconclusive.


Engagement at The State Level

## 
## Call:
## lm(formula = mean_engagement_score ~ month, data = incentive_period_data)
## 
## Residuals:
##          1          2          3          4          5          6 
## -0.0438095  0.0630476 -0.0100952  0.0067619 -0.0163810  0.0004762 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 1.254381   0.054681  22.940 2.14e-05 ***
## month       0.073143   0.009495   7.704  0.00153 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.03972 on 4 degrees of freedom
## Multiple R-squared:  0.9369, Adjusted R-squared:  0.9211 
## F-statistic: 59.34 on 1 and 4 DF,  p-value: 0.001528


Recognizing that the incentive program naturally increased the number of records collected, we analyzed mean engagement scores instead of total engagement. This helped us understand trends in engagement quality independent of the number of records. A linear model fitted to the mean engagement scores showed a significant positive trend over time, with a coefficient of 0.073143 and a p-value of 0.00153. This indicates a statistically significant increase in average engagement, irrespective of the overall number of records or unique members. With a Multiple R-squared value of 0.9369, we concluded that month accounts for 93.69% of the variability in mean engagement score.


Limitations: While we identified a significant increase in mean engagement score, it is difficult to discern whether this improvement was due to increased engagement quality, better record-keeping, or a mix of both. Further analysis with more detailed qualitative data could help clarify the source of this increase.


## 
## Call:
## lm(formula = count ~ month, data = incentive_period_data)
## 
## Residuals:
##       1       2       3       4       5       6 
## -104.52  204.02  -19.44 -122.90   10.65   32.19 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  2212.15     181.05  12.219 0.000258 ***
## month         137.46      31.44   4.372 0.011945 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 131.5 on 4 degrees of freedom
## Multiple R-squared:  0.827,  Adjusted R-squared:  0.7837 
## F-statistic: 19.12 on 1 and 4 DF,  p-value: 0.01194


A linear model analysis at a significance level of \(\alpha = 0.05\) showed a significant increase in the number of records collected as the program progressed, with a p-value of 0.0119. This finding will help us draw inferences about the quality of records being collected throughout the program.

## 
## Call:
## lm(formula = count_level_0/unique_members ~ month, data = incentive_period_data)
## 
## Residuals:
##        1        2        3        4        5        6 
## -0.02805 -0.07332  0.06460  0.06284  0.11406 -0.14013 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)  
## (Intercept)  0.53993    0.14912   3.621   0.0223 *
## month        0.06271    0.02589   2.422   0.0726 .
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1083 on 4 degrees of freedom
## Multiple R-squared:  0.5945, Adjusted R-squared:  0.4932 
## F-statistic: 5.865 on 1 and 4 DF,  p-value: 0.07262
## 
## Call:
## lm(formula = count_level_1/unique_members ~ month, data = incentive_period_data)
## 
## Residuals:
##        1        2        3        4        5        6 
##  0.29298  0.01719 -0.30746 -0.25461 -0.10207  0.35397 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)   
## (Intercept)   3.3800     0.4250   7.953  0.00135 **
## month        -0.2827     0.0738  -3.830  0.01861 * 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.3087 on 4 degrees of freedom
## Multiple R-squared:  0.7858, Adjusted R-squared:  0.7322 
## F-statistic: 14.67 on 1 and 4 DF,  p-value: 0.01861
## 
## Call:
## lm(formula = count_level_2/unique_members ~ month, data = incentive_period_data)
## 
## Residuals:
##         1         2         3         4         5         6 
## -0.119839  0.164312 -0.024988 -0.003902  0.024714 -0.040297 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  1.29494    0.14477   8.945 0.000864 ***
## month        0.02497    0.02514   0.993 0.376818    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1052 on 4 degrees of freedom
## Multiple R-squared:  0.1978, Adjusted R-squared:  -0.002707 
## F-statistic: 0.9865 on 1 and 4 DF,  p-value: 0.3768
## 
## Call:
## lm(formula = count_level_3/unique_members ~ month, data = incentive_period_data)
## 
## Residuals:
##        1        2        3        4        5        6 
## -0.04449  0.15187 -0.11004 -0.04144  0.02801  0.01610 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.24025    0.13752   1.747 0.155556    
## month        0.25217    0.02388  10.560 0.000455 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.09989 on 4 degrees of freedom
## Multiple R-squared:  0.9654, Adjusted R-squared:  0.9567 
## F-statistic: 111.5 on 1 and 4 DF,  p-value: 0.0004549


To evaluate the quality of records being recorded we chose to look at the individual records individually.


For Level 0 records, we observed a slight upward trend, with a positive coefficient of 0.06271 and a p-value of 0.0726. Although this trend is not statistically significant at \(\alpha = 0.05\), it is important to note that even if it had been significant, it wouldn’t necessarily imply a lack of program success. Instead, it might indicate that incentives were less effective in clubhouses that faced challenges with attendance logging. However, the trend does suggest that more engagement is being documented overall.


For Level 1 records, we observed a significant decrease, as indicated by a negative coefficient of -0.2827 and a p-value of 0.01861. Level 1 records represent instances where attendance was logged, but no specific engagement activities were captured. The decrease in Level 1 records, combined with an increase in the overall number of records, suggests a positive impact of the program. This result indicates that more specific engagement activities are being captured over time.


For Level 2 records, the p-value of 0.3768 suggests no significant change in their number. This is somewhat expected, as Level 2 represents the baseline for the type of record clubhouses were already collecting prior to the program. These records remained steady throughout the program.


For Level 3 records, we found a highly significant increase, with a p-value of 0.000455. This increase, along with the rise in the total number of records over the program’s duration, suggests a substantial improvement in the quality of engagement data being collected. While we cannot definitively determine whether this quality increase was due to better record-keeping, increased engagement, or a combination of both, it is a positive outcome. It also suggests that subsequent phases of our investigation into the impact of engagement on clubhouses will benefit from richer data quality.



Summary of Individual Clubhouse Engagement, by Month



Monthly Summary Per Clubhouse



Summary Per Member within Each Clubhouse, Aggregated to Monthly Level



Program Effectiveness, by Clubhouse

##             Df Sum Sq Mean Sq F value   Pr(>F)    
## clubhouse    9  723.5   80.39      18 7.64e-13 ***
## Residuals   50  223.3    4.47                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = normalized_engagement_score ~ clubhouse, data = clubhouse_monthly_summary_details %>% filter(month >= 3 & month <= 8))
## 
## $clubhouse
##                                 diff          lwr        upr     p adj
## Clubhouse 01-Clubhouse 05  5.3000000   1.26084312  9.3391569 0.0025375
## Clubhouse 02-Clubhouse 05  7.7166667   3.67750979 11.7558235 0.0000029
## Clubhouse 03-Clubhouse 05  1.7116667  -2.32749021  5.7508235 0.9206564
## Clubhouse 04-Clubhouse 05 -1.0733333  -5.11249021  2.9658235 0.9964532
## Clubhouse 06-Clubhouse 05 10.1466667   6.10750979 14.1858235 0.0000000
## Clubhouse 07-Clubhouse 05  0.9050000  -3.13415688  4.9441569 0.9990475
## Clubhouse 08-Clubhouse 05  6.3916667   2.35250979 10.4308235 0.0001322
## Clubhouse 09-Clubhouse 05  4.9750000   0.93584312  9.0141569 0.0057855
## Clubhouse 10-Clubhouse 05  1.8800000  -2.15915688  5.9191569 0.8688816
## Clubhouse 02-Clubhouse 01  2.4166667  -1.62249021  6.4558235 0.6159630
## Clubhouse 03-Clubhouse 01 -3.5883333  -7.62749021  0.4508235 0.1213094
## Clubhouse 04-Clubhouse 01 -6.3733333 -10.41249021 -2.3341765 0.0001391
## Clubhouse 06-Clubhouse 01  4.8466667   0.80750979  8.8858235 0.0079374
## Clubhouse 07-Clubhouse 01 -4.3950000  -8.43415688 -0.3558431 0.0230103
## Clubhouse 08-Clubhouse 01  1.0916667  -2.94749021  5.1308235 0.9959749
## Clubhouse 09-Clubhouse 01 -0.3250000  -4.36415688  3.7141569 0.9999998
## Clubhouse 10-Clubhouse 01 -3.4200000  -7.45915688  0.6191569 0.1635354
## Clubhouse 03-Clubhouse 02 -6.0050000 -10.04415688 -1.9658431 0.0003866
## Clubhouse 04-Clubhouse 02 -8.7900000 -12.82915688 -4.7508431 0.0000001
## Clubhouse 06-Clubhouse 02  2.4300000  -1.60915688  6.4691569 0.6086855
## Clubhouse 07-Clubhouse 02 -6.8116667 -10.85082354 -2.7725098 0.0000402
## Clubhouse 08-Clubhouse 02 -1.3250000  -5.36415688  2.7141569 0.9839551
## Clubhouse 09-Clubhouse 02 -2.7416667  -6.78082354  1.2974902 0.4401545
## Clubhouse 10-Clubhouse 02 -5.8366667  -9.87582354 -1.7975098 0.0006119
## Clubhouse 04-Clubhouse 03 -2.7850000  -6.82415688  1.2541569 0.4178620
## Clubhouse 06-Clubhouse 03  8.4350000   4.39584312 12.4741569 0.0000004
## Clubhouse 07-Clubhouse 03 -0.8066667  -4.84582354  3.2324902 0.9996210
## Clubhouse 08-Clubhouse 03  4.6800000   0.64084312  8.7191569 0.0118650
## Clubhouse 09-Clubhouse 03  3.2633333  -0.77582354  7.3024902 0.2121671
## Clubhouse 10-Clubhouse 03  0.1683333  -3.87082354  4.2074902 1.0000000
## Clubhouse 06-Clubhouse 04 11.2200000   7.18084312 15.2591569 0.0000000
## Clubhouse 07-Clubhouse 04  1.9783333  -2.06082354  6.0174902 0.8314260
## Clubhouse 08-Clubhouse 04  7.4650000   3.42584312 11.5041569 0.0000061
## Clubhouse 09-Clubhouse 04  6.0483333   2.00917646 10.0874902 0.0003432
## Clubhouse 10-Clubhouse 04  2.9533333  -1.08582354  6.9924902 0.3362675
## Clubhouse 07-Clubhouse 06 -9.2416667 -13.28082354 -5.2025098 0.0000000
## Clubhouse 08-Clubhouse 06 -3.7550000  -7.79415688  0.2841569 0.0886416
## Clubhouse 09-Clubhouse 06 -5.1716667  -9.21082354 -1.1325098 0.0035268
## Clubhouse 10-Clubhouse 06 -8.2666667 -12.30582354 -4.2275098 0.0000006
## Clubhouse 08-Clubhouse 07  5.4866667   1.44750979  9.5258235 0.0015594
## Clubhouse 09-Clubhouse 07  4.0700000   0.03084312  8.1091569 0.0468628
## Clubhouse 10-Clubhouse 07  0.9750000  -3.06415688  5.0141569 0.9982942
## Clubhouse 09-Clubhouse 08 -1.4166667  -5.45582354  2.6224902 0.9749371
## Clubhouse 10-Clubhouse 08 -4.5116667  -8.55082354 -0.4725098 0.0176168
## Clubhouse 10-Clubhouse 09 -3.0950000  -7.13415688  0.9441569 0.2750158


Since the various clubhouses have differences in membership numbers and average daily attendance, we decided to normalize the total engagement scores to account for the variation in Total engagement score. This is the measure we will use for Program Effectiveness.

With Clubhouse 05 being ineligible for incentive pay, it served as a reference for the purposes of this analysis.


The ANOVA results indicate a significant effect of the clubhouse on the normalized engagement score, with a p-value of 4.8e-09. This shows that the differences in engagement scores among the clubhouses are statistically significant, meaning that some clubhouses perform better or worse in terms of normalized engagement compared to others.


To try to determine which clubhouses performed when compared against one another we chose to run a Tukey HSD test. When compared head to head against each other clubhouse, many of the results comparisons were not statistically significant at a level of \(\alpha\ = 0.05\). Takeaways from these results:

Top Performers These clubhouses consistently showed higher engagement scores when compared to other clubhouses:

  • Clubhouse 06 consistently showed significantly higher engagement scores compared to other clubhouses. It was significantly different from seven other clubhouses in pairwise comparisons, indicating strong overall performance. Clubhouse 06 had one of the highest positive mean differences when compared to other clubhouses.

  • Clubhouse 02 Clubhouse also performed strongly, with significantly higher engagement scores compared to San Antonio and several other clubhouses. Clubhouse 02 consistently showed high engagement scores and was significantly outperformed only by Clubhouse 06.

  • Clubhouse 08 performed well, with higher engagement scores compared to San Antonio and some other clubhouses. It demonstrated significant positive differences, indicating that its engagement levels were better than many of the other clubhouses.


Mid Performers These clubhouses did not show consistent significant differences in head to head comparisons, but in general showed lower engagement scores than other clubhouses,

  • Clubhouse 01 generally performed better than some clubhouses but did not significantly outperform many others, placing it in the middle category. It had mixed results in pairwise comparisons, neither consistently high nor consistently low.

  • Clubhouse 10 showed engagement scores that were not significantly different from Clubhouse 05 or many others, placing it firmly in the middle. It did, however, have a significant negative difference when compared to Clubhouse 06, suggesting it wasn’t among the top performers.

  • Clubhouse 09 showed mixed performance, with scores that were significantly higher compared to San Antonio but lower compared to the top performers. It is classified as a mid-level performer due to its mixed pairwise results.

Under Performers

  • Clubhouse 03 significantly underperformed compared to Clubhouse 06, Clubhouse 02, and Clubhouse 08. It also had negative mean differences relative to San Antonio.

  • Clubhouse 04 showed significantly lower scores compared to Clubhouse 06, Clubhouse 02, and Clubhouse 08. Its performance was consistently lower, and it was outperformed by most other clubhouses in pairwise comparisons.

  • Clubhouse 07 served as a reference in many comparisons and consistently showed lower engagement scores compared to top performers like Clubhouse 06, Clubhouse 02, and Clubhouse 08. It was outperformed in several pairwise tests, indicating it was one of the lower performers in this study.


## 
## Call:
## lm(formula = normalized_engagement_score ~ month + flourish + 
##     previous_system, data = clubhouse_monthly_summary_details %>% 
##     filter(month >= 3 & month <= 8))
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.3829 -1.5129  0.1729  1.4523  6.0066 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                 0.8019     1.7984   0.446 0.657489    
## month                       0.6653     0.1771   3.757 0.000429 ***
## flourish                    5.1717     1.3523   3.824 0.000347 ***
## previous_systemFlourish 2  -0.4469     0.8729  -0.512 0.610775    
## previous_systemKintone      6.6408     1.1712   5.670 6.03e-07 ***
## previous_systemSalesforce  -4.0425     0.9562  -4.227 9.37e-05 ***
## previous_systemUnknown      4.2108     1.1712   3.595 0.000711 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.342 on 53 degrees of freedom
## Multiple R-squared:  0.6929, Adjusted R-squared:  0.6581 
## F-statistic: 19.93 on 6 and 53 DF,  p-value: 5.031e-12


We fit a linear model to explain the variability in engagement scores among clubhouses, accounting for 69.29% of the variability (Multiple R-Squared = 0.6929). Below are the significant factors influencing the engagement levels at each clubhouse, based on a significance level of \(\alpha = 0.05\).

Month * Month had a significantly positive effect on engagement scores, indicating that each successive month during the program led to increased engagement levels. (\(0.000429 < 0.05\)) * This result is consistent with the overall trend of program effectiveness across Texas

Flourish * Clubhouses that used Flourish 3 to collect their data showed a significant improvement in engagement scores compared to those that did not (\(p = 0.000347 < 0.05\)).

Previous system: The data system used during this study was Flourish 3, so it served as the reference for this model. Two clubhouses were already using Flourish 3 prior to the start of this study.
Flourish 2: Clubhouses previously using Flourish 2 had slightly lower engagement scores compared to those who previously used other systems, but this difference was not significant (\(p = 0.610775 > 0.05\)). Kintone and Unknown Systems: Clubhouses that transitioned from Kintone or Unknown* systems experienced a significant positive increase in engagement scores during the incentive period (\(p = 6.03e-07\) for Kintone, \(p = 0.000711\) for Unknown). Salesforce: Clubhouses that previously used Salesforce showed significantly lower engagement scores compared to others, This could indicate a potential challenge in adapting to the new system (\(p = 9.37e-05 < 0.05\)).


Since the model only accounts for a little more than 69% of the variability in engagement scores, it’s possible there are other data points we should be looking for.

  • While the previous system was unknown, we do know that it is not one of the other systems listed


Engagement Levels, by Clubhouse


##             Df Sum Sq Mean Sq F value   Pr(>F)    
## clubhouse    9 13.594  1.5105   23.84 3.64e-15 ***
## Residuals   50  3.167  0.0633                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = mean_engagement_score ~ clubhouse, data = clubhouse_monthly_summary_details %>% filter(month >= 3 & month <= 8))
## 
## $clubhouse
##                                   diff         lwr         upr     p adj
## Clubhouse 01-Clubhouse 05  0.383333333 -0.09768336  0.86435003 0.2276840
## Clubhouse 02-Clubhouse 05  1.155000000  0.67398330  1.63601670 0.0000000
## Clubhouse 03-Clubhouse 05  0.311666667 -0.16935003  0.79268336 0.5063649
## Clubhouse 04-Clubhouse 05 -0.160000000 -0.64101670  0.32101670 0.9823716
## Clubhouse 06-Clubhouse 05  1.355000000  0.87398330  1.83601670 0.0000000
## Clubhouse 07-Clubhouse 05 -0.001666667 -0.48268336  0.47935003 1.0000000
## Clubhouse 08-Clubhouse 05  0.525000000  0.04398330  1.00601670 0.0223172
## Clubhouse 09-Clubhouse 05  0.115000000 -0.36601670  0.59601670 0.9984167
## Clubhouse 10-Clubhouse 05  0.186666667 -0.29435003  0.66768336 0.9525754
## Clubhouse 02-Clubhouse 01  0.771666667  0.29064997  1.25268336 0.0001031
## Clubhouse 03-Clubhouse 01 -0.071666667 -0.55268336  0.40935003 0.9999669
## Clubhouse 04-Clubhouse 01 -0.543333333 -1.02435003 -0.06231664 0.0156557
## Clubhouse 06-Clubhouse 01  0.971666667  0.49064997  1.45268336 0.0000008
## Clubhouse 07-Clubhouse 01 -0.385000000 -0.86601670  0.09601670 0.2227137
## Clubhouse 08-Clubhouse 01  0.141666667 -0.33935003  0.62268336 0.9924533
## Clubhouse 09-Clubhouse 01 -0.268333333 -0.74935003  0.21268336 0.7030996
## Clubhouse 10-Clubhouse 01 -0.196666667 -0.67768336  0.28435003 0.9353413
## Clubhouse 03-Clubhouse 02 -0.843333333 -1.32435003 -0.36231664 0.0000185
## Clubhouse 04-Clubhouse 02 -1.315000000 -1.79601670 -0.83398330 0.0000000
## Clubhouse 06-Clubhouse 02  0.200000000 -0.28101670  0.68101670 0.9287645
## Clubhouse 07-Clubhouse 02 -1.156666667 -1.63768336 -0.67564997 0.0000000
## Clubhouse 08-Clubhouse 02 -0.630000000 -1.11101670 -0.14898330 0.0026026
## Clubhouse 09-Clubhouse 02 -1.040000000 -1.52101670 -0.55898330 0.0000001
## Clubhouse 10-Clubhouse 02 -0.968333333 -1.44935003 -0.48731664 0.0000009
## Clubhouse 04-Clubhouse 03 -0.471666667 -0.95268336  0.00935003 0.0588332
## Clubhouse 06-Clubhouse 03  1.043333333  0.56231664  1.52435003 0.0000001
## Clubhouse 07-Clubhouse 03 -0.313333333 -0.79435003  0.16768336 0.4988008
## Clubhouse 08-Clubhouse 03  0.213333333 -0.26768336  0.69435003 0.8981183
## Clubhouse 09-Clubhouse 03 -0.196666667 -0.67768336  0.28435003 0.9353413
## Clubhouse 10-Clubhouse 03 -0.125000000 -0.60601670  0.35601670 0.9970015
## Clubhouse 06-Clubhouse 04  1.515000000  1.03398330  1.99601670 0.0000000
## Clubhouse 07-Clubhouse 04  0.158333333 -0.32268336  0.63935003 0.9835775
## Clubhouse 08-Clubhouse 04  0.685000000  0.20398330  1.16601670 0.0007691
## Clubhouse 09-Clubhouse 04  0.275000000 -0.20601670  0.75601670 0.6738194
## Clubhouse 10-Clubhouse 04  0.346666667 -0.13435003  0.82768336 0.3559522
## Clubhouse 07-Clubhouse 06 -1.356666667 -1.83768336 -0.87564997 0.0000000
## Clubhouse 08-Clubhouse 06 -0.830000000 -1.31101670 -0.34898330 0.0000256
## Clubhouse 09-Clubhouse 06 -1.240000000 -1.72101670 -0.75898330 0.0000000
## Clubhouse 10-Clubhouse 06 -1.168333333 -1.64935003 -0.68731664 0.0000000
## Clubhouse 08-Clubhouse 07  0.526666667  0.04564997  1.00768336 0.0216184
## Clubhouse 09-Clubhouse 07  0.116666667 -0.36435003  0.59768336 0.9982301
## Clubhouse 10-Clubhouse 07  0.188333333 -0.29268336  0.66935003 0.9499560
## Clubhouse 09-Clubhouse 08 -0.410000000 -0.89101670  0.07101670 0.1572219
## Clubhouse 10-Clubhouse 08 -0.338333333 -0.81935003  0.14268336 0.3897594
## Clubhouse 10-Clubhouse 09  0.071666667 -0.40935003  0.55268336 0.9999669


An ANOVA was conducted to determine whether there were significant differences in mean engagement scores among different clubhouses during the program duration (March - August).

The p-value for the clubhouse variable is 3.64e-15, which is highly significant (p < 0.05), suggesting that there are significant differences in mean engagement scores between the different clubhouses.


As with the Program effectivness analysis, Clubhouse 05 served as a control group for the purposes of this analysis.


Top Performing Clubhouses: Clubhouse 06, Clubhouse 02, and Clubhouse 08 consistently showed significantly higher mean engagement scores compared to Clubhouse 05 and other clubhouses. Clubhouse 06 also outperformed other clubhouses in the pairwise comparisons.

Mid Performing Clubhouses Clubhouse 07, Clubhouse 01, and Clubhouse 03 all had non-significant differences in engagement scores compared to Clubhouse 05, suggesting that their performance was not statistically different from that of the reference group.

Under Performing Clubhouses Clubhouse 04, Clubhouse 10, and Clubhouse 09 had significantly lower engagement scores compared to the top-performing clubhouses. 

## 
## Call:
## lm(formula = mean_engagement_score ~ month + flourish + previous_system, 
##     data = clubhouse_monthly_summary_details %>% filter(month >= 
##         3 & month <= 8))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.46324 -0.13114 -0.01985  0.17253  0.35324 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)               -0.12338    0.17179  -0.718    0.476    
## month                      0.09137    0.01691   5.402 1.59e-06 ***
## flourish                   1.24000    0.12917   9.599 3.45e-13 ***
## previous_systemFlourish 2 -0.11083    0.08338  -1.329    0.189    
## previous_systemKintone     1.00750    0.11187   9.006 2.87e-12 ***
## previous_systemSalesforce -0.42750    0.09134  -4.680 2.02e-05 ***
## previous_systemUnknown     0.80750    0.11187   7.218 2.01e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2237 on 53 degrees of freedom
## Multiple R-squared:  0.8417, Adjusted R-squared:  0.8238 
## F-statistic: 46.97 on 6 and 53 DF,  p-value: < 2.2e-16


We fit a linear model to explain the variability in mean engagement scores among clubhouses, accounting for 84.17% of the variability (Multiple R-Squared = 0.8417). Below are the significant factors influencing the engagement levels at each clubhouse, based on a significance level of \(\alpha = 0.05\).

Month * Month had a significantly positive effect on mean engagement scores, indicating that each successive month during the program led to increased engagement levels. (\(1.59e-06 < 0.05\)) * This result is consistent with the overall trend of program effectiveness across Texas

Flourish * Clubhouses that used Flourish 3 to collect their data showed a significant improvement in mean engagement scores compared to those that did not (\(p = 3.45e-13 < 0.05\)).

Previous system: The data system used during this study was Flourish 3, so it served as the reference for this model. Two clubhouses were already using Flourish 3 prior to the start of this study.
Flourish 2: Clubhouses previously using Flourish 2 had slightly lower engagement scores compared to those who previously used other systems, but this difference was not significant (\(p = 2.01e-09> 0.05\)). Kintone and Unknown Systems: Clubhouses that transitioned from Kintone or Unknown* systems experienced a significant positive increase in engagement scores during the incentive period (\(p = 2.87e-12\) for Kintone, \(p = 2.01e-09\) for Unknown). Salesforce: Clubhouses that previously used Salesforce showed significantly lower engagement scores compared to others, This could indicate a potential challenge in adapting to the new system (\(p = 2.01e-09 < 0.05\)).


Ultimately these results mirror the results of the findings for the normalized engagement score (Program Effectiveness). The main difference is that this model accounts for more variability in the data than the one for program effectiveness. This means that it is possible there are more factors that play into program effectiveness than there are that affect overall clubhouse engagement. It is possible that other factors are influencing our models as well. Our next step would be to look at individual member characteristics to see how it affects program effectiveness.

  • While the previous system was unknown, we do know that it is not one of the other systems listed


Summary

We are not finished with our exploration of this data and the concept of providing incentives to clubhouses in order to achieve better collection habits. There are many other factors that we would like to explore, including: * Other variables that might affect engagement and and quality data collection * How individual member demographics play into the overall program participation and engagement within a clubhouse * How increased engagement affects efficacy of the Clubhouse Model overall.