Study reminder / overview
In this document, we present a first draft of the analysis process for the Purrble-at-Oxford RCT – given that I (Petr) have no formal education in any of this, I’ll much appreciate careful sense of whether I’m doing the right things / have not forgotten something important.
As a reminder, this is a 2-arm RCT, with 1:1 allocation
- Purrble + SSI: (Students first receive Purrble, then access to SSI 1 week later)
- Waitlist: treatment as usual (no Purrble nor SSI), with Purrble received after week 4 of the study.
The study has ran during the last 4 weeks of Oxford Trinity term. Working together with the peer counselling support, we recruited n=99 participants, who were randomised to Purrble (n=49) and waitlist (n=50) conditions. Eligibility criteria included GAD7 > 10 on the screener, living in Oxford during the term, and aged 18-25.
The primary pre-registered outcome was to test for differences in GAD7 outcomes between control and active group across pre/mid/post datapoints.
Known concerns
Note that we had initially aimed to recruit a much higher n (n=150-200) which unfortunately ended up being impossible due to Claudia leaving the team at a very short notice (and failing to set the recruitment channels sufficiently beforehand).
–> we could be under-powered for the primary/secondary effects and might need to consider this a pilot rather than definitive study
The timing of the deployment has also been less than ideal, for the same reasons. Our original plan has been to deploy from week3-of-term to week7-of-term (out of 10-week term). Due to the challenges, we had to deploy week6-of-term to week10-of-term, where it was likely that the main stressors would be already disappearing towards the last two weeks of the study. Our planned 2 week follow-up has also been affected.
Data collection timepoints
| Outcome measure | Screener | Baseline (wk0) | wk1 | wk2 | wk3 | wk4 | post-deployment process analysis |
|---|---|---|---|---|---|---|---|
| Primary/secondary outcomes (GAD, PHQ, ER beliefs, ERQ-SE, SHS, BHS) | GAD7 only | X | X | X | |||
| Engagement measure (TWEETS) | x | x | x | ||||
| Simpler weekly survey (wellbeing (GAD2, PHQ2, Purrble use) | x | x | x | x | x | interviews and co-designs | |
| SSI offered to active group | XX |
The analysis below is focused predominantly on the main assessment points, i.e., wk0, wk2, wk4, and the main primary/secondary outcomes.
Note – we deliberately ignore the screener GAD in the analysis (where we selected people based on GAD cut-off value) to reduce regression to the mean. This also means that a couple of our participants have GAD < 10 on wk0, which is fine and not a sign of recruitement eligibility fail.
Sample and retention
The tables below show the ages, gender, and undergraduate / postgraduate student distribution for the 99 participants who have been randomised into the study. Overall, we also saw a good retention in the study, with all waitlist participants and most Purrble participants submitting their weekly surveys on time.
| gender | student | age | |
|---|---|---|---|
| Male :20 | Undergraduate student:68 | Min. :18.00 | |
| Female :74 | Postgraduate student :30 | 1st Qu.:20.00 | |
| Non-binary / third gender: 5 | Other : 1 | Median :21.00 | |
| NA | NA | Mean :21.37 | |
| NA | NA | 3rd Qu.:23.00 | |
| NA | NA | Max. :25.00 |
| condition | wk0 | wk2 | wk4 |
|---|---|---|---|
| Waitlist | 100% | 100% | 100% |
| Purrble | 100.0% | 91.8% | 87.8% |
Identifying ‘completers’
While we have not pre-specified how we will identify ‘completers’ for the Purrble+SSI condition, beyond submitting all surveys; and all pre-registered analyses are on intent-to-treat basis. However, as part of exploratory analyses, it will be interesting to identify two main situations where we would assume null effects (corresponding to the two intervention components) and that, essentially, the participants ‘dropped out’ from the ‘treatment’:
First, if participants do not go through the SSI at all. This is easy to track by looking at SSI completion rates in the Purrble group, which are tracked by Qualtrics.
Second, perhaps more importantly, if a participant does not engage with Purrble, it is unlikely that they would receive any benefits. This is somewhat difficult to measure directly. The closest measure we have is the question “How often did you engage with Purrble this week” (Not at all, Several days, More than half the days, Nearly every day, More than once a day), which has been asked active group participants during the weekly surveys (i.e., wk1, wk2, wk3, wk4).
Descriptive analysis
Missing data and ranges check
First, let’s have a look at the range & any missing-ness for the primary / secondary outcome contructs(e.g., if participants were to have skipped one or more questions).
| wk | GAD_total | PHQ_total | ERQ_total | ERbeliefs_total | bhs_total | shs_total | |
|---|---|---|---|---|---|---|---|
| wk0:99 | Min. : 2.00 | Min. : 1.00 | Min. :12.00 | Min. : 4.00 | Min. : 0.000 | Min. : 1.00 | |
| wk2:95 | 1st Qu.: 8.00 | 1st Qu.: 8.00 | 1st Qu.:30.25 | 1st Qu.: 9.00 | 1st Qu.: 2.000 | 1st Qu.: 9.00 | |
| wk4:93 | Median :11.00 | Median :12.00 | Median :38.00 | Median :12.00 | Median : 4.000 | Median :12.00 | |
| NA | Mean :11.35 | Mean :12.26 | Mean :38.70 | Mean :11.98 | Mean : 4.476 | Mean :11.71 | |
| NA | 3rd Qu.:15.00 | 3rd Qu.:16.00 | 3rd Qu.:46.00 | 3rd Qu.:15.00 | 3rd Qu.: 7.000 | 3rd Qu.:15.00 | |
| NA | Max. :21.00 | Max. :27.00 | Max. :67.00 | Max. :19.00 | Max. :12.000 | Max. :21.00 | |
| NA | NA’s :1 | NA’s :1 | NA’s :1 | NA’s :2 | NA’s :1 | NA’s :1 |
All seems fine in terms of outcome ranges; and missingness is very low, with just one person (cgbtzr) missing all their surveys in wk4 (thus the 1 NA across outcomes), and one another forgetting a question in ERbeliefs only. We’ll remove cgbtzr from the dataset for now, leaving 92 survey answers in wk4.
Cronbach alphas for all scales
Now, let’s have a look at the alpha values for each of the scales – all of these look pretty, with each alpha being \(>.8\).
| scale | std.alpha | raw_alpha |
|---|---|---|
| gad | 0.8394746 | 0.8363000 |
| phq | 0.8244805 | 0.8230519 |
| erq | 0.8736644 | 0.8691469 |
| erb | 0.8295610 | 0.8291674 |
| bhs | 0.8606546 | 0.8595370 |
| shs | 0.8652646 | 0.8651804 |
| tweets | 0.8817511 | 0.8782698 |
Note –the alphas were calculated by psych::alpha() over the respective scale answers whole dataset (i.e., wk0,wk2,wk4 and both conditions). There is nothing to suggest these should differ when broken down to conditions / datapoints, but let me know if that is what I should double check too.
Plots for primary/secondary measures
Just as a way of helping us eye-ball the data, the graphs below plot the data for the main outcome measures, with simple geom_smooth (method = “lm”) overlay. I’ve also included a violin_plots in lieu of raincloud_plots, as those are cumbersome to work with (but we can have them for any final publication).
Plot TWEETS (engagement measure)
Note that TWEETS are only available for wk2 and wk4 datapoints (as students had nothing to report on at baseline); and that the measure is obviously only available to Purrble group participants. As a reminder, TWEETS has 9 question scored on a 0-4 scale (strongly disagree to strongly agree), with three possible subscales (behaviour, cognition, affect) which we could analyse separately in explorative analyses.
The exact survey questions as presented in qualtrics are below.
Participants were asked: “Thinking about using Purrble last week, I
feel that:” 
A few descriptives to start with – all looks good.
| TWEETS_total | wk | |
|---|---|---|
| Min. : 1.00 | wk0: 0 | |
| 1st Qu.:19.50 | wk2:45 | |
| Median :24.00 | wk4:42 | |
| Mean :22.85 | NA | |
| 3rd Qu.:27.00 | NA | |
| Max. :34.00 | NA |
And now distribution across the two weeks – we see that, on average, the TWEETS scores have not changed massively, but that there is some movement within participants week to week. Note that the raincloud graphs misses 7 participants who did not have full TWEETS scores for both wk2 and wk4.
Outcomes
Across all variables, we start by calculating mixed linear models including time (wk_num), experimental condition (condition) and a random effect to account for nesting within participants. We consider two possible random effects: either 1 | PurrbleID (varied intercept only and keeping slopes same for all) or wk_num | PurrbleID varied slope and intercept)).
We then add further covariates into the model as per pre-registration: baseline anxiety (GAD-at-wk0) and gender (gender). Note that we cannot include TWEETS as an engagement covariate as that was asked of the Purrble group only.
Finally, we also calculate a table of both within and between effect-sizes for each of the outcomes.
GAD
The GAD analysis already includes the ‘gad_baseline’ covariate automatically, so the only other covariate to add the gender, as per the four models below.
We see that adding random slopes does not improve the fit. Model 2 (~wk + condition + (1|Purrble) + gender) is a marginally best of the four.
Purrble is not statistically significant in any of the models
##
## ===========================================================================
## Dependent variable:
## -------------------------------------------
## GAD_total
## 1 | ID 1 | ID wk | ID wk | ID
## (1) (2) (3) (4)
## ---------------------------------------------------------------------------
## wk_num -0.836*** -0.835*** -0.832*** -0.831***
## (0.109) (0.109) (0.111) (0.111)
## t = -7.647 t = -7.642 t = -7.473 t = -7.469
## p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## conditionPurrble -0.760 -0.752 -0.603 -0.587
## (0.739) (0.753) (0.733) (0.748)
## t = -1.029 t = -0.998 t = -0.822 t = -0.786
## p = 0.304 p = 0.319 p = 0.411 p = 0.433
##
## genderMale -0.482 -0.498
## (0.933) (0.927)
## t = -0.517 t = -0.537
## p = 0.606 p = 0.592
##
## genderNon-binary / third gender -0.077 -0.181
## (1.736) (1.718)
## t = -0.045 t = -0.105
## p = 0.965 p = 0.917
##
## Constant 13.392*** 13.489*** 13.312*** 13.413***
## (0.560) (0.597) (0.541) (0.578)
## t = 23.903 t = 22.601 t = 24.600 t = 23.195
## p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## ---------------------------------------------------------------------------
## isSingular? NO NO NO NO
## Observations 286 286 286 286
## Log Likelihood -791.087 -788.648 -790.223 -787.790
## Akaike Inf. Crit. 1592.173 1591.296 1594.446 1593.580
## Bayesian Inf. Crit. 1610.453 1616.888 1620.038 1626.484
## ===========================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the two (1 | PurrbleID) models (first row) and the twt (wk_num | PurrbleID) models (second row), in the same order as per the table above.
PHQ
PHQ can include both baseline-GAD and gender as co-variates. Note that all of the random slopes models threw a isSingular fit warning (i.e., over-fitting risk), and that they also do not perform better than the simpler (1|PurrbleID). Model 3 seems to be best fit.
Purrble is not statistically significant in any of the models, although approaches significance (p = 0.061) in the best fitting model.
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## PHQ_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num -0.742*** -0.732*** -0.733*** -0.742*** -0.732*** -0.733***
## (0.110) (0.110) (0.110) (0.110) (0.112) (0.112)
## t = -6.737 t = -6.661 t = -6.663 t = -6.737 t = -6.564 t = -6.572
## p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## conditionPurrble -1.260 -1.383 -1.487 -1.265 -1.072 -1.198
## (0.967) (0.801) (0.793) (0.967) (0.787) (0.780)
## t = -1.302 t = -1.727 t = -1.874 t = -1.307 t = -1.363 t = -1.535
## p = 0.193 p = 0.085 p = 0.061 p = 0.192 p = 0.173 p = 0.125
##
## GAD_baseline 0.691*** 0.710*** 0.744*** 0.760***
## (0.102) (0.100) (0.100) (0.098)
## t = 6.765 t = 7.073 t = 7.441 t = 7.723
## p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## genderMale 2.322* 2.233*
## (0.986) (0.970)
## t = 2.355 t = 2.302
## p = 0.019 p = 0.022
##
## genderNon-binary / third gender 1.411 1.522
## (1.828) (1.790)
## t = 0.772 t = 0.850
## p = 0.441 p = 0.396
##
## Constant 14.370*** 5.291*** 4.543** 14.373*** 4.431** 3.750**
## (0.712) (1.472) (1.477) (0.713) (1.431) (1.438)
## t = 20.177 t = 3.594 t = 3.076 t = 20.171 t = 3.097 t = 2.608
## p = 0.000 p = 0.0004 p = 0.003 p = 0.000 p = 0.002 p = 0.010
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO YES YES YES
## Observations 286 286 286 286 286 286
## Log Likelihood -818.315 -800.709 -795.433 -818.314 -798.846 -793.686
## Akaike Inf. Crit. 1646.629 1613.418 1606.866 1650.629 1613.692 1607.372
## Bayesian Inf. Crit. 1664.909 1635.354 1636.113 1676.221 1642.940 1643.932
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the three (1 | PurrbleID) models (first row) and the three (wk_num | PurrbleID) models (second row), in the same order as per the table above:
ER Beliefs
ER beliefs can include both baseline-GAD and gender as co-variates. None of the random slope models had isSingular errors, and model 6 appears to work best according to AIC. QQ plots seem okay.
Purrble is significant across all model that include GAD-baseline covariate.
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## ERbeliefs_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num 0.234** 0.229** 0.229** 0.239** 0.232** 0.232**
## (0.072) (0.072) (0.072) (0.082) (0.082) (0.081)
## t = 3.264 t = 3.197 t = 3.195 t = 2.911 t = 2.846 t = 2.847
## p = 0.002 p = 0.002 p = 0.002 p = 0.004 p = 0.005 p = 0.005
##
## conditionPurrble 1.042 1.076* 1.194* 1.070 1.121* 1.238*
## (0.555) (0.525) (0.525) (0.554) (0.523) (0.524)
## t = 1.878 t = 2.050 t = 2.273 t = 1.931 t = 2.143 t = 2.364
## p = 0.061 p = 0.041 p = 0.024 p = 0.054 p = 0.033 p = 0.019
##
## GAD_baseline -0.236*** -0.245*** -0.242*** -0.252***
## (0.067) (0.066) (0.067) (0.066)
## t = -3.532 t = -3.688 t = -3.626 t = -3.800
## p = 0.0005 p = 0.0003 p = 0.0003 p = 0.0002
##
## genderMale -0.956 -0.998
## (0.653) (0.650)
## t = -1.465 t = -1.535
## p = 0.143 p = 0.125
##
## genderNon-binary / third gender -1.797 -1.734
## (1.210) (1.210)
## t = -1.485 t = -1.433
## p = 0.138 p = 0.152
##
## Constant 11.026*** 14.135*** 14.480*** 11.009*** 14.191*** 14.556***
## (0.414) (0.964) (0.977) (0.424) (0.972) (0.985)
## t = 26.662 t = 14.661 t = 14.817 t = 25.948 t = 14.605 t = 14.778
## p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 285 285 285 285 285 285
## Log Likelihood -682.060 -677.931 -674.402 -677.358 -673.067 -669.518
## Akaike Inf. Crit. 1374.121 1367.862 1364.804 1368.716 1362.134 1359.036
## Bayesian Inf. Crit. 1392.383 1389.777 1394.024 1394.284 1391.353 1395.561
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the three (1 | PurrbleID) models (first row) and the three (wk_num | PurrbleID) models (second row), in the same order as per the table above:
ERQ - Self-efficacy
NOTE – Caution required: I need to sense check with James how ERQ-SE is supposed to be scored. At the moment, all is summed into a single construct without any reverse coding. This seems to make sense on face validity and the cronbach alpha is good: 0.8736644; however, I would like to make sure.
ERQ-SE can include both baseline-GAD and gender as co-variates. None of the random slope models had isSingular errors, and model 6 appears to work best according to AIC. QQ plots seem a bit off but probably still okay.
Purrble is significant across all random-slopes models (wk_num | Purrble), but not for any (1|Purrble) ones. Theoretically, this could be quite understandable, as we assume that ERQ efficacy would change iff Purrble is used repeatedly and successfully to down-regulate own emotions; and we also know from TWEETS that some people find Purrble much more useful than others.
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## ERQ_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num 0.583** 0.574** 0.573** 0.597** 0.582** 0.580**
## (0.208) (0.207) (0.207) (0.224) (0.223) (0.223)
## t = 2.811 t = 2.766 t = 2.763 t = 2.667 t = 2.611 t = 2.606
## p = 0.005 p = 0.006 p = 0.006 p = 0.008 p = 0.010 p = 0.010
##
## conditionPurrble 2.768 2.884 3.213 3.907* 3.972* 4.388*
## (1.884) (1.815) (1.839) (1.852) (1.782) (1.802)
## t = 1.469 t = 1.590 t = 1.747 t = 2.110 t = 2.229 t = 2.434
## p = 0.142 p = 0.112 p = 0.081 p = 0.035 p = 0.026 p = 0.015
##
## GAD_baseline -0.685** -0.695** -0.677** -0.690**
## (0.231) (0.232) (0.228) (0.229)
## t = -2.967 t = -2.989 t = -2.971 t = -3.014
## p = 0.004 p = 0.003 p = 0.003 p = 0.003
##
## genderMale -0.636 -1.228
## (2.286) (2.238)
## t = -0.278 t = -0.549
## p = 0.781 p = 0.584
##
## genderNon-binary / third gender -5.237 -5.683
## (4.232) (4.174)
## t = -1.237 t = -1.361
## p = 0.217 p = 0.174
##
## Constant 36.153*** 45.164*** 45.519*** 35.583*** 44.509*** 45.014***
## (1.384) (3.319) (3.405) (1.439) (3.311) (3.393)
## t = 26.129 t = 13.610 t = 13.368 t = 24.725 t = 13.441 t = 13.268
## p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 286 286 286 286 286 286
## Log Likelihood -1000.838 -997.126 -992.253 -997.878 -994.179 -989.116
## Akaike Inf. Crit. 2011.676 2006.253 2000.506 2009.757 2004.357 1998.233
## Bayesian Inf. Crit. 2029.956 2028.189 2029.753 2035.349 2033.605 2034.793
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the three (1 | PurrbleID) models (first row) and the three (wk_num | PurrbleID) models (second row), in the same order as per the table above:
## [1] "PurrbleID" "purrble_completer" "ssi_complete"
## [4] "complete_both" "GAD7_1" "GAD7_2"
## [7] "GAD7_3" "GAD7_4" "GAD7_5"
## [10] "GAD7_6" "GAD7_7" "PHQ9_1"
## [13] "PHQ9_2" "PHQ9_3" "PHQ9_4"
## [16] "PHQ9_5" "PHQ9_6" "PHQ9_7"
## [19] "PHQ9_8" "PHQ9_9" "ERQ_SE_1"
## [22] "ERQ_SE_2" "ERQ_SE_3" "ERQ_SE_4"
## [25] "ERQ_SE_5" "ERQ_SE_6" "ERQ_SE_7"
## [28] "ERQ_SE_8" "ERQ_SE_9" "ERQ_SE_10"
## [31] "base_bhs_1" "base_bhs_2" "base_bhs_3"
## [34] "base_bhs_4" "base_shs_1" "base_shs_2"
## [37] "base_shs_3" "ERbeliefs_1" "ERbeliefs_2"
## [40] "ERbeliefs_3" "ERbeliefs_4" "TWEETS_1"
## [43] "TWEETS_2" "TWEETS_3" "TWEETS_4"
## [46] "TWEETS_5" "TWEETS_6" "TWEETS_7"
## [49] "TWEETS_8" "TWEETS_9" "condition"
## [52] "wk" "age" "gender"
## [55] "college" "student" "GAD_total"
## [58] "PHQ_total" "ERQ_total" "ERbeliefs_total"
## [61] "bhs_total" "shs_total" "TWEETS_total"
## [64] "wk_num" "GAD_baseline"SHS - State Hope Scale
SHS can include both baseline-GAD and gender as co-variates. None of the random slope models had isSingular errors, and model 3 appears to work best according to AIC. QQ plots are a bit off, but likely still fine.
Purrble is significant across any model, and increases as further correlates are added.
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## shs_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num 0.280** 0.274** 0.273** 0.278** 0.273** 0.272**
## (0.101) (0.101) (0.101) (0.104) (0.105) (0.104)
## t = 2.772 t = 2.718 t = 2.710 t = 2.664 t = 2.615 t = 2.609
## p = 0.006 p = 0.007 p = 0.007 p = 0.008 p = 0.009 p = 0.010
##
## conditionPurrble 1.731* 1.767* 1.977** 1.600* 1.715* 2.001**
## (0.755) (0.731) (0.732) (0.754) (0.730) (0.731)
## t = 2.293 t = 2.418 t = 2.702 t = 2.123 t = 2.348 t = 2.735
## p = 0.022 p = 0.016 p = 0.007 p = 0.034 p = 0.019 p = 0.007
##
## GAD_baseline -0.252** -0.256** -0.250** -0.257**
## (0.093) (0.093) (0.093) (0.093)
## t = -2.703 t = -2.767 t = -2.688 t = -2.779
## p = 0.007 p = 0.006 p = 0.008 p = 0.006
##
## genderMale -0.230 -0.242
## (0.909) (0.909)
## t = -0.253 t = -0.266
## p = 0.801 p = 0.791
##
## genderNon-binary / third gender -3.365* -3.388*
## (1.686) (1.686)
## t = -1.997 t = -2.009
## p = 0.046 p = 0.045
##
## Constant 10.341*** 13.655*** 13.828*** 10.406*** 13.661*** 13.836***
## (0.565) (1.344) (1.362) (0.561) (1.343) (1.364)
## t = 18.302 t = 10.163 t = 10.153 t = 18.560 t = 10.174 t = 10.143
## p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 286 286 286 286 286 286
## Log Likelihood -778.168 -776.075 -771.832 -777.841 -775.821 -771.589
## Akaike Inf. Crit. 1566.337 1564.149 1559.665 1569.682 1567.642 1563.178
## Bayesian Inf. Crit. 1584.617 1586.085 1588.913 1595.274 1596.890 1599.738
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the three (1 | PurrbleID) models (first row) and the three (wk_num | PurrbleID) models (second row), in the same order as per the table above:
BHS - Beck Hopelesness scale
BHS can include both baseline-GAD and gender as co-variates. None of the random slope models had isSingular errors, and model 3 appears to work best according to AIC. QQ plots are a bit off but probably still okay.
Purrble is not significant in any of the models
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## bhs_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num -0.192** -0.188** -0.189** -0.193** -0.189** -0.189**
## (0.063) (0.063) (0.063) (0.066) (0.066) (0.066)
## t = -3.020 t = -2.970 t = -2.973 t = -2.908 t = -2.847 t = -2.848
## p = 0.003 p = 0.003 p = 0.003 p = 0.004 p = 0.005 p = 0.005
##
## conditionPurrble -0.672 -0.703 -0.725 -0.791 -0.779 -0.803
## (0.517) (0.496) (0.499) (0.513) (0.494) (0.497)
## t = -1.301 t = -1.417 t = -1.453 t = -1.542 t = -1.577 t = -1.614
## p = 0.194 p = 0.157 p = 0.147 p = 0.124 p = 0.115 p = 0.107
##
## GAD_baseline 0.195** 0.203** 0.187** 0.196**
## (0.063) (0.063) (0.063) (0.063)
## t = 3.081 t = 3.218 t = 2.959 t = 3.102
## p = 0.003 p = 0.002 p = 0.004 p = 0.002
##
## genderMale 1.033 1.006
## (0.620) (0.618)
## t = 1.666 t = 1.628
## p = 0.096 p = 0.104
##
## genderNon-binary / third gender 0.250 0.349
## (1.149) (1.149)
## t = 0.217 t = 0.304
## p = 0.829 p = 0.762
##
## Constant 5.190*** 2.629** 2.307* 5.250*** 2.773** 2.446**
## (0.383) (0.910) (0.927) (0.395) (0.914) (0.930)
## t = 13.547 t = 2.891 t = 2.490 t = 13.288 t = 3.035 t = 2.629
## p = 0.000 p = 0.004 p = 0.013 p = 0.000 p = 0.003 p = 0.009
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 286 286 286 286 286 286
## Log Likelihood -655.064 -652.329 -649.451 -653.758 -651.435 -648.619
## Akaike Inf. Crit. 1320.129 1316.657 1314.902 1321.516 1318.870 1317.239
## Bayesian Inf. Crit. 1338.409 1338.593 1344.150 1347.108 1348.118 1353.799
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001QQ plots for the three (1 | PurrbleID) models (first row) and the three (wk_num | PurrbleID) models (second row), in the same order as per the table above:
Summary of selected models
For convenience, see each of the ‘best’ models selected for the outcome variables above in the two tables below (as stargazer for some reason refuses to add these into one):
An interesting set of observations is that:
those constructs most directly associated with emotion regulation (ER Beliefs & ERQ sel-efficacy) also seem to be most strongly associated with Purrble impacts.
we see a tendency for positive Purrble impacts across all of the constructs (suggesting that these could be significant for larger samples).
##
## =====================================================================
## Dependent variable:
## -------------------------------------
## GAD_total PHQ_total ERbeliefs_total
## (1) (2) (3)
## ---------------------------------------------------------------------
## wk_num -0.835*** -0.733*** 0.232**
## (0.109) (0.110) (0.081)
## t = -7.642 t = -6.663 t = 2.847
## p = 0.000 p = 0.000 p = 0.005
##
## conditionPurrble -0.752 -1.487 1.238*
## (0.753) (0.793) (0.524)
## t = -0.998 t = -1.874 t = 2.364
## p = 0.319 p = 0.061 p = 0.019
##
## GAD_baseline 0.710*** -0.252***
## (0.100) (0.066)
## t = 7.073 t = -3.800
## p = 0.000 p = 0.0002
##
## genderMale -0.482 2.322* -0.998
## (0.933) (0.986) (0.650)
## t = -0.517 t = 2.355 t = -1.535
## p = 0.606 p = 0.019 p = 0.125
##
## genderNon-binary / third gender -0.077 1.411 -1.734
## (1.736) (1.828) (1.210)
## t = -0.045 t = 0.772 t = -1.433
## p = 0.965 p = 0.441 p = 0.152
##
## Constant 13.489*** 4.543** 14.556***
## (0.597) (1.477) (0.985)
## t = 22.601 t = 3.076 t = 14.778
## p = 0.000 p = 0.003 p = 0.000
##
## ---------------------------------------------------------------------
## Observations 286 286 285
## Log Likelihood -788.648 -795.433 -669.518
## Akaike Inf. Crit. 1591.296 1606.866 1359.036
## Bayesian Inf. Crit. 1616.888 1636.113 1395.561
## =====================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001##
## ================================================================
## Dependent variable:
## --------------------------------
## ERQ_total shs_total bhs_total
## (1) (2) (3)
## ----------------------------------------------------------------
## wk_num 0.580** 0.273** -0.189**
## (0.223) (0.101) (0.063)
## t = 2.606 t = 2.710 t = -2.973
## p = 0.010 p = 0.007 p = 0.003
##
## conditionPurrble 4.388* 1.977** -0.725
## (1.802) (0.732) (0.499)
## t = 2.434 t = 2.702 t = -1.453
## p = 0.015 p = 0.007 p = 0.147
##
## GAD_baseline -0.690** -0.256** 0.203**
## (0.229) (0.093) (0.063)
## t = -3.014 t = -2.767 t = 3.218
## p = 0.003 p = 0.006 p = 0.002
##
## genderMale -1.228 -0.230 1.033
## (2.238) (0.909) (0.620)
## t = -0.549 t = -0.253 t = 1.666
## p = 0.584 p = 0.801 p = 0.096
##
## genderNon-binary / third gender -5.683 -3.365* 0.250
## (4.174) (1.686) (1.149)
## t = -1.361 t = -1.997 t = 0.217
## p = 0.174 p = 0.046 p = 0.829
##
## Constant 45.014*** 13.828*** 2.307*
## (3.393) (1.362) (0.927)
## t = 13.268 t = 10.153 t = 2.490
## p = 0.000 p = 0.000 p = 0.013
##
## ----------------------------------------------------------------
## Observations 286 286 286
## Log Likelihood -989.116 -771.832 -649.451
## Akaike Inf. Crit. 1998.233 1559.665 1314.902
## Bayesian Inf. Crit. 2034.793 1588.913 1344.150
## ================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001Estimates for d
Finally, we are also interested in seeing the effect sizes on all off the outcome variables. The numbers below are computed by the MOTE package, using t-stats for conditionPurrble from the selected models above; and keeping the full N of the sample (i.e., n = 49 for Purrble and n = 50 for waitlist) as we have had everyone fill out the wk0 pack, and thus included into the lmer calculations.
| outcome | d_s |
|---|---|
| gad | \(d_s\) = -0.20, 95% CI [-0.60, 0.19] |
| phq | \(d_s\) = -0.38, 95% CI [-0.77, 0.02] |
| erb | \(d_s\) = 0.48, 95% CI [0.07, 0.87] |
| erq | \(d_s\) = 0.49, 95% CI [0.09, 0.89] |
| shs | \(d_s\) = 0.55, 95% CI [0.14, 0.94] |
| bhs | \(d_s\) = -0.30, 95% CI [-0.69, 0.10] |
Weekly questionnaires – explorative
Shorter versions of the outcome measures—GAD2, PHQ2, SHS, BHS—were asked off participants weekly – either as embedded in the broader surveys (wk0, wk2, wk4), or as part of smaller questionnaires (wk1, wk3).
Plots for weekly measures
Just as a way of helping us eye-ball the data, the graphs below plot the data each of the measures outcome measures, with simple geom_smooth (method = “lm”) overlay. I’ve also included a violin_plots in lieu of raincloud_plots, as those are cumbersome to work with (but we can have them for any final publication).
Outcomes
Let’s also re-run the models as per before, but this time looking at the weekly outcomes.
GAD-2
There is no significant effect of Purrble on anxiety, although the difference is in the expected direction. Model 3 (wk|ID) is performing best according to AIC.
##
## ===========================================================================
## Dependent variable:
## -------------------------------------------
## GAD_total
## 1 | ID 1 | ID wk | ID wk | ID
## (1) (2) (3) (4)
## ---------------------------------------------------------------------------
## wk_num -0.284*** -0.284*** -0.286*** -0.286***
## (0.035) (0.035) (0.040) (0.040)
## t = -8.062 t = -8.060 t = -7.069 t = -7.067
## p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## conditionPurrble -0.245 -0.248 -0.222 -0.234
## (0.256) (0.261) (0.248) (0.253)
## t = -0.958 t = -0.949 t = -0.898 t = -0.928
## p = 0.339 p = 0.343 p = 0.370 p = 0.354
##
## genderMale -0.139 -0.130
## (0.324) (0.313)
## t = -0.429 t = -0.415
## p = 0.669 p = 0.679
##
## genderNon-binary / third gender 0.055 0.204
## (0.601) (0.579)
## t = 0.091 t = 0.353
## p = 0.928 p = 0.725
##
## Constant 4.202*** 4.228*** 4.192*** 4.213***
## (0.192) (0.205) (0.180) (0.192)
## t = 21.852 t = 20.602 t = 23.332 t = 21.905
## p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## ---------------------------------------------------------------------------
## isSingular? NO NO NO NO
## Observations 472 472 472 472
## Log Likelihood -800.503 -800.213 -794.730 -794.447
## Akaike Inf. Crit. 1611.005 1614.427 1603.461 1606.893
## Bayesian Inf. Crit. 1631.790 1643.526 1632.560 1644.306
## ===========================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001PHQ-2
We see similar result as above, with Purrblenot statistically significant in any of the models, although it approaches significance (p = 0.066) in the best fitting model (Model 3).
##
## ====================================================================================================
## Dependent variable:
## --------------------------------------------------------------------
## PHQ_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## ----------------------------------------------------------------------------------------------------
## wk_num -0.130*** -0.128*** -0.128*** -0.130*** -0.128*** -0.128***
## (0.034) (0.034) (0.034) (0.035) (0.035) (0.035)
## t = -3.803 t = -3.753 t = -3.743 t = -3.685 t = -3.622 t = -3.618
## p = 0.0002 p = 0.0002 p = 0.0002 p = 0.0003 p = 0.0003 p = 0.0003
##
## conditionPurrble -0.384 -0.417 -0.476 -0.411 -0.378 -0.454
## (0.297) (0.264) (0.258) (0.296) (0.263) (0.258)
## t = -1.296 t = -1.581 t = -1.841 t = -1.386 t = -1.439 t = -1.759
## p = 0.196 p = 0.114 p = 0.066 p = 0.166 p = 0.151 p = 0.079
##
## GAD_baseline 0.175*** 0.182*** 0.182*** 0.186***
## (0.034) (0.033) (0.033) (0.033)
## t = 5.217 t = 5.574 t = 5.456 t = 5.682
## p = 0.00000 p = 0.00000 p = 0.00000 p = 0.000
##
## genderMale 0.814* 0.801*
## (0.322) (0.321)
## t = 2.531 t = 2.494
## p = 0.012 p = 0.013
##
## genderNon-binary / third gender 0.867 0.814
## (0.595) (0.594)
## t = 1.457 t = 1.371
## p = 0.146 p = 0.171
##
## Constant 3.107*** 0.803 0.530 3.121*** 0.689 0.482
## (0.218) (0.484) (0.480) (0.221) (0.480) (0.479)
## t = 14.222 t = 1.660 t = 1.105 t = 14.096 t = 1.437 t = 1.006
## p = 0.000 p = 0.097 p = 0.270 p = 0.000 p = 0.151 p = 0.315
##
## ----------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 472 472 472 472 472 472
## Log Likelihood -802.207 -792.521 -788.556 -801.926 -792.000 -788.308
## Akaike Inf. Crit. 1614.414 1597.042 1593.112 1617.852 1600.000 1596.615
## Bayesian Inf. Crit. 1635.199 1621.984 1626.368 1646.951 1633.255 1638.185
## ====================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001SHS - State Hope Scale
As before, Purrble is significant across any model, and increases as further correlates are added.
##
## ====================================================================================================
## Dependent variable:
## --------------------------------------------------------------------
## shs_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## ----------------------------------------------------------------------------------------------------
## wk_num 0.345*** 0.342*** 0.341*** 0.342*** 0.340*** 0.339***
## (0.082) (0.082) (0.082) (0.094) (0.094) (0.094)
## t = 4.212 t = 4.176 t = 4.168 t = 3.645 t = 3.618 t = 3.611
## p = 0.00003 p = 0.00003 p = 0.00004 p = 0.0003 p = 0.0003 p = 0.0004
##
## conditionPurrble 1.813* 1.856** 2.026** 1.535* 1.649* 1.849**
## (0.716) (0.690) (0.695) (0.710) (0.687) (0.692)
## t = 2.532 t = 2.688 t = 2.917 t = 2.161 t = 2.400 t = 2.671
## p = 0.012 p = 0.008 p = 0.004 p = 0.031 p = 0.017 p = 0.008
##
## GAD_baseline -0.251** -0.256** -0.242** -0.247**
## (0.088) (0.088) (0.087) (0.087)
## t = -2.860 t = -2.910 t = -2.770 t = -2.829
## p = 0.005 p = 0.004 p = 0.006 p = 0.005
##
## genderMale -0.252 -0.174
## (0.864) (0.861)
## t = -0.292 t = -0.202
## p = 0.771 p = 0.840
##
## genderNon-binary / third gender -2.719 -2.609
## (1.598) (1.591)
## t = -1.701 t = -1.640
## p = 0.089 p = 0.101
##
## Constant 10.193*** 13.497*** 13.658*** 10.334*** 13.481*** 13.620***
## (0.527) (1.263) (1.287) (0.518) (1.255) (1.281)
## t = 19.336 t = 10.689 t = 10.614 t = 19.964 t = 10.743 t = 10.630
## p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000 p = 0.000
##
## ----------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 472 472 472 472 472 472
## Log Likelihood -1213.689 -1211.250 -1207.648 -1210.285 -1208.129 -1204.653
## Akaike Inf. Crit. 2437.379 2434.501 2431.296 2434.571 2432.258 2429.306
## Bayesian Inf. Crit. 2458.164 2459.443 2464.552 2463.669 2465.514 2470.876
## ====================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001BHS - Beck Hopelesness scale
Model 3 appears to work best according to AIC, with results akin to the main analyses – Purrble is not significant in any of the models, although the differences are in the right direction. Model 3 works best according to AIC.
##
## =================================================================================================
## Dependent variable:
## -----------------------------------------------------------------
## bhs_total
## 1 | ID 1 | ID 1 | ID wk | ID wk | ID wk | ID
## (1) (2) (3) (4) (5) (6)
## -------------------------------------------------------------------------------------------------
## wk_num -0.199*** -0.197*** -0.197*** -0.201** -0.198** -0.197**
## (0.058) (0.058) (0.058) (0.064) (0.064) (0.064)
## t = -3.450 t = -3.414 t = -3.411 t = -3.142 t = -3.102 t = -3.094
## p = 0.001 p = 0.001 p = 0.001 p = 0.002 p = 0.002 p = 0.002
##
## conditionPurrble -0.624 -0.655 -0.701 -0.704 -0.713 -0.759
## (0.475) (0.456) (0.458) (0.472) (0.454) (0.456)
## t = -1.314 t = -1.436 t = -1.531 t = -1.490 t = -1.569 t = -1.663
## p = 0.189 p = 0.151 p = 0.126 p = 0.137 p = 0.117 p = 0.097
##
## GAD_baseline 0.179** 0.187** 0.174** 0.183**
## (0.058) (0.058) (0.058) (0.058)
## t = 3.086 t = 3.237 t = 3.005 t = 3.158
## p = 0.003 p = 0.002 p = 0.003 p = 0.002
##
## genderMale 0.987 0.957
## (0.569) (0.568)
## t = 1.733 t = 1.687
## p = 0.084 p = 0.092
##
## genderNon-binary / third gender 0.660 0.738
## (1.053) (1.054)
## t = 0.627 t = 0.700
## p = 0.531 p = 0.484
##
## Constant 5.247*** 2.892*** 2.572** 5.288*** 2.987*** 2.667**
## (0.352) (0.835) (0.849) (0.364) (0.839) (0.854)
## t = 14.923 t = 3.463 t = 3.028 t = 14.511 t = 3.559 t = 3.124
## p = 0.000 p = 0.001 p = 0.003 p = 0.000 p = 0.0004 p = 0.002
##
## -------------------------------------------------------------------------------------------------
## isSingular? NO NO NO NO NO NO
## Observations 472 472 472 472 472 472
## Log Likelihood -1044.356 -1041.692 -1038.784 -1042.224 -1039.818 -1036.958
## Akaike Inf. Crit. 2098.712 2095.384 2093.569 2098.448 2095.636 2093.916
## Bayesian Inf. Crit. 2119.497 2120.326 2126.825 2127.547 2128.892 2135.485
## =================================================================================================
## Note: *p<0.05; **p<0.01; ***p<0.001