1 Preparing for analysis

Required Packages:

library(tidyverse)
library(corrplot)
library(BayesFM)
library(blavaan)
library(brms)

Initial steps: 1. Read the data from our behavioral stability dataset. 2. Remove columns with session information. 3. Transform integer variables into numeric variables (which is required by the BayesFM package).

2 Determining the Number of Factors for Factor Analysis

Because we have a large number of variables to draw from, it is necessary to first reduce our variables into factors that capture commonalities between the variables. If we can reasonably assume that our variables could be split into a maximum of five factors, they could be split as follows:

path <- '
f1 =~ n_major_block + n_jumps + n_edges + n_sustained_reading 
f2 =~ prop_edits_before_sentence + prop_edits_before_word + prop_edits_within_word + len_prodseq
f3 =~ prop_lookbacks_before_sentence + prop_lookbacks_before_word + prop_lookbacks_within_word + len_nonlookbacks
f4 =~ lookback_duration + prob_long_lookback_duration 
f5 =~ prob_long_event_duration_before_sentence + prob_long_event_duration_before_word + prob_long_event_duration_within_word
'

We iteratively combined factors based on similarities between the variables, creating 5 possible paths with 1, 2, 3, 4, and 5 factors each.

A factor containing all possible variables would look like this:

path <- '
f1 =~ n_major_block + n_jumps + n_edges + n_sustained_reading + prop_edits_before_sentence + prop_edits_before_word + prop_edits_within_word + len_prodseq + prop_lookbacks_before_sentence + prop_lookbacks_before_word + prop_lookbacks_within_word + len_nonlookbacks + lookback_duration + prob_long_lookback_duration + prob_long_event_duration_before_sentence + prob_long_event_duration_before_word + prob_long_event_duration_within_word
'

We then ran Bayesian Confirmatory Factor Analysis models for all possible paths using weakly informative priors. Models were fitted with 5,000 iterations (2,500 warm-up).

mydp <- dpriors(lambda="normal(0,1)", nu="normal(0,1)", alpha="normal(0,1)", beta="normal(0,10)", tau="normal(0,1.5)", rho="beta(2,2)", theta="gamma(1,2)", psi="gamma(1,2)", ibpsi="wishart(3, iden)")
cfamodel <- bcfa(path, dp = mydp, data=df, n.chains=3, bcontrol = list(cores = 3), burnin=2500, sample=2500)

We then used blavCompare to compare the Bayesian confirmatory factor analysis models.

For example:

bc1 <- blavCompareMod(cfamodel, cfamodel2) %>% mutate(across(model, ~recode(., model1 = "cfamodel",
                           model2 = "cfamodel2")),
     comp = c(NA, unique(str_c(model, collapse = " vs "))))

[repeated for bc2, bc3, bc4]

modelcomps <- bind_rows(bc4, bc3, bc2, bc1) %>% 
  filter(!is.na(elpd_diff) | model == model[1]); modelcomps

cfamodel4 with 4 factors was the most informative.


WAIC estimates: 
 object1:  3973.987 
 object2:  3712.066 

WAIC difference & SE: 
 -130.961   26.746 

LOO estimates: 
 object1:  3974.773 
 object2:  3712.697 

LOO difference & SE: 
 -131.038   26.683 

Laplace approximation to the log-Bayes factor
(experimental; positive values favor object1): -123.351 

WAIC estimates: 
 object1:  3712.066 
 object2:  3707.43 

WAIC difference & SE: 
   -2.318    2.616 

LOO estimates: 
 object1:  3712.697 
 object2:  3707.592 

LOO difference & SE: 
   -2.553    2.620 

Laplace approximation to the log-Bayes factor
(experimental; positive values favor object1):   -1.299 

WAIC estimates: 
 object1:  3707.43 
 object2:  3541.617 

WAIC difference & SE: 
  -82.906   16.258 

LOO estimates: 
 object1:  3707.592 
 object2:  3541.486 

LOO difference & SE: 
  -83.053   16.287 

Laplace approximation to the log-Bayes factor
(experimental; positive values favor object1):  -76.113 

WAIC estimates: 
 object1:  3541.617 
 object2:  3537.663 

WAIC difference & SE: 
   -1.977    3.833 

LOO estimates: 
 object1:  3541.486 
 object2:  3537.658 

LOO difference & SE: 
   -1.914    3.826 

Laplace approximation to the log-Bayes factor
(experimental; positive values favor object1):    1.272 
# A tibble: 5 × 7
  model       elpd se_elpd elpd_diff elpd_diff_se elpd_diff_ratio comp          
  <chr>      <dbl>   <dbl>     <dbl>        <dbl>           <dbl> <chr>         
1 cfamodel5 -1769.    40.5     NA           NA             NA     <NA>          
2 cfamodel4 -1771.    40.5     -1.91         3.83           0.500 cfamodel5 vs …
3 cfamodel3 -1854.    43.3    -83.1         16.3            5.10  cfamodel4 vs …
4 cfamodel2 -1856.    43.2     -2.55         2.62           0.974 cfamodel3 vs …
5 cfamodel  -1987.    43.9   -131.          26.7            4.91  cfamodel2 vs …

3 Final Selected Factors

Factor 1: Non-linearity

Factor 2: Revisions

Factor 3: Lookbacks

Factor 4: Pauses

4 Performing Cluster Analysis

Using the most informative model with 4 factors, cluster analysis using hclust was run to group sessions into clusters on the basis of their behavior in terms of these factors. We looked at several possible solutions for cluster models and narrowed it down to 4 reasonable options for models with 3+ clusters, which were variations of “complete” and “Ward D” methods for hclust and “Euclidean” and “Maximum” methods for dist(). The solution with the most evenly distributed clusters—and the clusters which made sense upon initial qualitative observations—was the following:

Df <- read_csv("data/by_ppt_weighted_factors.csv") %>% 
  select(f1:f4)

res.dist <- dist(Df, method = "euclidean")
hc1 <- hclust(res.dist, method = "complete")

plot(hc1, cex = 0.6, hang = -1, labels = Df$participant, ylab="", yaxt="n", xlab="", xaxt="n")

# selects and indicates boundaries for 3 clusters
rect.hclust(hc1, k = 3, border = 2:5)

Other possible cluster solutions:

# Option 2 -- weaker (selected similar participants not grouped together because of one extra large group)
res.dist <- dist(Df, method = "maximum")
hc2 <- hclust(res.dist, method = "complete")

# Option 3 -- similar to Option 1
res.dist <- dist(Df, method = "euclidian")
hc3 <- hclust(res.dist, method = "ward.D")

# Option 4 -- similar to 1
res.dist <- dist(Df, method = "maximum")
hc4 <- hclust(res.dist, method = "ward.D")

Each participant completed 3 writing sessions, and each session was assigned to a cluster on the basis of the process behavior.

5 Factor Averages per Cluster

# Get data
data <- read_csv("data/data_wide.csv") %>% 
  mutate(condition = str_c("c", cluster, "_", factor)) %>% 
  select(participant:value, condition)

# Define the model formula
formula <- bf(value ~ 0 + condition + (prompt + session_no|participant), family = gaussian())

# Define the priors 
get_prior(formula, data)

# The priors are regulating (weakly informative):
priors <- set_prior("student_t(7, 0, .5)", class = 'sd') +
  set_prior("lkj(2)", class = 'L') + 
  # Weakly informative priors for all slope coefficients
  set_prior("student_t(7, 0, 2.5)", class = 'b')

# Fit the Bayesian model using the brm function
fit <- brm(
  formula = formula, 
  data = data,
  sample_prior = 'yes',
  seed = 123,          # Set a seed for reproducibility
  prior = priors,
  refresh = 2000,      # Show update after 2000 iterations
  chains = 4,          # The number of MCMC chains
  cores = 4,           # Use multiple cores to parallelise chains (one per chain)
  iter = 10000,         # The number of iterations in each chain
  warmup = 5000,       # The number of warmup iterations per chain
  control = list(max_treedepth = 16, # improve sampling
                 adapt_delta = 0.99))
# A tibble: 4 × 4
  factor c1                   c2                 c3                  
  <chr>  <chr>                <chr>              <chr>               
1 f1     -0.17 [-1.03, 0.68]  2.84 [1.77, 3.92]  -3.6 [-4.89, -2.31] 
2 f2     -1.68 [-2.54, -0.84] 4.14 [3.08, 5.2]   -2.07 [-3.33, -0.77]
3 f3     1.66 [0.8, 2.52]     1.28 [0.25, 2.33]  -5.42 [-6.73, -4.1] 
4 f4     0.08 [-0.76, 0.93]   0.32 [-0.72, 1.36] -0.46 [-1.73, 0.83] 

6 Cluster Changes by Participant

How many clusters did participants’ sessions appear in?

read_csv("data/by_ppt_weighted_factors.csv") %>% 
  mutate(cluster = cutree(hc1, 3)) %>% 
  group_by(participant) %>%
  mutate(switch_nclust = length(unique(cluster))) %>%
  ungroup() %>%
  select(participant, switch_nclust) %>%
  unique() %>% 
  count(switch_nclust) %>% 
  select(No_Clusters_per_Participant = switch_nclust,
         n)
# A tibble: 2 × 2
  No_Clusters_per_Participant     n
                        <int> <int>
1                           1    16
2                           2    14
Cluster changes by participant

Figure 6.1: Cluster changes by participant

When a participant switched clusters, were they already further from the cluster centroid than those who did not switch?

switching distance_from M N SD
no switch distance_from_cent 4.96 43 1.97
no switch distance_from_cl1 7.32 43 3.35
no switch distance_from_cl2 8.25 43 4.44
no switch distance_from_cl3 10.46 43 3.37
switched to 1 distance_from_cent 5.00 8 2.02
switched to 1 distance_from_cl1 6.84 8 3.28
switched to 1 distance_from_cl2 7.35 8 4.55
switched to 1 distance_from_cl3 10.03 8 4.20
switched to 2 distance_from_cent 5.70 4 1.81
switched to 2 distance_from_cl1 5.70 4 1.81
switched to 2 distance_from_cl2 7.47 4 3.93
switched to 2 distance_from_cl3 9.50 4 2.17
switched to 3 distance_from_cent 5.33 5 1.51
switched to 3 distance_from_cl1 5.95 5 1.87
switched to 3 distance_from_cl2 7.01 5 2.77
switched to 3 distance_from_cl3 9.08 5 1.29

7 Cluster and Text Quality

We analyzed the extent to which the emergent patterns of behavior could predict text quality. For this analysis, we used Bayesian multivariate mixed-effects models to estimate whether measures of text quality were dependent on cluster assignment.

# Define the model formula
# Random intercepts for participants, prompts, and session_no (test occasion)
# Fixed effect for cluster
# model is multivariate with corrections of random effect across outcome variable
formula <- bf(rating ~ cluster + (prompt + session_no|participant), family = cumulative()) +
           bf(oc_ratio ~ cluster + (prompt + session_no|participant), family = gaussian()) +
           bf(lexd ~ cluster + (prompt + session_no|participant), family = gaussian()) +
           bf(M_sent_len ~ cluster + (prompt + session_no|participant), family = gaussian()) +
           bf(M_word_len ~ cluster + (prompt + session_no|participant), family = gaussian()) +
           bf(sentence_count ~ cluster + (prompt + session_no|participant), family = poisson()) +
           bf(word_count ~ cluster + (prompt + session_no|participant), family = poisson()) 
  

# Define the priors 
get_prior(formula, data)

prior_sd <- set_prior("student_t(7, 0, .5)", class = 'sd', 
                  resp = c("rating", "ocratio", "lexd",
                           "Msentlen", "Mwordlen", "sentencecount",
                           "wordcount"))

prior_cor <- set_prior("lkj(2)", class = 'L')

# Weakly informative priors for all slope coefficients
prior_rating <- c(prior(student_t(7, 0, .5), class = 'b', resp = "rating"),
                  prior(student_t(7, 0, 2.5), class = 'Intercept', resp = "rating"))
                  
prior_ocratio <- c(prior(student_t(7, 0, .1), class = 'b', resp = "ocratio"),
                   prior(student_t(7, 0.9, .25), class = 'Intercept', resp = "ocratio"))

prior_lexd <- c(prior(student_t(7, 0, 2.5), class = 'b', resp = "lexd"),
                prior(student_t(7, 80, 10), class = 'Intercept', resp = "lexd"))

prior_msentlen <- c(prior(student_t(7, 0, .5), class = 'b', resp = "Msentlen"),
                    prior(student_t(7, 21, 2), class = 'Intercept', resp = "Msentlen"))

prior_mwordlen <- c(prior(student_t(7, 0, .1), class = 'b', resp = "Mwordlen"),
                   prior(student_t(7, 5, .5), class = 'Intercept', resp = "Mwordlen"))

prior_sentcount <- c(prior(student_t(7, 0, .25), class = 'b', resp = "sentencecount"),
                     prior(student_t(7, 3.5, .25), class = 'Intercept', resp = "sentencecount"))

prior_wordcount <- c(prior(student_t(7, 0, .1), class = 'b', resp = "wordcount"),
                     prior(student_t(7, 6.5, .5), class = 'Intercept', resp = "wordcount"))

# Combine the priors
priors <- c(prior_rating, prior_ocratio, prior_lexd, prior_msentlen, 
            prior_mwordlen, prior_sentcount, prior_wordcount, prior_sd, prior_cor)

# Fit the Bayesian model using the brm function
fit <- brm(
  formula = formula, 
  data = data,
  sample_prior = 'yes',
  seed = 123,          # Set a seed for reproducibility
  prior = priors,
  refresh = 2000,      # Show update after 2000 iterations
  chains = 4,          # The number of MCMC chains
  cores = 4,           # Use multiple cores to parallelise chains (one per chain)
  iter = 10000,         # The number of iterations in each chain
  warmup = 5000,       # The number of warmup iterations per chain
  control = list(max_treedepth = 16, # improve sampling
                 adapt_delta = 0.99))

7.1 Posterior cellmeans by cluster and text quality measure

Table 7.1: Posterior cellmeans by outcome variable and cluster (multivariate model).
Cluster Text quality rating Open-to-closed class words Lexical diversity Mean sentence length Mean word length No. of sentences No. of words
1 4.16 [3.61, 4.73] 0.9 [0.83, 0.97] 82.92 [78.72, 86.96] 20.99 [19.94, 22.02] 4.73 [4.64, 4.81] 28.02 [25.59, 30.61] 578.13 [525.64, 635.07]
2 4.22 [3.61, 4.85] 0.91 [0.82, 0.99] 81.95 [77.04, 86.77] 21.13 [19.97, 22.29] 4.7 [4.59, 4.8] 28.73 [25.62, 31.97] 587.61 [528.62, 652.76]
3 4.05 [3.4, 4.74] 0.88 [0.79, 0.98] 83.28 [77.85, 88.64] 21.3 [20.05, 22.59] 4.73 [4.63, 4.84] 25.48 [22.38, 29] 572.87 [513.47, 641.22]
Note:
Estimates are shown with upper and lower bounds of 95% PIs.

7.2 Model comparison

For model comparisons we used out-of-sample predictions estimated using Pareto smoothed importance-sampling leave-one-out cross-validation (Vehtari et al., 2015; 2016). Predictive performance was estimated as the sum of the expected log predictive density (\(\widehat{elpd}\)) and the difference \(\Delta\widehat{elpd}\) between models. The advantage of using leave-one-out cross-validation is that models with more parameters are penalised to prevent overfit.

The model described above was compared to a model without the fixed effect of cluster. The results are shown in Table 7.1. The difference \(\Delta\widehat{elpd}\) indicates that adding cluster to the multivariate model does not increase its predictive performance.

Table 7.2: Model comparisons for effect of cluster. Predictive performance was indicated as expected log pointwise predictive density (\(\widehat{elpd}\)). The top row shows the model with the highest predictive performance, i.e. the highest \(\widehat{elpd}\); the differences \(\Delta\widehat{elpd}\) are relative to the model with the highest predictive performance.
Model \(\Delta\widehat{elpd}\) \(\widehat{elpd}\)
Baseline – -1431 (18)
Cluster -6 (3) -1437 (18)
Note:
Standard errors are shown in parentheses.
## Model convergence

### Traceplots
mcmc_plot(fit, pars = "^b_", type = "trace")

mcmc_plot(fit, pars = "^sd_", type = "trace")

8 References

Vehtari, A., Simpson, D., Gelman, A., Yao, Y., & Gabry, J. (2015). Pareto Smoothed Importance Sampling. arXiv [stat.CO]. arXiv. https://arxiv.org/abs/1507.02646

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. In arXiv [stat.CO]. arXiv. http://arxiv.org/abs/1507.04544

9 Session info

R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.3.1

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Chicago
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] ggthemes_4.2.4        magrittr_2.0.3        kableExtra_1.3.4.9000
 [4] brms_2.19.0           blavaan_0.5-2.1200    Rcpp_1.0.11          
 [7] BayesFM_0.1.5         corrplot_0.92         lubridate_1.9.2      
[10] forcats_1.0.0         stringr_1.5.0         dplyr_1.1.2          
[13] purrr_1.0.1           readr_2.1.4           tidyr_1.3.0          
[16] tibble_3.2.1          ggplot2_3.4.2         tidyverse_2.0.0      

loaded via a namespace (and not attached):
  [1] tensorA_0.36.2       rstudioapi_0.14      jsonlite_1.8.5      
  [4] estimability_1.4.1   farver_2.1.1         rmarkdown_2.25      
  [7] vctrs_0.6.3          CompQuadForm_1.4.3   base64enc_0.1-3     
 [10] webshot_0.5.5        htmltools_0.5.5      distributional_0.3.2
 [13] sass_0.4.6           parallelly_1.36.0    StanHeaders_2.26.28 
 [16] bslib_0.5.0          htmlwidgets_1.6.2    plyr_1.8.8          
 [19] sandwich_3.0-2       emmeans_1.8.7        zoo_1.8-12          
 [22] cachem_1.0.8         igraph_1.5.0.1       mime_0.12           
 [25] lifecycle_1.0.3      pkgconfig_2.0.3      colourpicker_1.2.0  
 [28] Matrix_1.6-0         R6_2.5.1             fastmap_1.1.1       
 [31] future_1.33.0        shiny_1.7.4.1        digest_0.6.31       
 [34] colorspace_2.1-0     ps_1.7.5             crosstalk_1.2.0     
 [37] labeling_0.4.2       fansi_1.0.4          timechange_0.2.0    
 [40] httr_1.4.6           abind_1.4-5          compiler_4.3.1      
 [43] bit64_4.0.5          withr_2.5.0          backports_1.4.1     
 [46] inline_0.3.19        shinystan_2.6.0      highr_0.10          
 [49] QuickJSR_1.0.8       pkgbuild_1.4.2       gtools_3.9.4        
 [52] loo_2.6.0            tools_4.3.1          pbivnorm_0.6.0      
 [55] httpuv_1.6.11        future.apply_1.11.0  threejs_0.3.3       
 [58] glue_1.6.2           quadprog_1.5-8       callr_3.7.3         
 [61] nlme_3.1-162         promises_1.2.0.1     grid_4.3.1          
 [64] checkmate_2.2.0      reshape2_1.4.4       generics_0.1.3      
 [67] gtable_0.3.3         tzdb_0.4.0           hms_1.1.3           
 [70] xml2_1.3.4           utf8_1.2.3           pillar_1.9.0        
 [73] markdown_1.7         vroom_1.6.3          posterior_1.4.1     
 [76] later_1.3.1          lattice_0.21-8       bit_4.0.5           
 [79] tidyselect_1.2.0     miniUI_0.1.1.1       knitr_1.43          
 [82] gridExtra_2.3        bookdown_0.36        svglite_2.1.2       
 [85] stats4_4.3.1         xfun_0.39            bridgesampling_1.1-2
 [88] matrixStats_1.0.0    DT_0.28              rstan_2.32.3        
 [91] stringi_1.7.12       yaml_2.3.7           evaluate_0.21       
 [94] codetools_0.2-19     nonnest2_0.5-5       cli_3.6.1           
 [97] RcppParallel_5.1.7   shinythemes_1.2.0    xtable_1.8-4        
[100] systemfonts_1.0.4    munsell_0.5.0        processx_3.8.2      
[103] jquerylib_0.1.4      lavaan_0.6-16        globals_0.16.2      
[106] coda_0.19-4          parallel_4.3.1       rstantools_2.3.1.1  
[109] ellipsis_0.3.2       prettyunits_1.1.1    dygraphs_1.1.1.6    
[112] bayesplot_1.10.0     Brobdingnag_1.2-9    listenv_0.9.0       
[115] viridisLite_0.4.2    mvtnorm_1.2-2        scales_1.2.1        
[118] xts_0.13.1           crayon_1.5.2         tmvnsim_1.0-2       
[121] rlang_1.1.1          rvest_1.0.3          mnormt_2.1.1        
[124] shinyjs_2.1.0