In this context, the overarching question we aim to address is: Can subtypes based separately on resting state functional connectivity, structural imaging, and diffusion tensor imaging measures improve our understanding of the underlying neural mechanisms that contribute to typical and atypical development in cognitive functioning and psychopathology across development?
Answering this question is of paramount importance for several reasons. First, subtyping allows researchers to account for the vast heterogeneity in brain structure and function, often observed in typical and atypical development (Insel & Cuthbert, 2015)[1]. By identifying distinct subgroups of individuals with similar neurobiological profiles, we can better understand the complex interplay between different neural mechanisms and their influence on cognitive functioning and psychopathology. This, in turn, can help refine our models of brain development and enable more targeted interventions for cognitive and psychiatric disorders.
Second, the use of subtyping based on multiple neuroimaging modalities, such as resting state functional connectivity (RSFC), structural magnetic resonance imaging (sMRI), and diffusion tensor imaging (DTI), provides a comprehensive and integrative view of the brain’s organization and connectivity (Fox & Greicius, 2010)[2]. By examining subtypes derived from different imaging measures, researchers can elucidate the unique and shared contributions of functional and structural brain networks to cognitive functioning and psychopathology. This multimodal approach can help overcome the limitations of studying individual imaging measures in isolation and ultimately lead to a more holistic understanding of the brain’s role in development. Third, the identification of neuroimaging subtypes can inform early identification and intervention efforts for cognitive and psychiatric disorders. By revealing distinct patterns of brain organization and connectivity that are associated with different developmental trajectories, subtyping can help identify at-risk individuals and inform the development of targeted interventions that address the specific neurobiological underpinnings of cognitive and psychiatric disorders (Casey et al., 2014)[3]. This is particularly important during critical periods of development, when early intervention can have lasting effects on cognitive and mental health outcomes (Johnson et al., 2016)[4].
In summary, addressing the question of whether subtypes based separately on RSFC, sMRI, and DTI measures can improve our understanding of the underlying neural mechanisms that contribute to typical and atypical development in cognitive functioning and psychopathology is of critical importance. By leveraging the power of subtyping and multimodal neuroimaging approaches, we can advance our understanding of the complex neural mechanisms that drive development, ultimately leading to more effective interventions and improved outcomes for those affected by cognitive and psychiatric disorders.
Data-driven clustering has led to significant advancements in identifying patterns within complex datasets. However, the absence of a standardized approach results in inconsistencies regarding the reliability, stability, and reproducibility of the derived clusters. We propose a comprehensive framework for data-driven clustering, encompassing Identification, Validation, Evaluation, Prediction, and Replication (IVEPR), to ensure that subtyping provides meaningful information beyond traditional analyses. This framework aims to improve the quality of clustering research, serving as a guide for future studies and fostering meaningful advancements in the field. Introduction: Data-driven clustering techniques have transformed the way researchers identify and analyze patterns in complex datasets. To address the lack of standardization and determine if subtyping reveals valuable information beyond traditional methods, we propose the IVEPR framework. This comprehensive framework is based on previous work and recent studies employing standardized analysis procedures in data-driven clustering, consisting of the following key components: 1. Identification: The first step involves identifying meaningful clusters within the dataset using appropriate clustering algorithms, feature extraction techniques, and parameter settings. Establishing a standardized procedure ensures that the derived clusters are biologically or theoretically relevant and comparable across studies. 2. Validation: This step assesses the robustness and reliability of the identified clusters using cross-validation, bootstrapping, or other resampling techniques. Validation ensures the stability of clusters across different data subsets, minimizing the impact of noise or artifacts. 3. Evaluation: The framework emphasizes objective evaluation metrics to assess the quality of the derived clusters and determine if subtyping provides meaningful information beyond traditional analyses. This step includes comparing differences across clusters regarding measures of interest using statistical techniques such as ANCOVA. Standardized evaluation metrics facilitate the comparison of different clustering algorithms and parameter settings. 4. Prediction: Developing predictive models to forecast the outcomes or behaviors of the derived clusters demonstrates the practical utility and generalizability of the clustering solutions. Accurate predictions indicate that the subtyping process reveals valuable information that might not be obtainable through traditional approaches. 5. Replication: Reproducibility is vital in scientific research. The IVEPR framework underscores the importance of replication studies to confirm the robustness and generalizability of the identified clusters. A standardized procedure minimizes the likelihood of false discoveries and enhances the credibility of the findings.
Conclusion: The IVEPR framework offers a systematic approach to standardize data-driven clustering techniques, ensuring the reliability, stability, and reproducibility of the derived clusters. By adopting this framework, researchers can establish whether subtyping provides meaningful information beyond traditional analyses, contributing to meaningful advancements in clustering research and paving the way for future studies built on a solid foundation.
Resting-State Functional Connectivity (RSFC): RSFC has emerged as a robust and reliable tool for investigating the brain’s functional organization. Numerous studies have demonstrated its stability over time, making it an ideal candidate for longitudinal research designs (Van Dijk et al., 2010; Shehzad et al., 2009). Furthermore, RSFC is not affected by task performance or state changes, unlike task-based fMRI, which can be influenced by various factors such as fatigue, motivation, and attention (Biswal et al., 1995). The relatively long acquisition time of resting-state data allows for a higher signal-to-noise ratio, leading to increased statistical power in detecting functional networks (Fox and Raichle, 2007). These factors contribute to the growing consensus among neuroimagers that RSFC is the most stable and reliable form of functional neuroimaging data available (Damoiseaux et al., 2006). To fully leverage RSFC’s power, it is critical to use robust and reliable methods for data analysis and interpretation, such as subtyping approaches. These approaches can help address the underlying heterogeneity of the population and achieve more accurate and reliable predictions.
Structural Imaging: Structural imaging techniques, like MRI, provide detailed information about brain anatomy, including gray and white matter volumes, cortical thickness, and surface area. These measures have been shown to be sensitive to individual differences in cognitive functioning and psychopathology (Kanai & Rees, 2011). Structural imaging data are relatively stable and less susceptible to state-dependent fluctuations than functional data, making them suitable for longitudinal studies and subtyping approaches. Moreover, structural imaging can identify neuroanatomical markers that distinguish different subtypes, which can further inform the understanding of the biological basis of cognitive and psychiatric disorders. By utilizing subtyping approaches on structural imaging data, researchers can better address the heterogeneity within populations and improve the accuracy of predictions and diagnostic precision.
Diffusion Tensor Imaging (DTI): DTI measures the diffusion of water molecules along axonal fibers, enabling the visualization and quantification of white matter tracts in the brain. DTI provides valuable information about the brain’s structural connectivity, which plays a crucial role in the efficient transfer of information between functionally specialized brain regions (Le Bihan, 2003). The reliability and stability of DTI measures make them suitable for longitudinal and cross-sectional investigations of brain connectivity (Basser & Jones, 2002). By applying subtyping approaches to DTI data, researchers can capture the heterogeneity in white matter connectivity patterns, which may underlie differences in cognitive functioning and psychopathology. Moreover, DTI-based subtypes can inform the understanding of the structural connectivity’s role in the emergence, progression, and treatment of cognitive and psychiatric disorders.
In conclusion, resting-state functional connectivity, structural imaging, and diffusion tensor imaging provide valid and reliable measures for deriving neuroimaging subtypes. Utilizing these measures allows researchers to address the underlying heterogeneity in the population, leading to more accurate predictions and a better understanding of cognitive functioning and psychopathology in children and adolescents.
I. Data-driven Subtyping via Bootstrap Enhanced Leiden Community Detection The first step in the analysis involves data-driven subtyping using a reproducibility-motivated method called bootstrap enhanced Leiden Community Detection (Bassett et al., 2015)[3]. This method is used to identify distinct subgroups of individuals with similar functional connectivity or structural characteristics, based on resting-state functional connectivity (RSFC), diffusion tensor imaging (DTI), and structural magnetic resonance imaging (sMRI) data.
Bootstrap enhanced Leiden Community Detection offers several advantages over traditional clustering methods, such as hierarchical clustering and k-means clustering. First, it accounts for the uncertainty in the estimated connectivity matrices by employing a bootstrapping procedure, which enhances the stability and reproducibility of the identified subtypes (Bassett et al., 2015)[3]. Second, it allows for a more accurate representation of the underlying modular structure of the brain, as it detects communities of brain regions with similar connectivity patterns in a network-based manner (Newman, 2006)[4]. Third, it is a data-driven approach, meaning that it does not rely on a priori assumptions about the number or structure of subtypes, making it less prone to biases and more suitable for exploratory analyses.
Subtype Classification and Feature Importance using Gradient Boosted Decision Trees and SHAPLEY Additive Explanations Once the subtypes have been identified, the next step is to evaluate their stratification using XGboost Gradient boosted decision trees (Chen & Guestrin, 2016)[5]. This is a powerful machine learning algorithm used for multiclass classification, which has been shown to outperform other classification methods in terms of accuracy and computational efficiency (Fernández-Delgado et al., 2014)[6].The use of Xgboost Gradient boosted decision trees is particularly well-suited for this analysis for several reasons. First, it is capable of handling large and high-dimensional data sets, which are common in neuroimaging research (Woo et al., 2017)[7]. Second, it can automatically handle missing data and multicollinearity, which are common challenges in neuroimaging studies (Hastie et al., 2009)[8]. Third, it has a built-in feature selection mechanism, which can help identify the most important features that contribute to subtype classification (Chen & Guestrin, 2016)[5]. To further understand the importance of each feature in the classification process, the SHAP (Shapley additive explanations) method is used (Lundberg & Lee, 2017)[9]. SHAP values provide a unified measure of feature importance, allowing researchers to interpret the impact of each feature on the classification outcome in a consistent and interpretable manner.
Assessing Overlap using Chi-square Tests and Odds Ratios After classifying the individuals into different subtypes based on their functional, structural, and DTI profiles, it is crucial to assess the degree of overlap between these subtypes. This is done using chi-square tests and odds ratios, which are statistical methods for determining the association between categorical variables (Agresti, 2002)[10]. The use of chi-square tests and odds ratios is particularly suitable for this analysis for several reasons. First, they are non-parametric methods, which do not rely on assumptions about the underlying distribution of the data (Sheskin, 2003)[11]. This makes them more robust and suitable for analyzing the complex and heterogeneous nature of neuroimaging data. Second, they provide an intuitive way to quantify the degree of association between different subtypes, allowing researchers to determine whether the identified subtypes are indeed distinct or if there is a significant overlap between them (Agresti, 2002)[10]. Third, they can be easily applied to large and high-dimensional data sets, making them well-suited for neuroimaging research (Woo et al., 2017)[7].
Applications and Implications of Neuroimaging Subtypes The identification of neuroimaging subtypes has several important applications and implications for understanding cognitive functioning and psychopathology in children and adolescents.
Neurobiological heterogeneity: Identifying distinct subtypes based on neuroimaging measures can help account for the neurobiological heterogeneity that exists among children and adolescents (Casey et al., 2014)[12]. By grouping individuals with similar functional connectivity or structural characteristics, researchers can better understand the unique neural mechanisms that underlie cognitive functioning and psychopathology in different subpopulations.
Understanding brain development: The use of subtyping approaches can shed light on the complex and dynamic processes of brain development during childhood and adolescence (Giedd et al., 1999)[1]. By examining changes in brain volume, cortical thickness, and white matter connectivity, researchers can gain valuable insights into the associations between brain development and the emergence of cognitive and psychiatric disorders (Casey et al., 2014)[12].
Early identification and intervention: Identifying subtypes of children and adolescents based on their neurobiological profiles can potentially lead to the development of more targeted and effective interventions for cognitive and psychiatric disorders (Insel, 2014)[13]. Early identification and intervention are particularly important in this age group, as intervening early can result in better long-term outcomes (Johnson et al., 2016)[14].
Improved diagnostic precision: The use of neuroimaging subtypes can help differentiate between disorders with overlapping symptoms but distinct neurobiological profiles, leading to more accurate diagnoses and more personalized treatment approaches (Di Martino et al., 2014)[2].
Longitudinal study designs: Incorporating neuroimaging subtypes in longitudinal studies allows researchers to track changes in brain structure and connectivity over time, providing valuable information on the developmental trajectories of cognitive functioning and psychopathology (Casey et al., 2014)[12]. This information can be crucial for identifying risk factors and understanding the progression of various disorders.
Integration with other measures: Neuroimaging subtypes can be integrated with other measures, such as genetic, behavioral, and environmental data, to provide a more comprehensive understanding of the factors that contribute to cognitive functioning and psychopathology in children and adolescents (Craddock et al., 2013)[15]. By combining data from multiple sources, researchers can develop a more holistic view of the complex interplay between biology, environment, and behavior in shaping cognitive and psychiatric outcomes.
To ensure the robustness and reliability of the identified subtypes, validation is a crucial step in the analysis process. Validation in this context refers to using a split sample approach, where the sample is divided into two parts. The subtypes and profiles obtained in Sample 1 are compared to those obtained in Sample 2 to assess the consistency of the results. This direct comparison supports the initial validation of the identified subtypes, ensuring that the subtypes are reproducible and not merely artifacts of the data or methodological choices. Validation is necessary for data-driven clustering as it enhances reproducibility, reliability, and robustness across the findings. By validating the subtypes, researchers can be more confident that the identified subgroups are indeed distinct and not merely the result of random fluctuations in the data. This ensures that the findings are more likely to generalize to other samples and populations, thus increasing the utility of the subtypes for understanding the neural mechanisms underlying cognitive functioning and psychopathology. Moreover, validation plays a crucial role in building the rationale for the importance of addressing the overarching question and justifying the use of the specific methods employed. The validation process demonstrates that the identified subtypes are not only data-driven but also consistent across different samples, thus providing evidence for their robustness and reliability. This, in turn, strengthens the argument that these subtypes are necessary for understanding the complex neural mechanisms driving cognitive functioning and psychopathology and for guiding the development of targeted interventions. In summary, addressing the question of whether subtypes based separately on resting state functional connectivity, structural imaging, and diffusion tensor imaging measures can improve our understanding of the underlying neural mechanisms that contribute to typical and atypical development in cognitive functioning and psychopathology is of critical importance. The validation process, through the use of split sample comparisons, ensures the reproducibility, reliability, and robustness of the findings, ultimately leading to more accurate insights and better-informed interventions for cognitive and psychiatric disorders. By leveraging the power of subtyping, multimodal neuroimaging approaches, and rigorous validation techniques, we can advance our understanding of the complex neural mechanisms that drive development, ultimately leading to more effective interventions and improved outcomes for those affected by cognitive and psychiatric disorders.
Cognitive and behavioral difficulties in children and adolescents are highly heterogeneous, with different individuals exhibiting distinct patterns of symptoms and responses to treatment (Cuthbert & Insel, 2013)[1]. This heterogeneity presents significant challenges for accurate diagnosis, effective treatment of cognitive and psychiatric disorders, and understanding the underlying neurobiological mechanisms (Craddock et al., 2013)[2]. A critical question guiding our investigation is: How can subtyping based on neuroimaging measures help us better understand the heterogeneity of cognitive and behavioral difficulties? Neuroimaging-based subtypes offer the potential to advance our understanding of this heterogeneity by identifying distinct subpopulations characterized by specific patterns of brain organization, which may relate to cognitive functioning and psychopathology (Insel, 2014)[3]. Developing these subtypes could provide valuable insights into the neurobiological mechanisms underlying cognitive and behavioral difficulties and help tailor interventions more effectively to individual needs (Casey et al., 2014)[4].
To address this overarching question, we employ several analytical tools in a sequential manner. First, we conduct subtype confirmatory factor analysis (CFA) invariance testing to evaluate whether the same cognitive and psychopathology constructs are being measured across the subtypes. This analysis ensures that any observed differences across subtypes are not due to measurement artifacts but rather reflect true underlying differences in the constructs of interest (Millsap, 2011)[6]. Establishing measurement invariance is crucial before comparing the subtypes on various outcomes, as it provides a solid foundation for the subsequent analyses. Once measurement invariance has been established, we use analysis of covariance (ANCOVA) to test for differences across the subtypes regarding measures of interest, such as cognitive functioning and psychopathology. This approach allows us to determine whether the neuroimaging-based subtypes provide meaningful information about the data above and beyond what could be obtained without subtyping (Tabachnick & Fidell, 2013)[5]. Additionally, the ANCOVA helps identify potential interactions between subtypes and other factors, offering insights into how neurobiological differences might contribute to variations in cognitive and behavioral difficulties.
In conclusion, by investigating the importance of neuroimaging-based subtyping and employing robust analytical tools such as subtype CFA invariance testing and ANCOVA, we aim to enhance our understanding of the heterogeneity in cognitive and behavioral difficulties. This research has the potential to contribute significantly to the development of more accurate diagnostic tools and personalized treatment strategies for cognitive and psychiatric disorders in children and adolescents.
Cognitive and behavioral outcomes in children and adolescents are influenced by a complex interplay of genetic, environmental, and neurobiological factors (Casey et al., 2014)[1]. Developing accurate and reliable predictive models for these outcomes is essential for early identification and intervention, which can lead to better long-term outcomes (Johnson et al., 2016)[2]. A critical question that guides our investigation is: How can subtyping based on neuroimaging measures improve our ability to predict cognitive and behavioral outcomes in children and adolescents, and do these subtypes provide meaningful information for better prediction? Conditional random forests (CRFs) are a powerful tool for addressing this question (Breiman, 2001)[3]. CRFs are an ensemble learning method that combines multiple decision trees to generate more accurate and stable predictions. There are several reasons why CRFs are particularly suited to assess feature importance with factors and investigate the predictive utility of neuroimaging-based subtypes: 1. Non-parametric: Unlike parametric models that make assumptions about the underlying data distribution, CRFs can model complex and non-linear relationships between predictors and outcomes without such assumptions (Cutler et al., 2007)[4]. This flexibility makes them well-suited for analyzing the heterogeneous and high-dimensional neuroimaging data used to derive subtypes. In contrast, parametric models may fail to capture the true relationships in the data, leading to biased and less accurate predictions. 2. Robustness: CRFs are less prone to overfitting compared to single decision trees, as they rely on the aggregation of multiple trees, each constructed using a random subset of the data (Breiman, 2001)[3]. This “bagging” approach reduces the overall model variance and increases stability, ensuring that the models generalize well to new, unseen data. Overfitting is a significant concern when evaluating the predictive utility of subtypes, as models that perform well on the training data but poorly on new data are of limited practical use. 3. Feature importance: CRFs provide an intuitive and interpretable measure of feature importance by quantifying the decrease in prediction accuracy when a specific feature is permuted, or randomly shuffled (Strobl et al., 2007)[5]. This “permutation importance” allows us to assess the relative importance of subtypes in predicting cognitive and behavioral outcomes compared to other factors, such as demographic and SES variables. Understanding the relative importance of these factors can help inform targeted interventions and policy decisions. 4. Interaction detection: CRFs can naturally account for interactions between features by allowing decision trees to split on multiple features at different levels of the tree hierarchy (Friedman & Popescu, 2008)[6]. This capability makes them an ideal tool for investigating the complex relationships between neuroimaging-based subtypes, demographic factors, and cognitive and behavioral outcomes, providing a more comprehensive understanding of the factors that contribute to these outcomes. To evaluate whether subtypes enhance our predictive ability, we will compare the prediction errors (i.e., the differences between actual and predicted scores) between CRF models with and without the subtypes. A significant reduction in prediction error when subtypes are included in the model would indicate that they provide meaningful information for better prediction. Compared to gradient boosted decision trees (GBDTs), which were used to classify subtypes and derive the most important features for distinguishing them, CRFs are a more appropriate model for assessing feature importance in this context. GBDTs are designed for boosting weak learners to improve overall model performance, but they can be prone to overfitting when dealing with complex interactions and high-dimensional data (Friedman, 2001)[7]. In contrast, CRFs are more robust to overfitting and can better handle the complexity of the relationships between neuroimaging-based
The pervasive heterogeneity of cognitive and behavioral difficulties in children and adolescents necessitates the development of reliable and robust methods to uncover meaningful subtypes. For a scientific journal such as Neuron, it is essential to demonstrate that the subtypes identified are reproducible, reliable, and robust across different samples and analyses. Therefore, we address the overarching question: How reproducible are these subtypes, their differences, and predictive ability? To answer this question, we employ three key strategies:
Split Sample/Full Sample Approach: We begin by dividing the full sample of 10,000+ children (9-10 years old) at their baseline visit into two split samples, each containing approximately 5,400 children. By conducting analyses separately in these two samples, we can evaluate the consistency of our findings across different datasets. Results that reproduce across both samples will be considered “reproducible” and “robust”. This approach not only allows us to assess the stability of our subtypes and their differences, but also their predictive ability in different populations. Bootstrapping: Bootstrapping is a statistical resampling method widely used in evaluating the reliability, reproducibility, and stability of statistical analyses (Efron & Tibshirani, 1993)[1]. It involves generating multiple samples with replacement from the original sample, which provides an estimate of the sampling distribution of a statistic. Bootstrapping can be applied to any type of sample distribution and is not sensitive to outliers or skewness. By resampling the data, it provides an estimate of the variability in the sample, allowing researchers to quantify the uncertainty and robustness of their results. Additionally, bootstrapping offers a means to calculate confidence intervals and perform hypothesis tests without relying on assumptions about the underlying distribution, making it a valuable tool for evaluating the stability, reliability, and reproducibility of statistical analyses in the ABCD dataset. Subsampling: Subsampling is a technique for evaluating the reliability, reproducibility, and stability of statistical analyses by randomly selecting different subsets of the data, ranging from 10% to 90% of the full sample (Vardi & Zhang, 2000)[2]. This allows for a systematic assessment of the robustness of the results obtained from a full dataset. By examining how sensitive the results are to changes in the size and composition of the sample, we can determine the degree of confidence we have in our findings. This is especially important when working with large datasets, where small changes in the data can have significant impacts on the results. To implement these methods, we first partition each original sample to have an equal number of subtypes (n=910 per subtype). This partitioning is designed to protect against increasing the imbalance in the number of subtypes pulled during the bootstrap resampling. Each partitioned sample is then resampled with replacement (bootstrapping) - 66% of the individuals from the original sample are used in each bootstrapped iteration. We perform ANOVA on the cognitive functioning measures for each new bootstrapped sample. A total of 1000 bootstrapped analyses are performed for each sample. In addition, the partitioned samples from each iteration are combined to create a “full sample”. This full sample is then resampled with replacement and run through ANOVA analyses.
By employing these rigorous methods, we ensure that our data-driven clustering approach enhances the reproducibility, reliability, and robustness of our findings. This is crucial for the credibility and impact of our research in the field of cognitive and behavioral difficulties in children and adolescents, as it provides a solid foundation for further studies and the development of more accurate identification and targeted interventions for cognitive and psychiatric disorders in this age group.