NIH Cognitive battery scoring

Overall Correlation Matrices

Note: these correlations matrices have lower power because the correlations are only calculated with complete cases across all the sampled tasks - only 50% of the entire sample has data for all three EF tasks. Or to be interpreted as correlations for our most reliable participants. Individual correlations reported later for each individual task are more accurate.

Entire ORA Sample

Current scoring

NIH scoring

Elie’s Scoring

Montreal Sample

Current scoring

NIH scoring

Elie’s scoring

German Sample

Current scoring

NIH scoring

Elie’s scoring

UK Sample

Current Scoring

NIH Scoring

Elie’s Scoring

Flanker

Current scoring - Federico et al., 2016

ORA Sample with Current Flanker Scoring

The correlation between the Flanker task as scored currently and age in the entire ORA sample is:

ORA Sample with Truncated Flanker Scoring

The correlation between the Flanker task with negative scores truncated to 0 and age in the entire ORA sample is:

Montreal Sample with Current Flanker Scoring

The correlation between age and Flanker scores as scored currently in the MTL sample is:

German Sample with Current Flanker Scoring

The correlation between age and Flanker scores as scored currently in the GER sample is:

UK Sample with Current Flanker Scoring

The correlation between age and Flanker scores as scored currently in the UK sample is:

Scoring Description

Measures of the efficiency of the three attentional networks are obtained via simple subtractions of RTs (or percentage of errors) between conditions. The so-called conflict effect is calculated by subtracting the mean RTs of the congruent flanking conditions from the mean RTs of incongruent flanking conditions. The two conditions differ only in the information given by the flankers. When the images are congruent, they provide a facilitating effect on the discrimination of the target stimulus, whereas incongruent flankers distract participants

TL,DR: Percentage error in incongruent trials - percentage error in congruent trials.

This scoring results in a ‘conflict effect’ Flanker score that can range from -1 to 1.

The score on the template currently calculates this by doing: > Percentage correct in congruent trials - percentage correct in incongruent trials

which is mathematically the same.

  • Children that perform similarly in incongruent trials as congruent trials (i.e., who aren’t distracted or enhanced by flankers) get scores closer to 0.
  • Children who perform worse in the incongruent trials than the congruent trials (i.e., are highly distracted by the flankers) get scores closer to 1.
  • Children who perform better in the incongruent trials than the congruent trials get scores closer to -1 (i.e., they perform better under conditions of distraction).

Federico et al. processed the data exactly like we did, excluding kids based on RTs and setting a floor/ceilling for the rest of the RTs.

They had 24 practice trials and 96 test trials compared to our 4 practice trials and 24 test trials.

So this conflict effect score really measures the sensitivity of children’s attention to distractions.

Federico et al. reported a significant effect of age in their anova F(2, 63) = 6.89 p < .002; partial g2 = .18:

Percentage of errors was higher in children aged 6 years compared to both children aged 8 (p < .02) and 10 years (p < .002) (children aged 6: 17.01% vs. children aged 6: 8.35% vs. children aged 6: 6.02%).
Flanker error percentage by age reported by Federico et al. 2017
Flanker error percentage by age reported by Federico et al. 2017

Source: Federico, F., Marotta, A., Martella, D., & Casagrande, M. (2016). Development in attention functions and social processing: Evidence from the Attention Network Test. British Journal of Developmental Psychology, 35(2), 169–185. doi:10.1111/bjdp.12154

NIH Scoring - Zelazo et al., 2013

ORA Sample with NIH Scoring

This correlation between the Flanker task with the NIH scoring and age in the entire ORA sample is:

MTL Sample with NIH Scoring

This correlation between the Flanker task with the NIH scoring and age in the MTL sample is:

GER Sample with NIH Scoring

This correlation between the Flanker task with the NIH scoring and age in the GER sample is:

UK Sample with NIH Scoring

This correlation between the Flanker task with the NIH scoring and age in the UK sample is:

Scoring Description

The NIH toolbox version of the Flanker task uses a simple proportion of correct responses combining across congruent and incongruent trials. Their version of the task has 40 trials because they have a 2nd, harder block due to using the task for adults too, but the block designed for kids 3-6yo is 20 trials. They wanted their Flanker score to be on a scale of 0-5, so the formula for calculating it is:

Flanker Accuracy Score = 0.250 * Number of Correct Responses

This boils down to a proportion of correct responses across all trials, whether congruent and incongruent.

Zelazo et al. reported a correlation of .84 between age and Flanker scores calculated as such in kids between 3 and 6yo on 87 df.

Sources:

  1. Zelazo, P. D., Anderson, J. E., Richler, J., Wallner-Allen, K., Beaumont, J. L., & Weintraub, S. (2013). II. NIH Toolbox Cognition Battery (CB): Measuring executive function and attention. Monographs of the Society for Research in Child Development, 78(4), 16–33. https://doi.org/10.1111/mono.12032
  1. https://www.nihtoolbox.org/app/uploads/2022/05/Toolbox_Scoring_and_Interpretation_Guide_for_iPad_v1.7-5.25.21.pdf

Incongruent Accuracy Scoring

ORA Sample with Incongruent Accuracy Scoring

This correlation between the Flanker task with the incongruent accuracy scoring and age in the entire ORA sample is:

MTL Sample with incongruent accuracy Scoring

This correlation between the Flanker task with the incongruent accuracy scoring and age in the MTL sample is:

GER Sample with incongruent accuracy Scoring

This correlation between the Flanker task with the incongruent accuracy scoring and age in the GER sample is:

UK Sample with incongruent accuracy Scoring

This correlation between the Flanker task with the incongruent accuracy scoring and age in the UK sample is:

Scoring Description

Instead, using measures from only the incongruent (Flanker) and post-switch (DCCS) conditions may be more informative - though obviously these measures wouldn’t control for participants’ general ability to perform the task, they may better tap the constructs we’re interested in:

% correct in Flanker incongruent would be a measure of how well children can inhibit distractors Post-switch accuracy in DCCS would measure how well children can perform after a switch has been made

DCCS

Current DCCS scoring - Federico et al., 2016

ORA Sample with Current DCCS Scoring

The correlation between age and DCCS as currently scored in the ORA sample is:

ORA Sample with Truncated DCCS Scoring

The correlation between age and DCCS with negative scores truncated to 0 in the ORA sample is:

Montreal Sample with Current DCCS Scoring

The correlation between age and DCCS as scored currently in the MTL sample is:

German Sample with Current DCCS Scoring

The correlation between age and DCCS as scored currently in the GER sample is:

UK Sample with Current DCCS Scoring

The correlation between age and DCCS as scored currently in the UK sample is:

Scoring Description

Following the same logic adopted for the Flanker task, the magnitude of the switching effect will be computed as the difference in the proportion of accurate trials between the pre-switch and post-switch trials. This measure, resulting in a score ranging from -1 to 1, will be used as the outcome measure of the DCCS task.

TL,DR: pre-switch accuracy - post-switch accuracy

  • Kids who scored similarly high/low on both blocks get scores closer to 0.
  • Kids who score higher on the pre-switch block than the post-switch block (i.e., bad at rule switching) get scores close to 1
  • Kids who score lower on the pre-switch block than the post-switch block (i.e., somehow enhanced performance after the rule-switch)

Source: Federico, F., Marotta, A., Martella, D., & Casagrande, M. (2016). Development in attention functions and social processing: Evidence from the Attention Network Test. British Journal of Developmental Psychology, 35(2), 169–185. doi:10.1111/bjdp.12154

NIH Scoring - Zelazo et al., 2013

ORA Sample with NIH DCCS Scoring

The correlation between age and DCCS with NIH scoring in the ORA sample is:

Montreal Sample with NIH DCCS Scoring

The correlation between age and DCCS with NIH scoring in the MTL sample is:

German Sample with NIH DCCS Scoring

The correlation between age and DCCS with NIH scoring in the GER sample is:

UK Sample with NIH DCCS Scoring

The correlation between age and DCCS with NIH scoring in the UK sample is:

Scoring Description

Proportion of correct answers out of 10 trials in the pre-switch and post-switch conditions combined. They also included the final mixed block in their scoring, which we have decided to omit from our analysis.

Zelazo et al. reported a correlation of .79 between age and DCCS scores calculated as such in kids between 3 and 6yo on 82 df.

Sources: 1. Zelazo, P. D., Anderson, J. E., Richler, J., Wallner-Allen, K., Beaumont, J. L., & Weintraub, S. (2013). II. NIH Toolbox Cognition Battery (CB): Measuring executive function and attention. Monographs of the Society for Research in Child Development, 78(4), 16–33. https://doi.org/10.1111/mono.12032 2. https://www.nihtoolbox.org/app/uploads/2022/05/Toolbox_Scoring_and_Interpretation_Guide_for_iPad_v1.7-5.25.21.pdf

Post-Switch Accuracy Scoring

ORA Sample with Post-Switch Accuracy DCCS Scoring

The correlation between age and DCCS with Post-Switch Accuracy scoring in the ORA sample is:

Montreal Sample with Post-Switch Accuracy DCCS Scoring

The correlation between age and DCCS with Post-Switch Accuracy scoring in the MTL sample is:

German Sample with Post-Switch Accuracy DCCS Scoring

The correlation between age and DCCS with Post-Switch Accuracy scoring in the GER sample is:

UK Sample with Post-Switch Accuracy DCCS Scoring

The correlation between age and DCCS with Post-Switch Accuracy scoring in the UK sample is:

Scoring Description

Instead, using measures from only the incongruent (Flanker) and post-switch (DCCS) conditions may be more informative - though obviously these measures wouldn’t control for participants’ general ability to perform the task, they may better tap the constructs we’re interested in:

% correct in Flanker incongruent would be a measure of how well children can inhibit distractors Post-switch accuracy in DCCS would measure how well children can perform after a switch has been made

WM

Current WM Scoring

ORA Sample with Current WM Scoring

The correlation between the WM task as scored currently and age in the entire ORA sample is:

Montreal Sample with Current WM Scoring

The correlation between the WM task as scored currently and age in the MTL sample is:

German Sample with Current WM Scoring

The correlation between the WM task as scored currently and age in the GER sample is:

UK Sample with Current WM Scoring

The correlation between the WM task as scored currently and age in the UK sample is:

Scoring Description

Children get points for trials answered correctly (2 points for 1st attempt correct and 1 point for 2nd attempt correct) - this is identical to the method used by Tulsky et al., 2013 below.

Then this transformation is applied to make the scores comparable to the Flanker & DCCS.

The sum of the scores of the two blocks (maximum score = 24) is multiplied by -1, summed to 12 and divided by twelve. The resulting score (ranging from -1 and 1) constitutes the working memory score.

  • Children who scored high on the WM (i.e., large WM capacity) get scores closer to -1.
  • Children who scored low on the WM (i.e., low WM capacity) get scores closer to 1.

NIH Scoring - Tulsky et al., 2013

ORA Sample with NIH WM Scoring

The correlation between the WM task with NIH scoring and age in the entire ORA sample is:

Montreal Sample with NIH WM Scoring

The correlation between the WM task with NIH scoring and age in the MTL sample is:

German Sample with NIH WM Scoring

The correlation between the WM task with NIH scoring and age in the GER sample is:

UK Sample with NIH WM Scoring

The correlation between the WM task with NIH scoring and age in the UK sample is:

Scoring Description

From the same monograph about developing the NIH cognitive toolbox, WM was just developed separately from Flanker & DCCS and by a different group.

For both phases, for each list length, participants receive a score of 2 points if they are correct on the first trial. A second trial at a given list length is only administered when participants fail the first trial. Participants receive a score of 1 point only for a given list length if they fail the first trial at that list length but pass the second trial. Test scores consist of combined total trials correct on the 1-List and 2-List phases of the task.

TL,DR: sum of all points in both blocks of the WM.

Tulsky et al. reported a correlation of .52 between age and WM scores calculated as such in kids between 3 and 6yo on 111 df.