Purpose

This log documents inter-rater reliability monitoring during formal title and abstract screening. Screening is conducted in 20 successive non-overlapping blocks of approximately 5% of the corpus. Each block is assessed using independent pre-resolution decisions only.

Binary Fleiss’ kappa measures agreement on the operational decision to retain or exclude a reference. Nominal Fleiss’ kappa measures agreement across the full INC, MAYBE, and EXC decision framework.

The final PDF is accompanied by this RMarkdown source file and the CSV exports used for each block, providing a complete computational audit trail.

Monitoring rules

Measure Decision structure Threshold Protocol action
Binary Fleiss’ kappa INC/MAYBE retained vs EXC ≥ 0.75 A single block below threshold triggers full recalibration.
Nominal Fleiss’ kappa INC, MAYBE, and EXC ≥ 0.60 A single low block is recorded and monitored; three consecutive low blocks trigger targeted MAYBE recalibration.

Binary kappa is treated as the primary safeguard because it measures agreement on whether a reference advances to full-text screening. Nominal kappa captures whether reviewers apply the full three-category framework consistently, particularly the MAYBE category.

Block 01 | References #1–341 | 0–5% of corpus

Block integrity and residual duplicates

Block 01 contained 341 imported references. Of these, 338 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 338
Residual duplicates verified in Covidence 3
References included in IRR calculations 338
Residual duplicates removed from Block 01
Covidence # Title
#139 Transforming growth factor-β signaling: Tumorigenesis and targeting for cancer therapy
#439 A Reasonable Officer: Examining the Relationships Among Stress, Training, and Performance in a Highly Realistic Lethal Force Scenario
#39 Corrigendum to ‘Beyond Change Scores: Employing an Improved Statistical Approach to Analyze the Impact of Entry Fitness on Physical Performance During British Army Basic Training in Men and Women’

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 338
Nominal Fleiss’ kappa 0.591
Raw agreement 82.5%
Nominal disagreements 59/338
INC/MAYBE disagreements 18/338

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 338
Binary Fleiss’ kappa 0.691
Raw agreement 87.9%
Binary conflicts 41/338
Conflict rate (95% Wilson CI) 12.1% (9.1–16.0%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.800
Retention-specific agreement 77.3%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 227 67.2%
Conflicting retention decision (EXC / INC or MAYBE) 41 12.1%
Retained by both reviewers (INC and/or MAYBE) 70 20.7%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 26 37.1%
INC / MAYBE 18 25.7%
MAYBE / MAYBE 26 37.1%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 338
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.691
Nominal Fleiss’ kappa 0.591
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 02 | References #342–682 | 5–10% of corpus

Block integrity and residual duplicates

Block 02 contained 341 imported references. Of these, 339 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 2 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 339
Residual duplicates verified in Covidence 2
References included in IRR calculations 339
Residual duplicates removed from Block 02
Covidence # Title
#643 Bell 2015 : Effects of 13-Hour 20-Minute Work Shifts on Law Enforcement Officers’ Sleep, Cognitive Abilities, Health, Quality of Life, and Work Performance: The Phoenix Study
#768 Bishop 1991: Limitations to heavy work at 21 degrees C of personnel wearing the U.S. military chemical defense ensemble

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 339
Nominal Fleiss’ kappa 0.587
Raw agreement 81.1%
Nominal disagreements 64/339
INC/MAYBE disagreements 15/339

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 339
Binary Fleiss’ kappa 0.651
Raw agreement 85.5%
Binary conflicts 49/339
Conflict rate (95% Wilson CI) 14.5% (11.1–18.6%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.753
Retention-specific agreement 75.4%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 215 63.4%
Conflicting retention decision (EXC / INC or MAYBE) 49 14.5%
Retained by both reviewers (INC and/or MAYBE) 75 22.1%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 31 41.3%
INC / MAYBE 15 20.0%
MAYBE / MAYBE 29 38.7%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 339
References with zero votes 2
References excluded from IRR calculation 2
Binary Fleiss’ kappa 0.651
Nominal Fleiss’ kappa 0.587
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 03 | References #683–1023 | 10–15% of corpus

Block integrity and residual duplicates

Block 03 contained 341 imported references. Of these, 338 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 338
Residual duplicates verified in Covidence 3
References included in IRR calculations 338
Residual duplicates removed from Block 02
Covidence # Title
#1155 [Evaluation of Borg’s perceived exertion scale in cardiac rehabilitation]
#1252 Differences between U.S. Army Trainees and Active Duty Soldiers in Performing Physically Demanding Occupational Tasks: 2950 Board #233 June 1 3:30 PM - 5:00 PM
#1295 Systematic Review and Meta-Analysis: Iron Deficiency and Iron Deficiency Anemia Among Military Recruits Undergoing Basic Combat Training

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 338
Nominal Fleiss’ kappa 0.669
Raw agreement 86.1%
Nominal disagreements 47/338
INC/MAYBE disagreements 12/338

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 338
Binary Fleiss’ kappa 0.734
Raw agreement 89.6%
Binary conflicts 35/338
Conflict rate (95% Wilson CI) 10.4% (7.5–14.1%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.830
Retention-specific agreement 80.4%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 231 68.3%
Conflicting retention decision (EXC / INC or MAYBE) 35 10.4%
Retained by both reviewers (INC and/or MAYBE) 72 21.3%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 22 30.6%
INC / MAYBE 12 16.7%
MAYBE / MAYBE 38 52.8%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 338
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.734
Nominal Fleiss’ kappa 0.669
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 04 | References #1024–1365 | 15–20% of corpus

Block integrity and residual duplicates

Block 04 contained 342 imported references. Of these, 342 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 0 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 342
References with exactly two independent votes 342
Residual duplicates verified in Covidence 0
References included in IRR calculations 342

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 342
Nominal Fleiss’ kappa 0.605
Raw agreement 82.5%
Nominal disagreements 60/342
INC/MAYBE disagreements 23/342

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 342
Binary Fleiss’ kappa 0.735
Raw agreement 89.2%
Binary conflicts 37/342
Conflict rate (95% Wilson CI) 10.8% (8.0–14.6%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.817
Retention-specific agreement 81.0%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 226 66.1%
Conflicting retention decision (EXC / INC or MAYBE) 37 10.8%
Retained by both reviewers (INC and/or MAYBE) 79 23.1%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 19 24.1%
INC / MAYBE 23 29.1%
MAYBE / MAYBE 37 46.8%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 342
References with exactly two independent votes 342
References with zero votes 0
References excluded from IRR calculation 0
Binary Fleiss’ kappa 0.735
Nominal Fleiss’ kappa 0.605
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 05 | References #1366–1707 | 20–25% of corpus

Block integrity and residual duplicates

Block 05 contained 342 imported references. Of these, 339 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 342
References with exactly two independent votes 339
Residual duplicates verified in Covidence 3
References included in IRR calculations 339
Residual duplicates removed from Block 02
Covidence # Title
#2181 Eastern district AAHPERD 2007 research keynote: Effects of current trends in youth fitness on physical performance assessments and training methodologies of the United States Military Academy and the Army
#2339 Occupational group as a moderator of the job satisfaction-job performance relationship
#2412 Making whole blood available in austere medical environments: donor performance and safety

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 339
Nominal Fleiss’ kappa 0.516
Raw agreement 81.7%
Nominal disagreements 62/339
INC/MAYBE disagreements 19/339

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 339
Binary Fleiss’ kappa 0.640
Raw agreement 87.3%
Binary conflicts 43/339
Conflict rate (95% Wilson CI) 12.7% (9.6–16.6%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.804
Retention-specific agreement 72.3%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 240 70.8%
Conflicting retention decision (EXC / INC or MAYBE) 43 12.7%
Retained by both reviewers (INC and/or MAYBE) 56 16.5%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 16 28.6%
INC / MAYBE 19 33.9%
MAYBE / MAYBE 21 37.5%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 342
References with exactly two independent votes 339
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.640
Nominal Fleiss’ kappa 0.516
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 06 | References #1708-2047 | 25–30% of corpus

Block integrity and residual duplicates

Block 06 contained 339 imported references. Of these, 336 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 339
References with exactly two independent votes 336
Residual duplicates verified in Covidence 3
References included in IRR calculations 336
Residual duplicates removed from Block 02
Covidence # Title
#2690 Psychological, physical, and cognitive factors that influence tactical performance during a military relevant virtual reality scenario
#2693 Psychological, Physical, And Cognitive Associations With Tactical Performance During A Military Relevant Virtual Reality Scenario: 1553…American College of Sports Medicine (ACSM) Annual Meeting, May 27-30, 2025, Atlanta, Georgia
#2709 U.S. Army Physical Demands Study: Development of the Occupational Physical Assessment Test for Combat Arms soldiers

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 336
Nominal Fleiss’ kappa 0.602
Raw agreement 84.5%
Nominal disagreements 52/336
INC/MAYBE disagreements 16/336

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 336
Binary Fleiss’ kappa 0.705
Raw agreement 89.3%
Binary conflicts 36/336
Conflict rate (95% Wilson CI) 10.7% (7.8–14.5%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.832
Retention-specific agreement 77.5%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 238 70.8%
Conflicting retention decision (EXC / INC or MAYBE) 36 10.7%
Retained by both reviewers (INC and/or MAYBE) 62 18.5%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 17 27.4%
INC / MAYBE 16 25.8%
MAYBE / MAYBE 29 46.8%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 339
References with exactly two independent votes 336
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.705
Nominal Fleiss’ kappa 0.602
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 07 | References #2048-2388 | 30–35% of corpus

Block integrity and residual duplicates

Block 07 contained 341 imported references. Of these, 337 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 4 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 337
Residual duplicates verified in Covidence 4
References included in IRR calculations 337
Residual duplicates removed from Block 02
Covidence # Title
#2987 Establishing A Cognitive, Health, Physical, And Social-emotional Toolkit To Predict Soldier Performance: 486
#3211 How Effective Is Initial Military-Specific Training in the Development of Physical Performance of Soldiers?
#3352 United States Special Operations Command South Training Program’s Effectiveness: A Case Study
#3372 Preliminary examination of timesharing in United States Air Force ROTC cadets

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 337
Nominal Fleiss’ kappa 0.692
Raw agreement 87.5%
Nominal disagreements 42/337
INC/MAYBE disagreements 14/337

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 337
Binary Fleiss’ kappa 0.781
Raw agreement 91.7%
Binary conflicts 28/337
Conflict rate (95% Wilson CI) 8.3% (5.8–11.7%)

Interpretation: Excellent agreement; threshold met.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.866
Retention-specific agreement 83.7%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 237 70.3%
Conflicting retention decision (EXC / INC or MAYBE) 28 8.3%
Retained by both reviewers (INC and/or MAYBE) 72 21.4%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 13 18.1%
INC / MAYBE 14 19.4%
MAYBE / MAYBE 45 62.5%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 337
References with zero votes 4
References excluded from IRR calculation 4
Binary Fleiss’ kappa 0.781
Nominal Fleiss’ kappa 0.692
Preceding low nominal-kappa blocks 0
Protocol action Continue

Decision: Screening may continue without recalibration.

Rationale: Both pre-specified agreement thresholds were met.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 08 | References #2389-2730 | 35–40% of corpus

Block integrity and residual duplicates

Block 08 contained 342 imported references. Of these, 339 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 342
References with exactly two independent votes 339
Residual duplicates verified in Covidence 3
References included in IRR calculations 339
Residual duplicates removed from Block 02
Covidence # Title
3440 Heart Rate Reserve: An Objective Measure of Soldiers’ Physical Exertion During Field Operations: 2955 Board #238 June 1 3:30 PM - 5:00 PM
#3568 Predictors Of Physical Readiness In Rotary Wing Naval Aviators: 1479
#3647 β-Alanine supplementation and military performance

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 339
Nominal Fleiss’ kappa 0.628
Raw agreement 85.0%
Nominal disagreements 51/339
INC/MAYBE disagreements 13/339

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 339
Binary Fleiss’ kappa 0.702
Raw agreement 88.8%
Binary conflicts 38/339
Conflict rate (95% Wilson CI) 11.2% (8.3–15.0%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.820
Retention-specific agreement 77.6%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 235 69.3%
Conflicting retention decision (EXC / INC or MAYBE) 38 11.2%
Retained by both reviewers (INC and/or MAYBE) 66 19.5%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 19 28.8%
INC / MAYBE 13 19.7%
MAYBE / MAYBE 34 51.5%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 342
References with exactly two independent votes 339
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.702
Nominal Fleiss’ kappa 0.628
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 09 | References #2731-3070 | 40–45% of corpus

Block integrity and residual duplicates

Block 09 contained 340 imported references. Of these, 338 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 2 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 340
References with exactly two independent votes 338
Residual duplicates verified in Covidence 2
References included in IRR calculations 338
Residual duplicates removed from Block 02
Covidence # Title
3896 Key Factors Influencing Purchasing Of Dietary Supplements Among Female Service Academy Cadets
#3961 The Benefit of Mental Skills Training on Performance and Stress Response in Military Personnel

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 338
Nominal Fleiss’ kappa 0.630
Raw agreement 84.9%
Nominal disagreements 51/338
INC/MAYBE disagreements 18/338

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 338
Binary Fleiss’ kappa 0.742
Raw agreement 90.2%
Binary conflicts 33/338
Conflict rate (95% Wilson CI) 9.8% (7.0–13.4%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.843
Retention-specific agreement 80.7%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 236 69.8%
Conflicting retention decision (EXC / INC or MAYBE) 33 9.8%
Retained by both reviewers (INC and/or MAYBE) 69 20.4%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 20 29.0%
INC / MAYBE 18 26.1%
MAYBE / MAYBE 31 44.9%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 340
References with exactly two independent votes 338
References with zero votes 2
References excluded from IRR calculation 2
Binary Fleiss’ kappa 0.742
Nominal Fleiss’ kappa 0.630
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 10 | References #3071-3415 | 45–50% of corpus

Block integrity and residual duplicates

Block 10 contained 345 imported references. Of these, 343 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 2 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 345
References with exactly two independent votes 343
Residual duplicates verified in Covidence 2
References included in IRR calculations 343
Residual duplicates removed from Block 02
Covidence # Title
4795 Effects of low speed wind on the recognition/identification and pass-through communication tasks of auditory situation awareness afforded by military hearing protection/enhancement devices and tactical communication and protective systems
#4804 Novel Cooling Strategies for Military Training and Operations

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 343
Nominal Fleiss’ kappa 0.620
Raw agreement 83.1%
Nominal disagreements 58/343
INC/MAYBE disagreements 26/343

Interpretation: Substantial agreement; threshold met.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 343
Binary Fleiss’ kappa 0.773
Raw agreement 90.7%
Binary conflicts 32/343
Conflict rate (95% Wilson CI) 9.3% (6.7–12.9%)

Interpretation: Excellent agreement; threshold met.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Complementary binary agreement diagnostics

The following measures are reported as complementary diagnostics of binary agreement during the initial monitoring phase. They do not replace the pre-specified binary Fleiss’ kappa threshold and do not independently determine the protocol action.

Binary Gwet’s AC1 provides an additional chance-corrected estimate of agreement. Retention-specific agreement describes agreement specifically on the operationally sensitive decision to retain a reference for full-text screening.

Complementary binary agreement diagnostics
Metric Result
Binary Gwet’s AC1 0.842
Retention-specific agreement 83.8%

Interpretation: These complementary measures are reported descriptively and diagnostically. No progression threshold is currently assigned to binary Gwet’s AC1 or retention-specific agreement. The pre-specified protocol action remains determined by binary and nominal Fleiss’ kappa.

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 228 66.5%
Conflicting retention decision (EXC / INC or MAYBE) 32 9.3%
Retained by both reviewers (INC and/or MAYBE) 83 24.2%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 16 19.3%
INC / MAYBE 26 31.3%
MAYBE / MAYBE 41 49.4%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 345
References with exactly two independent votes 343
References with zero votes 2
References excluded from IRR calculation 2
Binary Fleiss’ kappa 0.773
Nominal Fleiss’ kappa 0.620
Preceding low nominal-kappa blocks 0
Protocol action Continue

Decision: Screening may continue without recalibration.

Rationale: Both pre-specified agreement thresholds were met.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Midpoint Monitoring Analysis | Blocks 01–10 | 0–50% of corpus

This midpoint analysis provides a retrospective descriptive and exploratory assessment of inter-rater reliability and screening uncertainty across the first 50% of the corpus. It does not modify the pre-specified progression rules or retrospectively alter any block-level decision. Binary Fleiss’ kappa remains the primary operational safeguard. Gwet’s AC1 and retention-specific agreement are examined as complementary diagnostics, consistent with the monitoring strategy established during the initial screening phase.

Where additional measures are introduced at the midpoint, including exclusion-specific agreement and the retention–exclusion specific-agreement gap, they are treated as retrospective descriptive diagnostics only.

Longitudinal agreement profile

The first-half monitoring profile is presented below for Blocks 01–10. Block-level estimates are retained to show fluctuations across the screening sequence. Binary Fleiss’ kappa remains the primary operational monitoring measure, while nominal kappa, Gwet’s AC1, retention-specific agreement, and conflict rates are reported as complementary diagnostics.

Agreement monitoring across Blocks 01–10
Block N Nominal κ Binary κ Gwet AC1 Retention-specific Binary agreement Binary conflicts
01 338 0.591 0.691 0.800 77.3% 87.9% 41 (12.1%)
02 339 0.587 0.651 0.753 75.4% 85.5% 49 (14.5%)
03 338 0.669 0.734 0.830 80.4% 89.6% 35 (10.4%)
04 342 0.605 0.735 0.817 81.0% 89.2% 37 (10.8%)
05 339 0.516 0.640 0.804 72.3% 87.3% 43 (12.7%)
06 336 0.602 0.705 0.832 77.5% 89.3% 36 (10.7%)
07 337 0.692 0.781 0.866 83.7% 91.7% 28 (8.3%)
08 339 0.628 0.702 0.820 77.6% 88.8% 38 (11.2%)
09 338 0.630 0.742 0.843 80.7% 90.2% 33 (9.8%)
10 343 0.620 0.773 0.842 83.8% 90.7% 32 (9.3%)

Decision-specific agreement

Because overall binary agreement may be driven disproportionately by references jointly excluded by both reviewers, decision-specific agreement was examined separately for retained and excluded references. The retention-specific measure reflects agreement on the decision to retain a reference, whereas exclusion-specific agreement provides the corresponding measure for exclusion. Their difference is reported descriptively and is not used as an additional progression threshold.

Decision-specific agreement and uncertainty burden across Blocks 01–10
Block Retention-specific Exclusion-specific Excl. - Ret. gap Both retained Both excluded INC/MAYBE disagreement MAYBE annotations / 100 refs
01 77.3% 91.7% +14.4 pp 70 (20.7%) 227 (67.2%) 18 (5.3%) 29.6
02 75.4% 89.8% +14.4 pp 75 (22.1%) 215 (63.4%) 15 (4.4%) 31.0
03 80.4% 93.0% +12.5 pp 72 (21.3%) 231 (68.3%) 12 (3.6%) 32.2
04 81.0% 92.4% +11.4 pp 79 (23.1%) 226 (66.1%) 23 (6.7%) 36.3
05 72.3% 91.8% +19.5 pp 56 (16.5%) 240 (70.8%) 19 (5.6%) 25.4
06 77.5% 93.0% +15.5 pp 62 (18.5%) 238 (70.8%) 16 (4.8%) 29.8
07 83.7% 94.4% +10.7 pp 72 (21.4%) 237 (70.3%) 14 (4.2%) 37.1
08 77.6% 92.5% +14.9 pp 66 (19.5%) 235 (69.3%) 13 (3.8%) 33.0
09 80.7% 93.5% +12.8 pp 69 (20.4%) 236 (69.8%) 18 (5.3%) 31.7
10 83.8% 93.4% +9.6 pp 83 (24.2%) 228 (66.5%) 26 (7.6%) 40.8

Midpoint pooled comparison

To characterise the evolution of agreement during the first half of screening, Blocks 01–05 and Blocks 06–10 were compared as two successive 25% screening periods. Agreement coefficients were recalculated from the pooled pre-resolution reviewer decisions rather than obtained by averaging block-level estimates. Reference-level bootstrap 95% confidence intervals are reported for the principal agreement coefficients. Differences between periods are interpreted descriptively and do not constitute a formal estimate of the effect of recalibration or guide revision.

Pooled midpoint comparison with reference-level bootstrap 95% confidence intervals
Metric Blocks 01–05 Blocks 06–10 Change Blocks 01–10
References analysed 1,696 1,693 — 3,389
Nominal Fleiss’ kappa 0.596 [0.554–0.636] 0.635 [0.597–0.671] +0.039 0.615 [0.589–0.643]
Binary Fleiss’ kappa 0.692 [0.648–0.729] 0.742 [0.706–0.779] +0.050 0.717 [0.689–0.743]
Gwet’s binary AC1 0.801 [0.772–0.829] 0.840 [0.816–0.865] +0.039 0.821 [0.803–0.839]
AC1 - binary kappa gap 0.109 0.098 -0.011 0.104
Binary raw agreement 87.9% 90.1% +2.2 pp 89.0%
Binary conflict rate 12.1% 9.9% -2.2 pp 11.0%
Retention-specific agreement 77.4% [74.2–80.3] 80.8% [78.0–83.6] +3.4 pp 79.1% [77.0–81.1]
Exclusion-specific agreement 91.7% [90.6–92.9] 93.4% [92.4–94.4] +1.6 pp 92.6% [91.8–93.3]
Exclusion - retention gap 14.3 pp 12.5 pp -1.8 pp 13.5 pp
INC/MAYBE disagreement rate 5.1% 5.1% +0.0 pp 5.1%
MAYBE annotations / 100 references 30.9 34.5 +3.6 32.7

Guide-version context

Two substantive revisions of the screening guide occurred during the first half of screening. Guide V01 was used for Blocks 01–02. Following recalibration after Block 02, Guide V02 was introduced and used for Blocks 03–05. A second substantive revision following Block 05 resulted in Guide V03, which was used for Blocks 06–10. Other recalibration episodes during this period did not result in a new guide version. Guide-version periods are shown in the longitudinal figures for descriptive context and are not interpreted as experimental intervention periods.

Longitudinal agreement trajectories

Block-level agreement coefficients were examined longitudinally to assess the evolution of reviewer agreement across the first 50% of screening. The prespecified operational thresholds are displayed for binary and nominal Fleiss’ kappa. Gwet’s AC1 is shown as a complementary chance-corrected diagnostic and has no independent progression threshold.

Note. Binary Fleiss’ κ remains the primary operational monitoring statistic. Gwet’s AC1 is complementary and has no independent progression threshold. Vertical lines indicate transitions from Guide V01 to V02 after Block 02 and from Guide V02 to V03 after Block 05.

Decision-specific agreement trajectories

Retention-specific and exclusion-specific agreement were examined separately to determine whether high overall binary agreement was primarily driven by shared exclusion decisions. In accordance with the monitoring plan, retention-specific agreement is also examined at the midpoint as a potential candidate for prospective operational interpretation. No threshold was prespecified for this measure; therefore, its ability to distinguish blocks associated with substantive guide revision from those not requiring such revision is evaluated separately below.

Note. Retention-specific agreement reflects concordance on retaining references, whereas exclusion-specific agreement reflects concordance on exclusion. Vertical lines indicate transitions from Guide V01 to V02 after Block 02 and from Guide V02 to V03 after Block 05. No retention-specific progression threshold was prespecified.

Retention-specific agreement and recalibration outcome

The observation phase specified for retention-specific agreement was completed at Block 10. Blocks that triggered a full recalibration were therefore examined according to the documented outcome of the recalibration meeting. The primary classification follows the broader operational definition of substantive recalibration, including the introduction of numbered operational clarifications or a new Guide version. A sensitivity analysis applies the narrower criterion of whether the meeting resulted in a new Guide version.

Blocks 07 and 10 are not included in this comparison because they met the prespecified progression criteria and did not trigger a full recalibration meeting. Candidate cut-points are exploratory and are not applied retrospectively.

Retention-specific agreement and documented recalibration outcome
Block Guide Binary κ Retention-specific Expanded outcome Guide-version sensitivity
01 Guide V01 0.691 77.3% Substantive recalibration No guide revision
02 Guide V01 0.651 75.4% Substantive recalibration Guide revision
03 Guide V02 0.734 80.4% Reiteration-only recalibration No guide revision
04 Guide V02 0.735 81.0% Reiteration-only recalibration No guide revision
05 Guide V02 0.640 72.3% Substantive recalibration Guide revision
06 Guide V03 0.705 77.5% Reiteration-only recalibration No guide revision
08 Guide V03 0.702 77.6% Reiteration-only recalibration No guide revision
09 Guide V03 0.742 80.7% Reiteration-only recalibration No guide revision
Exploratory evaluation of retention-specific agreement as a candidate monitoring threshold
Scenario Substantive n Reiteration n Substantive range Reiteration range Separation gap Complete separation Candidate cut-point Exact Wilcoxon p
Expanded operational-change definition 3 5 72.3–77.3% 77.5–81.0% +0.2 pp Yes 77.4% 0.036
Guide-version-only sensitivity 2 6 72.3–75.4% 77.3–81.0% +2.0 pp Yes 76.4% 0.071

Note. The dashed line is a data-derived exploratory cut-point rather than a prespecified progression threshold. The exact Wilcoxon test is reported only as a rank-based descriptive diagnostic because the number of recalibration episodes, particularly substantive episodes, is very small. The sensitivity analysis examines whether the apparent separation depends on defining substantive change by Guide-version revision alone.

MAYBE uncertainty profile

MAYBE-note annotations were examined as an operational record of uncertainty encountered during screening. The unit of analysis is the reviewer annotation, not the individual reference. Accordingly, annotation counts should not be interpreted as counts of unique MAYBE references.

Because the Sel-crit category was introduced prospectively beginning with Block 06 and Blocks 01–05 were not retrospectively recoded, the complete category composition cannot be compared directly between the two periods. The principal period comparison therefore uses only categories available throughout Blocks 01–10. Sel-crit is retained separately as a prospective monitoring category.

The Sel-crit category accounted for 89 annotations during Blocks 06–10 (15.2% of the observed MAYBE-note annotations in that period). No corresponding percentage is calculated for Blocks 01–05 because the category was not yet available.

Harmonised comparison

Harmonised MAYBE-note composition across categories coded throughout Blocks 01–10
Category Description Blocks 01–05 Blocks 06–10 Change
Oper operational performance indicator not operationalised 166 (31.7%) 134 (27.1%) -4.6 pp
Gen-task generic test/task versus tactical anchoring 110 (21.0%) 104 (21.0%) +0.0 pp
Ctx-org contextual/organisational performance 96 (18.3%) 91 (18.4%) +0.1 pp
Pop ambiguous population or civilian proxy 65 (12.4%) 98 (19.8%) +7.4 pp
Abs abstract unavailable or incomplete 32 (6.1%) 23 (4.6%) -1.5 pp
Doc uncertain document type 21 (4.0%) 27 (5.5%) +1.4 pp
Admin administrative productivity as a proxy 15 (2.9%) 5 (1.0%) -1.9 pp
Equip personal/corrective equipment 12 (2.3%) 7 (1.4%) -0.9 pp
Rehab recovery/prevention versus performance 7 (1.3%) 6 (1.2%) -0.1 pp

The harmonised comparison removes Sel-crit from both periods so that the denominator is based on the same category framework. Differences are descriptive and may reflect both evolving application of the screening guidance and variation in the references encountered.

Block-level uncertainty composition

Note. Values represent the percentage of categorised MAYBE-note annotations within each block. NC = not coded. Sel-crit was introduced prospectively at Block 06 and was not retrospectively assigned to Blocks 01–05. Vertical lines indicate transitions from Guide V01 to V02 and from Guide V02 to V03.

Exploratory diagnostic associations

A small set of block-level associations was examined to assess whether the complementary diagnostics behaved in an interpretable manner across Blocks 01–10. These analyses are exploratory. With only ten blocks, they are not intended as confirmatory hypothesis tests, and p-values are reported only as descriptive aids.

Several of the agreement measures are derived from the same underlying decision table and are therefore mathematically related. Associations among them must not be interpreted as independent evidence. MAYBE-note associations may additionally reflect differences in block composition.

Preselected exploratory block-level diagnostic associations
Association N Spearman ρ Descriptive p
Retained-vote share vs AC1 - binary κ gap 10 -0.697 0.025
MAYBE annotation burden vs nominal κ 10 +0.697 0.025
MAYBE annotation burden vs retention-specific agreement 10 +0.915 <0.001
Gen-task annotations / 100 references vs binary κ 10 -0.321 0.365

AC1–kappa divergence and decision prevalence

Note. The figure is diagnostic rather than inferential. A larger AC1–κ gap at lower retained-decision prevalence would be consistent with the intended use of AC1 as a complementary diagnostic of the influence of decision distribution on κ. It would not establish AC1 as a superior or more accurate agreement coefficient.