Purpose

This log documents inter-rater reliability monitoring during formal title and abstract screening. Screening is conducted in 20 successive non-overlapping blocks of approximately 5% of the corpus. Each block is assessed using independent pre-resolution decisions only.

Binary Fleiss’ kappa measures agreement on the operational decision to retain or exclude a reference. Nominal Fleiss’ kappa measures agreement across the full INC, MAYBE, and EXC decision framework.

The final PDF is accompanied by this RMarkdown source file and the CSV exports used for each block, providing a complete computational audit trail.

Monitoring rules

Measure Decision structure Threshold Protocol action
Binary Fleiss’ kappa INC/MAYBE retained vs EXC ≥ 0.75 A single block below threshold triggers full recalibration.
Nominal Fleiss’ kappa INC, MAYBE, and EXC ≥ 0.60 A single low block is recorded and monitored; three consecutive low blocks trigger targeted MAYBE recalibration.

Binary kappa is treated as the primary safeguard because it measures agreement on whether a reference advances to full-text screening. Nominal kappa captures whether reviewers apply the full three-category framework consistently, particularly the MAYBE category.

Block 01 | References #1–341 | 0–5% of corpus

Block integrity and residual duplicates

Block 01 contained 341 imported references. Of these, 338 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 3 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 338
Residual duplicates verified in Covidence 3
References included in IRR calculations 338
Residual duplicates removed from Block 01
Covidence # Title
#139 Transforming growth factor-β signaling: Tumorigenesis and targeting for cancer therapy
#439 A Reasonable Officer: Examining the Relationships Among Stress, Training, and Performance in a Highly Realistic Lethal Force Scenario
#39 Corrigendum to ‘Beyond Change Scores: Employing an Improved Statistical Approach to Analyze the Impact of Entry Fitness on Physical Performance During British Army Basic Training in Men and Women’

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 338
Nominal Fleiss’ kappa 0.591
Raw agreement 82.5%
Nominal disagreements 59/338
INC/MAYBE disagreements 18/338

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 338
Binary Fleiss’ kappa 0.691
Raw agreement 87.9%
Binary conflicts 41/338
Conflict rate (95% Wilson CI) 12.1% (9.1–16.0%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 227 67.2%
Conflicting retention decision (EXC / INC or MAYBE) 41 12.1%
Retained by both reviewers (INC and/or MAYBE) 70 20.7%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 26 37.1%
INC / MAYBE 18 25.7%
MAYBE / MAYBE 26 37.1%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 338
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.691
Nominal Fleiss’ kappa 0.591
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)

Block 02 | References #342–682 | 5–10% of corpus

Block integrity and residual duplicates

Block 01 contained 341 imported references. Of these, 339 had exactly two independent pre-resolution decisions and were included in the inter-rater reliability analyses. The remaining 2 references had no recorded votes. Verification in Covidence confirmed that these records were residual duplicates removed during screening.

Block integrity summary
Metric N
References imported 341
References with exactly two independent votes 339
Residual duplicates verified in Covidence 2
References included in IRR calculations 339
Residual duplicates removed from Block 02
Covidence # Title
#643 Bell 2015 : Effects of 13-Hour 20-Minute Work Shifts on Law Enforcement Officers’ Sleep, Cognitive Abilities, Health, Quality of Life, and Work Performance: The Phoenix Study
#768 Bishop 1991: Limitations to heavy work at 21 degrees C of personnel wearing the U.S. military chemical defense ensemble

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 339
Nominal Fleiss’ kappa 0.587
Raw agreement 81.1%
Nominal disagreements 64/339
INC/MAYBE disagreements 15/339

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 339
Binary Fleiss’ kappa 0.651
Raw agreement 85.5%
Binary conflicts 49/339
Conflict rate (95% Wilson CI) 14.5% (11.1–18.6%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Pre-resolution decision distribution

The table below reports reference-level pre-resolution decision patterns among records with exactly two independent votes. A reference was classified as retained where both reviewers selected either INC or MAYBE. These results do not represent final post-arbitration outcomes.

Reference-level pre-resolution decision patterns
Pattern N Percentage
Excluded by both reviewers (EXC / EXC) 215 63.4%
Conflicting retention decision (EXC / INC or MAYBE) 49 14.5%
Retained by both reviewers (INC and/or MAYBE) 75 22.1%

Among references retained by both reviewers, the following nominal decision patterns were observed:

Nominal patterns among references retained by both reviewers
Pattern N Percentage_of_retained
INC / INC 31 41.3%
INC / MAYBE 15 20.0%
MAYBE / MAYBE 29 38.7%

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 339
References with zero votes 2
References excluded from IRR calculation 2
Binary Fleiss’ kappa 0.651
Nominal Fleiss’ kappa 0.587
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)