Purpose

This log documents inter-rater reliability monitoring during formal title and abstract screening. Screening is conducted in 20 successive non-overlapping blocks of approximately 5% of the corpus. Each block is assessed using independent pre-resolution decisions only.

Binary Fleiss’ kappa measures agreement on the operational decision to retain or exclude a reference. Nominal Fleiss’ kappa measures agreement across the full INC, MAYBE, and EXC decision framework.

The final PDF is accompanied by this RMarkdown source file and the CSV exports used for each block, providing a complete computational audit trail.

Monitoring rules

Measure Decision structure Threshold Protocol action
Binary Fleiss’ kappa INC/MAYBE retained vs EXC ≥ 0.75 A single block below threshold triggers full recalibration.
Nominal Fleiss’ kappa INC, MAYBE, and EXC ≥ 0.60 A single low block is recorded and monitored; three consecutive low blocks trigger targeted MAYBE recalibration.

Binary kappa is treated as the primary safeguard because it measures agreement on whether a reference advances to full-text screening. Nominal kappa captures whether reviewers apply the full three-category framework consistently, particularly the MAYBE category.

Block 01 | References #1–341 | 0–5% of corpus

Nominal kappa

The nominal analysis treats INC, MAYBE, and EXC as three distinct categories. It therefore assesses agreement in the use of MAYBE as well as agreement on inclusion and exclusion decisions.

Nominal agreement results
Metric Result
References analysed 338
Nominal Fleiss’ kappa 0.591
Raw agreement 82.5%
Nominal disagreements 59/338
INC/MAYBE disagreements 18/338

Interpretation: Below the pre-specified threshold; the use of MAYBE decisions is monitored.

Code used

# Three-category agreement
nominal_pairs <- compact_to_two_raters(block_decisions)

kappa_nominal <- irr::kappam.fleiss(
  nominal_pairs,
  exact = FALSE
)$value

Binary kappa

The binary analysis collapses INC and MAYBE into a retained category, contrasted with EXC. It assesses agreement on whether each reference advances to full-text screening.

Binary agreement results
Metric Result
References analysed 338
Binary Fleiss’ kappa 0.691
Raw agreement 87.9%
Binary conflicts 41/338
Conflict rate (95% Wilson CI) 12.1% (9.1–16.0%)

Interpretation: Below the pre-specified threshold; full recalibration is required.

Code used

# Binary recode: EXC = 0; INC and MAYBE = 1
binary_pairs <- nominal_pairs
binary_pairs[binary_pairs %in% c(1, 2)] <- 1
binary_pairs[binary_pairs == 0] <- 0

kappa_binary <- irr::kappam.fleiss(
  binary_pairs,
  exact = FALSE
)$value

Block summary and protocol decision

Block-level monitoring summary
Metric Result
References imported 341
References with exactly two independent votes 338
References with zero votes 3
References excluded from IRR calculation 3
Binary Fleiss’ kappa 0.691
Nominal Fleiss’ kappa 0.591
Preceding low nominal-kappa blocks 0
Protocol action Full recalibration

Decision: Full recalibration required before the next block.

Rationale: Binary agreement fell below the pre-specified threshold of 0.75, indicating insufficient agreement on the retain-versus-exclude decision.

Code used

decision <- protocol_decision(
  binary_kappa = kappa_binary,
  nominal_kappa = kappa_nominal,
  prior_low_nominal_blocks = PRIOR_CONSECUTIVE_LOW_NOMINAL_BLOCKS
)