An R tool for assessing inter-rater reliability

Background

• I was doing a scoping review about the spatial patterning of emergency calls to the police.
• I wanted to use two raters to assess reliability.
• We each independently uploaded the same RIS file to ASReview.
• We each used the same screening criteria.
• Result: two .csv files from ASReview.
• …what now?

Development

• The tool began as a single-use R script.
• Based on input from colleagues, it slowly developed features and became a report.
• Software and stats input from Maas van Steenbergen.
• Now it is a (more or less) reusable tool.

Demo

Source material on GitHub.

Example report available on rpubs.

Thanks for listening!