Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden

Maya M. Jeyaraman, Rasheda Rabbani, Leslie Copstein, Reid C. Robson, Nameer Al-Yousif, Michelle Pollock, Jun Xia, Chakrapani Balijepalli, Kimberly Hofer, Samer Mansour, Mir S. Fazeli, Mohammed T. Ansari, Andrea C. Tricco, Ahmed M. Abou-Setta

Research output: Journal PublicationArticlepeer-review

8 Citations (Scopus)

Abstract

Objective: To assess the real-world interrater reliability (IRR), interconsensus reliability (ICR), and evaluator burden of the Risk of Bias (RoB) in Nonrandomized Studies (NRS) of Interventions (ROBINS-I), and the ROB Instrument for NRS of Exposures (ROB-NRSE) tools. Study Design and Setting: A six-center cross-sectional study with seven reviewers (2 reviewer pairs) assessing the RoB using ROBINS-I (n = 44 NRS) or ROB-NRSE (n = 44 NRS). We used Gwet's AC1 statistic to calculate the IRR and ICR. To measure the evaluator burden, we assessed the total time taken to apply the tool and reach a consensus. Results: For ROBINS-I, both IRR and ICR for individual domains ranged from poor to substantial agreement. IRR and ICR on overall RoB were poor. The evaluator burden was 48.45 min (95% CI 45.61 to 51.29). For ROB-NRSE, the IRR and ICR for the majority of domains were poor, while the rest ranged from fair to perfect agreement. IRR and ICR on overall RoB were slight and poor, respectively. The evaluator burden was 36.98 min (95% CI 34.80 to 39.16). Conclusions: We found both tools to have low reliability, although ROBINS-I was slightly higher. Measures to increase agreement between raters (e.g., detailed training, supportive guidance material) may improve reliability and decrease evaluator burden.

Original languageEnglish
Pages (from-to)140-147
Number of pages8
JournalJournal of Clinical Epidemiology
Volume128
DOIs
Publication statusPublished - Dec 2020

Keywords

  • Evaluator burden
  • Interconsensus reliability
  • Interrater reliability
  • Nonrandomized studies
  • RoB instrument for NRS of exposures
  • ROBINS-I

ASJC Scopus subject areas

  • Epidemiology

Fingerprint

Dive into the research topics of 'Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden'. Together they form a unique fingerprint.

Cite this