Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

Maya M. Jeyaraman, Reid C. Robson, Leslie Copstein, Nameer Al-Yousif, Michelle Pollock, Jun Xia, Chakrapani Balijepalli, Kimberly Hofer, Samer Mansour, Mir S. Fazeli, Mohammed T. Ansari, Andrea C. Tricco, Rasheda Rabbani, Ahmed M. Abou-Setta

Research output: Journal PublicationArticlepeer-review

Abstract

Objectives: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). Study design and Setting: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. Results: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. Conclusion: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

Original languageEnglish
Pages (from-to)157-167
Number of pages11
JournalJournal of Clinical Epidemiology
Volume136
DOIs
Publication statusPublished - Aug 2021

Keywords

  • Evaluator burden
  • Inter-consensus reliability
  • Inter-rater reliability
  • Non-randomized studies
  • RoB instrument for NRS of exposures
  • ROBINS-I

ASJC Scopus subject areas

  • Epidemiology

Fingerprint

Dive into the research topics of 'Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies'. Together they form a unique fingerprint.

Cite this