TY - JOUR
T1 - Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies
AU - Jeyaraman, Maya M.
AU - Robson, Reid C.
AU - Copstein, Leslie
AU - Al-Yousif, Nameer
AU - Pollock, Michelle
AU - Xia, Jun
AU - Balijepalli, Chakrapani
AU - Hofer, Kimberly
AU - Mansour, Samer
AU - Fazeli, Mir S.
AU - Ansari, Mohammed T.
AU - Tricco, Andrea C.
AU - Rabbani, Rasheda
AU - Abou-Setta, Ahmed M.
N1 - Publisher Copyright:
© 2021
PY - 2021/8
Y1 - 2021/8
N2 - Objectives: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). Study design and Setting: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. Results: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. Conclusion: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.
AB - Objectives: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). Study design and Setting: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. Results: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. Conclusion: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.
KW - Evaluator burden
KW - Inter-consensus reliability
KW - Inter-rater reliability
KW - Non-randomized studies
KW - ROBINS-I
KW - RoB instrument for NRS of exposures
UR - http://www.scopus.com/inward/record.url?scp=85107594850&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2021.04.017
DO - 10.1016/j.jclinepi.2021.04.017
M3 - Article
C2 - 33979663
AN - SCOPUS:85107594850
SN - 0895-4356
VL - 136
SP - 157
EP - 167
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -