Interrater agreement statistics under the two-rater dichotomous-response case with correlated decisionsInterrater Agreement Statistics • February 14th, 2024
Measurement of the interrater agreement (IRA) is critical in various disciplines. To correct for potential confounding chance agreement in IRA, Cohen’s κ and many other methods have been proposed. However, owing to the varied strategies and assumptions across these meth- ods, there is a lack of practical guidelines on how these methods should be preferred even for the common two-rater dichotomous rating. To fill the gaps in the literature, we system- atically review nine IRA methods and propose a generalized framework that can simulate the correlated decision processes behind the two raters to compare those reviewed meth- ods under comprehensive practical scenarios. Based on the new framework, an estimand of “true” chance-corrected IRA is defined by accounting for the “probabilistic certainty” and serves as the comparison benchmark. We carry out extensive simulations to evaluate the performance of the reviewed IRA measures, and an agglomerative hierarchical clustering analysis is condu
