KAPPA AND SIMILAR AGREEMENT COEFFICIENTS Sample Clauses

KAPPA AND SIMILAR AGREEMENT COEFFICIENTS. The Kappa agreement coefficient was presented to computational linguists and cognitive scientist who study discourse by Xxxx Xxxxxxxx in 1996. However, the first kappa implementation was introduced by Xxxxx (1960), long before Xxxxxxxx (1996). Xxxxx’x Kappa (1960) can be used to measure the degree of agreement, by correcting the agreement by chance between two annotators, where each annotator annotates subjects on a nominal scale (Fleiss, 1971). Another version of kappa by Xxxxx is Xxxxx’x Weighted Kappa (1968), which is very similar to the normal kappa, but additionally weights the different disagreement types. Xxxxx’x Kappa is a member of a chance-corrected agreement measurement family that measures agreement between two annotators, which were mostly introduced in the 1950s and 60s. The most known three examples are ς (Xxxxxx, 1954), g (Xxxxx, 1955), and κ (Xxxxx, 1960) (Xxxxxxxx, & Poesio 2008). All of them have the following formula: ç, g, u = A0– Ae 1–Ae (Equation 2.5) Where, Aç = Ag = Au = Σ P(k|i ) P(k|i ) e e e k<K 1 2 The difference between ς, π, and κ lies in the calculation of P(k|i), where P(k|i) is the probability that annotator i will assign an arbitrary item to category k (Xxxxx, 1988; Xxx, & Xxxxx, 2003; Xxxxxxxx, & Poesio 2008). If annotators had made random picks: • ς: Assumes uniform distribution. For any two annotator im, in, and any two categories kj, kl, P(kj|in) = P(kS|in) • π: Assumes same distribution for each annotator. For any two annotators im, in, and any category k, P(k|in) = P(k|in) • κ: Assumes separate distribution for each annotator. The above statements lead to the following chance agreement formulas (Xxxxxxxx, & Poesio 2008): Aç = Σk<K 1 = 1 (Equation 2.6) e Ag = Σ k2 k Σ n2 e k<K (2K) = 4K2 k<X x (Equation 2.7) Au = Σ ni1k . ni2k = 1 Σ n n (Equation 2.8) e k<K K K K2 k<K i1k ik Ag ≤ Aç and Ag ≤ Ak correlations are extracted from the above formulas, and the e e e e relation of Aç and Xx cannot be exactly extracted. In order to stay on the safe side, the