Xxxxx’x κ. We can also consider the case where an estimate of the IAA matrix M is not available and we only have access to a scalar representation of the inter-annotator agreement like Xxxxx’x κ. In this case we can only estimate one parameter and hence the matrix T has to be parameterized by a single parameter that can be estimated. n One particular example is the case where the noise is uniform among classes. Under these hypotheses, T is a ma- trix with all values 1 − p on the diagonal and p off the dominant, the exact details about the generation of T can be found in Appendix D.1. For each annotator we produce their prediction according to the matrix T . We run exper- iments for the number of annotators H = 10, 7, 3, 2. We report here the results for H = 10, and 4 classes, all the other plots are in D.1. In Fig. 4 (as well as the the plots in the appendix) we can be observed that the error in the esti- mation decreases as √1 with n number of samples, which is in agreement with the bound provided in Theorem 4.3. We also observed that, as expected, the estimation becomes diagonal. C−1 more accurate as the number of annotators increases. Lemma 5.1 (Relationship between p and κ). In the case of classification with uniform noise for two homogeneous annotators with noise rate p, i.e if a is one annotator, P(ya = i|y = j) = p if i j. If the distribution of the ground-truth labels is uniform, it holds that: p = (1 − C−1)(1 − √κ) (22) with κ the Xxxxx’x kappa coefficient of the two annotators (see Appendix A). Proof. The proof can be found in Appendix C.6. − − −
Xxxxx’x κ a Proposed by Xxxxx (1960), κ characterizes chance agreement as individual distributions per coder, calculated as shown in Equations 9-10 using our definition of agreement (Aπ) as shown earlier.
Xxxxx’x κ. Xxxxx’x κ coefficient measures the concordance between two judges classifications of m elements into k mutually exclusive categories. Xxxxx defined the coefficient as “the proportion of chance-expected disagreements which do not occur, or alternatively, it is the proportion of agreement after chance agreement is removed from consideration” [5]. The coefficient is defined as κ = Po − Pc , 1 − Pc where Po is the proportion of units for which the judges agreed (relative observed agreement among raters) and Pc is the proportion of units for which agreement is expected by chance (chance-expected agreement). In order to compute these proportions, we will use the so-called contingency matrix, as shown in Table 3. This is a square matrix of order the number of categories k. The (i, j)-entry, denoted ci,j, is the number of times that an item was assigned to the i-th category by judge J1 and to the j-th category by judge J2. In this way, the elements of the form ci,i are precisely the agreements in the evaluations. J1 Category 2 c1,2 c2,2 · · · · · ·
Xxxxx’x κ. In [13] Xxxxxxxxxx describes two inter-annotator agreement metrics: Xxxxx’x κ [12] and original Krippendorff α [14]. κ = pAO − pAE 1 − pAE α = 1 pDO pDE − κ is defined as the difference of observed agreement (pAO ) and chance agree- ment (pAE ) while α involves the observed non-agreement (pDO ) and chance non-agreement (pDE ). Passonneau shows that α = κ, so in the further part of the text we concentrate on the procedure of calculating κ. Annotator A Label X Label Y Σ Annotator B Label X 47 14 61 Label Y 10 29 39 Σ 57 43 100 Table 2. A sample coincidence matrix α and κ can be calculated when coincidence matrix is available for multi- ple annotators’ decisions (with data how frequent each of the annotators were choosing each label). For example, when it is defined as in Table 2, we can calculate: pAO = 47 + 29 = 0.76 100 57 61 43 39 pAE = 100 ∗ 100 + 100 ∗ 100 = 0.5154 ∗ Observed agreement shows diagonally in the matrix. Expected agreement is based on the probability of selection of each label by each annotator. For instance, when annotator A selected label X in 57% decisions and annotator B in 61% decisions, the chance that they accidentally chose the same label A is (57/100) (61/100). The sum of probabilities for all labels gives the expected agreement. Finally for Table 2 we have: κ = pAO − pAE 1 − pAE = 0.5 For the coreference annotation agreement assessment, crucial decision is to choose how to represent coreference annotation in coincidence matrix similar to Table 2. We present some approaches in the following sections. For the reason described in the near-identity section, we suggest calculating the agreement for each text separately and then averaging it.
