Inter Annotator Agreement. For most tasks, Xxxxx’x Kappa is reported as a measure of IAA, and is consid- ered the standard measure (XxXxxx, 2012). But for Named Entity Recognition, Kappa is not the most relevant measure, as noted in multiple studies (Xxxxxxxx & Xxxxxxxxxx, 2005; Xxxxxx et al., 2011). This is because Kappa needs the num- ber of negative cases, which isn’t known for named entities. There is no known number of items to consider when annotating entities, as they are a sequence of tokens. A solution is to calculate the Kappa on the token level, but this has two associated problems. Firstly, annotators do not annotate words individually, but look at sequences of one or more tokens, so this method does not reflect the annotation task very well. Secondly, the data is extremely unbalanced, with the un-annotated tokens (labelled "O") vastly outnumbering the actual entities, un- fairly increasing the Kappa score. A solution is to only calculate the Kappa for tokens where at least one annotator has made an annotation, but this tends to underestimate the IAA. Because of these issues, the pairwise F1 score calculated without the O label is usually seen as a better measure for IAA in Named Entity 42 CHAPTER 3. DATA SET Xxxxx’x Kappa on all tokens 0.82 Xxxxx’x Kappa on annotated tokens only 0.67 F1 score 0.95 Table 3.4: Inter-annotator agreement measures on 100 sentence test document. Calculated by doing pairwise comparisons between all combinations of annotators and averaging the results. Recognition (Xxxxxxx et al., 2012). However, as the token level Kappa scores can also provide some insight, we provide all three measures but focus on the F1 score. The scores are provided in Table 3.4. These scores are calculated by averaging the results of pairwise comparisons across all annotators. We also cal- culated these scores by comparing all the annotators against the annotations we did ourselves, and obtained the same F1 score and slightly lower Kappa (-0.02).
Inter Annotator Agreement. ON ANNOTATION EFFORT OF XXXX ET AL. (2003) Xxxx et al. (2003) used Xxxxx et al.’s (1999) kappa statistic methodologies to measure various aspects of the inter-annotator agreement on their RST based corpus. Five topics were presented to fully cover the typical agreement issue of those kinds of corpora. The first topic deals with unit segmentation and the rest of them suggest methodologies for the issues emerging with the hierarchical structure of the corpora. Essentially, in all the methodologies for hierarchical aspects, hierarchical structure was flattened to a linear table by considering each possible segment pairs as units which constitute the source data to compute the kappa statistic. The following is a suitable example, which is a modified portion of a sample annotation from the study of Xxxxx et al. (1999), to clarify the claim above. In Figure 4, there are two nuclearity segmentation examples for two levels that represent two hierarchical discourse structures of the same text: Segmentation 1 N S N S 1 0 Segmentation 2 As a result of flattening, the following data table is constructed from the discourse structure above: [0,0] none N [0,1] N N [0,2] N None [1,1] none S [1,2] none None [2,2] S S The constructed agreement table is used as the input to the kappa statistic. For this sample the attributes of the kappa statistic are 2 annotators (Segmentation 1, Segmentation 2), 3 categories (N, S, none), and 9 samples (segment pairs). In the light of this explanation, five inter-annotator agreement aspects are as follows:
1. Unit Level (kw and ku): Xxxxx et al. (1999) present two kinds of kappa statistics to measure agreement on elementary discourse units which are calculated considering two different approaches. In the first case (kw), it is assumed that the unit boundaries can be the end of any word. The second case (ku) suggests taking the unit boundaries as the locations that at least one annotator annotated as boundary. The two approaches have different chance factors because units and unit numbers those are included in measurement changes. The change of chance factor directly affects the results. In Xxxxx et al.’s (1999) sample corpus, measurements of kw are around 0.90 while ku measurements are around 0.75. This is a nice example that illustrates that the results depend on not only on the selected statistical methodologies but also on their application manner.
2. Spans Level (ks): This statistic suggests measuring the hierarchical discourse segment...
Inter Annotator Agreement. Table 3.3 show the overall statistics of the FriendsQA dataset. There is a total of 1,222 dialogues, 10,610 questions, and 21,262 answer spans in this dataset after pruning (Section 3.7). There are at least 2 answers to each question since there are 2 phases during annotation, each of which will acquire an answer to the same question. Note that annotators were not asked to paraphrase questions during the second phase of the first round (R1 in Table 3.3), so the number of questions in R1 is about twice less than ones from the other rounds. The final inter-annotator agreement scores are 81.82% and 53.55% for the F1 and exact matching scores respectively, indicating high-quality annotation in our dataset.
Inter Annotator Agreement. The need to ascertain the agreement and reliabil- ity between coders for segmentation was recognized 3Georgescul et al. (2006, p. 48) note that both FPs and FNs are weighted by 1/N−k, and although there are “equiprobable possibilities to have a [FP] in an interval of k units”, “the total number of equiprobable possibilities to have a [FN] in an inter- val of k units is smaller than (N k)”, making the interpretation of a full miss as a FN less probable than as a FP. by Passonneau and Xxxxxx (1993), who adapted the percentage agreement metric by Xxxx et al. (1992,
Inter Annotator Agreement. Similarity alone is not a sufficiently insightful mea- sure of reliability, or agreement, between coders.
