Traditional IAA Sample Clauses

Traditional IAA. We measured the Inter-Annotator Agreement (IAA) using Krippendorff’s α, achieving a value of 0.342. This is lower than the recommended threshold of 0.667, but we should note that this value repre- sents the agreement level before curation, and as such, it is more representative of the curation dif- ficulty rather than of the quality of the final con- solidated annotations. We used the IAA during the campaign to allocate curation roles and to remove low-performing annotators. We further studied the IAA by ranking the an- notators by their performance with respect to the ground truth on the subset of documents they xxxx- xxxxx. We split then the annotators into two groups: top and low based on subjective assessment by the curators after the end of the curation campaign, this assessment was then further confirmed numerically (see Annex E for details). Their respective average α were 0.415 and 0.250. Finally, we considered the α of the group of the curators, in order to make an approximate estimation of the coherence of the curated dataset, as we expect these curators to con- sistently curate the data with at least the same coher- ence they had when annotating documents. There are only two such curators, whose α is of 0.588, which is lower but close to the recommended value.