Common Contracts

Nicolas Stefanovitch

November 10th, 2023

• Filed
  November 10th, 2023

In this paper we report on the complexity of persuasion technique annotation in the context of a large multilingual annotation campaign involving 6 languages and approximately 40 annotators. We highlight the techniques that appear to be difficult for humans to annotate and elaborate on our findings on the causes of this phenomenon. We introduce Holistic IAA, a new word embedding-based annotator agreement metric and we report on various ex- periments using this metric and its correlation with the traditional Inter Annotator Agreement (IAA) metrics. However, given somewhat lim- ited and loose interaction between annotators, i.e., only a few annotators annotate the same document subsets, we try to devise a way to assess the coherence of the entire dataset and strive to find a good proxy for IAA between an- notators tasked to annotate different documents and in different languages, for which classical IAA metrics can not be applied.