Group of Relations Level (krr. This statistic suggests measuring the hierarchical rhetorical relation annotations when the relations are grouped according to their rhetorical similarity. Xxxx et al. (2003) presented agreement results for all of Xxxxx et al.’s (1999) inter- annotator agreement statistics. There were two kinds of results: in the first result set, the evolution of agreement among raters by time was illustrated, and in the second result set, the final annotation agreements in various corpus subset annotations that were performed by different annotator pairs were presented. Both of the results were the agreements among two to three annotators for about 30 documents. As mentioned before, Xxx et al. (2003) preferred to use pre-segmented elementary discourse units during their final annotation, except the training session annotations and the illustrative final annotation. They presented agreement results for unit levels (for pre-segmented and not pre-segmented elementary discourse units). However, they did not indicate whether they had used kw or ku to calculate. For the spans level, nuclearity level, relation level and group of relations level they closely followed Xxxxx et al. (1999). In Xxxx et al’s (2003) measurements, the inter-annotator agreement that changed among time indicates that there were 0.10 to 0.15 increase during the annotation for all levels. Unit level agreement results that were performed on not pre-segmented text increased from 0.87 to 0.97. For pre-segmented data, unit level agreement was expected 1.00, but the measurements are between 0.95 and 1.00. The reason of the lesser agreement than expected agreement was explained as the annotators’ misusing of the annotation tool. As seen, exterior or human factors may change the reliability of annotated corpus even if the task was clearly defined. At the spans level, agreement increased from 0.77 to 0.89. At the nuclearity level, agreement increased from 0.70 to 0.85. At the relation level, agreement increased from 0.60 to 0.75. Finally, agreement increased from 0.64 to 0.78 at the group of relations level. These results indicate that the reliability of this corpus can be sustained for all levels (unit, spans, nuclearity, relation, and group of relations) by training annotators. Another important inference from the results is that annotating grouped relations or annotating individual relations does not significantly affect the reliability of the annotations. The inter-annotator agreements of the final annotations...
