Common use of Overall estimate of inter-annotator agreement Clause in Contracts

Overall estimate of inter-annotator agreement. As a first overall estimate of the inter-annotator agreement, Table 1 shows for each annotator, the mean absolute shift of the boundaries computed on all the phones of the 18 sentences. It corresponds to about 520 boundaries per annotator. We can observe a fairly good overall agreement between the annotators and the expert. Therefore the annotation of the IFCASL corpus can be used to develop and assess new automatic segmentation tools. However, our results show that it will not be possible to require an automatic boundary accuracy better than ± 10 ms. As the threshold of 20 ms is commonly used to compare the performance of human and automatic labelers, we computed the percentage of labels whose boundaries are shifted by less than 20 ms with respect to the boundaries set by the expert annotator. The average percentage of 93%, with a confidence interval at the 95% confidence level of ± 2.2%, corresponds well with the results reported by Xxxxx for native speech [3]. On the one hand, the agreement may have been slightly facilitated in some cases by the fact that annotators had started from the automatic alignment. However, the annotators were instructed to adjust any incorrect boundaries and place them as precisely as possible. But, on the other hand, the task was more complex because of the non-native speech. Table 1. Shifts of boundaries for each annotator. Annotator Mean absolute shift (ms) Shift <= 20ms #1 7.1 94.1% #2 9.1 90.1% #3 7.6 93.3% #4 6.7 95.3% #5 8.0 93.2% #6 7.4 91.9% #7 7.6 92.9% all 7.6 93.0%

Overall estimate of inter-annotator agreement. As a first overall estimate of the inter-annotator agreement, Table 1 shows for each annotator, the mean absolute shift of the boundaries computed on all the phones of the 18 sentences. It corresponds to about 520 boundaries per annotator. We can observe a fairly good overall agreement between the annotators and the expert. Therefore the annotation of the IFCASL corpus can be used to develop and assess new automatic segmentation tools. However, our results show that it will not be possible to require an automatic boundary accuracy better than ± 10 ms. As the threshold of 20 ms is commonly used to compare the performance of human and automatic labelers, we computed the percentage of labels whose boundaries are shifted by less than 20 ms with respect to the boundaries set by the expert annotator. The average percentage of 93%, with a confidence interval at the 95% confidence level of ± 2.2%, corresponds well with the results reported by Xxxxx Hosom for native speech [3]. On the one hand, the agreement may have been slightly facilitated in some cases by the fact that annotators had started from the automatic alignment. However, the annotators were instructed to adjust any incorrect boundaries and place them as precisely as possible. But, on the other hand, the task was more complex because of the non-native speech. Table 1. Shifts of boundaries for each annotator. Annotator Mean absolute shift (ms) Shift <= 20ms #1 7.1 94.1% #2 9.1 90.1% #3 7.6 93.3% #4 6.7 95.3% #5 8.0 93.2% #6 7.4 91.9% #7 7.6 92.9% all 7.6 93.0%