Common Contracts
1 similar Inter-Annotator Agreement contracts
LING83800Inter-Annotator Agreement • June 18th, 2024
When our models consist of supervised machine learning trained on human annotations, agree- ment between multiple human annotators imposes a upper bound for performance, if only be- cause it is difficult if not impossible to measure system performance exceeding that of human annotators. And, when human annotators do not show substantial agreement, this suggests the annotation task may be ill-defined, or too challenging.
