Predicting word sense annotation agreementWord Sense Annotation Agreement • August 13th, 2015
High agreement is a common objective when annotating data for word senses. However, a number of factors make per- fect agreement impossible, e.g. the limi- tations of sense inventories, the difficulty of the examples or the interpretation pre- ferences of the annotators. Estimating po- tential agreement is thus a relevant task to supplement the evaluation of sense an- notations. In this article we propose two methods to predict agreement on word- annotation instances. We experiment with a continuous representation and a three- way discretization of observed agreement. In spite of the difficulty of the task, we find that different levels of agreement can be identified—in particular, low-agreement examples are easier to identify.
