Common Contracts
1 similar Word-Sense Distinguishability and Inter-Coder Agreement contracts
yDepartment of Computer ScienceWord-Sense Distinguishability and Inter-Coder Agreement • November 4th, 2007
It is common in NLP that the categories into which text is classi ed do not have fully objective def- initions. Examples of such categories are lexical distinctions such as part-of-speech tags and word- sense distinctions, sentence level distinctions such as phrase attachment, and discourse level distinc- tions such as topic or speech-act categorization. This paper presents an approach to analyzing the agreement among human judges for the purpose of formulating a re ned and more reliable set of category designations. We use these techniques to analyze the sense tags assigned by ve judges to the noun interest. The initial tag set is taken from Longman's Dictionary of Contemporary English. Through this process of analysis, we automatically identify and assign a revised set of sense tags for the data. The revised tags exhibit high reliabil- ity as measured by Cohen's . Such techniques are important for formulating and evaluating both human and automated classi cation systems.
