Multiwords (WP6 Sample Clauses

Multiwords (WP6. 1) Evaluation of MWE automatic extraction methods is often done comparing the results against manually compiled lexica of MWEs, or against existing ontologies or terminologies. Such resources are taken as gold standards. Another frequent and often complementary evaluation approach is through human/native-speaker judgments. Here only a few examples taken from recent works in MWE extraction will be reported. Xxxxxx et al. (2010) used the Cambridge International Dictionary of English and the Cambridge International Dictionary of Phrasal Verbs. After the comparison, human judges are asked to analyze the results, mainly to see if MWE not found in reference dictionaries should be included in the final MWE list. Interestingly, inter-annotator agreement in this kind of task is normally quite low (cf. Xxxxxx 2010, Xxxxx and Xxxxxxx 2002). This may be a reflection of the fact that the notion of MWEs and collocation is quite subjective, domain-specific, and also somewhat vague (cfr. Xxxxxx 2010: 141). Xxxxxxx and Xxxxxx (2009:80) performed evaluation by comparing the accuracy of a baseline system (a system applying a window-based method) with the accuracy of their syntactic-based system (i.e. deriving candidate from a deep parsing output) at difference levels of significance. Evaluation was done by sampling a (small) subset of MW (pairs) extracted at the different significance levels (altogether their test set consists of 250 pair per language) and have 2 humans judge each pair (in its original context) on a 5-point scale (i.e. ungrammatical, regular combination, named entity, collocation, compound, idiom). They report a satisfactory K value for inter-annotator agreement (k=68.7) and demonstrate the better performance of the syntactic-method. The reported precision at 0 and 3 levels of confidence are: English=42-58, Spanish=39-42, French=46-35, Italian=32-37. Xxxxxxx et al. (2010) use the Xxxxx ontology as a gold standard and use common precision, recall and F- measure to measure the performance of their MW terms identification methods. Using a frequency threshold of 5 (i.e., considering only candidate that occur at least 5 times in the corpus), precision is good, 74,14%, but recall is quite low, 6,42%. Changing the value of the threshold, of course, precision and recall change considerably. As the authors notice, if a higher recall is needed (for instance, if the aim is the creation of a terminological dictionary), then the threshold can be lowered to 1, thus obtain...