Common Contracts
1 similar Inter-Annotator Agreement (Iaa) contracts
Agreement is overrated:Inter-Annotator Agreement (Iaa) • September 21st, 2019
Inter-Annotator Agreement (IAA) is used as a means of assessing the quality of NLG evalu- ation data, in particular, its reliability. Accord- ing to existing scales of IAA interpretation – see, for example, Lommel et al. (2014), Liu et al. (2016), Sedoc et al. (2018) and Amidei et al. (2018a) – most data collected for NLG evaluation fail the reliability test. We con- firmed this trend by analysing papers pub- lished over the last 10 years in NLG-specific conferences (in total 135 papers that included some sort of human evaluation study). Follow- ing Sampson and Babarczy (2008), Lommel et al. (2014), Joshi et al. (2016) and Amidei et al. (2018b), such phenomena can be ex- plained in terms of irreducible human lan- guage variability. Using three case studies, we show the limits of considering IAA as the only criterion for checking evaluation relia- bility. Given human language variability, we propose that for human evaluation of NLG, correlation coefficients and agreement coeffi- cients
