Common Contracts
1 similar null contracts
Inter-annotator Agreement: By Hook or by CrookAugust 21st, 2018
Through an extended case study, this paper reveals the metaphorical skeletons hidden in statistical cupboards of selective reporting, casting a new light on inter-annotator agreement (IAA) measures. Strategic decisions and their impacts on IAA were tracked in an extended corpus study of rhetorical functions in scientific research abstracts. A search of the research notes of the principal investigator resulted in 142 notes tagged with #IAA that were written between 2013 and 2017. The strategic decisions and their actual or perceived impacts on IAA were logged. A root cause analysis was also conducted to identify the causal factors that reduce IAA. The results show numerous strategic decisions, which using template analysis, were grouped into three categories, namely methodological, statistical and rhetorical. High IAA may be attributed to sound or cogent methodological choices, but it could also be due to manipulating the statistical smoke and rhetorical mirrors. With no standardized co
