Amazon’s Mechanical Turk. Amazon’s Mechanical Turk is recognised as a promising platform for crowdsourcing. Although requesters have full control over how much a worker is paid on completing a task, many of them seem to pay between $0.01 and $0.10 for a task taking “a few minutes”. It is much less than the MTurk suggested amount of at least the equivalent to the minimum federal wage of $8 per hour or $0.13 per minute (Downs et al., 2010). Interestingly, Xxxxx & Xxxxx (2009) investigated the relationship between financial incentives and the performance of AMT workers by varying the amount of payment. They found that increased financial incentives improved the quantity, but not the quality, of work performed by participants. It was explained that workers who were paid more were no more motivated than workers paid less, because they perceived their work to be more valuable. Xxxx et al. (2009) also found in their experiment that paying $0.10 per task generated lower quality of data than paying $0.05. In order to obtain reliable results, quality control is very important when using Amazon Mechanical Turk, as such monetary incentive based crowdsourcing services are the most susceptible to fraud (Xxxxx et al., 2009; Xxxxx, 2011). In order to obtain reliable results, quality control is important (Kazai, 2011). Collecting multiple assessments for each task is probably the most popular strategy to control quality and has been used extensively in almost every experiment. Multiple assessments can be simply aggregated as an average (Xxxxxx & Xxxxxxx, 2009; Xxxx & Xxxxxxxxx, 2010), voting scheme or weighted sum as suggested by Xxxxxx & Xxxx (2008). Xxxx et al. (2009) found that seven crowdsourced workers per task seemed to be a reasonable number for their experiment. They utilised a two-phase framework for acquiring high quality data from AMT. In the first phase, validation, a small number of tasks were sent to AMT to find malicious workers (via inter-person agreement), the optimal number of workers per task, and the optimal pay rate. The second phase, large-scale submission phase, submitted all tasks using the parameters from the first phase. Xxxxxxx et al. (2008) tried to encourage the participants to follow the task protocol by injecting gold standard into the image annotation process. If the annotations provided deviated significantly from the gold standard, the gold standard would be shown to the participants to warn them after the task was submitted. Xxxx & Xxxxxxxxx (2010) attempted to el...
