Crowdsourcing. In order to build large datasets and train machine learning algorithms, one of the most promising tools is LSUN (Yu et al): a partially automated labeling scheme, leveraging deep learning with humans in the loop. Starting from a large set of candidate images for each category, LSUN iteratively samples a subset, ask people to label them, classify the others with a trained model, split the set into positives, negatives, and unlabeled based on the classification confidence, and then iterate with the unlabeled set.
Crowdsourcing. Images can be used for crowdsourcing projects. In such a case an image of specimen or label will be displayed in a specific portal for general public (for citizen scientists). Citizen scientists will use their skills to categorise images, transcribe labels etc. Often the participation involves a learning process to raise the skills which are needed for the task. Learning taxonomy and morphology Images of specimens as illustrations can help with identification and learning for certain features of natural objects, e.g. to learn similarities or differences of systematic groups etc. Images of botanical specimens, for example, can be used in this way. This can be done in classroom or at home. OCCURRENCE DATA School study projects Based on specimen occurrence data (and related metadata), various educational study projects can be carried out.
Crowdsourcing. T2.1.1: Crowdsourcing: requirements and policy definition, start M1, duration 9 months (SHIFT 4M, EF 4M, NTUA 4M, AIT 4M, NISV 4M) This task will define requirements for crowdsourcing of metadata in the context of Europeana Sounds. Requirements will be gathered through focus groups sessions, desk research and complimentary questionnaires circulated to the Europeana Network. The User Advisory Panel (see B3.2c) will be consulted. The requirements will differentiate between two target groups: the general public and experts. This sub-task will also define preconditions for all end-user contributions (exchange policies) created in the context of WP2 to be offered to the content providers. This will be reported as part of D2.1.
Crowdsourcing. Finally and in order to involve the crowd in the ideation of use cases we have been in contact with the potential stakeholders of the IoT Lab. Through interviews and the organisation of workshops with the end-users we have gathered also relevant use cases. Together with WP5 we have defined guidelines on how to perform such kind of workshops/interviews as we can see in 6.3. Another important point for the contribution from the crowd on the scenarios is that we have added in the IoT Lab website the possibility to the crowd to contribute with ideas for the project. We will through the duration of the project evaluate and consider new scenarios coming from the community.
Crowdsourcing. Crowdsourcing makes use of unknown, usually large, populations to carry out online tasks (Xxxx, 2006). These range from evaluating collections of items, building physical artefacts and social networks, labelling images, rating films, digitising text, spelling correction and assessing recommendations (Xxxx et al., 2011). Xxxxxxxx Xxxxxx & Xxxxxxxx Xxxxxx-xx- Xxxxxxx survey over 40 articles that define crowdsourcing to propose a single definition: “Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task. The undertaking of the task, of variable complexity and modularity, and in which the crowd should participate bringing their work, money, knowledge and/or experience, always entails mutual benefit. The user will receive the satisfaction of a given type of need, be it economic, social recognition, self-esteem, or the development of individual skills, while the crowdsourcer will obtain and utilize to their advantage that what the user has brought to the venture, whose form will depend on the type of activity undertaken.” Crowdsourcing has grown in popularity online, with huge success stories such as Wikipedia and Project Gutenberg utilising the ‘cognitive surplus’ (Shirky, 2010) of many thousands of anonymous web users to develop and edit major sources of digital content for mass consumption. Another interesting development has been the emergence and increasing popularity of Crowdsourcing Systems (e.g. Amazon’s Mechanical Turk system, Crowdflower, and others) as a means of accessing very large crowds to complete simple computational tasks. In fact crowdsourcing systems have been identified as a field where rapid growth is expected in coming years (Xxxx et al., 2011). However, there are several issues facing the development of crowdsourcing systems ranging from ethical concerns to ensuring quality of data collected using unknown populations. Here the focus is on Amazon’s Mechanical Turk as the crowdsourcing platform and its use for data collection in Natural Language Processing (NLP) and Information Retrieval (IR) tasks. In contrast, many cultural heritage projects that harness crowdsourcing focus on extending the long-standing tradition of volunteering and altruism, utilising communities of interested users to support large-scale digitisat...
Crowdsourcing. All annotation tasks are conducted on the Amazon Mechanical Turk. TALEN, a web-based tool for named entity annotation [16], is extended for our QA annotation such that it displays a dialogue segmented into a sequence of utterances with utterance IDs and speaker names on the left panel. It allows the annotator to select a span of words with left-click. At the completion of the click, a pop-up window will show and reveal the available labels that could be used to tag the current span. In our case, the labels would be the question ID (For example, this span of texts is the answer to what question). On the right panel it will ask crowd workers to generate questions regarding this dialogue and then select spans in the dialogue that contain the correct information. Prior to the annotation, crowd workers are required to pass a simple quiz regarding the dialogue context, to verify if they have a good understanding in this context and the required knowledge to use this web interface. The actual annotation task remains hidden until they pass this quiz. Upon the submission, a series of validations will take place and make sure the question and answer format are acceptable (Section 3.4).
