Active Learning. The main difference between standard passive learning and active learning is that, instead of strictly separating between a training and a testing phase, the active learner performs loops of training and testing, thereby incorporating the information flow obtained from the teacher (e.g. a human supervi- sor) into the loop. Fig. 1 shows a schematic flow chart of a generic active learning algorithm. We note that, while in general active learning can be used in many different contexts, we use it for object classification in this work. ICT-FP7-600877-XXXXXXX Deliverable D2.3 Figure 1: Active Learning flow chart. After an initial training step, the classifier is presented new test data and reports label predictions and confidence values (here: uncertainties). These are used to ask a human supervisor for new ground truth labels, which subsequently are added to the current training data. Then, the training process is repeated with the extended training data until a stopping criterion is met. One important question in active learning is how to select the data samples for which semantic information, i.e. in our case class labels, are requested from the human supervisor. We refer to this as the which-question problem. The most used method to address it is uncertainty sampling, and we also use this in our implementation. Thus, we compute an uncertainty along with the prediction of a newly observed sample. Then, we use a confidence threshold #c and decide to ask for a ground truth label yˆ for all those data samples which, in the current learning epoch, have been classified with a confidence lower than #c.
Active Learning. During the annotation phase of the project, there were some impediments, on which Chapter 2.2 will elaborate. In order to collect more data for a more robust data set and explore different ways to improve the performance of model, a web-based online Active Learning system was developed in parallel to the conventional offline data collection and offline model tuning. Active Learning is a sub-branch of Machine Learning in which the learning system will interactively query the user to obtain the desired data from the user[9, 11]. In the implementation of the Active Learning model for Dialogue Question Answering, the system will first analyze the performance of the Machine Learning model against a fixed set of test cases, then prompt the user who use the system to produce data that could potentially assist in the improvement of the model, and finally the system will periodically use the collected and targeted data to train and improve the model. The main focus of the this thesis will be on the Active Learning system built to assist the Question Answering model, including its architecture and much of the engineering details.
