User Guidance Sample Clauses

User Guidance. ‌ The Active Learning System provide guidance on user input by two means: selecting the scene and selecting question type for user to contribute to. For the scenes, the goal is to ensure an even distribution among all scenes, excluding ones that will be reserved for testing. For question types, the goal is to encourage user to contribute more on the question types that: 1. annotated the least; 2. the model showed insufficient performance on. On the other hand, since continuously requesting the same scene and question type would result in bad user experience, the system make use of random generation with probability to ensure enough freshness in the user experience. Since the data set has abundant number of scenes available, for scene selection, we use an uniform distribution for the scenes which has least count of annotation: P (Si) =  1 Nmin Si = Smin (3.1)  0 Si /= Smin where Si denotes the count of annotation of Scene i, Smin denotes the the min of all Si and Nmin denotes | {Si | Si = Smin} |. For the question type selection, we start by defining the normalized Q question type count QˆiN = QiN , where QiN is the count of annotations on N question type Qi and QN is the total count of annotations. f (Q) = 3 − (QˆiN + QiF1 + QiEM ) (3.2) f (Q) P (Q) = c − 1 e3 − 1 × (e − 1) + 1 (3.3) Here QiF1 and QiEM denotes the F-1 score and Exact Match for question type Qi and c is a scaling factor used to control the upper bound of the probability of a single question type. Since QiN , QiF1 and QiEM all have range [0, 1], f (Q) has range of [0, 3]. The exponential function will then have range of [1, e3]. The factor c scales the probability linearly range to [1, c]. It denotes the ratio of probability of a question type with minimal statistics being chose and one of an answer with maximal statistics being chose. The exponential function guarantees the discrepancy between low stat and high stat will rise rapidly while c ensures a question type with perfect statistics c still have 1 chance of selected with respect to a question type with minimal statistics. The final probability is computed by normalizing P (Qi) for all question types. In order to prevent the side effect that revealing the test case statistics in user guidance could cause biases in our results, a set of data disjoint with the training data and test data is separated and used for obtaining statistics for user guidance purpose only. Here QiF1 and QiEM are the statistics from such dev data sets. The t...

User Guidance