A METRIC FOR EVALUATING CONFIDENCE. This section gives the definition of our metric for evaluating con- fidence. In principle, the task of estimating confidence for each hypothesized word is to have an estimate of which words of the outputs of LVCSR models are likely to be correct and which are not reliable. In this paper, however, we focus on estimating cor- rectly recognized words and evaluate confidence according to re- call/precision rates of estimating correctly recognized words. The following gives a procedure for evaluating the agreement among the outputs of multiple LVCSR models as an estimate of correctly recognized words. First, let us suppose that we have two outputs Hyp1 and Hyp2 of two LVCSR models, each of which is represented as a sequence of hypothesized words. Next, two sequences Hyp1 and Hyp2 of hypothesized words are aligned by DP matching. Then, words that are aligned together and have an identical lexical form are collected into a list named agreed word list. Suppose that we have two sequences Hyp1 and Hyp2 of hypothesized words as below: Hyp1 = w11, ... , w1i,... , w1k
A METRIC FOR EVALUATING CONFIDENCE. This section gives the definition of our metric for evaluating con- fidence. In principle, the task of estimating confidence for each hypothesized word is to have an estimate of which words of the outputs of LVCSR models are likely to be correct and which are not reliable. In this paper, however, we focus on estimating cor- rectly recognized words and evaluate confidence according to re- call/precision rates of estimating correctly recognized words. The following gives a procedure for evaluating the agreement among the outputs of multiple LVCSR models as an estimate of correctly recognized words. First, let us suppose that we have two outputs Hyp1 and Hyp2 of two LVCSR models, each of which is represented as a sequence of hypothesized words. Next, two sequences Hyp1 and Hyp2 of hypothesized words are aligned by DP matching. Then, words that are aligned together and have an identical lexical form are collected into a list named agreed word list. Suppose that we have two sequences Hyp1 and Hyp2 of hypothesized words as below: Hyp1 = w11, ... , w1i,... , w1k Hyp2 = w21, ... , w2j,... , w2l Then, the agreed word list is constructed by collecting those words w1i (= w2j ) that satisfy the constraint: w1i and w2j are aligned to- gether by DP matching, and w1i and w2j are lexically identical. Finally, the following recall/precision rates are calculated by com- paring the agreed word list with the reference sentence considering both the lexical form and the position of each word. # of correct words in the agreed word list Newspaper Article Sentences) speech data [8]. Recall = # of words in the reference sentence 1 When running the decoder with HMMs without the short pause state, we remove the powerless frames from the input speech, and use the lan- guage model trained without punctuation symbols (i.e., comma and pe- riod). 2 The reason why we evaluate acoustic models both with and without the short pause states is that, from the preliminary evaluation result, this difference proved to be among those most effective in achieving high con- fidence.
