Answer Passage Retrieval. This section describes another selection-based QA task, called answer pas- sage retrieval, that finds the answer context from a larger dataset, the en- tire Wikipedia. SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia. An automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda6 is presented so they can be coherently used for sentence selection tasks. Each paragraph in Wikipedia is first indexed by Lucene using {1,2,3}- grams, where the paragraphs are separated by WikiExtractor7 and segmented by NLP4J8 (28.7M+ paragraphs are indexed). Each answer sentence from 6enwiki-20160820-pages-articles.xml.bz2 0xxxxxx.xxx/xxxxxxx/wikiextractor 0xxxxxx.xxx/xxxxxxxx/xxx0x WikiQA SelQA SQuAD (ρ, цc, цp),t ≥ 0.3 ( 92.00, 1 203, 96.86) (90.00, 7 446, 94.28) (100.00, 93 928, 95.61) (ρ, цc, цp),t ≥ 0.4 ( 94.00, 1 139, 91.71) (94.00, 7 133, 90.31) (100.00, 93 928, 95.61) (ρ, цc, цp),t ≥ 0.5 (100.00, 1 051, 84.62) (98.00, 6 870, 86.98) (100.00, 93 928, 95.61) k = (1, 5, 10, 20) (4.39, 12.47, 16.59, 22.39) (20.01, 34.07, 40.29, 46.40) (19.90, 35.08, 40.96, 46.74) Table 3.16: Statistics of the silver-standard dataset (first three rows) and the accuracies of answer retrieval in % (last row). ρ: robustness of the silver-standard in %, цc/p: #/% of retrieved silver-standard passages (coverage). the corpora in Table 3.16 is then queried to Lucene, and the top-5 ranked paragraphs are retrieved. The cosine similarity between each sentence in these paragraphs and the answer sentence is measured for n-grams, say n1,2,3. A weight is assigned to each n-gram score, say λ1,2,3, and the weighted sum i=1 is measured: t = P3 λi · ni. The fixed weights of λ1,2,3 = (0.25, 0.35, 0.4) are used for the experiments, which can be improved in the future. If there exists a sentence whose t ≥ θ, the paragraph consisting of that sentence is considered the silver-standard answer passage. Table 3.16 shows how robust these silver-standard passages are based on human judgment (ρ) and how many passages are collected (γ) for θ = [0.3, 0.5], where the human judgment is performed on 50 random samples for each case. For answer retrieval, a dataset is created by θ = 0.4, which gives ρ ≥ 94% accuracy and γp > 90% coverage, respectively.9 Finally, each question is queried to Lucene and the top-k paragraphs are retrieved from the entire Wikipedia. 9SQuAD ma...
