Test Set Pruning. We also tried several different values of the rank cutoff for test set pruning, using the full training setup and testing on the dev set. The results are in Table 3.3. For F1 evalua- tion, which is on a very small set of sentences, we selected 500 as the value with the best speed/performance tradeoff. However, when reranking our entire MT corpus, we used a value of 200, sacrificing a tiny bit of performance for an extra factor of 2 in speed.7 7Using a rank cutoff of 200, the reranking step takes slightly longer than serially running both baseline parsers, and generating k-best lists takes slightly longer than getting 1-best parses, so in total, joint parsing takes about 2.3 times as long as monolingual parsing. With a rank cutoff of 500, total parsing time is scaled by a factor of around 3.8. ϵ Chinese F1 English F1 Total F1 Tree Pairs 15 85.78 77.75 82.05 1,463,283 20 85.88 77.27 81.90 1,819,261 25 86.37 78.92 82.91 2,204,988 30 85.97 79.18 82.83 2,618,686 40 86.10 78.12 82.40 3,521,423 50 85.95 78.50 82.50 4,503,554 100 86.28 79.02 82.91 8,997,708 Table 3.2: Training set pruning study. F1 on dev set after training with different values of the ϵ parameter for training set pruning. Cutoff Chinese F1 English F1 Total F1 Time (s) 50 86.34 79.26 83.04 174 100 86.61 79.31 83.22 307 200 86.67 79.39 83.28 509 500 86.76 79.41 83.34 1182 1000 86.80 79.39 83.35 2247 2000 86.78 79.35 83.33 4476 10,000 86.71 79.37 83.30 20,549 Table 3.3: Test set pruning study. F1 on dev set obtained using different cutoffs for test set pruning. k Joint Chinese F1 Parsing English F1 Chinese Oracle F1 English F1 1 84.95 76.75 84.95 76.75 10 86.23 78.43 90.05 81.99 25 86.64 79.27 90.99 83.37 50 86.61 79.10 91.82 84.14 100 86.71 79.37 92.23 84.73 150 86.67 79.47 92.49 85.17 Table 3.4: Sensitivity to k study. Joint parsing and oracle F1 obtained on dev set using different maximum values of k when generating baseline k-best lists. Parsing Model Chinese F1 English F1 Total F1 Monolingual 83.6 81.2 82.5 Bilingual 86.0 83.8 84.9 Table 3.5: Final evaluation. Comparison of F1 on test set between baseline parsers and joint parser.
Test Set Pruning. Because the size of (T, Tr) grows as O(k2), the time spent iterating through all these tree pairs can grow unreasonably long, particularly when reranking a set of sentence pairs the size of a typical MT corpus. To combat this, we use a simple pruning technique to limit the number of tree pairs under consideration. To prune the list of tree pairs, first we rank them according to the metric: wSOURCELL · SOURCELL + xXXXXXXXX · TARGETLL Then, we simply remove all tree pairs whose ranking falls below some empirically deter- mined cutoff. As we show in Section 3.5.3, by using this technique we are able to speed up reranking by a factor of almost 20 without an appreciable loss of performance.
