Training Parsing Models for MT Sample Clauses

Training Parsing Models for MT. ‌ One seldom-discussed yet fairly important issue in syntactic MT is tokenization. Both parsers and MT systems are very sensitive to tokenization conditions, but the datasets used for these tasks typically have very different tokenization schemes. In order to deal with this concern in our experiments, we modified the treebanks in Tables 2.1 and 2.3 as follows: • The surface word yields of the individual sentences in each treebank were retok- enized according to the same guidelines as the MT corpora using a tokenizer obtained from BBN. • The gold parse trees were coerced to match the new tokenization using Procrustes, a best-fit tree retokenization tool.6 • For the bilingual treebanks, the gold word alignments were also transformed to match the new tokenization by duplicating or collapsing alignment links as tokens were split or merged, respectively. All parsing models (both the monolingual baselines and the bilingual models) were retrained on the retokenized treebanks before they were used to annotate the MT training data. 5The tuning and test corpora each had four English references per sentence pair.‌ 6Thanks to Xxxxx Xxxxxx for sharing this software.