Comparative Evaluation. In order to have an impression how the result is compared to the state of the art, the test sentences were translated with several available MT systems, to have an impression how useful they would be. The systems selected for comparison were one SMT and four RMT systems. The test sentences were translated, and the translations for the test words were identified and compared to the reference translation. Like for the absolute evaluation, total (rank1) and partial (rank2) identity were computed, as well as the overall scores. Tab. 5-1 shows the evaluation result. It can be seen that the LT-Xfr system clearly shows the best performance of all systems in all categories. It has much better scores than all RMT systems, and also better scores than Google. It is absolute 20% better than the least-performing MT system, and still 7% better than the best-performing one. Even the fallback frequency-based (LT-Xfr-freq) version outperforms all RMT systems, and is better than Google in three of six categories (Verbs1, Verbs/1+2, Adj/1+2).
Comparative Evaluation. The Consultant will evaluate feasible alternatives utilizing evaluation criteria agreed upon in the MM. The Consultant will compile the evaluation matrix to show the differences between each alternative in a manner appropriate for public use. The Consultant will assist the Department in presenting the results of comparative alternative evaluation in the public meeting and in soliciting public and agencies opinions on the alternatives recommended for elimination and alternatives that may be advanced in the PD&E study.
