engleski

Machine Translation Evaluation: a Case study of Croatian-English and Russian-English MT Systems

From the earliest days of machine translation (MT), evaluation has been an inherent and significant part of efforts invested into machine translation research. In this paper, an overview of the history of MT evaluation is presented, with emphasis on one of the most comprehensive MT evaluation projects, undertaken in the 1960s, namely the Automatic Language Processing Advisory Committee Report. Furthermore, strategies and problems pertaining to MT evaluation are discussed, with emphasis on the distinction between subjective criteria, such as comprehensibility, and the objective, quantifiable criteria, such as error quantification and analysis. Within the practical part of the paper, the results of testing four MT systems – one for the language pair Croatian-English, and three for the language pair Russian-English – are shown. The systems were tested on three textual samples belonging to general, fictional and scientific genres. The analysis of the results included a comprehensibility poll which included five informants (native or proficient target language speakers) for each target language, as well as quantification of errors and error type assessment across genres and across individual MT systems. Finally, cumulative results are given for each MT system and for each language pair. As a conclusion, recent developments in the field of MT evaluation are presented, including automatic evaluation methods, such as IBM’ s measures BLEU and NIST.

machine translation; MT evaluation; evaluation criteria

nije evidentirano