Modeling Monolingual Character Alignment for Automatic Evaluation of Chinese Translation

• Published in: ACM transactions on Asian and low-resource language information processing 2016-12, Vol.15 (3), p.1-18
• Main Authors: Li, Maoxi, Wang, Mingwen, Li, Hanxi, Xu, Fan
• Format: Article
• Language: English
• ISSN: 2375-4699; 2375-4702
• DOI: 10.1145/2815619

Automatic evaluation of machine translations is an important task. Most existing evaluation metrics rely on matching the same word or letter n -grams. This strategy leads to poor results on Chinese translations because one has to rely merely on matching identical characters. In this article, we propose a new evaluation metric that allows different characters with the same or similar meaning to match. An Indirect Hidden Markov Model (IHMM) is proposed to align the Chinese translation with human references at the character level. In the model, the emission probabilities are estimated by character similarity, including character semantic similarity and character surface similarity, and transition probabilities are estimated by a heuristic distance-based distortion model. When evaluating the submitted output of English-to-Chinese translation systems in the IWSLT’08 CT-EC and NIST’08 EC tasks, the experimental results indicate that the proposed metric has a significantly better correlation with human evaluation than the state-of-the-art machine translation metrics (i.e., BLEU, Meteor Universal, and TESLA-CELAB). This study shows that it is important to allow different characters to match in the evaluation of Chinese translations and that the IHMM is a reasonable approach for the alignment of Chinese characters.