Optimizing Tokenization Choice for Machine Translation across Multiple Target Languages

Tokenization is very helpful for Statistical Machine Translation (SMT), especially when translating from morphologically rich languages. Typically, a single tokenization scheme is applied to the entire source-language text and regardless of the target language. In this paper, we evaluate the hypothe...

Full description

Saved in:
Bibliographic Details
Published in:Prague bulletin of mathematical linguistics 2017-06, Vol.108 (1), p.257-269
Main Authors: Zalmout, Nasser, Habash, Nizar
Format: Article
Language:English
Subjects:
Arabic language
Chinese languages
Closed circuit television
Complexity
Context
English language
French language
Languages
Machine translation
Morphological complexity
Performance enhancement
Russian language
Spanish language
Translating
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Tokenization is very helpful for Statistical Machine Translation (SMT), especially when translating from morphologically rich languages. Typically, a single tokenization scheme is applied to the entire source-language text and regardless of the target language. In this paper, we evaluate the hypothesis that SMT performance may benefit from different tokenization schemes for different words within the same text, and also for different target languages. We apply this approach to Arabic as a source language, with five target languages of varying morphological complexity: English, French, Spanish, Russian and Chinese. Our results show that different target languages indeed require different source-language schemes; and a context-variable tokenization scheme can outperform a context-constant scheme with a statistically significant performance enhancement of about 1.4 BLEU points.
ISSN:1804-0462
0032-6585
1804-0462
DOI:10.1515/pralin-2017-0025