What makes multilingual BERT multilingual?

Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of no...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2020-10
Main Authors: Chi-Liang, Liu, Tsung-Yuan Hsu, Yung-Sung Chuang, Hung-yi, Lee
Format: Article
Language:English
Subjects:
Multilingualism
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.
ISSN:2331-8422