Building efficient and effective OpenQA systems for low-resource languages

• Published in: Knowledge-based systems 2024-10, Vol.302, p.112243, Article 112243
• Main Authors: Budur, Emrah, Özçelik, Rıza, Soylu, Dilara, Khattab, Omar, Güngör, Tunga, Potts, Christopher
• Format: Article
• Language: English
• Subjects: Low-resource languages; Machine translation; Open domain question answering; OpenQA; Question answering
• ISSN: 0950-7051
• DOI: 10.1016/j.knosys.2024.112243

Question answering (QA) is the task of answering questions posed in natural language with free-form natural language answers extracted from a given passage. In the OpenQA variant, only a question text is given, and the system must retrieve relevant passages from an unstructured knowledge source and use them to provide answers, which is the case in the mainstream QA systems on the Web. QA systems currently are mostly limited to the English language due to the lack of large-scale labeled QA datasets in non-English languages. In this paper, we show that effective, low-cost OpenQA systems can be developed for low-resource contexts. The key ingredients are (1) weak supervision using machine-translated labeled datasets and (2) a relevant unstructured knowledge source in the target language context. Furthermore, we show that only a few hundred gold assessment examples are needed to reliably evaluate these systems. We apply our method to Turkish as a challenging case study, since English and Turkish are typologically very distinct and Turkish has limited resources for QA. We present SQuAD-TR, a machine translation of SQuAD2.0, and we build our OpenQA system by adapting ColBERT-QA and retraining it over Turkish resources and SQuAD-TR using two versions of Wikipedia dumps spanning two years. We obtain a performance improvement of 24–32% in the Exact Match (EM) score and 22–29% in the F1 score compared to the BM25-based and DPR-based baseline QA reader models. Our results show that SQuAD-TR makes OpenQA feasible for Turkish, which we hope encourages researchers to build OpenQA systems in other low-resource languages. We make all the code, models, and the dataset publicly available at https://github.com/boun-tabi/SQuAD-TR. •We show OpenQA is feasible in low-resource language contexts without gold labels for training.•Key enabler of OpenQA in low-resource languages: weak supervision and unstructured data.•We demonstrate that only a few hundred gold examples suffice to evaluate OpenQA.•Growing knowledge sources impact OpenQA results based on retrievers’ noise handling capability.•We release SQuAD-TR, a large scale Turkish QA dataset derived from SQuAD2.0.