Read-All-in-Once (RAiO): Multi-Layer Contextual Architecture for Long-Text Machine Reading Comprehension

Machine reading comprehension (MRC) is a cutting-edge technology in natural language processing (NLP), which focuses on teaching machines to read and understand the meaning of texts based on the emergence of large-scale datasets and neural network models. Recently, with the successful development of...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2023, Vol.11, p.77873-77879
Main Authors: Phan, Tuan-Anh, Jung, Jason J., Bui, Khac-Hoai Nam
Format: Article
Language:English
Subjects:
Bit error rate
Computational modeling
Context
Context modeling
Data mining
Datasets
Documents
long-text machine reading comprehension
Multilayers
Natural language processing
Neural networks
Question answering (information retrieval)
question-answering system
Reading comprehension
Task analysis
Teaching machines
Text processing
Texts
Transformers
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Machine reading comprehension (MRC) is a cutting-edge technology in natural language processing (NLP), which focuses on teaching machines to read and understand the meaning of texts based on the emergence of large-scale datasets and neural network models. Recently, with the successful development of pre-trained transformer models (e.g., BERT), MRC has advanced significantly, surpassing human parity in several public datasets and being applied in various NLP tasks (e.g., QA systems). Nevertheless, long document MRC is still a remain challenge since the transformer-based models are limited by the input length. For instance, several well-known pre-trained language models such as BERT and RoBERTa are limited by 512 tokens. This study aims to provide a new simple approach for long document MRC. Specifically, recent state-of-the-art models follow the architecture with two crucial stages for reading long texts in order to enable local and global context representations. In this study, we present a new architecture that is able to enrich the global information of the context with one stage by exploiting the interaction of different levels of semantic units of the context (i.e., sentence and word level). Therefore, we name the proposed model as RAiO (Read-All-in-Once) approach. For the experiment, we evaluate RAiO on two benchmark long document MRC datasets such as NewsQA and NLQuAD. Accordingly, the experiment shows promising results of the proposed approach compared with strong baselines in this research field.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2023.3298100