Loading…

In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, w...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-02
Main Authors: Kuratov, Yuri, Bulatov, Aydar, Anokhin, Petr, Sorokin, Dmitry, Sorokin, Artyom, Burtsev, Mikhail
Format: Article
Language:English
Subjects:
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to \(10^4\) elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to \(11\times 10^6\) elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.
ISSN:2331-8422