esCorpius-m: A Massive Multilingual Crawling Corpus with a Focus on Spanish

In recent years, transformer-based models have played a significant role in advancing language modeling for natural language processing. However, they require substantial amounts of data and there is a shortage of high-quality non-English corpora. Some recent initiatives have introduced multilingual...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences 2023-11, Vol.13 (22), p.12155
Main Authors: Gutiérrez-Fandiño, Asier, Pérez-Fernández, David, Armengol-Estapé, Jordi, Griol, David, Kharitonova, Ksenia, Callejas, Zoraida
Format: Article
Language:English
Subjects:
Automation
common crawl
corpus
crawling
Data analysis
dataset
Datasets
Digital archives
Financial analysis
Language
Legal research
Machine translation
massive
multilingual
Multilingualism
Natural language
Sentiment analysis
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent years, transformer-based models have played a significant role in advancing language modeling for natural language processing. However, they require substantial amounts of data and there is a shortage of high-quality non-English corpora. Some recent initiatives have introduced multilingual datasets obtained through web crawling. However, there are notable limitations in the results for some languages, including Spanish. These datasets are either smaller compared to other languages or suffer from lower quality due to insufficient cleaning and deduplication. In this paper, we present esCorpius-m, a multilingual corpus extracted from around 1 petabyte of Common Crawl data. It is the most extensive corpus for some languages with such a level of high-quality content extraction, cleanliness, and deduplication. Our data curation process involves an efficient cleaning pipeline and various deduplication methods that maintain the integrity of document and paragraph boundaries. We also ensure compliance with EU regulations by retaining both the source web page URL and the WARC shared origin URL.
ISSN:2076-3417
2076-3417
DOI:10.3390/app132212155