Language Statistics at Different Spatial, Temporal, and Grammatical Scales

In recent decades, the field of statistical linguistics has made significant strides, which have been fueled by the availability of data. Leveraging Twitter data, this paper explores the English and Spanish languages, investigating their rank diversity across different scales: temporal intervals (ra...

Full description

Saved in:
Bibliographic Details
Published in:Entropy (Basel, Switzerland) Switzerland), 2024-08, Vol.26 (9), p.734
Main Authors: Sánchez-Puig, Fernanda, Lozano-Aranda, Rogelio, Pérez-Méndez, Dante, Colman, Ewan, Morales-Guzmán, Alfredo J, Rivera Torres, Pedro Juan, Pineda, Carlos, Gershenson, Carlos
Format: Article
Language:English
Subjects:
Analysis
Datasets
geolocalization
Information management
Language
language models
Languages
Linguistics
ngrams
rank diversity
Social networks
Spanish language
Statistics
Time
Twitter
Words (language)
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent decades, the field of statistical linguistics has made significant strides, which have been fueled by the availability of data. Leveraging Twitter data, this paper explores the English and Spanish languages, investigating their rank diversity across different scales: temporal intervals (ranging from 3 to 96 h), spatial radii (spanning 3 km to over 3000 km), and grammatical word ngrams (ranging from 1-grams to 5-grams). The analysis focuses on word ngrams, examining a time period of 1 year (2014) and eight different countries. Our findings highlight the relevance of all three scales with the most substantial changes observed at the grammatical level. Specifically, at the monogram level, rank diversity curves exhibit remarkable similarity across languages, countries, and temporal or spatial scales. However, as the grammatical scale expands, variations in rank diversity become more pronounced and influenced by temporal, spatial, linguistic, and national factors. Additionally, we investigate the statistical characteristics of Twitter-specific tokens, including emojis, hashtags, and user mentions, revealing a sigmoid pattern in their rank diversity function. These insights contribute to quantifying universal language statistics while also identifying potential sources of variation.
ISSN:1099-4300
1099-4300
DOI:10.3390/e26090734