Rank diversity of languages: generic behavior in computational linguistics

Statistical studies of languages have focused on the rank-frequency distribution of words. Instead, we introduce here a measure of how word ranks change in time and call this distribution rank diversity. We calculate this diversity for books published in six European languages since 1800, and find t...

Full description

Saved in:
Bibliographic Details
Published in:PloS one 2015-04, Vol.10 (4), p.e0121898-e0121898
Main Authors: Cocho, Germinal, Flores, Jorge, Gershenson, Carlos, Pineda, Carlos, Sánchez, Sergio
Format: Article
Language:English
Subjects:
Analysis
Computer applications
Datasets
Europe
Frequency distribution
Human communication
Humans
Language
Languages
Linguistics
Models, Theoretical
Natural language processing
Random walk
Rankings
Variation
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Statistical studies of languages have focused on the rank-frequency distribution of words. Instead, we introduce here a measure of how word ranks change in time and call this distribution rank diversity. We calculate this diversity for books published in six European languages since 1800, and find that it follows a universal lognormal distribution. Based on the mean and standard deviation associated with the lognormal distribution, we define three different word regimes of languages: "heads" consist of words which almost do not change their rank in time, "bodies" are words of general use, while "tails" are comprised by context-specific words and vary their rank considerably in time. The heads and bodies reflect the size of language cores identified by linguists for basic communication. We propose a Gaussian random walk model which reproduces the rank variation of words in time and thus the diversity. Rank diversity of words can be understood as the result of random variations in rank, where the size of the variation depends on the rank itself. We find that the core size is similar for all languages studied.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0121898