How Variable May a Constant Be? Measures of Lexical Richness in Perspective

A well-known problem in the domain of quantitative linguistics and stylistics concerns the evaluation of the lexical richness of texts. Since the most obvious measure of lexical richness, the vocabulary size (the number of different word types), depends heavily on the text length (measured in word t...

Full description

Saved in:
Bibliographic Details
Published in:Computers and the humanities 1998-01, Vol.32 (5), p.323-352
Main Authors: Tweedie, Fiona J., Baayen, R. Harald
Format: Article
Language:English
Subjects:
Confidence interval
Discourse analysis
Function words
Language
Linguistics
Mathematical constants
Mathematics and linguistics
Monte Carlo simulation
Parametric models
Quantitative studies
Statistics
Stylistics
Trajectories
Vocabulary
Vocabulary size
Word frequency analysis
Words
Words (language)
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:A well-known problem in the domain of quantitative linguistics and stylistics concerns the evaluation of the lexical richness of texts. Since the most obvious measure of lexical richness, the vocabulary size (the number of different word types), depends heavily on the text length (measured in word tokens), a variety of alternative measures has been proposed which are claimed to be independent of the text length. This paper has a threefold aim. Firstly, we have investigated to what extent these alternative measures are truly textual constants. We have observed that in practice all measures vary substantially and systematically with the text length. We also show that in theory, only three of these measures are truly constant or nearly constant. Secondly, we have studied the extent to which these measures tap into different aspects of lexical structure. We have found that there are two main families of constants, one measuring lexical richness and one measuring lexical repetition. Thirdly, we have considered to what extent these measures can be used to investigate questions of textual similarity between and within authors. We propose to carry out such comparisons by means of the empirical trajectories of texts in the plane spanned by the dimensions of lexical richness and lexical repetition, and we provide a statistical technique for constructing confidence intervals around the empirical trajectories of texts. Our results suggest that the trajectories tap into a considerable amount of authorial structure without, however, guaranteeing that spatial separation implies a difference in authorship.
ISSN:0010-4817
1574-020X
1572-8412
1574-0218
DOI:10.1023/A:1001749303137