From traces to measures: Large language models as a tool for psychological measurement from text

Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-10
Main Authors: Simons, Joseph J P, Wong, Liang Ze, Bhattacharya, Prasanta, Loh, Brandon Siyuan, Gao, Wei
Format: Article
Language:English
Subjects:
Annotations
Large language models
Psychology
Target recognition
Workflow
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Large language models are increasingly being used to label or rate psychological features in text data. This approach helps address one of the limiting factors of digital trace data - their lack of an inherent target of measurement. However, this approach is also a form of psychological measurement (using observable variables to quantify a hypothetical latent construct). As such, these ratings are subject to the same psychometric considerations of reliability and validity as more standard psychological measures. Here we present a workflow for developing and evaluating large language model based measures of psychological features which incorporate these considerations. We also provide an example, attempting to measure the previously established constructs of attitude certainty, importance and moralization from text. Using a pool of prompts adapted from existing measurement instruments, we find they have good levels of internal consistency but only partially meet validity criteria.
ISSN:2331-8422