(Ir)rationality and cognitive biases in large language models

Do large language models (LLMs) display rational reasoning? LLMs have been shown to contain human biases due to the data they have been trained on; whether this is reflected in rational reasoning remains less clear. In this paper, we answer this question by evaluating seven language models using tas...

Full description

Saved in:
Bibliographic Details
Published in:Royal Society open science 2024-06, Vol.11 (6), p.240255-14
Main Authors: Macmillan-Scott, Olivia, Musolesi, Mirco
Format: Article
Language:English
Subjects:
Artificial intelligence
Bias
Cognition & reasoning
cognitive bias
Cognitive psychology
Decision making
Experiments
Human performance
Language
Large language models
Rationality
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Do large language models (LLMs) display rational reasoning? LLMs have been shown to contain human biases due to the data they have been trained on; whether this is reflected in rational reasoning remains less clear. In this paper, we answer this question by evaluating seven language models using tasks from the cognitive psychology literature. We find that, like humans, LLMs display irrationality in these tasks. However, the way this irrationality is displayed does not reflect that shown by humans. When incorrect answers are given by LLMs to these tasks, they are often incorrect in ways that differ from human-like biases. On top of this, the LLMs reveal an additional layer of irrationality in the significant inconsistency of the responses. Aside from the experimental results, this paper seeks to make a methodological contribution by showing how we can assess and compare different capabilities of these types of models, in this case with respect to rational reasoning.
ISSN:2054-5703
2054-5703
DOI:10.1098/rsos.240255