Exploring and Benchmarking the Planning Capabilities of Large Language Models

Classical and natural language planning tasks remain a difficult domain for modern large language models (LLMs). In this work, we lay the foundations for improving planning capabilities of LLMs. First, we construct a comprehensive benchmark suite encompassing both classical planning benchmarks and n...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-11
Main Authors: Bohnet, Bernd, Nova, Azade, Parisi, Aaron T, Swersky, Kevin, Goshvadi, Katayoon, Dai, Hanjun, Schuurmans, Dale, Fiedel, Noah, Sedghi, Hanie
Format: Article
Language:English
Subjects:
Algorithms
Benchmarks
Context
Large language models
Natural language
Performance evaluation
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Classical and natural language planning tasks remain a difficult domain for modern large language models (LLMs). In this work, we lay the foundations for improving planning capabilities of LLMs. First, we construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. This suite includes algorithms to methodically generate instances of tasks with varying levels of difficulty, allowing for rigorous and systematic evaluation of LLM performance. Next, we investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance. In addition, we demonstrate the positive impact of fine-tuning LLMs on optimal planning paths. We also probe the efficacy of chain-of-thought reasoning methods to improve LLM planning performance. Moreover, we probe the performance of the proposed methods in out-of-distribution scenarios, assessing the ability to generalize to novel and unseen planning challenges. Finally, we investigate model's failure modes and reveal insights that hold true across different benchmarks.
ISSN:2331-8422