Context Matters: Investigating Its Impact on ChatGPT's Bug Fixing Performance

In this study, we explore the role of contextual information in enhancing ChatGPT's capabilities in bug fixing. Our focus is specifically on the "Wrong Answer" problem, where a program executes without error but fails to produce the correct output. Our approach draws inspiration from...

Full description

Saved in:
Bibliographic Details
Main Authors: Qu, Xianshan, Zuo, Fei, Li, Xiaopeng, Rhee, Junghwan
Format: Conference Proceeding
Language:English
Subjects:
Bug Fixing
Chatbots
Computer bugs
Context modeling
Contextual Information
Debugging
Large Language Models
Merging
Problem Descriptions
Software engineering
Test Cases
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this study, we explore the role of contextual information in enhancing ChatGPT's capabilities in bug fixing. Our focus is specifically on the "Wrong Answer" problem, where a program executes without error but fails to produce the correct output. Our approach draws inspiration from human debugging practices, which heavily rely on understanding both the intended task of the program and the specific scenarios in which it fails, such as unit test cases. We evaluate ChatGPT's performance with various types and levels of contextual data. The results reveal three key insights. First, providing the model with a mix of correct and incorrect test cases sharpens its debugging skills. Second, giving ChatGPT detailed descriptions of the problems substantially enhances its ability to identify and resolve errors. Third, merging detailed problem descriptions with various test cases leads to a synergistic outcome. This combined approach significantly elevates the efficiency of the bug-fixing process compared to employing each type of contextual information individually. Our paper presents a thorough analysis based on these findings. It offers an extensive exploration of why and how contextual information can be strategically utilized to enhance ChatGPT's debugging effectiveness. Furthermore, this investigation enriches our comprehension of the underlying mechanisms by which contextual cues amplify the model's capacity for solving problems.
ISSN:2770-8209
DOI:10.1109/SERA61261.2024.10685568