Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation

We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent...

Full description

Saved in:
Bibliographic Details
Main Authors: Fakhoury, Sarah, Naik, Aaditya, Sakkas, Georgios, Chakraborty, Saikat, Musuvathi, Madan, Lahiri, Shuvendu
Format: Conference Proceeding
Language:English
Subjects:
Accuracy
Codes
LLM4Code
Natural languages
Software engineering
User intent formulation
user study
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.
ISSN:2574-1934
DOI:10.1145/3639478.3643525