Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation

We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent...

Full description

Saved in:
Bibliographic Details
Main Authors: Fakhoury, Sarah, Naik, Aaditya, Sakkas, Georgios, Chakraborty, Saikat, Musuvathi, Madan, Lahiri, Shuvendu
Format: Conference Proceeding
Language:English
Subjects:
Accuracy
Codes
LLM4Code
Natural languages
Software engineering
User intent formulation
user study
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page 391
container_issue
container_start_page 390
container_title
container_volume
creator Fakhoury, Sarah
Naik, Aaditya
Sakkas, Georgios
Chakraborty, Saikat
Musuvathi, Madan
Lahiri, Shuvendu
description We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.
doi_str_mv 10.1145/3639478.3643525
format conference_proceeding
fullrecord <record><control><sourceid>acm_CHZPO</sourceid><recordid>TN_cdi_ieee_primary_10554982</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>10554982</ieee_id><sourcerecordid>acm_books_10_1145_3639478_3643525</sourcerecordid><originalsourceid>FETCH-LOGICAL-a1625-185e834bbbff14e53ee5689721d3303d07be0738599fd2662cd786cd93cb3e4d3</originalsourceid><addsrcrecordid>eNqNkD1PwzAURQ0Iiap0ZmHwyJJix3Zss6EqlEpBDLRzZMcvYJEmlZ1W9N-TfkxMTG8491zpXYTuKJlSysUjy5jmUk1ZxplIxQWaaKkVJ0QSQVJ6iUapkDyhmvGrP-wGTWL0lggxJKiWI7TNfzZNF3z7ifsvwHldQ9X7HbQQI-5qXBRv2JoIDi8h9okLB4YXbQ_BHIN41jnA80EIpvdd-4RXEQL-6Lduj03rcL7e-OAr0-B8Z5rtMXSLrmvTRJic7xitXvLl7DUp3ueL2XORGJqlIqFKgGLcWlvXlINgACJTWqbUMUaYI9ICkUwJrWuXZllaOamyymlWWQbcsTG6P_V6ACg3wa9N2Jd0-J9rlQ54esKmWpe2677jwMrDyOV55PI8cmmDh3oQHv4psF8mUXgX</addsrcrecordid><sourcetype>Publisher</sourcetype><iscdi>true</iscdi><recordtype>conference_proceeding</recordtype></control><display><type>conference_proceeding</type><title>Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation</title><source>IEEE Xplore All Conference Series</source><creator>Fakhoury, Sarah ; Naik, Aaditya ; Sakkas, Georgios ; Chakraborty, Saikat ; Musuvathi, Madan ; Lahiri, Shuvendu</creator><creatorcontrib>Fakhoury, Sarah ; Naik, Aaditya ; Sakkas, Georgios ; Chakraborty, Saikat ; Musuvathi, Madan ; Lahiri, Shuvendu</creatorcontrib><description>We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.</description><identifier>ISBN: 9798400705021</identifier><identifier>EISSN: 2574-1934</identifier><identifier>EISBN: 9798400705021</identifier><identifier>DOI: 10.1145/3639478.3643525</identifier><identifier>CODEN: IEEPAD</identifier><language>eng</language><publisher>New York, NY, USA: ACM</publisher><subject>Accuracy ; Codes ; LLM4Code ; Natural languages ; Software engineering ; User intent formulation ; user study</subject><ispartof>2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2024, p.390-391</ispartof><rights>2024 Copyright held by the owner/author(s)</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><orcidid>0000-0002-8486-7749 ; 0000-0002-3100-0455 ; 0000-0002-2482-7892 ; 0000-0002-6889-7171 ; 0000-0002-1071-8038 ; 0000-0002-4446-4777</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/10554982$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>309,310,780,784,789,790,27925,54555,54932</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/10554982$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Fakhoury, Sarah</creatorcontrib><creatorcontrib>Naik, Aaditya</creatorcontrib><creatorcontrib>Sakkas, Georgios</creatorcontrib><creatorcontrib>Chakraborty, Saikat</creatorcontrib><creatorcontrib>Musuvathi, Madan</creatorcontrib><creatorcontrib>Lahiri, Shuvendu</creatorcontrib><title>Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation</title><title>2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)</title><addtitle>ICSE-COMPANION</addtitle><description>We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.</description><subject>Accuracy</subject><subject>Codes</subject><subject>LLM4Code</subject><subject>Natural languages</subject><subject>Software engineering</subject><subject>User intent formulation</subject><subject>user study</subject><issn>2574-1934</issn><isbn>9798400705021</isbn><isbn>9798400705021</isbn><fulltext>true</fulltext><rsrctype>conference_proceeding</rsrctype><creationdate>2024</creationdate><recordtype>conference_proceeding</recordtype><sourceid>6IE</sourceid><recordid>eNqNkD1PwzAURQ0Iiap0ZmHwyJJix3Zss6EqlEpBDLRzZMcvYJEmlZ1W9N-TfkxMTG8491zpXYTuKJlSysUjy5jmUk1ZxplIxQWaaKkVJ0QSQVJ6iUapkDyhmvGrP-wGTWL0lggxJKiWI7TNfzZNF3z7ifsvwHldQ9X7HbQQI-5qXBRv2JoIDi8h9okLB4YXbQ_BHIN41jnA80EIpvdd-4RXEQL-6Lduj03rcL7e-OAr0-B8Z5rtMXSLrmvTRJic7xitXvLl7DUp3ueL2XORGJqlIqFKgGLcWlvXlINgACJTWqbUMUaYI9ICkUwJrWuXZllaOamyymlWWQbcsTG6P_V6ACg3wa9N2Jd0-J9rlQ54esKmWpe2677jwMrDyOV55PI8cmmDh3oQHv4psF8mUXgX</recordid><startdate>20240414</startdate><enddate>20240414</enddate><creator>Fakhoury, Sarah</creator><creator>Naik, Aaditya</creator><creator>Sakkas, Georgios</creator><creator>Chakraborty, Saikat</creator><creator>Musuvathi, Madan</creator><creator>Lahiri, Shuvendu</creator><general>ACM</general><scope>6IE</scope><scope>6IL</scope><scope>CBEJK</scope><scope>RIE</scope><scope>RIL</scope><orcidid>https://orcid.org/0000-0002-8486-7749</orcidid><orcidid>https://orcid.org/0000-0002-3100-0455</orcidid><orcidid>https://orcid.org/0000-0002-2482-7892</orcidid><orcidid>https://orcid.org/0000-0002-6889-7171</orcidid><orcidid>https://orcid.org/0000-0002-1071-8038</orcidid><orcidid>https://orcid.org/0000-0002-4446-4777</orcidid></search><sort><creationdate>20240414</creationdate><title>Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation</title><author>Fakhoury, Sarah ; Naik, Aaditya ; Sakkas, Georgios ; Chakraborty, Saikat ; Musuvathi, Madan ; Lahiri, Shuvendu</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a1625-185e834bbbff14e53ee5689721d3303d07be0738599fd2662cd786cd93cb3e4d3</frbrgroupid><rsrctype>conference_proceedings</rsrctype><prefilter>conference_proceedings</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Accuracy</topic><topic>Codes</topic><topic>LLM4Code</topic><topic>Natural languages</topic><topic>Software engineering</topic><topic>User intent formulation</topic><topic>user study</topic><toplevel>online_resources</toplevel><creatorcontrib>Fakhoury, Sarah</creatorcontrib><creatorcontrib>Naik, Aaditya</creatorcontrib><creatorcontrib>Sakkas, Georgios</creatorcontrib><creatorcontrib>Chakraborty, Saikat</creatorcontrib><creatorcontrib>Musuvathi, Madan</creatorcontrib><creatorcontrib>Lahiri, Shuvendu</creatorcontrib><collection>IEEE Electronic Library (IEL) Conference Proceedings</collection><collection>IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume</collection><collection>IEEE Xplore All Conference Proceedings</collection><collection>IEEE</collection><collection>IEEE Proceedings Order Plans (POP All) 1998-Present</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Fakhoury, Sarah</au><au>Naik, Aaditya</au><au>Sakkas, Georgios</au><au>Chakraborty, Saikat</au><au>Musuvathi, Madan</au><au>Lahiri, Shuvendu</au><format>book</format><genre>proceeding</genre><ristype>CONF</ristype><atitle>Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation</atitle><btitle>2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)</btitle><stitle>ICSE-COMPANION</stitle><date>2024-04-14</date><risdate>2024</risdate><spage>390</spage><epage>391</epage><pages>390-391</pages><eissn>2574-1934</eissn><isbn>9798400705021</isbn><eisbn>9798400705021</eisbn><coden>IEEPAD</coden><abstract>We introduce a novel workflow, TiCoder, designed to enhance the trust and accuracy of LLM-based code generation through interactive and guided intent formalization. TiCoder partially formalizes ambiguous intent in natural language prompts by generating a set of tests to distinguish common divergent behaviours in generated code suggestions. We evaluate the code generation accuracy improvements provided by TiCoder at scale across four competitive LLMs, and evaluate the cost-benefit trade off of evaluating tests surfaced by TiCoder through a user study with 15 participants.</abstract><cop>New York, NY, USA</cop><pub>ACM</pub><doi>10.1145/3639478.3643525</doi><tpages>2</tpages><orcidid>https://orcid.org/0000-0002-8486-7749</orcidid><orcidid>https://orcid.org/0000-0002-3100-0455</orcidid><orcidid>https://orcid.org/0000-0002-2482-7892</orcidid><orcidid>https://orcid.org/0000-0002-6889-7171</orcidid><orcidid>https://orcid.org/0000-0002-1071-8038</orcidid><orcidid>https://orcid.org/0000-0002-4446-4777</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier ISBN: 9798400705021
ispartof 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2024, p.390-391
issn 2574-1934
language eng
recordid cdi_ieee_primary_10554982
source IEEE Xplore All Conference Series
subjects Accuracy
Codes
LLM4Code
Natural languages
Software engineering
User intent formulation
user study
title Exploring the Effectiveness of LLM based Test-driven Interactive Code Generation: User Study and Empirical Evaluation
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-07T14%3A12%3A58IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-acm_CHZPO&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=proceeding&rft.atitle=Exploring%20the%20Effectiveness%20of%20LLM%20based%20Test-driven%20Interactive%20Code%20Generation:%20User%20Study%20and%20Empirical%20Evaluation&rft.btitle=2024%20IEEE/ACM%2046th%20International%20Conference%20on%20Software%20Engineering:%20Companion%20Proceedings%20(ICSE-Companion)&rft.au=Fakhoury,%20Sarah&rft.date=2024-04-14&rft.spage=390&rft.epage=391&rft.pages=390-391&rft.eissn=2574-1934&rft.isbn=9798400705021&rft.coden=IEEPAD&rft_id=info:doi/10.1145/3639478.3643525&rft.eisbn=9798400705021&rft_dat=%3Cacm_CHZPO%3Eacm_books_10_1145_3639478_3643525%3C/acm_CHZPO%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-a1625-185e834bbbff14e53ee5689721d3303d07be0738599fd2662cd786cd93cb3e4d3%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_id=info:pmid/&rft_ieee_id=10554982&rfr_iscdi=true