chore: 저장소 구조 정리 및 문서화, 첫 커밋
- src/dist 산출물 분리 원칙 정리(.gitignore, .gitattributes) - 루트 및 주요 폴더(config/scripts/prompts/tests/src, 런타임 폴더 5종)에 안내용 README.md 추가 - CHANGELOG.md, LICENSE, docs/ops/05-release-and-versioning.md 추가 - docs/README.md 문서 지도 갱신
This commit is contained in:
Yun Chan
commit
56a6e2da93
159 changed files with 145825 additions and 0 deletions
625
docs/research/_raw/tdd-red/arxiv-2312-04687-llm4tdd.raw.html
Normal file
625
docs/research/_raw/tdd-red/arxiv-2312-04687-llm4tdd.raw.html
Normal file
|
|
@ -0,0 +1,625 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2312.04687] LLM4TDD: Best Practices for Test Driven Development Using Large Language Models</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2312.04687"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2312.04687: LLM4TDD: Best Practices for Test Driven Development Using Large Language Models"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="LLM4TDD: Best Practices for Test Driven Development Using Large Language Models" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2312.04687v1" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="In today's society, we are becoming increasingly dependent on software systems. However, we also constantly witness the negative impacts of buggy software. Program synthesis aims to improve software correctness by automatically generating the program given an outline of the expected behavior. For decades, program synthesis has been an active research field, with recent approaches looking to incorporate Large Language Models to help generate code. This paper explores the concept of LLM4TDD, where we guide Large Language Models to generate code iteratively using a test-driven development methodology. We conduct an empirical evaluation using ChatGPT and coding problems from LeetCode to investigate the impact of different test, prompt and problem attributes on the efficacy of LLM4TDD."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="LLM4TDD: Best Practices for Test Driven Development Using Large..."/>
|
||||
<meta name="twitter:description" content="In today's society, we are becoming increasingly dependent on software systems. However, we also constantly witness the negative impacts of buggy software. Program synthesis aims to improve..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="LLM4TDD: Best Practices for Test Driven Development Using Large Language Models" /><meta name="citation_author" content="Piya, Sanyogita" /><meta name="citation_author" content="Sullivan, Allison" /><meta name="citation_date" content="2023/12/07" /><meta name="citation_online_date" content="2023/12/07" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2312.04687" /><meta name="citation_arxiv_id" content="2312.04687" /><meta name="citation_abstract" content="In today's society, we are becoming increasingly dependent on software systems. However, we also constantly witness the negative impacts of buggy software. Program synthesis aims to improve software correctness by automatically generating the program given an outline of the expected behavior. For decades, program synthesis has been an active research field, with recent approaches looking to incorporate Large Language Models to help generate code. This paper explores the concept of LLM4TDD, where we guide Large Language Models to generate code iteratively using a test-driven development methodology. We conduct an empirical evaluation using ChatGPT and coding problems from LeetCode to investigate the impact of different test, prompt and problem attributes on the efficacy of LLM4TDD." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2312.04687"
|
||||
dc:identifier="/abs/2312.04687"
|
||||
dc:title="LLM4TDD: Best Practices for Test Driven Development Using Large Language Models"
|
||||
trackback:ping="/trackback/2312.04687" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2312.04687</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 7 Dec 2023]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>LLM4TDD: Best Practices for Test Driven Development Using Large Language Models</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Piya,+S" rel="nofollow">Sanyogita Piya</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Sullivan,+A" rel="nofollow">Allison Sullivan</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled LLM4TDD: Best Practices for Test Driven Development Using Large Language Models, by Sanyogita Piya and Allison Sullivan</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2312.04687">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2312.04687v1">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>In today's society, we are becoming increasingly dependent on software systems. However, we also constantly witness the negative impacts of buggy software. Program synthesis aims to improve software correctness by automatically generating the program given an outline of the expected behavior. For decades, program synthesis has been an active research field, with recent approaches looking to incorporate Large Language Models to help generate code. This paper explores the concept of LLM4TDD, where we guide Large Language Models to generate code iteratively using a test-driven development methodology. We conduct an empirical evaluation using ChatGPT and coding problems from LeetCode to investigate the impact of different test, prompt and problem attributes on the efficacy of LLM4TDD.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"><tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Machine Learning (cs.LG)</td>
|
||||
</tr><tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2312.04687">arXiv:2312.04687</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2312.04687v1">arXiv:2312.04687v1</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2312.04687" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2312.04687</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Allison Sullivan [<a href="/show-email/d9f20772/2312.04687" rel="nofollow">view email</a>] <br/> <strong>[v1]</strong>
|
||||
Thu, 7 Dec 2023 20:37:54 UTC (714 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled LLM4TDD: Best Practices for Test Driven Development Using Large Language Models, by Sanyogita Piya and Allison Sullivan</div><li><a href="/pdf/2312.04687" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2312.04687v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2312.04687" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2312.04687&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2312.04687&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2023-12" rel="nofollow">2023-12</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2312.04687?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2312.04687?context=cs.LG" rel="nofollow">cs.LG</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2312.04687">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2312.04687" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2312.04687" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2312.04687&description=LLM4TDD: Best Practices for Test Driven Development Using Large Language Models"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2312.04687&title=LLM4TDD: Best Practices for Test Driven Development Using Large Language Models"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2312.04687" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -0,0 +1,629 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2505.09027] Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2505.09027"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2505.09027: Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2505.09027v1" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code..."/>
|
||||
<meta name="twitter:description" content="We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation" /><meta name="citation_author" content="Cui, Yi" /><meta name="citation_date" content="2025/05/13" /><meta name="citation_online_date" content="2025/05/13" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2505.09027" /><meta name="citation_arxiv_id" content="2505.09027" /><meta name="citation_abstract" content="We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2505.09027"
|
||||
dc:identifier="/abs/2505.09027"
|
||||
dc:title="Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation"
|
||||
trackback:ping="/trackback/2505.09027" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2505.09027</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 13 May 2025]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Cui,+Y" rel="nofollow">Yi Cui</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation, by Yi Cui</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2505.09027">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2505.09027v1">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"> <tr>
|
||||
<td class="tablecell label">Comments:</td>
|
||||
<td class="tablecell comments mathjax">arXiv admin note: text overlap with <a href="https://arxiv.org/abs/2409.05177" data-arxiv-id="2409.05177" class="link-https">arXiv:2409.05177</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Artificial Intelligence (cs.AI)</td>
|
||||
</tr><tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2505.09027">arXiv:2505.09027</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2505.09027v1">arXiv:2505.09027v1</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2505.09027" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2505.09027</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Yi Cui [<a href="/show-email/6b42e462/2505.09027" rel="nofollow">view email</a>] <br/> <strong>[v1]</strong>
|
||||
Tue, 13 May 2025 23:47:12 UTC (1,298 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation, by Yi Cui</div><li><a href="/pdf/2505.09027" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2505.09027v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2505.09027" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2505.09027&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2505.09027&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2025-05" rel="nofollow">2025-05</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2505.09027?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2505.09027?context=cs.AI" rel="nofollow">cs.AI</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2505.09027">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2505.09027" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2505.09027" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2505.09027&description=Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2505.09027&title=Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2505.09027" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
633
docs/research/_raw/tdd-red/arxiv-2506-09289-utboost.raw.html
Normal file
633
docs/research/_raw/tdd-red/arxiv-2506-09289-utboost.raw.html
Normal file
|
|
@ -0,0 +1,633 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2506.09289] UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2506.09289"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2506.09289: UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2506.09289v1" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench"/>
|
||||
<meta name="twitter:description" content="The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench" /><meta name="citation_author" content="Yu, Boxi" /><meta name="citation_author" content="Zhu, Yuxuan" /><meta name="citation_author" content="He, Pinjia" /><meta name="citation_author" content="Kang, Daniel" /><meta name="citation_date" content="2025/06/10" /><meta name="citation_online_date" content="2025/06/10" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2506.09289" /><meta name="citation_arxiv_id" content="2506.09289" /><meta name="citation_abstract" content="The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2506.09289"
|
||||
dc:identifier="/abs/2506.09289"
|
||||
dc:title="UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench"
|
||||
trackback:ping="/trackback/2506.09289" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2506.09289</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 10 Jun 2025]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Yu,+B" rel="nofollow">Boxi Yu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Zhu,+Y" rel="nofollow">Yuxuan Zhu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=He,+P" rel="nofollow">Pinjia He</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Kang,+D" rel="nofollow">Daniel Kang</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench, by Boxi Yu and 3 other authors</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2506.09289">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2506.09289v1">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"><tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Computation and Language (cs.CL)</td>
|
||||
</tr> <tr>
|
||||
<td class="tablecell label"><abbr title="Association of Computing Machinery Classification">ACM</abbr> classes:</td>
|
||||
<td class="tablecell acm-classes">D.0; I.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2506.09289">arXiv:2506.09289</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2506.09289v1">arXiv:2506.09289v1</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2506.09289" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2506.09289</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr> <tr>
|
||||
<td class="tablecell label">Journal reference:</td>
|
||||
<td class="tablecell jref">ACL 2025</td>
|
||||
</tr>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Boxi Yu [<a href="/show-email/40665989/2506.09289" rel="nofollow">view email</a>] <br/> <strong>[v1]</strong>
|
||||
Tue, 10 Jun 2025 22:56:49 UTC (223 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench, by Boxi Yu and 3 other authors</div><li><a href="/pdf/2506.09289" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2506.09289v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2506.09289" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2506.09289&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2506.09289&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2025-06" rel="nofollow">2025-06</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2506.09289?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2506.09289?context=cs.CL" rel="nofollow">cs.CL</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2506.09289">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2506.09289" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2506.09289" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2506.09289&description=UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2506.09289&title=UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2506.09289" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -0,0 +1,629 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2602.07900] Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2602.07900"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2602.07900: Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2602.07900v2" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget?
|
||||
To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Rethinking the Value of Agent-Generated Tests for LLM-Based..."/>
|
||||
<meta name="twitter:description" content="Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents" /><meta name="citation_author" content="Chen, Zhi" /><meta name="citation_author" content="Sun, Zhensu" /><meta name="citation_author" content="Shi, Yuling" /><meta name="citation_author" content="Peng, Chao" /><meta name="citation_author" content="Gu, Xiaodong" /><meta name="citation_author" content="Lo, David" /><meta name="citation_author" content="Jiang, Lingxiao" /><meta name="citation_date" content="2026/02/08" /><meta name="citation_online_date" content="2026/04/09" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2602.07900" /><meta name="citation_arxiv_id" content="2602.07900" /><meta name="citation_abstract" content="Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2602.07900"
|
||||
dc:identifier="/abs/2602.07900"
|
||||
dc:title="Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents"
|
||||
trackback:ping="/trackback/2602.07900" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2602.07900</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 8 Feb 2026 (<a href="https://arxiv.org/abs/2602.07900v1">v1</a>), last revised 9 Apr 2026 (this version, v2)]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Chen,+Z" rel="nofollow">Zhi Chen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Sun,+Z" rel="nofollow">Zhensu Sun</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Shi,+Y" rel="nofollow">Yuling Shi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Peng,+C" rel="nofollow">Chao Peng</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Gu,+X" rel="nofollow">Xiaodong Gu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Lo,+D" rel="nofollow">David Lo</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Jiang,+L" rel="nofollow">Lingxiao Jiang</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, by Zhi Chen and 6 other authors</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2602.07900">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2602.07900v2">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking <a href="http://agents.This" rel="external noopener nofollow" class="link-external link-http">this http URL</a> raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget?
|
||||
<br>To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"><tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Artificial Intelligence (cs.AI)</td>
|
||||
</tr><tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2602.07900">arXiv:2602.07900</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2602.07900v2">arXiv:2602.07900v2</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2602.07900" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2602.07900</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Zhi Chen [<a href="/show-email/c1d11f2e/2602.07900" rel="nofollow">view email</a>] <br/> <strong><a href="/abs/2602.07900v1" rel="nofollow">[v1]</a></strong>
|
||||
Sun, 8 Feb 2026 10:26:31 UTC (372 KB)<br/>
|
||||
<strong>[v2]</strong>
|
||||
Thu, 9 Apr 2026 13:23:28 UTC (1,239 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents, by Zhi Chen and 6 other authors</div><li><a href="/pdf/2602.07900" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2602.07900v2" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2602.07900" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2602.07900&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2602.07900&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2026-02" rel="nofollow">2026-02</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2602.07900?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2602.07900?context=cs.AI" rel="nofollow">cs.AI</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2602.07900">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2602.07900" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2602.07900" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2602.07900&description=Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2602.07900&title=Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2602.07900" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -0,0 +1,633 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2603.08806] Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2603.08806"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2603.08806: Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2603.08806v1" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at https://github.com/f-labs-io/tdad-paper-code."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using..."/>
|
||||
<meta name="twitter:description" content="We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications" /><meta name="citation_author" content="Rehan, Tzafrir" /><meta name="citation_date" content="2026/03/09" /><meta name="citation_online_date" content="2026/03/09" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2603.08806" /><meta name="citation_arxiv_id" content="2603.08806" /><meta name="citation_abstract" content="We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at https://github.com/f-labs-io/tdad-paper-code." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2603.08806"
|
||||
dc:identifier="/abs/2603.08806"
|
||||
dc:title="Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications"
|
||||
trackback:ping="/trackback/2603.08806" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2603.08806</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 9 Mar 2026]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Rehan,+T" rel="nofollow">Tzafrir Rehan</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications, by Tzafrir Rehan</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2603.08806">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2603.08806v1">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at <a href="https://github.com/f-labs-io/tdad-paper-code" rel="external noopener nofollow" class="link-external link-https">this https URL</a>.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"> <tr>
|
||||
<td class="tablecell label">Comments:</td>
|
||||
<td class="tablecell comments mathjax">9 pages, 2 figures, open benchmark at <a href="https://github.com/f-labs-io/tdad-paper-code" rel="external noopener nofollow" class="link-external link-https">this https URL</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Artificial Intelligence (cs.AI)</td>
|
||||
</tr> <tr>
|
||||
<td class="tablecell label"><abbr title="Association of Computing Machinery Classification">ACM</abbr> classes:</td>
|
||||
<td class="tablecell acm-classes">I.2.2; I.2.11; D.2.1; D.2.5</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2603.08806">arXiv:2603.08806</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2603.08806v1">arXiv:2603.08806v1</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2603.08806" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2603.08806</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Tzafrir Rehan [<a href="/show-email/60ce6bdb/2603.08806" rel="nofollow">view email</a>] <br/> <strong>[v1]</strong>
|
||||
Mon, 9 Mar 2026 18:04:54 UTC (242 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications, by Tzafrir Rehan</div><li><a href="/pdf/2603.08806" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2603.08806v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2603.08806" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by-sa/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-sa-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2603.08806&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2603.08806&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2026-03" rel="nofollow">2026-03</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2603.08806?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2603.08806?context=cs.AI" rel="nofollow">cs.AI</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2603.08806">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2603.08806" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2603.08806" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2603.08806&description=Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2603.08806&title=Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2603.08806" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
639
docs/research/_raw/tdd-red/arxiv-2603-17973-tdad.raw.html
Normal file
639
docs/research/_raw/tdd-red/arxiv-2603-17973-tdad.raw.html
Normal file
|
|
@ -0,0 +1,639 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head><script>document.documentElement.classList.add('js');</script> <title>[2603.17973] TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
|
||||
<link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
|
||||
<link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
|
||||
<meta name="msapplication-TileColor" content="#da532c">
|
||||
<meta name="theme-color" content="#ffffff">
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20260318" />
|
||||
<link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
|
||||
<link rel="stylesheet" type="text/css" media="screen" href="/static/base/1.0.1/css/arxiv-header-footer.css?v=20260626" />
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/accordion.js" ></script>
|
||||
<script language="javascript" src="/static/browse/0.3.4/js/optin-modal.js?v=20250819"></script>
|
||||
|
||||
<link rel="canonical" href="https://arxiv.org/abs/2603.17973"/>
|
||||
<meta name="description" content="Abstract page for arXiv paper 2603.17973: TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis"><meta property="og:type" content="website" />
|
||||
<meta property="og:site_name" content="arXiv.org" />
|
||||
<meta property="og:title" content="TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis" />
|
||||
<meta property="og:url" content="https://arxiv.org/abs/2603.17973v2" />
|
||||
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
|
||||
<meta property="og:image:width" content="1200" />
|
||||
<meta property="og:image:height" content="700" />
|
||||
<meta property="og:image:alt" content="arXiv logo"/>
|
||||
<meta property="og:description" content="AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD."/>
|
||||
<meta name="twitter:site" content="@arxiv"/>
|
||||
<meta name="twitter:card" content="summary"/>
|
||||
<meta name="twitter:title" content="TDAD: Test-Driven Agentic Development - Reducing Code Regressions..."/>
|
||||
<meta name="twitter:description" content="AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on..."/>
|
||||
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
|
||||
<meta name="twitter:image:alt" content="arXiv logo"/>
|
||||
<link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/> <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script> <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
|
||||
<script src="/static/browse/0.3.4/js/toggle-labs.js?20241022" type="text/javascript"></script>
|
||||
<script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis" /><meta name="citation_author" content="Alonso, Pepe" /><meta name="citation_author" content="Yovine, Sergio" /><meta name="citation_author" content="Braberman, Victor A." /><meta name="citation_date" content="2026/03/18" /><meta name="citation_online_date" content="2026/03/19" /><meta name="citation_pdf_url" content="https://arxiv.org/pdf/2603.17973" /><meta name="citation_arxiv_id" content="2603.17973" /><meta name="citation_abstract" content="AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD." />
|
||||
</head>
|
||||
|
||||
<body ><div class="flex-wrap-footer">
|
||||
<a href="#content" class="ds-skip-link">Skip to main content</a>
|
||||
|
||||
|
||||
|
||||
<header class="ds-site-header">
|
||||
<a aria-hidden="true" tabindex="-1" href="https://arxiv.org/IgnoreMe" class="is-sr-only"></a>
|
||||
|
||||
<a href="https://arxiv.org/" class="ds-site-header-logo" aria-label="archive home">
|
||||
<img src="/static/base/1.0.1/images/arxiv-logo-primary-light.svg" alt="archive">
|
||||
</a>
|
||||
|
||||
<button type="button" id="ds-nav-toggle" class="ds-site-header-nav-toggle"
|
||||
aria-label="Open menu" aria-controls="ds-site-header-nav" aria-expanded="false">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<line x1="3" y1="6" x2="21" y2="6"/>
|
||||
<line x1="3" y1="12" x2="21" y2="12"/>
|
||||
<line x1="3" y1="18" x2="21" y2="18"/>
|
||||
</svg>
|
||||
</button>
|
||||
|
||||
<nav class="ds-site-header-nav" id="ds-site-header-nav" aria-label="Main navigation"><a id="arxiv-search-toggle" href="https://arxiv.org/search"
|
||||
aria-controls="arxiv-search-overlay" aria-expanded="false">
|
||||
<svg class="ds-nav-icon" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true" focusable="false">
|
||||
<circle cx="11" cy="11" r="8"/>
|
||||
<line x1="21" y1="21" x2="16.65" y2="16.65"/>
|
||||
</svg>
|
||||
Search
|
||||
</a>
|
||||
<a href="https://arxiv.org/user/create">Submit</a>
|
||||
<a href="https://info.arxiv.org/about/donate.html">Donate</a>
|
||||
<span class="ds-site-header-divider" aria-hidden="true"></span>
|
||||
<a href="https://arxiv.org/login" class="ds-site-header-login">Log in</a>
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<div class="arxiv-search-overlay" id="arxiv-search-overlay" hidden>
|
||||
<div class="arxiv-search-panel" role="search">
|
||||
<form method="GET" action="https://arxiv.org/search">
|
||||
<label for="arxiv-search-input" class="is-sr-only">Search arXiv</label>
|
||||
<input type="text" name="query" id="arxiv-search-input" autocomplete="off"
|
||||
placeholder="Search papers by title, author, abstract, or ID...">
|
||||
<input type="hidden" name="searchtype" value="all">
|
||||
<input type="hidden" name="source" value="header">
|
||||
</form>
|
||||
<div class="arxiv-search-hint">
|
||||
Press Enter to search · <a href="https://arxiv.org/search/advanced">Advanced search</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<main>
|
||||
<div id="content">
|
||||
<!--
|
||||
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
|
||||
<rdf:Description
|
||||
rdf:about="/abs/2603.17973"
|
||||
dc:identifier="/abs/2603.17973"
|
||||
dc:title="TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis"
|
||||
trackback:ping="/trackback/2603.17973" />
|
||||
</rdf:RDF>
|
||||
--><div id="abs-outer">
|
||||
|
||||
<div class="leftcolumn">
|
||||
<div class="subheader">
|
||||
<h1>Computer Science > Software Engineering</h1>
|
||||
</div>
|
||||
|
||||
<div class="header-breadcrumbs-mobile">
|
||||
<strong>arXiv:2603.17973</strong> (cs)
|
||||
</div>
|
||||
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
|
||||
<div id="content-inner">
|
||||
<div id="abs">
|
||||
<div class="dateline">
|
||||
[Submitted on 18 Mar 2026 (<a href="https://arxiv.org/abs/2603.17973v1">v1</a>), last revised 19 Mar 2026 (this version, v2)]</div>
|
||||
<h1 class="title mathjax"><span class="descriptor">Title:</span>TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis</h1>
|
||||
<div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&query=Alonso,+P" rel="nofollow">Pepe Alonso</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Yovine,+S" rel="nofollow">Sergio Yovine</a>, <a href="https://arxiv.org/search/cs?searchtype=author&query=Braberman,+V+A" rel="nofollow">Victor A. Braberman</a></div> <div id="download-button-info" hidden>View a PDF of the paper titled TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis, by Pepe Alonso and 1 other authors</div>
|
||||
<a class="mobile-submission-download" href="/pdf/2603.17973">View PDF</a>
|
||||
<a class="mobile-submission-download" href="https://arxiv.org/html/2603.17973v2">HTML (experimental)</a>
|
||||
|
||||
|
||||
|
||||
<blockquote class="abstract mathjax">
|
||||
<span class="descriptor">Abstract:</span>AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at <a href="https://github.com/pepealonso95/TDAD" rel="external noopener nofollow" class="link-external link-https">this https URL</a>.
|
||||
</blockquote>
|
||||
|
||||
<!--CONTEXT-->
|
||||
<div class="metatable">
|
||||
<table summary="Additional metadata"> <tr>
|
||||
<td class="tablecell label">Comments:</td>
|
||||
<td class="tablecell comments mathjax">Toolpaper, 7 pages, 7 tables, 3 figures, 1 algorithm. Submitted to ACM AIWare 2026 (Data and Benchmark Track)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Subjects:</td>
|
||||
<td class="tablecell subjects">
|
||||
<span class="primary-subject">Software Engineering (cs.SE)</span>; Artificial Intelligence (cs.AI)</td>
|
||||
</tr> <tr>
|
||||
<td class="tablecell label"><abbr title="Mathematical Subject Classification">MSC</abbr> classes:</td>
|
||||
<td class="tablecell msc-classes">68N30, 68T05</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"><abbr title="Association of Computing Machinery Classification">ACM</abbr> classes:</td>
|
||||
<td class="tablecell acm-classes">D.2.5; I.2.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label">Cite as:</td>
|
||||
<td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2603.17973">arXiv:2603.17973</a> [cs.SE]</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxividv">(or <span class="arxivid">
|
||||
<a href="https://arxiv.org/abs/2603.17973v2">arXiv:2603.17973v2</a> [cs.SE]</span> for this version)
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="tablecell label"> </td>
|
||||
<td class="tablecell arxivdoi"> <a href="https://doi.org/10.48550/arXiv.2603.17973" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2603.17973</a><div class="button-and-tooltip">
|
||||
<button class="more-info" aria-describedby="more-info-desc-1">
|
||||
<svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
|
||||
<span class="visually-hidden">Focus to learn more</span>
|
||||
</button>
|
||||
<!-- tooltip description -->
|
||||
<div role="tooltip" id="more-info-desc-1">
|
||||
<span class="left-corner"></span> arXiv-issued DOI via DataCite</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr></table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="submission-history">
|
||||
<h2>Submission history</h2> From: Pepe Alonso [<a href="/show-email/26108619/2603.17973" rel="nofollow">view email</a>] <br/> <strong><a href="/abs/2603.17973v1" rel="nofollow">[v1]</a></strong>
|
||||
Wed, 18 Mar 2026 17:38:22 UTC (18 KB)<br/>
|
||||
<strong>[v2]</strong>
|
||||
Thu, 19 Mar 2026 17:12:15 UTC (19 KB)<br/>
|
||||
</div>
|
||||
</div>
|
||||
<!--end leftcolumn-->
|
||||
<div class="extra-services"> <div class="full-text">
|
||||
<a name="other"></a>
|
||||
<span class="descriptor">Full-text links:</span>
|
||||
<h2>Access Paper:</h2>
|
||||
<ul>
|
||||
<div id="download-button-info" hidden>
|
||||
View a PDF of the paper titled TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis, by Pepe Alonso and 1 other authors</div><li><a href="/pdf/2603.17973" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2603.17973v2" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2603.17973" class="abs-button download-eprint">TeX Source
|
||||
</a></li></ul>
|
||||
<div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
|
||||
<img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
|
||||
<span>view license</span>
|
||||
</a></div>
|
||||
</div>
|
||||
<!--end full-text--> <div class="browse">
|
||||
<h3 class="browse-context-heading">Current browse context:</h3>
|
||||
<div class="current">cs.SE</div>
|
||||
|
||||
<div class="prevnext">
|
||||
<span class="arrow">
|
||||
<a class="abs-button prev-url" href="/prevnext?id=2603.17973&function=prev&context=cs.SE"
|
||||
accesskey="p" title="previous in cs.SE (accesskey p)" rel="nofollow">< prev</a>
|
||||
</span>
|
||||
<span class="is-hidden-mobile"> | </span> <span class="arrow">
|
||||
<a class="abs-button next-url" href="/prevnext?id=2603.17973&function=next&context=cs.SE" accesskey="n"
|
||||
title="next in cs.SE (accesskey n)" rel="nofollow">next ></a>
|
||||
</span><br/>
|
||||
</div><div class="list">
|
||||
<a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.SE/new" rel="nofollow">new</a>
|
||||
<span class="is-hidden-mobile"> | </span>
|
||||
<a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.SE/recent" rel="nofollow">recent</a>
|
||||
<span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.SE/2026-03" rel="nofollow">2026-03</a>
|
||||
</div><div class="abs-switch-cat">
|
||||
Change to browse by:
|
||||
<div class="switch context-change">
|
||||
<a href="/abs/2603.17973?context=cs" rel="nofollow">cs</a><br class="is-hidden-mobile">
|
||||
<a class="subclass" href="/abs/2603.17973?context=cs.AI" rel="nofollow">cs.AI</a><br class="is-hidden-mobile">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="extra-ref-cite">
|
||||
<h3>References & Citations</h3>
|
||||
<ul>
|
||||
<li><a class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2603.17973">NASA ADS</a></li><li><a class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2603.17973" target="_blank" rel="noopener">Google Scholar</a></li>
|
||||
<li><a class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2603.17973" target="_blank" rel="noopener">Semantic Scholar</a></li>
|
||||
</ul>
|
||||
<div style="clear:both;"></div>
|
||||
</div>
|
||||
|
||||
<div class='extra-ref-cite'>
|
||||
<button type="button" id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</button>
|
||||
<span id='bib-cite-loading' hidden='true'>Loading...</span>
|
||||
</div>
|
||||
|
||||
<div id='bib-cite-modal' class='bib-modal' hidden='true'>
|
||||
<div class='bib-modal-content'>
|
||||
<div class='bib-modal-title'>
|
||||
<h2>BibTeX formatted citation</h2>
|
||||
<button type="button" class='bib-modal-close' aria-label="Close">×</button>
|
||||
</div>
|
||||
<div>
|
||||
<textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
|
||||
</div>
|
||||
<div>
|
||||
<span>Data provided by: </span>
|
||||
<a id='bib-cite-source-api'></a>
|
||||
</div>
|
||||
</div>
|
||||
</div><div class="bookmarks">
|
||||
<div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2603.17973&description=TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis"
|
||||
title="Bookmark on BibSonomy">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
|
||||
alt="BibSonomy"/>
|
||||
</a>
|
||||
<a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2603.17973&title=TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis"
|
||||
title="Bookmark on Reddit">
|
||||
<img src="/static/browse/0.3.4/images/icons/social/reddit.png"
|
||||
alt="Reddit"/>
|
||||
</a>
|
||||
</div> </div>
|
||||
<!--end extra-services-->
|
||||
<!-- LABS AREA -->
|
||||
<div id="labstabs">
|
||||
<div class="labstabs" role="tablist" aria-label="arXivLabs tools"><input type="radio" name="tabs" id="tabone"checked="checked"class="labs-tab-input">
|
||||
<label for="tabone" role="tab" aria-selected="true" aria-controls="tabpanel-one" id="tab-label-one">Bibliographic Tools</label>
|
||||
<div class="tab labs-display-bib" role="tabpanel" id="tabpanel-one" aria-labelledby="tab-label-one">
|
||||
<h1>Bibliographic and Citation Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="bibex-toggle" type="checkbox" class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/bibex/bibex.js?20241202">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Bibliographic Explorer Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="connectedpapers-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
|
||||
aria-labelledby="label-for-connected-papers">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Connected Papers Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
|
||||
</div>
|
||||
</div><div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="litmaps-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
|
||||
aria-labelledby="label-for-litmaps">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Litmaps Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="scite-toggle"
|
||||
type="checkbox"
|
||||
class="lab-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
|
||||
aria-labelledby="label-for-scite">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">scite.ai Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="labs-content-placeholder labs-display" style="display: none;"></div>
|
||||
<div style="min-height: 15px" id="connectedpapers-output"></div>
|
||||
<div style="min-height: 15px" id="litmaps-open-in"></div>
|
||||
<div style="min-height: 15px" id="scite-open-in"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="tabtwo" class="labs-tab-input">
|
||||
<label for="tabtwo" role="tab" aria-selected="false" aria-controls="tabpanel-two" id="tab-label-two">Code, Data, Media</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-two" aria-labelledby="tab-label-two">
|
||||
<h1>Code, Data and Media Associated with this Article</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="alphaxiv-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/alphaxiv.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-alphaxiv">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">alphaXiv Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-alphaxiv">alphaXiv</span> <em>(<a href="https://alphaxiv.org/" target="_blank">What is alphaXiv?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="catalyzex-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/catalyzex.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Links to Code Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="dagshub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/dagshub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">DagsHub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="gotitpub-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/gotitpub.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">GotitPub Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="huggingface-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/huggingface.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-huggingface">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Huggingface Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-huggingface">Hugging Face</span> <em>(<a href="https://huggingface.co/huggingface" target="_blank">What is Huggingface?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="sciencecast-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/sciencecast.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">ScienceCast Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="alphaxiv-output" style="display:none"></div>
|
||||
<div id="catalyzex-output" style="display:none"></div>
|
||||
<div id="dagshub-output" style="display:none"></div>
|
||||
<div id="gotitpub-output" style="display:none"></div>
|
||||
<div id="sciencecast-output" style="display:none"></div>
|
||||
<div id="huggingface-output" style="display:none"></div>
|
||||
</div>
|
||||
|
||||
|
||||
<input type="radio" name="tabs" id="labstabs-demos-input" class="labs-tab-input">
|
||||
<label for="labstabs-demos-input" id="labstabs-demos-label" role="tab" aria-selected="false" aria-controls="tabpanel-demos">Demos</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-demos" aria-labelledby="labstabs-demos-label">
|
||||
<h1>Demos</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="replicate-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/replicate.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Replicate Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="spaces-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/spaces.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input
|
||||
id="txyz-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/txyz.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Spaces Toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="replicate-output"></div>
|
||||
<div id="spaces-output"></div>
|
||||
<div id="txyz-output"></div>
|
||||
</div>
|
||||
<input type="radio" name="tabs" id="tabfour" class="labs-tab-input">
|
||||
<label for="tabfour" role="tab" aria-selected="false" aria-controls="tabpanel-four" id="tab-label-four">Related Papers</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-four" aria-labelledby="tab-label-four">
|
||||
<h1>Recommenders and Search Tools</h1>
|
||||
<div class="toggle">
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="influenceflower-toggle"
|
||||
data-script-url="/static/browse/0.3.4/js/influenceflower.js"
|
||||
type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Link to Influence Flower</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
|
||||
</div>
|
||||
</div>
|
||||
<div class="columns is-mobile lab-row">
|
||||
<div class="column lab-switch">
|
||||
<label class="switch">
|
||||
<input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
|
||||
<span class="slider"></span>
|
||||
<span class="is-sr-only">Core recommender toggle</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="column lab-name">
|
||||
<span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
|
||||
</div>
|
||||
</div></div>
|
||||
<div id="influenceflower-output"></div>
|
||||
<div id="influenceflower-output-graph" style="display:none">
|
||||
<ul class="flower-tabs">
|
||||
<li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
|
||||
<li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
|
||||
</ul>
|
||||
<div class="flower-tab-content">
|
||||
<div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
|
||||
<div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
|
||||
<div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
|
||||
<div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="coreRecommenderOutput"></div>
|
||||
<div id="iarxivOutput"></div>
|
||||
</div>
|
||||
|
||||
<input type="radio" name="tabs" id="tabfive" class="labs-tab-input">
|
||||
<label for="tabfive" role="tab" aria-selected="false" aria-controls="tabpanel-five" id="tab-label-five">
|
||||
About arXivLabs
|
||||
</label>
|
||||
<div class="tab" role="tabpanel" id="tabpanel-five" aria-labelledby="tab-label-five">
|
||||
<div class="columns">
|
||||
<div class="column">
|
||||
<h1>arXivLabs: experimental projects with community collaborators</h1>
|
||||
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
|
||||
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
|
||||
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
|
||||
</div>
|
||||
<div class="column is-narrow is-full-mobile">
|
||||
<p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<!-- END LABS AREA -->
|
||||
<div class="endorsers">
|
||||
<a href="/auth/show-endorsers/2603.17973" class="endorser-who" rel="nofollow">Which authors of this paper are endorsers?</a> |
|
||||
<a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
|
||||
<span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
|
||||
</div>
|
||||
<script type="text/javascript" language="javascript">mathjaxToggle();</script>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
<footer class="ds-site-footer">
|
||||
<div class="ds-site-footer-grid">
|
||||
<div class="ds-site-footer-main">
|
||||
<div class="ds-site-footer-ack">
|
||||
We gratefully acknowledge support from
|
||||
our <strong>major funders</strong>,
|
||||
<a href="https://info.arxiv.org/about/ourmembers.html"><strong>member institutions</strong></a><span class="ack-member-inline" hidden>, <strong></strong></span>,
|
||||
and all contributors.
|
||||
</div>
|
||||
<nav class="ds-site-footer-links" aria-label="Site navigation">
|
||||
<a href="https://info.arxiv.org/about">About</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help">Help</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/contact.html">Contact</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/subscribe">Subscribe</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/license/index.html">Copyright</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://info.arxiv.org/help/web_accessibility.html">Accessibility</a>
|
||||
<span class="ds-site-footer-sep" aria-hidden="true">·</span>
|
||||
<a href="https://status.arxiv.org" target="_blank" rel="noopener noreferrer">Operational Status<span class="is-sr-only"> (opens in new tab)</span></a>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
<div class="ds-site-footer-funders" aria-label="Major funders">
|
||||
<div class="ds-site-footer-funders-label">Major funding support from</div>
|
||||
<div class="ds-site-footer-funders-logos">
|
||||
<a class="ds-funder-link" href="https://www.simonsfoundation.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation.png" alt="Simons Foundation">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.sfi.org.bm/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/simons-foundation-international.png" alt="Simons Foundation International">
|
||||
</a>
|
||||
<a class="ds-funder-link" href="https://www.schmidtsciences.org/" target="_blank" rel="noopener noreferrer">
|
||||
<img class="ds-funder-logo" src="/static/base/1.0.1/images/funders/schmidt-sciences.png" alt="Schmidt Sciences">
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer> </div>
|
||||
|
||||
<script src="/static/base/1.0.1/js/arxiv-header.js?v=20260626"></script>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
1682
docs/research/_raw/tdd-red/nng-form-placeholders.raw.html
Normal file
1682
docs/research/_raw/tdd-red/nng-form-placeholders.raw.html
Normal file
File diff suppressed because one or more lines are too long
1556
docs/research/_raw/tdd-red/nng-form-white-space.raw.html
Normal file
1556
docs/research/_raw/tdd-red/nng-form-white-space.raw.html
Normal file
File diff suppressed because one or more lines are too long
1704
docs/research/_raw/tdd-red/nng-good-visual-design.raw.html
Normal file
1704
docs/research/_raw/tdd-red/nng-good-visual-design.raw.html
Normal file
File diff suppressed because one or more lines are too long
1579
docs/research/_raw/tdd-red/nng-web-form-design.raw.html
Normal file
1579
docs/research/_raw/tdd-red/nng-web-form-design.raw.html
Normal file
File diff suppressed because one or more lines are too long
110
docs/research/_raw/tdd-red/playwright-best-practices.raw.html
Normal file
110
docs/research/_raw/tdd-red/playwright-best-practices.raw.html
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1,190 @@
|
|||
---
|
||||
description: Review UI code for Vercel Web Interface Guidelines compliance
|
||||
argument-hint: <file-or-pattern>
|
||||
---
|
||||
|
||||
# Web Interface Guidelines
|
||||
|
||||
Review these files for compliance: $ARGUMENTS
|
||||
|
||||
Read files, check against rules below. Output concise but comprehensive—sacrifice grammar for brevity. High signal-to-noise.
|
||||
|
||||
## Rules
|
||||
|
||||
### Accessibility
|
||||
|
||||
- Icon-only buttons need `aria-label`
|
||||
- Form controls need `<label>` or `aria-label`
|
||||
- Interactive elements need keyboard handlers (`onKeyDown`/`onKeyUp`)
|
||||
- `<button>` for actions, `<a>`/`<Link>` for navigation (not `<div onClick>`)
|
||||
- Images need `alt` (or `alt=""` if decorative)
|
||||
- Decorative icons need `aria-hidden="true"`
|
||||
- Async updates (toasts, validation) need `aria-live="polite"`
|
||||
- Use semantic HTML (`<button>`, `<a>`, `<label>`, `<table>`) before ARIA
|
||||
- Headings hierarchical `<h1>`–`<h6>`; include skip link for main content
|
||||
- `scroll-margin-top` on heading anchors
|
||||
- Meaningful media needs captions, transcripts, or descriptions as applicable
|
||||
- Media controls need keyboard support; decorative media needs assistive-tech hiding
|
||||
|
||||
### Focus States
|
||||
|
||||
- Interactive elements need visible focus: `focus-visible:ring-*` or equivalent
|
||||
- Never `outline-none` / `outline: none` without focus replacement
|
||||
- Use `:focus-visible` over `:focus` (avoid focus ring on click)
|
||||
- Group focus with `:focus-within` for compound controls
|
||||
- Sticky headers/footers/overlays must not cover the focused element
|
||||
|
||||
### Forms
|
||||
|
||||
- Inputs need `autocomplete` and meaningful `name`
|
||||
- Use correct `type` (`email`, `tel`, `url`, `number`) and `inputmode`
|
||||
- Never block paste (`onPaste` + `preventDefault`)
|
||||
- Labels clickable (`htmlFor` or wrapping control)
|
||||
- Disable spellcheck on emails, codes, usernames (`spellCheck={false}`)
|
||||
- Checkboxes/radios: label + control share single hit target (no dead zones)
|
||||
- Submit button stays enabled until request starts; spinner during request
|
||||
- Errors inline next to fields; focus first error on submit
|
||||
- Placeholders end with `…` and show example pattern
|
||||
- `autocomplete="off"` on non-auth fields to avoid password manager triggers
|
||||
- Warn before navigation with unsaved changes (`beforeunload` or router guard)
|
||||
|
||||
### Animation
|
||||
|
||||
- Honor `prefers-reduced-motion` (provide reduced variant or disable)
|
||||
- Animate `transform`/`opacity` only (compositor-friendly)
|
||||
- Never `transition: all`—list properties explicitly
|
||||
- Set correct `transform-origin`
|
||||
- SVG: transforms on `<g>` wrapper with `transform-box: fill-box; transform-origin: center`
|
||||
- Animations interruptible—respond to user input mid-animation
|
||||
- Autoplay motion >5 seconds alongside other content needs pause, stop, or hide controls
|
||||
- Muted decorative loops must stop under `prefers-reduced-motion`
|
||||
|
||||
### Typography
|
||||
|
||||
- `…` not `...`
|
||||
- Curly quotes `“` `”` not straight `"`
|
||||
- Non-breaking spaces: `10 MB`, `⌘ K`, brand names
|
||||
- Loading states end with `…`: `"Loading…"`, `"Saving…"`
|
||||
- `font-variant-numeric: tabular-nums` for number columns/comparisons
|
||||
- Use `text-wrap: balance` or `text-pretty` on headings (prevents widows)
|
||||
|
||||
### Content Handling
|
||||
|
||||
- Text containers handle long content: `truncate`, `line-clamp-*`, or `break-words`
|
||||
- Flex children need `min-w-0` to allow text truncation
|
||||
- Handle empty states—don't render broken UI for empty strings/arrays
|
||||
- User-generated content: anticipate short, average, and very long inputs
|
||||
|
||||
### Images
|
||||
|
||||
- `<img>` needs explicit `width` and `height` (prevents CLS)
|
||||
- Below-fold images: `loading="lazy"`
|
||||
- Above-fold critical images: `priority` or `fetchpriority="high"`
|
||||
|
||||
### Performance
|
||||
|
||||
- Large lists (>50 items): virtualize (`virtua`, `content-visibility: auto`)
|
||||
- No layout reads in render (`getBoundingClientRect`, `offsetHeight`, `offsetWidth`, `scrollTop`)
|
||||
- Batch DOM reads/writes; avoid interleaving
|
||||
- Prefer uncontrolled inputs; controlled inputs must be cheap per keystroke
|
||||
- Add `<link rel="preconnect">` for CDN/asset domains
|
||||
- Critical fonts: `<link rel="preload" as="font">` with `font-display: swap`
|
||||
- Prefer `<video autoplay muted loop playsinline>` over animated GIF; provide a still alternative
|
||||
- Short non-essential loops: Safari H.264 MP4 `<picture>` source, `prefers-reduced-motion` media condition, and still fallback
|
||||
|
||||
### Navigation & State
|
||||
|
||||
- URL reflects state—filters, tabs, pagination, expanded panels in query params
|
||||
- Links use `<a>`/`<Link>` (Cmd/Ctrl+click, middle-click support)
|
||||
- Deep-link all stateful UI (if uses `useState`, consider URL sync via nuqs or similar)
|
||||
- Destructive actions need confirmation modal or undo window—never immediate
|
||||
|
||||
### Touch & Interaction
|
||||
|
||||
- `touch-action: manipulation` (prevents double-tap zoom delay)
|
||||
- `-webkit-tap-highlight-color` set intentionally
|
||||
- `overscroll-behavior: contain` in modals/drawers/sheets
|
||||
- During drag: disable text selection, `inert` on dragged elements
|
||||
- Drag/swipe/pinch/path gestures need tap/click and keyboard alternatives unless essential
|
||||
- `autoFocus` sparingly—desktop only, single primary input; avoid on mobile
|
||||
|
||||
### Safe Areas & Layout
|
||||
|
||||
- Full-bleed layouts need `env(safe-area-inset-*)` for notches
|
||||
- Avoid unwanted scrollbars: `overflow-x-hidden` on containers, fix content overflow
|
||||
- Flex/grid over JS measurement for layout
|
||||
|
||||
### Dark Mode & Theming
|
||||
|
||||
- `color-scheme: dark` on `<html>` for dark themes (fixes scrollbar, inputs)
|
||||
- `<meta name="theme-color">` matches page background
|
||||
- Native `<select>`: explicit `background-color` and `color` (Windows dark mode)
|
||||
|
||||
### Locale & i18n
|
||||
|
||||
- Dates/times: use `Intl.DateTimeFormat` not hardcoded formats
|
||||
- Numbers/currency: use `Intl.NumberFormat` not hardcoded formats
|
||||
- Detect language via `Accept-Language` / `navigator.languages`, not IP
|
||||
- Brand names, code tokens, identifiers: wrap with `translate="no"` to prevent garbled auto-translation
|
||||
|
||||
### Hydration Safety
|
||||
|
||||
- Inputs with `value` need `onChange` (or use `defaultValue` for uncontrolled)
|
||||
- Date/time rendering: guard against hydration mismatch (server vs client)
|
||||
- `suppressHydrationWarning` only where truly needed
|
||||
|
||||
### Hover & Interactive States
|
||||
|
||||
- Buttons/links need `hover:` state (visual feedback)
|
||||
- Interactive states increase contrast: hover/active/focus more prominent than rest
|
||||
|
||||
### Content & Copy
|
||||
|
||||
- Active voice: "Install the CLI" not "The CLI will be installed"
|
||||
- Title Case for headings/buttons (Chicago style)
|
||||
- Numerals for counts: "8 deployments" not "eight"
|
||||
- Specific button labels: "Save API Key" not "Continue"
|
||||
- Error messages include fix/next step, not just problem
|
||||
- Second person; avoid first person
|
||||
- `&` over "and" where space-constrained
|
||||
|
||||
### Anti-patterns (flag these)
|
||||
|
||||
- `user-scalable=no` or `maximum-scale=1` disabling zoom
|
||||
- `onPaste` with `preventDefault`
|
||||
- `transition: all`
|
||||
- `outline-none` without focus-visible replacement
|
||||
- Inline `onClick` navigation without `<a>`
|
||||
- `<div>` or `<span>` with click handlers (should be `<button>`)
|
||||
- Images without dimensions
|
||||
- Large arrays `.map()` without virtualization
|
||||
- Form inputs without labels
|
||||
- Icon buttons without `aria-label`
|
||||
- Hardcoded date/number formats (use `Intl.*`)
|
||||
- `autoFocus` without clear justification
|
||||
- Animated GIF when compressed video is suitable
|
||||
- Gesture-only action without tap/click and keyboard alternative
|
||||
|
||||
## Output Format
|
||||
|
||||
Group by file. Use `file:line` format (VS Code clickable). Terse findings.
|
||||
|
||||
```text
|
||||
## src/Button.tsx
|
||||
|
||||
src/Button.tsx:42 - icon button missing aria-label
|
||||
src/Button.tsx:18 - input lacks label
|
||||
src/Button.tsx:55 - animation missing prefers-reduced-motion
|
||||
src/Button.tsx:67 - transition: all → list properties
|
||||
|
||||
## src/Modal.tsx
|
||||
|
||||
src/Modal.tsx:12 - missing overscroll-behavior: contain
|
||||
src/Modal.tsx:34 - "..." → "…"
|
||||
|
||||
## src/Card.tsx
|
||||
|
||||
✓ pass
|
||||
```
|
||||
|
||||
State issue + location. Skip explanation unless fix non-obvious. No preamble.
|
||||
|
|
@ -0,0 +1,190 @@
|
|||
---
|
||||
description: Review UI code for Vercel Web Interface Guidelines compliance
|
||||
argument-hint: <file-or-pattern>
|
||||
---
|
||||
|
||||
# Web Interface Guidelines
|
||||
|
||||
Review these files for compliance: $ARGUMENTS
|
||||
|
||||
Read files, check against rules below. Output concise but comprehensive—sacrifice grammar for brevity. High signal-to-noise.
|
||||
|
||||
## Rules
|
||||
|
||||
### Accessibility
|
||||
|
||||
- Icon-only buttons need `aria-label`
|
||||
- Form controls need `<label>` or `aria-label`
|
||||
- Interactive elements need keyboard handlers (`onKeyDown`/`onKeyUp`)
|
||||
- `<button>` for actions, `<a>`/`<Link>` for navigation (not `<div onClick>`)
|
||||
- Images need `alt` (or `alt=""` if decorative)
|
||||
- Decorative icons need `aria-hidden="true"`
|
||||
- Async updates (toasts, validation) need `aria-live="polite"`
|
||||
- Use semantic HTML (`<button>`, `<a>`, `<label>`, `<table>`) before ARIA
|
||||
- Headings hierarchical `<h1>`–`<h6>`; include skip link for main content
|
||||
- `scroll-margin-top` on heading anchors
|
||||
- Meaningful media needs captions, transcripts, or descriptions as applicable
|
||||
- Media controls need keyboard support; decorative media needs assistive-tech hiding
|
||||
|
||||
### Focus States
|
||||
|
||||
- Interactive elements need visible focus: `focus-visible:ring-*` or equivalent
|
||||
- Never `outline-none` / `outline: none` without focus replacement
|
||||
- Use `:focus-visible` over `:focus` (avoid focus ring on click)
|
||||
- Group focus with `:focus-within` for compound controls
|
||||
- Sticky headers/footers/overlays must not cover the focused element
|
||||
|
||||
### Forms
|
||||
|
||||
- Inputs need `autocomplete` and meaningful `name`
|
||||
- Use correct `type` (`email`, `tel`, `url`, `number`) and `inputmode`
|
||||
- Never block paste (`onPaste` + `preventDefault`)
|
||||
- Labels clickable (`htmlFor` or wrapping control)
|
||||
- Disable spellcheck on emails, codes, usernames (`spellCheck={false}`)
|
||||
- Checkboxes/radios: label + control share single hit target (no dead zones)
|
||||
- Submit button stays enabled until request starts; spinner during request
|
||||
- Errors inline next to fields; focus first error on submit
|
||||
- Placeholders end with `…` and show example pattern
|
||||
- `autocomplete="off"` on non-auth fields to avoid password manager triggers
|
||||
- Warn before navigation with unsaved changes (`beforeunload` or router guard)
|
||||
|
||||
### Animation
|
||||
|
||||
- Honor `prefers-reduced-motion` (provide reduced variant or disable)
|
||||
- Animate `transform`/`opacity` only (compositor-friendly)
|
||||
- Never `transition: all`—list properties explicitly
|
||||
- Set correct `transform-origin`
|
||||
- SVG: transforms on `<g>` wrapper with `transform-box: fill-box; transform-origin: center`
|
||||
- Animations interruptible—respond to user input mid-animation
|
||||
- Autoplay motion >5 seconds alongside other content needs pause, stop, or hide controls
|
||||
- Muted decorative loops must stop under `prefers-reduced-motion`
|
||||
|
||||
### Typography
|
||||
|
||||
- `…` not `...`
|
||||
- Curly quotes `“` `”` not straight `"`
|
||||
- Non-breaking spaces: `10 MB`, `⌘ K`, brand names
|
||||
- Loading states end with `…`: `"Loading…"`, `"Saving…"`
|
||||
- `font-variant-numeric: tabular-nums` for number columns/comparisons
|
||||
- Use `text-wrap: balance` or `text-pretty` on headings (prevents widows)
|
||||
|
||||
### Content Handling
|
||||
|
||||
- Text containers handle long content: `truncate`, `line-clamp-*`, or `break-words`
|
||||
- Flex children need `min-w-0` to allow text truncation
|
||||
- Handle empty states—don't render broken UI for empty strings/arrays
|
||||
- User-generated content: anticipate short, average, and very long inputs
|
||||
|
||||
### Images
|
||||
|
||||
- `<img>` needs explicit `width` and `height` (prevents CLS)
|
||||
- Below-fold images: `loading="lazy"`
|
||||
- Above-fold critical images: `priority` or `fetchpriority="high"`
|
||||
|
||||
### Performance
|
||||
|
||||
- Large lists (>50 items): virtualize (`virtua`, `content-visibility: auto`)
|
||||
- No layout reads in render (`getBoundingClientRect`, `offsetHeight`, `offsetWidth`, `scrollTop`)
|
||||
- Batch DOM reads/writes; avoid interleaving
|
||||
- Prefer uncontrolled inputs; controlled inputs must be cheap per keystroke
|
||||
- Add `<link rel="preconnect">` for CDN/asset domains
|
||||
- Critical fonts: `<link rel="preload" as="font">` with `font-display: swap`
|
||||
- Prefer `<video autoplay muted loop playsinline>` over animated GIF; provide a still alternative
|
||||
- Short non-essential loops: Safari H.264 MP4 `<picture>` source, `prefers-reduced-motion` media condition, and still fallback
|
||||
|
||||
### Navigation & State
|
||||
|
||||
- URL reflects state—filters, tabs, pagination, expanded panels in query params
|
||||
- Links use `<a>`/`<Link>` (Cmd/Ctrl+click, middle-click support)
|
||||
- Deep-link all stateful UI (if uses `useState`, consider URL sync via nuqs or similar)
|
||||
- Destructive actions need confirmation modal or undo window—never immediate
|
||||
|
||||
### Touch & Interaction
|
||||
|
||||
- `touch-action: manipulation` (prevents double-tap zoom delay)
|
||||
- `-webkit-tap-highlight-color` set intentionally
|
||||
- `overscroll-behavior: contain` in modals/drawers/sheets
|
||||
- During drag: disable text selection, `inert` on dragged elements
|
||||
- Drag/swipe/pinch/path gestures need tap/click and keyboard alternatives unless essential
|
||||
- `autoFocus` sparingly—desktop only, single primary input; avoid on mobile
|
||||
|
||||
### Safe Areas & Layout
|
||||
|
||||
- Full-bleed layouts need `env(safe-area-inset-*)` for notches
|
||||
- Avoid unwanted scrollbars: `overflow-x-hidden` on containers, fix content overflow
|
||||
- Flex/grid over JS measurement for layout
|
||||
|
||||
### Dark Mode & Theming
|
||||
|
||||
- `color-scheme: dark` on `<html>` for dark themes (fixes scrollbar, inputs)
|
||||
- `<meta name="theme-color">` matches page background
|
||||
- Native `<select>`: explicit `background-color` and `color` (Windows dark mode)
|
||||
|
||||
### Locale & i18n
|
||||
|
||||
- Dates/times: use `Intl.DateTimeFormat` not hardcoded formats
|
||||
- Numbers/currency: use `Intl.NumberFormat` not hardcoded formats
|
||||
- Detect language via `Accept-Language` / `navigator.languages`, not IP
|
||||
- Brand names, code tokens, identifiers: wrap with `translate="no"` to prevent garbled auto-translation
|
||||
|
||||
### Hydration Safety
|
||||
|
||||
- Inputs with `value` need `onChange` (or use `defaultValue` for uncontrolled)
|
||||
- Date/time rendering: guard against hydration mismatch (server vs client)
|
||||
- `suppressHydrationWarning` only where truly needed
|
||||
|
||||
### Hover & Interactive States
|
||||
|
||||
- Buttons/links need `hover:` state (visual feedback)
|
||||
- Interactive states increase contrast: hover/active/focus more prominent than rest
|
||||
|
||||
### Content & Copy
|
||||
|
||||
- Active voice: "Install the CLI" not "The CLI will be installed"
|
||||
- Title Case for headings/buttons (Chicago style)
|
||||
- Numerals for counts: "8 deployments" not "eight"
|
||||
- Specific button labels: "Save API Key" not "Continue"
|
||||
- Error messages include fix/next step, not just problem
|
||||
- Second person; avoid first person
|
||||
- `&` over "and" where space-constrained
|
||||
|
||||
### Anti-patterns (flag these)
|
||||
|
||||
- `user-scalable=no` or `maximum-scale=1` disabling zoom
|
||||
- `onPaste` with `preventDefault`
|
||||
- `transition: all`
|
||||
- `outline-none` without focus-visible replacement
|
||||
- Inline `onClick` navigation without `<a>`
|
||||
- `<div>` or `<span>` with click handlers (should be `<button>`)
|
||||
- Images without dimensions
|
||||
- Large arrays `.map()` without virtualization
|
||||
- Form inputs without labels
|
||||
- Icon buttons without `aria-label`
|
||||
- Hardcoded date/number formats (use `Intl.*`)
|
||||
- `autoFocus` without clear justification
|
||||
- Animated GIF when compressed video is suitable
|
||||
- Gesture-only action without tap/click and keyboard alternative
|
||||
|
||||
## Output Format
|
||||
|
||||
Group by file. Use `file:line` format (VS Code clickable). Terse findings.
|
||||
|
||||
```text
|
||||
## src/Button.tsx
|
||||
|
||||
src/Button.tsx:42 - icon button missing aria-label
|
||||
src/Button.tsx:18 - input lacks label
|
||||
src/Button.tsx:55 - animation missing prefers-reduced-motion
|
||||
src/Button.tsx:67 - transition: all → list properties
|
||||
|
||||
## src/Modal.tsx
|
||||
|
||||
src/Modal.tsx:12 - missing overscroll-behavior: contain
|
||||
src/Modal.tsx:34 - "..." → "…"
|
||||
|
||||
## src/Card.tsx
|
||||
|
||||
✓ pass
|
||||
```
|
||||
|
||||
State issue + location. Skip explanation unless fix non-obvious. No preamble.
|
||||
6242
docs/research/_raw/tdd-red/wcag22.raw.html
Normal file
6242
docs/research/_raw/tdd-red/wcag22.raw.html
Normal file
File diff suppressed because one or more lines are too long
Loading…
Add table
Add a link
Reference in a new issue