Papers
arxiv:2602.02235

Agent-Based Software Artifact Evaluation

Published on Jul 14
Authors:
,
,
,
,

Abstract

A structured scoring rubric and an AI agent system are developed to automate the evaluation of software engineering artifacts by providing executable verification criteria for LLM-based agents.

Artifact Evaluation (AE) has become a standard mechanism for linking software engineering papers to their supporting artifacts, but growing submission volume makes purely manual AE increasingly difficult to sustain. Although recent advances in LLM-based agents make AE automation increasingly plausible, current descriptive badge policies define badge semantics but provide no executable verification criteria, leaving both human reviewers and agents without a detailed basis for judgment. To address this gap, we construct ArtifactGuide, a structured scoring rubric grounded in ACM policy, expert-informed calibration, and artifact-based validation, and we design ArtifactCopilot, an agent collecting review evidence following a fixed sequence under ArtifactGuide and deriving the final badge decision from accumulated evidence. We evaluate our framework on 60 real artifacts from recent software engineering conferences using human-adjudicated badges as reference. The results show that ArtifactGuide improves the AE performance of coding agents over official ACM badge-policy prompts, increasing three-run mean exact badge agreement by 10.55 to 28.34 percentage points. Across all evaluated systems and prompting protocols, ArtifactCopilot achieves the highest badge-level agreement at 70.56% and is the only system that completed all repeated runs successfully while producing a review report in every run. A controlled user study with 8 experienced researchers suggests that ArtifactCopilot reports improve reviewer confidence, help reviewers locate evidence, and understand evaluation scope more clearly. Further analysis translates insights from automated AE into practical guidance for designing higher-quality artifacts with clearer review routes, more explicit claim-to-output links, and more concrete reuse paths.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2602.02235
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2602.02235 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2602.02235 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2602.02235 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.