VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications
Abstract
Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.
Community
We introduce VisualArtifactSpec 1.0 (VAS), a benchmark for evaluating agent-authored visual artifacts.
VAS includes 211 creation and modification tasks across 4 artifact families:
📊 Presentations: creating decks from scratch, working with templates, and editing existing slides.
🎨 Static graphics: posters, infographics, banners, and multi-image series.
🎬 Animations and videos: animated graphics, animated videos, and video editing.
🌐 Web interfaces: content browsing, data exploration, interactive tools, and workflows.
We evaluate two aspects separately:
① Task Fulfillment: Does the artifact satisfy explicit task requirements?
② Artifact Performance: How well do its content, organization, visuals, and delivery serve the intended context?
Across nine model–harness configurations, we find that meeting requirements does not necessarily make an artifact suitable for its intended context. For example, science cards may contain all the required information but differ substantially in visual hierarchy and readability on a phone.
We welcome feedback on the benchmark, evaluation protocol, and agent results!
Get this paper in your agent:
hf papers read 2611.21382 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper