Papers
arxiv:2611.21382

VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications

Published on Nov 16
· Submitted by
Shuyi Ouyang
on Nov 18
Authors:
,
,
,
,
,

Abstract

Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.

Community

Paper author Paper submitter

We introduce VisualArtifactSpec 1.0 (VAS), a benchmark for evaluating agent-authored visual artifacts.

VAS includes 211 creation and modification tasks across 4 artifact families:

📊 Presentations: creating decks from scratch, working with templates, and editing existing slides.
🎨 Static graphics: posters, infographics, banners, and multi-image series.
🎬 Animations and videos: animated graphics, animated videos, and video editing.
🌐 Web interfaces: content browsing, data exploration, interactive tools, and workflows.

We evaluate two aspects separately:
① Task Fulfillment: Does the artifact satisfy explicit task requirements?
② Artifact Performance: How well do its content, organization, visuals, and delivery serve the intended context?

Across nine model–harness configurations, we find that meeting requirements does not necessarily make an artifact suitable for its intended context. For example, science cards may contain all the required information but differ substantially in visual hierarchy and readability on a phone.

We welcome feedback on the benchmark, evaluation protocol, and agent results!

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.21382
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.21382 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.21382 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.