chore: publish rSkill OpenRAL/rskill-smolvla-robotwin v0.1.0

README.md

@@ -0,0 +1,123 @@
+---
+tags:
+  - OpenRAL
+  - rskill
+  - smolvla
+  - lerobot
+  - vla
+  - aloha_agilex
+  - robotwin
+  - bimanual
+  - manipulation
+license: apache-2.0
+language:
+  - en
+---
+
+# rskill-smolvla-robotwin
+
+> **OpenRAL rSkill** — SmolVLA (0.45 B) finetuned on the **RoboTwin 2.0** unified
+> dual-arm dataset (50 bimanual SAPIEN tasks, aloha-agilex embodiment), packaged for
+> use with the [OpenRAL](https://github.com/OpenRAL/openral) robot agent framework.
+
+This package wraps
+[`lerobot/smolvla_robotwin`](https://huggingface.co/lerobot/smolvla_robotwin) with a
+`rskill.yaml` manifest that adds capability checking, license surfacing, latency
+budgets, and local registry integration. It does **not** copy model weights.
+
+## What this skill does
+
+A multi-task dual-arm policy for the RoboTwin 2.0 benchmark
+([Chen et al., arXiv 2506.18088](https://arxiv.org/abs/2506.18088)). Action chunks of
+length 50 across three RGB views (head + per-wrist) driving a 14-DoF dual-arm joint
+command on the AgileX "aloha-agilex" embodiment.
+
+| Field | Value |
+| --- | --- |
+| Actions | `generalist`, `pick`, `place`, `transfer` |
+| Objects | `block`, `pot`, `cup`, `hammer` |
+| Scenes  | `tabletop` |
+| Embodiment | `aloha_agilex` |
+| Action space | 14-D joint position |
+| Cameras | `camera1` (head), `camera2` (left wrist), `camera3` (right wrist), 256×256 |
+
+## How it works
+
+OpenRAL loads the upstream LeRobot SmolVLA policy from `hf://lerobot/smolvla_robotwin`
+and uses the in-tree `smolvla` adapter to run chunked inference. RoboTwin itself runs in a
+separate SAPIEN/CuRobo sidecar; the sidecar returns three RGB views plus the 14-D
+aloha-agilex joint state, and the adapter replays 50-action chunks as absolute 14-D joint
+position commands.
+
+## Sensors / observation contract
+
+| Direction | Key | Shape | Notes |
+| --- | --- | --- | --- |
+| in | `observation.images.camera1` | `(3, 256, 256)` RGB | Head / overhead view, re-keyed from RoboTwin `head_camera`. |
+| in | `observation.images.camera2` | `(3, 256, 256)` RGB | Left wrist view, re-keyed from RoboTwin `left_camera`. |
+| in | `observation.images.camera3` | `(3, 256, 256)` RGB | Right wrist view, re-keyed from RoboTwin `right_camera`. |
+| in | `observation.state` | `(14,) float32` | aloha-agilex dual-arm joint state. |
+| out | action chunk | `(50, 14) float32` | Absolute dual-arm joint position commands. |
+
+## Manifest summary
+
+| Field | Value |
+| --- | --- |
+| `name` | `OpenRAL/rskill-smolvla-robotwin` |
+| `version` | `0.1.0` |
+| `license` | `apache-2.0` |
+| `role` | `s1` |
+| `model_family` | `smolvla` |
+| `embodiment_tags` | `aloha_agilex` |
+| `runtime` / `quantization.dtype` | `pytorch` / `bf16` |
+| `weights_uri` | `hf://lerobot/smolvla_robotwin` |
+| `state_contract.dim` / `action_contract.dim` | `14` / `14` |
+| `chunk_size` / `n_action_steps` | `50` / `50` |
+| `latency_budget.per_chunk_ms` | `250.0` |
+| `evaluated_tasks` | `robotwin` |
+
+## How to run it
+
+RoboTwin runs on **SAPIEN** out-of-process via a Python 3.10 sidecar
+([ADR-0061](../../docs/adr/0061-robotwin-dual-arm-benchmark-backend.md)) — its stack is
+incompatible with the openral 3.12 venv. Provision the sidecar venv, then:
+
+```bash
+# openral-side wire (pyzmq + msgpack)
+just sync --all-packages --group robotwin --inexact
+
+# single task
+openral benchmark scene \
+  --config scenes/benchmark/robotwin_lift_pot.yaml \
+  --rskill rskills/smolvla-robotwin
+
+# the 5-task suite
+openral benchmark run --suite robotwin --vla smolvla:rskills/smolvla-robotwin
+```
+
+See ADR-0061 for the SAPIEN+RoboTwin sidecar provisioning recipe
+(`OPENRAL_ROBOTWIN_AUTO_PROVISION=1` or the manual conda recipe).
+
+## Provenance
+
+- **Weights:** [`lerobot/smolvla_robotwin`](https://huggingface.co/lerobot/smolvla_robotwin)
+  (Apache-2.0), base [`lerobot/smolvla_base`](https://huggingface.co/lerobot/smolvla_base).
+- **Dataset:** [`lerobot/robotwin_unified`](https://huggingface.co/datasets/lerobot/robotwin_unified)
+  (Apache-2.0; `pepijn223/robotwin_unified_v3` renamed).
+- **Eval protocol:** RoboTwin official — 100 episodes/task, sim built-in success,
+  `episode_length=300`. No locally-reproduced official numbers shipped yet (`eval/` is empty).
+  The current website artifact is a 150-step GPU `openral benchmark scene` smoke clip
+  (`robotwin_smolvla-robotwin_fail.mp4`, `success=False`) for visual validation only;
+  populate `eval/` with `openral benchmark run --suite robotwin` on the eval host.
+
+> **STATE NOTE:** the live RoboTwin sidecar returns a 14-D aloha-agilex state, and the
+> official `policy_preprocessor.json` normalization stats expect `observation.state`
+> shape `(14,)`; `rskill.yaml` pins `state_contract.dim: 14` accordingly
+> (ADR-0061 §Live verification).
+
+## License
+
+This rSkill wrapper, the upstream `lerobot/smolvla_robotwin` checkpoint, and the
+`lerobot/robotwin_unified` dataset are Apache-2.0. The package does not copy weights into
+this repository; runtime loading still emits OpenRAL's unverified-provenance warning until
+the planned signing control exists.

SKILL.md

@@ -0,0 +1,74 @@
+---
+name: smolvla-robotwin
+description: >-
+  S1 Vision-Language-Action policy. Capabilities: generalist, pick, place, transfer on block, pot, cup, hammer. SmolVLA (0.45 B, lerobot/smolvla_base) finetuned on the RoboTwin 2.0 unified dataset (50 dual-arm tasks, aloha-agilex embodiment, SAPIEN). Multi-task: action chunks of length 50 across three RGB views (head + per-wrist) driving a 14-DoF dual-arm joint command. Runs on the RoboTwin scene backend through the out-of-process py3.10 SAPIEN sidecar (ADR-0061). Discovery view of an OpenRAL rSkill — NOT directly runnable by an agent harness; it runs via rSkill.from_pretrained + the robot HAL.
+metadata:
+  openral_rskill: true            # generated discovery view of an rSkill
+  schema_version: 0.1
+  rskill_id: OpenRAL/rskill-smolvla-robotwin
+  manifest: ./rskill.yaml
+  role: s1
+  kind: vla
+  model_family: smolvla
+  embodiment_tags: [aloha_agilex]
+  actions: [generalist, pick, place, transfer]
+  objects: [block, pot, cup, hammer]
+  scenes: [tabletop]
+  sensors_required: ['rgb:observation.images.camera1', 'rgb:observation.images.camera2', 'rgb:observation.images.camera3']
+  state_dim: 6
+  action_dim: 14
+  runtime: pytorch
+  quantization: bf16/pytorch
+  chunk_size: 50
+  n_action_steps: 50
+  latency_budget: {per_chunk_ms: 250.0}
+  license_code: Apache-2.0
+  license_weights: apache-2.0
+  weights_uri: hf://lerobot/smolvla_robotwin
+  source_repo: hf://lerobot/smolvla_robotwin
+  paper_url: https://arxiv.org/abs/2506.18088
+---
+
+# smolvla-robotwin — rSkill discovery view
+
+> **Generated view, not a hand-written skill.** This `SKILL.md` is a discovery-only
+> mirror of [`rskill.yaml`](./rskill.yaml), produced by `tools/generate_rskill_skillmd.py`.
+> It lets tools that read the standard agent-skill format find and reason about this
+> OpenRAL rSkill. The `rskill.yaml` manifest is the single source of truth
+> (CLAUDE.md §1.3). Do not edit by hand — edit the manifest and regenerate.
+
+## What it is
+
+An OpenRAL **Vision-Language-Action policy** (`role: s1`, `kind: vla`). SmolVLA (0.45 B, lerobot/smolvla_base) finetuned on the RoboTwin 2.0 unified dataset (50 dual-arm tasks, aloha-agilex embodiment, SAPIEN). Multi-task: action chunks of length 50 across three RGB views (head + per-wrist) driving a 14-DoF dual-arm joint command. Runs on the RoboTwin scene backend through the out-of-process py3.10 SAPIEN sidecar (ADR-0061).
+
+## Capabilities
+
+- **Verbs:** generalist · pick · place · transfer
+- **Objects:** block · pot · cup · hammer
+- **Scenes:** tabletop
+- **Embodiments:** aloha_agilex
+
+## Why this is discovery-only
+
+An agent skill is natural-language instructions loaded into an LLM's context. An rSkill
+is an executable artifact: it carries a typed capability/embodiment contract, model weights,
+a runtime, and a license/provenance gate — none of which fit in freeform markdown. So an
+agent can use this view to *select* the right skill, but cannot *execute* it by loading
+this file. Execution always goes through the OpenRAL loader and the robot HAL.
+
+## License
+
+- **Code:** Apache-2.0.
+- **Weights:** `apache-2.0` — permissive / commercial-use OK
+
+## How to actually run it (not via an agent harness)
+
+```python
+from openral_rskill import rSkill
+
+skill = rSkill.from_pretrained("OpenRAL/rskill-smolvla-robotwin")
+# the loader validates embodiment / sensors / runtime / quantization against the target
+# RobotDescription and enforces the weight-license gate before any weights load.
+```
+
+See [`rskill.yaml`](./rskill.yaml) for the authoritative, validated manifest.

rskill.yaml

@@ -0,0 +1,146 @@
+# rSkill manifest — OpenRAL packaging format V1 (CLAUDE.md §6.4)
+# Wraps: lerobot/smolvla_robotwin            (Apache-2.0)
+# Base:  lerobot/smolvla_base                (arxiv:2506.01844)
+# Dataset: lerobot/robotwin_unified (== pepijn223/robotwin_unified_v3, renamed)
+#
+# Official LeRobot SmolVLA finetune on the RoboTwin 2.0 unified dataset
+# (Chen et al., arXiv 2506.18088). Multi-task across the 50 dual-arm
+# tasks on the aloha-agilex embodiment. Routed via the in-tree `smolvla`
+# adapter (model_family: smolvla).
+#
+# Wired from the checkpoint's config.json (the inference authority):
+#   type: smolvla
+#   input_features:
+#     observation.state              shape (6,)         — see STATE NOTE below
+#     observation.images.camera1     shape (3,256,256)  — head / overhead
+#     observation.images.camera2     shape (3,256,256)  — left wrist
+#     observation.images.camera3     shape (3,256,256)  — right wrist
+#   output_features:
+#     action                          shape (14,)        — 14-DoF dual-arm joints
+#   chunk_size: 50  n_action_steps: 50  flow-matching steps: 10
+#   normalization: STATE/ACTION MEAN_STD, VISUAL IDENTITY
+#
+# Camera-name mapping
+# -------------------
+# RoboTwin's native LeRobot env cameras are head_camera / left_camera /
+# right_camera; the RoboTwin sidecar (tools/robotwin_sidecar.py) re-keys
+# them to camera1 / camera2 / camera3 (head→left→right) so they match
+# this checkpoint's input_features and the robots/aloha_agilex sensors —
+# no per-skill alias remap needed.
+#
+# STATE NOTE: the RoboTwin sidecar reset returns a 14-D aloha-agilex state
+# vector, and the official policy_preprocessor.json normalization stats also
+# expect observation.state shape (14,). `state_contract.dim` therefore follows
+# the live env + processor contract.
+
+# ── Identity ───────────────────────────────────────────────────────────────
+schema_version: "0.1"
+name: "OpenRAL/rskill-smolvla-robotwin"
+version: "0.1.0"
+license: "apache-2.0"
+role: "s1"
+kind: "vla"  # learnable Vision-Language-Action policy
+
+# ── Policy identity ────────────────────────────────────────────────────────
+model_family: "smolvla"
+
+# ── Compatibility contract ─────────────────────────────────────────────────
+# RoboTwin 2.0 default embodiment: AgileX dual-arm (robots/aloha_agilex).
+embodiment_tags:
+  - "aloha_agilex"
+
+# Three RGB streams matching the checkpoint's input_features. The
+# aloha_agilex RobotDescription declares matching camera1 / camera2 /
+# camera3 sensors so the compatibility check resolves cleanly.
+sensors_required:
+  - modality: "rgb"
+    vla_feature_key: "observation.images.camera1"
+    min_width: 256
+    min_height: 256
+  - modality: "rgb"
+    vla_feature_key: "observation.images.camera2"
+    min_width: 256
+    min_height: 256
+  - modality: "rgb"
+    vla_feature_key: "observation.images.camera3"
+    min_width: 256
+    min_height: 256
+
+# Output side (ADR-0013). 14-DoF dual-arm joint command. For aloha_agilex
+# the loader auto-fills n_dof (14) + vla_action_key from
+# robots/aloha_agilex/robot.yaml.
+actuators_required:
+  - kind: "joint_position"
+    control_mode_semantics:
+      mode: "absolute"
+
+# ── Runtime / weights ──────────────────────────────────────────────────────
+runtime: "pytorch"
+quantization:
+  dtype: "bf16"
+  backend: "pytorch"
+weights_uri: "hf://lerobot/smolvla_robotwin@967623a0f38c7e1236c66b3893c830398d793ff7"
+
+# ── Preprocessing (all knobs needed to interpret IO) ───────────────────────
+# The official checkpoint ships the lerobot PolicyProcessorPipeline JSONs
+# (policy_preprocessor.json + policy_postprocessor.json) plus their
+# `*_normalizer_processor.safetensors` siblings, so we point at them directly —
+# `materialize_processor_dir` auto-pulls the sibling safetensors. No
+# dataset-stats rebuild fallback needed. Trained on raw SAPIEN renders (no flip).
+processors:
+  preprocessor_uri: "hf://lerobot/smolvla_robotwin/policy_preprocessor.json"
+  postprocessor_uri: "hf://lerobot/smolvla_robotwin/policy_postprocessor.json"
+image_preprocessing:
+  flip_180: false
+state_contract:
+  dim: 14
+
+# ── Execution semantics ────────────────────────────────────────────────────
+# Matches the checkpoint's config.json (chunk_size=50, n_action_steps=50).
+chunk_size: 50
+n_action_steps: 50
+latency_budget:
+  per_chunk_ms: 250.0   # SmolVLA bf16 ≈ 80–150 ms/chunk + SAPIEN step headroom
+
+# ── Provenance ─────────────────────────────────────────────────────────────
+# No benchmarks shipped yet — see eval/.gitkeep. Populated by
+# `openral benchmark run --suite robotwin` once the SAPIEN sidecar venv is
+# provisioned (ADR-0061). RoboTwin leaderboard SmolVLA-class baselines for
+# reference (Pi0 46.4/16.3 Easy/Hard) are paper-cited, not reproduced here.
+paper_url: "https://arxiv.org/abs/2506.18088"
+dataset_uri: "hf://lerobot/robotwin_unified"
+source_repo: "hf://lerobot/smolvla_robotwin"
+
+description: >
+  SmolVLA (0.45 B, lerobot/smolvla_base) finetuned on the RoboTwin 2.0
+  unified dataset (50 dual-arm tasks, aloha-agilex embodiment, SAPIEN).
+  Multi-task: action chunks of length 50 across three RGB views
+  (head + per-wrist) driving a 14-DoF dual-arm joint command. Runs on the
+  RoboTwin scene backend through the out-of-process SAPIEN sidecar (ADR-0061).
+
+# ── Task-data gate (ADR-0060) ──────────────────────────────────────────────
+# Multi-task checkpoint trained on the full robotwin_unified set, so it is
+# valid on every RoboTwin scene. Declaring the scene id `robotwin` makes the
+# benchmark runner accept all `robotwin/<task>` scenes (family match) and
+# refuse non-RoboTwin scenes.
+evaluated_tasks: ["robotwin"]
+
+# ── Action vocabulary (ADR-0022) ───────────────────────────────────────────
+# GENERALIST marks the multi-task foundation checkpoint; the specific verbs
+# cover the bimanual-coordination / pick-place / stacking task families.
+actions:
+  - "generalist"
+  - "pick"
+  - "place"
+  - "transfer"
+objects:
+  - "block"
+  - "pot"
+  - "cup"
+  - "hammer"
+scenes:
+  - "tabletop"
+
+# ── Per-checkpoint action contract (ADR-0019) ──────────────────────────────
+action_contract:
+  dim: 14