Copy rskill from AdrianLlopart/rskill-rtdetr-coco-r18

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.onnx.data filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,154 @@
+---
+tags:
+  - OpenRAL
+  - rskill
+  - detector
+  - object-detection
+  - rt-detr
+  - onnx
+  - coco
+license: apache-2.0
+language:
+  - en
+base_model:
+  - PekingU/rtdetr_r18vd_coco_o365
+---
+
+# rskill-rtdetr-coco-r18
+
+> **OpenRAL rSkill** — RT-DETR (Real-Time DEtection TRansformer) with a
+> ResNet-18vd backbone (r18vd), trained on COCO and exported to ONNX. Runs
+> as a perception producer on the camera tee and publishes `ObjectsMetadata`
+> to `/openral/perception/objects`. **No actuators.** This skill uses
+> `kind: detector` (ADR-0034); it emits no `Action` chunks and drives no
+> `ros2_control` joints.
+
+## What it does
+
+RT-DETR r18vd detects 80 COCO-category objects in each camera frame and
+publishes per-frame `ObjectsMetadata` events containing bounding boxes,
+class labels, and confidence scores. The runtime `ObjectsDetector`
+(in `openral_perception`) reads the `detector` manifest block at configure
+time to initialise the ONNX inference session and bind the class-id →
+label mapping.
+
+## Latency
+
+| Host           | Latency (ms) |
+|----------------|-------------|
+| NVIDIA GPU     | ~15–30      |
+| x86 CPU        | ~35–50      |
+
+Budget declared in manifest: `per_chunk_ms: 50.0`.
+
+## Weights
+
+The `model.onnx` file is not committed to the repository (binary artefact; see
+`.gitignore`). Reproduce it with an **ephemeral** overlay environment (does not
+mutate the project venv):
+
+```bash
+uv run --isolated --no-project \
+    --with "transformers>=4.45,<5" --with "torch>=2.2" --with torchvision \
+    --with onnx --with onnxscript \
+    python tools/export_rtdetr_onnx.py \
+    --out rskills/rtdetr-coco-r18/model.onnx \
+    --model-id PekingU/rtdetr_r18vd_coco_o365
+```
+
+> **Do NOT** run this via `uv sync --group onnx-export` — `uv sync` reconciles
+> the project venv to the synced group set and prunes `pydantic`/`structlog`
+> (and other deps) the source-on-PYTHONPATH dev/test setup relies on, breaking
+> the unit tests.
+>
+> `--isolated --no-project` is required: a plain `uv run --with` overlays on the
+> project venv, whose `torchvision` is built against a different `torch` than the
+> overlay's — importing `RTDetrForObjectDetection` then dies with
+> `operator torchvision::nms does not exist`. The isolated form builds a clean
+> ephemeral env (project venv untouched). `transformers<5` keeps the stable
+> RTDetr `forward` (logits + pred_boxes) signature; `onnx`+`onnxscript` are
+> required by the torch ≥2.7 ONNX exporter. GPU footprint: ~0.2 GB at 640² fp32 —
+> runs on an 8 GB card, no quantization needed.
+
+The torch 2.9 new exporter splits the model into two files that must be kept
+together in the same directory:
+
+| File               | Description                                  | sha256 (first 16 hex)      | Size  |
+|--------------------|----------------------------------------------|----------------------------|-------|
+| `model.onnx`       | ONNX graph (references `model.onnx.data`)    | `bda4dbeceff130ce...`       | 2.3 MB |
+| `model.onnx.data`  | External weight data (loaded by ORT)         | `8dff132e55df1bef...`        | 78 MB |
+
+Full sha256 values (reproduced with `transformers 4.x` + `torch 2.9` + `onnxscript`;
+the new torch exporter is not bit-reproducible across toolchain versions, so treat
+these as a same-host integrity check, not a cross-version guarantee):
+- `model.onnx`:      `bda4dbeceff130cec050e9757c9d95e217526a00730fb5f1558f960a6b316c63`
+- `model.onnx.data`: `8dff132e55df1befdf394a672a29906e38df7653be66705c47bd2a41634567b2`
+
+The published copies on the HF Hub repo are the canonical artefacts; the local
+export above must match them on the same toolchain.
+
+| Field        | Value                                                                |
+|--------------|----------------------------------------------------------------------|
+| `model_id`   | `PekingU/rtdetr_r18vd_coco_o365`                                     |
+| `opset`      | 18 (torch 2.9 new exporter; opset 17 target auto-bumped by exporter) |
+| `input`      | `pixel_values` — shape `(batch, 3, 640, 640)`, float32, range [0,1] |
+| `outputs`    | `logits (1, 300, 80)` pre-sigmoid; `pred_boxes (1, 300, 4)` cxcywh  |
+
+## Upstream model / training
+
+This rSkill packages an RT-DETR (Real-Time DEtection TRansformer) object
+detector with a **ResNet-18vd backbone (`r18vd`)**, exported to ONNX. It
+copies no PyTorch policy weights — the ONNX graph is produced from the
+upstream Transformers checkpoint by `tools/export_rtdetr_onnx.py` (see the
+**Weights** section above for the exact command and sha256 digests).
+
+| Field | Value |
+| --- | --- |
+| Architecture | RT-DETR, `r18vd` backbone |
+| Source repo | [`PekingU/rtdetr_r18vd_coco_o365`](https://huggingface.co/PekingU/rtdetr_r18vd_coco_o365) |
+| Training data | COCO (80 categories), pretrained on Objects365 (`o365`) |
+| Export tool | `tools/export_rtdetr_onnx.py` → `model.onnx` + `model.onnx.data` |
+| Paper | [arxiv:2304.08069](https://arxiv.org/abs/2304.08069) — *DETRs Beat YOLOs on Real-time Object Detection* |
+| License | apache-2.0 |
+
+## Supported robots / embodiments
+
+This detector is **embodiment-agnostic**: it consumes any RGB camera stream
+and emits `ObjectsMetadata`. All known embodiment tags are declared in the
+manifest; the `sensors_required` entry has no `vla_feature_key`, so the
+loader accepts any camera key — not just `camera1`.
+
+## Sensors / observation contract
+
+| Direction | Key | Modality | Shape / format | Notes |
+| --- | --- | --- | --- | --- |
+| in | any RGB camera | RGB `sensor_msgs/Image` | min 640 × 480 | `vla_feature_key` unset — any camera name accepted |
+| (preprocessing) | — | — | resized to 640 × 640, `/255` → float32 `[0,1]`, NCHW | `pixel_values` `(batch, 3, 640, 640)` |
+| out | COCO-80 detections | `ObjectsMetadata` | per object: `label`, `confidence`, `bbox` | published to `/openral/perception/objects` |
+
+The detector emits **no** `Action` chunks and has no proprioception
+(`observation.state`) contract.
+
+## Manifest summary
+
+| Field | Value |
+| --- | --- |
+| `name` | `AdrianLlopart/rskill-rtdetr-coco-r18` |
+| `version` | `0.1.0` |
+| `license` | `apache-2.0` |
+| `role` | `s1` |
+| `kind` | `detector` (ADR-0034 perception producer) |
+| `embodiment_tags` | all 17 canonical embodiment tags (any robot with RGB camera) |
+| `runtime` / `quantization.dtype` | `onnx` / `fp32` |
+| `weights_uri` | `rskill://rskills/rtdetr-coco-r18` |
+| `latency_budget.per_chunk_ms` | `50.0` |
+| `detector.labels` | 80 COCO categories |
+| `detector.input_size` | `[640, 640]` |
+| `detector.score_threshold` | `0.5` |
+
+Full schema: [`openral_core.schemas.RSkillManifest`](../../python/core/src/openral_core/schemas.py).
+
+## License
+
+Weights: Apache-2.0 (PaddlePaddle RT-DETR public release).
+See [arxiv:2304.08069](https://arxiv.org/abs/2304.08069) for the paper.

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bda4dbeceff130cec050e9757c9d95e217526a00730fb5f1558f960a6b316c63
+size 2313746

model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dff132e55df1befdf394a672a29906e38df7653be66705c47bd2a41634567b2
+size 81199104

rskill.yaml

@@ -0,0 +1,187 @@
+# rSkill manifest — RT-DETR COCO detector (kind: detector, ADR-0034)
+#
+# Packages a PaddlePaddle RT-DETR-L model exported to ONNX for use as
+# an OpenRAL perception producer. The skill runs on the camera tee and
+# publishes ObjectsMetadata to /openral/perception/objects.
+#
+# This is a pure perception producer: it emits no Action chunks, drives
+# no actuators, and has no proprioception contract. The runtime
+# ObjectsDetector (openral_perception) reads the detector block at
+# configure time to initialise the ONNX session and label map.
+#
+# LICENSE: weights are Apache-2.0 (PaddlePaddle RT-DETR public release).
+
+schema_version: "0.1"
+name: "AdrianLlopart/rskill-rtdetr-coco-r18"
+version: "0.1.0"
+license: "apache-2.0"
+role: "s1"
+kind: "detector"  # ADR-0034: perception producer; emits ObjectsMetadata, no Action
+
+# ── Compatibility contract ──────────────────────────────────────────────────
+# A detector is camera-based, not manipulator-specific. All embodiments are
+# declared because the only requirement is an RGB camera stream.
+embodiment_tags:
+  - "aloha"
+  - "franka_panda"
+  - "g1"
+  - "google_robot"
+  - "gr1"
+  - "h1"
+  - "mobile_base"
+  - "openarm"
+  - "panda_mobile"
+  - "pusht"
+  - "rizon4"
+  - "sawyer"
+  - "so100_follower"
+  - "so101_follower"
+  - "ur10e"
+  - "ur5e"
+  - "widowx"
+
+# The detector reads any RGB camera stream. vla_feature_key is omitted so
+# the loader accepts any camera key, not just "camera1".
+sensors_required:
+  - modality: "rgb"
+    min_width: 640
+    min_height: 480
+
+# Detectors command no actuators.
+actuators_required: []
+
+# ── Runtime / weights ───────────────────────────────────────────────────────
+runtime: "onnx"
+quantization:
+  dtype: "fp32"
+  backend: "onnx"
+
+# RT-DETR-L exported to ONNX. References the in-tree rSkill package.
+# The loader resolves rskill:// to the local rskills/<id>/ directory.
+weights_uri: "rskill://rskills/rtdetr-coco-r18"
+
+# ── Execution semantics ─────────────────────────────────────────────────────
+# chunk_size is 1: the detector runs per-frame. The safety supervisor never
+# sees detector output (no Action chunks), but chunk_size=1 keeps the
+# execution tick consistent with the camera frame rate.
+chunk_size: 1
+latency_budget:
+  # RT-DETR-L on an NVIDIA GPU is typically 15-30 ms; allow 50 ms budget
+  # to cover CPU fallback on smaller hosts.
+  per_chunk_ms: 50.0
+
+# ── Detector contract (ADR-0034) ────────────────────────────────────────────
+# 80 COCO labels (subset shown for readability — full list matches the
+# 0-indexed COCO category order used by the exported model).
+detector:
+  labels:
+    - "person"
+    - "bicycle"
+    - "car"
+    - "motorcycle"
+    - "airplane"
+    - "bus"
+    - "train"
+    - "truck"
+    - "boat"
+    - "traffic light"
+    - "fire hydrant"
+    - "stop sign"
+    - "parking meter"
+    - "bench"
+    - "bird"
+    - "cat"
+    - "dog"
+    - "horse"
+    - "sheep"
+    - "cow"
+    - "elephant"
+    - "bear"
+    - "zebra"
+    - "giraffe"
+    - "backpack"
+    - "umbrella"
+    - "handbag"
+    - "tie"
+    - "suitcase"
+    - "frisbee"
+    - "skis"
+    - "snowboard"
+    - "sports ball"
+    - "kite"
+    - "baseball bat"
+    - "baseball glove"
+    - "skateboard"
+    - "surfboard"
+    - "tennis racket"
+    - "bottle"
+    - "wine glass"
+    - "cup"
+    - "fork"
+    - "knife"
+    - "spoon"
+    - "bowl"
+    - "banana"
+    - "apple"
+    - "sandwich"
+    - "orange"
+    - "broccoli"
+    - "carrot"
+    - "hot dog"
+    - "pizza"
+    - "donut"
+    - "cake"
+    - "chair"
+    - "couch"
+    - "potted plant"
+    - "bed"
+    - "dining table"
+    - "toilet"
+    - "tv"
+    - "laptop"
+    - "mouse"
+    - "remote"
+    - "keyboard"
+    - "cell phone"
+    - "microwave"
+    - "oven"
+    - "toaster"
+    - "sink"
+    - "refrigerator"
+    - "book"
+    - "clock"
+    - "vase"
+    - "scissors"
+    - "teddy bear"
+    - "hair drier"
+    - "toothbrush"
+  input_size: [640, 640]
+  score_threshold: 0.5
+
+# ── Provenance ──────────────────────────────────────────────────────────────
+paper_url: "https://arxiv.org/abs/2304.08069"
+source_repo: "hf://PekingU/rtdetr_r18vd_coco_o365"
+
+description: >
+  RT-DETR-L (Real-Time DEtection TRansformer, large variant) trained on
+  COCO and exported to ONNX. Runs on the camera tee and publishes
+  ObjectsMetadata to /openral/perception/objects. 80 COCO categories.
+  Apache-2.0 weights. Reference latency ~20 ms on GPU, ~45 ms on CPU.
+  See ADR-0034 for the detector rSkill kind contract.
+
+# ADR-0022 — action vocabulary for the reasoner LLM tool palette.
+# DETECT signals that this skill is a perception producer.
+actions:
+  - "detect"
+
+objects:
+  - "person"
+  - "cup"
+  - "bottle"
+  - "bowl"
+  - "chair"
+  - "table"
+scenes:
+  - "tabletop"
+  - "kitchen"
+  - "indoor"