S2HAND

Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models

Singyu Muk (Independent Researcher) · Xuanren Song (Tsinghua University) · Kalin Stefanov (Japan Advanced Institute of Science and Technology)

S2HAND specializes the hand predictions of Sapiens2 while preserving its native 308-keypoint whole-body interface. S2HAND-MT routes 40 adapted non-wrist finger heatmaps through a copied pose decoder while preserving the shared Transformer, auxiliary task heads, and other 268 pose heatmaps. S2HAND-Pose additionally adapts three Fisher-selected Transformer blocks for stronger hand recovery. The optional TailGuard policy conditionally re-runs uncertain hands with a detected hand crop.

Source code and release documentation are maintained in the private ronggong000/S2HAND repository while the accompanying manuscript is under review.

Source code · H2SMV-FPV generator · Training protocol · Evaluation protocol

Model summary

Need	Model
Exact public routing for 268 non-finger heatmaps and unchanged auxiliary heads	S2HAND-MT-5B
Best released hand accuracy	S2HAND-Pose-5B
Smaller whole-body pose model	S2HAND-Pose-0.4B, 0.8B, or 1B

Native-pipeline AUC@0.20 (%; TailGuard off):

Model	NewDays TEST	VISOR TEST	H2O TEST	EPICK VAL	H2SMV-FPV TEST
Public Sapiens2-5B	61.48	56.12	69.02	55.16	47.93
S2HAND-MT-5B	62.57	59.42	70.79	58.54	49.01
S2HAND-Pose-5B	70.04	74.88	73.96	74.35	56.14

Each row uses its deployable detector/crop pipeline; missed hands count as failures. EPICK is a validation split and H2SMV-FPV is synthetic. See the source repository for strict metric definitions and cross-model comparisons.

Automatic base + delta loading

Five final model checkpoints and their loading script live in this repository. No GitHub checkout or local research environment is required. The loader downloads an exact official Sapiens2 pose base, verifies its SHA256, downloads the selected S2HAND delta, and applies replacement tensors with the final protected decoder route. Downloads use the Hugging Face cache; models sharing a 5B base reuse that same file. It does not write another full merged checkpoint.

The repository also contains training/S2HAND-Training-WarmStart-5B, a compact base delta used only to reproduce the common 5B training initialization and the diagonal-Fisher block probe. It is not a deployment model. The final S2HAND-MT-5B and S2HAND-Pose-5B checkpoints branch independently from this common warm-start; neither final checkpoint is initialized from the other.

Quick start

Python 3.12+ and PyTorch 2.7+:

pip install torch safetensors huggingface_hub
# S2HAND files are public; no login is required to download this repository.
hf download mage0001/S2HAND s2hand.py sapiens2_backbone.py --local-dir S2HAND
cd S2HAND
python s2hand.py --model S2HAND-MT-5B --device cuda

The last command downloads, checks and loads the model, then exits if no input is provided. To download weights without constructing the model, add --download-only. The official backbone source is bundled under the included Sapiens2 License.

Use in an existing pipeline

import torch
from s2hand import load_model

model = load_model("S2HAND-Pose-0.4B", device="cuda")
# Your existing person-crop pipeline supplies RGB tensors [B,3,1024,768].
# Normalize RGB values in [0,1] with ImageNet mean/std:
# mean = [0.485,0.456,0.406], std = [0.229,0.224,0.225].
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    heatmaps = model(normalized_rgb)  # [B,308,256,192]

For a command-line tensor input, save your normalized RGB batch as safetensors under key images, then run:

python s2hand.py --model S2HAND-Pose-0.4B --device cuda --bf16   --input-tensor crops.safetensors --output heatmaps.safetensors

This is the model loader and heatmap forward pass. Detection, person-crop affine transforms, UDP coordinate decoding, optional flip testing and TailGuard remain steps in the caller's pipeline. Matching paper scores requires the corresponding paper preprocessing and evaluation protocol; the loader does not substitute a new image-resize or detector protocol.

Models

Name	Learned Transformer blocks, zero-based	Delta size
S2HAND-MT-5B	None; complete copied pose decoder	3.31 MiB
S2HAND-Pose-5B	38,39,40 plus decoder	1019.04 MiB
S2HAND-Pose-1B	28,29,30 plus decoder	410.58 MiB
S2HAND-Pose-0.8B	17,18,19 plus decoder	284.78 MiB
S2HAND-Pose-0.4B	16,17,18 plus decoder	195.50 MiB

The smaller Pose models use their own same-size official bases. They are not compressed versions of the 5B checkpoint. The two 5B variants share one cached base, which alone is approximately 19.07 GiB; the delta size is not total model memory. Parameters remain FP32, matching the research runtime; --bf16 enables inference autocast. Meta initialization avoids allocating a second random model.

Exact routing and versions

The public decoder is copied before applying the delta. Only the 40 non-wrist finger channels use the adapted decoder. The other 268 channels, including wrist indices 41 and 62, use the frozen public decoder. Pose variants share their adapted backbone across both decoder routes; exact preservation of protected outputs is not guaranteed for those variants. MT does not update learned Transformer weights. The loader preserves the original BF16 RoPE runtime buffer semantics, including the explicit rebased 5B override. A generic base+delta load without this routing is insufficient to reproduce the final model interface.

Official base revisions and the S2HAND weight revision are pinned in the script. Per-model manifests record both SHA256 values and replacement tensor lists. Keep the downloaded base and delta in the cache to reuse them on subsequent runs. --cache-dir controls the cache location. --base-path, --delta-path, --manifest-path and --offline support existing local files without downloading another base. For completely offline use, keep sapiens2_backbone.py beside the script.

See VALIDATION.md for the checks performed before publishing this loader. The verified standalone loader is included here. Training, evaluation, TailGuard, and dataset tooling are maintained in the accompanying source repository. The private source release candidate v0.1.0-rc3 includes the complete clean HInt validation replay and end-to-end TailGuard command.

Intended use and limitations

• Intended for non-commercial research on whole-body and hand pose estimation, subject to the included Sapiens2 license.
• S2HAND remains a top-down estimator. Native-pipeline accuracy depends on the person/hand detector and crop protocol; a checkpoint result is not comparable across undeclared crop settings.
• S2HAND-MT gives an architectural exact-route guarantee for protected pose heatmaps and leaves the shared backbone/auxiliary heads unchanged. S2HAND-Pose updates shared Transformer blocks and therefore provides measured, not exact, non-hand preservation.
• TailGuard requires an external hand detector and adds conditional inference; it is not baked into these weights.
• The model can fail under severe blur, truncation, occlusion, side ambiguity, unusual optics, or detector failure. Do not use it for safety-critical or identity-sensitive decisions.

Citation

@misc{muk2027s2hand,
  title  = {S2HAND: Accuracy-Preservation Trade-Offs for Hand Specialization in Whole-Body Pose Models},
  author = {Muk, Singyu and Song, Xuanren and Stefanov, Kalin},
  year   = {2027}
}

Attribution

Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived model artifacts are distributed under the included Sapiens2 License.

Contact

Singyu Muk — ronggong000@gmail.com