Normalize Motius Pipeline.from_pretrained metadata
Browse files- README.md +31 -23
- model_index.json +34 -13
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
library_name:
|
| 3 |
pipeline_tag: other
|
| 4 |
tags:
|
| 5 |
- motion-generation
|
|
@@ -14,18 +14,18 @@ license: other
|
|
| 14 |
# Go to Zero (MotionMillion)
|
| 15 |
|
| 16 |
Million-scale, 3B/7B-parameter autoregressive text-to-motion model ("Go to Zero",
|
| 17 |
-
ICCV 2025 **Highlight**) integrated into the
|
| 18 |
reproduction is **fully self-contained and independent of the original
|
| 19 |
repository** at runtime: the HumanVQVAE (FSQ tokenizer) and the LLaMA
|
| 20 |
autoregressive transformer live in
|
| 21 |
-
`
|
| 22 |
exercised.
|
| 23 |
|
| 24 |
| | |
|
| 25 |
|---|---|
|
| 26 |
| **Task** | Text-to-Motion (T2M), zero-shot |
|
| 27 |
| **Bundle / Pipeline** | `MotionMillionBundle` / `MotionMillionPipeline` |
|
| 28 |
-
| **Processed HF artifact** | [`7B-train`](https://huggingface.co/ZeyuLing/
|
| 29 |
| **Motion representation** | **humanml3d_272** (272-dim, 30 fps) — *identical layout to MotionStreamer-272* |
|
| 30 |
| **Tokenizer** | HumanVQVAE + **FSQ** (levels `[8,8,8,5,5,5]`, codebook 64000) |
|
| 31 |
| **AR model** | LLaMA-style 3B / 7B AR transformer (RoPE, length-causal text cross-attn) |
|
|
@@ -39,14 +39,14 @@ exercised.
|
|
| 39 |
|
| 40 |
## Weights
|
| 41 |
|
| 42 |
-
Current
|
| 43 |
|
| 44 |
| Artifact | Location | Contents | Status |
|
| 45 |
|---|---|---|---|
|
| 46 |
-
| Go-to-Zero 7B train-only HumanML3D-272 | [`ZeyuLing/
|
| 47 |
-
| Go-to-Zero 3B train-only HumanML3D-272 | [`ZeyuLing/
|
| 48 |
-
| local mirror, 7B | `checkpoints/gotozero/
|
| 49 |
-
| local mirror, 3B | `checkpoints/gotozero/
|
| 50 |
|
| 51 |
The public leaderboard uses the **train-only** checkpoints for both 7B and 3B.
|
| 52 |
The older `t2m_*_all.zip` checkpoints were trained with all HumanML3D splits and
|
|
@@ -55,13 +55,13 @@ should not be used for HumanML3D test-set leaderboard rows.
|
|
| 55 |
**Use directly from the Hub (recommended):**
|
| 56 |
|
| 57 |
```python
|
| 58 |
-
from
|
| 59 |
|
| 60 |
pipe = MotionMillionPipeline.from_pretrained(
|
| 61 |
-
"ZeyuLing/
|
| 62 |
device="cuda",
|
| 63 |
)
|
| 64 |
-
# Use "ZeyuLing/
|
| 65 |
# Cast the AR to bf16 for memory-efficient inference:
|
| 66 |
import torch
|
| 67 |
pipe.bundle.ar.to(dtype=torch.bfloat16)
|
|
@@ -79,20 +79,20 @@ bundle = MotionMillionBundle(
|
|
| 79 |
)
|
| 80 |
```
|
| 81 |
|
| 82 |
-
Package the
|
| 83 |
|
| 84 |
```bash
|
| 85 |
python3 scripts/eval/convert_motionmillion_checkpoint.py \
|
| 86 |
--ar checkpoints/motionmillion/pretrained_models/motionmillion_7B.pth \
|
| 87 |
-
--out_dir checkpoints/gotozero/
|
| 88 |
-
--repo_id ZeyuLing/
|
| 89 |
--text_model_source checkpoints/flan-t5-xl \
|
| 90 |
--verify
|
| 91 |
```
|
| 92 |
|
| 93 |
Use the same converter with the 3B train-only checkpoint and
|
| 94 |
-
`--out_dir checkpoints/gotozero/
|
| 95 |
-
ZeyuLing/
|
| 96 |
|
| 97 |
---
|
| 98 |
|
|
@@ -112,10 +112,10 @@ text -> Flan-T5-XL -> LLaMA-style AR (greedy, EOS-stopped, ≤150 tokens)
|
|
| 112 |
|
| 113 |
The upstream demo sampler caps generation at 50 motion tokens (about 100 frames
|
| 114 |
at 30 fps), which truncates many HumanML3D test clips. The model block size
|
| 115 |
-
supports the full benchmark range, so the
|
| 116 |
`max_sample_steps=150` and writes exact-GT-length MS272 predictions.
|
| 117 |
|
| 118 |
-
Convert to HumanML3D-263 with `
|
| 119 |
(`motion272_to_hml263`).
|
| 120 |
|
| 121 |
---
|
|
@@ -136,12 +136,12 @@ Reproduce generation with the packaged artifact:
|
|
| 136 |
```bash
|
| 137 |
NGPU=6 TOTAL_SHARDS=6 LIMIT=0 \
|
| 138 |
OUT=outputs/evaluation/t2m/humanml3d_official_test/ms272/gotozero_7b_train \
|
| 139 |
-
ARTIFACT=checkpoints/gotozero/
|
| 140 |
STEPS=150 DTYPE=bf16 \
|
| 141 |
bash scripts/eval/run_motionmillion_official272_exactlen_genonly.sh
|
| 142 |
```
|
| 143 |
|
| 144 |
-
Swap `OUT` and `ARTIFACT` to the `gotozero_3b_train` / `
|
| 145 |
paths for the 3B train-only row.
|
| 146 |
|
| 147 |
### MotionStreamer-272 evaluator (native space)
|
|
@@ -209,7 +209,7 @@ use it as a secondary diagnostic.
|
|
| 209 |
|
| 210 |
## Implementation notes
|
| 211 |
|
| 212 |
-
- **
|
| 213 |
`fsq.py` (Finite Scalar Quantization), `resnet.py` / `modules.py` (Haar
|
| 214 |
patch/unpatch) / `encdec.py` / `vqvae.py` (HumanVQVAE tokenizer) and
|
| 215 |
`llama.py` (LLaMA AR). The `args` namespace was refactored into explicit
|
|
@@ -220,8 +220,16 @@ use it as a secondary diagnostic.
|
|
| 220 |
state next to the `trans` weights; a tolerant unpickler stubs the unimportable
|
| 221 |
classes so only the tensor `state_dict` is read (repo-independent).
|
| 222 |
- **Complete text encoder artifact**: Flan-T5-XL is frozen and stored under
|
| 223 |
-
`text_encoder/` in the
|
| 224 |
resolves that artifact-local directory automatically.
|
| 225 |
- **Representation parity verified**: on smoke pairs the de-normalised prediction
|
| 226 |
matches GT per-block scale almost exactly (e.g. position-block std 0.514 vs
|
| 227 |
0.514; rot6d-block std 0.495 vs 0.489), confirming MotionMillion-272 ≡ MS-272.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: motius
|
| 3 |
pipeline_tag: other
|
| 4 |
tags:
|
| 5 |
- motion-generation
|
|
|
|
| 14 |
# Go to Zero (MotionMillion)
|
| 15 |
|
| 16 |
Million-scale, 3B/7B-parameter autoregressive text-to-motion model ("Go to Zero",
|
| 17 |
+
ICCV 2025 **Highlight**) integrated into the motius Model Zoo. Our
|
| 18 |
reproduction is **fully self-contained and independent of the original
|
| 19 |
repository** at runtime: the HumanVQVAE (FSQ tokenizer) and the LLaMA
|
| 20 |
autoregressive transformer live in
|
| 21 |
+
`motius.models.motion.motionmillion.network`. Only the T2M inference path is
|
| 22 |
exercised.
|
| 23 |
|
| 24 |
| | |
|
| 25 |
|---|---|
|
| 26 |
| **Task** | Text-to-Motion (T2M), zero-shot |
|
| 27 |
| **Bundle / Pipeline** | `MotionMillionBundle` / `MotionMillionPipeline` |
|
| 28 |
+
| **Processed HF artifact** | [`7B-train`](https://huggingface.co/ZeyuLing/Motius-MotionMillion-7B-HumanML272), [`3B-train`](https://huggingface.co/ZeyuLing/Motius-MotionMillion-3B-HumanML272) |
|
| 29 |
| **Motion representation** | **humanml3d_272** (272-dim, 30 fps) — *identical layout to MotionStreamer-272* |
|
| 30 |
| **Tokenizer** | HumanVQVAE + **FSQ** (levels `[8,8,8,5,5,5]`, codebook 64000) |
|
| 31 |
| **AR model** | LLaMA-style 3B / 7B AR transformer (RoPE, length-causal text cross-attn) |
|
|
|
|
| 39 |
|
| 40 |
## Weights
|
| 41 |
|
| 42 |
+
Current motius artifact (diffusers-style `from_pretrained`):
|
| 43 |
|
| 44 |
| Artifact | Location | Contents | Status |
|
| 45 |
|---|---|---|---|
|
| 46 |
+
| Go-to-Zero 7B train-only HumanML3D-272 | [`ZeyuLing/Motius-MotionMillion-7B-HumanML272`](https://huggingface.co/ZeyuLing/Motius-MotionMillion-7B-HumanML272) | `fsq.safetensors` + `ar.safetensors` + `mm_config.json` + `model_index.json` + `mean.npy` / `std.npy` + `text_encoder/` | public Hub artifact; Flan-T5-XL is packaged as safetensors |
|
| 47 |
+
| Go-to-Zero 3B train-only HumanML3D-272 | [`ZeyuLing/Motius-MotionMillion-3B-HumanML272`](https://huggingface.co/ZeyuLing/Motius-MotionMillion-3B-HumanML272) | same layout | public Hub artifact; Flan-T5-XL is packaged as safetensors |
|
| 48 |
+
| local mirror, 7B | `checkpoints/gotozero/motius_7b_train_humanml272` | same layout | optional local cache |
|
| 49 |
+
| local mirror, 3B | `checkpoints/gotozero/motius_3b_train_humanml272` | same layout | optional local cache |
|
| 50 |
|
| 51 |
The public leaderboard uses the **train-only** checkpoints for both 7B and 3B.
|
| 52 |
The older `t2m_*_all.zip` checkpoints were trained with all HumanML3D splits and
|
|
|
|
| 55 |
**Use directly from the Hub (recommended):**
|
| 56 |
|
| 57 |
```python
|
| 58 |
+
from motius.pipelines.motionmillion import MotionMillionPipeline
|
| 59 |
|
| 60 |
pipe = MotionMillionPipeline.from_pretrained(
|
| 61 |
+
"ZeyuLing/Motius-MotionMillion-7B-HumanML272",
|
| 62 |
device="cuda",
|
| 63 |
)
|
| 64 |
+
# Use "ZeyuLing/Motius-MotionMillion-3B-HumanML272" for the 3B train-only artifact.
|
| 65 |
# Cast the AR to bf16 for memory-efficient inference:
|
| 66 |
import torch
|
| 67 |
pipe.bundle.ar.to(dtype=torch.bfloat16)
|
|
|
|
| 79 |
)
|
| 80 |
```
|
| 81 |
|
| 82 |
+
Package the motius artifact from local upstream weights:
|
| 83 |
|
| 84 |
```bash
|
| 85 |
python3 scripts/eval/convert_motionmillion_checkpoint.py \
|
| 86 |
--ar checkpoints/motionmillion/pretrained_models/motionmillion_7B.pth \
|
| 87 |
+
--out_dir checkpoints/gotozero/motius_7b_train_humanml272 \
|
| 88 |
+
--repo_id ZeyuLing/Motius-MotionMillion-7B-HumanML272 \
|
| 89 |
--text_model_source checkpoints/flan-t5-xl \
|
| 90 |
--verify
|
| 91 |
```
|
| 92 |
|
| 93 |
Use the same converter with the 3B train-only checkpoint and
|
| 94 |
+
`--out_dir checkpoints/gotozero/motius_3b_train_humanml272 --repo_id
|
| 95 |
+
ZeyuLing/Motius-MotionMillion-3B-HumanML272` for the 3B artifact.
|
| 96 |
|
| 97 |
---
|
| 98 |
|
|
|
|
| 112 |
|
| 113 |
The upstream demo sampler caps generation at 50 motion tokens (about 100 frames
|
| 114 |
at 30 fps), which truncates many HumanML3D test clips. The model block size
|
| 115 |
+
supports the full benchmark range, so the motius reproduction uses
|
| 116 |
`max_sample_steps=150` and writes exact-GT-length MS272 predictions.
|
| 117 |
|
| 118 |
+
Convert to HumanML3D-263 with `motius.motion.representation.convert`
|
| 119 |
(`motion272_to_hml263`).
|
| 120 |
|
| 121 |
---
|
|
|
|
| 136 |
```bash
|
| 137 |
NGPU=6 TOTAL_SHARDS=6 LIMIT=0 \
|
| 138 |
OUT=outputs/evaluation/t2m/humanml3d_official_test/ms272/gotozero_7b_train \
|
| 139 |
+
ARTIFACT=checkpoints/gotozero/motius_7b_train_humanml272 \
|
| 140 |
STEPS=150 DTYPE=bf16 \
|
| 141 |
bash scripts/eval/run_motionmillion_official272_exactlen_genonly.sh
|
| 142 |
```
|
| 143 |
|
| 144 |
+
Swap `OUT` and `ARTIFACT` to the `gotozero_3b_train` / `motius_3b_train_humanml272`
|
| 145 |
paths for the 3B train-only row.
|
| 146 |
|
| 147 |
### MotionStreamer-272 evaluator (native space)
|
|
|
|
| 209 |
|
| 210 |
## Implementation notes
|
| 211 |
|
| 212 |
+
- **motius-native runtime**: `motius/models/motion/motionmillion/network/` holds
|
| 213 |
`fsq.py` (Finite Scalar Quantization), `resnet.py` / `modules.py` (Haar
|
| 214 |
patch/unpatch) / `encdec.py` / `vqvae.py` (HumanVQVAE tokenizer) and
|
| 215 |
`llama.py` (LLaMA AR). The `args` namespace was refactored into explicit
|
|
|
|
| 220 |
state next to the `trans` weights; a tolerant unpickler stubs the unimportable
|
| 221 |
classes so only the tensor `state_dict` is read (repo-independent).
|
| 222 |
- **Complete text encoder artifact**: Flan-T5-XL is frozen and stored under
|
| 223 |
+
`text_encoder/` in the motius artifact. `MotionMillionBundle.from_pretrained`
|
| 224 |
resolves that artifact-local directory automatically.
|
| 225 |
- **Representation parity verified**: on smoke pairs the de-normalised prediction
|
| 226 |
matches GT per-block scale almost exactly (e.g. position-block std 0.514 vs
|
| 227 |
0.514; rot6d-block std 0.495 vs 0.489), confirming MotionMillion-272 ≡ MS-272.
|
| 228 |
+
|
| 229 |
+
## Direct Loading
|
| 230 |
+
|
| 231 |
+
```python
|
| 232 |
+
from motius import Pipeline
|
| 233 |
+
|
| 234 |
+
pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-MotionMillion-3B-HumanML272")
|
| 235 |
+
```
|
model_index.json
CHANGED
|
@@ -1,19 +1,40 @@
|
|
| 1 |
{
|
| 2 |
"_class_name": "MotionMillionPipeline",
|
| 3 |
-
"_diffusers_version": "hftrainer",
|
| 4 |
-
"pipeline": {
|
| 5 |
-
"library": "hftrainer",
|
| 6 |
-
"class_name": "MotionMillionPipeline",
|
| 7 |
-
"module": "hftrainer.pipelines.motionmillion"
|
| 8 |
-
},
|
| 9 |
-
"bundle": {
|
| 10 |
-
"library": "hftrainer",
|
| 11 |
-
"class_name": "MotionMillionBundle",
|
| 12 |
-
"module": "hftrainer.models.motion.motionmillion"
|
| 13 |
-
},
|
| 14 |
"weights": {
|
| 15 |
"fsq": "fsq.safetensors",
|
| 16 |
"ar": "ar.safetensors",
|
| 17 |
"text_encoder": "text_encoder"
|
| 18 |
-
}
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
{
|
| 2 |
"_class_name": "MotionMillionPipeline",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
"weights": {
|
| 4 |
"fsq": "fsq.safetensors",
|
| 5 |
"ar": "ar.safetensors",
|
| 6 |
"text_encoder": "text_encoder"
|
| 7 |
+
},
|
| 8 |
+
"_library_name": "motius",
|
| 9 |
+
"format_version": 1,
|
| 10 |
+
"pipeline_class": "motius.pipelines.motionmillion.MotionMillionPipeline",
|
| 11 |
+
"bundle_class": "motius.models.motionmillion.MotionMillionBundle",
|
| 12 |
+
"tasks": [
|
| 13 |
+
"text_to_motion"
|
| 14 |
+
],
|
| 15 |
+
"required_files": [
|
| 16 |
+
"ar.safetensors",
|
| 17 |
+
"fsq.safetensors",
|
| 18 |
+
"mean.npy",
|
| 19 |
+
"mm_config.json",
|
| 20 |
+
"std.npy",
|
| 21 |
+
"text_encoder/.gitattributes",
|
| 22 |
+
"text_encoder/README.md",
|
| 23 |
+
"text_encoder/config.json",
|
| 24 |
+
"text_encoder/generation_config.json",
|
| 25 |
+
"text_encoder/model-00001-of-00002.safetensors",
|
| 26 |
+
"text_encoder/model-00002-of-00002.safetensors",
|
| 27 |
+
"text_encoder/model.safetensors.index.json",
|
| 28 |
+
"text_encoder/special_tokens_map.json",
|
| 29 |
+
"text_encoder/spiece.model",
|
| 30 |
+
"text_encoder/tokenizer.json",
|
| 31 |
+
"text_encoder/tokenizer_config.json"
|
| 32 |
+
],
|
| 33 |
+
"api": {
|
| 34 |
+
"loader": "motius.Pipeline.from_pretrained",
|
| 35 |
+
"task_methods": [
|
| 36 |
+
"infer_text_to_motion"
|
| 37 |
+
]
|
| 38 |
+
},
|
| 39 |
+
"artifact_format": "motius-motionmillion-v1"
|
| 40 |
+
}
|