Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

YuE2-3B · OrbitQuant W4A4

Music generation from style and lyrics (m-a-p/YuE2-3B) with packed 4-bit transformer weights and activations. Output: 48 kHz stereo.

Model

392 transformer projections in OrbitQuant W4A4 (224 packed modules). Embeddings, norms, auxiliary projections and the audio VAE stay in source precision; output heads are INT8 or W4A4 only in the profiles below.
Checkpoint 3.04 GB vs 7.26 GB BF16 (−58.2%). Separate 0.53 GB VAE unchanged. No fine-tuning or distillation.

Run

hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --mode fast --output out

One request per process is the default. A long-lived process serving many requests adds --warm-compile (compiled VAE decoder: 0.2 s instead of 0.7 s per request after a one-time 2 s cache load).

`--mode`	Changes	Logits
`off`	reference decode path	reference
`exact`	lookahead decode loop, ABC head cropped to the sampler's classes, model resident during VAE decode, compiled VAE decoder	bit-identical to `off`
`fast`	+ KV handoff to NAR, compiled NAR pointwise ops, INT8 output heads (DP4A GEMV)	KL 0.0015 semantic / 0.0002 ABC
`lowmem`	+ INT8 token embeddings	as `fast`
`turbo`	`lowmem` with W4A4 output heads	KL 0.016 / 0.0009

Requirements: Linux x86_64, Python 3.12, PyTorch 2.10.0 + CUDA 12.8; the OrbitQuant 0.9.8 wheel is bundled in src/wheels, native kernels 1.0.5 under src/runtime/kernels. Kernels: SM89 (validated on RTX 4090) and SM120 (built, not run). The first request compiles the VAE decoder (~25 s, cached afterwards).

Measurements

RTX 4090 (48-core host), identical input and seed. "Warm" is the second request of a long-lived process with --warm-compile; "process" is one src/run.py call from interpreter start to the written files (imports and model load included). Peaks are for the whole process.

Mode	Audio	Warm request	audio s / wall s	Whole process	NVML peak	torch peak
`off`	168.9 s	17.94 s	9.42	—	5.39 GiB	4.31 GiB
`exact`	168.9 s	15.85 s	10.66	21.6 s	5.33 GiB	4.05 GiB
`fast`	140.7 s	12.26 s	11.47	18.0 s	4.64 GiB	3.37 GiB
`lowmem`	152.2 s	13.49 s	11.28	19.3 s	4.47 GiB	3.37 GiB
`turbo`	178.2 s	15.57 s	11.44	21.7 s	4.71 GiB	3.56 GiB

exact keeps the off tokens and logits bit for bit; its waveform differs from off only at FP16 rounding in the VAE. INT8/W4A4 heads change sampling at the same seed, so compare those modes by audio seconds per wall second. Of a whole process, about 4 s are Python and library imports plus model load; the rest is generation. Wall time also depends on the host CPU: on a slow 4-core host the same exact request took 17.7 s and off 28.3 s. Raw data: evaluation/profiles/.

Files

src/: runtime, kernels and kernel sources; src/kernel-binaries.json lists the binaries per architecture, src/runtime-lock.json pins versions, SHA256SUMS covers every file. examples/: prompt file and WAV demos (BF16, W4A4, fast INT8 KV).

License: CC BY-NC 4.0 for the weights (inherited from M-A-P / YuE2); code components keep their own licenses. No commercial rights.

Downloads last month: 998

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for WaveCut/YuE2-3B-OrbitQuant-W4A4

Base model

m-a-p/YuE2-3B

Finetuned

(1)

this model