Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

YuE2-3B · OrbitQuant W4A4

Music generation from style and lyrics, with packed 4-bit transformer weights and activations. Output: 48 kHz stereo. The default prompt is the Russian synth-pop song «До утра».

Model

392 transformer projections quantized, represented by 224 grouped packed modules. Embeddings, normalization, auxiliary projections and the audio VAE remain in source precision; the output heads can optionally run as per-row INT8 (fast/lowmem profiles).
28 AR/NAR transformer layers: hidden width 2048, MLP 6144, 16 query heads, 8 KV heads. The AR branch produces an ABC score and semantic tokens; the NAR flow transformer produces acoustic latents; the convolutional VAE decodes audio.
Checkpoint: 3.04 GB, down from 7.26 GB (58.2% smaller). The separate 0.53 GB VAE is unchanged. No fine-tuning or distillation.

Runtime v11

Runtime v11 keeps the W4A4 weights and kernels of v10 and removes the pipeline overheads that dominated wall time on consumer GPUs:

decode loop reads sampled tokens one step behind the GPU (no host stall per token; exact),
the ABC head evaluates only the ordinary-text classes the sampler can use (exact),
the transformer stays resident during VAE decode instead of a CPU round trip (exact),
the VAE decoder runs through torch.compile per uniform tile (24 ms per 512-frame tile on RTX 4090 vs 81 ms eager; numerics differ at FP16 rounding level),
fast: the NAR reuses the semantic KV cache instead of a second prefill, NAR pointwise chains are compiled, and both output heads run as per-row INT8 through a DP4A GEMV (937 GB/s on RTX 4090). INT8 heads change logits at the rounding level, so the sampled arrangement differs from exact at the same seed,
lowmem: fast plus INT8 token embeddings,
turbo: lowmem with the output heads quantized by the OrbitQuant W4A4 recipe instead of INT8 (semantic-head KL 0.016, top-1 0.79 versus 0.0015 / 0.914 for INT8); fastest per audio second and lowest memory, largest logit deviation.

Kernels ship for SM89 (RTX 40 / Ada, validated on RTX 4090) and SM120 (Blackwell consumer, built with CUDA 12.8 but not executed on Blackwell in this campaign). PyTorch 2.10.0 / CUDA 12.8, Python 3.12, Linux x86_64, OrbitQuant 0.9.7 / native 1.0.4.

Run

hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --v11 exact --output song      # same tokens as v10 fast+INT8 KV
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --v11 fast --output song-fast   # INT8 heads, KV handoff
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --v11 lowmem --output song-mem  # + INT8 embeddings
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --v11 turbo --output song-turbo # W4A4 heads

The first request in a process compiles the VAE decoder (about 25 s; cached by Inductor afterwards).

Measurements (RTX 4090, community cloud $0.34/h, same prompt and seed, warm second request in a fresh process)

Profile	Audio	Generation	audio s / wall s	NVML peak	torch peak
v10 published bundle (fast, RMS fusion, INT8 KV)	168.9 s	28.27 s	5.97	5.38 GiB	4.63 GiB
v11 `exact` (same sampled tokens as v10)	168.9 s	17.68 s	9.55	5.60 GiB	4.31 GiB
v11 `fast`	140.7 s	14.55 s	9.67	4.93 GiB	3.66 GiB
v11 `fast`, CFG 1.5	140.6 s	16.44 s	8.55	5.22 GiB	4.01 GiB
v11 `lowmem`	152.2 s	15.85 s	9.60	4.60 GiB	3.25 GiB
v11 `turbo`	178.2 s	17.29 s	10.31	4.49 GiB	3.20 GiB

Per-token decode on the same GPU (short context): semantic 2.47 → 2.11 ms (exact), ABC 2.78 → 2.30 ms (fast).

Quality gates (RTX 4090, evaluation/v11/): exact reproduces the v10 logits bit for bit over 128 forced positions and its waveform matches the v10 song at 0.001 dB mean band delta (JS 0.00000; only the FP16 VAE rounding differs). INT8 heads change the semantic-head distribution by a mean forward KL of 0.0015 (top-1 agreement 0.914) and the ABC head by 0.0002 (top-1 1.0); W4A4 heads (turbo) by 0.016 / 0.0009. The resulting fast/lowmem/turbo songs differ in arrangement from exact (1.8–1.9 dB mean band delta, JS 0.02–0.06 against the v10 song, the same class of deviation as the published INT8-KV profile). Compare those profiles by audio seconds per wall second, not by total time.

Audio and limitations

WAV demos: original BF16, W4A4 reference, fast INT8 KV. Spectral similarity is a diagnostic, not proof of perceptual equivalence. Russian was tested on this prompt; upstream advertises Chinese and English.

Code and kernels are under src/; metrics under evaluation/; prompts and audio under examples/. SHA256SUMS and runtime-lock.json pin the shipped artifacts; src/v11-build.json lists the per-architecture kernel binaries.

License: CC BY-NC 4.0 for model weights, inherited from M-A-P / YuE2. Code components retain their respective licenses. This release does not grant commercial rights.

Downloads last month: 998

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for WaveCut/YuE2-3B-OrbitQuant-W4A4

Base model

m-a-p/YuE2-3B

Finetuned

(1)

this model