Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

YuE2-3B-OrbitQuant-W4A4 / src /ROUND6_EXPERIMENTS.md

WaveCut

Runtime update: released native 1.0.5, memory flags (VAE tile 256, lazy KV handoff), verified-digest cache, background VAE load, chunked INT8 head quantization, compile paths only with --warm-compile; measurements refreshed

cb66b66 verified about 22 hours ago

preview code

Raw

History Blame Contribute Delete

3.39 kB

Attention scheduling and semantic sampling

RTX PRO 4500 Blackwell; Torch2.10/CUDA12.8; OrbitQuant0.9.5/native1.0.2. The Russian default song, packed weights, 8 acoustic steps and FP16 VAE remain fixed. All timings below are experiments preceding runtime v4 integration; the model card reports the integrated release separately.

Experiment	Evidence	Decision
Tighter FlashAttention max_k buckets	At 2509 used tokens, one isolated call fell from17.591 to11.997us. Actual teacher-forced model decode was essentially unchanged. Whole song28.57s versus28.50s and about0.2GiB more memory.	Preserve audio/source; do not enable.
Restrict eager semantic sampling to32769 legal classes	Isolated0.4182 to0.4066ms; whole song28.41s, identical PCM.	Too small for a standalone speed claim.
Capture legal semantic sampling in a CUDA graph	Isolated0.1372ms. Two fresh-process requests preserved PCM; warm27.38s versus28.50s. Interleaved semantic phase15.817 to14.705s.	Integrate with request-owned buffers, RNG and cleanup.

The attention test covered both one request and two unequal CFG branches, used lengths1998/2509/4096/7000/10000, and the production16-query/8-KV-head128-wide layout. Outputs passed an FP32 sliced reference and NaN-poisoned future-cache checks. Different reduction scheduling still changed the autoregressive trajectory: full-song normalized log-band energy differed by1.89dB, JS divergence0.0434. This was not rejected for bitwise differences; it lacked a model-level speed/memory gain.

The sampler comparison ran a warmup followed by baseline/captured/captured/baseline. Mean full times28.316 versus27.050s also include VAE warmup variance; the semantic-only times isolate the approximately1.11s sampler gain. All five decoded waveforms were identical. Shared-process VRAM peaks are not compared with fresh-process peaks.

The sampler tests cover advanced RNG preservation through capture, independent interleaved request generators, legal vocabulary, repetition penalties/window expiry, minimum-length EOS, explicit close and invalid inputs. Runtime tests additionally cover CFG, greedy fallback, cancellation cleanup, repeatability and expert restoration.

Private experiment artifacts are preserved in WaveCut/YuE2-3B-OrbitQuant-lab-private at revisions94883acf4314f545aedf68c8f09a3087b9b2da3e (attention/cropping), d78f8b9acdb46cd61858d6b39864689afa31374b (captured sampler), and acfb177adfb11dceb89fb8cc6b0382554c1c4e19 (interleaved full songs). Public integrated measurements live in evaluation/runtime-v4/. Every experiment has a listening milestone; a human rating is not a blocking gate.

Next measurable hypotheses

Keep repetition history in a GPU ring, avoiding per-token host tensor construction and copies. Preserve duplicate frequencies and partially filled windows.
Capture ABC sampling with its ordinary-text vocabulary plus ABC_END. Its100-token history and larger vocabulary may yield a different tradeoff.
Integrate sampling after logits/CFG in the decode graph, removing the additional replay and logit copy. Keep request RNG and the host termination check intact.
Revisit UINT16 pair-table storage only with an interleaved native measurement; the earlier approximately0.7% apparent gain is insufficient for a library release.

These are hypotheses, not shipped features or performance claims.