SERVING.md: quote the measured KV capacity with its config instead of the auto-profile ceiling

800e44f verified 3 months ago

5 kB

Serving GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78

Reference deployment for 4x RTX PRO 6000 Blackwell (SM120a, 96 GB each, PCIe, no NVLink). This is the configuration the checkpoint was validated on; the KLD and throughput numbers in RESULTS.md come from it.

Docker image

The serving image is public on Docker Hub:

verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a

Digest-pinned (what docker-compose.yml uses by default):

verdictai/glm52-exl3-sparkinfer@sha256:2bb9e804a283d1da3b7e3425ff87375121285141d0d0a40d3dc09d41bf881a10

Pull it with:

docker pull verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a

An earlier tag, v21-mtp78tr3-gg6722c1d-si1a88b38-cu132-sm120a (sha256:9b1befc116b080f4d1328ede8ebe6f0896e10959e816c5db5da2b5445ae8423b), is also public and is the one referenced by the bundled server.sh.

The image carries vLLM with the b12x kernels and SparkInfer's EXL3 path built for sm_120a. It is architecture-specific: CUTE_DSL_ARCH=sm_120a and TORCH_CUDA_ARCH_LIST=12.0a are set inside the container, so it will not run on non-Blackwell parts without a rebuild.

Quick start

Both docker-compose.yml and server.sh ship inside the checkpoint. From the model directory:

./server.sh start      # docker compose up -d
./server.sh logs       # tail
./server.sh status
./server.sh stop

server.sh is a thin wrapper: it exports the environment below and calls docker compose. Every value is overridable from the shell.

docker compose

services:
  glm52:
    image: ${IMAGE:-verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a}
    container_name: glm52-exl3-sparkinfer
    restart: unless-stopped
    ports:
      - "${BIND_ADDRESS:-127.0.0.1}:${PORT:-9200}:8000"
    gpus: all
    shm_size: "32g"
    ipc: host
    ulimits:
      memlock: -1
      nofile: 1048576
    volumes:
      - ${MODEL_DIR:-.}:${MODEL_DIR:-.}:ro
      - ${CACHE_DIR:-~/.cache/glm52-exl3-sparkinfer}:/cache
    environment:
      CUDA_VISIBLE_DEVICES: "${CUDA_VISIBLE_DEVICES:-3,1,2,0}"
      CUDA_DEVICE_ORDER: PCI_BUS_ID
      CUTE_DSL_ARCH: sm_120a
      TORCH_CUDA_ARCH_LIST: 12.0a
      PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
      # EXL3 / b12x kernel selection
      VLLM_USE_B12X_MOE: "1"
      VLLM_USE_B12X_FP8_GEMM: "1"
      VLLM_USE_B12X_SPARSE_INDEXER: "1"
      B12X_W4A16_TC_DECODE: "1"
      B12X_MOE_FORCE_A16: "1"
      B12X_MLA_SM120_UNIFIED: "1"
      # PCIe all-reduce (no NVLink on this rig)
      VLLM_ENABLE_PCIE_ALLREDUCE: "1"
      VLLM_PCIE_ALLREDUCE_BACKEND: b12x
      VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE: "64KB"
      VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE: "84KB"
      # Decode context parallel
      VLLM_USE_B12X_DCP_A2A: "1"
      VLLM_DCP_A2A_MAX_TOKENS: "16"
      VLLM_DCP_GLOBAL_TOPK: "1"
      VLLM_DCP_SHARD_DRAFT: "1"
      # REQUIRED for the tr3 MTP layer-78 draft: keeps the small-m (m=1..N)
      # draft GEMMs inside the Trellis cudagraph decode window. Leaving the
      # default (4) raises "eager parity path entered during CUDA graph
      # capture (m=3)".
      VLLM_EXL3_TRELLIS_MIN_M: "1"
      VLLM_DISABLE_SHARED_EXPERTS_STREAM: "1"

The full file in the checkpoint carries additional tuning that is not required for correctness; the entries above are the ones that matter.

Serve parameters

Setting	Value	Note
tensor parallel	4
decode context parallel	4	`dcp_comm_backend=a2a`, KV interleave 64
MTP	3	layer 78 draft head, `draft_sample_method=greedy`
KV cache dtype	`nvfp4_ds_mla`	fp8 used for the KLD run, see RESULTS.md
max model len	524288	1M-native; 524288 is the validated setting
max num batched tokens	3072	the only config lever that moved throughput
gpu memory utilization	0.95	measured 457,728 KV tokens at 8K ctx / 2,048 batched, nvfp4_ds_mla
index topk pattern	`FFFSSS...`	see `GLM52_INDEX_TOPK_PATTERN` in `server.sh`

Notes and gotchas

VLLM_EXL3_TRELLIS_MIN_M=1 is mandatory with the MTP-78 head. This is the single most common cause of a failed boot on this checkpoint.
GPU order 3,1,2,0 is deliberate on the reference rig: GPU 1 drives the desktop and has roughly 800 MiB less usable memory, so it is placed off rank 0. Drop this if your GPUs are headless.
Mixed bit widths. This checkpoint stores routed experts at mixed 3/4-bit with a per-tensor bit map (tier_bitmap.json). A loader that assumes a single uniform bitrate will not load it. See R7_SERVING_SPEC.md.
Split layout. Non-expert tensors (attention, norms, router gates, shared experts, embed, head, MTP layer 78) live in model-layer-*.safetensors; routed experts live in r7-experts-layer-*.safetensors. Both sets are required and both are mapped by model.safetensors.index.json.