Hugging Face's logo

brandonmusic
/
GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78

Trellis
Safetensors
glm_moe_dsa
exl3
quantization
glm
Mixture of Experts
mixed-precision
modelopt
Model card Files
xet
 Community
4
GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78
File size: 5,003 Bytes
cc9c1b6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
800e44f
cc9c1b6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
# Serving GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78

Reference deployment for 4x RTX PRO 6000 Blackwell (SM120a, 96 GB each, PCIe,
no NVLink). This is the configuration the checkpoint was validated on; the KLD
and throughput numbers in [RESULTS.md](RESULTS.md) come from it.

## Docker image

The serving image is public on Docker Hub:

```
verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a
```

Digest-pinned (what `docker-compose.yml` uses by default):

```
verdictai/glm52-exl3-sparkinfer@sha256:2bb9e804a283d1da3b7e3425ff87375121285141d0d0a40d3dc09d41bf881a10
```

Pull it with:

```bash
docker pull verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a
```

An earlier tag, `v21-mtp78tr3-gg6722c1d-si1a88b38-cu132-sm120a`
(`sha256:9b1befc116b080f4d1328ede8ebe6f0896e10959e816c5db5da2b5445ae8423b`), is
also public and is the one referenced by the bundled `server.sh`.

The image carries vLLM with the b12x kernels and SparkInfer's EXL3 path built
for `sm_120a`. It is architecture-specific: `CUTE_DSL_ARCH=sm_120a` and
`TORCH_CUDA_ARCH_LIST=12.0a` are set inside the container, so it will not run
on non-Blackwell parts without a rebuild.

## Quick start

Both `docker-compose.yml` and `server.sh` ship inside the checkpoint. From the
model directory:

```bash
./server.sh start      # docker compose up -d
./server.sh logs       # tail
./server.sh status
./server.sh stop
```

`server.sh` is a thin wrapper: it exports the environment below and calls
`docker compose`. Every value is overridable from the shell.

## docker compose

```yaml
services:
  glm52:
    image: ${IMAGE:-verdictai/glm52-exl3-sparkinfer:v26-gg-v20final-scopefix-archkey-vllm5517197-sibe0edca-cu132-sm120a}
    container_name: glm52-exl3-sparkinfer
    restart: unless-stopped
    ports:
      - "${BIND_ADDRESS:-127.0.0.1}:${PORT:-9200}:8000"
    gpus: all
    shm_size: "32g"
    ipc: host
    ulimits:
      memlock: -1
      nofile: 1048576
    volumes:
      - ${MODEL_DIR:-.}:${MODEL_DIR:-.}:ro
      - ${CACHE_DIR:-~/.cache/glm52-exl3-sparkinfer}:/cache
    environment:
      CUDA_VISIBLE_DEVICES: "${CUDA_VISIBLE_DEVICES:-3,1,2,0}"
      CUDA_DEVICE_ORDER: PCI_BUS_ID
      CUTE_DSL_ARCH: sm_120a
      TORCH_CUDA_ARCH_LIST: 12.0a
      PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
      # EXL3 / b12x kernel selection
      VLLM_USE_B12X_MOE: "1"
      VLLM_USE_B12X_FP8_GEMM: "1"
      VLLM_USE_B12X_SPARSE_INDEXER: "1"
      B12X_W4A16_TC_DECODE: "1"
      B12X_MOE_FORCE_A16: "1"
      B12X_MLA_SM120_UNIFIED: "1"
      # PCIe all-reduce (no NVLink on this rig)
      VLLM_ENABLE_PCIE_ALLREDUCE: "1"
      VLLM_PCIE_ALLREDUCE_BACKEND: b12x
      VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE: "64KB"
      VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE: "84KB"
      # Decode context parallel
      VLLM_USE_B12X_DCP_A2A: "1"
      VLLM_DCP_A2A_MAX_TOKENS: "16"
      VLLM_DCP_GLOBAL_TOPK: "1"
      VLLM_DCP_SHARD_DRAFT: "1"
      # REQUIRED for the tr3 MTP layer-78 draft: keeps the small-m (m=1..N)
      # draft GEMMs inside the Trellis cudagraph decode window. Leaving the
      # default (4) raises "eager parity path entered during CUDA graph
      # capture (m=3)".
      VLLM_EXL3_TRELLIS_MIN_M: "1"
      VLLM_DISABLE_SHARED_EXPERTS_STREAM: "1"
```

The full file in the checkpoint carries additional tuning that is not required
for correctness; the entries above are the ones that matter.

## Serve parameters

| Setting | Value | Note |
|---|---|---|
| tensor parallel | 4 | |
| decode context parallel | 4 | `dcp_comm_backend=a2a`, KV interleave 64 |
| MTP | 3 | layer 78 draft head, `draft_sample_method=greedy` |
| KV cache dtype | `nvfp4_ds_mla` | fp8 used for the KLD run, see RESULTS.md |
| max model len | 524288 | 1M-native; 524288 is the validated setting |
| max num batched tokens | 3072 | the only config lever that moved throughput |
| gpu memory utilization | 0.95 | measured 457,728 KV tokens at 8K ctx / 2,048 batched, nvfp4_ds_mla |
| index topk pattern | `FFFSSS...` | see `GLM52_INDEX_TOPK_PATTERN` in `server.sh` |

## Notes and gotchas

- **`VLLM_EXL3_TRELLIS_MIN_M=1` is mandatory** with the MTP-78 head. This is the
  single most common cause of a failed boot on this checkpoint.
- **GPU order `3,1,2,0`** is deliberate on the reference rig: GPU 1 drives the
  desktop and has roughly 800 MiB less usable memory, so it is placed off rank 0.
  Drop this if your GPUs are headless.
- **Mixed bit widths.** This checkpoint stores routed experts at mixed 3/4-bit
  with a per-tensor bit map (`tier_bitmap.json`). A loader that assumes a single
  uniform bitrate will not load it. See `R7_SERVING_SPEC.md`.
- **Split layout.** Non-expert tensors (attention, norms, router gates, shared
  experts, embed, head, MTP layer 78) live in `model-layer-*.safetensors`;
  routed experts live in `r7-experts-layer-*.safetensors`. Both sets are
  required and both are mapped by `model.safetensors.index.json`.