Instructions to use zai-org/GLM-5.3-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-5.3-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zai-org/GLM-5.3-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zai-org/GLM-5.3-Flash")
model = AutoModelForMultimodalLM.from_pretrained("zai-org/GLM-5.3-Flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/GLM-5.3-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-5.3-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-5.3-Flash

SGLang

How to use zai-org/GLM-5.3-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-5.3-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-5.3-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-5.3-Flash with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-5.3-Flash
```

GLM-5.3-Flash runs great on a single, regular DGX Spark (vLLM, 200K context, HumanEval 97%)

#50

by wiklif - opened 1 day ago

Discussion

wiklif

1 day ago

Hi Z.ai team and everyone,

First, thank you for GLM-5.3-Flash. I wanted to share that it runs well on one ordinary DGX Spark-class box. Mine is an ASUS Ascent GX10 with the NVIDIA GB10 chip and 128 GB of unified memory. There is no cluster, and there is no second Spark. It is served by vLLM, and I use it every day for agentic coding.

Numbers (release v0.3.0)

Measured on the reference box: ASUS Ascent GX10, GB10, 121.6 GiB usable, aarch64.


decode, code	~29 tok/s (29.4; 26-29 across boots). DFlash2 k=7, acceptance length 5.5, temperature 0, `reasoning_effort` low
decode, prose	12.4-13.5 tok/s
prefill	~640 tok/s (643 on a 70K prompt, 632 on a 198K prompt)
context	200K (204,800 tokens). A 198K-token prompt: 313 s to the first token, zero preemptions
HumanEval pass@1	97.0% (159/164) at `reasoning_effort` low
HumanEval+ (EvalPlus)	93.9%
loop probe, long code	0 of 16 samples degenerate
memory	model 99.6 GiB, KV cache 6.2-7.5 GiB

Compared with the original, on your API

I ran the exact same HumanEval harness against glm-5.3-flash on the z.ai API. Everything was the same: prompts, code extraction, reasoning_effort low, temperature 0 with do_sample: false. Both sets of answers were scored with the same sandboxed tests.

	HumanEval	HumanEval+
original, z.ai API	95.1% (156/164)	89.6% (147/164)
2-bit experts, one Spark	97.0% (159/164)	93.9% (154/164)

I'm not claiming the local build is better. What it does show is no measurable loss on coding tasks at 2 bits per expert weight.

(A small wish: if the API exposed logprobs, the community could publish a proper KLD against the original instead of task results only.)

How it fits

The FP8 checkpoint is 306 GiB. The trick is that ~97% of the parameters are routed experts, and each token uses only 8 of 288 per layer.

Routed experts at 2 bits. These are tensor-symmetric codes {-4, -1, 1, 4} with one scale per 32 weights, computed by hand-written SASS kernels. The engine is vLLM-Moet.
Everything else at the checkpoint's native precision. That covers MLA attention, the 34 KDA layers, the shared expert, the three dense layers, embeddings and the router. The BF16 part is packed losslessly to 12 bits.
A 2 GiB FP4 pool keeps the most frequently routed experts at 4 bits, re-chosen continuously from the live routing. Without it, HumanEval drops from 95.7% to 92.1%.
GPTQ-calibrated 2-bit codes. One Hessian per layer, taken from the model's real inputs on source code. This took HumanEval from 95.7% to 97.0% and lowered perplexity on code, English and Polish.
DFlash2 (incoai/GLM-5.3-Flash-DFlash2) instead of the MTP head. The MTP head is a full MoE layer, so drafting with it was ~46% of every decode step. DFlash2 proposes a block of 8 in one pass, which gave +49% on code.
NoPE sparse MLA on sm_121 via chriswritescode-dev/glm-5.3-flash-sm120 (from vllm#53963). It zero-pads the 512-wide latent to the 576-wide GLM_NSA geometry, which is exact.

Things worth knowing

Pass reasoning_effort explicitly. The default is max, which is far too slow for local agent work. With low, I also recommend capping thinking with thinking_token_budget (1536 works well for coding in OpenCode). Without the cap, the model sometimes ignores low and thinks for a long time.
Long agent sessions and vLLM's prefix cache. The KDA state is checkpointed every 7168 tokens. If a small-block drafter lowers cache_config.block_size, prefill chunks stop aligning to those checkpoints. An occasional cached state then belongs to the wrong position, and the turn that restores it produces gibberish from its first token. Replaying a real session reproduced it deterministically. The fix is a small change in Scheduler._mamba_block_aligned_split: it uses the Mamba group's block size instead (patch 12 in the repo). If you run vLLM + DFlash2 with a small drafter block and see rare gibberish turns, this may be why.
Limits. DFlash2 k=7 leaves room for one long stream at a time. An agent turn pays a roughly constant ~16 s of prefill, because the KDA state forces a 7168-token cache block. The drafter is CC-BY-NC-ND-4.0.

Reproduce it

The patches, build scripts, a boot script, the measurement tools and a step-by-step guide are here:

https://github.com/lrozewicz/vLLM-Moet-GB10

Guide: docs/models/glm-5.3-flash.md

Happy to answer questions, or to run a specific benchmark if you'd like to see one.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment