Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NVFP4

NVFP4 quantized version of DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 (Cold Fusion — reduced-thinking Qwen3.8 fine-tune).

Original author: DavidAU. This is a quantization of their base model. All model quality/behavior credit goes to DavidAU.

Quantization details

• Tool: llm-compressor 0.13.0 (with compressed-tensors 0.18.0)
• Format: nvfp4-pack-quantized, W4A4, group size 16
• Input activations: dynamic-local, scale fp8_e4m3fn
• Calibration: 512 samples
• Size: 18GB (from 52GB BF16, ~2.9× compression)
• Architecture: text-only Qwen3_5ForCausalLM (Qwen3.5-GDN hybrid)

Serving with vLLM

IMPORTANT — FlashInfer GDN decode requires a small patch on vLLM ≤0.27.1. vLLM main is unservable for Qwen3.5-GDN (the 3.2 MB/token recurrent state gets paged as KV → 50 GB at 2K context). The working stack is vLLM 0.27.1 + the backported FlashInfer GDN decode kernel (vLLM PR #41966's _fi_decode → flashinfer.gdn_decode.gated_delta_rule_decode_pretranspose).

Reproducible patch + full setup flow:

• Patch script: scripts/backport-gdn-flashinfer-decode.sh
• Unified diffs: patches/vllm-0271-gdn-backport/
• Learnings: docs/vllm-gdn-flashinfer-pr-build-learnings.md (all in the gpu-cluster-infra repo)

# Requires: vLLM 0.27.1 + flashinfer nightly 0.6.18 (gdn_decode) + the backport patch
vllm serve <this-repo> --served-model-name cold-fusion-nvfp4 --trust-remote-code \
  --quantization compressed-tensors \
  --gdn-decode-backend flashinfer \
  --max-model-len 131072 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

Also note: remove processor_config.json from the local model dir (text-only Qwen3_5TextConfig — the multimodal processor registration mismatches), and pip uninstall tilelang + import tvm_ffi preload to avoid a tvm::ffi double-registration crash during CUDA graph capture.

Benchmark (measured on 1× RTX 5090, 32 GB, verified 2026-08-25)

Config	Decode single	Aggregate	Max ctx
128K / 8 seqs (recommended)	73.5 tok/s	405 tok/s @ 6 conc	131,072
256K / 2 seqs (max context)	24.7 tok/s	60 tok/s @ 2 conc	262,144
Stock vLLM 0.27.1 (no patch, Triton decode)	0.4 tok/s	—	65,536

The backported FlashInfer GDN decode fixes the stock Triton decode path (which was too slow to be usable), enabling practical serving on a single RTX 5090.

Tool calling

Works with --enable-auto-tool-choice --tool-call-parser qwen3_xml (verified).

Note

The checkpoint key layout was converted from model.language_model.* to flat model.* for vLLM's text-only Qwen3_5ForCausalLM class, and vLLM's registry needs the text-only architecture entry (PR #40471).