Qwen3.8-27B-Coder390-DFlash2-W4A16

The DFlash2 block drafter from nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2 (FP8/DFlash2-FP8) re-quantized to W4A16 compressed-tensors (pack-quantized, group 128, symmetric) — 1.28 GB, to sit next to the repo's 18.1 GB INT4/W4A16 target.

Why W4A16

The fp8 block-128 drafter does not load in the vLLM draft path on sm86 (bf16-vs-fp8 matmul dtype crash in the draft loop). This repo re-quantizes the fp8 drafter weights with a plain round-to-nearest symmetric group quant (no calibration set — the fp8 source is itself an already-low-precision artifact, and RTN from it matches the reference syvai/Qwen3.8-27B-DFlash2-W4A16 layout exactly: same 153-tensor set, same shapes, config.json byte-identical; only the quantized values differ, since the source drafter is the Coder390 one).

What is quantized

tensors	precision	note
layers.N.self_attn.{q,k,v,o}_proj, layers.N.mlp.{gate,up,down}_proj (5 layers)	int4 RTN, g128, symmetric	pack-quantized, weight_scale fp16, weight_shape int64
fc (5120 × 25600, projects the target's layer 5/19/33/47/61 hidden states)	int4 RTN, g128, symmetric
grouped-conv kernel_projections, candidate_selector, norms	bf16	passthrough; vLLM builds them unquantized

Use

vllm serve nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2 \
  --speculative-config '{"method":"dflash","model":"fedesito/Qwen3.8-27B-Coder390-DFlash2-W4A16","num_speculative_tokens":7}' \
  --attention-backend FLASH_ATTN --kv-cache-dtype bfloat16

The drafter shares the target's embeddings and lm_head (it ships neither). Requires a vLLM with DFlash2 support (0.27.1 + the drafter backport, or main with PR #52816).

Integrity: SHA256SUMS covers config.json and model.safetensors.

Apache-2.0, like the source drafter. Source: nerkyor; reference W4A16 layout: Inco, syvai.