lued/Qwen3.8-27B-TeichAI-Fable-INT8-W8A16-MTP

Qwen3.8-27B TeichAI Fable · INT8 W8A16 · BF16 MTP

A Fable-5 distilled, Ampere-optimized quantization for dual RTX 3090 inference.

Base model · Qwen3.8-27B · vLLM · llm-compressor

This is a numerical W8A16 quantization of TeichAI/Qwen3.8-27B-Fable-Distill, a Fable-5 light tune (distill) of Qwen/Qwen3.8-27B. All model credit belongs to TeichAI (the Fable-5 tune) and Qwen (the base model); refer to the upstream model card for architecture, capabilities, and usage guidance. This repository changes numerics only.

This build targets vLLM on Ampere-class GPUs where native FP8 tensor-core execution is unavailable. It retains the model's vision tower, thinking controls, native context, and the byte-preserved BF16 MTP head. The Fable-5 tune is a language-model weight change; the vision tower, recurrent GDN gates, lm_head, and MTP head fall outside the W8A16 target set and are carried through at source precision.

Relationship to the sibling quantization

This checkpoint is the Fable-5-distilled counterpart of lued/Qwen3.8-27B-INT8-W8A16-MTP. The repositories share:

• the identical W8A16 compressed-tensors recipe (400 packed GEMMs, same BF16 preservation set, group-128 symmetric data-free RTN);
• the identical native BF16 MTP head (15 tensors, byte-identical to Qwen's original MTP — same sha256 as the sibling release);
• the identical tokenizer, chat template, processor configs, and bounded-shard layout;
• the same total checkpoint footprint (29.44 GiB).

The only deviation is the base language-model weights: this checkpoint's LM weights are TeichAI's Fable-5 light-tune weights instead of stock Qwen3.8-27B.

Fable-5 distillation (inherited from the base)

Per the upstream card, the source is a light tune of Qwen/Qwen3.8-27B on the Fable-5 datasets (armand0e/claude-fable-5-claude-code, armand0e/Fable-5-Chat) plus a large corpus of private Fable 5 data, trained with Unsloth and Hugging Face TRL. Upstream-reported benchmark results versus the base: ARC Challenge 0.637 (0.591), ARC Challenge (Easy) 0.832 (0.782), BoolQ 0.911 (0.896). These are the upstream card's claims, not measurements made here. The W8A16 step touches only the 400 GEMMs; the recurrent GDN gates (96), vision tower (333 tensors), lm_head, and MTP head are preserved at source precision.

Quantization fidelity

Measured on this exact checkpoint against its own BF16 teacher (the TeichAI Fable-Distill source) on 2026-08-18 with tools/measure_kld.py (teacher-forced, full-vocabulary FP32 log-softmax, FP64 accumulation, nats/token; PyTorch 2.11.0+cu128, Transformers 5.10.1, compressed-tensors 0.17.1). This is the v0.2 revision: every preserved tensor is a bit-exact copy of the source (input_layernorm, post_attention_layernorm, q_norm/k_norm, vision tower, embeddings, lm_head, GDN gates); only the 400 projection GEMMs are quantized.

Candidate	Mean KLD vs BF16	Top-1 agreement	Weight files
This W8 / BF16 A16	0.000700	98.29%	29.44 GiB

The KLD result measures checkpoint-weight drift over 467 teacher-forced positions (fixed 8-prompt short suite). It is not a functional quality score; tool use, coding, multimodal quality, and long-context recall require separate behavioral evaluation. A greedy KV-cache smoke generation (finite logits, non-degenerate output) passes on every release.

Short-prompt suite

Prompt	Category	Tokens	Positions	Mean KLD	Top-1 agreement
factual	factual	13	12	0.000291	1.0000
code-fib	code	87	86	0.000512	0.9884
physics-uncertainty	physics	86	85	0.000668	0.9765
math-train	math	58	57	0.000920	0.9825
sql-top5	technical	60	59	0.000823	1.0000
narrative-clock	narrative	68	67	0.000857	0.9701
instruction-stack	instruction	43	42	0.000774	0.9762
history-industrial	history	60	59	0.000538	0.9831
Token-weighted mean			467	0.000700	98.29%

kld_result.json in this repository is the single source of truth for these numbers.

Checkpoint profile

Property	Value
Quantization	Data-free symmetric RTN W8A16, group size 128
Structural audit	STRUCTURAL AUDIT PASS — packed set, ignore list, index contract, value round-trip (every packed tensor dequantizes within the INT8 bound), and preserved bit-identity (all 784 non-quantized tensors byte-equal to source)
MTP audit	MTP AUDIT PASS (15 tensors dtype-, shape-, and bit-identical to the source)
Quantized GEMMs	400: 192 MLP, 64 full-attention, 144 dense GDN projections
Preserved precision	BF16 vision tower, lm_head, MTP (15 tensors, byte-identical to Qwen's original MTP), recurrent GDN gates (96)
Shard layout	7 main shards (5.34 GB max) + model_mtp.safetensors (849,400,392 bytes)
Runtime	vLLM; this repository is not a GGUF checkpoint

MTP preservation

The MTP head is the exact BF16 MTP from the original Qwen3.8-27B — never re-quantized:

• 15 tensors in model_mtp.safetensors (849,400,392 bytes).
• sha256 1d8268aa85ace093a561e3e7b63b9d390dac1cd55a90cd55b5ec509c3c9da9fe — identical to the MTP bytes in the Qwen3.8-27B source and every family release. Verify with sha256sum model_mtp.safetensors.
• All 15 tensors verified bfloat16, shape-equal, and bit-equal to the TeichAI Fable-Distill source (strict audit, not a count check); the built model_mtp.safetensors sha256 matches the family canonical hash.

Reproduction contract

The immutable source is TeichAI/Qwen3.8-27B-Fable-Distill commit ac2b133895580be05ee90a6a7cdcc23cdf998bd0. The exact selective W8A16 configuration is recorded in recipe.yaml and config.json. A conforming rebuild must take native MTP from that source, reproduce the source-derived packed projection set, and prove all 15 MTP tensors bit-identical before resharing for publication.

Status

v0.2 revision (2026-08-18). This checkpoint is built on the v0.2 pipeline: every preserved (non-quantized) tensor is a bit-exact copy of the TeichAI source — machine-audited (STRUCTURAL AUDIT PASS including the value round-trip and preserved bit-identity checks; MTP AUDIT PASS) — and KLD re-measured on the v0.2 bytes at 0.000700 nats/token, 98.29% top-1 (v0.1: 0.000660/98.50%; both within the family's measured range, and the difference is noise-scale — see the family CHANGELOG.md for the v0.1→v0.2 provenance story). Originally published 2026-08-18 at commit f41c1353d1cf238f631d42a1d0845d0471666f44 (v0.1); the v0.2 revision is a new commit on the same repo — prior commits remain fetchable by hash, so existing downloads keep working. Serving was validated on the v0.1 bytes: boots through llama-swap from the lued/* cache entry, loads the CompressedTensorsWNA16 → MarlinLinearKernel path, detects the native MTP draft model (method=mtp, 3 speculative tokens), and serves a smoke request with the qwen3 reasoning parser. The BF16 parent source remains cached; deletion is the pipeline's final step and has not been requested.