Qwen3.8-27B-Uncensored-INT8-QEngine

Community INT8 conversion of Qwen3.8-27B-Uncensored-BF16 for the QEngine inference runtime. This is a third-party conversion, not an official Alibaba Cloud/Qwen release.

Quantization and files

• Main weights: symmetric W8A16, group size 128; generated with tools/quant_int8.py (search=0).
• MTP head: original BF16 weights.
• KV scales: sourced from a local Qwen3.8-27B-INT4 checkpoint.
• Includes model configuration, tokenizer, chat template, and bundled Apache-2.0 license.
• model.safetensors: 30,766,875,432 bytes; model_mtp.safetensors: 849,400,424 bytes.

QENGINE_INT8.md records quantization provenance. The exact upstream Hugging Face repository ID and converter revision are absent from supplied metadata; no upstream source repository is asserted here.

Inference

Requires a QEngine build supporting this Qwen3.8 INT8 format and MTP head. Tested with ghcr.io/bpx03/qengine:1.3.5.2-vision, vision enabled, and the OpenAI-compatible /v1/chat/completions API. These weights are QEngine-specific; Transformers/vLLM compatibility has not been tested and must not be assumed.

Server settings: QE_MAX_SEQS=4; QE_COMM=auto selects copy-engine prefill transfers; QE_DECODE_COMM=auto calibrates decode transport (this rig selected pinned host memory over P2P). Defaults when requests omit sampling fields: temperature 0.60, top-p 0.95, top-k 20, min-p 0.00.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b-qengine","messages":[{"role":"user","content":"Explain quantization in one paragraph."}],"max_tokens":256}'

Benchmark

Measured 2026-11-07 with scripts/bench.sh from club-3090 commit 0a955527. RUNS=3, WARMUPS=1; sampler: temperature 0.6, top-p 0.95, top-k 20, min-p 0.0. Three RTX 3090 GPUs; QEngine-reported context 262,144 tokens, four served slots, FP8 E4M3 KV cache.

Prompt shape	Wall throughput (mean)	Decode/prefill throughput (mean)	TTFT	Samples
Narrative, 1,000-token cap	134.00 tok/s	135.28 decode tok/s	70–71 ms	3/3 usable; decode CV 1.2%
Code, 800-token cap	171.74 tok/s	175.15 decode tok/s	70–71 ms	3/3 usable; decode CV 3.5%
Prefill, ~10K input tokens	—	3,574.80 prompt tok/s	2,758–2,980 ms	3; CV 2.3%
Prefill, ~90K input tokens	—	3,199.55 prompt tok/s	28,128 ms	1; directional only

Power caveat: the full rig report measured all three GPUs capped at 250 W (GPU 0: 71% of 350 W default; GPU 1: 59% of 420 W; GPU 2: 71% of 350 W). Treat throughput above as power-limited, not stock-power results. The 10K prefill CV is 2.3%, slightly above club-3090's <2% prefill norm.

Separate N=4 concurrency probe

Five rounds; four simultaneous streams; 16K input and 256 generated tokens per stream; greedy sampling (temperature 0, top-p 1, top-k 0, min-p 0). All rounds completed 4/4 with zero errors or silent-empty turns. Probe sampler differs from the canonical benchmark and results are not directly comparable.

Round	Reported aggregate tok/s	Per-stream tok/s	Median TTFT	TTFT p95	VRAM MiB
1	42.0	66.9	13,773 ms	22,727 ms	69,940
2	43.1	68.9	13,336 ms	22,132 ms	69,940
3	42.9	67.9	13,373 ms	22,329 ms	69,940
4	42.6	75.6	13,608 ms	22,433 ms	69,940
5	42.3	74.3	13,738 ms	22,659 ms	69,940

Probe summary: steady per-stream decode 74.31 tok/s; script-reported aggregate 42.31 tok/s; retention 109.6% (minimum 98%); p50/p95 TTFT 13,738/22,659 ms; post-warm VRAM growth 0 MiB. max_num_seqs=4 validated. Full report and run-level details: QEngine issue #20.

Limitations

• Results describe this QEngine build and three-GPU rig, not other hardware or runtimes.
• All GPUs were power-capped as noted above; do not quote these as stock-power throughput.
• The 90K prefill result has one measured sample. No BF16 quality comparison was performed.
• The upstream checkpoint repository and converter revision were not present in supplied metadata; verify source provenance and license before redistribution.
• Follow upstream acceptable-use and safety requirements. “Uncensored” is a model label, not a safety property.

License

The bundled license identifies Apache License 2.0 and copyright Alibaba Cloud. Preserve upstream notices when redistributing.