Qwick-3.5-9B-FP8

日本語

Qwick-3.5-9B-FP8 is the FP8 E4M3 ModelOpt export of horiuchinobuyuki/Qwick-3.5-9B. Its weight files total 11.91 GB, 36.7% less than the BF16 checkpoint. It was tested with vLLM 0.23.0 and the Cutlass linear backend on an NVIDIA RTX PRO 6000 Blackwell GPU.

Developed by Nobuyuki Horiuchi.

Qwick-3.5-9B fine-tunes Qwen3.5-9B for shorter reasoning at comparable quality; this repository is its FP8 E4M3 export. See the base model card for that motivation and the full benchmark table.

Evaluation scope: quantization calibration used text only. This exported checkpoint was also measured on all 900 MMMU validation rows below. The measurement is report-only, not the 10,500-row test split, and broad multimodal serving remains outside release qualification; therefore the Hub pipeline stays text-generation.

Quantization checks

These runs measure the exported weights. Completion length includes thinking and the final answer.

Benchmark	Qwick BF16	Qwick-3.5-9B-FP8	Score change	Token change
MMLU-Pro screen (672)	80.060%	78.720%	-1.339 pp	+2.12%
GPQA-Diamond (198)	78.283%	76.263%	-2.020 pp	-1.07%
IFEval prompt strict (541), BF16 reference	89.649%	88.170%	-1.479 pp	+2.27%
MMMU validation official-compatible budget (900)	74.556%	72.778%	-1.778 pp	+4.66%

MMLU-Pro uses 48 questions from each of 14 subjects rather than the complete 12,032-question suite. GPQA and IFEval are complete. All three use thinking, temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 1.5, repetition penalty 1.0, and at most 32,768 generated tokens. IFEval uses dataset revision 966cd89545d6b6acfd7638bc708b98261ca58e84; final content is graded while the full trace is retained. This rerun replaces every earlier temperature-zero IFEval public measurement and corrects reporting only; it did not trigger model reselection. The source BF16 checkpoint's synthetic holdout used a separately frozen temperature-zero internal release gate, which passed; it is not a public performance benchmark and was not rerun for quantization.

The BF16 model's full benchmark table and training history are available on its model card. Detailed quantization results are in EVALUATION.md and quantization_results.json.

Complete vision validation

Model	Official-compatible budget	Strict budget	Mean full-trace tokens	Delta vs Qwick BF16
Qwen3.5-9B BF16	664/900 (73.778%)	662/900 (73.556%)	6476.48	-0.778 pp
Qwick BF16	671/900 (74.556%)	667/900 (74.111%)	4798.66	+0.000 pp
Qwick FP8	655/900 (72.778%)	655/900 (72.778%)	5022.40	-1.778 pp
Qwick NVFP4	635/900 (70.556%)	626/900 (69.556%)	5437.84	-4.000 pp

All four rows use the complete 900-row validation split, the same official-compatible final-content grader, thinking at temperature 1.0, a 32,768-token output cap, and 131,072-token server context. Qwick BF16, FP8, and NVFP4 use the same model-independent prompt-hash seed for every item. Qwick-3.5-9B-FP8 minus Qwick BF16 is -1.778 pp; its subject-stratified paired descriptive interval is [-4.222, +0.778] pp and exact McNemar is p=0.1982. Under that interval, the run does not resolve a vision difference. Official random fallbacks, strict scores, length stops, and full-trace integrity are retained in EVALUATION.md and quantization_results.json. This report-only measurement is not the 10,500-row test split.

Serving measurement

The following pair used one model per GPU, 256 measured requests after 16 warmups, fixed 512-token inputs and 512-token outputs, ignore_eos, and concurrency 128.

Metric	Qwick BF16	Qwick-3.5-9B-FP8	Change
Request throughput	5.618 req/s	7.921 req/s	+41.00%
Output throughput	2876.2 tok/s	4055.6 tok/s	+41.00%
Mean time to first token	4339.5 ms	3125.8 ms	-27.97%
Mean time per output token, excluding first	36.080 ms	25.490 ms	-29.35%

This is a hardware- and configuration-specific saturation measurement, not a single-request latency result.

Run with vLLM

python -m pip install -r requirements.txt
vllm serve horiuchinobuyuki/Qwick-3.5-9B-FP8 \
  --dtype auto \
  --linear-backend cutlass \
  --language-model-only \
  --max-model-len 65536

The tested server used eager execution and unquantized KV cache. The model occupied 10.37 GiB before KV-cache allocation. Context capacity depends on the available memory and serving settings.

Quantization

• Producer: NVIDIA ModelOpt 0.45.0
• Source: the sealed BF16 Qwick checkpoint
• Quantized: 200 text-backbone linear matrices, weights and input activations
• Kept in BF16: embeddings, LM head, visual tower, and the excluded linear-attention projections
• KV cache: unquantized
• Calibration: 512 × 512 tokens from WikiText-103-v1 train, revision b08601e04326c79dfdd32d625aee71d232d685c3

Only the pinned WikiText training split was supplied for calibration; benchmark and reserved-holdout records were not calibration inputs. See provenance.json for hashes and exact export settings.

Intended use and limitations

This release is intended for text inference and deployment experiments on NVIDIA GPUs supported by the serving stack. Validate it on the prompts and hardware used in production.

• The quantized evaluation is smaller than the BF16 release evaluation: 672 MMLU-Pro questions, 198 GPQA questions, and 541 IFEval prompts.
• HMMT, JMMLU, LiveCodeBench, and the reserved synthetic holdout were not rerun after quantization. MMMU validation was rerun on the exported checkpoint with one matched sampling seed per item.
• Accuracy and generation length can vary with the backend, kernel, sampling implementation, and decode settings.
• The reported serving benchmark uses fixed synthetic token lengths on one Blackwell system; it is not a general latency or energy claim.

License and citation

Qwick-3.5-9B-FP8 is released under Apache-2.0. It is derived from Qwick-3.5-9B and Qwen/Qwen3.5-9B@c202236235762e1c871ad0ccb60c8ee5ba337b9a. See NOTICE for attribution.

@misc{horiuchi2026qwick35fp8,
  author       = {Nobuyuki Horiuchi},
  title        = {Qwick-3.5-9B-FP8},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/horiuchinobuyuki/Qwick-3.5-9B-FP8}}
}