Qwengram-2B GGUF runtime validation
Canonical checkpoint: REAL-15M + linear750. The milestone comparison records the frozen 15M evaluation and the superseded conservative 10M selection.
| Precision | Stock NLL | Qwengram NLL | Reader gain [95% CI] | Gain retention [95% CI] | Perplexity reduction vs stock |
|---|---|---|---|---|---|
| BF16 | 2.538120 | 2.486529 | 0.051591 [0.037129, 0.066689] | 100% | 5.03% |
| Q8_0 | 2.539465 | 2.487978 | 0.051487 [0.037551, 0.066078] | 99.8% [98.3%, 101.8%] | 5.02% |
| Q6_K | 2.551537 | 2.497033 | 0.054504 [0.039340, 0.070659] | 105.6% [98.1%, 112.6%] | 5.30% |
| Q4_K_M | 2.577519 | 2.525079 | 0.052440 [0.037291, 0.067937] | 101.6% [95.1%, 108.2%] | 5.11% |
Matched test
The matched CPU test scores 8,128 tokens from the first 64 consecutive 256-token WikiText-2 raw test chunks, scoring the last 127 tokens per chunk. All runs use eight threads and context/batch/microbatch 256, with no warmup. Reader gain is NLL(stock) - NLL(Qwengram); retention divides each quantized gain by the BF16 gain. Paired 95% intervals use 10,000 resamples of 16 consecutive four-chunk blocks, seed 1234. The external PLE is Ivan Fioravanti's Q4_1 sidecar.
The four Qwengram runs were repeated for this 15M release. Stock controls reuse the existing verified runs: the backbone tensors, tokenizer, input text and CPU runtime binary hashes are unchanged. Per-chunk scores and model hashes are in results.json; exact commands and logs are in logs.
All 335 backbone tensors and nine tokenizer fields match exactly within each stock/Qwengram pair. All 11 reader/arbiter tensors remain bit-exact FP32 across BF16, Q8_0, Q6_K and Q4_K_M. Their values match the hash-verified 15M reader and matching linear750 arbiter. See packaging, verification and Q6 verification.
The Q4_1 PLE sidecar is host mapped and has SHA256 66db3ab390f4dd5063ecc89cc180f4713898577682347001bf64ab8e328527a1. The existing sidecar validation checks 4,096 addressed rows against independent reference addressing and dequantization, including prefill, decoding, EOS boundaries and reset. The reader checkpoint change does not change PLE addressing or dequantization.
This CPU test uses a quantized Q4_1 PLE and a WikiText-2 slice. The frozen Kaggle study uses the original FP8 PLE and different evaluation streams. These scores are separate benchmarks.
Generation checks
On the tested AMD BC-250, BF16, Q8_0, Q6_K, Q4_K_M with full Vulkan offload (-ngl 99) matched the corresponding CPU eight-token greedy continuation for The capital of France is. These short checks do not establish broad GPU parity.
The loader rejected missing and invalid PLE sidecars. See smoke.json and the separate Q6 output.