Measured results and historical reference
Columns Qwen base and Qwen community refer to our historical W4A16 quantizations, not BF16 originals. Qwen community uses JonathanColetti-Qwen3.8-27B-Uncensored upstream weights; it is not an abliteration developed by this publisher. VeriLoop base means the unmodified upstream model quantized by us. Ablated columns are evaluation records only.
| Task | Qwen base W4A16 | Qwen community W4A16 | VeriLoop base W4A16 | VeriLoop ablated W4A16 | VeriLoop base GGUF | VeriLoop ablated GGUF |
|---|---|---|---|---|---|---|
| mmlu | 130/150 (86.67%) | 126/150 (84.00%) | 126/150 (84.00%) | 114/150 (76.00%) | 102/150 (68.00%) | 102/150 (68.00%) |
| cmmlu | 135/150 (90.00%) | 134/150 (89.33%) | 127/150 (84.67%) | 113/150 (75.33%) | 92/150 (61.33%) | 97/150 (64.67%) |
| mmlu_pro | 132/150 (88.00%) | 125/150 (83.33%) | 118/150 (78.67%) | 106/150 (70.67%) | 96/150 (64.00%) | 103/150 (68.67%) |
| ceval | 127/150 (84.67%) | 127/150 (84.67%) | 126/150 (84.00%) | 114/150 (76.00%) | 90/150 (60.00%) | 105/150 (70.00%) |
| arc | 144/150 (96.00%) | 144/150 (96.00%) | 121/150 (80.67%) | 128/150 (85.33%) | 135/150 (90.00%) | 137/150 (91.33%) |
| truthfulqa | 132/150 (88.00%) | 120/150 (80.00%) | 88/150 (58.67%) | 107/150 (71.33%) | 91/150 (60.67%) | 91/150 (60.67%) |
| gsm8k | 99/100 (99.00%) | 99/100 (99.00%) | 95/100 (95.00%) | 88/100 (88.00%) | 37/100 (37.00%) | 47/100 (47.00%) |
| math500 | 75/100 (75.00%) | 77/100 (77.00%) | 75/100 (75.00%) | 73/100 (73.00%) | 68/100 (68.00%) | 67/100 (67.00%) |
| bbh | 140/150 (93.33%) | 138/150 (92.00%) | 113/150 (75.33%) | 98/150 (65.33%) | 72/150 (48.00%) | 82/150 (54.67%) |
| humaneval | 143/164 (87.20%) | 137/164 (83.54%) | 147/164 (89.63%) | 147/164 (89.63%) | 149/164 (90.85%) | 148/164 (90.24%) |
| ifeval | 89/100 (89.00%) | 93/100 (93.00%) | 76/100 (76.00%) | 69/100 (69.00%) | 63/100 (63.00%) | 60/100 (60.00%) |
| Total | 1346/1514 (88.90%) | 1320/1514 (87.19%) | 1212/1514 (80.05%) | 1157/1514 (76.42%) | 995/1514 (65.72%) | 1039/1514 (68.63%) |
Comparison boundary
Historical runs were on 2026-08-30; VeriLoop runs were on 2026-10-10. Both use seed 20260816, the same 11 categories and 1,514 samples, thinking enabled, greedy decoding and a 32,768-token output budget. evaluation-item-alignment.json records exact ordered item-ID comparisons against the current frozen set. Matching IDs do not establish identical dataset revisions or prompt bytes in the historical run.
Historical W4A16 used a 262,144-token context, a previous vLLM environment and the Qwen3 reasoning parser. Current W4A16 used context 65,536, vLLM 0.29.0, and raw-content extraction without that parser. Both used tensor parallelism 2, concurrency 16 and FP8 E5M2 KV cache. Current GGUF used llama.cpp, layer split 1:1, concurrency 8, context 40,960 per slot and Q8_0 KV cache. These are historical reference comparisons, not strictly identical-runtime controlled experiments. The GGUF/W4A16 difference also includes engine and KV-cache differences.
There is no full BF16 baseline for VeriLoop in this evaluation. These sampled/custom scores are not official leaderboard scores and do not reproduce the upstream private VeriLoop Harness or its agent/tool benchmarks. No conclusion about the validity of the authors' reported results follows from this table.
Data coverage
evaluation-items.json contains every saved per-item metric for all six runs (9,084 records), including IDs, correctness, token counts, errors, and available truncation/finish metadata. Saved answer tails and benchmark gold text are excluded. The original harness retained only answer tails, not complete generated responses; complete response replay is therefore unavailable. evaluation-question-indices.json supplies all current IDs and prompt hashes, without redistributing benchmark question text. Dataset loaders and scoring rules are supplied in evaluation-legacy-scorer.py (the frozen implementation, including historical comments), with evaluation-protocol.py and evaluation-adapter.py documenting current extraction and code execution. Dataset rights remain with their owners; model Apache-2.0 licensing does not relicense third-party datasets.
evaluation-refusal-items.json contains the saved metadata for 400 refusal probes, with no harmful prompts or generated answers. Marker counts are heuristic measurements, not semantic compliance judgments. No ablation implementation or weight-modification recipe is included in this release.
evaluation-marker-definitions.json contains the exact lexical lists and normalization rules. The scoring source files are archival copies: to import the adapter, restore the module filenames legacy_capability.py, protocol.py and capability.py. Rebuilding the frozen questions requires the original public datasets and matching the supplied item IDs and prompt hashes; those prerequisites and the original local code-execution image are not bundled. This is an audit package, not a claim of a self-contained one-command reproduction.
Standards and validity
Each category score is correct/n; total is sum(correct)/1514, not an unweighted average of category percentages. Main scores retain empty and truncated cases in the denominator. The historical summaries did not record the newer empty and finish-reason fields; absent fields must not be interpreted as zero. All six runs recorded zero request errors. Historical truncation totals were 28 and 42.
The rubric is the included legacy custom implementation: multiple choice extracts an answer letter; numerical tasks use normalization and numeric/string comparisons; BBH uses normalized exact matching; HumanEval executes extracted code against tests; IFEval implements only a subset of instruction rules. These are not the official complete evaluation implementations. Current HumanEval runs used isolated Docker execution, no network, read-only filesystem, non-root user, dropped capabilities, no host mounts and a 15-second timeout. The adapter retains the original local image name as provenance; that image is not distributed and cannot be assumed reproducible from its name alone.
Current decoding: temperature 0, top_p 1, max output 32768, thinking enabled with the source template. The paired test uses a fixed question snapshot SHA256 57ccbc22395b05ac6d13b408727fd821cec91ebb3ad84c01d276ed77b644d71a. Dataset revisions were not comprehensively pinned; IDs, prompt hashes and loader code are provided to expose that reproducibility limitation. See the runtime version/hash file for the exact recorded software metadata.
Current empty counts are W4A16 base 9, W4A16 ablated 8, GGUF base 4, GGUF ablated 7. The <=2% per-category empty-rate gate fails for original W4A16 HumanEval: 5/164 (3.05%). Its measured 147/164 remains reported with those failures included; not all validity gates passed. The other three runs passed that gate.
Protocol corrections preceded retained formal results: removing the incompatible Qwen3 parser recovered 33/33 missing-answer preflight cases; current extraction takes content after a closing think tag, preserves normally stopped direct answers, and treats unclosed length-limited reasoning as empty. GGUF removes an echoed opening think prefix and uses the standalone source template override; all 1514 current prompt strings and token sequences were checked against that template. Superseded preflight results are excluded.
GGUF math diagnostics tested two failed examples with cache disabled and CPU F16 KV; both remained failures. One also failed native source BF16 CPU inference. This did not establish a GGUF-only fault, and one example is not a full BF16 baseline. Pauses during diagnosis make elapsed time unsuitable as a throughput benchmark.
Refusal protocol: 100 public mlabonne/harmful_behaviors test items, thinking disabled, temperature 0, max output 2048. First 16 overlap the earlier candidate-selection set; the remaining 84 are separately reported in evaluation.json. Four lexical metrics (local17 full text, local17 first 30 whitespace words, normalized Heretic33 and EGA26) are retained together; long-answer topic words can produce false positives. Token-budget hits are disclosed. No semantic refusal rate is inferred from missing markers.