Add v3 task benchmark results and historical EvalScope wall times

README.md

@@ -30,7 +30,7 @@ tags:
   - local-inference
 ---
 
-[![Qwen3.8 Flash CIRU Strix IU4](assets/qwen38-flash-ciru-strix-iu4.jpg)](https://llm.ciru.ai/research)
+[![Qwen3.8 Flash CIRU Strix IU4](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4/resolve/v3.0.0/assets/qwen38-flash-ciru-strix-iu4.jpg)](https://llm.ciru.ai/research)
 
 # Qwen3.8-Flash-CIRU-STRIX-IU4 · runtime v3.0.0
 
@@ -42,14 +42,16 @@ Use the [v3 source archive](runtime/v3.0.0/ciru-runtime-v3.0.0-source.tar.gz) or
 
 Measured on one Ryzen AI Max+ 395 / gfx1151 / 128 GB shared-memory NixOS machine, with one model workload at a time. Inputs are identical token IDs, cold prompt cache, 128 generated tokens, one slot and 262,144-token configured capacity. Nonthinking sampler: temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5, repeat penalty 1, frequency penalty 0 and seed 123. EOS is honored.
 
-| Input tokens | Profile | Prompt tok/s | Generation tok/s | Whole request (s) |
-| ---: | --- | ---: | ---: | ---: |
-| 4,096 | Previous CIRU | 392.00 | 22.52 | 16.34 |
-| 4,096 | CIRU v3 | 455.65 | 24.60 | 14.41 |
-| 4,096 | Halo | 381.49 | 35.30 | 14.69 |
-| 65,536 | Previous CIRU | 284.49 | 13.33 | 239.99 |
-| 65,536 | CIRU v3 | 369.81 | 24.22 | 182.57 |
-| 65,536 | Halo | 263.42 | 23.28 | 254.37 |
+| Input tokens | Profile | Prompt tok/s | Generation tok/s | First streamed piece (s) | Whole request (s) |
+| ---: | --- | ---: | ---: | ---: | ---: |
+| 4,096 | Previous CIRU | 392.00 | 22.52 | 10.70 | 16.34 |
+| 4,096 | CIRU v3 | 455.65 | 24.60 | 9.25 | 14.41 |
+| 4,096 | Halo | 381.49 | 35.30 | 11.09 | 14.69 |
+| 65,536 | Previous CIRU | 284.49 | 13.33 | 230.46 | 239.99 |
+| 65,536 | CIRU v3 | 369.81 | 24.22 | 177.32 | 182.57 |
+| 65,536 | Halo | 263.42 | 23.28 | 248.91 | 254.37 |
+
+**Versus the previous CIRU profile, v3 reduces whole-request time by 11.80% at 4K and 23.93% at 64K.** At 64K, prompt throughput improves 29.99% and generation throughput 81.66%. These are separate metrics; the first-piece times above are the first streamed content-field event.
 
 MTP 2 is a useful optional setting for the tested lower-acceptance long requests, but it was not promoted as the general default. Across 20 short HumanEval requests, v3 MTP 6 measured 53.24 generation tok/s versus 39.63 with MTP 2. Both passed 20/20 base and extended tests. Keeping MTP 6 avoids that short-coding regression; target verification retains the full vocabulary at either depth.
 
@@ -82,6 +84,86 @@ V3 also completed **261,888 input tokens plus 128 generated tokens** at **257.44
 
 The final inference source passes 69 QSA mapping/state/guard cases with flags on and off, 33 actual ROCm operator reference cases and the existing 30 batch allocator tests. A separate short four-prefix diagnostic matches 15,892,480 F32 logits byte-for-byte against the previous runner. Radix selection can change threshold-tie membership and selected-list order; general bitwise equivalence is not claimed.
 
+## Additional task benchmarks · 9 September 2026
+
+The post-release **Ornith difficulty panel** measures complete task execution, including tool work, scoring and harness overhead. All arms ran sequentially on the same Ciru machine using one server slot: previous CIRU v2.0.1 and v3 use MTP6; the unchanged Halo Vulkan arm uses MTP3. These are the same serving profiles described above.
+
+| Benchmark stage | Previous CIRU | CIRU v3 | Halo MTP3 |
+| --- | ---: | ---: | ---: |
+| **Complete mixed panel, after readiness** | **29m 17.11s** | **25m 18.32s** | **24m 34.43s** |
+| Additional model load | 31.17s | 31.15s | 33.62s |
+| Short scored stage | 5m 20.33s | 5m 20.85s | 4m 35.60s |
+| Hermes, two rounds combined | 12m 28.01s | 11m 37.57s | 10m 08.12s |
+| Shared 63K-token history seeding | 3m 26.33s | 2m 42.80s | 3m 21.00s |
+| Long hard stage, history already loaded | 4m 43.92s | 2m 41.90s | 3m 22.96s |
+| Total generated tokens | 33,294 | 33,451 | 31,058 |
+
+V3 takes **13.59% less total time than previous CIRU**, saving 3m 58.79s. Halo finishes 43.89s sooner than v3. Long hard-stage wall time falls **42.98%** against previous CIRU; output lengths differ, so this is a workload result rather than an equal-output decode comparison. Short-stage time is effectively unchanged. Stage rows are components of the full panel; coding health checks and other overhead also contribute.
+
+The panel includes 22 short academic tasks, six Hermes scenarios repeated twice, eight hard tasks returning to a shared long history, and 10 short / eight long coding health checks. Cases were selected from earlier Ornith disagreements and failures. They do not estimate general dataset accuracy or leaderboard performance.
+
+| Profile | Short IFEval strict | Short GSM8K | Short HumanEval | Long IFEval strict | Long GSM8K | Long HumanEval |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Previous CIRU | 5/8 | 8/8 | 5/6 | 2/2 | 2/2 | 2/4 |
+| CIRU v3 | 5/8 | 8/8 | 5/6 | 2/2 | 2/2 | 2/4 |
+| Halo MTP3 | 6/8 | 8/8 | 5/6 | 2/2 | 2/2 | 3/4 |
+
+| Profile | Hermes native full passes | Native mean points / 100 | Reviewed end states |
+| --- | ---: | ---: | ---: |
+| Previous CIRU | 7/12 | 80.83 | 11/12 |
+| CIRU v3 | 11/12 | 95.83 | 12/12 |
+| Halo MTP3 | 11/12 | 95.83 | 12/12 |
+
+Native scores are retained. Reviewed end states separately account for demonstrated grader artifacts; previous CIRU and Halo each had a memory-case wording mismatch. All three arms passed the short 10-task and long eight-task coding health checks on base and extended tests. Two Hermes repetitions are a bounded comparison, not a reliable failure-rate estimate.
+
+Short-task generation rates were **38.44 / 39.09 / 43.97 tok/s** for previous CIRU / v3 / Halo. Long hard-stage generation rates were **21.75 / 35.93 / 31.25 tok/s**. Native short tasks use temperature 0, seed 15035, nonthinking, a 32,768-token output allowance, first samples and no answer repair. Hermes uses temperature 0.6, top-p 0.95, top-k 20, thinking enabled and full remaining context. Long history seeding is timed separately. Interrupted or instrumented captures are excluded.
+
+[Full hard-panel report and case review](benchmarks/v3-followup-20260909/HARD-PANEL.md) · [Measured results JSON](benchmarks/v3-followup-20260909/results.json) · [Wall times CSV](benchmarks/v3-followup-20260909/v3-benchmark-walltimes.csv) · [Interactive research page](https://llm.ciru.ai/research/qwen38-v2/)
+
+### Coding throughput and MTP depth
+
+The separate 20-task HumanEval regression panel quantifies the high-acceptance coding workload. Every profile below passed **20/20 base and 20/20 extended tests**. These are summed API request times, excluding grading overhead.
+
+| Profile | Generated tokens | Prompt tok/s | Generation tok/s | Sum of request times |
+| --- | ---: | ---: | ---: | ---: |
+| Previous CIRU MTP6 | 3,179 | 148.53 | 53.33 | 75.49s |
+| CIRU v3 MTP6 | 3,179 | 219.51 | 53.24 | 70.67s |
+| CIRU v3 MTP2 | 3,212 | 226.25 | 39.63 | 91.57s |
+| Halo MTP3 | 3,241 | 176.10 | 49.48 | 79.15s |
+
+Previous CIRU and v3 MTP6 produced identical token IDs on all 20 tasks. V3 reduces their summed request time by **6.39%**, principally through prefill; decode throughput is effectively unchanged. Different output lengths are retained for MTP2 and Halo. This reinforces retaining MTP6 as the general profile while exposing MTP2 as a workload-specific option.
+
+## Historical full and sampled quality results
+
+These results used the same released model artifacts on earlier runtimes. They are historical results, **not full-suite v3 reruns**; the new v3 checks are shown separately above.
+
+| Benchmark | Historical score | Scope |
+| --- | ---: | --- |
+| HumanEval | 160/164 (97.56%) | Full local-custom chat run |
+| HumanEval+ | 155/164 (94.51%) | Full 164-task EvalPlus extended tests |
+| ARC-Challenge | 1,143/1,172 (97.53%) | Full EvalScope dataset |
+| GPQA-Diamond | 46/50 (92.0%) | Sampled subset |
+| MMLU-Pro | 61/70 (87.14%) | Five questions per subject |
+| GSM8K | 97/100 (97.0%) | Sampled subset |
+| IFEval prompt strict | 92/100 (92.0%) | Sampled subset |
+| ToolEval Standard | 115/138 points (83.33%) | 69 local-custom cases |
+| ToolEval Hard | 23/30 points (76.67%) | 15 local-custom cases |
+
+### Recorded EvalScope wall time
+
+The completed **29 August H96 / MTP1** run used one request at a time and uncapped natural-EOS generation. The table records stage start/end intervals for 1,492 items, excluding setup and earlier failed attempts.
+
+| Dataset | Items | Wall time |
+| --- | ---: | ---: |
+| ARC-Challenge | 1,172 | 24m 42s |
+| GPQA-Diamond | 50 | 1h 33m 21s |
+| MMLU-Pro | 70 | 36m 37s |
+| GSM8K | 100 | 20m 19s |
+| IFEval | 100 | 22m 50s |
+| **Quality suite total** | **1,492** | **3h 17m 50s** |
+
+Including the separate performance probe, the run took **3h 20m 46s**. This is a different workload from the 25-minute v3 mixed panel. No measured full EvalScope wall time is available for v3. [Timestamp-derived records](benchmarks/v3-followup-20260909/results.json) · [Historical quality methodology](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/blob/v1.1/docs/BENCHMARKS.md)
+
 ## Download, build and run
 
 The unchanged model files total **135,962,881,135 bytes (126.625 GiB)**, excluding runtime and reports. The target is 73.945 GiB, the PLE payload 48.828 GiB and the Q8 MTP head 3.852 GiB. Allow at least 160 GiB for model storage/verification plus SDK and build space. Use fast NVMe and a Strix Halo machine with 128 GiB unified memory.

benchmarks/v3-followup-20260909/HARD-PANEL.md

@@ -0,0 +1,103 @@
+# Ornith hard-panel comparison: CIRU v3 and Halo
+
+All three builds completed the existing Ornith 1.5 difficulty panel with **one request at a time**, MTP enabled, and the same native tasks and graders. CIRU v3 took **25m 18.32s**, compared with **29m 17.11s** before and **24m 34.43s** for Halo. Model load is shown separately.
+
+V3 changed total wall time by **-13.59% versus previous CIRU** and **+2.98% versus Halo**. These are complete measured runs, including tool work, scoring, and harness overhead.
+
+| Build | Full wall after readiness | Load | Full wall including load | Total output tokens |
+|---|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 29m 17.11s | 31.17s | 29m 48.28s | 33,294 |
+| CIRU v3, MTP6 | 25m 18.32s | 31.15s | 25m 49.47s | 33,451 |
+| Halo Vulkan, MTP3 | 24m 34.43s | 33.62s | 25m 08.05s | 31,058 |
+
+Halo finished **43.89 seconds sooner than v3 in this run**. V3 had faster prompt processing and long-history generation; Halo had faster short-task generation, fewer total output tokens, and one additional passing case in short IFEval and long HumanEval.
+
+Hardware: Ciru, AMD Ryzen AI Max+395 / gfx1151, 128 GB unified memory, NixOS. The test uses the previously qualified one-slot profiles.
+
+## Correctness
+
+Scores are kept separate by task family. The panel was selected from historical Ornith disagreements and failures; its rates do not estimate general model quality. One native sample per short/long task, exactly two Hermes rounds, no answer repair.
+
+| Build | Short IFEval strict | Short GSM8K | Short HumanEval | Long IFEval strict | Long GSM8K | Long HumanEval |
+|---|---:|---:|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 5/8 | 8/8 | 5/6 | 2/2 | 2/2 | 2/4 |
+| CIRU v3, MTP6 | 5/8 | 8/8 | 5/6 | 2/2 | 2/2 | 2/4 |
+| Halo Vulkan, MTP3 | 6/8 | 8/8 | 5/6 | 2/2 | 2/2 | 3/4 |
+
+| Build | Hermes native full passes / 12 | Native mean points / 100 | Reviewed end states / 12 | Short health, base + extended | Long health, base + extended |
+|---|---:|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 7/12 | 80.83 | 11/12 | 10/10 + 10/10 | 8/8 + 8/8 |
+| CIRU v3, MTP6 | 11/12 | 95.83 | 12/12 | 10/10 + 10/10 | 8/8 + 8/8 |
+| Halo Vulkan, MTP3 | 11/12 | 95.83 | 12/12 | 10/10 + 10/10 | 8/8 + 8/8 |
+
+Health cases are easy regression sentinels; passing them does not imply high general coding quality. Native Hermes grades and reviewed task end states are distinct. Raw native outcome flags are retained in the data; the reviewed column annotates demonstrated grader artifacts without changing native scores. See the reviewed partials below.
+
+## Short-task speed
+
+| Build | Entire scored short stage | API request wall sum | Output tokens | New prompt tokens/s | Generated tokens/s |
+|---|---:|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 320.33s | 260.79s | 8,911 | 269.15 | 38.44 |
+| CIRU v3, MTP6 | 320.85s | 260.45s | 9,311 | 350.34 | 39.09 |
+| Halo Vulkan, MTP3 | 275.60s | 216.37s | 8,255 | 272.61 | 43.97 |
+
+## Hermes agent speed and work
+
+| Build | Round | Wall | Output tokens | API calls | New prompt tokens/s | Generated tokens/s | Reviewed end states |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 1 | 352.55s | 6,933 | 34 | 285.20 | 35.21 | 6/6 |
+| Previous CIRU, MTP6 | 2 | 395.46s | 8,963 | 37 | 264.48 | 35.62 | 5/6 |
+| CIRU v3, MTP6 | 1 | 381.56s | 8,814 | 34 | 375.33 | 36.88 | 6/6 |
+| CIRU v3, MTP6 | 2 | 316.01s | 7,246 | 33 | 364.35 | 37.45 | 6/6 |
+| Halo Vulkan, MTP3 | 1 | 307.74s | 6,705 | 36 | 293.48 | 41.86 | 6/6 |
+| Halo Vulkan, MTP3 | 2 | 300.38s | 7,380 | 35 | 282.75 | 41.78 | 6/6 |
+
+A longer answer or extra tool work can increase wall time despite higher token throughput. Each round has fresh scenario state. The two repetitions expose variation; they do not establish a reliable failure probability.
+
+## Shared long history
+
+| Build | Seed prompt tokens | Seed wall | Seed prompt tokens/s | Long health wall | Long hard wall | Long hard output tokens | Long hard generation tokens/s |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Previous CIRU, MTP6 | 63,000 | 206.33s | 306.51 | 61.91s | 283.92s | 5,498 | 21.75 |
+| CIRU v3, MTP6 | 63,000 | 162.80s | 388.50 | 40.67s | 161.90s | 5,090 | 35.93 |
+| Halo Vulkan, MTP3 | 63,000 | 201.00s | 314.53 | 48.24s | 202.96s | 5,717 | 31.25 |
+
+The same retained archive and task text were rendered with each served Qwen template/tokenizer. The physical server capacity is 262,144 tokens; the original long-workload boundary remains 65,536. Seeding is charged explicitly. These are returns to a shared history, not independently populated histories.
+
+## Case-level agent outcomes
+
+| Case | Previous R1 / R2 | V3 R1 / R2 | Halo R1 / R2 |
+|---|---:|---:|---:|
+| HA-02 | 100 / 50 | 100 / 100 | 100 / 50 |
+| HA-08 | 100 / 100 | 100 / 100 | 100 / 100 |
+| HA-10 | 100 / 100 | 100 / 100 | 100 / 100 |
+| HA-11 | 50 / 50 | 100 / 50 | 100 / 100 |
+| HA-12 | 100 / 100 | 100 / 100 | 100 / 100 |
+| HA-19 | 85 / 35 | 100 / 100 | 100 / 100 |
+
+## Reviewed partials and failures
+
+- **Previous CIRU, MTP6, hermes-1, HA-11, native 50:** The model used a broad edit despite the request for a narrow patch. The required skill content and Verification section survived. The native action penalty remains.
+- **Previous CIRU, MTP6, hermes-1, HA-19, native 85:** Deployment failed, was corrected, and succeeded within one shell tool call. The saved output and follow-up reads verify DEPLOY_OK and the release manifest. The native grader expects a later separate deployment call, so the 85-point result reflects command grouping. The native score remains unchanged.
+- **Previous CIRU, MTP6, hermes-2, HA-19, native 35:** The first attempt failed because deploy.sh lacked executable permission. The harness then explicitly denied the chmod-and-deploy request. The model respected that denial and correctly reported that deployment was incomplete. The task end state was not achieved; respecting the denial was appropriate.
+- **Previous CIRU, MTP6, hermes-2, HA-02, native 50:** All three facts were retained within the memory limit. The native grader at core.mjs:1010 only recognizes “not tag” or “not the tag”; the saved memory says “never tag”. This is a wording-matching artifact. The native score and outcome flag remain unchanged; the reviewed end state is successful.
+- **CIRU v3, MTP6, hermes-2, HA-11, native 50:** The model used a broad edit despite the request for a narrow patch. The required skill content and Verification section survived. The native action penalty remains.
+- **Halo Vulkan, MTP3, hermes-2, HA-02, native 50:** All three facts were retained within the memory limit. The native grader at core.mjs:1010 only recognizes “not tag” or “not the tag”; the saved memory says “never tag”. This is a wording-matching artifact. The native score and outcome flag remain unchanged; the reviewed end state is successful.
+
+## Reproducibility and limits
+
+- Runtime/model source identities, exact launch commands, samplers, templates, prompts, reasoning, native responses, tool traces, scoring decisions, and timings are retained with each run.
+- Previous CIRU and v3 use the same weights. Their runtime and batch/ubatch profiles differ; this measures the complete promoted change. Halo uses its unchanged Vulkan source and published compatible Q8 head. Its larger Q4_K_XL model makes the external result a serving-package comparison.
+- MTP depth is distinct from slot count: previous CIRU and v3 use depth 6; Halo uses depth 3. Every run has one server slot.
+- Native short tasks retain temperature 0, seed 15035, no thinking, and a 32768-token output allowance. Hermes retains temperature 0.6, top-p 0.95, top-k 20, thinking enabled and full remaining context. Full settings are in protocol locks and actual requests.
+- Native PP is newly processed prompt tokens divided by prompt phase seconds. TG follows both runtimes’ native convention: sum(output tokens minus one per request) divided by decode phase seconds. Wall throughput includes additional work and is separate.
+- Short quality was split into a scored three-case smoke and the remaining nineteen cases. Each is counted once. Scorer startup/cleanup is included equally for all builds.
+- Interrupted clarification captures and the canceled concurrency launch are preserved separately and excluded from this completed comparison. No reserve tasks or Ornith-specific BF16 token probes were consumed.
+- Hermes SSE timing fields were normalized from the preserved raw responses after capture; original recorder summaries are retained. This required no new inference.
+
+The original v3 release passed its existing gates before publication. This additional hard-panel test was requested afterward. Published v3 artifacts remain unchanged.
+
+[GitHub v3 release](https://github.com/ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4/releases/tag/v3.0.0) · [Hugging Face v3](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4/tree/v3.0.0)
+
+Evidence checks passed: 81 new official speed rows, 24 official quality rows and 2,972 archived files verified by SHA-256. Raw evidence remains retained by the lab.
+
+[Public structured results](results.json) · [Research page](https://llm.ciru.ai/research/qwen38-v2/)

benchmarks/v3-followup-20260909/SHA256SUMS

@@ -0,0 +1,4 @@
+a8ba7ae6bc569e70dd8cfd36c24aee508d492f9008b2a147caac8d869f2fac40  HARD-PANEL.md
+6c61ed3673fad4ddac8b80bad2421acdd6375120e42ab24fd1813cc02a600e64  results.json
+f7eed8bbdc9d39ffe1ec1aca6efa8ffda8c10202f88cc89730916e127adfaae9  v3-benchmark-walltimes.csv
+de7e2635d44c0ddece7b29392dfeeac0224fa5ac660d76f88dcb237052437201  v3-serving.csv

benchmarks/v3-followup-20260909/results.json

@@ -0,0 +1,738 @@
+{
+  "schema_version": 1,
+  "updated": "2026-09-09",
+  "classification": "measured-results-only",
+  "hardware": "Ciru: Ryzen AI Max+ 395, gfx1151, 128 GB unified memory, NixOS",
+  "baseline": "Locally qualified v2.0.1, MTP6, b2048/u512; no separate public v2.0.1 tag.",
+  "v3_profile": "v3.0.0, same weights, MTP6, b1024/u1024, F16 target KV, Q8 draft KV, one slot, context 262144.",
+  "halo_profile": "Pristine 5f851647fe5ed795dfd6c0a3fba543114879e874, Vulkan, Unsloth UD-Q4_K_XL target, EasiiX Q8 head, MTP3, native defaults.",
+  "serving": {
+    "Previous CIRU": {
+      "4096": {
+        "rows": 2,
+        "PP": 392.00366125294556,
+        "TG": 22.51733524535651,
+        "TG_range": [
+          22.05163555418191,
+          23.003129150072414
+        ],
+        "TTFP_s": 10.69582200050354,
+        "request_s": 16.336945295333862,
+        "drafted": 588,
+        "accepted": 156,
+        "idle_ram_GiB": 100.72896957397461,
+        "peak_ram_GiB": 103.4262580871582,
+        "peak_GTT_GiB": 88.57872772216797,
+        "peak_VRAM_GiB": 0.5218048095703125,
+        "max_delta_ram_GiB": 2.1545333862304688
+      },
+      "65536": {
+        "rows": 1,
+        "PP": 284.4915510098569,
+        "TG": 13.332401115050645,
+        "TG_range": [
+          13.332401115050644,
+          13.332401115050644
+        ],
+        "TTFP_s": 230.45938777923584,
+        "request_s": 239.98778319358826,
+        "drafted": 283,
+        "accepted": 79,
+        "idle_ram_GiB": 100.59429168701172,
+        "peak_ram_GiB": 101.31758499145508,
+        "peak_GTT_GiB": 88.57872772216797,
+        "peak_VRAM_GiB": 0.5173988342285156,
+        "max_delta_ram_GiB": 0.7232933044433594
+      }
+    },
+    "CIRU v3": {
+      "4096": {
+        "rows": 6,
+        "PP": 455.6541833933926,
+        "TG": 24.599275541650545,
+        "TG_range": [
+          23.714691718344156,
+          25.19420366676837
+        ],
+        "TTFP_s": 9.245681206385294,
+        "request_s": 14.40951939423879,
+        "drafted": 1680,
+        "accepted": 480,
+        "idle_ram_GiB": 103.05811818440755,
+        "peak_ram_GiB": 106.01566696166992,
+        "peak_GTT_GiB": 90.2929458618164,
+        "peak_VRAM_GiB": 0.5292892456054688,
+        "max_delta_ram_GiB": 2.4259910583496094
+      },
+      "65536": {
+        "rows": 3,
+        "PP": 369.80548798995795,
+        "TG": 24.219011568105987,
+        "TG_range": [
+          24.166916898916444,
+          24.30324323910564
+        ],
+        "TTFP_s": 177.32060686747232,
+        "request_s": 182.56692719459534,
+        "drafted": 717,
+        "accepted": 258,
+        "idle_ram_GiB": 102.90898895263672,
+        "peak_ram_GiB": 103.66057205200195,
+        "peak_GTT_GiB": 90.2929458618164,
+        "peak_VRAM_GiB": 0.5248832702636719,
+        "max_delta_ram_GiB": 0.7370414733886719
+      }
+    },
+    "Halo": {
+      "4096": {
+        "rows": 4,
+        "PP": 381.49292683846284,
+        "TG": 35.30120650212089,
+        "TG_range": [
+          34.77293274914806,
+          35.57767783656343
+        ],
+        "TTFP_s": 11.094179451465607,
+        "request_s": 14.69240266084671,
+        "drafted": 528,
+        "accepted": 328,
+        "idle_ram_GiB": 102.4269323348999,
+        "peak_ram_GiB": 104.8825912475586,
+        "peak_GTT_GiB": 93.07597351074219,
+        "peak_VRAM_GiB": 1.85498046875,
+        "max_delta_ram_GiB": 2.7664871215820312
+      },
+      "65536": {
+        "rows": 2,
+        "PP": 263.42074609163245,
+        "TG": 23.278333535262185,
+        "TG_range": [
+          23.22488713070601,
+          23.33202649489396
+        ],
+        "TTFP_s": 248.91091644763947,
+        "request_s": 254.3688863515854,
+        "drafted": 276,
+        "accepted": 160,
+        "idle_ram_GiB": 103.14214324951172,
+        "peak_ram_GiB": 103.72273635864258,
+        "peak_GTT_GiB": 93.17262649536133,
+        "peak_VRAM_GiB": 1.9886512756347656,
+        "max_delta_ram_GiB": 0.597412109375
+      }
+    }
+  },
+  "optional_mtp2": {
+    "4096": {
+      "rows": 4,
+      "PP": 453.08718324929157,
+      "TG": 29.386021312616883,
+      "TG_range": [
+        28.85317053177984,
+        29.90677798267714
+      ],
+      "TTFP_s": 9.292197465896606,
+      "request_s": 13.615030109882355,
+      "drafted": 448,
+      "accepted": 284,
+      "idle_ram_GiB": 102.58193302154541,
+      "peak_ram_GiB": 105.33094024658203,
+      "peak_GTT_GiB": 89.85221862792969,
+      "peak_VRAM_GiB": 0.5273895263671875,
+      "max_delta_ram_GiB": 2.2389984130859375
+    },
+    "65536": {
+      "rows": 2,
+      "PP": 373.0762034331891,
+      "TG": 24.880485510349693,
+      "TG_range": [
+        24.87814118364711,
+        24.882830278916934
+      ],
+      "TTFP_s": 175.76338636875153,
+      "request_s": 180.8699688911438,
+      "drafted": 232,
+      "accepted": 136,
+      "idle_ram_GiB": 102.45176124572754,
+      "peak_ram_GiB": 103.16925811767578,
+      "peak_GTT_GiB": 89.85221862792969,
+      "peak_VRAM_GiB": 0.5229873657226562,
+      "max_delta_ram_GiB": 0.7927474975585938
+    }
+  },
+  "coding20": {
+    "Previous CIRU": {
+      "rows": 20,
+      "PP": 148.5313314164119,
+      "TG": 53.327449701233775,
+      "TG_range": [
+        47.918969023381464,
+        57.79477805890751
+      ],
+      "TTFP_s": 0.811668586730957,
+      "request_s": 3.774738335609436,
+      "drafted": 3162,
+      "accepted": 2664,
+      "idle_ram_GiB": 105.73584384918213,
+      "peak_ram_GiB": 108.11417388916016,
+      "peak_GTT_GiB": 88.58134460449219,
+      "peak_VRAM_GiB": 0.5263214111328125,
+      "max_delta_ram_GiB": 0.6338348388671875,
+      "generated_tokens": 3179,
+      "prompt_seconds": 15.181982,
+      "generation_seconds": 59.237784999999995,
+      "total_request_seconds": 75.49476671218872
+    },
+    "CIRU v3 MTP6": {
+      "rows": 20,
+      "PP": 219.50646380829988,
+      "TG": 53.2389694088275,
+      "TG_range": [
+        48.25151585152767,
+        57.44670920116761
+      ],
+      "TTFP_s": 0.5656813144683838,
+      "request_s": 3.533544683456421,
+      "drafted": 3162,
+      "accepted": 2664,
+      "idle_ram_GiB": 107.95920333862304,
+      "peak_ram_GiB": 110.46823501586914,
+      "peak_GTT_GiB": 90.29296112060547,
+      "peak_VRAM_GiB": 0.5336418151855469,
+      "max_delta_ram_GiB": 0.4556770324707031,
+      "generated_tokens": 3179,
+      "prompt_seconds": 10.273046,
+      "generation_seconds": 59.336235,
+      "total_request_seconds": 70.67089366912842
+    },
+    "CIRU v3 MTP2": {
+      "rows": 20,
+      "PP": 226.2516078412486,
+      "TG": 39.63335865299028,
+      "TG_range": [
+        36.41629100986472,
+        41.0896075498267
+      ],
+      "TTFP_s": 0.5504455924034118,
+      "request_s": 4.578490233421325,
+      "drafted": 2180,
+      "accepted": 2119,
+      "idle_ram_GiB": 107.43915119171143,
+      "peak_ram_GiB": 109.95774841308594,
+      "peak_GTT_GiB": 89.85222625732422,
+      "peak_VRAM_GiB": 0.530792236328125,
+      "max_delta_ram_GiB": 0.4302635192871094,
+      "generated_tokens": 3212,
+      "prompt_seconds": 9.96678,
+      "generation_seconds": 80.538216,
+      "total_request_seconds": 91.56980466842651
+    },
+    "Halo MTP3": {
+      "rows": 20,
+      "PP": 176.1027271635372,
+      "TG": 49.48345935868638,
+      "TG_range": [
+        46.09872313796549,
+        51.46158955201079
+      ],
+      "TTFP_s": 0.701993727684021,
+      "request_s": 3.9575579285621645,
+      "drafted": 2502,
+      "accepted": 2415,
+      "idle_ram_GiB": 106.45168437957764,
+      "peak_ram_GiB": 109.05359268188477,
+      "peak_GTT_GiB": 91.11787796020508,
+      "peak_VRAM_GiB": 1.8333740234375,
+      "max_delta_ram_GiB": 0.5126419067382812,
+      "generated_tokens": 3241,
+      "prompt_seconds": 12.805026,
+      "generation_seconds": 65.092458,
+      "total_request_seconds": 79.15115857124329
+    }
+  },
+  "full_capacity": {
+    "input_tokens": 261888,
+    "output_tokens": 128,
+    "rows": 1,
+    "PP": 257.4360682269633,
+    "TG": 17.99524727099241,
+    "TG_range": [
+      17.99524727099241,
+      17.99524727099241
+    ],
+    "TTFP_s": 1017.3788130283356,
+    "request_s": 1024.442313671112,
+    "drafted": 214,
+    "accepted": 90,
+    "idle_ram_GiB": 102.63951873779297,
+    "peak_ram_GiB": 104.53916549682617,
+    "peak_GTT_GiB": 90.98435974121094,
+    "peak_VRAM_GiB": 0.5237464904785156,
+    "max_delta_ram_GiB": 1.8996467590332031
+  },
+  "hard_panel": [
+    {
+      "profile": "Previous CIRU (v2.0.1)",
+      "wall": {
+        "after_ready_seconds": 1757.111019048025,
+        "load_seconds": 31.165580605040304,
+        "total_seconds": 1788.2766006630845
+      },
+      "output_tokens": 33294,
+      "short": {
+        "groups": {
+          "ifeval": {
+            "passed": 5,
+            "total": 8
+          },
+          "gsm8k": {
+            "passed": 8,
+            "total": 8
+          },
+          "humaneval": {
+            "passed": 5,
+            "total": 6
+          }
+        },
+        "whole_stage_wall_seconds": 320.33387160894927,
+        "output_tokens": 8911,
+        "request_wall_seconds_sum": 260.78946709632874,
+        "pp_tps": 269.1525896956804,
+        "tg_tps": 38.43611943974174
+      },
+      "long": {
+        "groups": {
+          "gsm8k": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 956
+          },
+          "humaneval": {
+            "passed": 2,
+            "total": 4,
+            "generated_tokens": 3945
+          },
+          "ifeval": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 597
+          }
+        },
+        "wall_seconds": 283.9170971038984,
+        "generated_tokens": 5498,
+        "tg_tps": 21.749810096227776
+      },
+      "seed_seconds": 206.33202179009095,
+      "seed_input_tokens": 63000,
+      "hermes": {
+        "cases": 12,
+        "native_full_passes": 7,
+        "native_mean_points": 80.83333333333333,
+        "reviewed_end_states": 11,
+        "rounds": [
+          {
+            "round": "hermes-1",
+            "wall_seconds": 352.545,
+            "outcome_count": 6,
+            "generated_tokens": 6933,
+            "api_calls": 34,
+            "native_performance": {
+              "wall_seconds": 352.545,
+              "completed_http_requests": 34,
+              "http_errors": [],
+              "generated_tokens": 6933,
+              "new_prefill_tokens": 29018,
+              "timed_generated_tokens": 6899,
+              "request_prefill_seconds_sum": 101.74647099999997,
+              "request_decode_seconds_sum": 195.91637000000003,
+              "pp_new_tokens_per_s": 285.1990807622213,
+              "tg_pooled_per_request_tokens_per_s": 35.21400483277635,
+              "generation_tokens_per_wall_second": 19.665574607496914,
+              "cache_hit_tokens": 103843,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 6
+          },
+          {
+            "round": "hermes-2",
+            "wall_seconds": 395.462,
+            "outcome_count": 4,
+            "generated_tokens": 8963,
+            "api_calls": 37,
+            "native_performance": {
+              "wall_seconds": 395.462,
+              "completed_http_requests": 37,
+              "http_errors": [],
+              "generated_tokens": 8963,
+              "new_prefill_tokens": 23018,
+              "timed_generated_tokens": 8926,
+              "request_prefill_seconds_sum": 87.02983600000002,
+              "request_decode_seconds_sum": 250.60983700000006,
+              "pp_new_tokens_per_s": 264.48400982853735,
+              "tg_pooled_per_request_tokens_per_s": 35.6171174557685,
+              "generation_tokens_per_wall_second": 22.664630229958885,
+              "cache_hit_tokens": 129379,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 5
+          }
+        ]
+      },
+      "health": [
+        {
+          "label": "health-C1",
+          "wall_seconds": 38.401602292084135,
+          "generated_tokens": 1640,
+          "total": 10,
+          "base_pass": 10,
+          "plus_pass": 10
+        },
+        {
+          "label": "long-health-C1",
+          "wall_seconds": 61.90521405194886,
+          "generated_tokens": 1332,
+          "total": 8,
+          "base_pass": 8,
+          "plus_pass": 8
+        }
+      ]
+    },
+    {
+      "profile": "CIRU v3",
+      "wall": {
+        "after_ready_seconds": 1518.3177031480009,
+        "load_seconds": 31.14865791995544,
+        "total_seconds": 1549.4663619579514
+      },
+      "output_tokens": 33451,
+      "short": {
+        "groups": {
+          "ifeval": {
+            "passed": 5,
+            "total": 8
+          },
+          "gsm8k": {
+            "passed": 8,
+            "total": 8
+          },
+          "humaneval": {
+            "passed": 5,
+            "total": 6
+          }
+        },
+        "whole_stage_wall_seconds": 320.8469852799317,
+        "output_tokens": 9311,
+        "request_wall_seconds_sum": 260.453138589859,
+        "pp_tps": 350.3353163709068,
+        "tg_tps": 39.09385075023305
+      },
+      "long": {
+        "groups": {
+          "gsm8k": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 920
+          },
+          "humaneval": {
+            "passed": 2,
+            "total": 4,
+            "generated_tokens": 3573
+          },
+          "ifeval": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 597
+          }
+        },
+        "wall_seconds": 161.89970062102657,
+        "generated_tokens": 5090,
+        "tg_tps": 35.93143891173188
+      },
+      "seed_seconds": 162.79952381795738,
+      "seed_input_tokens": 63000,
+      "hermes": {
+        "cases": 12,
+        "native_full_passes": 11,
+        "native_mean_points": 95.83333333333333,
+        "reviewed_end_states": 12,
+        "rounds": [
+          {
+            "round": "hermes-1",
+            "wall_seconds": 381.558,
+            "outcome_count": 6,
+            "generated_tokens": 8814,
+            "api_calls": 34,
+            "native_performance": {
+              "wall_seconds": 381.558,
+              "completed_http_requests": 34,
+              "http_errors": [],
+              "generated_tokens": 8814,
+              "new_prefill_tokens": 32229,
+              "timed_generated_tokens": 8780,
+              "request_prefill_seconds_sum": 85.868804,
+              "request_decode_seconds_sum": 238.09672100000003,
+              "pp_new_tokens_per_s": 375.3283905060562,
+              "tg_pooled_per_request_tokens_per_s": 36.87577033032722,
+              "generation_tokens_per_wall_second": 23.100026732502005,
+              "cache_hit_tokens": 97854,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 6
+          },
+          {
+            "round": "hermes-2",
+            "wall_seconds": 316.008,
+            "outcome_count": 6,
+            "generated_tokens": 7246,
+            "api_calls": 33,
+            "native_performance": {
+              "wall_seconds": 316.008,
+              "completed_http_requests": 33,
+              "http_errors": [],
+              "generated_tokens": 7246,
+              "new_prefill_tokens": 23730,
+              "timed_generated_tokens": 7213,
+              "request_prefill_seconds_sum": 65.12937900000001,
+              "request_decode_seconds_sum": 192.581227,
+              "pp_new_tokens_per_s": 364.35170063574526,
+              "tg_pooled_per_request_tokens_per_s": 37.454325701227354,
+              "generation_tokens_per_wall_second": 22.929799245588722,
+              "cache_hit_tokens": 100868,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 6
+          }
+        ]
+      },
+      "health": [
+        {
+          "label": "health-C1",
+          "wall_seconds": 36.41694383090362,
+          "generated_tokens": 1640,
+          "total": 10,
+          "base_pass": 10,
+          "plus_pass": 10
+        },
+        {
+          "label": "long-health-C1",
+          "wall_seconds": 40.67496939992998,
+          "generated_tokens": 1333,
+          "total": 8,
+          "base_pass": 8,
+          "plus_pass": 8
+        }
+      ]
+    },
+    {
+      "profile": "Halo MTP3",
+      "wall": {
+        "after_ready_seconds": 1474.4292242299998,
+        "load_seconds": 33.61671800306067,
+        "total_seconds": 1508.0459434330696
+      },
+      "output_tokens": 31058,
+      "short": {
+        "groups": {
+          "ifeval": {
+            "passed": 6,
+            "total": 8
+          },
+          "gsm8k": {
+            "passed": 8,
+            "total": 8
+          },
+          "humaneval": {
+            "passed": 5,
+            "total": 6
+          }
+        },
+        "whole_stage_wall_seconds": 275.59877648192924,
+        "output_tokens": 8255,
+        "request_wall_seconds_sum": 216.36825466156006,
+        "pp_tps": 272.613431910844,
+        "tg_tps": 43.97370591373599
+      },
+      "long": {
+        "groups": {
+          "gsm8k": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 889
+          },
+          "humaneval": {
+            "passed": 3,
+            "total": 4,
+            "generated_tokens": 4156
+          },
+          "ifeval": {
+            "passed": 2,
+            "total": 2,
+            "generated_tokens": 672
+          }
+        },
+        "wall_seconds": 202.96475215093233,
+        "generated_tokens": 5717,
+        "tg_tps": 31.245162582225024
+      },
+      "seed_seconds": 201.0019980610814,
+      "seed_input_tokens": 63000,
+      "hermes": {
+        "cases": 12,
+        "native_full_passes": 11,
+        "native_mean_points": 95.83333333333333,
+        "reviewed_end_states": 12,
+        "rounds": [
+          {
+            "round": "hermes-1",
+            "wall_seconds": 307.739,
+            "outcome_count": 6,
+            "generated_tokens": 6705,
+            "api_calls": 36,
+            "native_performance": {
+              "wall_seconds": 307.739,
+              "completed_http_requests": 36,
+              "http_errors": [],
+              "generated_tokens": 6705,
+              "new_prefill_tokens": 26972,
+              "timed_generated_tokens": 6669,
+              "request_prefill_seconds_sum": 91.90304900000001,
+              "request_decode_seconds_sum": 159.30165700000003,
+              "pp_new_tokens_per_s": 293.48319009524914,
+              "tg_pooled_per_request_tokens_per_s": 41.86397132077539,
+              "generation_tokens_per_wall_second": 21.78794367954663,
+              "cache_hit_tokens": 106425,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 6
+          },
+          {
+            "round": "hermes-2",
+            "wall_seconds": 300.382,
+            "outcome_count": 5,
+            "generated_tokens": 7380,
+            "api_calls": 35,
+            "native_performance": {
+              "wall_seconds": 300.382,
+              "completed_http_requests": 35,
+              "http_errors": [],
+              "generated_tokens": 7380,
+              "new_prefill_tokens": 19237,
+              "timed_generated_tokens": 7345,
+              "request_prefill_seconds_sum": 68.03446299999999,
+              "request_decode_seconds_sum": 175.788464,
+              "pp_new_tokens_per_s": 282.7537567247353,
+              "tg_pooled_per_request_tokens_per_s": 41.78317412227915,
+              "generation_tokens_per_wall_second": 24.568715835169883,
+              "cache_hit_tokens": 119772,
+              "definitions": "Parsed retained SSE/JSON terminal usage and native timings. PP=prompt_n/prompt_seconds; TG=sum(predicted_n-1)/decode_seconds. Actual round wall includes tools and verifier work. No generation repeated."
+            },
+            "reviewed_outcome_count": 6
+          }
+        ]
+      },
+      "health": [
+        {
+          "label": "health-C1",
+          "wall_seconds": 40.445588815025985,
+          "generated_tokens": 1649,
+          "total": 10,
+          "base_pass": 10,
+          "plus_pass": 10
+        },
+        {
+          "label": "long-health-C1",
+          "wall_seconds": 48.2424732230138,
+          "generated_tokens": 1335,
+          "total": 8,
+          "base_pass": 8,
+          "plus_pass": 8
+        }
+      ]
+    }
+  ],
+  "source_hashes": {
+    "strix-v3-20260908/comparison.json": "147619c25d1abd861c3689a71c384d789fdcd23ae43755615192cf5f6f94e4df",
+    "strix-v3-20260908/hard-panel/comparison.json": "37c27d24704a162788030b2fa4528e7321492934d1b5fa511f4dff4d8ac0471a"
+  },
+  "verification": {
+    "status": "PASS",
+    "official_speed_rows": 81,
+    "official_quality_rows": 24,
+    "archive_files": 2972,
+    "report_sha256": "9b696fd62b38929f7664366abc8104eebd14d9618e952f65cffc2ef952394ca0",
+    "native_scores_unchanged": true,
+    "reviewed_memory_wording_artifacts_apply_to_previous_and_halo": true
+  },
+  "historical_evalscope": {
+    "date": "2026-08-29",
+    "profile": "qwen38-flash-next-h96-mtp-n1-uncapped",
+    "classification": "historical, not rerun on v3",
+    "quality_span_seconds": 11869.696209669113,
+    "including_perf_span_seconds": 12046.300354480743,
+    "stages": [
+      {
+        "stage": "arc",
+        "start_epoch": 1787963313.8788962,
+        "end_epoch": 1787964795.6449215,
+        "seconds": 1481.7660253047943,
+        "tasks": 1172
+      },
+      {
+        "stage": "gpqa_diamond",
+        "start_epoch": 1787964795.763053,
+        "end_epoch": 1787970396.8102598,
+        "seconds": 5601.047206878662,
+        "tasks": 50
+      },
+      {
+        "stage": "mmlu_pro",
+        "start_epoch": 1787970396.8787158,
+        "end_epoch": 1787972593.5807705,
+        "seconds": 2196.7020547389984,
+        "tasks": 70
+      },
+      {
+        "stage": "gsm8k",
+        "start_epoch": 1787972593.6486495,
+        "end_epoch": 1787973813.100671,
+        "seconds": 1219.452021598816,
+        "tasks": 100
+      },
+      {
+        "stage": "ifeval",
+        "start_epoch": 1787973813.1662111,
+        "end_epoch": 1787975183.575106,
+        "seconds": 1370.408894777298,
+        "tasks": 100
+      },
+      {
+        "stage": "perf-1024x1024",
+        "start_epoch": 1787975183.63826,
+        "end_epoch": 1787975360.1792507,
+        "seconds": 176.54099082946777
+      }
+    ],
+    "scores": {
+      "arc": [
+        1143,
+        1172
+      ],
+      "gpqa_diamond": [
+        46,
+        50
+      ],
+      "mmlu_pro": [
+        61,
+        70
+      ],
+      "gsm8k": [
+        97,
+        100
+      ],
+      "ifeval": [
+        92,
+        100
+      ]
+    }
+  }
+}

benchmarks/v3-followup-20260909/v3-benchmark-walltimes.csv

@@ -0,0 +1,4 @@
+profile,full_wall_seconds,load_seconds,short_stage_seconds,hermes_two_rounds_seconds,long_seed_seconds,long_hard_seconds
+Previous CIRU (v2.0.1),1757.111019048025,31.165580605040304,320.33387160894927,748.0070000000001,206.33202179009095,283.9170971038984
+CIRU v3,1518.3177031480009,31.14865791995544,320.8469852799317,697.566,162.79952381795738,161.89970062102657
+Halo MTP3,1474.4292242299998,33.61671800306067,275.59877648192924,608.121,201.0019980610814,202.96475215093233

benchmarks/v3-followup-20260909/v3-serving.csv

@@ -0,0 +1,7 @@
+input_tokens,profile,prompt_tps,generation_tps,first_piece_seconds,request_seconds,repetitions
+4096,Previous CIRU,392.00366125294556,22.51733524535651,10.69582200050354,16.336945295333862,2
+65536,Previous CIRU,284.4915510098569,13.332401115050645,230.45938777923584,239.98778319358826,1
+4096,CIRU v3,455.6541833933926,24.599275541650545,9.245681206385294,14.40951939423879,6
+65536,CIRU v3,369.80548798995795,24.219011568105987,177.32060686747232,182.56692719459534,3
+4096,Halo,381.49292683846284,35.30120650212089,11.094179451465607,14.69240266084671,4
+65536,Halo,263.42074609163245,23.278333535262185,248.91091644763947,254.3688863515854,2