V1 benchmark summary
Old production vs V1
| Context | Acceptance old | Acceptance V1 | Delta | Decode gain | Prefill gain |
|---|---|---|---|---|---|
| Mixed 8k | 54.02% | 75.09% | +21.07 pp | +42.41% | +8.07% |
| Mixed 32k | 40.91% | 61.34% | +20.43 pp | +45.56% | +5.60% |
| Mixed 48k | 55.33% | 54.84% | -0.49 pp | +9.96% | +6.10% |
| Mixed 110k | 54.52% | 58.64% | +4.12 pp | +13.35% | +6.72% |
Chunk 1024 vs 2048 on V1
| Context | Acceptance 1024 | Acceptance 2048 | Delta | Decode gain | Prefill gain |
|---|---|---|---|---|---|
| Mixed 32k | 55.29% | 61.34% | +6.05 pp | +6.28% | +4.89% |
| Mixed 110k | 43.53% | 58.64% | +15.11 pp | +22.45% | +4.79% |
Baked equivalence
| Context | Runtime-tuned | Baked V1 |
|---|---|---|
| Mixed 8k | 0.7509293680 | 0.7509293680 |
| Mixed 32k | 0.6134185304 | 0.6134185304 |
| Mixed 48k | 0.5483870968 | 0.5483870968 |
| Mixed 110k | 0.5864197531 | 0.5864197531 |
Baked equivalence: PASS