Benchmark Updates: GPQA-Diamond, AIME26, LongBenchV2
I'm now running benchmarks with my Ninfer fork and this profile
Results so far:
| Benchmark | KV cache | Context / RoPE | Score |
|---|---|---|---|
| GPQA-Diamond (3-round mean) | INT8 G64 | 262,144 | 87.88 ± 2.62 % |
| GPQA-Diamond (hq A/B) | hq-e8-2b | 262,144 | 89.90 % |
| AIME26 (3-round mean) | INT8 G64 | 262,144 | 93.33 ± 3.34 % |
| AIME26 (hq A/B) | hq-e8-2b | 262,144 | 96.67 % |
| LBv2 short (3-round mean) | INT8 G64 | 262,144 | 67.41 ± 1.94 % |
| LBv2 short (hq A/B) | hq-e8-2b | 262,144 | 66.67 % |
| LBv2 medium (3-round mean) | hq-e8-2b | 524,288 · YaRN 2 | 59.07 ± 2.27 % |
| LBv2 long (3-round mean) | hq-e8-2b | 786,432 · YaRN 3 | 38.89 ± 2.33 % |
Note: hq is my own HyperQuant implmentation to support long context (~1M) within a single 5090 card. The hq kv gives equivalent quality to int8 kv across all benchmarks being measured. The quality drop on the large context is mostly from YaRN scailing. And since it is slower, using nvfp4 is more reasonable unless you need a really large context.
I'm currently running additional benchmarks to verify coding capabilities. After completing one round each of Terminal Bench 2.1 and SWE-Bench Pro, I plan to update the artifact with DFlash2 support. Including DFlash2 weights will increase the artifact size, but they can be optionally loaded into VRAM.
where i can get hq is my own HyperQuant implmentation ? I think that 555,000 nvfp4 is the best value you can get without any significant loss in quality. I don't think there's much point in it, really. Thanks for the tests.
I'm porting it back onto the newest upstream ninfer engine changes. I'll let you know along with the model profile update once it is complete.
HyperQuant is the only option that enables 1M context (YaRN 4) on a single 5090. I'm experimenting with it out of personal curiosity and trying various methods to compensate for quality on the engine side.
@koloved I've added details regarding the HyperQuant implementation. You can find the actual implementation in my ninfer fork branch (feat/hyperquant).