The Noise Floor of a Quantization Comparison: Engine, restart, batch and kernel drift, measured against 8-bit and 4-bit effects in two decision models
Abstract
A quantized model is usually judged by comparing it with its BF16 original, but that comparison changes more than the weights. It often changes the serving engine, it always runs on a different server instance, it may change how requests are batched, and on some stacks the library selects a different kernel depending only on which packages import. We measure each of these sources on the same 2,400 public decisions for two of Cloudflare's decision models, Clef-flash (9B) and Clef (27B), on an NVIDIA GB10, and state every quantization effect as the excess over them. Sources that are not quantization move between 0 and 0.92% of decisions. llama.cpp restarts are bit-identical on every record for both models. vLLM restarts flip 0, 5 and 20 decisions at BF16, FP8 and NVFP4 on Clef-flash and 4, 0 and 22 on Clef; a third instance of Clef's quantized builds (exploratory) leaves FP8 at 0 and puts NVFP4 at 4 to 22, depending on which two instances are compared. An engine change at BF16 flips 6 to 16, a batch-size change 8 to 15, and the Gated DeltaNet kernel transformers selects unasked 10 to 14. Quantization inside one engine moves more: for Clef-flash and Clef, 0.88% and 1.04% at Q8_0, 2.12% and 1.62% at FP8, 5.79% and 3.50% at Q4_K_M, and 5.79% and 5.67% at NVFP4. On both models, every 4-bit interval excludes every non-quantization source. Two of our five pre-registered predictions failed, and the same two fail when the tests are rerun on the 27B: FP8 is not inside restart noise, and changing engine moves fewer decisions than 8-bit quantization does. Accuracy hides most of this. Over the full 31,275 public questions, Clef's Q4_K_M build moves accuracy by a net six questions (-0.02 points [-0.20, +0.16]) while 436 of its decisions get worse (exploratory), and NVFP4 costs Clef-flash 0.95 points and Clef 0.65. We release the code, quantization recipes, per-record outputs and the pre-registration.
Community
We measured how much a quantized model's decisions move for reasons that have nothing to do with quantization: a different serving engine, a restart, a different batch size or kernel. On Cloudflare's Clef and Clef-flash, those alone move up to 0.92% of decisions, about what Q8_0 moves (0.88% and 1.04%). 4-bit moves 3.5% to 5.8%.
Code, per-record outputs and the pre-registration are on GitHub, and all six quantized builds are in the collection linked above.
I did exactly this once — moved a model to 4-bit and swapped the serving engine in the same sprint, then spent a week arguing about which of the two cost me a couple of points of agreement on a decision task. Turns out the answer was neither; the restart alone moved the number. Framing effects as excess over controls is the right unit of account, and the restart drift is the part nobody believes until they reproduce it themselves. What I'd want beyond this is the floor as a curve over concurrency rather than a point estimate — kernel and batching effects don't look iid to me, they cluster at high batch and long context, which is exactly where a decision model actually lives, so a single operating point probably understates the swing under load. And put per-request disagreement next to aggregate accuracy: two configs can match on the aggregate while flipping a visible slice of individual decisions, and for a decision model those flipped cases are the whole product.
The second one is the paper's main metric. Every rate in it counts flipped decisions on the same records, and the full-set table puts agreement next to accuracy. That's how Clef's 4-bit GGUF build can move accuracy by a net six questions while 436 of its decisions get worse.
The concurrency curve is a fair gap. We measured one operating point per engine: vLLM at 16 requests in flight, llama.cpp one sequence at a time, and the native reference at batch 1 against batch 8. Li et al. (arXiv:2610.42148) sweep vLLM from 1 to 64 clients and see up to 3.2% of answers change on their setup, which is well above anything we measured. So our floor holds for those operating points, not for a server under heavier load.
I like that the paper separates restart, engine, batch and kernel drift before blaming quantization, since a quantized build is usually compared on a different server and often a different engine too. Accuracy moving by six questions while 436 decisions get worse is a striking contrast. Did those 436 worse decisions cluster in particular kinds of questions, or look spread evenly?
Get this paper in your agent:
hf papers read 2611.15671 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 6
Level5Tech/clef-flash-NVFP4
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper