pipenetwork commited on
Commit
8160d90
·
verified ·
1 Parent(s): cfeee33

Add NVFP4/MXFP4 quant comparison

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -66,6 +66,26 @@ already-quantized build; because expert subsetting is along the expert axis and
66
  affine-quant groups run along the hidden axis, it is **bit-identical to pruning the bf16
67
  source then requantizing**.
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  ## ⚠️ Loading requires the bundled `inkling_mlx` loader
70
 
71
  The `inkling_mm_model` architecture is **not** in stock `mlx-lm` / `mlx-vlm`, so this
 
66
  affine-quant groups run along the hidden axis, it is **bit-identical to pruning the bf16
67
  source then requantizing**.
68
 
69
+ ## Quantization scheme: affine int4 (not NVFP4 / MXFP4)
70
+
71
+ MLX supports FP4 modes and Thinking Machines ships an
72
+ [Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4) checkpoint — so
73
+ for the record, we benchmarked round-trip reconstruction error (‖W − Ŵ‖ / ‖W‖ vs bf16)
74
+ on real Inkling expert weights:
75
+
76
+ | Scheme | bits/weight | reconstruction error |
77
+ |---|---:|---:|
78
+ | **affine int4** (group 64) | 4.50 | **~9.1%** |
79
+ | nvfp4 (group 16) | 4.50 | ~10.2% |
80
+ | mxfp4 (group 32) | 4.25 | ~12.3% |
81
+
82
+ Affine int4 is the most faithful: it is *asymmetric* (per-group scale **and** zero-point,
83
+ 16 uniform levels), which centers on Inkling's near-Gaussian expert weights better than
84
+ symmetric FP4's fixed non-uniform levels (scale only, no zero-point). FP4's real payoff is
85
+ heavy-tailed *activations* and native Blackwell FP4 tensor cores — neither helps weight
86
+ fidelity on Apple Silicon, where MLX would dequantize FP4 anyway. So these builds use
87
+ affine int4; a Mac port of the NVFP4 checkpoint would be *lower* quality at best-equal size.
88
+
89
  ## ⚠️ Loading requires the bundled `inkling_mlx` loader
90
 
91
  The `inkling_mm_model` architecture is **not** in stock `mlx-lm` / `mlx-vlm`, so this