What is the target size for NVFP4?
#5
by musa334 - opened
The FP8 quant is ~24.5GB and this NVFP4 is ~22.5GB. Why the size gap is so small? I thought NVFP4 would have size ~20GB, and performs similar to Q5 quant...
Sadly NVFP4 has scaling factors and a 16 group size
what is the actual or theoretical advantage to inference with nvfp4 instead of fp8?
what is the actual or theoretical advantage to inference with nvfp4 instead of fp8?
memory requirements and speed (nvfp4 is smaller and faster)
and real life figures?