Update README.md

README.md

@@ -20,13 +20,18 @@ tags:
 
 [Training Code](https://github.com/LLM360/xllm) - [Evaluation Code](https://github.com/LLM360/Eval360-V2) - [Pretraining Data](https://huggingface.co/datasets/IFM/K2-Horizon-Pretrain-Data) - [Midtraining Data](https://huggingface.co/datasets/IFM/K2-Horizon-Midtrain-Data)
 
+> [!NOTE]
 > This repository contains an FP8-quantized version of [IFM/K2-Horizon-375B-A23B](https://huggingface.co/IFM/K2-Horizon-375B-A23B). 
 > 
-> Linear layers are quantized to FP8 with block-wise weight scales (128 * 128 blocks) and dynamic per-group (group size 128) FP8 activations. Only the routed experts are quantized. The self-attention projections, the first three dense MLP layers, the MoE router (mlp.gate), the shared experts, and lm_head are all kept in BF16. Weights are stored in the `compressed-tensors` format.
->  
-> The FP8 model achieves evaluation performance that is nearly identical to the original full-precision model, while providing the efficiency benefits of reduced-precision inference.
->
-> Serving note: the routed experts' intermediate size (1792) is not splittable into whole 128-wide quantization blocks at the usual tensor-parallel sizes (TP=4, TP=8), so expert parallelism is required.
+> Only the routed-expert linear layers are quantized to FP8:
+> - **Weights**: static FP8, one scale per 128*128 block.
+> - **Activations**: dynamic FP8, one scale per 1*128 group along the input-channel dim.
+> 
+> All other linear layers (attention, shared experts, routers, the first 3 dense layers, and lm_head) are kept in BF16.
+> 
+> The FP8 model performs closely in line with the original BF16 model on our evaluations, while reducing memory footprint and enabling faster inference on FP8-capable hardware.
+> 
+> **Serving note**: the routed experts' intermediate size (1792) is not splittable into whole 128-wide quantization blocks at the usual tensor-parallel sizes (TP=4, TP=8), so expert parallelism is required.