---
language:
- en
license: other
license_name: meta-llama-community-license
license_link: https://llama.meta.com/llama3-3/community-license
base_model: nex-agi/Nex-N2-mini
tags:
- fp8
- quantization
- rtn
- moe
- vllm
- nex-agi
---

# Nex-N2-mini-FP8-RTN

FP8 block-quantized version of [Nex-N2-mini](https://huggingface.co/nex-agi/Nex-N2-mini) using per-expert RTN (round-to-nearest) quantization.

Quantized with RTN in 128×128 blocks using per-expert weight format. MoE expert weights are stored as `float8_e4m3fn`; all other layers (attention, layernorms, router gates, vision encoder, lm_head, embeddings) remain in BF16.

## Format

- **Quant method:** `fp8` (native vLLM format, not `compressed-tensors`)
- **Weight block size:** `[128, 128]`
- **Scale type:** `bfloat16` per-block `weight_scale_inv`
- **Checkpoint size:** ~36 GB (vs ~70 GB BF16 original)
- **Serving:** Requires vLLM with `--linear-backend triton` on Blackwell GPUs

## Inference

```bash
docker run --rm -it --gpus all --shm-size=32g \
  -v /path/to/Nex-N2-mini-FP8-RTN:/models/model \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  /models/model \
    --host 0.0.0.0 --port 8000 \
    --served-model-name Nex-N2-mini-FP8-RTN \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.80 \
    --trust-remote-code \
    --linear-backend triton
```

For tool calling and reasoning, add:

```
    --enable-auto-tool-choice \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

## Technical notes

This is a hand-rolled RTN quantization (no calibration data, no `llm-compressor`). Each expert weight is independently block-scaled using per-block `absmax / 448.0` scaling. See [fp8-moe-rtn-quant](obsidian://open?vault=hermes&file=skills%2Ffp8-moe-rtn-quant) skill for the full methodology and pitfalls.