---
license: apache-2.0 
base_model:
- thinkingmachines/Inkling
library_name: transformers
pipeline_tag: text-generation
tags:
- amd-quark
- mxfp4
- rocm
- tokenspeed
---

# Inkling-MXFP4

## Model Overview

- **Model Architecture:** Thinking Machines Lab Inkling
  - **Input:** Text, Image, Audio
  - **Output:** Text
- **Inference Engine:** [TokenSpeed](https://github.com/lightseekorg/tokenspeed)
- **Model Optimizer:** [AMD Quark](https://github.com/amd/quark) (0.12.post1+rocm72.torch2.11)
  - **Quantized layers:** MoE routed experts only
  - **Weight quantization:** OCP MXFP4, static
  - **Activation quantization:** OCP MXFP4, dynamic

This model was built by applying AMD Quark MXFP4 quantization to the BF16 Thinking Machines Lab Inkling checkpoint. The quantization targets the MoE routed experts, while attention layers and shared experts are kept in BF16.

## Environment

The quantization workflow was prepared on an AMD gfx950 system. The inspected container environment was:

- **GPU:** AMD MI350/MI355
- **Target graphics version:** gfx950
- **ROCm:** 7.2.1
- **amdgpu driver:** 6.16.13
- **OS:** Linux 6.8.0-84, x86_64
- **Python:** 3.12.3
- **PyTorch:** 2.13.0+rocm7.1
- **AMD Quark:** 0.12.post1+rocm72.torch2.11
- **Safetensors:** 0.8.0
- **Transformers:** 5.13.1

Create and activate the Quark environment:

```bash
python3 -m venv ~/.venv-quark
source ~/.venv-quark/bin/activate
```

Install the required packages:

```bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1
python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simple
python -m pip install safetensors transformers accelerate tqdm
```

## Model Quantization

The model was quantized with the Quark file-to-file flow. This avoids loading the full BF16 checkpoint into GPU memory at once, which is important for very large MoE checkpoints. Run the quantization script:

```bash
python quantize_quark.py \
  --model_dir /path/to/model \
  --output_dir /path/to/output \
  --quant_scheme mxfp4 \
  --file2file_quantization
```

The script applies the model-specific exclusion policy automatically in file-to-file mode. The resulting checkpoint stores MXFP4 routed-expert weights and scales while preserving non-routed-expert components in BF16.

## Deployment

This model can be served with [TokenSpeed](https://github.com/lightseekorg/tokenspeed):

```bash
tokenspeed serve \
  --model lightseekorg/Inkling-MXFP4 \
  --attn-tp-size 4 \
  --moe-tp-size 4 \
  --max-model-len 81920 \
  --max-num-seqs 16 \
  --max-prefill-tokens 8192 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 \
  --disable-cuda-graph-padding \
  --trust-remote-code \
  --dtype bfloat16 \
  --disable-kvstore \
  --kvstore-ratio 0 \
  --block-size 128 \
  --host 127.0.0.1 \
  --port 22015
```

## Evaluation

The following validation results are placeholders and will be updated before public release.

| Benchmark | BF16 Reference | MXFP4 |
| --- | ---: | ---: |
| BFCL exact calls | 78.3% | 79.1% |
| BFCL all-live macro | 75.4% | 75.3% |
| MMAU | 77.2% | 76.0% |
| GPQA Diamond | 88.1% | 85.4% |
| AIME26 | 96.4% | 96.7% |