local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit

Kimi-K3.1-Pro EXL3 2-bit

This repository contains two representations of the same routed-expert quant:

Revision	Contents	Intended use
main	TP-independent exl3-v1 checkpoint; nonexpert tensors retain their source bytes	Prepare an EXL3-aware runtime
gb202	Production-signed Flare package for eight SM120 GPUs	Serve directly without Python, Torch, compilation or autotuning

The immutable prepared release is gb202-tp8-c4-p3072-dflash2-k7-20261108. It includes TP8/DCP8, DFlash2 K7, vision, four concurrent requests and 3072-token prefill graphs. The maximum context is 262,144 tokens per request. Shared physical KV capacity measured on the qualified host is 1,281,024 tokens, not a 1.28M-token single-request window. Prepared files occupy 706.46 GiB; the raw checkpoint is 742.235 GiB.

Run the prepared package

Requirements: Linux x86-64, eight RTX PRO 6000 Blackwell 96 GB GPUs with peer access enabled, NVIDIA Container Toolkit and enough disk for the package cache. Driver 615.71.10 was tested. Other GPU architectures and TP sizes are not qualified by this release. A host with fewer or smaller GPUs cannot load this TP8 package by changing --gpus.

mkdir -p "$PWD/flare-cache"
docker run --detach --name kimi-k31 \
  --user 0:0 --gpus '"device=0,1,2,3,4,5,6,7"' --ipc host \
  --ulimit memlock=-1:-1 --ulimit nofile=65536:65536 \
  --env NVIDIA_DRIVER_CAPABILITIES=compute,utility --env HOME=/root \
  --publish 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/flare-cache,dst=/root/.cache/flare" \
  ghcr.io/local-inference-lab/flare:k31-exl3-tp8-c4-dflash2-20261108 \
  run local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --revision gb202-tp8-c4-p3072-dflash2-k7-20261108 --filename model.safetensors \
  --gpus 0,1,2,3,4,5,6,7 --host 0.0.0.0 --port 8000 \
  --max-concurrency 4 --mtp-draft-tokens 7 --memory-limit=-0G \
  --prefix-cache-memory 4G --prefix-cache-disk-limit 0 \
  --dangerously-accept-publisher local-inference-lab \
  --dangerously-approve-key 937c0143f6856d3ee8ad164f62e3d804bd825d7a8750d0accb9a5849f703a8c1

docker logs -f kimi-k31

The first run downloads the prepared package; subsequent runs reuse the cache. The image is approximately 160 MB and does not contain the model weights. Its immutable digest is sha256:775016c31860bb4451aad0aa16d4773fc0b4a317c90c72ed02d2eb5bcc5c9496. The approval flags explicitly trust the LIL key shown below; omit them and use an interactive container if you prefer the first-use trust prompt. They do not disable signature verification. No private signing key or passphrase is needed to run the model.

The example exposes the API only on the host's loopback address. For remote access, change the host-side port binding and configure --api-key-file before exposing the service. Stop it gracefully with docker stop --time 90 kimi-k31.

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Kimi-K3.1-Pro","messages":[{"role":"user","content":"Write a Python LRU cache with tests."}],"reasoning_effort":"medium","stream":true}'

Chat Completions supports low, medium, high and max; omitted effort defaults to max. Use Chat Completions for exact medium: this runtime's Responses adapter aliases that effort to high. Preserve assistant reasoning and tool-call history when using a multi-turn client.

Runtime settings

Flag	Default	Value used above
--max-concurrency	Package capacity, 4	4; lowering it is allowed, increasing beyond the prepared capacity is not
--mtp-draft-tokens	Package depth, 7	7
--memory-limit	Configuration/package reserve	-0G uses available memory while retaining the native safety guard
--prefix-cache-memory	Configuration, otherwise disabled	4G; automatic intake buffers are additional
--prefix-cache-disk-limit	Configuration; applies when disk caching is enabled	0, disk retention disabled

Prefill 3072, the eight-rank topology, vision and the 262,144-token context limit are prepared into the package. Changing those capacities requires an export, not an unsupported runtime flag. Cache capacity depends on the memory available at startup; check the Allocate cache line in the server log.

Measured serving performance

Conditions: eight RTX PRO 6000 Blackwell Workstation Edition 96 GB GPUs, AMD EPYC 9575F 64-core host, driver 615.71.10, 600 W GPU limits and 16,365 MHz memory clocks (the host's +6000 offset). The image does not apply power limits or overclocks. Export used CUDA 13.2 and Torch 2.14.1+cu132; serving used the native binary in the Docker image above.

Workload	Result
One coding request, median of three prompts, reasoning low	109.04 decode tok/s
Four simultaneous coding requests, common decode interval	158.45 decode tok/s aggregate
Cold 32,768-token prefill, median of three prompts	1,951.16 tok/s
Physical KV shared across requests and cached histories	1,281,024 tokens
Four concurrent image checks	4/4 passed

Decode excludes prefill. Sampling used the recipe defaults: temperature 1, top-p 0.95 and no request seed override. Generated lengths and speculative acceptance vary, so these figures are measurements rather than fixed throughput guarantees. The signed package has the same graph catalogs, cache geometry, speculation configuration and device-allocation sizes as the qualified development package. Four simultaneous approximately 250k-token contexts totaling 1,000,345 prompt tokens completed without OOM, with all exact-recall markers correct. Those long-context checks used the identical prepared computation before production signing.

Estonia and Hotel Lights

The task runner is llm-inference-bench at 52216d7. Each profile uses 30 resamples at concurrency 4 with reasoning_effort=medium, recipe sampling defaults and no seed override. Estonia uses the standard 40,000-token output cap; Hotel Lights omits the client output cap. These are consistency tests of two fixed prompts, not 60 independent benchmark problems.

Profile	Prompt tokens	Correct	Mean completion tokens	Median completion tokens	p90 completion tokens
Estonia	133,140	30/30	672.53	663.5	938.7
Hotel Lights	179	30/30	10,494.67	10,332	13,927.7

Both signed-Docker runs completed without request errors, stalls or truncation. Completion counts include reasoning and the visible answer. The two profiles ran on separate eight-GPU instances on the same host. Machine-readable results are in benchmark-results.json.

Reproduce either profile against a running instance:

git clone https://github.com/local-inference-lab/llm-inference-bench.git
cd llm-inference-bench
git checkout 52216d70865a015b34689626d04a80b89c9dc45f
python3 -m venv .venv
.venv/bin/pip install httpx rich psutil
LLM_BENCH_NO_UPDATE_CHECK=1 .venv/bin/python llm_decode_bench.py \
  --host http://127.0.0.1:8000 --model Kimi-K3.1-Pro \
  --test-profile estonia --profile-concurrency 4 --profile-runs 30 \
  --reasoning-effort medium --completion-stats-save-text \
  --display-mode plain --no-hw-monitor --output estonia-medium.json

For Hotel Lights, replace --test-profile estonia with --test-profile hotel-lights and choose a different output filename. Do not reuse the runner's aggregate_gen_tok_s field as simultaneous wall-clock throughput: at this revision it divides by summed request durations.

How the raw checkpoint was encoded

The source is moonshotai/Kimi-K3.1-Pro at 6d0a0202956796543364d666ab19c6ad1a494ee1. All 82,432 routed experts were encoded from its MXFP4 weights into TP-independent exl3-v1 with the lut_e4m3 codebook. The encoder uses direct tile Viterbi, a 128-step tail-biting context and three-bit scale search. Sign patterns 0 and 6 are evaluated per expert and the lower source-relative coefficient squared error is selected. Activation calibration, LDLQ and QAD are not used.

The raw checkpoint's 2,748 nonexpert tensors retain their source bytes, including attention indexers, BF16 dense/shared projections, vision, embeddings, output head and FP32 parameters. The original source model card, construction metadata, expert manifest and file hashes remain on the immutable raw checkpoint revision.

The prepared runtime is not a BF16-dense reference: it explicitly uses online MXFP8 for eligible dense/vision/draft projections and an MXFP8 draft head. The DSA index-query projection remains BF16 and is output-sharded. Routed expert codes remain EXL3 and use the A16 execution path. Target and draft KV retain the recipe's FP8 layouts; recurrent state is not indiscriminately converted to FP8. No full-model KLD comparison with official MXFP4 is claimed.

Re-export from the pinned sources

Component	Revision
Flare preparation and serving	01327bf2
FlashInfer/B12X	6a842a7d
Raw EXL3 checkpoint	38fbbe2b
DFlash2 draft	RadixArk/Kimi-K3.1-Dflash2-Preview@ea257b3

Both source repositories publish branch release/k31-exl3-tp8-20261108. The container uses Flare's standard Dockerfile at the pinned revision; no uncommitted source overlay is required. FlashInfer is needed for export, not installed in the native serving image: its compiled kernels are in the package.

Preparation needs CUDA 13.2, the Flare build prerequisites, substantial host RAM, and disk for both the raw checkpoint and prepared package. The export host had about 1.1 TiB RAM. Keep preparation configuration outside Git because Ignition writes a development signing key beside it.

git clone --branch release/k31-exl3-tp8-20261108 https://github.com/voipmonitor/flare.git
git clone --recursive --branch release/k31-exl3-tp8-20261108 https://github.com/local-inference-lab/flashinfer.git
git -C flare checkout 01327bf2b0bb4a1626a313bac64a73a749da2e9b
git -C flashinfer checkout 6a842a7d054e6e3e1db32f4c984ca1b9c0460a97
K31_PREP="$PWD/k31-preparation"
mkdir -p "$K31_PREP"
curl -fL https://huggingface.co/local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit/resolve/gb202-tp8-c4-p3072-dflash2-k7-20261108/runtime.toml \
  -o "$K31_PREP/runtime.toml"

cd flare
uv sync --locked --extra prepare --extra test
uv pip install --python .venv/bin/python --no-deps ./heliopause
uv pip install --python .venv/bin/python --no-deps -e ../flashinfer
.venv/bin/python -m ignition.patches
K31_TORCH_SITE=$(.venv/bin/python -c 'import site; print(site.getsitepackages()[0])')
for K31_PATCH in torch-2.14.1-mutation-regions.patch \
  torch-2.14.1-scheduler-membership.patch \
  torch-2.14.1-lazy-input-mutation-events.patch
do
  patch --fuzz=0 --directory "$K31_TORCH_SITE" -p1 < "tools/compiler-patches/$K31_PATCH"
done

.venv/bin/hf download local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --revision 38fbbe2ba5d46141eaa4d662c128bdf8d810746b --local-dir "$K31_PREP/checkpoint"
export PYTHONPATH="$PWD/../flashinfer:$PWD/../flashinfer/flashinfer/experimental/b12x/_compat:$PWD"
ulimit -n 65536
.venv/bin/python -m ignition export "$K31_PREP/checkpoint" \
  --config "$K31_PREP/runtime.toml" \
  --repo-id local-inference-lab/Kimi-K3.1-Pro-EXL3-2bit \
  --gpus 0-7 --max-concurrency 4 --image-concurrency 4 --prefill-chunk-size 3072 \
  --speculative-method dflash2 --mtp-draft-tokens 7 --speculative-token-capacity 32 \
  --speculative-graph-budget 16 --online-mxfp8 --draft-head-quantization mxfp8 \
  --autotune-effort default --compile-workers 8

Do not apply torch-2.14.1-no-clone-custom-op-decomposition.patch; it was not used for this package. The published graph export used the same settings plus --reuse-weights against an already validated package with identical packed weights. Omit that flag for the first export on a fresh host. Download the signed release to obtain the exact captured binaries; autotuning another host can select different kernel plans.

Build the serving image with docker build -t kimi-k31-flare . from the pinned Flare checkout. Production signing used ignition sign on the completed development package, without reloading the model or changing weights/graphs. Private signing material is not distributed. The public profile is publisher.json.

The source model terms are preserved in LICENSE. AI assistance was used for the encoder adaptation, runtime integration, packaging and validation.