local-inference-lab

/

Kimi-K3.1-Pro-EXL3-2bit

Like 2

Follow

local-inference-lab 210

We run an 8x DGX Spark (GB10, SM121) cluster on a switched CX-7 RoCE fabric (one rank per host) and would love to serve this model there.

What we validated so far on Sparks, at this release's pins (flare 01327bf2 / flashinfer 6a842a7d):

• Flare builds and runs natively on arm64/SM121; the flare binary correctly detects hardware revision gb10 (then 404s, since only gb202-tp8-* is published).
• The Ignition prep environment works on GB10: torch 2.14.1+cu132 on driver 580.x, the b12x io_uring loader, and the b12x RoCE proxy all run; b12x kernel autotune (attention.varlen, gemm.blockscaled_precision) compiles and measures on sm_121, observed during a GLM-5.3-Flash TP2 multi-host prep across two Sparks (rendezvous and fabric checkpoint streaming working end to end).
• For Kimi K3.1 we attempted an 8-rank multi-host export (one rank per Spark). The HostRendezvous 2-4 rank guard was easy to widen experimentally (collective_ladder already accepts world size 8), but kimi3.recipe.prepare_variant rejects multi-host, and from the code the reason looks structural: the DCP latent/index page pools are exchanged through CUDA IPC (pool/dcp/routing), which cannot cross hosts.

Questions:

1. Is a gb10 revision of this repo planned (given flare download already resolves @gb10)? Any rough timeline?
2. Are the fabric-gather work branches (tagged-dcp-gather, pcie-sequence-gather, experimental-spark-ring) headed toward multi-host kimi31 prep/serve over RoCE?
3. Anything you'd like tested on an 8-Spark switched RoCE fabric? Happy to run experiments and report numbers.

Thanks for publishing the full stack - the multi-host, preparation, and handoff docs made it straightforward to get this far.