Anyone get this working on 4x RTX 6000 Pro? In VLLM or SGLANG

extra_special_tokens in tokenizer_config.json is a list of strings, but transformers (4.49+) expects a dict. The original Moonshot Kimi-K2.5 has it as {}
the PRISM-REAP quant introduced the list by accident. The tokens are already registered in added_tokens_decoder so nothing is lost by setting it to an empty dict.

Try this:

cd /path/to/Kimi-K2.5-PRISM-REAP-530B-A32B-Ex0bit
python3 -c "import json; f='tokenizer_config.json'; d=json.load(open(f)); d['extra_special_tokens']={}; json.dump(d,open(f,'w'),indent=2,ensure_ascii=False);
print('Fixed!')"

i ran it in vllm with this:

export NCCL_P2P_LEVEL=SYS
export NCCL_IB_DISABLE=1
export NCCL_MIN_NCHANNELS=8

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export disable_custom_all_reduce=True
export VLLM_USE_FLASHINFER_MOE_FP16=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export OMP_NUM_THREADS=8
export VLLM_SLEEP_WHEN_IDLE=1
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2,3

vllm serve \
    /nvme/models/safetensors/Kimi-K2.5-PRISM-REAP-530B-A32B-Ex0bit \
    --served-model-name Kimi-K2.5-PRISM-REAP-530B-A32B-A32B-Ex0bit \
    --swap-space 16 \
    --max-num-seqs 4 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.95 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --mm-encoder-tp-mode data \
    --compilation_config.pass_config.fuse_allreduce_rms true \
    --compilation-config '{"level": 3, "cudagraph_mode": "full"}' \
    --disable-custom-all-reduce \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

Yeah, but is spits nonsense

Works great on my setup, benchmarks posted on the discord ;) /r/BlackwellPerformance ftw

Anyone get this working on 4x RTX 6000 Pro? In VLLM or SGLANG

extra_special_tokens in tokenizer_config.json is a list of strings, but transformers (4.49+) expects a dict. The original Moonshot Kimi-K2.5 has it as {}
the PRISM-REAP quant introduced the list by accident. The tokens are already registered in added_tokens_decoder so nothing is lost by setting it to an empty dict.

Try this:

cd /path/to/Kimi-K2.5-PRISM-REAP-530B-A32B-Ex0bit
python3 -c "import json; f='tokenizer_config.json'; d=json.load(open(f)); d['extra_special_tokens']={}; json.dump(d,open(f,'w'),indent=2,ensure_ascii=False);
print('Fixed!')"

i ran it in vllm with this:

export NCCL_P2P_LEVEL=SYS
export NCCL_IB_DISABLE=1
export NCCL_MIN_NCHANNELS=8

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export disable_custom_all_reduce=True
export VLLM_USE_FLASHINFER_MOE_FP16=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export OMP_NUM_THREADS=8
export VLLM_SLEEP_WHEN_IDLE=1
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2,3

vllm serve \
    /nvme/models/safetensors/Kimi-K2.5-PRISM-REAP-530B-A32B-Ex0bit \
    --served-model-name Kimi-K2.5-PRISM-REAP-530B-A32B-A32B-Ex0bit \
    --swap-space 16 \
    --max-num-seqs 4 \
    --max-model-len 200000 \
    --gpu-memory-utilization 0.95 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --mm-encoder-tp-mode data \
    --compilation_config.pass_config.fuse_allreduce_rms true \
    --compilation-config '{"level": 3, "cudagraph_mode": "full"}' \
    --disable-custom-all-reduce \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000

Could you please share your vllm version and transformer version? I got weird output using same params on vllm==0.17.0.

pip freeze | egrep -i "(vllm|transformers)"

transformers==4.57.6
vllm==0.17.1