Patch Updates
#1
pinned
by Minachist - opened
I added a patch to speed up TTFT, please re-download the vllm-patch folder and rebuild dockerfile. Weights are unchanged.
Minachist pinned discussion
I've finished testing and adding all the patches I had prepared for vLLM to run Qwen3.8-Flash-Next.
If you run into any bugs or other issues, please let me know in this discussion.
I'm testing it with 4x3090 and 768gb ddr4. So far it is the best one. With kv offload it can go over 3mil ctx.
Thank you!
I'm currently working on patches to offload experts to RAM. I'm not very familiar with this kind of work, so it may take a while.
24x2gb and 32x2gb setups are now supported. Throughput and latency are solid, and it can still handle 1M context. You do need a lot of RAM though.