~140 t/s decode on 2x RTX 5090
#7
by geveent - opened
Thank you for this quant. It's become my daily driver for agentic coding.
Setup:
2x RTX 5090 (32 GB each), Xeon (Emerald Rapids), Ubuntu 24.04
Your patched vLLM, 2x32GB build, in Docker
Single request, 262K context
Results:
Decode: about 138–144 t/s
Prefill: about 9,000 t/s
For comparison, the llama.cpp UD-Q4_K_XL on the same box gave 52–56 t/s decode and about 2,000 t/s prefill, so roughly 2.6x faster decode and 4.5x faster prefill. It runs much faster than I expected.
One tip for anyone with a desktop on GPU 0: stop gdm (or the display manager) before launching. Otherwise the shipped gpu-memory-utilization 0.97 fails because the desktop is holding VRAM.
Thank you so much! Great work!