Evals
Do you have any comparisons to MTP of Qwen 3.6 35B or other speculative decoding methods?
Happy to run some comparisons! Will update here.
How to reproduce the acceptance rates in L20?
Do you mind sharing the vllm version and command? thanks!
Do you mind sharing the vllm version and command? thanks!
vllm-0.25.0，4*L20
vllm serve /dev/shm/Qwen3.6-35B-A3B \
--trust-remote-code \
--enable-prompt-tokens-details \
--tensor-parallel-size 4 \
--max-num-seqs 128 \
--max-num-batched-tokens=8K \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--enable-prefix-caching --mamba-cache-mode align \
--enable-expert-parallel \
--override-generation-config '{"temperature":0.6}' \
--served-model-name default_model \
--max-model-len 262144 \
--language-model-only \
--gpu-memory-utilization 0.7 \
--enable-mfu-metrics \
--speculative-config '{"model":"/dev/shm/Qwen3.6-35B-A3B-speculator.dspark","num_speculative_tokens":8,"method":"dspark"}' \
--port 9260
https://github.com/vllm-project/speculators/blob/main/scripts/evaluate/evaluate.py
python3 evaluate.py \
--target http://127.0.0.1:9260/v1 \
--dataset RedHatAI/speculator_benchmarks \
--data-column-mapper 'kind=generative_column_mapper,column_mappings.text_column=prompt' \
throughput \
--subsets HumanEval \
--max-requests 80 \
--max-concurrency 8 \
--output-dir /tmp/dspark_user/guidellm_official/HumanEval
Huh, Could you try a newer version of vLLM with temperature 0? Thanks!
Huh, Could you try a newer version of vLLM with temperature 0? Thanks!
vllm-0.27.1 with temperature 0.6 seems good.
Awesome! Glad it worked out!
Huh, Could you try a newer version of vLLM with temperature 0? Thanks!
vllm-0.27.1 with temperature 0.6 seems good.
Thank you. Are you able to compare it to MTP from the official checkpoint? I would like to know the difference between DSpark from RedHatAI and MTP in your experiments.