williamliao

/

Qwen3.8-27B-NVFP4-GGUF

Like 10

I'd love to, but I'm basically a hobbyist testing what can realistically run on consumer hardware
Proper MMLU / GSM8K / HumanEval comparisons really need enough VRAM to run the higher-precision/reference variants under the same setup, which I unfortunately don't have.

I can benchmark the variants that fit on my hardware, but without a consistent baseline, I can't really provide a fair apples-to-apples comparison or tell how much quality is actually lost due to quantization.

If anyone with enough VRAM can help run MMLU / GSM8K / HumanEval across the variants, I'd love to include the results.

I'd love to, but I'm basically a hobbyist testing what can realistically run on consumer hardware
Proper MMLU / GSM8K / HumanEval comparisons really need enough VRAM to run the higher-precision/reference variants under the same setup, which I unfortunately don't have.

I can benchmark the variants that fit on my hardware, but without a consistent baseline, I can't really provide a fair apples-to-apples comparison or tell how much quality is actually lost due to quantization.

If anyone with enough VRAM can help run MMLU / GSM8K / HumanEval across the variants, I'd love to include the results.

Well you can use as reference other people benchmark just to compare number.
But yeah i get it don't worry the model is good.

just measure KLD and top-1 like unsloth do, that should be enough. there's also BenchLocal and tool-eval-bench. tool-eval-bench has complete ifeval, mmlu and gsm8k and it's easy to set up.

ran the models through BenchLocal, and the bigger one was better than (old version now) unsloth Q6_K_XL, but the speed/context was limited even on 32GB, and the smaller one is the fastest nvfp4 I found so far, but it thinks for much longer than the bigger one/other good nvfp4 quants. And makes mistakes, so, probably too broken somewhere.

Anyone having an issue where it caps output at 16384 tokens. I can't override it with any settings and I keep running into it.
Tks is great though, 30-40 with 5060ti and 4060

I tested this directly with llama.cpp and couldn't reproduce the 16384-token cap.

Native /completion:

n_predict = 16500
tokens_predicted = 16500
stop_type = limit

OpenAI-compatible /v1/chat/completions:

completion_tokens = 16500
finish_reason = length

So the GGUF and llama.cpp's OpenAI-compatible endpoint can both generate beyond 16384 tokens.

Could you share:

• your llama.cpp version/build
• the frontend/client you're using
• your llama-server command
• the max_tokens / max_completion_tokens value actually sent in the request

It looks like the 16384 cap is being introduced somewhere in the client/configuration rather than by the model itself.

I don't have max tokens set anywhere but I did add it to my pi config just to test it out but it didn't work

llama-b10628-bin-win-cuda-13.3-x64
Pi

cmd: "C:\\llama-swap\\llama\\llama-server --model C:\\llama-swap\\Models\\Qwen3.8-27B-NVFP4-Quality-v2.gguf --port ${PORT} --host 0.0.0.0 --cache-ram 8192 --flash-attn on --ctx-size 140000 --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 4096 --ubatch-size 768 --threads 6 --parallel 1 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 --jinja -ngl 999 -sm layer -ts 1,2 -mg 1 --no-prefill-assistant --spec-type draft-mtp --spec-draft-n-max 3 --mmproj C:\\llama-swap\\Models\\mmproj-Qwen3.8-bf16.gguf --no-mmproj-offload --chat-template-kwargs \"{\\\"preserve_thinking\\\":true,\\\"reasoning_effort\\\":\\\"medium\\\"}\" --chat-template-file C:\\llama-swap\\Models\\qwen_chat_template.jinja --reasoning-format deepseek --repeat-penalty 1.1 --no-context-shift"

I think I found it — this looks like a Pi-side limit rather than llama.cpp or the GGUF.

Pi had a known bug where its llama.cpp provider hardcoded the output limit to 16384 tokens:
https://github.com/earendil-works/pi/issues/6994

It was fixed in PR #7034:
https://github.com/earendil-works/pi/pull/7034

and the fix shipped in Pi v0.82.0.

Could you run:

pi --version

If you're on anything older than v0.82.0, updating Pi should fix it.

If you're already on a newer version, could you also confirm whether you're using Pi's built-in llama.cpp provider or a separate llama.cpp provider/extension?

I verified the model directly against llama.cpp and both /completion and /v1/chat/completions successfully generated 16500 tokens, so llama.cpp itself is able to go past the 16384 boundary.

I'm definitely on the latest pi, so it's odd.

I'll play around a bit more

Thank you

so i updated pi to increase maxtokens in the model config and it generated 44000 tokens in one shot. I'm not understanding why it behaves different than say the unsloth Q4 variant, which doesn't seem to try to bash out so much. But the performance on NVFP4 is great, just trying to understand the behavior