Would you consider GSQ-RCO IQ3_S + MTP for a Qwen3.8-27B coding-focused fine-tune?
Hi,
first of all, thanks for the Qwen3.8-27B GSQ-RCO releases. The IQ3_S results around 3.5 bpw, especially together with MTP, are really interesting for 24 GB GPUs.
I recently asked DavidAU whether GSQ-RCO could also be interesting for his Qwen3.8-27B TURBO / NEO-CODER-MAX variant:
https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF/discussions/8
David has now made the source weights available and explicitly mentioned that they can be used to generate additional quant types:
https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU
This made me wonder whether this checkpoint might be an interesting candidate for your GSQ-RCO pipeline as well, particularly an IQ3_S (~3.5 bpw) + MTP build.
There is also another interesting reference point for this use case: cdiamond's Qwen3.8-27B iMatrix-NVFP4-MTP build:
https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF
That model takes a very different approach, a workload-aware mixed-precision quant targeted at coding/agentic use and Blackwell hardware, but it works extremely well as a practical 24 GB reference. A GSQ-RCO version of a coding-focused Qwen3.8-27B checkpoint would make for a particularly interesting comparison between these two quantization strategies.
I unfortunately don't have the GPU resources locally to reproduce the full GSQ/RCO optimization pipeline properly, my system is based on a single RTX PRO 4000 Blackwell 24 GB.
I can, however, test the resulting GGUF quite extensively and share the results back.
In addition to the usual LLM/runtime benchmarks, throughput, prompt processing, MTP performance and acceptance, VRAM utilization, and behavior with long contexts. I can also test the model in a real-world coding agent/TUI workflow using Crush (my own modification).
That would allow me to compare not only benchmark numbers, but also practical behavior during longer agentic coding sessions: tool use, code edits, instruction following, recovery from mistakes, consistency across multiple steps, and whether differences between the quants are actually noticeable in day-to-day use.
A comparison I'd find particularly interesting would be:
original Qwen3.8-27B GSQ-RCO IQ3_S + MTP
DavidAU NEO-CODER-MAX GSQ-RCO IQ3_S + MTP
DavidAU's existing DI-Matrix Q4_K_S + MTP
cdiamond Qwen3.8-27B iMatrix-NVFP4-MTP as a Blackwell-focused reference
It could also be interesting from a quantization perspective to see whether the RCO tensor allocation changes meaningfully between the original Qwen3.8-27B weights and a heavily modified/tuned checkpoint.
Would this be something you'd be interested in running through your pipeline?
No worries at all if the compute cost makes it impractical, I thought it was worth asking since the source is now available.
If you do decide to generate one, I'd be happy to provide both the raw benchmark results and some real-world feedback from the 24 GB Blackwell + Crush setup.
Thanks for the suggestion!
Just to give some context we’re only two people working on these quantizations with fairly limited hardware resources. Because of that we unfortunately can’t work on multiple models in parallel. For now our priority is to keep up with new and popular model releases such as Qwen3.8-Flash-Next, Qwen3.6-35B-A3B and similar models that are likely to have broader demand.
We’d definitely like to work on more custom or community-requested models as well but at the moment we can’t really give an ETA for when we’ll have the capacity to get to them. Thanks for understanding and for your interest in our quants!
Thank you very much for the honest feedback and also for even considering these quants despite your limited resources.
Of course, there's no rush, I just wanted to suggest it in case it fits into your plans at some point.
I'd be very happy if you could take this up at a later date, but I completely understand that you have to prioritize and can't do everything at once.
Thank you in advance if it ever works out, and continued success with your current releases!
You can use this project to run out your own keep-sets first:
https://github.com/ranxianglei/sglang-expert-profile/tree/main
After there are enough people's keep-sets data files, they can be integrated into a more universal keep-sets to customize the pruning version. After pruning is completed, the IQ3_S quantization technology of GSQ-RCO can be used to obtain the locally available model with the highest cost performance.
I have a 4090 GPU with 48G and 96G of RAM locally, but I'm not sure if it's sufficient to complete the pruning of the 27B model and the IQ3_S quantization. I'm still exploring.
Almost all finetunes suck and statistically they perform worse than the base model.
Dont trust stats u havent faked urself. Here, for me its on Opus 4.7~ level of code quality with the XXS, having enough room to get spoonfed by mazemaker.online
Adaptive KV-Cache, 131k context, etc etc etc... and still having >2.5gb~ room to play on 16gb VRam.
https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming
llama.cpp-adaptive-kv-streaming && \
./build/bin/llama-server \
--model ~/<...>/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf \
--host 0.0.0.0 --port 8080 \
--ctx-size 131072 \
--kv-stream-stage-mib 2048 \
-ctk q8_0 -ctv q4_0 \
-fa on -ngl 99 -t 8 -np 1 \
-b 512 -ub 512 \
--no-mmproj-offload \
--temp 1.0 --top-k 20 --min-p 0.00 --top-p 0.95 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
--reasoning on --reasoning-preserve \
--reasoning-format deepseek --reasoning-budget 32000 \
--chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 24 \
--spec-ngram-mod-n-max 32 \
--image-min-tokens 1024 --image-max-tokens 2048 \
--jinja --cont-batching