Is it even worth trying to run the 1-bit version?
According to https://unsloth.ai/docs/models/qwen3.8-next, I can run the 1-bit version of this model, since I have 80 GB of memory available (16 GB VRAM + 64 GB RAM). Is it worth it? Wouldn't it be dumber than the Qwen 3.8 27B q4/q3 model I'm currently working with?
Not so far in my testing. "Write a single html file containing the clasic mobile phone game "snake"" results in a reasoning response of garbage Ie "The user wants to create a new parameter (presiteell). Wait, I need to first understand the logic of what we need to do.
Actually, looking at the user's question "N<_PLACE>
Let me" before abruptly cancelling generation. Speeds not bad at 18t/s (24 layers offloaded, 24 layers in RAM) So somethigns not right wither in the model or in my llama.cpp build - might be a cache issue @ q8
Yeah it was my own fault. I had KV cache q8_0, for now f16 works. As for is it worth it for you, maybe, it's an active 6b model, so it might be worth it on the 16GB VRAM card instead of splitting a dense 27b model, but 1bit is probably not it, wait to see the size of the q3. And for what it'se worth, it one shot a working snake html clone for me, which is pretty decent on 1bit.
4,731 tokens
4min 27s
17.66 t/s
Yeah it was my own fault. I had KV cache q8_0, for now f16 works. As for is it worth it for you, maybe, it's an active 6b model, so it might be worth it on the 16GB VRAM card instead of splitting a dense 27b model, but 1bit is probably not it, wait to see the size of the q3. And for what it'se worth, it one shot a working snake html clone for me, which is pretty decent on 1bit.
4,731 tokens
4min 27s
17.66 t/s
Thanks for the reply! Glad to hear that
@AtticusG3 Could you share your llamacpp configuration? I'm trying to run it on a fork of unsloth's build, but it crashes.
UD-IQ1_S
--jinja
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--cache-type-k f16 --cache-type-v f16
--presence-penalty 0.0 --repeat-penalty 1.0
-sm layer -np 1
--fit on
--flash-attn on -c 15000
--spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 24
kv q8 crashes had to use f16
no mtp (support is wip)
pp is 300 t/s
decode 25 t/s, ngram helping to 40 t/s
system is 5070ti + 5060ti both full, 128gb DDR4
likes to randomly stop and only output 1 word on additional prompts when testing this in pi
decode speed is so far encouraging for heavy ram offload though
@AtticusG3 Could you share your llamacpp configuration (and what is the GPU VRAM and RAM)? I'm trying to run it on a fork of unsloth's build on RTX 5090 and 96RAM, but it crashes.
I'm running it on https://github.com/ggml-org/llama.cpp/pull/27742, 32GB V100 GPU, 128 GB System RAM
# Qwen3.8-Flash-Next (qwen4exp): 125B-A6B + 51B n-gram PLE. Isolated
# llama.cpp (Unsloth PR #27742 + local QSA/RoPE graph fix). UD-IQ1_S is a
# test quant (below the Q4_K_M floor). Attention/shared experts on V100;
# first 27 layers' routed experts in RAM (--n-cpu-moe 27); later 21 on V100.
# f16 KV (q8_0 Hadamard on QSA produced garbage). 24 GPU expert layers + f16
# OOMs the 1.2GB compute buffer. N-gram table stays in RAM.
- model_id: qwen38-flash-next
name: "Qwen3.8 Flash-Next 125B A6B UD-IQ1_S (test)"
aliases: [qwen3.8-flash-next]
command: >
/home/atticsug3/infra/llama.cpp-qwen4exp/build/bin/llama-server
--model /home/atticsug3/models/qwen38-flash-next/UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf
--port {port} --host 127.0.0.1
--n-gpu-layers 999
--n-cpu-moe 27
--override-tensor per_layer_token_embd=CPU
--load-mode none
--ctx-size 262144 --threads 16 --parallel 1 --cont-batching --flash-attn on
--cache-type-k f16 --cache-type-v f16
--jinja --metrics
--temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--split-mode none --device CUDA0
env: ["CUDA_VISIBLE_DEVICES=1", "GGML_CUDA_PDL=0", "LLAMA_ATTN_ROT_DISABLE=1"]
@AtticusG3 Could you share your llamacpp configuration (and what is the GPU VRAM and RAM)? I'm trying to run it on a fork of unsloth's build on RTX 5090 and 96RAM, but it crashes.
I'm running it on https://github.com/ggml-org/llama.cpp/pull/27742
# Qwen3.8-Flash-Next (qwen4exp): 125B-A6B + 51B n-gram PLE. Isolated # llama.cpp (Unsloth PR #27742 + local QSA/RoPE graph fix). UD-IQ1_S is a # test quant (below the Q4_K_M floor). Attention/shared experts on V100; # first 27 layers' routed experts in RAM (--n-cpu-moe 27); later 21 on V100. # f16 KV (q8_0 Hadamard on QSA produced garbage). 24 GPU expert layers + f16 # OOMs the 1.2GB compute buffer. N-gram table stays in RAM. - model_id: qwen38-flash-next name: "Qwen3.8 Flash-Next 125B A6B UD-IQ1_S (test)" aliases: [qwen3.8-flash-next] command: > /home/atticsug3/infra/llama.cpp-qwen4exp/build/bin/llama-server --model /home/atticsug3/models/qwen38-flash-next/UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --port {port} --host 127.0.0.1 --n-gpu-layers 999 --n-cpu-moe 27 --override-tensor per_layer_token_embd=CPU --load-mode none --ctx-size 262144 --threads 16 --parallel 1 --cont-batching --flash-attn on --batch-size 512 --ubatch-size 256 --cache-type-k f16 --cache-type-v f16 --jinja --metrics --reasoning off --reasoning-format deepseek --reasoning-preserve --temperature 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --split-mode none --device CUDA0 env: ["CUDA_VISIBLE_DEVICES=1", "GGML_CUDA_PDL=0", "LLAMA_ATTN_ROT_DISABLE=1"]
How are you using this yaml with llamacpp ?
The yaml is my model-router config, the llama-server stanza is written in it
i'm running flash next as IQ3_M on 16GB VRAM and 64GB RAM. it runs even at usable speed and does seem both smarter and faster than the quantized version of the dense 27B model. (subjective, no systematic test)
however over time KV cache fills up and i start running out of RAM.
sadly Q4 does not work anymore, its too much.
but you can probably go higher than Q1 at least.