peculiar-ragdoll

/

Tiel-Coder-35B-A3B-GGUF-MTP

like 78

Sorry, what are you asking here? Those logs are basically showing that things have changed in the cache and it needs to reprocess. this is normal.

I would also recommend not to go full q4_0/q4_0 on kv, as it can and will cause problems.

if you can compile your own binaries and are running cuda, you can split k/v to q8_0/q4_0, which in my testing, will still work really well. (this is the -DGGML_CUDA_FA_ALL_QUANTS=ON env)

You may also want to try some of these other options that i use (i'm on a 5090 so i get that you can't do everything, but pay attention to the cram/cache related stuff.

D:\ai\loaders\llamacpp\llama-server.exe ^
  --model .\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf ^
  --alias "200k" ^
  --mmproj Tiel-Coder-35B-A3B-mmproj-BF16.gguf ^
  --no-mmproj-offload ^
  --load-mode none ^
  --n-cpu-moe 0 -ngl all  ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.01 ^
  --repeat-penalty 1.0 ^
  --repeat-last-n 64 ^
  --port 8001 ^
  --ctx-size 200000 ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q4_0 ^
  --spec-draft-ngl all ^
  --fit off ^
  -ctkd q4_0 ^
  -ctvd q4_0 ^
  --no-kv-unified ^
  -b 2048 ^
  -ub 256 ^
  --image-min-tokens 1024 ^
  --image-max-tokens 4096 ^
  --host 0.0.0.0 ^
  --jinja ^
  --no-context-shift --threads -1 ^
  --chat-template-kwargs "{\"preserve_thinking\":true}" ^
  --reasoning on ^
  --reasoning-preserve ^
  --reasoning-budget 8192 ^
  --reasoning-budget-message "\nFinal Answer:" ^
  --reasoning-format deepseek ^
  --metrics -lv 4 ^
  --prio 2 ^
  --ctx-checkpoints 32 ^
  -cram 8192 ^
  --cache-idle-slots ^
  --checkpoint-min-step 8192 ^
  --cache-reuse 256 ^
  --spec-type draft-mtp,ngram-mod,ngram-map-k4v ^
  --spec-draft-n-max 3 --spec-draft-p-min 0.2 ^
  --keep -1 ^
  --no-warmup ^
  --chat-template-file .\froggeric_qwen-chat_template-v22.4.jinja

Hi! :) SolutionsDealer, can you describe what happens in your experience? Does the model just crash? Does it get slow and take a long time to re-load the KV? I can see the log you posted, but it would help me understand the problem if you describe your experience and what happens to the model and the server, not just the log.

Veldierin, thank you so much for you input! :) I see two big issues in your config that will hurt your performance significantly:

1. First of all, you use "--chat-template-file .\froggeric_qwen-chat_template-v22.4.jinja". which is the wrong template for TielCoder! Run the TielCoder directly without adding a chat template, and you get the baked in https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates which is the modified version of froggeric. The modifications I've made is a massive part of why my benchmarks are that much better. I do not recommend anyone overwriting it with stock froggeric, unless they are willing to take a performance hit.
2. You use temperature 1.0, which is not recommended for agentic coding, which is TielCoder's stated use case. Temperature 0.6 is the recommended setting

@peculiar-ragdoll - ah you got me on the template, sorry. you're right, i just cookie-cutter working templates into batch files and didnt adjust it to take into account your own proper template. I'll fix that 😃

but i generally run my qwens at temp 1.0 for my coding work as per Unsloth's and Qwen's recommendations when using thinking, and dont have problems. i've considered 0.6 but havent really A/B'd outputs either way.

There is a nuance to that recommendation: for qwen 3.6, the temp 1.0 recommended is for thinking general, and 0.6 is for agentic coding even with thinking afaik. For qwen 3.8-27b, the recommendation is 1.0 across the board. So for qwen3.6 agentic coding with thinking, you might be leaving some gains on the table! Temp 0.7-0.8 is also a nice balance for agentic coding, if you don’t want to be as aggressive, and want to leave some space for divergent thinking on harder problems.

i dont know much about how to use this area. Thanks to veldierin for such a nice command i applied all and its working fine with me. I actually know how to play with MOE little bit so used all those args and peculiar-ragdoll also guided well. I wanna tell you that model start wrong tool calling while attaching your chat template without it model works fine. I have one question as i dont have much knowledge of Ai model development but i can see Tiel is finetuned version of ornith and ornith is Text generation model then so i think Tiel is Text Gen model not the Image to text so why we need to add mmproj file. though i added but default webui of llama.cpp doesn't reflect if the model is capable of reading images. can you elaborate. Thanks
I am using llama.cpp 10549 default. Activated MCP that its webui offers. Its not just wrong tool calling but it started hellucinating and stuck in loop of calling hugging face mcp severs tool which was un related to my query. in loop until it reached tool calling limit. i again told the mold about its behaiour but it couldn't get escape then i detached the template and its working fine now.
As i wanted to learn more about llama.cpp args please share some more knowledge to play with it. thanks

You don't need to add the mmproj if you don't want image to text, that's just for the people that want that functionality :) Without the mmproj tielcoder is a simple blind text-to-text coder model, which is good enough when you're not doing frontend work where the model needs to look at screen shots and art references etc.

When you say "I am using llama.cpp 10549 default", what does this number 10549 mean? Context? That's not enough for this model. I recommend using 131k or 262k context for agentic coding, which in terms of KV takes about 2.5 and 5 GB RAM at full precission KV, or around 1.5-2.5 GB RAM at q8_0 KV.

You don't need to add the mmproj if you don't want image to text, that's just for the people that want that functionality :) Without the mmproj tielcoder is a simple blind text-to-text coder model, which is good enough when you're not doing frontend work where the model needs to look at screen shots and art references etc.

When you say "I am using llama.cpp 10549 default", what does this number 10549 mean? Context? That's not enough for this model. I recommend using 131k or 262k context for agentic coding, which in terms of KV takes about 2.5 and 5 GB RAM at full precission KV, or around 1.5-2.5 GB RAM at q8_0 KV.

Thanks for your patience. i got fixed the mmproj issue . thanks.