Tiel-Coder-35B vs Ornith-1.5 — Small Local Agent Benchmark
Quick internal test comparison I ran locally on 2× RTX 5060 Ti 16GB using BeeLlama, 32K context, MTP depths 1/3/5, 3 seeds each.
Compared:
- Tiel-Coder-35B Q4_K_XL + official MTPv2
- Ornith-1.5 AD-Q4_K + Shisa12K-MTP
- Ornith-1.5 25G ICE + official MTPv2
Tests:
- Tool calling: multi-step mock API workflow with an irrelevant trap tool that must not be called
- Planning: constrained route-planning task with a known optimal solution
- Coding: autonomous Kotlin/Coroutines bug-fix task using real file tools and ./gradlew test, including hidden tests
Results:
- Tool calling: all 9/9
- Planning: Tiel 93, Shisa 79, ICE 93
- Coding: Tiel 3/3, Shisa 2/3, ICE 0/3
Tiel was also much more efficient in coding: 9–20 tool calls / 10–17K reasoning tokens, compared with up to 32 calls / 43K tokens for ICE.
Decode speed and MTP acceptance were very similar across all three (~112–117 tok/s, ~88% acceptance), with Tiel-Coder-35B being the fastest in this test.
Thanks a lot for the benchmark! <3 Glad to see my baby is holding up for others too :)
@MojoManagement thanks for the tests!
ICE is a quantization method, it doesn't change model's behavior or training.
I've done some local tests too after reading this, @peculiar-ragdoll that's an amazing work, thanks!
I'll run 4 level ICE quantizations with Tiel's imatrix, chat template & MTPv2 then update the table on the Model Card.
Result is amazing and quick with Abliterated trunk.
And got better KLD measurements.
This is really the second-life for Ornith 1.5 35b-a3b.