Smaller GGUF for stock llama.cpp (4.8 GB, fits a 12 GB GPU)
#3
by mike18 - opened
Hi Jared, we've published GGUF builds of Kev-9B that load in stock llama-server (llama.cpp from 9 October 2026 or newer) and answer on /v1/systemone:
https://huggingface.co/turintech/Kev-9B-Artemis-GGUF
The default file is 4.8 GB against 17.9 GB for the BF16 and needs 9.1 GiB of GPU memory instead of 20.0 GiB (NVIDIA A100), so it should fit a 12 GB GPU. It gives the same answer as the original on 92.2% of 2,000 test decisions (92.8% on held-out decisions), with accuracy 67.8% against 69.0%. The ggml-org Q4_K_M is a little closer to the original (93.2%) and 25% larger; the card has the full comparison.
These are independent conversions by TurinTech. Thanks for releasing Kev under Apache-2.0. Happy to answer questions here.