Christopher Dobler

dobler

8

·

AI & ML interests

ML

Recent Activity

reacted to satgeze's post with 🔥 about 1 month ago

First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists. Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture. What's in the repo: - IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN - IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it - Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects) - Build instructions for the fork, including the two gotchas that cost me three build attempts Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself. https://huggingface.co/satgeze/Hy3-1M-GGUF Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.

reacted to satgeze's post with 🤗 about 1 month ago

First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists. Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture. What's in the repo: - IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN - IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it - Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects) - Build instructions for the fork, including the two gotchas that cost me three build attempts Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself. https://huggingface.co/satgeze/Hy3-1M-GGUF Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.

liked a dataset about 1 month ago

Glint-Research/Fable-5-traces

View all activity

Organizations

None yet

models 0

None public yet

datasets 0

None public yet