GestaltLabs/Ornstein-Hermes-3.6-27B-MLX-8bit

Ornstein-hermes-3.6-27b — MLX 8-bit

MLX 8-bit build of GestaltLabs/Ornstein-hermes-3.6-27b — a Hermes-format function-calling fine-tune of Ornstein-3.6-27B (Qwen 3.6 27B multimodal). Optimized for Apple Silicon via MLX.

Approximate size: 29.5 GB — Near-lossless 8-bit. ~30 GB; runs on 36 GB+ Macs / DGX Spark.

Support This Work

I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

Support on Ko-fi

---

All MLX Quants

Repo	Bits	Approx. size	Notes
-MLX-16bit	bf16	55.6 GB	Reference quality.
-MLX-8bit	8	29.5 GB	Near-lossless.
-MLX-6bit	6	22.6 GB	High fidelity.
-MLX-4bit	4	15.6 GB	Most popular.

For GGUF (llama.cpp / Ollama / LM Studio) builds, see Ornstein-Hermes-3.6-27b-GGUF.

Picking a quant on Apple Silicon

• 128 GB Mac Studio / DGX Spark → bf16 or 8-bit
• 64 GB M-series → 8-bit or 6-bit
• 36 GB M-series → 6-bit
• 24 GB M-series → 4-bit

Usage

mlx-vlm (CLI)

pip install mlx-vlm
mlx_vlm.generate \
  --model GestaltLabs/Ornstein-Hermes-3.6-27b-MLX-8-bit \
  --image https://example.com/image.jpg \
  --prompt "Describe this image in detail."

mlx-vlm (Python)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("GestaltLabs/Ornstein-Hermes-3.6-27b-MLX-8-bit")
config = model.config

messages = [{"role": "user", "content": "What's in this image?"}]
prompt = apply_chat_template(processor, config, messages, num_images=1)

output = generate(model, processor, prompt, image=["./image.jpg"], max_tokens=512, verbose=True)
print(output)

LM Studio

LM Studio's MLX engine auto-detects this repo and applies the Qwen3 chat template.

Hermes Tool-Calling Format

The model was trained on Hermes-style function calling. Expected message flow:

<|im_start|>system
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags.
<tools>
[{"name": "get_weather", "description": "...", "parameters": {...}}]
</tools>
<|im_end|>
<|im_start|>user
What's the weather in Tokyo?<|im_end|>
<|im_start|>assistant
<think>The user wants weather info. I'll call get_weather.</think>
<tool_call>{"name": "get_weather", "arguments": {"city": "Tokyo"}}</tool_call><|im_end|>
<|im_start|>tool
<tool_response>{"temp_c": 18, "condition": "cloudy"}</tool_response><|im_end|>
<|im_start|>assistant
It's 18°C and cloudy in Tokyo.<|im_end|>

Quantization Details

Source	GestaltLabs/Ornstein-hermes-3.6-27b (bf16)
Format	MLX safetensors, 8-bit (group size 64, affine)
Tool	mlx-vlm mlx_vlm.convert
Hardware	NVIDIA GB10 (DGX Spark) via the MLX CUDA backend

License

Apache 2.0 — inherited from Qwen 3.6 base.

Citation

If you use this model, please consider citing the dataset:

@dataset{lougen_acta_2026,
  author = {DJLougen},
  title = {Acta: A Premium Curated Sample of High-Quality Agentic Tool-Use Conversations},
  year = {2026},
  url = {https://huggingface.co/datasets/DJLougen/Acta}
}