Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpw

Ornith-1.5-122B - EXL3 - EAQ v1 - 4.0 bpw

Quantization of ornith-ai/Ornith-1.5-122B (source revision 38c106ad) to about 4 bits per weight with EAQ v1.

Format: standard EXL3. EAQ v1 is the standard EXL3 format (quant_method: exl3, codebook mul1; tensors stored as trellis / suh / svh / mul1) produced by the EAQ conversion. It loads in exllamav3; the tree used for conversion and for every measurement on this page is based on exllamav3 v1.6.1 (other exllamav3 versions were not tested).

Summary

Base model	ornith-ai/Ornith-1.5-122B, 125.09 B parameters, BF16
Architecture	qwen3_5_moe (Qwen3_5MoeForConditionalGeneration): 48 layers, 3:1 linear (GDN) / full attention, 256 routed experts with top-8 routing plus a shared expert, 1 MTP layer, vision tower
Format	EXL3, codebook mul1
Average bits	trellis average 4.049 bpw; 4.19 bpw overall over all 125.09 B parameters including the 16-bit embeddings, vision tower and norms
Size	65.5 GB of weights (65,542,591,754 bytes, 61.0 GiB) in 9 safetensors shards
Context	262,144 positions in the base model config
VRAM	63.4 GiB peak after loading, measured (small cache)
Licence	MIT, as declared by the base model

Bit layout (MEASURED)

Component	Width	Tensors quantized	Size on disk
Routed experts (48 layers x 256 experts x gate/up/down)	4 bit	36,864	58.28 GB (89.0%)
Shared experts	5 bit	144	0.28 GB
Attention (full-attention layers)	5 bit	48	0.59 GB
GDN / linear attention	5 bit	108	2.02 GB
LM head	6 bit	1	0.57 GB
MTP layer (experts, attention, shared experts, fc)	4 bit (norms 16 bit)	776	1.27 GB
Vision tower	16 bit (unquantized)	0	0.90 GB
Token embeddings	16 bit (unquantized)	0	1.53 GB
Norms, router gates, other small tensors	16 bit	0	0.08 GB
Total		37,941	65.52 GB tensor bytes

Structure check against the source checkpoint (MEASURED): all 1,949 source tensors are accounted for, 0 missing and 0 unexpected quantized tensors, 0 per-tensor width mismatches, and all 76,658 floating-point tensors in the pack are finite.

Quality versus BF16 (MEASURED)

Evaluation. 100 held-out conversations (code, tool calling, agentic, reasoning, chat, multilingual), 92,067 scored tokens, compared with the BF16 model. For every token position the full next-token distribution of the BF16 model (teacher) is compared with the quantized model's (student): KL(BF16 || quant) in nats, whether the arg-max token agrees (top-1 agreement), and the negative log-likelihood of the actual next token under each model. Scored tokens are the assistant turn (reasoning text, answer text, tool calls, end of turn); prompt context is not scored. Both models ran with TF32 disabled, over contexts of up to 4,096 tokens. Note: the evaluation prompts were rendered with the Qwen3.8 chat template, not Ornith's own template, so absolute numbers can differ for prompts in Ornith's native format. Confidence intervals are 95% bootstrap intervals that resample whole conversations.

Headline

Metric	Value
Mean KL(BF16 || quant)	0.01347 [95% CI 0.0119, 0.0152] nats/token
Top-1 agreement with BF16	96.7% [95% CI 96.4%, 97.0%]
NLL of the real next token	0.6678 quant vs 0.6705 BF16 (difference -0.0027)
Median per-token KL	0.0011
90th / 95th / 99th / 99.9th percentile KL	0.032 / 0.056 / 0.157 / 0.673
Tokens with KL > 1	0.061% (56 of 92,067)
Tokens with KL > 0.1	2.09%
Largest single-token KL	6.66

The quantized model's NLL on the reference text is within 0.003 nats of the BF16 model's (slightly lower, which is within noise and not a claim of improvement). No other quantization of this model was measured, so there is no comparison to another quant.

By category

Categories prefixed "thinking:" are conversations that contain a reasoning trace.

Category	Conversations	Scored tokens	KL	KL 95% CI	Top-1 agree	NLL BF16	NLL quant
thinking: code	8	21,441	0.0077	[0.0069, 0.0087]	97.3%	0.6031	0.6036
code	13	7,087	0.0092	[0.0056, 0.0187]	98.4%	0.3144	0.3106
thinking: chat	4	5,071	0.0114	[0.0098, 0.0135]	96.3%	0.8141	0.8136
agentic	14	13,867	0.0134	[0.0113, 0.0168]	97.3%	0.4265	0.4178
thinking: stem	8	13,957	0.0138	[0.0121, 0.0159]	96.1%	0.7419	0.7381
thinking: agentic	6	6,912	0.0154	[0.0128, 0.0188]	96.1%	0.6203	0.6150
chat	10	5,986	0.0181	[0.0140, 0.0234]	95.4%	1.1353	1.1320
multilingual	14	8,055	0.0188	[0.0140, 0.0257]	95.8%	0.8913	0.8912
thinking: tool	8	5,425	0.0201	[0.0144, 0.0271]	95.4%	0.9568	0.9557
tool	15	4,266	0.0233	[0.0144, 0.0399]	96.8%	0.6378	0.6398
all	100	92,067	0.0135	[0.0119, 0.0152]	96.7%	0.6705	0.6678

Per-category intervals are wide because each category has only 4 to 15 conversations; differences between neighbouring categories are mostly within noise. Tool calling and chat are the highest-KL categories and code the lowest.

Grouped view

Group	Conversations	Scored tokens	KL	KL 95% CI	Top-1 agree
code (code + thinking: code)	21	28,528	0.0081	[0.0068, 0.0101]	97.6%
tool calling and agentic (incl. thinking variants)	43	30,470	0.0164	[0.0143, 0.0193]	96.6%
chat, multilingual, STEM (incl. thinking variants)	36	33,069	0.0154	[0.0137, 0.0177]	95.9%

By token type

Token type	Scored tokens	KL	Top-1 agree
reasoning text (inside think)	37,707	0.0128	96.2%
answer text	49,751	0.0139	96.8%
tool-call structure tokens	2,148	0.0081	99.5%
tool name	481	0.0113	98.8%
argument key	239	0.0032	99.2%
argument value	1,505	0.0233	97.1%
end-of-turn token	236	0.0248	97.0%

Tool-call structure, tool names and argument keys agree with BF16 on 99.5%, 98.8% and 99.2% of tokens; argument values (free-form strings and numbers) are the least stable tool-call part, with the second-highest KL of any token type.

KL distribution

The mean is dominated by a small tail: the median token has KL 0.0011, while 0.061% of tokens have KL above 1.

Error through the depth of the network

Relative hidden-state error against BF16 after each decoder layer (each model running on its own states) grows smoothly with depth, from 0.5% after layer 0 to about 14% at the output of layer 47, peaking at about 23% around layers 43-44. Expert routing agreement follows the same trend: the router's top-1 expert matches BF16 for 99.1% of tokens in layer 0 and 78.4% in layer 47 (mean over layers 90.3%).

Sanity test (MEASURED)

In exllamav3: the model loads in 30 s using 63.4 GiB of VRAM, and greedy generation is correct on 4 of 4 test prompts (capital of France, 17 x 23, a Python add function, the first five primes) and is deterministic across a repeat run. These are smoke tests, not benchmarks.

Note on the source checkpoint: 13 all-zero experts

13 routed experts in the source checkpoint have weights so small (about 1e-18 to 1e-34) that they are exactly zero in fp16. They are kept as exact zeros in this quant (verified after conversion: dequantized values are exactly 0.0 and finite). The affected experts, as (layer, expert index):

Layer	Expert indices
3	72, 166, 180, 252
4	62, 235, 239
5	210
6	188
7	241
8	82
10	6
13	40

Usage

Requires exllamav3 (tested with a v1.6.1-based tree). The model needs about 64 GiB of VRAM plus cache.

from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/Ornith-1.5-122B-EXL3-EAQv1-4-bpw")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=32768)
model.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

messages = [{"role": "user", "content": "Write a Python function that reverses a linked list."}]
prompt = tokenizer.hf_render_chat_template(messages, add_generation_prompt=True)
print(generator.generate(prompt=prompt, max_new_tokens=512, completion_only=True, encode_special_tokens=True))

Recommended sampling (from the base model's generation_config.json): temperature 0.6, top-k 20, top-p 0.95. Stop tokens: 248046 and 248044.

Files

model-0000N-of-00009.safetensors (weights), model.safetensors.index.json, config.json (with quantization_config), quantization_config.json (per-tensor storage map), tokenizer and template files, generation_config.json, preprocessor configs, and images/ (the charts on this page). Tokenizer, chat template, generation config and preprocessor files are byte-identical to the base model's.

Not tested

Vision inputs, MTP speculative decoding, long-context behaviour, throughput, and public benchmark scores have not been measured for this quant.

Credits

• Ornith AI for the base model, Ornith-1.5-122B.
• turboderp for exllamav3 and the EXL3 format.
• Quantized by MiaAI Lab with the EAQ conversion.