Sieve-0.8B

Sieve-0.8B is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is the first 16 of 24 blocks of Qwen/Qwen3.5-0.8B-Base, fully fine-tuned, and a scalar scorer head: the smallest model of the family. The code is at github.com/sthanika-ai/Sieve.

Summary

Backbone	Qwen/Qwen3.5-0.8B-Base (revision dc7cdfe2), blocks 1-16 of 24
Design	option fork
Trained weights	full fine-tune of blocks 1-16 (332.1M parameters) and a scalar head; 0.59B parameters run
Question types	choice and noul
Decision Index	Full score 23.1 on the leaderboard (edition 0.3.2, board data of 2026-10-14); public index 32.26 on the 0.3 suite (our own run)
Temperature	T = 1.1208
Token limits	32,768 tokens for the state, the question and the longest option; trained on inputs up to 8,192
GPU memory	the weights take about 1.2 GB in bf16

How it works

The state and the question are encoded once. Every option is listed in the prompt, in an order fixed by a hash of each option's text. The encoded state is then copied once per option, and each option is scored as its own continuation of that copy. No option sees another, so reordering the options cannot change the answer: that holds by construction, not by training. A scalar head reads the last token of each option; a softmax over the options, divided by one built-in temperature, gives the probabilities.

	parameters
run at inference (embeddings + blocks 1-16 of the base, scorer head)	0.59 B
trained (blocks 1-16 + scorer head; embeddings frozen)	332.6 M
full base model (Qwen/Qwen3.5-0.8B-Base text model), for reference	0.75 B

type	options	returns
choice	the caller's keys, optionally described	a probability per key
noul	yes / no, scored as a 2-option choice	P(yes)

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused convolution kernel

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-0.8B")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. load_sieve downloads Qwen3.5-0.8B-Base once and loads the released weights of blocks 1-16 (full_weights.safetensors) and the head over it.
• The model's own call. m.choice(state, question, options) takes a plain list of options and returns {"decision", "index", "confidence", "probs"}.
• Server. sieve-serve --model sthanika-ai/Sieve-0.8B.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-0.8B --out runs/Sieve-0.8B.
• Token limits. A request longer than 32,768 tokens (state, question and longest option) is refused, never trimmed.

Results

Decision Index

Sieve-0.8B is on the Decision Index leaderboard, edition 0.3.2 (board data of 2026-10-14), with a Full score of 23.1. The Full score counts the public benchmarks 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%. Our own run of the edition 0.3 public suite, with the official kit, gives a public index of 32.26. The two numbers measure different things: the public index is the public suite alone, so it is higher than the Full score, which is mostly private tests.

On the board, Sieve-0.8B is third on PhishNChips phishing decisions (92.7% public, 71.8% private) and has a calibration error of 8.8 points.

• Coverage. All 140,178 scoreable requests over 43 benchmarks were run: 140,176 answered and 2 refused (their options have identical text), with no truncation and no error. The index averages 37 of the benchmarks.
• Results. sthanika-ai/Sieve-0.8B-decision-index-results.

area	0.3 public	0.2.1
Knowledge & Reasoning	14.3	11.9
Language Understanding	26.6	26.8
Retrieval & Classification	54.8	54.8
Tools & Automation	51.2	51.2
Arts & Human Taste	13.5	11.3
Index	32.26	31.46

Calibration

One temperature, T = 1.1208, fitted by NLL on a 10,013-question calibration split that covers every track and was never trained on. It never changes which option wins. On a separate 2,388-question validation split (accuracy 75.8%):

	NLL	ECE	confidence − accuracy
raw (T = 1)	0.650	0.029	+0.013
built in (T = 1.12)	0.645	0.024	−0.005

Latency

Single process, one request at a time, bf16, A100 80GB, 400 uniformly sampled suite requests: median 69 ms, mean 92 ms, 80th percentile 91 ms.

Training

• Base. Qwen/Qwen3.5-0.8B-Base, a hybrid of Gated DeltaNet and full-attention blocks; only blocks 1-16 of 24 run.
• Method. Full fine-tune of blocks 1-16 (332.1 M parameters, fp32) plus a scalar head (LayerNorm → 512 → 1); the token embeddings stay frozen. Trained through the same option fork that serves the model.
• Recipe. 1 epoch (5,947 steps), AdamW, lr 5e-6 with 3% warmup and cosine decay to 10%, batch 32 questions, gradient-norm clip 1.0. The learning rate was chosen on a 38,060-question pilot (1e-5, 5e-6 and 2.5e-6 tried; 2e-5 was clearly worse) and compared with a LoRA run of the same size.
• Data. 190,296 questions from 155 decision tracks of a public + synthetic mix, ordered so every part of the epoch sees every track. It covers intent and topic classification, NLI and fact-checking, commonsense and knowledge questions, safety and moderation, preference judging, tool selection and routing, retrieval and long documents, math word problems and rule-based synthetic reasoning (dates, numbers, policies, lookups), plus synthetic phishing-email decisions written for this mix.
  • 32,181 questions (16.9%) come from the train or dev splits of datasets whose test splits appear in the Decision Index: iSarcasmEval 6,803, HellaSwag 5,000, WinoGrande 5,000, CLINC150 3,227, GSM8K 3,167, ESCI 2,989, BANKING77 2,403, RAGTruth 1,378, HoVer 1,308, When2Call 464, ContractNLI 442. No test split was used.
  • Each public dataset keeps its own licence.

Limitations

• Knowledge-heavy and multi-step benchmarks (MMLU-Pro, GPQA, BBH, MuSR) remain weak; a 0.59 B model carries little world knowledge.
• iSarcasmEval, RAGTruth, HLE, CRUXEval and the home-appliance simulator score 0 on the index.
• Single training run, one seed; the learning rate was chosen on a pilot, not tuned on the full run.
• Requires transformers==5.17 (qwen3_5 architecture).

Files

file	contents
full_weights.safetensors	blocks 1-16 of the fine-tuned backbone
head.safetensors	the scalar scorer head
config.json	backbone, read layer, temperature, context limit and training settings
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt	the Qwen3.5 tokenizer

License

Apache-2.0 for the weights and head in this repository, and for the code at github.com/sthanika-ai/Sieve. The Qwen3.5 base model is Apache-2.0. The training datasets keep their own licences.