Crivo-150M

A compact typed-decision encoder for agents and pipelines: on the Pareto frontier of the Decision Index.

Crivo-150M reads a state and a set of typed questions (choice, noul, score) and returns a probability for every option in a single encoder pass. It does not generate text. It is built for the classify, route, gate and verify decisions that agents and pipelines make thousands of times, where a generative model is slow and expensive.

Highlights

• Best in its size class. Maintainer-run Full score of 22.21 on the Decision Index 0.3: first of the 7 submissions at or below 150M parameters, and higher than every submission below 0.75B, including models up to 4× its size.
• One pass, every option scored. choice, noul and score questions share one encoder; all questions of a request are batched, and each returns a full probability distribution.
• Fast and local. Median 20 ms per request on an A100 and 63 ms on a laptop CPU (Apple M2), with no API calls.
• Reproducible and clean. The complete public-suite run is published; training used open data only, no outputs of Jev or any other proprietary decision API, and a three-level contamination check found 0 exact matches with the 0.3 suite.

Architecture	Ettin / ModernBERT encoder, 22 layers, hidden 768, [OPT] readout + MLP scorer
Parameters	149M
Base model	cross-encoder/ettin-reranker-150m-v1
Decision Index 0.3	Full 22.21 (maintainer-run) · public index 36.39 (self-run, 140,178 / 140,178 requests answered)
Latency, 1× A100 40 GB, bf16	median 20 ms · mean 30 ms · p95 51 ms per request
Latency, Apple M2 CPU, 8 threads, fp32	median 63 ms · p95 427 ms per request (813 short requests, 1–3 questions each)
Context	trained up to 8,192 tokens; reads longer inputs whole, up to 32,768, without truncation
Language	English
License	CC BY-NC 4.0 (see License)

Crivo follows the typed-decision interface popularized by TypeSafe's Jev (a state plus typed questions in, a probability for every option out) and runs unmodified in the open Decision Index kit. It is an independent open model, not affiliated with or endorsed by TypeSafe. Crivo is Portuguese for "sieve"; passar pelo crivo, to pass through the sieve, means to be judged and sorted.

Resumo em português

Crivo-150M é um encoder de decisões tipadas com 149M de parâmetros (Ettin/ModernBERT). Ele lê um state e perguntas choice, noul ou score e devolve uma probabilidade para cada opção numa única passada, sem gerar texto. No Decision Index 0.3 tem Full 22,21, avaliado pelos mantenedores com testes privados: é o primeiro entre os 7 modelos de até 150M, está na fronteira de Pareto e fica à frente de todos os modelos abaixo de 0,75B. O índice público é 36,39, com o run completo publicado. Foi treinado só com dados abertos, com verificação de contaminação contra a suíte de avaliação. Funciona em inglês. Licença CC BY-NC 4.0 (uso não comercial).

Contents

• Results
• Quick start
• Model details
• Training
• Contamination firewall
• Intended use and limitations
• License
• See also
• Acknowledgements
• Citation

Results

Decision Index 0.3 — Full score

The board ranks models by the Full score: 20% the public index, 50% private tests of the same skills and 30% private tasks from new domains. The private parts are run by the maintainers. Snapshot of 2026-10-12, 148 submissions:

Full score	Public index	Private, same skills	Private, new domains	Overall rank
22.21	36.39	27.74	9.77	110 / 148

Comparison with models of similar size (Full score; served parameters as reported by the board):

Model	Parameters	Full
Bekko System One v0 68M	68M	8.42
Dinah-0	150M	13.87
tasksource Decider nano	149M	14.90
Crivo-150M	149M	22.21
Bekko System One v0 400M	400M	15.24
DecisionTune 1.0	395M	16.72
vLLM-SR d3-edge	594M	20.24
jiwo 0.8B v2 (next point on the frontier)	752M	25.57

Crivo-150M holds up on private tests of familiar skills (27.74) but drops on tasks from new domains (9.77). This is the main gap to close, and the reason to validate it on your own task before deployment. The chart above is generated from the board's data/index.json by assets/plot_pareto.py.

Decision Index 0.3 — public index (self-run, complete)

Run with the official kit (apolinario/decision-index, commit 9eb2dbe, edition 0.3) on the complete public suite: every one of the 140,178 scoreable requests answered, none unsupported, nothing truncated. Scores are chance-corrected skill (0 = random guessing, 100 = perfect), weighted as the board does. Full results: tardellirs/decision-index-results → runs/crivo-150m.

Public index	Knowledge & Reasoning	Language	Retrieval & Classification	Tools & Automation	Arts & Human Taste
36.39	11.8	41.8	52.2	60.7	10.1

Per-benchmark skill (0–100, chance-corrected) — click to expand

★ = gold benchmark (weight 1.2 within its area). The index uses the native metric listed, chance-corrected.

Benchmark	Native metric	Crivo-150M
Knowledge & Reasoning
GPQA Diamond ★	accuracy	0.0
GSM8K	accuracy	2.0
ChessBench	accuracy	0.0
MuSR	accuracy	26.7
SATA-Bench	case exact accuracy	0.0
CRUXEval	accuracy	7.3
CLadder	accuracy	61.0
HLE ★	accuracy	0.0
MMLU-Pro ★	accuracy	3.4
BBH fixed-option tasks ★	accuracy	11.6
WinoGrande ★	accuracy	21.9
Language Understanding
ContractNLI	macro-F1	74.7
ANLI ★	macro-F1	19.0
HellaSwag ★	accuracy	48.9
ACOS	per-review F1	38.9
FinEntity	macro-F1	76.7
iSarcasmEval	sarcasm F1, track A English	0.0
VAST	macro-F1	61.4
NLI4CT	macro-F1	15.1
RAGTruth	F1 on hallucinated class	44.3
Retrieval & Classification
BANKING77 ★	macro-F1	90.6
CLINC150+OOS ★	macro-F1	91.8
BRIGHT ★	nDCG@10	18.6
Amazon ESCI	macro-F1	29.8
PhishNChips	accuracy (verdict)	4.0
HoVer	accuracy	69.4
Tools & Automation
BFCL ★	case exact accuracy	79.0
ToolRet	nDCG@10	55.7
API-Bank ★	accuracy	29.4
Home appliance simulator	case exact accuracy	79.5
When2Call	accuracy	62.5
Arts & Human Taste
BPoMP	accuracy	0.0
Humicroedit	accuracy	4.5
POP909-CL	accuracy	1.2
cfcolor	accuracy	0.6
Habermas Machine	accuracy	10.1
New Yorker caption matching	accuracy	44.1

• Strong: tool selection (BFCL 79.0, ToolRet 55.7), intent classification (BANKING77 90.6, CLINC150 91.8), long-document NLI (ContractNLI 74.7, HoVer 69.4), hallucination detection (RAGTruth 44.3), entity and aspect sentiment (FinEntity 76.7, ACOS 38.9) and stance (VAST 61.4).
• Weak: knowledge-heavy and multi-step reasoning (GPQA, HLE, MMLU-Pro, GSM8K, chess), since a 150M encoder carries little world knowledge and does not reason step by step, and subjective "taste" benchmarks.

Held-out decision tasks (our own evaluation)

To estimate behaviour on decisions outside the public suite we used two external sets that never entered training: fastino/fast-decisions (17 product domains, dev split) and the English test split of mertkayacs/jevalt-bench. 2,964 requests, 5,989 choice/noul questions, accuracy and multi-class Brier (lower is better).

Set	Questions	Accuracy	Brier ↓
All	5,989	70.4	0.404
fast-decisions (17 domains)	2,600	52.3	0.642
jevalt-bench (en)	3,389	84.4	0.222

These are small proxies for the private "new domains" part of the board, not substitutes for it. jevalt-bench draws on some of the same upstream families as our training data (e.g. MASSIVE-style intents, typed-decisions workflows), which may favour the model there; fast-decisions has no such overlap.

Quick start

Standalone (no kit)

from huggingface_hub import snapshot_download
import sys; path = snapshot_download("tardellirs/crivo-150m"); sys.path.insert(0, path)
from crivo import Crivo

model = Crivo(path)                                 # CUDA if available, else CPU
out = model.answer(
    state={"ticket": "My card was charged twice for the same order."},
    questions={
        "intent": {"type": "choice", "instructions": "What does the customer want?",
                   "criteria": {"refund": "Money back for the duplicate charge",
                                "cancel": "Cancel the order",
                                "info": "Just asking for information"}},
        "urgent": {"type": "noul", "instructions": "The customer needs a fast response."},
    })
print(out["intent"]["choice"], out["intent"]["probabilities"])   # refund {'refund': 0.72, 'cancel': 0.18, 'info': 0.10}
print(out["urgent"]["noul"])                                     # 0.33

Question types:

type	criteria	answer
choice	{key: description or null}, 2+ options (no upper limit)	choice + probabilities over every key
noul	optional {"true": ..., "false": ...} describing both poles	noul = P(statement in instructions is true)
score	ordered list of level descriptions, lowest first	score = expected level + probabilities

state and instructions can be strings or JSON objects (rendered as compact JSON with sorted keys, exactly as in training).

Decision Index kit

pip install "decision-index[transformers] @ git+https://github.com/apolinario/decision-index@9eb2dbe"
hf download tardellirs/crivo-150m --local-dir crivo-150m && cd crivo-150m
python -m decision_index pipeline --engine decision_index_engine:DecisionEngine --edition 0.3 --out runs/crivo-150m

The engine (decision_index_engine.py) defaults to the settings of the reported run: max_length=32768, token_budget=65536, batch_size=32, bf16 autocast on CUDA. It raises Unsupported only for a question above max_length tokens — none in the 0.3 suite.

Model details

Input. Each question is one sequence:

[CLS] instructions [SEP] state [SEP] [OPT] option_1 [OPT] option_2 … [OPT] option_n

where option_i is key: description (or just key). noul is two options (false, true); score levels are options 0: …, 1: …, …. A new special token [OPT] (initialized from [SEP]) marks each option.

Readout. The encoder's hidden state at each [OPT] marker, plus a learned question-type embedding, goes through a small MLP scorer (LayerNorm → Linear → GELU → Linear) to one logit per option; a softmax over the options of the question gives the distribution. All questions of a request are batched together; options of a question see each other (full bidirectional attention), and option order was randomized during training so the model does not rely on position.

Context. ModernBERT/Ettin uses rotary position embeddings with alternating local (128-token) and global attention. Training used sequences up to 8,192 tokens. Inputs longer than that (in the 0.3 suite: 324 requests, almost all ToolRet requests carrying tool documents of ~23k tokens) are read whole with rotary positions up to 32,768 rather than refused or truncated. On inputs up to 8,192 tokens the extended setting is a no-op: on 666 sampled requests (3,378 questions, all 42 benchmarks) it reproduced the 8k run exactly (same choice on 3,378/3,378, max probability difference < 1e-4).

Files. model.safetensors (fp32 weights), config.json (ModernBERT config + decision block), tokenizer files, model.py (architecture, rendering, tokenization), crivo.py (standalone API), decision_index_engine.py (kit engine).

Training

Full fine-tune of cross-encoder/ettin-reranker-150m-v1 in three stages, then a weight average. Every training example is one question with a target distribution over its options (one-hot for labeled data, soft where the source provides probabilities or human/teacher distributions).

Data

Block	Questions	Sources	Purpose
Broad typed decisions	749,943	Bekko System One dataset v0: all 153 training subsets, capped at 6,000 decisions per subset; its 21 ranking subsets re-rendered as per-candidate relevance questions in the suite's retrieval layout (3,000 per subset)	breadth across task families (NLI, QA, classification, safety, ranking, tool use, agent traces, reasoning)
Train splits of the public benchmarks' sources	209,707	train/dev splits (never test) of ANLI, HoVer, When2Call, RAGTruth, VAST, iSarcasmEval, ACOS, NLI4CT, Amazon ESCI, ContractNLI, BANKING77, CLINC150, HellaSwag, WinoGrande, New Yorker caption contest (matching, safe fold), API-Bank — rendered by the kit's own builders with the split swapped	learn the exact request formats and the same skills
Procedural decisions	51,762	tasksource/procedural-typed-decisions (13 generator families, ≤4,000 each)	rule application, arithmetic, state tracking, evidence sufficiency, with exact labels
Tool selection	15,005	Team-ACE/ToolACE, glaiveai/glaive-function-calling-v2 converted to "which supplied tools does this request need"	function selection (BFCL-style)
Causal reasoning	10,000	causalNLP/cladder variants not used by the suite (aggregate, commonsense, anti-commonsense, non-commonsense, easy, hard), minus any prompt present in the suite's pool	causal inference questions (CLadder-style)
Entity-level financial sentiment	9,798	SEntFiN 1.0 (MIT) headlines with per-entity sentiment	FinEntity-style skill from a sibling source
Smart-home control	5,063	280 synthetic houses (randomized rooms, devices, states, groups, policies) generated with the kit's row builder and request templates; no suite or dev case reproduced	multi-question device control (home-appliance style)

No data labeled by Jev or any other proprietary decision API was used.

Recipe

Stage	Init	Data	Steps	LR (encoder / head)	Max length	Tokens per batch
1	Ettin reranker 150M	blocks 1–3 (1,011,412 questions), 1 epoch	36,104	3e-5 / 3e-4	2,048	16,384
2	stage 1	tool selection, CLadder, home control (each ×2), financial sentiment, + 24% replay of stage 1 (311,287)	10,640	2e-5 / 1e-4	2,048	16,384
3	stage 2	long-context: API-Bank with the catalog fitted to the suite's length, ContractNLI / RAGTruth / NLI4CT, + 13% replay of stage 2 (56,690)	12,265	1.5e-5 / 5e-5	8,192	12,288
Final	—	uniform weight average of stages 2 and 3

Common settings: AdamW (β = 0.9 / 0.98, ε = 1e-6, weight decay 0.01 on matrices), linear warmup (5% stage 1, 3% after) and linear decay, bf16 autocast, gradient clipping 1.0, length-bucketed batches, option order shuffled for every choice example. Loss per question: cross-entropy against the target distribution + 0.5 × Brier score.

The weight average combined stage 3's long-context gain (API-Bank) with stage 2's slightly better short-context accuracy; on our panels it was better than either stage alone.

Compute. One NVIDIA A100 40 GB on Lambda Cloud: ~4.3 GPU-hours of training plus ~2 GPU-hours of evaluation; the whole project, including the complete public suite run, cost about US$14.

Development panels

Checkpoints were selected only on our own panels — the held-out decision sets above and a stratified sample of the public benchmarks' pools for format checks — never on the board's scores.

Contamination firewall

The index counts any test item found in an entrant's training data as wrong, so contamination was checked at three levels:

1. Leaf-level filter on the broad and procedural blocks: every string of ≥25 normalized characters in a training example was compared against every string in the rebuilt pools of all 42 benchmarks (1.4M requests, a superset of the test rows); 12,904 matching examples were dropped (mostly ESCI products, GSM8K, ContractNLI contracts, SGD).
2. Builder-level check on the public-source train splits: each generated question compared against the test-split output of the same builder; 939 colliding rows and 1,200 internal duplicates were dropped.
3. Final exact check of every training question (all three stages, ~1.4M) against all 563,686 questions of the final 0.3 suite (normalized state + instructions): 0 matches.

Benchmarks whose sources have no train split, or whose train split overlaps the evaluation pool (GPQA, HLE, MMLU-Pro, GSM8K, BBH, ChessBench, MuSR, SATA-Bench, CRUXEval, POP909, BPoMP, FinEntity, PhishNChips), were used for evaluation only.

Intended use and limitations

Intended use

Fast, local, probabilistic decisions over text or JSON state, as a cheap first pass or a component inside an agent or pipeline:

• routing, intent and topic classification;
• policy and guardrail gates;
• tool selection;
• relevance and hallucination checks, NLI-style verification.

Out of scope

• Open-ended generation, extraction or any task that needs free-text output.
• Expert knowledge questions, math and multi-step reasoning.
• High-stakes decisions about people without human review.

Limitations

• New domains are the weak spot. On the board's private tasks from new domains the model scores 9.77, against 27.74 on private tests of familiar skills. Validate on your own data before relying on it for an unfamiliar task.
• English only. Training data is almost entirely English; other languages are untested and likely weak.
• Little world knowledge, no step-by-step reasoning. Expect near-chance results on expert QA (GPQA, MMLU-Pro, HLE), math word problems, chess and code execution.
• Format sensitivity. The model was trained on the suite's request layouts; very different phrasings of the same task may lose accuracy. Descriptive option text (key: description) works better than bare keys.
• The home-appliance score is optimistic. Our synthetic smart-home cases reuse the suite's request templates (with new houses), which likely explains most of the 79.5 on that benchmark; small models score near zero on the private counterpart.
• Long inputs beyond 8,192 tokens are read without truncation but were never seen in training; quality there is uncertain.
• Calibration is reasonable on our panels (Brier above) but no temperature was fitted; treat probabilities as scores to threshold on your own data.

License

CC BY-NC 4.0. Several training sources are licensed for non-commercial use only (for example ANLI, CC BY-NC 4.0, and some Bekko dataset subsets), so the weights are released for non-commercial use. Upstream dataset licenses and terms continue to apply to the data. The base model (Ettin reranker 150M) is Apache-2.0.

See also

• Crivo-32M, a 32M-parameter sibling for tighter latency budgets.
• Crivo collection.

Acknowledgements

• Ettin (JHU CLSP) and the Ettin reranker used as the starting point.
• Bekko System One (hotchpotch) for the open training dataset.
• Dinah-0 (Lukitaduarte) for the [OPT] readout design this model follows.
• tasksource for the procedural typed-decision data.
• The Decision Index maintainers and its open kit, which made a reproducible comparison possible.

Citation

@misc{crivo150m2026,
  title        = {Crivo-150M: a small typed-decision encoder},
  author       = {Tardelli Stekel},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/tardellirs/crivo-150m}
}