shisa-ai/shisa-de-2.1

Shisa DE-2.1

Shisa DE-2.1 (Decision Engine 2.1) is a general-purpose decision model. It takes one state and a list of named questions and returns one typed answer per question: yes/no, or one of up to 256 options, each with a probability. No answer text is generated. Each answer is read from the next-token distribution at the answer boundary, restricted to the option codes.

DE-2.1 is based on Gemma 4 26B A4B. Its weight update is the average of eleven rank-16 LoRA runs on the language model's attention projections, trained with different seeds and merged into the weights. Each run was trained with softmax cross-entropy over the restricted option distribution, which is the same readout used at inference. The vision encoder is the base model's and was not trained.

DE-2.1 is a drop-in replacement for Shisa DE-2: the prompt layout, the readout contract and the inference policy are unchanged, and only the weights differ.

What changed from DE-2

	DE-2.1	DE-2
Weights	Mean of eleven runs' LoRA updates, merged	One training run (seed 13)
Option order in training	Re-rendered every epoch: source, reversed or shuffled	Shuffled once
Training data	15,373 questions per epoch from 31 families, redrawn every epoch	14,637 questions per epoch from 28 families, drawn once
New training families	Tool availability (when no listed tool fits), none-of-the-above options, evidence mentions
Decision Index 0.3, public index, release policy	60.89	58.89
When2Call, cannot-answer track	90.0%	59.5%

DE-2.1 started as an experiment testing out some of RSI-Jev's training techniques. Of those, only the model-souping had a measurable positive impact. While reviewing the results, we also made some improvements in data draw and ordering, as well as some targeted data improvements: rebalancing select data labels, adding tool-availability, none-option and evidence-mention supervision, and strengthening hidden-text training with scanner-labeled untrusted-data examples.

DE-2's published Decision Index of 58.96 is from another run of the same weights; two runs differ by about 0.15.

The Shisa DE models are designed to run on existing inference engines (vLLM, SGLang, llama.cpp, etc.) without requiring any modifications. You can interact with the models directly using the readout contract and inference policy described below; the Shisa DE SDK makes this easier.

Model details

Field	Value
Model id	shisa-ai/shisa-de-2.1
Developed by	Shisa AI
Model type	Causal language model; no task head is added to the graph
Base model	google/gemma-4-26B-A4B-it
Architecture	Gemma 4 mixture of experts: 30 layers, 128 experts with 8 active plus 1 shared, 2,816 hidden size, 262,144-token vocabulary
Parameters	25.2B total and 3.8B active per the base model card
Trained parameters	11.5M per run (LoRA rank 16, alpha 32, on q_proj, k_proj, v_proj, o_proj of the language model); the eleven-run mean is a rank-176 update, merged
Precision	bfloat16
Size on disk	51,612,009,636 bytes (48.1 GiB) in 3 safetensors shards
Max context used in evaluation	262,144 tokens
License	Apache-2.0

Serving

Readout contract

Send one request per question. Render the state, the question and the option list with the chat template, then read logprobs at the answer position with max_tokens=1. Restrict the logits to the valid option codes and normalize. Take the argmax for a choice answer, or the probability of the Yes row for a noul answer. The model's unrestricted next-token distribution is not part of the contract.

The scaffold is a system instruction and a JSON payload in the user message:

<bos><|turn>system
Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. Respond with only its uppercase letter, with no explanation or reasoning.<turn|>
<|turn>user
{"evidence": <state>, "criterion": <question>, "options": [{"letter": "A", "description": ...}, {"letter": "B", "description": ...}]}<turn|>
<|turn>model

Options are labelled A to Z, then AA, AB and so on up to 256. Every code is a single token. A noul question uses the option pair Yes/No.

Launch

DE-2.1 serves on mainline vLLM with no special options:

vllm serve shisa-ai/shisa-de-2.1

Then use the Shisa DE SDK, which applies the readout contract, the adaptive policy and the calibration below:

from shisa_de import DecisionModel

de = DecisionModel.from_endpoint("http://127.0.0.1:8000/v1", model="shisa-ai/shisa-de-2.1", api_key="")
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})

Use shisa-de 0.6.0 or later, which bundles DE-2.1's calibration; earlier versions read DE-2.1 correctly but return raw probabilities.

Requirements

• vLLM attention backend. Pass attention_config={"backend": "TRITON_ATTN"} on Hopper GPUs (H100, H200, H20): vLLM 0.30.0 otherwise runs this base model's 512-wide attention heads through a FlashAttention 4 path that lowers accuracy (measured on DE-2: 48.59 against 52.58 on a Decision Index run for the same weights). Triton and FlashInfer tested on SM90, SM120 did not have any accuracy issues.
• String content for text questions sent through the chat endpoint. The SDK reads text questions through /v1/completions and needs nothing here. If you send text questions through /v1/chat/completions yourself, serve with --chat-template-content-format string: the unmodified Gemma 4 chat template writes a space after the system text when the system message arrives as content parts and none when it arrives as a string, and DE-2.1 was trained and evaluated on the string rendering. Leave the option unset if you send images; with it set, vLLM 0.30.0 rejects image requests.
• Sampling defaults. Read probabilities with temperature 0 or 1 and without top_k or top_p truncation.

Adaptive inference policy

Based on empirical testing, we recommend the following adaptive inference policy (applied by the SDK automatically):

1. Write the user turn twice, separated by \n\nRead the same input again before answering:\n, and read the option codes.
2. If the top probability is below 0.7 and the question has 26 options or fewer, ask again with thinking on, let the model reason for up to 1,024 tokens, close the thought and read the codes after it. Otherwise keep the answer from step 1.

The same weights score:

Decision Index 0.3, skill by area	Single read	Read twice	Full policy
Knowledge & Reasoning	43.6	47.3	52.4
Language Understanding	61.0	61.9	62.5
Retrieval & Classification	62.0	64.8	65.1
Tools & Automation	72.9	76.3	76.8
Arts & Human Taste	36.9	40.9	41.4
Public index	56.47	59.23	60.89

The SDK's two steps are worth 4.4 points on the Decision Index.

There is of course a slight latency penalty, so if a fast or predictable response is required, one or both of these steps can be disabled:

One RTX PRO 6000 (TP=1):

Reading	p50	p95	Avg
Single read	37 ms	260 ms	103 ms
User turn read twice	43 ms	517 ms	163 ms
Read twice, think when unsure	55 ms	6,926 ms	1,197 ms

Two RTX PRO 6000s, tensor-parallel (TP=2):

Reading	p50	p95	Avg
Single read	31 ms	286 ms	91 ms
User turn read twice	44 ms	574 ms	152 ms
Read twice, think when unsure	58 ms	4,973 ms	896 ms

Response times are measured one request at a time with vLLM default settings, on one RTX PRO 6000 (TP=1) or tensor-parallel over two (TP=2), on 700 requests of the Decision Index board's sample mix after 10 warmup requests. A second GPU mostly shortens thinking. A read costs the same as DE-2's; under the full policy DE-2.1's average is higher (DE-2: 886 ms at TP=1, 610 ms at TP=2) because it thinks on more requests, 111 of the 700 against DE-2's 80.

A chat template with an opt-in repeat_input=true argument is provided for callers who serve through a plain chat endpoint. It covers step 1 only, is off by default, repeats the last user turn only, and must not be combined with a client that already repeats the input.

Calibration

Raw DE-2.1 probabilities are overconfident, less so than DE-2's, and by a different amount on each read. A post-hoc temperature is fitted per read and per answer kind by NLL grid search, and the SDK applies the one for the read each answer came from:

Read	T, noul	T, choice	ECE, noul	ECE, choice
Single read	1.98	1.53	0.048 → 0.019	0.057 → 0.037
Read twice	2.11	1.64	0.040 → 0.023	0.062 → 0.024
After thinking	5.23	5.23	0.320 → 0.086 (both kinds)

Fitted on all 19,597 questions of 59 of our evaluation suites, each suite weighted equally, served by vLLM 0.30.0 on one RTX PRO 6000. The single-read and read-twice temperatures were fitted on half of the question groups and measured on the other half (3,612 noul and 6,240 choice questions); the thinking temperature is one value fitted on 1,298 thoughts and measured by 5-fold cross-validation. Accuracy is unchanged, because tempering does not move the argmax.

DE-2's temperatures (2.69 / 1.81 single, 2.76 / 1.89 twice, 6.8 after thinking) do not fit DE-2.1: on the same held-out half they leave ECE at 0.034 / 0.045 single and 0.048 / 0.042 twice, and make read-twice yes/no answers worse than raw (0.040). After thinking, DE-2's 6.8 gives 0.064 on these rows against 0.086 for the cross-validated refit. Probabilities read after thinking remain the least reliable: those answers are right 62% of the time at 94% raw confidence. Refit on your own rows before using these probabilities to drive a threshold.

Training

Setting	Value
Method	Eleven LoRA runs, rank 16, alpha 32, dropout 0.05, language-model attention projections; the mean of their updates is merged
Objective	Cross-entropy over the listed option codes at the answer boundary
Schedule	3 epochs, peak learning rate 1.5e-4, 10% warmup then cosine decay
Batch	Up to 16 examples and 4,096 padded tokens, gradient accumulation 2
Data per epoch	15,373 questions: 7,215 English, 8,158 Japanese (seed 1, first epoch)
Option order	Every question is re-rendered each epoch in source, reversed or shuffled order

The 31 training families, by group:

• Distilled and procedural decision data in English and Japanese.
• Japanese judgement families: leakage of internal information, nuance, policy, register, and indirect refusals in context.
• Fraud and safety: antiscam messages with link and sender cues, prompt injection and hidden text.
• Agent workflow: compaction, guardrails, review triage and progress checks.
• Tool use: whether a listed tool fits the request, and when to decline, ask or call (new in DE-2.1).
• Wide-option and ordinal tasks: large-option financial intents, relation and sentiment, product, emotion and sentiment ratings.
• Exactness and logic: sibling criteria where one detail decides the answer, compound conditions, counting and arithmetic probes, none-of-the-above options and evidence mentions (the last two new in DE-2.1).
• A second classifier-benchmark suite of 49 short tasks.

Evaluation

Unless a table says otherwise, each system is shown at its best measured reading: DE-2.1 and DE-2 through the SDK's reading policy, Jev as the hosted jev-1.13.0 API, and OpenAI Decisions (gpt-6-luna) via the latest OpenAI SDK. All DE-2.1 and DE-2 figures are our own local runs; the Jev and OpenAI Decisions figures are the runs reported on the DE-2 card, on the same items. A dash means the system was not run.

Summary

Benchmark	Measure	DE-2.1	DE-2	Jev 1.13.0	OpenAI Decisions
Decision Index 0.3	Public index	60.89	58.89	57.96	47.32
Plan-RewardBench, text only	Macro accuracy	75.5%	72.8%	75.1%	69.4%

• Decision Index 0.3. Full public suite, 140,178 scored requests, official scorer; every request is answered. These are not board entries. The board ranks by a Full score that adds private tests only its maintainers run; Jev's public-index figure is from the board on 2026-10-06. Other public-index figures there: Clef 61.71, Rune 26B-A4B v3 58.28, Decider chat on Gemma-4-31B 57.79.
• Plan-RewardBench. 1,171 pairs of tool-agent trajectories, both presentation orders averaged, following the public JEV-27B-DEMO script.

Decision Index 0.3 by area

Decision Index 0.3, skill by area	DE-2.1	DE-2
Knowledge & Reasoning	52.4	48.6
Language Understanding	62.5	62.9
Retrieval & Classification	65.1	64.2
Tools & Automation	76.8	72.3
Arts & Human Taste	41.4	40.1
Public index	60.89	58.89

JevBench

The 231 public JevBench items, by tier:

Tier	DE-2.1	DE-2	Jev 1.13.0	OpenAI Decisions
Easy (48)	48	48	48	48
Medium (72)	72	72	71	70
Hard (111)	91	86	81	94

"Medium" is JevBench's standard tier. DE-2.1 and DE-2 are read through the full policy with DE-2's scaffold; Jev and OpenAI Decisions are their hosted APIs on the same items, one request per item. The easy and medium tiers are saturated. On the hard tier, by reading:

Hard tier (111), reading	DE-2.1	DE-2
Single read	82	86
Read twice	89	84
Full policy	91	86

Harness suites

66 internal suites, run against both models on the same questions under the three reads, questions of a case asked together through the SDK. Mean of each suite's primary metric:

Read	DE-2.1	DE-2
Single read	89.3	89.2
Read twice	89.9	89.8
Full policy	90.0	89.7

Single read through the harness, the same backend and questions as the DE-2 card's runs; primary metric per suite, a selection:

Suite	Metric	DE-2.1	DE-2	Jev 1.13.0	OpenAI Decisions
Hidden text v1	accuracy	0.901	0.930	0.556	0.556
Antiscam v5	accuracy	0.854	0.844	0.734	0.740
Counting probe	accuracy	0.967	0.967	0.867	0.793
Fresh-math probe	accuracy	0.950	0.950	0.717	0.375
Emotion, 6 classes	accuracy	0.715	0.670	0.465	0.440
Japanese indirect refusals	AUROC	0.926	0.919	0.844	0.806
Japanese leakage v3	AUROC	0.986	0.983	0.951	0.955
Japanese nuance	AUROC	0.974	0.965	0.921	0.898
Kev transfer v9	accuracy	0.778	0.766	0.824	0.734
Antiscam, original	accuracy	0.958	0.949	0.924	0.848
Slop	AUROC	0.967	0.966	0.970	0.888
Guardrails	AUROC	0.987	0.979	0.975	0.956
Compaction	AUROC	0.866	0.850	0.923	0.886
Agent progress v2	AUROC	0.983	0.924	0.992	0.971
Agent routing, 16 cases	accuracy	0.812	0.812	0.938	0.812

On the 63 suites all four systems answered, DE-2.1 has the top result on 35, Jev on 28, DE-2 on 19 and OpenAI Decisions on 7, counting a tie for each system that shares it.

Vision

DE-2.1's vision encoder is the base model's and was not trained, as in DE-2, so on images it is the base model's vision with DE-2.1's readout.

Image classification, glance suites (one forward pass per question):

Suite	Measure	DE-2.1	DE-2	Qwen3-VL-4B
POPE, object presence	Accuracy	0.855	0.865	0.880
POPE, object presence	AUROC	0.951	0.939	0.955
GQA, compositional yes/no	Accuracy	0.675	0.690	0.732
Oxford-IIIT Pet, 37 breeds	Accuracy	0.90	0.92	0.892
Caltech-101, 26-option form	Accuracy	0.96	0.96	0.977

DE-2.1 and DE-2 were run on the same machine and glance checkout (200 / 200 / 100 / 50 test items); the differences between them are a few items each. Jev 1.13.0 takes no image input; the Qwen3-VL-4B figures are its authors' published numbers.

Multimodal judging, following the public JEV-27B-DEMO protocols with both presentation orders averaged:

Benchmark	Measure	DE-2.1	DE-2	OpenAI Decisions	JEV-27B-VL
VL-RewardBench, 1,247 pairs	Overall accuracy	81.2%	80.0%	80.0%	78.3%
Multimodal RewardBench 2, 4 × 1,000 pairs	Average accuracy	61.4%	61.8%	64.3%	63.8%
AgentRewardBench, 1,302 trajectories with screenshots	AUROC	0.911	0.910	0.859	0.91
AgentRewardBench	Precision / recall at 0.5	73.2 / 66.0%	75.1 / 66.9%	82.0 / 58.7%	78.4 / 70.2%

OpenAI Decisions (gpt-6-luna) was run by us on the same items; the JEV-27B-VL figures are its authors' published numbers.

Limitations

• Not every eval improved over DE-2. Notably When2Call tool-call (88.3% vs 91.0% read once) and hidden-text v2 read twice (87.1% vs 91.4%).
• Overconfident raw probabilities. The SDK's calibration (shisa-de 0.6.0 and later) corrects most of this for the single and read-twice answers; DE-2's temperatures over-correct DE-2.1. Probabilities read after thinking stay the least reliable, and calibration was not fitted for image questions.
• Forecasting questions. On ForecastBench (retired from the index in edition 0.3) Brier skill is 24.6 read once, 25.3 read twice and 20.4 under the full policy (DE-2: 8.0, 0 and 0 on the same runner); thinking lowers it. Use a read without thinking when the probability is what you need.
• Rare-positive detection. When very few candidates are true, as in aspect extraction with dozens of pairs per input, the model raises many false alarms: precision is about 10% on ACOS (9.6% read once, 12.2% under the policy).
• Order sensitivity on multi-image comparisons. Swapping two candidate responses changes the verdict on 40% of Multimodal RewardBench 2 pairs. Ask both orders and average.
• Knowledge limits. Under the policy HLE (10.8 against DE-2's 11.8) and ChessBench (21.2 against 20.6) barely move; GPQA Diamond rises from 44.9 to 51.0.
• Thinking is limited to 26 options. Wider questions use the read-twice answer.
• Prompts longer than 65,536 tokens were not evaluated separately.