gemma3-270m-typed-decisions

Gemma 3 270M fully fine-tuned as a probabilistic decision model for LocalLLaMA/typed-decisions. Mode: fine-tuned on train (all four workflows, 1080 cases for training and 120 held out for epoch selection on validation KL).

Results

Scored on LocalLLaMA/typed-decisions all/test (400 cases, 2000 decisions), never seen in training.

Model	Kind	Accuracy ↑	KL from gold ↓	Brier ↓	ECE ↓	Latency
gemma3-270m-typed-decisions (Gemma 3 270M, full fine-tune)	fine-tuned on train	0.761	0.1319	0.0716	0.1480	68 ms / case (cuda)

ECE is computed per question and averaged over the 20 questions (closest to the README's reference rows); pooled over all decisions it is 0.1146.

By question type

	accuracy	kl	brier	ece
choice	0.732	0.167	0.085	0.112
noul	0.822	0.089	0.062	0.066
score	0.739	0.138	0.068	0.158
OVERALL	0.761	0.132	0.072	0.115

By workflow

	accuracy	kl	brier	ece
agent_trace_observability	0.740	0.095	0.057	0.163
customer_service	0.772	0.134	0.070	0.114
invoice_processing	0.784	0.217	0.115	0.087
security_incidents	0.750	0.081	0.045	0.151
OVERALL	0.761	0.132	0.072	0.115

Baselines on the same split: Prior accuracy 0.478 / KL 0.327; zero-shot Gemma 3 270M accuracy 0.279 / KL 1.068.

Full test distributions: test_predictions.jsonl; metrics: test_metrics.json. Gold is the mean of three teacher samples, so scores measure agreement with that teacher (self-agreement ≈ 0.735).

How it works

Each (case, question) pair is one prompt: the case JSON, the question's instructions, and its options listed as letters A, B, C, … with their descriptions. The model's next-token logits at the reply position, restricted to those letter tokens and softmaxed, are the answer distribution. Training minimises soft cross-entropy against the gold distribution. One forward pass per question (5 per case).

Usage

import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "rony/gemma3-270m-typed-decisions"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, attn_implementation="eager").eval()
LETTERS = "ABCDEFGHIJ"
SYSTEM = 'You are a precise decision model. Read the case, then answer the question with the single letter of the best option.'

def options(q):
    if q["type"] == "choice": return list(q["criteria"].items())
    if q["type"] == "noul":
        c = q.get("criteria") or {}
        return [("true", c.get("true", "The statement is true.")), ("false", c.get("false", "The statement is false."))]
    return [(str(i), d) for i, d in enumerate(q["criteria"])]

@torch.no_grad()
def decide(state, questions):
    out = {}
    for name, q in questions.items():
        opts = options(q)
        kind = {"choice": "Choose one option.", "noul": "Is the statement true or false?",
                "score": "Pick the level on the rubric."}[q["type"]]
        prompt = (f"{SYSTEM}\n\n### Case\n{json.dumps(state, separators=(',', ':'))}\n\n"
                  f"### Question ({name})\n{q['instructions']}\n{kind}\n\n### Options\n"
                  + "\n".join(f"{LETTERS[i]}. {l}: {d}" for i, (l, d) in enumerate(opts))
                  + "\n\nAnswer with one letter.")
        text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
        logits = model(**tok(text, add_special_tokens=False, return_tensors="pt")).logits[0, -1]
        ids = [tok.encode(LETTERS[i], add_special_tokens=False)[0] for i in range(len(opts))]
        p = torch.softmax(logits[ids].float(), -1).tolist()
        out[name] = dict(zip([l for l, _ in opts], p))
    return out