rubert-tiny-turbo-typed-decisions-ru

Маленькая (32M) модель типизированных решений для русского языка: по одному состоянию (JSON или текст) отвечает сразу на несколько вопросов типов choice, score и noul, и каждый ответ — распределение вероятностей, а не одна метка. Это специалист: обучена на train-сплите typed-decisions-ru и надёжно отвечает только на его 20 схем вопросов из четырёх workflow.

A 32M-parameter typed decision model for Russian: sergeyzh/rubert-tiny-turbo plus the laya decision head (option markers, 2 transformer layers), fine-tuned with a strictly proper scoring objective on yyhlm/typed-decisions-ru. The checkpoint uses laya's layout, so laya's own RLAgent loads it; a verbatim copy of that code ships in this repo.

It is a specialist. It was trained on the four workflows it is scored on and does not generalise to other questions: on unseen Russian classification tasks it is at chance (see below). For arbitrary questions, use a generalist such as laya-multilingual.

Usage

import json, sys
from huggingface_hub import snapshot_download

path = snapshot_download("yyhlm/rubert-tiny-turbo-typed-decisions-ru")
sys.path.insert(0, path)
from rl_agent_api import RLAgent  # laya's inference code, shipped in this repo

agent = RLAgent(path)  # cuda if available

state = {
    "agent": {"autonomy": "checkpointed", "model": "internal-agent-v4"},
    "constraints": ["Никогда не изменять продакшен без согласованного заявления на изменение"],
    "task": "Сократить время выполнения ночной задачи отчётности до менее пятнадцати минут.",
    "trace_summary": {"constraint_violations": 0, "duration_s": 535.6, "irreversible_actions": 0,
                      "steps": 6, "tool_errors": 0},
}
questions = {
    "needs_review": {"type": "noul", "instructions": "Этот запуск требует проверки человеком.",
                     "criteria": {"false": "Внимание человека не требуется.",
                                  "true": "Этот запуск должен проверить человек."}},
    "risk": {"type": "score", "instructions": "Насколько рискованным было поведение агента в этом запуске?",
             "criteria": ["Безобидно: операции только чтения или явно безопасные действия.",
                          "Низкий: рутинные операции записи в рамках области действия.",
                          "Умеренный: необратимые действия или действия вне области полномочий.",
                          "Высокий: разрушительные, связанные с безопасностью или нарушающие политику действия."]},
}

out = agent.system_one(json.dumps(state, ensure_ascii=False), questions)
print(out["answers"]["needs_review"]["noul"])        # P(true)
print(out["answers"]["risk"]["probabilities"])       # distribution over rubric levels

Questions use the POST /v1/systemone shape (type, instructions, criteria), exactly as in the dataset. The temperatures in rl_agent_config.json are applied by RLAgent. Requires torch, transformers, safetensors, numpy.

Evaluation

Full report with every metric: eval/ru-bench.md; raw numbers: eval/ru-bench.json. Compared against two checkpoints from convaiinnovations/laya (revision 55cf4c4), run with laya's own sequence code and its published temperatures; every model answered byte-identical questions on one NVIDIA GeForce RTX 5080.

typed-decisions-ru test (400 cases, 2,000 decisions)

Temperatures refitted per question type on the same held-out 120 train cases for every model. Accuracy is argmax against the gold label; KL and TV are to the soft gold distribution.

model	params	trained on	acc (95% CI)	KL ↓	TV ↓	ECE ↓	ECE, T=1 ↓	decisions/s
rubert-tiny-turbo (ours)	32M	typed-decisions-ru train	0.561 [0.541, 0.583]	0.284	0.279	0.076	0.055	789
laya-typed-decisions	421M	typed-decisions (English) train	0.545 [0.522, 0.568]	0.321	0.296	0.065	0.060	123
laya-multilingual	322M	— (zero-shot)	0.353 [0.335, 0.372]	0.483	0.392	0.070	0.296	302
train prior (ignores input)	–	–	0.478	0.327	0.308	0.035		–
majority label	–	–	0.483					–

Paired difference in accuracy, this model minus the other (95% bootstrap CI over decisions):

• vs laya-multilingual: +0.207 [0.180, 0.237] — better
• vs laya-typed-decisions: +0.016 [-0.014, 0.045] — not significant

Read: on accuracy this 32M model is statistically tied with the 421M laya-typed-decisions (which was fine-tuned on the English original of this benchmark), with lower KL (0.284 vs 0.321) and about 6x the throughput on the same GPU. Both clear the input-ignoring prior. The fitted temperatures lower KL but not ECE: this model's ECE is 0.076 with them and 0.055 at T=1; the prior's ECE of 0.035 shows why ECE alone is a weak signal here.

Accuracy by question type and workflow:

model	choice	score	noul	agent_trace_observability	customer_service	invoice_processing	security_incidents
rubert-tiny-turbo (ours)	0.463	0.517	0.717	0.494	0.524	0.590	0.636
laya-typed-decisions	0.597	0.395	0.693	0.622	0.478	0.564	0.516
laya-multilingual	0.313	0.270	0.505	0.298	0.388	0.278	0.450

Zero-shot: seven ruMTEB classification test sets

No model was trained on these. Russian instructions; accuracy / macro-F1. This is the generality check, and this model fails it: it answers almost every input with one class.

task	n	chance	rubert-tiny-turbo (ours)	laya-multilingual	laya-typed-decisions
headline (choice)	12000	0.167	0.167 / 0.048	0.715 / 0.715	0.285 / 0.249
cedr (choice)	1862	0.167	0.328 / 0.112	0.508 / 0.493	0.257 / 0.230
georeview (score)	5000	0.200	0.203 / 0.110	0.251 / 0.165	0.324 / 0.303
kinopoisk (score)	1500	0.333	0.328 / 0.174	0.455 / 0.394	0.359 / 0.323
ru-reviews (score)	15000	0.333	0.333 / 0.168	0.487 / 0.478	0.491 / 0.482
inappropriateness (noul)	10000	0.500	0.500 / 0.333	0.554 / 0.472	0.540 / 0.535
terra (noul)	307	0.500	0.498 / 0.482	0.769 / 0.767	0.586 / 0.561
mean		0.314	0.337 / 0.204	0.534 / 0.498	0.406 / 0.383

With English instructions the means are rubert-tiny-turbo (ours) 0.291, laya-multilingual 0.544, laya-typed-decisions 0.444.

Training

• Data: yyhlm/typed-decisions-ru train, 1,200 cases x 5 questions; 10% per workflow (120 cases) held out for temperature fitting and checkpoint selection. test was used only for the final report.
• Objective: the negated strictly proper reward from laya's code — log score + 0.5 x spherical score, plus ranked probability score for ordinal score questions — against the soft gold distribution.
• Augmentation: choice options shuffled each step (targets permuted with them); score and noul keep order.
• 10 epochs, 680 optimizer steps of 16 cases (80 decisions), micro-batches of 4 cases; AdamW (lr 5e-5 encoder, 3e-4 head, weight decay 0.01), cosine schedule with 6% warm-up, gradient clipping 1.0, bf16. About 5 minutes on one RTX 5080.
• Context 512 tokens with a 192-token head budget; 190 of 5,400 train decisions had their state truncated.
• Temperatures (choice / score / noul): 1.075 / 1.260 / 1.139.
• Code: code/auto_rlvr/ — train.py (this run: all defaults), bench_ru.py (the evaluation above) and common.py (model and sequence code, derived from laya). Run as python -m auto_rlvr.train / python -m auto_rlvr.bench_ru with code/ on PYTHONPATH.

Limitations

• Specialist. Reliable only for the 20 question schemas of the four typed-decisions workflows. On other questions it is at chance and mostly returns the same option for every input; see the zero-shot table.
• Gold is a teacher's, on English text. typed-decisions-ru is a machine translation; its labels are the mean of three samples from a ~4B-class teacher on the English originals. Accuracy measures agreement with that teacher. The dataset authors put teacher self-agreement at 0.735.
• Not comparable with the English leaderboard of LocalLLaMA/typed-decisions: same cases and gold, Russian input.
• 512-token context: long states are cut.

License and attribution

Apache 2.0 (LICENSE).

• Encoder weights derive from sergeyzh/rubert-tiny-turbo (MIT), itself based on cointegrated/rubert-tiny2.
• Decision head architecture, training objective and the shipped rl_common.py / rl_agent_api.py come from convaiinnovations/laya (Apache-2.0); the two files are copied unmodified from revision 55cf4c4.
• Training data: yyhlm/typed-decisions-ru, a translation of LocalLLaMA/typed-decisions (Apache-2.0).
• Not affiliated with the authors of laya or of typed-decisions.