polydec-base

polydec-base is the foundation model of ThePolysight.

It is a decision model, not a text generator. You give it a piece of text (a message, a document, a JSON record or a conversation) and one or more typed questions, and it answers all of them in a single call, with calibrated probabilities and a recommendation on whether to trust the answer.

What it does

Question type	Answers	Example
choice	one option from a list, with a probability for each	intent, department, topic
score	a level on an ordinal scale (expected value and most likely level)	urgency 0–2, quality 0–4
noul	a calibrated probability that a statement is true	spam, toxicity, "will this sale close?"

For every answer it also returns a decision: act (the answer is reliable enough to automate) or escalate (a person should review it). The questions are plain text, so the same model answers questions it has never seen, without retraining.

Simple and portable

• One small folder: weights (model.safetensors, 1.6 GB), configuration, encoder configuration and tokenizer. Nothing else is downloaded at load time, so it works offline.
• One call, all answers: the questions are scored in a single forward pass, with no token-by-token generation. About 30 ms per call on a laptop GPU (RTX 4060).
• Runs anywhere PyTorch runs: it picks CUDA, Apple MPS or CPU automatically.
• Same interface for every product model: fine-tuned models load and answer exactly like this one.

Architecture

Encoder	answerdotai/ModernBERT-large (394.8 M parameters)
Heads	2-layer question head, option scorer and act/escalate head (26.5 M parameters)
Total	421.3 M parameters
Input	up to 512 tokens (text + question + options)
Calibration	temperature scaling per question type and number of options

Each option is preceded by a [MASK] token; the model scores the hidden state of each [MASK] and turns the scores into calibrated probabilities.

Training

• Data: ~224,000 typed questions built from 10 public datasets: AG News, BoolQ, Banking77, CLINC150, MultiNLI, VitaminC, Civil Comments, Jailbreak Classification, HelpSteer2 and SaaS Sales Conversations. Instructions are paraphrased and long label lists are shortened with distractors, so the model learns to follow the question rather than a fixed label set.
• Objective: strictly proper scoring rules (logarithmic, spherical and, for ordinal scales, RPS), so the predicted probabilities match observed frequencies. The act/escalate head is trained with a reward of +1 for a correct automated answer, −3 for a wrong one and −0.5 for escalating, so it automates only when it is likely to be right.
• Run: 2 epochs, 7.9 hours on a single 8 GB GPU.

Results

Measured on 11,569 validation questions and on 2,116 questions from three zero-shot tasks never seen in training (emotion, SST-5 sentiment, prompt injections).

	Accuracy	NLL	ECE (calibration error)
Validation	81.2%	0.457	0.015
Zero-shot	48.8%	1.298	0.099

Decision quality (validation)
Accuracy when it decides to act	89.5%
Share of answers escalated to a person	25%
Accuracy of the 50% most confident answers	96.1%

Validation task	Accuracy
Banking77 (intent, 20 options)	99.1%
Jailbreak detection	99.2%
CLINC150 (intent)	97.5%
AG News (topic)	93.2%
Civil Comments (toxicity)	89.7%
MultiNLI (inference)	88.9%
VitaminC (fact checking)	86.1%
BoolQ (reading comprehension)	85.7%
SaaS sales (conversation outcome)	62.9%
HelpSteer2 (response quality, 0–4)	56.8%

Its strength is calibration: when it says 90%, it is right about 90% of the time, and the act/escalate decision uses that to automate the easy cases and route the hard ones to a person.

Limitations

• English only. Accuracy in other languages has not been measured and is expected to be lower.
• General-purpose. For a specific domain, fine-tune it; that is how our product models are built.
• Unseen questions are less reliable than trained ones. Check them on a few known examples and rely on the decision field.
• Subjective scales (response quality, conversation outcomes) remain hard: 57–63% accuracy.

License

polydec-base is licensed under the GNU General Public License v3.0 (see LICENSE and NOTICE). It is a derivative of ModernBERT-large, licensed under the Apache License 2.0 (see LICENSE-APACHE-2.0). The training datasets keep their own licenses.

Contact: adrian.b@thepolysight.com