---
license: gpl-3.0
language:
  - en
base_model: answerdotai/ModernBERT-large
pipeline_tag: text-classification
tags:
  - decision-model
  - classification
  - calibration
  - modernbert
  - polydec
datasets:
  - fancyzhx/ag_news
  - google/boolq
  - mteb/banking77
  - clinc/clinc_oos
  - nyu-mll/multi_nli
  - tals/vitaminc
  - google/civil_comments
  - jackhhao/jailbreak-classification
  - nvidia/HelpSteer2
  - DeepMostInnovations/saas-sales-conversations
---

# polydec-base

**polydec-base is the foundation model of ThePolysight.**

It is a **decision model**, not a text generator. You give it a piece of text (a message, a document, a JSON record or
a conversation) and one or more **typed questions**, and it answers all of them in a single call, with calibrated
probabilities and a recommendation on whether to trust the answer.

## What it does

| Question type | Answers | Example |
|---|---|---|
| `choice` | one option from a list, with a probability for each | intent, department, topic |
| `score` | a level on an ordinal scale (expected value and most likely level) | urgency 0–2, quality 0–4 |
| `noul` | a calibrated probability that a statement is true | spam, toxicity, "will this sale close?" |

For every answer it also returns a decision: **`act`** (the answer is reliable enough to automate) or **`escalate`**
(a person should review it). The questions are plain text, so the same model answers questions it has never seen,
without retraining.

## Simple and portable

- **One small folder:** weights (`model.safetensors`, 1.6 GB), configuration, encoder configuration and tokenizer.
  Nothing else is downloaded at load time, so it works offline.
- **One call, all answers:** the questions are scored in a single forward pass, with no token-by-token generation.
  About 30 ms per call on a laptop GPU (RTX 4060).
- **Runs anywhere PyTorch runs:** it picks CUDA, Apple MPS or CPU automatically.
- **Same interface for every product model:** fine-tuned models load and answer exactly like this one.

## Architecture

| | |
|---|---|
| Encoder | `answerdotai/ModernBERT-large` (394.8 M parameters) |
| Heads | 2-layer question head, option scorer and act/escalate head (26.5 M parameters) |
| Total | **421.3 M parameters** |
| Input | up to 512 tokens (text + question + options) |
| Calibration | temperature scaling per question type and number of options |

Each option is preceded by a `[MASK]` token; the model scores the hidden state of each `[MASK]` and turns the scores
into calibrated probabilities.

## Training

- **Data:** ~224,000 typed questions built from 10 public datasets: AG News, BoolQ, Banking77, CLINC150, MultiNLI,
  VitaminC, Civil Comments, Jailbreak Classification, HelpSteer2 and SaaS Sales Conversations. Instructions are
  paraphrased and long label lists are shortened with distractors, so the model learns to follow the question rather
  than a fixed label set.
- **Objective:** strictly proper scoring rules (logarithmic, spherical and, for ordinal scales, RPS), so the predicted
  probabilities match observed frequencies. The act/escalate head is trained with a reward of +1 for a correct
  automated answer, −3 for a wrong one and −0.5 for escalating, so it automates only when it is likely to be right.
- **Run:** 2 epochs, 7.9 hours on a single 8 GB GPU.

## Results

Measured on 11,569 validation questions and on 2,116 questions from three **zero-shot** tasks never seen in training
(emotion, SST-5 sentiment, prompt injections).

| | Accuracy | NLL | ECE (calibration error) |
|---|---|---|---|
| Validation | **81.2%** | 0.457 | **0.015** |
| Zero-shot | 48.8% | 1.298 | 0.099 |

| Decision quality (validation) | |
|---|---|
| Accuracy when it decides to `act` | **89.5%** |
| Share of answers escalated to a person | 25% |
| Accuracy of the 50% most confident answers | **96.1%** |

| Validation task | Accuracy |
|---|---|
| Banking77 (intent, 20 options) | 99.1% |
| Jailbreak detection | 99.2% |
| CLINC150 (intent) | 97.5% |
| AG News (topic) | 93.2% |
| Civil Comments (toxicity) | 89.7% |
| MultiNLI (inference) | 88.9% |
| VitaminC (fact checking) | 86.1% |
| BoolQ (reading comprehension) | 85.7% |
| SaaS sales (conversation outcome) | 62.9% |
| HelpSteer2 (response quality, 0–4) | 56.8% |

Its strength is **calibration**: when it says 90%, it is right about 90% of the time, and the act/escalate decision
uses that to automate the easy cases and route the hard ones to a person.

## Limitations

- **English only.** Accuracy in other languages has not been measured and is expected to be lower.
- **General-purpose.** For a specific domain, fine-tune it; that is how our product models are built.
- **Unseen questions are less reliable** than trained ones. Check them on a few known examples and rely on the
  `decision` field.
- **Subjective scales** (response quality, conversation outcomes) remain hard: 57–63% accuracy.

## License

polydec-base is licensed under the **GNU General Public License v3.0** (see `LICENSE` and `NOTICE`). It is a
derivative of ModernBERT-large, licensed under the Apache License 2.0 (see `LICENSE-APACHE-2.0`). The training
datasets keep their own licenses.

Contact: adrian.b@thepolysight.com