jonasaise/laya-swedish-preference-gate

Laya Swedish Preference Gate (v4.1)

A ~10 ms judge for Swedish text. One forward pass of a fine-tuned Laya decision model (mmBERT-base backbone) answers one of five questions, as a "System 1" filter in front of a slow LLM judge.

Question	Input	Output
preference	prompt (or English source) + two responses	calibrated P(response A is better)
naturalness	two responses (prompt optional)	P(A is more natural, idiomatic Swedish)
quality	prompt + one response	1–5 score
natural_swedish	one response	P(natural Swedish vs translationese)
grammatical (new)	one text	P(grammatically correct); compare two texts, see limitations

Earlier releases: revision="v4", revision="v3", revision="v2", revision="v1".

Code, the full ablation and every number below: github.com/joenaess/laya-swedish-preference-gate. Training and evaluation pairs: jonasaise/laya-swedish-gate-v4-data.

What changed from v3

v4 and v4.1 add training data chosen in a controlled ablation (same recipe, seed and hyperparameters; only the data differs; all tests held out):

• DaLAJ-GED (Superlim 2): learner sentences vs the same sentence corrected, for naturalness and the new grammatical question.
• SweFAQ (Superlim 2): a question's own answer vs other answers from the same FAQ page, for preference.
• Europarl (Språkbanken): professional Swedish translations vs Kimi-k3 machine translations of English speeches, plus adequacy negatives (another paragraph's translation, a half translation, a word-shuffled translation, the untranslated English).
• v4.1 only: correct text from four registers against corrupted copies (a word deleted or two neighbours swapped), and wrong-source negatives from the adjacent paragraph of the same debate, to reduce grammatical over-flagging and fluent-but-wrong translations winning.

Held-out test (share where the reference wins)	v3	v4	v4.1
Fluent-vs-fluent probe, preference	79.9%	78.6%	79.3%
DPO validation, preference	98.9%	98.7%	98.8%
liu-nlp native vs OPUS-MT, naturalness	44.3%	52.2%	51.0%
FLORES+ human vs OPUS-MT, preference with source	48.5%	73.8%	71.9%
WMT24++ human vs OPUS-MT, preference with source	54.3%	74.6%	72.8%
WMT24++ human vs Kimi-k3, preference with source	43.1%	58.9%	59.8%
DaLAJ corrected vs learner, naturalness	55.9%	89.7%	88.0%
GerLangMod-sv original vs verb misplaced, naturalness	83.3%	95.5%	95.7%
SweFAQ own answer vs other, preference	60.3%	88.9%	88.6%
BLiMPPI-sv polarity items, naturalness	40.7%	86.4%	89.8%
MultiBLiMP-sv agreement, naturalness	83.1%	97.0%	97.5%
ScaLA-sv MCC (EuroEval's metric), single-response	0.146	0.341	0.466

Calibration: preference T = 1.674, naturalness T = 0.485, quality T = 1.226, natural_swedish T = 1.495, grammatical T = 1.427. ECE on the DPO validation pairs 0.0164.

A trap we hit on the way. Human-vs-MT pairs alone teach a gate that "the candidate least like the source wins": the intermediate arms preferred a fluent translation of the wrong sentence, half translations and nonsense. The adequacy negatives fix it (FLORES+ devtest, English source in the prompt, share where the reference wins):

The reference against …	v3	without adequacy negatives	v4	v4.1
the reference of a different sentence	91.0%	0.0%	78.0%	89.5%
LFM2-350M's translation (pilot's lowest chrF)	97.0%	3.5%	98.0%	97.5%
only the first half of the reference	72.0%	7.5%	100.0%	100.0%
the untranslated English	20.5%	100.0%	100.0%	100.0%
the untranslated English (naturalness, no prompt)	87.5%	38.5%	99.5%	99.5%

On 5600 translations by 28 LLMs, v4.1's translation score agrees with chrF at Spearman 0.38 per translation (v3: 0.30).

Pilot: a quick proxy for EuroEval Swedish

Scoring 100 Swedish answers per model with quality and averaging, over 28 open LLMs with Swedish results on the EuroEval leaderboard: Spearman with the Swedish rank 0.66 [0.33, 0.85], 0.63 [0.25, 0.81] after controlling for model size. A pilot (models up to 11B parameters, 6 of them in 4-bit), not a calibrated predictor.

Quickstart

uv pip install "git+https://github.com/joenaess/laya-swedish-preference-gate"

from laya_gate.predictor import SwedishPreferenceGate

gate = SwedishPreferenceGate("jonasaise/laya-swedish-preference-gate")

r = gate.predict_preference(
    prompt="Hur fungerar fotosyntesen?",
    chosen_candidate="Fotosyntesen är den process där växter omvandlar ljus till energi.",
    rejected_candidate="Det är när solen lyser på träd och gör magi.",
)
# {"preference": "candidate_a", "p_chosen": <float>, "is_valid_margin": <bool>, ...}

# translation quality: put the English source in the prompt
t = gate.predict_preference(
    prompt="Översätt följande text till svenska:
The meeting was postponed.",
    chosen_candidate="Mötet sköts upp.",
    rejected_candidate="Mötet ställdes in.",
)

s = gate.score_quality(prompt="Hur fungerar fotosyntesen?", response="Växter gör mat av solljus.")
g = gate.score_grammar("Människor är snälla där och alla varor är billiga.")
a, b = (gate.score_grammar(x)["p_correct"] for x in ("Hon har bott här.", "Hon har bo här."))

Limitations

• grammatical over-flags as a stand-alone verdict. At 0.5 it marks 25.2% of fluent LLM answers and 47.9% of WMT24++ references as incorrect (it was trained on single learner sentences). Compare two versions of a text instead. On lowercased text without punctuation (GerLangMod-sv) it is at chance (86.3%); v4 was at 47.8%.
• Wrong-content translations still win sometimes: v4 prefers a fluent translation of a different sentence in 10.5% of FLORES+ pairs (v3: 9.0%).
• Answer preference largely tracks length. On 129 human Chatbot Arena votes on Swedish answers, preference agrees with people in 58.9% of battles, close to "the longer answer wins" (64.3%); Kimi-k3 reaches 74.8%. Use the gate for Swedish form (grammar, translationese, translation adequacy, relevance), not to pick between two fluent answers.
• Pairwise preference, not fact-checking; 1,024-token window; one seed per training arm.
• Kimi-k3 is not a neutral reference for its own output: on WMT24++ it prefers the human translation over OPUS-MT in 81.0% of pairs but over its own translation in only 24.5%.

Small LLM-as-a-judge comparison (Qwen/Qwen3.5-2B)

Can a small generative LLM (Qwen/Qwen3.5-2B, 1.88B parameters, 24 GatedDeltaNet + full-attention layers) serve the exact same single-pass discriminator role when adapted with the same 5 questions (preference, naturalness, quality, natural_swedish, grammatical), temperature calibration, and v4.1 multi-question training recipe (python -m laya_gate.llm_judge)?

Published on the Hugging Face Hub: jonasaise/qwen3.5-2b-swedish-preference-gate.

QwenPreferenceGate formats each question with its option criteria (A/B or 1..5), applies token-level field truncation to 1,024 tokens, disables <think> reasoning, and projects the final hidden state onto the 7 option vectors in a single forward pass. For fine-tuning on a single 12 GB GPU, all 24 backbone layers, the final RMSNorm, and the 7-option decision head (1.37B trainable parameters) are trained with 8-bit PagedAdamW8bit and gradient checkpointing while freezing the 248k-row tied vocabulary embedding table, on a deterministic stratified 8,000-unit v4.1 sample (400 units per (source, question) group, 800 for fluent/quality), alongside a matched-budget Laya-307M control trained on the exact same 8,000 examples.

Cost and efficiency

Model	Parameters	Trainable	Train units	Train time	Peak VRAM	Pair p50	Single p50	Throughput (bs=8)
Qwen/Qwen3.5-2B (zero-shot)	1882M	0	0	—	—	57.7 ms	33.0 ms	17.1/s
Qwen/Qwen3.5-2B (v4.1 fine-tuned)	1882M	1373M	8,000	16.2 min (121.9s/1k)	7.22 GiB	57.9 ms	32.9 ms	17.2/s
Qwen/Qwen3.5-2B (v4.2b length-controlled)	1882M	1373M	+1,690	+3.3 min	7.22 GiB	57.7 ms	32.9 ms	17.2/s
Laya-307M (v4.1 matched 8k)	307M (322M total)	322M	8,000	6.7 min (50.4s/1k)	6.01 GiB	9.8 ms	6.0 ms	119.9/s
Laya-307M (v4.1 full)	307M (322M total)	322M	223,521	166.0 min (44.6s/1k)	6.01 GiB	9.8 ms	6.0 ms	119.9/s

Benchmark accuracy across all configurations and consensus fusion

Benchmark	Qwen3.5-2B zero-shot	Qwen3.5-2B v4.1 (8k)	Qwen3.5-2B v4.2b (+len)	Laya-307M matched (8k)	Laya-307M v4.1 (224k)	Consensus (Laya ∧ Qwen-FT)
In-domain original/preference	79.1%	93.6%	92.7%	70.9%	98.8%	99.4% (94% agree)
In-domain fluent/preference	75.8%	82.1%	82.4%	62.9%	76.6%	88.8% (76% agree)
Reasoning-on probe/preference	73.9%	82.3%	84.6%	62.9%	79.3%	88.0% (81% agree)
DaLAJ-GED naturalness	73.7%	86.7%	86.6%	60.0%	88.0%	93.9% (85% agree)
DaLAJ-GED grammatical	73.6%	75.0%	74.2%	53.6%	81.4%	84.7% (67% agree)
ScaLA-sv grammatical acc@0.5	not measured	not measured	not measured	not measured	not measured	81.0% (69% agree)
GerLangMod-sv naturalness	94.4%	96.1%	96.0%	64.8%	95.7%	99.2% (93% agree)
BLiMPPI-sv naturalness	69.5%	96.6%	94.9%	15.3%	89.8%	96.4% (93% agree)
MultiBLiMP-sv naturalness	84.6%	99.5%	100.0%	46.3%	97.5%	100.0% (97% agree)
WikiEdits-sv naturalness	51.8%	58.0%	56.3%	49.1%	56.6%	60.3% (71% agree)
SweFAQ preference	72.9%	93.9%	93.2%	77.2%	88.6%	96.9% (88% agree)
SweFAQ quality top-1	24.7%	40.2%	41.1%	16.9%	23.3%	23.7% (69% agree)
Length test (short correct, preference)	8.6%	44.0%	75.9%	16.9%	28.6%	27.3% (61% agree)
FLORES+ preference+source	43.6%	59.5%	58.3%	50.2%	71.9%	73.6% (67% agree)
WMT24++ OPUS-MT preference+source	61.8%	73.4%	70.4%	54.5%	72.8%	82.9% (70% agree)
WMT24++ Kimi-k3 preference+source	27.7%	40.7%	35.6%	47.6%	59.8%	50.5% (59% agree)
Europarl EN→SV Kimi-k3 preference+source	20.5%	45.5%	41.2%	43.6%	75.9%	67.1% (62% agree)
Europarl SV→EN→SV Kimi-k3 preference+source	13.6%	42.3%	38.4%	43.8%	82.8%	74.1% (52% agree)
Chatbot Arena arena_sv (preference)	63.6%	61.2%	62.0%	62.8%	58.9%	64.4% (67% agree)
ScaLA-sv grammatical MCC@0.5	0.307	0.391	0.323	-0.002	0.466	0.623 (69% agree)

Takeaways:

1. Fine-tuning transforms Qwen3.5-2B as a Swedish judge: full-backbone fine-tuning on the 8,000 stratified v4.1 units lifts Qwen3.5-2B on in-domain original/preference (79.1% → 93.6%), fluent/preference (75.8% → 82.1%), DaLAJ-GED (naturalness 73.7% → 86.7%), BLiMPPI-sv (69.5% → 96.6%), SweFAQ (72.9% → 93.9%), and short-correct length-controlled pairs (8.6% → 44.0%), while halving false-positive grammar flagging on clean human references.
2. Sample efficiency vs full-data encoder specialization: at 8,000 training units, the 1.88B causal LM is far more sample-efficient than the 307M bidirectional encoder trained from scratch on the task head. However, because Laya-307M trains 2.7× faster per unit and runs 5.9× faster at inference (9.8 ms vs 57.9 ms per pair, 119.9 vs 17.2 pairs/s), training Laya-307M on the full 223,521-unit v4.1 recipe surpasses fine-tuned Qwen3.5-2B on DaLAJ-GED (88.0% vs 86.7%), ScaLA-sv acceptability (MCC 0.466 vs 0.391), and every human-vs-MT translation benchmark (FLORES+ 71.9% vs 59.5%, WMT24++ Kimi-k3 59.8% vs 40.7%, Europarl EN→SV Kimi-k3 75.9% vs 45.5%), where generative LLMs retain a strong bias toward machine-generated phrasing.
3. Standalone vs cascade vs consensus fusion: because the bidirectional encoder (Laya-307M) and causal LM (Qwen3.5-2B) have complementary inductive biases (linguistic form/anti-MT adequacy vs pretrained instruction helpfulness), on the 69–94% of items where both agree (Laya ∧ Qwen-FT), accuracy is higher than either model's accuracy on all items — a selective-prediction result that abstains on the rest — on fluent/preference (88.8%), probe/preference (88.0%), DaLAJ-GED (93.9%), SweFAQ (96.9%), and ScaLA-sv acceptability (MCC 0.623), while standalone Laya-307M should be used for high-volume corpus filtering and source-grounded translation curation.
4. Length-controlled pairs (v4.2b) remove Qwen3.5-2B's length bias: warm-starting from v4.1 on 800 v4.2b length-controlled counterexample units + 800 replay units lifts length_test/short_correct/preference from 44.0% to 75.9% (overall length_test/preference: not measured).

Training data and licenses

Weights: Apache-2.0. Base model convaiinnovations/laya: apache-2.0. Training data:

• jonasaise/dpo-swedish-interpreted (mit) and argilla/ultrafeedback-binarized-preferences-cleaned (mit).
• Europarl Swedish-English: Koehn (2005), Europarl: A Parallel Corpus for Statistical Machine Translation, MT Summit X; enriched and distributed by Språkbanken, https://doi.org/10.23695/mdp4-2p72, CC BY 4.0.
• DaLAJ-GED (CC BY 4.0) and SweFAQ (CC BY-SA 4.0) from Superlim 2 (sbx/superlim-2), Språkbanken.
• Machine translations by Helsinki-NLP/opus-mt-en-sv (evaluation only) and Kimi-k3.

Evaluation only: liu-nlp/translationese-opensubtitles (Apache-2.0), FLORES+, WMT24++ (Apache-2.0), ScaLA-sv (mteb/scala_sv_classification), GerLangMod (Apache-2.0), BLiMPPI (arXiv:2610.13607; Swedish subset from UD SweLL/Talbanken CC BY-SA 4.0 and LinES CC BY-NC-SA 4.0), MultiBLiMP (jumelet/multiblimp, CC BY 4.0).

@software{laya_swedish_preference_gate,
  title  = {Laya Swedish Preference Gate},
  author = {Lind, Jonas},
  year   = {2026},
  url    = {https://huggingface.co/jonasaise/laya-swedish-preference-gate}
}