Mati83moni/PLLuM-12B-chat-2512-EXL3
Text Generation • 4B • Updated • 365
Mateusz Piesiak PRO
Mati83moni
AI & ML interests
Hybrid LLMs, MoE, Mamba, multilingual NLP, biomedical reasoning, efficient pretraining
Recent Activity
posted an update about 3 hours ago
Is "evaluation awareness" in a language model one direction, or eight?
We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.
What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.
The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.
Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight
Spaces:
https://huggingface.co/spaces/Mati83moni/evaluation-awareness-one-direction-or-eight
https://huggingface.co/spaces/Mati83moni/evaluation-awareness-pllum-12b published an article about 3 hours ago
Evaluation Awareness: One Direction, or Eight? updated a dataset about 4 hours ago
Mati83moni/evalaware-article-figuresOrganizations
None yet