Abstract
Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third item k is a witness judged through its legs (a, k) and (k, b). It reads witnesses at zero temperature, keeping the best and the number tied with it, passes on only bits comparing such counts, rebuilds its pair state from the input and these bits at every loop, and is trained from answers alone. Each choice removes one way soft attention fails at size: dispersed weight, unresolved gaps between learned scores, and a drifting soft state. In maximum retrieval it is right on at least 99.95% of sets 1,000 times larger than in training, on all ten seeds; trained with a soft count, the same network falls to 42% at 4,096 items. On CLRS-30 without hints, a general form of the layer (half its channels read at zero temperature, numbers embedded, state carried) beats the best published model, which uses hints on some tasks; the full cold layer is exact on every test input of 9 of the 12 tasks whose answers depend on numbers only through comparisons. From yes-or-no answers it decides 2-SAT and two-colouring exactly at 500 variables and, as our theory predicts, not parity equations. Beside a frozen 7B language model it chains family relations exactly at 1,000 statements after training on 16, where the model alone falls to 44%; finite checks on the weights prove every block that learned correct at every size, on acyclic contexts and given what the model reads. On CLUTRR it is right on every story of five to ten relations given the graphs, learning the 62 rules exactly; from text, with the model frozen, no hand-written rule and only a small read-in trained, it lifts an open 7B model from 33% to 65%, above every system we found without hand-written rules except frontier reasoning models.
Community
A higher-order attention layer whose state lives on pairs of items. Each pair (a, b) judges every third item k through its two legs (a, k) and (k, b), reads these witnesses at zero temperature (the best witness and the number tied with it), passes on only comparison bits, and rebuilds its pair state at every loop. It is trained from answers alone, without hints.
- CLRS-30: one block shared by all 30 algorithms averages 97.20% at 64 items, above the best published model even with hints (96.64%).
- Maximum retrieval: at least 99.95% on every one of ten seeds at 16,384 items, 1,000 times the training size.
- 2-SAT and two-colouring: exact at 500 variables, learned from yes-or-no answers.
- Beside a frozen 7B language model: chains family relations exactly at 1,000 statements after training on 16, and every trained block that learned is proven correct at every size (on acyclic contexts, given what the model reads).
Try all 30 algorithms in your browser: https://neovand.github.io/cold-attention
Code: https://github.com/NeoVand/cold-attention
Models: https://huggingface.co/mohsenvand
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- A Tool in the Forward Pass: Frozen Tape Modules as Layers of a Small Language Model (2026)
- LEV: Thinking Fast and Deep with Looped Decision Models (2026)
- Everything in its right place: attention is a radio head (2026)
- When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning (2026)
- P(Yes): 15$\times$ Improvements via Test-Time Scaling of Decision Models (2026)
- Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters (2026)
- Consider, Then Choose: Order-Invariant Decisions in System One Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.11937 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 10
mohsenvand/witness-clrs
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper