Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Paper • 2605.22791 • Published • 26
Alexandros Liapatis
alexliap
AI & ML interests
Generative AI + Traditional ML
Recent Activity
new activity 2 days ago
alexliap/greek-synth-v1:[bot] Conversion to Parquet upvoted a paper 3 days ago
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention updated a dataset 3 days ago
alexliap/greek-synth-v1Organizations
None yet
[bot] Conversion to Parquet
#1 opened 2 days ago
by 
parquet-converter
upvoted a paper 3 days ago
updated a
dataset 3 days ago
published a
dataset 3 days ago
liked a
dataset 4 days ago
reacted to Crownelius's post with 🔥 4 days ago
Post
4519
Howdy,
CompactAI-O is launching a tiny Model Golf, and the winner walks away with $50 in RunPod credits. Monthly. Every month. Show up, build, somebody wins.
What it is
Build the best language model you can under 100 million parameters, with at least a 1028-token context window. That's it. Any architecture, any tokenizer, any training scheme you can dream up at 3am. The only catch is it's gotta be open source (MIT, GPL, Apache, AGPL) take your pick.
It scratches the same itch as a Kaggle comp without the dataset\leaderboard nonsense. No fixed benchmark to game. No llama.cpp compatibility hoops. If you wanna train a 50M-param MoE with five experts and a tokenizer built on cookbooks, you can do that. Nothing stopping you.
The rules are listed in the discord and on the organization page if you're interested.
Why $50????
It's symbolic. It ain't gonna make anyone rich. But it's enough to cover a weekend of GPU time, enough to keep enthusiasts coming back, and not so much that it pulls in people who are just there for the money. Enthusiasts build interesting things. Interesting things move the field forward. A little incentive. I'd do it for $50 lol.
How to join
First round opens soon. Landing page is here:
→ CompactAI-O/Tiny-model-golf
For questions or to swap ideas, the Discord's open:
→ https://discord.gg/y2jTct6Cxv
Excited to see what yall come up with. ♥
— Shane
CompactAI-O is launching a tiny Model Golf, and the winner walks away with $50 in RunPod credits. Monthly. Every month. Show up, build, somebody wins.
What it is
Build the best language model you can under 100 million parameters, with at least a 1028-token context window. That's it. Any architecture, any tokenizer, any training scheme you can dream up at 3am. The only catch is it's gotta be open source (MIT, GPL, Apache, AGPL) take your pick.
It scratches the same itch as a Kaggle comp without the dataset\leaderboard nonsense. No fixed benchmark to game. No llama.cpp compatibility hoops. If you wanna train a 50M-param MoE with five experts and a tokenizer built on cookbooks, you can do that. Nothing stopping you.
The rules are listed in the discord and on the organization page if you're interested.
Why $50????
It's symbolic. It ain't gonna make anyone rich. But it's enough to cover a weekend of GPU time, enough to keep enthusiasts coming back, and not so much that it pulls in people who are just there for the money. Enthusiasts build interesting things. Interesting things move the field forward. A little incentive. I'd do it for $50 lol.
How to join
First round opens soon. Landing page is here:
→ CompactAI-O/Tiny-model-golf
For questions or to swap ideas, the Discord's open:
→ https://discord.gg/y2jTct6Cxv
Excited to see what yall come up with. ♥
— Shane
- 8 replies
Dataset sources
2
#1 opened 15 days ago
by alexliap
reacted to HannesVonEssen's post with ❤️ 14 days ago
Post
11595
📣 Hugging Face Visualizer, now as Chrome extension!
https://hfviewer.com
✨ After installing, Hugging Face model pages will have an architecture visualization on the model page itself!
🔗 Link:
https://chromewebstore.google.com/detail/hugging-face-viewer/mmadlggmpkpiockpjfepaohcllbnakej
Thanks for all the nice feedback so far! ❤️
https://hfviewer.com
✨ After installing, Hugging Face model pages will have an architecture visualization on the model page itself!
🔗 Link:
https://chromewebstore.google.com/detail/hugging-face-viewer/mmadlggmpkpiockpjfepaohcllbnakej
Thanks for all the nice feedback so far! ❤️
- 5 replies
liked a
model 15 days ago
reacted to qgallouedec's post with 🔥 15 days ago
Post
10063
Shipped hf-sandbox! 🥡
🧪 Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
👉 https://github.com/huggingface/hf-sandbox
🧪 Running an eval that executes model-generated C on a few thousand prompts? You probably don't want any of that on your laptop.
Just shipped hf-sandbox, a Modal-style sandbox API on top of Hugging Face Jobs. Spin up an isolated, ephemeral container, run untrusted code, get the result back. No Docker on your laptop, no infra to manage.
Just pip install hf-sandbox.
Early days (v0.1); feedback and issues very welcome:
👉 https://github.com/huggingface/hf-sandbox
- 1 reply
liked 2
models about 1 month ago
reacted to sergiopaniego's post with 🔥 about 1 month ago
Post
1402
Earlier this month, Apple introduced Simple Self-Distillation: a fine-tuning method that improves models on coding tasks just by sampling from the model and training on its own outputs with plain cross-entropy
And… it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team 🐎
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple 🍎
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train × T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
And… it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team 🐎
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple 🍎
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train × T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
updated a
dataset 2 months ago
published a
dataset 2 months ago
liked a
model 3 months ago
liked a
Space 3 months ago
liked 2
datasets 3 months ago
updated a
model 3 months ago