Banaxi PRO
Banaxi-Tech
AI & ML interests
SLMs, training from scratch, LoRA, TTS, Ternary models. AI Interpretability. BCI. Contact at banaxitech@gmail.com
Recent Activity
new activity about 2 hours ago
GODELEV/Rose-Medium:Leaderboard liked a model about 4 hours ago
SupraLabs/Supra2-Nano new activity about 4 hours ago
AxiomicLabs/Open_SLM_Leaderboard:Add Lumen-118M-BaseOrganizations
reacted to their post with 🤗❤️👀🚀🔥 about 22 hours ago
Post
937
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
- 2 replies
posted an update about 22 hours ago
Post
937
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget.
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
BananaMind
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through.
The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting.
Follow us for more:
@vovaRL
@Banaxi-Tech
Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd
And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
- 2 replies
replied to their post about 24 hours ago
Ik
replied to their post about 24 hours ago
bruh
replied to their post 1 day ago
Then ill make availible for download.
replied to their post 1 day ago
5 More followers https://huggingface.co/BananaMind
reacted to Nymbo's post with 🔥 1 day ago
Post
1781
Anthropic gave me six months of Claude Max 20x through the Claude for Open Source program, granted based on my Hugging Face work. Thank you 
Anthropic for supporting open source.
So far I've been pointing it at Markdown Minimap, an Obsidian plugin that adds a scrollable IDE-style minimap to your notes. This week I've been clearing a backlog of user-reported issues on it, with Claude often handling them end to end.
https://github.com/Nymbo/Markdown-Minimap — issues and PRs welcome.
So far I've been pointing it at Markdown Minimap, an Obsidian plugin that adds a scrollable IDE-style minimap to your notes. This week I've been clearing a backlog of user-reported issues on it, with Claude often handling them end to end.
https://github.com/Nymbo/Markdown-Minimap — issues and PRs welcome.
replied to LH-Tech-AI's post 1 day ago
Cant you train Pro and Ultra locally? id say Ultra would take about 14 days on your 2 5060 Tis
and Laguna s2.1?
no its "Qwen3.6-40B-Fable-Fusion-6-Core-Deckard-Eleanor-Heretic-Uncensored-NM-DAU-NEO-MAX-MTP-GGUF"
replied to their post 2 days ago
Yeah we will make them soon! We're currently focusing about our other models but soon we will test them.
replied to LH-Tech-AI's post 2 days ago
yea so i just dropped https://huggingface.co/BananaMind/BananaMind-2-SLMoE so its true
replied to LH-Tech-AI's post 2 days ago
posted an update 2 days ago
Post
1755
We're excited to release BananaMind 2 SLMoE, an experimental sequence-level mixture-of-experts model.
It uses only 8M parameters per message but has 25M total parameters, 13 experts (out of 64) are selected based on the message prefix and reused for the entire response.
We're testing with this sequence-level architecture to find out how big the capability loss actually is and how much of it can be fixed.
The long-term idea is that this could make very large sparse models usable on machines that can't fit them in RAM by putting the entire model (which is big) on disk and only loading the active parts into VRAM.
This architecture is still in research and shouldn't be used for production models.
We trained it on 60B tokens (of FineWeb-HQ, FineWeb-Edu, DCLM ,Cosmopedia v2, FineMath and NPSet-2) on 8 RTX Pro 6000s.
Check it out at BananaMind/BananaMind-2-SLMoE
Follow us for future models:
BananaMind
@vovaRL
@Banaxi-Tech
@DedeProGames
BananaMind 2 Pro in a few days. You've been waiting 22 days for it.
It uses only 8M parameters per message but has 25M total parameters, 13 experts (out of 64) are selected based on the message prefix and reused for the entire response.
We're testing with this sequence-level architecture to find out how big the capability loss actually is and how much of it can be fixed.
The long-term idea is that this could make very large sparse models usable on machines that can't fit them in RAM by putting the entire model (which is big) on disk and only loading the active parts into VRAM.
This architecture is still in research and shouldn't be used for production models.
We trained it on 60B tokens (of FineWeb-HQ, FineWeb-Edu, DCLM ,Cosmopedia v2, FineMath and NPSet-2) on 8 RTX Pro 6000s.
Check it out at BananaMind/BananaMind-2-SLMoE
Follow us for future models:
@vovaRL
@Banaxi-Tech
@DedeProGames
BananaMind 2 Pro in a few days. You've been waiting 22 days for it.
- 3 replies
replied to LH-Tech-AI's post 2 days ago
BananaMind 2.1.