Ahmad Chamma

AC-723

2 7 1

AI & ML interests

None yet

Recent Activity

liked a model 5 days ago

IFM/Jais-ASR

upvoted an article 5 days ago

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes

published an article 5 days ago

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes

View all activity

Organizations

liked a model 5 days ago

IFM/Jais-ASR

Automatic Speech Recognition • 6B • Updated 7 days ago • 727 • 13

upvoted an article 5 days ago

Article

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes

AC-723

•

5 days ago

• 2

published an article 5 days ago

Article

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes

AC-723

•

5 days ago

• 2

posted an update 5 days ago

Post

2,837

MixtureKit 1.1 is out 🚀 Turn the fine-tuned models you already have on the Hub into one Mixture-of-Experts model.

pip install mixturekit

Give it a base model and its fine-tunes (math, code, a language...). MixtureKit merges them, trains the routers that pick an expert for every token, and shows you what they learned:

🧩 3 composition methods behind one YAML: Branch-Train-MiX, Mixtral-style MoE, Branch-Train-Stitch
💻 Merges stream the experts tensor by tensor: they run on a laptop
🎯 Router training with a load balancing loss, optional LoRA
🔍 A routing visualizer: in the image, two Nile 4B experts split a mixed Egyptian sentence, Arabic script to one expert, Arabizi to the other
📦 Export to Mixtral / Qwen3-MoE, then GGUF for llama.cpp
🤗 mixturekit push uploads the result with a generated model card
✅ Llama 3, Qwen 2.5/3/3.5, Gemma 3/4 experts; tested on transformers 4.52 to 5.19

In the paper, a BTX model built with MixtureKit outperforms the dense baselines on multiple benchmarks.

▶️ Build, train and visualize an MoE in 15 min on a free Colab GPU: https://colab.research.google.com/github/MBZUAI-Paris/MixtureKit/blob/main/examples/build_your_own_moe.ipynb
📄 Paper: MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models (2512.12121)
💻 Code: https://github.com/MBZUAI-Paris/MixtureKit
📚 Docs: https://mbzuai-paris.github.io/MixtureKit/

Work with @guokan-shang and @oe2015 at @MBZUAI-Paris 🙌

authored 2 papers 5 days ago

MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models

Paper • 2512.12121 • Published Dec 13, 2025 • 11

Jais 2: A Family of Arabic-Centric Open Large Language Models

Paper • 2608.13580 • Published Jul 7 • 1

published 14 models 5 days ago

Ahmad Chamma

AI & ML interests

Recent Activity

Organizations

AC-723's activity

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes

Build your own Mixture-of-Experts from two fine-tunes in 15 minutes