MongoDB/AptMQL-Bench
Viewer • Updated • 3.19k • 511 • 2
Robin Vujanic
rvo
AI & ML interests
None yet
Recent Activity
updated a dataset about 18 hours ago
MongoDB/AptMQL-Bench liked a dataset 2 days ago
MongoDB/AptMQL-Bench published a dataset 10 days ago
MongoDB/AptMQL-BenchOrganizations
updated a
dataset about 18 hours ago
liked a
dataset 2 days ago
published a
dataset 10 days ago
upvoted a collection 3 months ago
updated 2
models 3 months ago
reacted to sergiopaniego's post with 🔥 6 months ago
Post
1,455
Earlier this month, Apple introduced Simple Self-Distillation: a fine-tuning method that improves models on coding tasks just by sampling from the model and training on its own outputs with plain cross-entropy
And… it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team 🐎
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple 🍎
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train × T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
And… it's already supported in TRL, built by Kashif Rasul. you can really feel the pace of development in the team 🐎
Paper by Ruixiang ZHANG, He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang at Apple 🍎
How it works: the model generates completions at a training-time temperature (T_train) with top_k/top_p truncation, then fine-tunes on them with plain cross-entropy. no labels or verifier needed
You can try it right away with this ready-to-run example (Qwen3-4B on rStar-Coder):
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd.py
or benchmark a checkpoint with the eval script:
https://github.com/huggingface/trl/blob/main/trl/experimental/ssd/ssd_eval.py
One neat insight from the paper: T_train and T_eval compose into an effective T_eff = T_train × T_eval, so a broad band of configs works well. even very noisy samples still help
Want to dig deeper?
Paper: Embarrassingly Simple Self-Distillation Improves Code Generation (2604.01193)
Trainer docs: https://huggingface.co/docs/trl/main/en/ssd_trainer
liked a
model 6 months ago
upvoted an article 6 months ago
Article
Training and Finetuning Embedding Models with Sentence Transformers
tomaarsen
• • 283 liked a
model 10 months ago
What is the max token size it can support?
2
#3 opened 11 months ago
by 
AayushShah
upvoted a collection about 1 year ago
Upload folder using huggingface_hub
#2 opened about 1 year ago
by 
Xenova
Upload folder using huggingface_hub
#2 opened about 1 year ago
by Xenova
Upload optimized ONNX model
#1 opened about 1 year ago
by Xenova
Upload optimized ONNX model
#1 opened about 1 year ago
by Xenova
updated 2
models about 1 year ago
liked 2
models about 1 year ago