BigScience Workshop

non-profit

https://bigscience.huggingface.co

bigscienceW

bigscience-workshop

Activity Feed

AI & ML interests

A one-year long research workshop on large language models: the Summer of Language Models 21 🌸

Recent Activity

nsubramani23 authored a paper 4 days ago

Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models

nsubramani23 authored a paper 4 days ago

LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization

nsubramani23 authored a paper 4 days ago

SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone

View all activity

loubnabnl

posted an update about 1 hour ago

Post

𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐢𝐧𝐠 🧬 𝐅𝐢𝐧𝐞𝐃𝐍𝐀: 𝐭𝐡𝐞 𝐛𝐞𝐬𝐭 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐭𝐨 𝐭𝐞𝐚𝐜𝐡 𝐚𝐧𝐲 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥 𝐃𝐍𝐀.
Training Qwen3-14B on it produces Carbon-X, the strongest open model on DNA benchmarks, on par with GPT-6.1 Sol.

It contains:
- Midtraining: 300B+ tokens of genomic sequences, papers and web pages mixing DNA with text
- SFT: 200K samples, from variant effect prediction to sequence question answering
- RL: 98K prompts with verifiable rewards

Data, models and the full recipe are open.
HuggingFaceBio/FineDNA

Enjoy this late Halloween treat! 🎃

albertvillanova

posted an update 3 days ago

Post

118

Day-0 Mistral Large 4 support in TRL 🚀

- LoRA SFT of the 1T-A52B model on one 8xB300 node, on top of the FP8 checkpoint
- Experts split 8 ways with expert parallelism, everything else sharded with FSDP, all set at load time
- Assistant-only loss out of the box with its training chat template
- ~32.7k tokens per step in ~8 s

Script + Slurm launcher 👇
https://github.com/huggingface/trl/tree/main/examples/sft_mistral_large_4
https://huggingface.co/posts/sergiopaniego/408182860646834

afaji

submitted a paper to Daily Papers 27 days ago

HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

Paper • 2610.03574 • Published about 1 month ago • 28

christopher

in bigscience/bloom 2 months ago

Security test - please ignore

#294 opened 3 months ago by

hackeronetestt

christopher

in bigscience/bloom 3 months ago

Fixed CO2 metadata in README.md

#293 opened 3 months ago by

Croc-Prog-HF

albertvillanova

posted an update 4 months ago

Post

4,582

🎉 KTO is now part of the stable TRL API

As of Promote KTO to stable API, KTOTrainer and KTOConfig have graduated from trl.experimental to the stable trl API. https://github.com/huggingface/trl/pull/6175

This one closes out a long road. Over the past 6+ months, the "Align KTO with DPO" effort landed ~90 PRs methodically bringing KTO up to the standard we hold for stable trainers, one carefully-scoped change at a time:
- Feature parity with DPO: full VLM support (incl. multi-image), sync_ref_model, PEFT + Liger, ZeRO-3 + PEFT dtype fix, pad_to_multiple_of, activation offloading, IterableDataset and dict eval_dataset, remove_unused_columns, and reference-logprob precomputation at init.
- Consistency with DPO: aligned method order and signatures, tokenization, _prepare_dataset, PEFT handling, ref-model preparation for distributed training, and config layout — plus a new DataCollatorForKTO and output format. Metrics moved into _compute_loss and simplified to direct averages via the shared _metrics attribute.
- Removing legacy baggage: dropped encoder-decoder support, BOS/EOS handling, null_ref_context, generate_during_eval, model_init, preprocess_logits_for_metrics, model/ref adapter names, and several dead config knobs.
- Coverage: a full test suite mirroring DPO, text collator tests, VLM tests, and slow tests.
- The promotion itself: the experimental → stable move (#6175) and shim cleanup (#6287), handled so downstream users get a clean deprecation path.

Honestly, this has been one of the more complex tasks I've taken on since joining the team, not because any single change was hard, but because it demanded sustained consistency across a ~2,000-line trainer, with every branch, comment, and edge case kept in lockstep with DPO.

Huge thanks to everyone who reviewed along the way (especially @qgallouedec ), the incremental review cadence is exactly what kept this maintainable.

KTO now sits on equal footing with our other flagship trainers. 🚀

2 replies

stas

posted an update 4 months ago

Post

1,742

I present to you a new experimental open book.

https://github.com/stas00/python-cookbook

I took my dense Python cheatsheet that I have been honing for many years and use a lot daily and turned it into a book of recipes.

Is this useful?

This is, of course, free, like other open books.

stas

posted an update 4 months ago

Post

500

The Art of Debugging Open Free book is now available in pdf/epub and finally sports a book cover

https://github.com/stas00/the-art-of-debugging#ebook-versions-of-the-book

While a lot of the focus is on Unix/Python/Pytorch, the methodology chapter is applicable to any Software Debugging.

It currently sports 161 packed pages in 5 solid chapters and more coming...

7 replies

stas

posted an update 4 months ago

Post

156

In parallel we announce a new open source repo:

https://github.com/Snowflake-AI-Research/Arctic-Platform

This is the framework for very fast RL (and future other optimizations rolled into it)

It currently has all the code you need to use or integrate Arctic RL into RL frameworks, with SkyRL and Verl available and more framework integrations coming.

Please kindly spread the word! Thank you!

stas

posted an update 4 months ago

Post

3,663

After many months of intense work the
Snowflake AI Research team is happy to present to you the new open source project: Arctic RL

https://snowflake.com/en/blog/engineering/arctic-rl-open-source-backend/

- Arctic RL integrates with VeRL and SkyRL today; enable ZoRRo with one config flag, no code changes required
- ZoRRo delivers up to 6x actor-update acceleration and a 3.5x end-to-end training speedup, reducing Arctic-Text2SQL-R2 training from ~5 days to ~36 hours on 32 H200 GPUs
- Arctic-Text2SQL-R2 achieved higher accuracy scores (48.7) than Gemini 3.1 Pro (47.9) and Claude 4.7 (47.3) on Snowflake's evaluated enterprise SQL benchmark under the tested conditions
- Two open source recipes ship with this release: a text-to-SQL recipe that improved BIRD dev accuracy from 59.92% to 70.35%, and a multi-hop QA recipe that improved average accuracy from 69.6% to 72.3%

4 replies

stas

posted an update 5 months ago

Post

150

PSA for DeepSpeed users - a long outstanding precision-related critical bug has been identified and fixed in https://github.com/deepspeedai/DeepSpeed/pull/8066 and a new release has been made.

The issue was about mixed precision mode downcasting buffers that had to be in fp32 - massively impacting correctness due to large static buffers - e.g. RoPE in Qwen3 models when using long sequence lengths 32K+.

Hopefully this fix brings Deepspeed to a close parity with FSDP2 which has been an issue since a long time.

You can still have the old behavior but you'd now need to manually configure it - by default the model's buffers will now remain in the original precision.

Please install deepspeed==0.19.2 which will do the right thing.

Thanks to Tunji Ruwase and Claude Opus 4.8 via Cursor for identifying and fixing the problem.