FineWeb

publisher: Hugging Face

stage: pretraining

contentType: web text

items: 24B documents

languages: English

notes: Refined CommonCrawl with aggressive quality filtering. Replaces RedPajama-1T as the de facto open pretraining corpus. The dataset behind most 2024-2025 open base models.


FineWeb-Edu

contentType: educational web text

items: 1.3B documents

notes: FineWeb subset filtered to educational content via Llama-3-70B classifier. Smaller models trained on FineWeb-Edu beat counterparts trained on full FineWeb at the same compute.


Common Crawl

publisher: Common Crawl Foundation

tokens: petabyte-scale raw

items: 250B+ webpages

languages: multilingual (200+)

released: monthly since 2008

notes: The raw web crawl every other web pretraining dataset is built on. Released monthly. Most labs filter it heavily before training.


RedPajama v2

publisher: Together AI

items: 100B+ documents

license: MIT (filters), CC-BY (subsets)

languages: English, German, French, Spanish, Italian

notes: Together AI's open reproduction of LLaMA-style pretraining data. v2 ships with 40+ pre-computed quality filters so labs can apply their own thresholds.


The Pile

publisher: EleutherAI

contentType: mixed (web + books + papers + code)

items: 22 sub-corpora

license: mixed (per-subcorpus)

languages: English-focused

notes: The original open pretraining corpus. Largely superseded by FineWeb / RedPajama for new training runs but still cited. Note: Books3 subset removed for copyright.


Dolma

publisher: Allen AI (AI2)

contentType: web + code + books + papers

items: mixed

license: AI2 ImpACT (Medium Risk)

notes: AI2's open pretraining corpus (the dataset behind OLMo). Documents source provenance and offers reproducible filters. The most-academically-cited open pretraining set.


RefinedWeb

publisher: TII

items: 600M documents

notes: Behind the Falcon series. Aggressive deduplication and filtering. Older but cleaner than CommonCrawl raw.


The Stack v2

publisher: BigCode (HuggingFace + ServiceNow)

contentType: source code

tokens: 67TB / 17B files

items: 17B files across 600+ languages

license: permissive only (filtered)

languages: 600+ programming languages

notes: The reference open code-pretraining corpus. v2 is sourced from Software Heritage, dedup'd at file and near-dup level. Behind StarCoder 2 and most open code models.


StarCoderData

publisher: BigCode

contentType: source code + jupyter + GitHub issues

items: 86 languages

license: mixed permissive

languages: 86 programming languages

notes: Pre-cleaned subset of The Stack used to train StarCoder. Smaller than The Stack v2 but easier to work with for small-scale code-model experiments.


Tulu 3 SFT Mixture

publisher: Allen AI

stage: instruction-tuning

contentType: instruction-response pairs

items: 939K instructions

notes: Tulu 3 is the strongest open post-training recipe in 2024-2025. The SFT mixture combines OpenAssistant, ShareGPT, math reasoning, code instructions, and persona-aligned prompts.


OpenHermes 2.5

publisher: Teknium / Nous Research

contentType: multi-turn conversations

items: 1M conversations

license: mixed (per-source)

notes: Aggregated from GPT-4 outputs, AiroborosLM, ShareGPT, and others. The dataset behind Nous Hermes and many community fine-tunes.


Glaive Function Calling v2

publisher: Glaive AI

contentType: function-calling traces

items: 113K examples

notes: Synthetic function-calling traces. The most-cited dataset for fine-tuning open models to do reliable JSON tool use.


OpenOrca

publisher: Open Orca team

contentType: reasoning traces

items: 4.2M GPT-4 + 0.7M GPT-3.5 examples

license: MIT

notes: Reproduction of Microsoft Orca paper. Augments FLAN tasks with GPT-4 chain-of-thought traces. Strong base for math/reasoning fine-tunes.


AgentInstruct

publisher: Microsoft Research

contentType: agentic task traces

items: 25M agent-shaped instruction pairs

notes: Synthetic dataset for agent training. Multi-step tool-use, reading comprehension, code, RAG, brain teasers, content creation. Behind Orca-3 and used in Phi-3.5.


UltraFeedback

publisher: OpenBMB

stage: dpo

contentType: preference pairs

items: 64K instructions x 4 model responses

notes: Most-used DPO dataset for open models. GPT-4 ranks 4 candidate responses per prompt across helpfulness, honesty, instruction-following, truthfulness.


Tulu 3 Preference Mixture

items: 270K preference pairs

notes: Curated preference pairs for the Tulu 3 DPO stage. The strongest open preference data for post-training in late 2024.


HelpSteer 2

publisher: NVIDIA

contentType: preference pairs (5-attribute)

items: 21K conversations

notes: NVIDIA preference data labeled across helpfulness, correctness, coherence, complexity, verbosity. Strong for multi-attribute reward modeling.


publisher: LAION

stage: multimodal

contentType: image-text pairs

items: 5.85B image-text pairs

languages: multilingual

notes: Largest open image-text dataset. Behind Stable Diffusion and many open image models. Original release was withdrawn for review; v2 reissued 2023.


publisher: DataComp

items: 1.4B filtered pairs

license: mixed

notes: Filtered subset of CommonPool. Reproducible filtering recipes are part of the contribution. Behind several open CLIP-class models trained in 2024.