Update README.md

README.md

@@ -5,26 +5,113 @@ tags:
 - sentence-transformers
 - sentence-similarity
 - feature-extraction
+license: apache-2.0
+language:
+- en
 pipeline_tag: sentence-similarity
 library_name: PyLate
 ---
 
-# PyLate
+<div align="center">
+<img src="https://cdn-uploads.huggingface.co/production/uploads/609bbe2f4932693ca2009d6a/kbQOAarw0eaApow3M9HIl.png" alt="LightOn" width="512">
 
-This is a [PyLate](https://github.com/lightonai/pylate) model trained. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator.
+[![Website](https://img.shields.io/badge/LightOn-Website-blue?logo=google-chrome)](https://lighton.ai)
+[![LinkedIn](https://img.shields.io/badge/LightOn-LinkedIn-0A66C2?logo=linkedin)](https://www.linkedin.com/company/lighton/)
+[![X](https://img.shields.io/badge/@LightOnIO-X-black?logo=x)](https://x.com/LightOnIO)
+
+  📚 [Collection](https://huggingface.co/collections/lightonai/denseon-and-lateon) | 📝 [Blog](https://huggingface.co/blog/lightonai/denseon-lateon)
+
+</div>
+
+<h1 align="center">LateOn-unsupervised</h1>
+
+<h3 align="center">State-of-the-Art ColBERT Retrieval Model by LightOn</h3>
+
+<p align="center">
+<a href="https://huggingface.co/lightonai/DenseOn">DenseOn</a> |
+<a href="https://huggingface.co/lightonai/LateOn">LateOn</a> |
+<a href="https://github.com/lightonai/pylate">PyLate</a> |
+<a href="https://github.com/lightonai/fast-plaid">FastPLAID</a>
+</p>
+
+> 🎯 **TL;DR**: The intermediate ColBERT checkpoint produced by **Stage 1 only** (unsupervised contrastive pre-training) of the LateOn pipeline Released as a **strong starting point for your own supervised fine-tuning, knowledge distillation, or downstream adaptation**.
+
+## About the LateOn / DenseOn Family
+
+State-of-the-art retrieval is increasingly dominated by closed models, either hidden behind APIs or trained on undisclosed data. This blocks reproducibility, prevents study of possible data leakage, and gatekeeps progress to a handful of private labs. We thus decided to gather and curate a large amount of data and explore various mixtures. We release all the data used in our explorations:
+- [Gathered pre-training data](https://huggingface.co/datasets/lightonai/embeddings-pre-training), 1.4B query-documents pairs alongside annotations used for non-destructive filtering (structural filtering, deduplication, cross-encoder pair relevancy)
+- [Best pre-training mixture](https://huggingface.co/datasets/lightonai/embeddings-pre-training-curated) found with already applied filters
+- [Fine-tuning datasets](https://huggingface.co/datasets/lightonai/embeddings-fine-tuning) with query, positive and 2048 mined documents alongside their scores for 1.88M samples.
+
+Based on our findings, we trained LateOn (multi-vector/ColBERT) and DenseOn (single vector/dense) models on a proprietary Apache 2.0-compatible training dataset and release those models as well.
+Both are built on the [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) backbone at 149M parameters, a size we believe sits at the sweet spot: large enough to handle real-world queries and documents, small enough to serve at high throughput in latency-sensitive production systems.
+For more information, please read our [blogpost](https://huggingface.co/blog/lightonai/denseon-lateon).
+
+## LateOn-unsupervised
+
+LateOn-unsupervised is the output of the **first stage** of the [LateOn](https://huggingface.co/lightonai/LateOn) training pipeline. It has been pre-trained on a large, filtered corpus of query-document pairs using in-batch contrastive learning, but has **not** yet been fine-tuned with mined hard negatives.
+
+For most production use cases, you should use the **fully-trained [LateOn](https://huggingface.co/lightonai/LateOn)** instead. This unsupervised checkpoint is intended for:
+
+- Researchers studying what each pipeline stage contributes
+- Practitioners who want to fine-tune on their own domain-specific data
+- Distillation experiments where you want to start from a strong but un-aligned base
+- Anyone running their own ablations on hard-negative mining strategies
+
+
+## Results
+
+### BEIR (14 datasets, NDCG@10)
+
+| Model | Average | Size (M) | Embed dim | ArguAna | CQADupstackRetrieval | ClimateFEVER | DBPedia | FEVER | FiQA2018 | HotpotQA | MSMARCO | NFCorpus | NQ | QuoraRetrieval | SCIDOCS | SciFact | TRECCOVID | Touche2020 |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| [ColBERTv2](https://huggingface.co/colbert-ir/colbertv2.0) | 48.63 | 110 | 128 | 46.50 | *38.30* | 17.60 | 45.20 | 78.50 | 35.40 | 67.50 | *46.00* | 33.70 | 52.40 | 85.50 | 15.40 | 68.90 | 72.60 | 26.00 |
+| [Jina-ColBERT-v2](https://huggingface.co/jinaai/jina-colbert-v2) | 51.85 | 600 | 128 | 36.60 | *40.80* | 23.90 | 47.10 | 80.50 | 40.80 | 76.60 | **46.90** | 34.60 | **64.00** | 88.70 | 18.60 | 67.80 | 83.40 | 27.40 |
+| [ColBERT-small](https://huggingface.co/answerdotai/answerai-colbert-small-v1) | 53.79 | 33 | 96 | 50.09 | 38.75 | 33.07 | 45.58 | 90.96 | 41.15 | 76.11 | 43.50 | 37.30 | 59.10 | 87.72 | 18.42 | 74.77 | 84.59 | 25.69 |
+| [GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) | 54.75 | 149 | 128 | 47.52 | 41.08 | 31.33 | 47.56 | 87.67 | 45.25 | 77.48 | 45.60 | 37.83 | 61.62 | 86.71 | 19.22 | 76.33 | **84.84** | 31.25 |
+| [ColBERT-Zero](https://huggingface.co/lightonai/ColBERT-Zero) | 55.39 | 149 | 128 | **52.82** | 41.41 | 35.90 | 47.43 | 90.52 | 42.50 | 79.45 | 45.95 | 37.21 | 61.82 | 85.19 | 19.84 | 76.33 | 78.27 | **36.24** |
+| [LateOn-unsupervised](https://huggingface.co/lightonai/LateOn-unsupervised) | 50.11 | 149 | 128 | 43.12 | **47.71** | 18.76 | 43.36 | 65.74 | 51.94 | 68.17 | 37.51 | 37.15 | 58.41 | 89.48 | 21.13 | **76.89** | 69.81 | 22.53 |
+| [LateOn](https://huggingface.co/lightonai/LateOn) | **57.22** | 149 | 128 | 50.52 | 47.36 | **39.67** | 45.99 | **92.02** | **53.12** | **79.98** | 45.67 | 37.79 | 63.91 | **89.67** | **21.90** | 76.61 | 83.60 | 30.52 |
+
+LateOn achieves 57.22 average NDCG@10, becoming the first ColBERT (and sub 150M parameter) model to break the 57 mark on BEIR by surpassing the previous best ColBERT model (ColBERT-Zero at 55.32) by almost two points and exceeding GTE-ModernColBERT-v1 by two point and a half, despite sharing the same backbone. 
+It is worth noting that we achieve this performance only with contrastive data and without the use of the prompt (which has been shown to be beneficial in the ColBERT-Zero study but makes it harder and more expensive to use). This work was focused on an exploration of contrastive data, but we believe we can achieve much stronger results through a KD phase and the addition of prompts to serve as query expansion. 
+Given such strong performances, one fair concern would be overfitting to BEIR. As highlighted in the decontaminated BEIR experiments, our models stay very strong on benches curated from any possible data leakage, especially in the multi-vector case and contrary to some other models.
+
+### Decontaminated BEIR (12 datasets, NDCG@10)
+
+Standard benchmarks risk overestimating model quality when training data overlaps with evaluation corpora. This is a non-negligible risk in our case, as our mixture explorations are mostly built on BEIR evaluation. To quantify this and ensure that our model has not memorized possible leakage, we built decontaminated versions of the BEIR datasets by removing samples found in both the mGTE training dataset and in our internal training datasets. 
+Since many retrieval models draw from similar public sources (Wikipedia, MS MARCO, Common Crawl, academic corpora), we expect significant overlap across models and believe the decontaminated benchmarks provide a meaningful, if imperfect, stress test. The decontaminated datasets are publicly available on HuggingFace.
+
+
+Despite being in the toughest position (as the decontamination is based on our data), LateOn and DenseOn stay consistent under decontamination. 
+LateOn keeps its #1 position and DenseOn stays in the top four (only falling behind our other strong multi-vector model, ColBERT-Zero). Neither model flinches, which is direct evidence of generalization rather than overfitting.
+
+More broadly, ColBERT models seems to generalize better under decontamination: all three ColBERT models hold or improve their ranking, and they take 2 of the top 3 decontaminated positions. Although DenseOn holds strong, some dense models are hit harder, for example `GTE-ModernBERT`, dropping from 8th to last, which is particularly interesting considering our base mixture is derived from theirs. This highlights the strength of our curation methodology. While other models such as `Qwen3-Embedding-0.6B` also drop some ranks, hinting at an overlap with the BEIR evaluation, it is worth noting that newer models, such as the new `jina-embeddings-v5` and `pplx-embed-v1-0.6b` seems to exhibit stronger evidence of generalization rather than overfitting.
+
+<div align="center">
+<img src="https://cdn-uploads.huggingface.co/production/uploads/61dd91afdb45389905634d35/9kMmY66jIZsLxrQ6a7v3L.png" alt="LightOn" width="1024">
+</div>
+
+## Related Checkpoints
+
+| Model | Stage | Link |
+|:---|:---|:---|
+| **LateOn-unsupervised** *(this card)* | Pre-training only | [`lightonai/LateOn-unsupervised`](https://huggingface.co/lightonai/LateOn-unsupervised) |
+| LateOn | Pre-training + fine-tuning *(recommended)* | [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn) |
+| DenseOn-unsupervised | Single-vector counterpart, pre-training only | [`lightonai/DenseOn-unsupervised`](https://huggingface.co/lightonai/DenseOn-unsupervised) |
+| DenseOn | Single-vector counterpart, full pipeline | [`lightonai/DenseOn`](https://huggingface.co/lightonai/DenseOn) |
 
 ## Model Details
 
 ### Model Description
 - **Model Type:** PyLate model
-<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
+- **Base model:** [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) 
 - **Document Length:** 300 tokens
 - **Query Length:** 32 tokens
 - **Output Dimensionality:** 128 tokens
 - **Similarity Function:** MaxSim
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
+- **Language:** English
+- **License:** Apache 2.0
 
 ### Model Sources
 
@@ -220,6 +307,8 @@ You can finetune this model on your own dataset.
 
 ### BibTeX
 
+#### DenseOn and LateOn
+
 ```bibtex
 @misc{sourty2026denseonlateon,
   title={DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models},
@@ -229,6 +318,8 @@ You can finetune this model on your own dataset.
 }
 ```
 
+#### PyLate
+
 ```bibtex
 @inproceedings{DBLP:conf/cikm/ChaffinS25,
   author       = {Antoine Chaffin and
@@ -255,6 +346,27 @@ You can finetune this model on your own dataset.
 }
 ```
 
+#### Sentence Transformers
+
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084"
+}
+```
+
+## Acknowledgements
+
+We thank Xin Zhang, Zach Nussbaum, Tom Aarsen, Bo Wang, Eugene Yang, Benjamin Clavié, Nandan Thakur, Oskar Hallström and Iacopo Poli for their valuable contributions and feedback. We are grateful to the teams behind Sentence Transformers and the BEIR benchmark, and to the open-source retrieval community, in particular the authors of Nomic Embed.
+
+This work was granted access to the HPC resources of IDRIS under GENCI allocations AS011016449, A0181016214, and A0171015706 (Jean Zay supercomputer). We also acknowledge the Barcelona Supercomputing Center (BSC-CNS) for providing access to MareNostrum 5 under EuroHPC AI Factory Fast Lane project EHPC-AIF-2025FL01-445.
+
+
 <!--
 ## Glossary