| --- |
| tags: |
| - multi-vector |
| - ColBERT |
| - PyLate |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - multilingual |
| - late-interaction |
| - retrieval |
| - pretrained |
| - loss:Distillation |
| pipeline_tag: sentence-similarity |
| library_name: PyLate |
| license: apache-2.0 |
| base_model: |
| - lightonai/GTE-ModernColBERT-v1 |
| --- |
| <img src="https://vago-solutions.ai/wp-content/uploads/2025/08/SauerkrautLM-Multi-ModernColBERT.png" width="500" height="auto"> |
| # SauerkrautLM-Multi-ModernColBERT |
|
|
| This model is a multilingual Late Interaction retriever that leverages: |
|
|
| **Continuous Pretraining** with 4.6 billion multilingual tokens using knowledge distillation from state-of-the-art reranker models. |
| **GTE-ModernColBERT Foundation** building upon the English-focused lightonai/GTE-ModernColBERT-v1 model. |
| **Multilingual Enhancement** extending English capabilities to European languages through targeted multilingual training. |
|
|
| ### 🎯 Core Features and Innovations: |
|
|
| - **Multilingual Continuous Pretraining**: Enhanced with 4,641,714,000 multilingual tokens covering 7 European languages while learning from powerful reranker models |
|
|
| - **English-to-Multilingual Transfer**: Successfully extends the strong English performance of GTE-ModernColBERT to European languages |
|
|
| - **Compressed Architecture**: Maintains the efficient 149M parameter design of ModernColBERT |
|
|
| ### 💪 From English Excellence to Multilingual Mastery |
|
|
| Starting from the strong **GTE-ModernColBERT-v1** foundation – a model optimized for English retrieval – we've expanded its capabilities through: |
| - **4.6 billion multilingual tokens** covering 7 European languages |
| - **Knowledge distillation** from state-of-the-art reranker models |
| - **Continuous pretraining** that preserves English strength while adding multilingual capabilities |
|
|
| This creates a truly multilingual retriever that maintains exceptional English performance while delivering strong results across European languages. |
|
|
|
|
|
|
| ## Model Overview |
|
|
| **Model:** `VAGOsolutions/SauerkrautLM-Multi-ModernColBERT`\ |
| **Base:** Continuous pretrained from [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) using knowledge distillation\ |
| **Architecture:** PyLate / ColBERT (Late Interaction) with ModernBERT backbone\ |
| **Languages:** Multilingual (optimized for 7 European languages: German, English, Spanish, French, Italian, Dutch, Portuguese)\ |
| **License:** Apache 2.0\ |
| **Model Size:** 149M parameters |
| **Additional Training:** 4.6B multilingual tokens via knowledge distillation |
|
|
| ### Model Description |
| - **Model Type:** Multi-vector embedding model with innovative Late Interaction architecture |
| - **Document Length:** 8192 tokens (32× longer than traditional BERT models) |
| - **Query Length:** 256 tokens (optimized for complex, multi-part queries) |
| - **Output Dimensionality:** 128 tokens (efficient vector representation) |
| - **Similarity Function:** MaxSim (enables precise token-level matching) |
| - **Training Method:** Continuous pretraining with knowledge distillation |
|
|
| ### Architecture |
|
|
| ``` |
| ColBERT( |
| (0): Transformer(CompressedModernBertModel) |
| (1): Dense(384 -> 128 dim, no bias) |
| ) |
| ``` |
|
|
| ## 🔬 Technical Innovations in Detail |
|
|
| ### Multilingual Continuous Pretraining |
|
|
| Our approach transforms an English-specialized model into a multilingual powerhouse: |
|
|
| 1. **Base Model Selection**: Starting with GTE-ModernColBERT-v1, which provides state-of-the-art English retrieval |
| 2. **Multilingual Enhancement**: 4,641,714,000 tokens across 7 European languages |
| 3. **Knowledge Distillation**: Learning from state-of-the-art reranker models throughout the training |
| 4. **Balanced Training**: Ensuring strong multilingual capabilities without degrading English performance |
|
|
| ### Architectural Advantages |
|
|
| SauerkrautLM-Multi-ModernColBERT leverages: |
|
|
| - **ModernBERT Efficiency**: Compressed architecture with 149M parameters |
| - **Late Interaction Benefits**: Token-level matching for precise retrieval |
| - **Cross-lingual Transfer**: Successfully extends English capabilities to multiple languages |
| - **Maintained Performance**: Preserves the strong English foundation while adding languages |
|
|
| This architecture combines the efficiency of ModernColBERT with true multilingual capabilities. |
|
|
| --- |
|
|
| ## 🔬 Benchmarks: Multilingual Retrieval Performance |
|
|
| Our evaluation demonstrates strong multilingual retrieval performance, successfully extending GTE-ModernColBERT's English excellence to European languages. |
|
|
| ### NanoBEIR Europe (multilingual retrieval) |
|
|
| Average nDCG@10 across seven European languages, showing the effectiveness of our multilingual continuous pretraining: |
|
|
| | Language | nDCG@10 | Performance Notes | |
| | -------- | -------- | ----------------- | |
| | en | **67.70** | Maintains exceptional English performance from base model | |
| | de | 51.21 | Strong german language transfer | |
| | es | 54.73 | Excellent spanish language capabilities | |
| | fr | 54.44 | Consistent cross-lingual performance | |
| | it | 53.87 | Balanced multilingual representation | |
| | nl | 52.15 | Effective on closely related languages | |
| | pt | 53.80 | Maintains quality across language families | |
|
|
| **Key Observations:** |
| - **Preserved English Excellence**: The continuous pretraining maintains the exceptional English performance (67.70 nDCG@10) from GTE-ModernColBERT |
| - **Strong Multilingual Addition**: All non-English languages achieve strong performance (51-55 nDCG@10) |
| - **Successful Transfer**: The model effectively transfers English capabilities to European languages |
| - **Balanced Performance**: Consistent results across different language families |
|
|
| --- |
|
|
| ### Why SauerkrautLM-Multi-ModernColBERT Matters for Production |
|
|
| - **Strong language capabilities for european languages**: Maintains state-of-the-art English while adding languages |
| - **Efficient Architecture**: 149M parameters deployable on standard infrastructure |
| - **True Multilingual**: Single model for 7 European languages |
| - **Knowledge Distillation Benefits**: Learns from models many times its size |
| - **Drop-in Replacement**: Can replace English-only ColBERT models with multilingual support |
|
|
| This model serves as an excellent solution for: |
| - Organizations expanding from English to European markets |
| - Multilingual search systems requiring strong English |
| - Cross-lingual retrieval applications |
| - Systems needing efficient multilingual models |
|
|
| --- |
|
|
| ### Real-World Applications |
|
|
| The combination of strong English foundation and multilingual capabilities enables: |
|
|
| 1. **Global Search Systems**: Single model for international deployments |
| 2. **E-commerce Expansion**: English-first companies entering European markets |
| 3. **Multilingual Documentation**: Technical documentation search across languages |
| 4. **Customer Support**: Unified search across multilingual knowledge bases |
| 5. **Research Applications**: Cross-lingual academic literature retrieval |
|
|
| ## 📈 Summary: English Excellence, Multilingual Capability |
|
|
| SauerkrautLM-Multi-ModernColBERT demonstrates how continuous pretraining can successfully extend an English-specialized model to multiple languages. By combining: |
|
|
| - **GTE-ModernColBERT's strong English foundation** |
| - **4.6 billion tokens of multilingual training** |
| - **Knowledge distillation from advanced rerankers** |
| - **Efficient ModernBERT architecture** |
|
|
| We've created a model that excels in English (67.70 nDCG@10) while delivering strong performance across all European languages. This makes it an ideal choice for organizations that need both exceptional English retrieval and comprehensive multilingual support in a single, efficient model. |
|
|
| --- |
|
|
| # Model |
|
|
| This is a multi-vector (ColBERT-style late interaction) embedding model. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator. |
|
|
|
|
| ## Usage |
|
|
| ### Sentence Transformers |
|
|
| This model can be used with [Sentence Transformers](https://www.sbert.net/) as a multi-vector (ColBERT-style late interaction) retriever via the `MultiVectorEncoder`: |
|
|
| ```bash |
| pip install "sentence-transformers>=6.0.0" |
| ``` |
|
|
| ```python |
| from sentence_transformers import MultiVectorEncoder |
| |
| model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-Multi-ModernColBERT") |
| |
| query = "Welcher Planet ist als der Rote Planet bekannt?" |
| documents = [ |
| "Venus wird wegen ihrer ähnlichen Größe und Nähe oft als Erdzwilling bezeichnet.", |
| "Mars, bekannt für sein rötliches Aussehen, wird oft als der Rote Planet bezeichnet.", |
| "Jupiter, der größte Planet in unserem Sonnensystem, hat einen markanten roten Fleck.", |
| "Saturn, berühmt für seine Ringe, wird manchmal für den Roten Planeten gehalten.", |
| ] |
| |
| query_embeddings = model.encode_query(query) |
| document_embeddings = model.encode_document(documents) |
| print(query_embeddings.shape, document_embeddings[0].shape) |
| # (32, 128) (29, 128) |
| |
| # MaxSim late-interaction scoring (higher is more relevant) |
| scores = model.similarity(query_embeddings, document_embeddings) |
| print(scores) |
| # tensor([[28.7305, 29.5820, 29.0117, 29.1172]]) |
| ``` |
|
|
| ### PyLate |
|
|
| First install the PyLate library: |
|
|
| ```bash |
| pip install -U pylate |
| ``` |
|
|
| ### Retrieval |
|
|
| PyLate provides a streamlined interface to index and retrieve documents using ColBERT models. The index leverages the Voyager HNSW index to efficiently handle document embeddings and enable fast retrieval. |
|
|
| #### Indexing documents |
|
|
| First, load the ColBERT model and initialize the Voyager index, then encode and index your documents: |
|
|
| ```python |
| from pylate import indexes, models, retrieve |
| |
| # Step 1: Load the ColBERT model |
| model = models.ColBERT( |
| model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT", |
| ) |
| |
| # Step 2: Initialize the Voyager index |
| index = indexes.Voyager( |
| index_folder="pylate-index", |
| index_name="index", |
| override=True, # This overwrites the existing index if any |
| ) |
| |
| # Step 3: Encode the documents |
| documents_ids = ["1", "2", "3"] |
| documents = ["document 1 text", "document 2 text", "document 3 text"] |
| |
| documents_embeddings = model.encode( |
| documents, |
| batch_size=32, |
| is_query=False, # Ensure that it is set to False to indicate that these are documents, not queries |
| show_progress_bar=True, |
| ) |
| |
| # Step 4: Add document embeddings to the index by providing embeddings and corresponding ids |
| index.add_documents( |
| documents_ids=documents_ids, |
| documents_embeddings=documents_embeddings, |
| ) |
| ``` |
|
|
| Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it: |
|
|
| ```python |
| # To load an index, simply instantiate it with the correct folder/name and without overriding it |
| index = indexes.Voyager( |
| index_folder="pylate-index", |
| index_name="index", |
| ) |
| ``` |
|
|
| #### Retrieving top-k documents for queries |
|
|
| Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries. |
| To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores: |
|
|
| ```python |
| # Step 1: Initialize the ColBERT retriever |
| retriever = retrieve.ColBERT(index=index) |
| |
| # Step 2: Encode the queries |
| queries_embeddings = model.encode( |
| ["query for document 3", "query for document 1"], |
| batch_size=32, |
| is_query=True, # # Ensure that it is set to False to indicate that these are queries |
| show_progress_bar=True, |
| ) |
| |
| # Step 3: Retrieve top-k documents |
| scores = retriever.retrieve( |
| queries_embeddings=queries_embeddings, |
| k=10, # Retrieve the top 10 matches for each query |
| ) |
| ``` |
|
|
| ### Reranking |
| If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank: |
|
|
| ```python |
| from pylate import rank, models |
| |
| queries = [ |
| "query A", |
| "query B", |
| ] |
| |
| documents = [ |
| ["document A", "document B"], |
| ["document 1", "document C", "document B"], |
| ] |
| |
| documents_ids = [ |
| [1, 2], |
| [1, 3, 2], |
| ] |
| |
| model = models.ColBERT( |
| model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT", |
| ) |
| |
| queries_embeddings = model.encode( |
| queries, |
| is_query=True, |
| ) |
| |
| documents_embeddings = model.encode( |
| documents, |
| is_query=False, |
| ) |
| |
| reranked_documents = rank.rerank( |
| documents_ids=documents_ids, |
| queries_embeddings=queries_embeddings, |
| documents_embeddings=documents_embeddings, |
| ) |
| ``` |
| ## Citation |
|
|
| ### BibTeX |
|
|
| #### SauerkrautLM‑Multi‑ModernColBERT |
|
|
| ```bibtex |
| @misc{SauerkrautLM-Multi-ModernColBERT, |
| title={SauerkrautLM-Multi-ModernColBERT}, |
| author={David Golchinfar}, |
| url={https://huggingface.co/VAGOsolutions/SauerkrautLM-Multi-ModernColBERT}, |
| year={2025} |
| } |
| ``` |
|
|
| #### GTE-ModernColBERT |
|
|
| ```bibtex |
| @misc{GTE-ModernColBERT, |
| title={GTE-ModernColBERT}, |
| author={Chaffin, Antoine}, |
| url={https://huggingface.co/lightonai/GTE-ModernColBERT-v1}, |
| year={2025} |
| } |
| ``` |
|
|
| #### Sentence Transformers |
|
|
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, |
| author = {Reimers, Nils and Gurevych, Iryna}, |
| booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing}, |
| month = {11}, |
| year = {2019}, |
| publisher = {Association for Computational Linguistics}, |
| url = {https://arxiv.org/abs/1908.10084} |
| } |
| ``` |
|
|
| #### PyLate |
|
|
| ```bibtex |
| @misc{PyLate, |
| title={PyLate: Flexible Training and Retrieval for Late Interaction Models}, |
| author={Chaffin, Antoine and Sourty, Raphaël}, |
| url={https://github.com/lightonai/pylate}, |
| year={2024} |
| } |
| ``` |
|
|
|
|
| ## Acknowledgements |
| We thank the PyLate team for providing the training framework that made this work possible, and the LightOn AI team for creating the excellent GTE-ModernColBERT base model. |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
