--- tags: - ColBERT - multi-vector - PyLate - sentence-transformers - sentence-similarity - feature-extraction - multilingual - late-interaction - retrieval - pretrained - loss:Distillation pipeline_tag: sentence-similarity library_name: PyLate license: apache-2.0 base_model: - EuroBERT/EuroBERT-210m --- # SauerkrautLM-EuroColBERT This model is a powerful multilingual Late Interaction retriever that leverages: **Continuous Pretraining** with 5.4 billion English tokens using knowledge distillation from state-of-the-art reranker models. **EuroBERT Foundation** building upon the multilingual EuroBERT-210m model specifically designed for European languages. **Late Interaction Architecture** initialized with PyLate for precise token-level matching and superior retrieval performance. ### 🎯 Core Features and Innovations: - **Continuous Pretraining with Distillation**: Enhanced with 5,430,249,475 English tokens while learning from powerful reranker models throughout the training process - **Strong Multilingual Foundation**: Built on EuroBERT/EuroBERT-210m, which was specifically trained for European language understanding - **Full 210M Parameters**: Preserving the complete capacity of the base model for maximum multilingual performance ### 💪 Standing on the Shoulders of Giants Starting from the exceptional **EuroBERT-210m** foundation – a model specifically designed for European languages – we've enhanced it with: - **5.4 billion additional English tokens** through continuous pretraining - **Knowledge distillation** from state-of-the-art reranker models - **Late Interaction architecture** for superior retrieval precision This creates a powerful multilingual retriever that combines the strengths of EuroBERT's native multilingual capabilities with enhanced retrieval performance. ## Model Overview **Model:** `VAGOsolutions/SauerkrautLM-EuroColBERT`\ **Base:** Continuous pretrained from [EuroBERT/EuroBERT-210m](https://huggingface.co/EuroBERT/EuroBERT-210m) using knowledge distillation\ **Architecture:** PyLate / ColBERT (Late Interaction)\ **Languages:** Multilingual (optimized for 7 European languages: German, English, Spanish, French, Italian, Dutch, Portuguese)\ **License:** Apache 2.0\ **Model Size:** 210M parameters **Additional Training:** 5.4B English tokens via knowledge distillation ### Model Description - **Model Type:** Multi-vector embedding model with innovative Late Interaction architecture - **Document Length:** 8192 tokens (32× longer than traditional BERT models) - **Query Length:** 256 tokens (optimized for complex, multi-part queries) - **Output Dimensionality:** 128 tokens (efficient vector representation) - **Similarity Function:** MaxSim (enables precise token-level matching) - **Training Method:** Continuous pretraining with knowledge distillation ### Architecture ``` ColBERT( (0): Transformer(ModernBertModel) (1): Dense(768 -> 128 dim, no bias) ) ``` ## 🔬 Technical Innovations in Detail ### Continuous Pretraining with Knowledge Distillation Our approach enhances the already powerful EuroBERT-210m through: 1. **Base Model Selection**: Starting with EuroBERT-210m, which provides exceptional multilingual understanding for European languages 2. **PyLate Initialization**: Converting the model to a Late Interaction architecture for superior retrieval capabilities 3. **Massive English Enhancement**: 5,430,249,475 English tokens to strengthen English performance 4. **Continuous Distillation**: Learning from state-of-the-art reranker models throughout the training process ### Architectural Advantages SauerkrautLM-EuroColBERT leverages: - **EuroBERT Foundation**: Native multilingual understanding from specialized European language pretraining - **Full Parameter Capacity**: All 210M parameters preserved for maximum performance - **Late Interaction Benefits**: Token-level matching for precise retrieval - **Enhanced English**: Significant improvement in English retrieval through targeted continuous pretraining This architecture combines the advantages of specialized multilingual pretraining with state-of-the-art retrieval techniques. --- ## 🔬 Benchmarks: Multilingual Retrieval Excellence Our evaluation demonstrates strong multilingual retrieval performance, building upon EuroBERT's foundation with enhanced capabilities through continuous pretraining. ### NanoBEIR Europe (multilingual retrieval) Average nDCG@10 across seven European languages, showing the effectiveness of our continuous pretraining approach: | Language | nDCG@10 | Performance Notes | | -------- | -------- | ----------------- | | en | **58.91** | Significantly enhanced through 5.4B token continuous pretraining | | de | 47.71 | Strong german language performance | | es | 52.15 | Excellent spanish language capabilities | | fr | 50.46 | Consistent cross-lingual transfer | | it | 49.85 | Balanced multilingual representation | | nl | 48.47 | Effective on lower-resource languages | | pt | 50.72 | Maintains quality across language families | **Key Observations:** - **Exceptional English Performance**: The continuous pretraining with 5.4B English tokens yields outstanding English results (58.91 nDCG@10) - **Strong Multilingual Balance**: All non-English languages maintain excellent performance (47-52 nDCG@10) - **EuroBERT Foundation Benefits**: The specialized European language pretraining provides robust cross-lingual capabilities - **Distillation Effectiveness**: Knowledge distillation from reranker models enhances retrieval quality across all languages --- ### Why SauerkrautLM-EuroColBERT Matters for Production - **Best of Both Worlds**: Combines EuroBERT's multilingual excellence with enhanced English retrieval - **Production Ready**: 210M parameters deployable on standard infrastructure - **State-of-the-Art Multilingual**: Superior performance across 7 European languages - **Knowledge Distillation Benefits**: Learns from models many times its size - **Flexible Foundation**: Ideal base for further fine-tuning on specific tasks This model serves as an excellent foundation for: - Multilingual search systems - Cross-lingual retrieval applications - European language document retrieval - Production systems requiring balanced multilingual performance --- ### Real-World Applications The combination of EuroBERT's multilingual foundation and our continuous pretraining enables: 1. **Enterprise Search**: Deploy across multinational organizations with confidence 2. **E-commerce**: Single model for product search across European markets 3. **Digital Libraries**: Multilingual document retrieval with high precision 4. **Customer Support**: Cross-lingual knowledge base search 5. **Research Applications**: Academic and scientific literature retrieval ## 📈 Summary: Enhanced Multilingual Retrieval SauerkrautLM-EuroColBERT represents the successful enhancement of an already excellent multilingual model. By combining: - **EuroBERT's specialized European language foundation** - **5.4 billion tokens of continuous pretraining** - **Knowledge distillation from state-of-the-art rerankers** - **PyLate's Late Interaction architecture** We've created a model that delivers exceptional multilingual retrieval performance while remaining practical for real-world deployment. The model particularly excels in English (58.91 nDCG@10) while maintaining strong performance across all European languages, making it an ideal choice for multilingual retrieval applications. --- # Model This is a multi-vector (ColBERT-style late interaction) embedding model. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator. ## Usage ### Sentence Transformers This model can be used with [Sentence Transformers](https://www.sbert.net/) as a multi-vector (ColBERT-style late interaction) retriever via the `MultiVectorEncoder`: ```bash pip install "sentence-transformers>=6.0.0" ``` ```python from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-EuroColBERT") query = "Which planet is known as the Red Planet?" documents = [ "Venus wird oft als Zwilling der Erde bezeichnet, wegen ihrer ähnlichen Größe.", "Mars, connue pour son apparence rougeâtre, est souvent appelée la planète rouge.", "Júpiter es el planeta más grande del sistema solar.", "Saturno è famoso per i suoi bellissimi anelli.", ] query_embeddings = model.encode_query(query) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings[0].shape) # (32, 128) (21, 128) # MaxSim late-interaction scoring (higher is more relevant) scores = model.similarity(query_embeddings, document_embeddings) print(scores) # tensor([[29.0540, 29.6528, 29.3835, 28.6055]]) ``` ### PyLate First install the PyLate library: ```bash pip install -U pylate ``` ### Retrieval PyLate provides a streamlined interface to index and retrieve documents using ColBERT models. The index leverages the Voyager HNSW index to efficiently handle document embeddings and enable fast retrieval. #### Indexing documents First, load the ColBERT model and initialize the Voyager index, then encode and index your documents: ```python from pylate import indexes, models, retrieve # Step 1: Load the ColBERT model model = models.ColBERT( model_name_or_path="VAGOsolutions/SauerkrautLM-EuroColBERT", ) # Step 2: Initialize the Voyager index index = indexes.Voyager( index_folder="pylate-index", index_name="index", override=True, # This overwrites the existing index if any ) # Step 3: Encode the documents documents_ids = ["1", "2", "3"] documents = ["document 1 text", "document 2 text", "document 3 text"] documents_embeddings = model.encode( documents, batch_size=32, is_query=False, # Ensure that it is set to False to indicate that these are documents, not queries show_progress_bar=True, ) # Step 4: Add document embeddings to the index by providing embeddings and corresponding ids index.add_documents( documents_ids=documents_ids, documents_embeddings=documents_embeddings, ) ``` Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it: ```python # To load an index, simply instantiate it with the correct folder/name and without overriding it index = indexes.Voyager( index_folder="pylate-index", index_name="index", ) ``` #### Retrieving top-k documents for queries Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries. To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores: ```python # Step 1: Initialize the ColBERT retriever retriever = retrieve.ColBERT(index=index) # Step 2: Encode the queries queries_embeddings = model.encode( ["query for document 3", "query for document 1"], batch_size=32, is_query=True, # # Ensure that it is set to False to indicate that these are queries show_progress_bar=True, ) # Step 3: Retrieve top-k documents scores = retriever.retrieve( queries_embeddings=queries_embeddings, k=10, # Retrieve the top 10 matches for each query ) ``` ### Reranking If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank: ```python from pylate import rank, models queries = [ "query A", "query B", ] documents = [ ["document A", "document B"], ["document 1", "document C", "document B"], ] documents_ids = [ [1, 2], [1, 3, 2], ] model = models.ColBERT( model_name_or_path="VAGOsolutions/SauerkrautLM-EuroColBERT", ) queries_embeddings = model.encode( queries, is_query=True, ) documents_embeddings = model.encode( documents, is_query=False, ) reranked_documents = rank.rerank( documents_ids=documents_ids, queries_embeddings=queries_embeddings, documents_embeddings=documents_embeddings, ) ``` ## Citation ### BibTeX #### SauerkrautLM‑EuroColBERT ```bibtex @misc{SauerkrautLM-EuroColBERT, title={SauerkrautLM-EuroColBERT}, author={David Golchinfar}, url={https://huggingface.co/VAGOsolutions/SauerkrautLM-EuroColBERT}, year={2025} } ``` #### EuroBERT-210m ```bibtex @misc{boizard2025eurobertscalingmultilingualencoders, title={EuroBERT: Scaling Multilingual Encoders for European Languages}, author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo}, year={2025}, eprint={2503.05500}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2503.05500}, } ``` #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, author = {Reimers, Nils and Gurevych, Iryna}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing}, month = {11}, year = {2019}, publisher = {Association for Computational Linguistics}, url = {https://arxiv.org/abs/1908.10084} } ``` #### PyLate ```bibtex @misc{PyLate, title={PyLate: Flexible Training and Retrieval for Late Interaction Models}, author={Chaffin, Antoine and Sourty, Raphaël}, url={https://github.com/lightonai/pylate}, year={2024} } ``` ## Acknowledgements We thank the PyLate team for providing the training framework that made this work possible, and the EuroBERT team for creating the excellent multilingual foundation model.