README.md

---
tags:
- multi-vector
- ColBERT
- PyLate
- sentence-transformers
- sentence-similarity
- feature-extraction
- multilingual
- late-interaction
- retrieval
- pretrained
- loss:Distillation
pipeline_tag: sentence-similarity
library_name: PyLate
license: apache-2.0
base_model:
- lightonai/GTE-ModernColBERT-v1
---
<img src="https://vago-solutions.ai/wp-content/uploads/2025/08/SauerkrautLM-Multi-ModernColBERT.png" width="500" height="auto">
# SauerkrautLM-Multi-ModernColBERT

This model is a multilingual Late Interaction retriever that leverages:

**Continuous Pretraining** with 4.6 billion multilingual tokens using knowledge distillation from state-of-the-art reranker models.
**GTE-ModernColBERT Foundation** building upon the English-focused lightonai/GTE-ModernColBERT-v1 model.
**Multilingual Enhancement** extending English capabilities to European languages through targeted multilingual training.

### 🎯 Core Features and Innovations:

- **Multilingual Continuous Pretraining**: Enhanced with 4,641,714,000 multilingual tokens covering 7 European languages while learning from powerful reranker models

- **English-to-Multilingual Transfer**: Successfully extends the strong English performance of GTE-ModernColBERT to European languages

- **Compressed Architecture**: Maintains the efficient 149M parameter design of ModernColBERT

### 💪 From English Excellence to Multilingual Mastery

Starting from the strong **GTE-ModernColBERT-v1** foundation – a model optimized for English retrieval – we've expanded its capabilities through:
- **4.6 billion multilingual tokens** covering 7 European languages
- **Knowledge distillation** from state-of-the-art reranker models
- **Continuous pretraining** that preserves English strength while adding multilingual capabilities

This creates a truly multilingual retriever that maintains exceptional English performance while delivering strong results across European languages.



## Model Overview

**Model:** `VAGOsolutions/SauerkrautLM-Multi-ModernColBERT`\
**Base:** Continuous pretrained from [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) using knowledge distillation\
**Architecture:** PyLate / ColBERT (Late Interaction) with ModernBERT backbone\
**Languages:** Multilingual (optimized for 7 European languages: German, English, Spanish, French, Italian, Dutch, Portuguese)\
**License:** Apache 2.0\
**Model Size:** 149M parameters 
**Additional Training:** 4.6B multilingual tokens via knowledge distillation

### Model Description
- **Model Type:** Multi-vector embedding model with innovative Late Interaction architecture
- **Document Length:** 8192 tokens (32× longer than traditional BERT models)
- **Query Length:** 256 tokens (optimized for complex, multi-part queries)
- **Output Dimensionality:** 128 tokens (efficient vector representation)
- **Similarity Function:** MaxSim (enables precise token-level matching)
- **Training Method:** Continuous pretraining with knowledge distillation

### Architecture

```
ColBERT(
  (0): Transformer(CompressedModernBertModel)
  (1): Dense(384 -> 128 dim, no bias)
)
```

## 🔬 Technical Innovations in Detail

### Multilingual Continuous Pretraining

Our approach transforms an English-specialized model into a multilingual powerhouse:

1. **Base Model Selection**: Starting with GTE-ModernColBERT-v1, which provides state-of-the-art English retrieval
2. **Multilingual Enhancement**: 4,641,714,000 tokens across 7 European languages
3. **Knowledge Distillation**: Learning from state-of-the-art reranker models throughout the training
4. **Balanced Training**: Ensuring strong multilingual capabilities without degrading English performance

### Architectural Advantages

SauerkrautLM-Multi-ModernColBERT leverages:

- **ModernBERT Efficiency**: Compressed architecture with 149M parameters
- **Late Interaction Benefits**: Token-level matching for precise retrieval
- **Cross-lingual Transfer**: Successfully extends English capabilities to multiple languages
- **Maintained Performance**: Preserves the strong English foundation while adding languages

This architecture combines the efficiency of ModernColBERT with true multilingual capabilities.

---

## 🔬 Benchmarks: Multilingual Retrieval Performance

Our evaluation demonstrates strong multilingual retrieval performance, successfully extending GTE-ModernColBERT's English excellence to European languages.

### NanoBEIR Europe (multilingual retrieval)

Average nDCG@10 across seven European languages, showing the effectiveness of our multilingual continuous pretraining:

| Language | nDCG@10 | Performance Notes |
| -------- | -------- | ----------------- |
| en       | **67.70** | Maintains exceptional English performance from base model |
| de       | 51.21    | Strong german language transfer |
| es       | 54.73    | Excellent spanish language capabilities |
| fr       | 54.44    | Consistent cross-lingual performance |
| it       | 53.87    | Balanced multilingual representation |
| nl       | 52.15    | Effective on closely related languages |
| pt       | 53.80    | Maintains quality across language families |

**Key Observations:**
- **Preserved English Excellence**: The continuous pretraining maintains the exceptional English performance (67.70 nDCG@10) from GTE-ModernColBERT
- **Strong Multilingual Addition**: All non-English languages achieve strong performance (51-55 nDCG@10)
- **Successful Transfer**: The model effectively transfers English capabilities to European languages
- **Balanced Performance**: Consistent results across different language families

---

### Why SauerkrautLM-Multi-ModernColBERT Matters for Production

- **Strong language capabilities for european languages**: Maintains state-of-the-art English while adding languages
- **Efficient Architecture**: 149M parameters deployable on standard infrastructure
- **True Multilingual**: Single model for 7 European languages
- **Knowledge Distillation Benefits**: Learns from models many times its size
- **Drop-in Replacement**: Can replace English-only ColBERT models with multilingual support

This model serves as an excellent solution for:
- Organizations expanding from English to European markets
- Multilingual search systems requiring strong English
- Cross-lingual retrieval applications
- Systems needing efficient multilingual models

---

### Real-World Applications

The combination of strong English foundation and multilingual capabilities enables:

1. **Global Search Systems**: Single model for international deployments
2. **E-commerce Expansion**: English-first companies entering European markets
3. **Multilingual Documentation**: Technical documentation search across languages
4. **Customer Support**: Unified search across multilingual knowledge bases
5. **Research Applications**: Cross-lingual academic literature retrieval

## 📈 Summary: English Excellence, Multilingual Capability

SauerkrautLM-Multi-ModernColBERT demonstrates how continuous pretraining can successfully extend an English-specialized model to multiple languages. By combining:

- **GTE-ModernColBERT's strong English foundation**
- **4.6 billion tokens of multilingual training**
- **Knowledge distillation from advanced rerankers**
- **Efficient ModernBERT architecture**

We've created a model that excels in English (67.70 nDCG@10) while delivering strong performance across all European languages. This makes it an ideal choice for organizations that need both exceptional English retrieval and comprehensive multilingual support in a single, efficient model.

---

# Model

This is a multi-vector (ColBERT-style late interaction) embedding model. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator.


## Usage

### Sentence Transformers

This model can be used with [Sentence Transformers](https://www.sbert.net/) as a multi-vector (ColBERT-style late interaction) retriever via the `MultiVectorEncoder`:

```bash
pip install "sentence-transformers>=6.0.0"
```

```python
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-Multi-ModernColBERT")

query = "Welcher Planet ist als der Rote Planet bekannt?"
documents = [
    "Venus wird wegen ihrer ähnlichen Größe und Nähe oft als Erdzwilling bezeichnet.",
    "Mars, bekannt für sein rötliches Aussehen, wird oft als der Rote Planet bezeichnet.",
    "Jupiter, der größte Planet in unserem Sonnensystem, hat einen markanten roten Fleck.",
    "Saturn, berühmt für seine Ringe, wird manchmal für den Roten Planeten gehalten.",
]

query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings[0].shape)
# (32, 128) (29, 128)

# MaxSim late-interaction scoring (higher is more relevant)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[28.7305, 29.5820, 29.0117, 29.1172]])
```

### PyLate

First install the PyLate library:

```bash
pip install -U pylate
```

### Retrieval

PyLate provides a streamlined interface to index and retrieve documents using ColBERT models. The index leverages the Voyager HNSW index to efficiently handle document embeddings and enable fast retrieval.

#### Indexing documents

First, load the ColBERT model and initialize the Voyager index, then encode and index your documents:

```python
from pylate import indexes, models, retrieve

# Step 1: Load the ColBERT model
model = models.ColBERT(
    model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT",
)

# Step 2: Initialize the Voyager index
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
    override=True,  # This overwrites the existing index if any
)

# Step 3: Encode the documents
documents_ids = ["1", "2", "3"]
documents = ["document 1 text", "document 2 text", "document 3 text"]

documents_embeddings = model.encode(
    documents,
    batch_size=32,
    is_query=False,  # Ensure that it is set to False to indicate that these are documents, not queries
    show_progress_bar=True,
)

# Step 4: Add document embeddings to the index by providing embeddings and corresponding ids
index.add_documents(
    documents_ids=documents_ids,
    documents_embeddings=documents_embeddings,
)
```

Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it:

```python
# To load an index, simply instantiate it with the correct folder/name and without overriding it
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
)
```

#### Retrieving top-k documents for queries

Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries.
To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores:

```python
# Step 1: Initialize the ColBERT retriever
retriever = retrieve.ColBERT(index=index)

# Step 2: Encode the queries
queries_embeddings = model.encode(
    ["query for document 3", "query for document 1"],
    batch_size=32,
    is_query=True,  #  # Ensure that it is set to False to indicate that these are queries
    show_progress_bar=True,
)

# Step 3: Retrieve top-k documents
scores = retriever.retrieve(
    queries_embeddings=queries_embeddings,
    k=10,  # Retrieve the top 10 matches for each query
)
```

### Reranking
If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank:

```python
from pylate import rank, models

queries = [
    "query A",
    "query B",
]

documents = [
    ["document A", "document B"],
    ["document 1", "document C", "document B"],
]

documents_ids = [
    [1, 2],
    [1, 3, 2],
]

model = models.ColBERT(
    model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT",
)

queries_embeddings = model.encode(
    queries,
    is_query=True,
)

documents_embeddings = model.encode(
    documents,
    is_query=False,
)

reranked_documents = rank.rerank(
    documents_ids=documents_ids,
    queries_embeddings=queries_embeddings,
    documents_embeddings=documents_embeddings,
)
```
## Citation

### BibTeX

#### SauerkrautLM‑Multi‑ModernColBERT

```bibtex
@misc{SauerkrautLM-Multi-ModernColBERT,
  title={SauerkrautLM-Multi-ModernColBERT},
  author={David Golchinfar},
  url={https://huggingface.co/VAGOsolutions/SauerkrautLM-Multi-ModernColBERT},
  year={2025}
}
```

#### GTE-ModernColBERT

```bibtex
@misc{GTE-ModernColBERT,
title={GTE-ModernColBERT},
author={Chaffin, Antoine},
url={https://huggingface.co/lightonai/GTE-ModernColBERT-v1},
year={2025}
}
```

#### Sentence Transformers

```bibtex
@inproceedings{reimers-2019-sentence-bert,
  title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
  author = {Reimers, Nils and Gurevych, Iryna},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  month = {11},
  year = {2019},
  publisher = {Association for Computational Linguistics},
  url = {https://arxiv.org/abs/1908.10084}
}
```

#### PyLate

```bibtex
@misc{PyLate,
  title={PyLate: Flexible Training and Retrieval for Late Interaction Models},
  author={Chaffin, Antoine and Sourty, Raphaël},
  url={https://github.com/lightonai/pylate},
  year={2024}
}
```


## Acknowledgements
We thank the PyLate team for providing the training framework that made this work possible, and the LightOn AI team for creating the excellent GTE-ModernColBERT base model.

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->