SauerkrautLM-Vision-Document-Retrieval
Collection
7 items • Updated • 9
SauerkrautLM-ColQwen3-8b-v0.1
🏆 #1 among 128-dim Models | State-of-the-Art Visual Document Retrieval
SauerkrautLM-ColQwen3-8b-v0.1 is the best-performing 128-dimensional embedding model for visual document retrieval, achieving 91.08 NDCG@5 on ViDoRe v1 - the highest score among all models with 128-dim embeddings.
🎯 Why Visual Document Retrieval?
Traditional OCR-based retrieval loses layout, tables, and visual context. Our visual approach:
- ✅ No OCR errors - Direct visual understanding
- ✅ Layout-aware - Understands tables, forms, charts
- ✅ End-to-end - Single model, no pipeline complexity
🏆 Key Achievements
| Benchmark | Score | Rank (128-dim) | Rank (All) |
|---|---|---|---|
| ViDoRe v1 | 91.08 | 🥇 #1 | #1 |
| MTEB v1+v2 | 82.91 | #2 | #5 |
| ViDoRe v3 | 58.55 | 🥇 #1 | #3 |
128-dim Models Comparison (XLarge Category)
| Model | Params | Dim | ViDoRe v1 | MTEB v1+v2 | ViDoRe v3 |
|---|---|---|---|---|---|
| SauerkrautLM-ColQwen3-8b-v0.1 ⭐ | 8.0B | 128 | 91.08 | 82.91 | 58.55 |
| EvoQwen2.5-VL-Retriever-7B-v1 | 7.0B | 128 | 90.68 | 83.41 | - |
| colnomic-embed-multimodal-7b | 7.0B | 128 | 89.72 | 81.30 | 57.64 |
Our 8B model achieves the highest ViDoRe v1 and v3 scores among ALL 128-dim models!
Detailed Benchmark Results
📊 ViDoRe v1 (NDCG@5) - Click to expand
| Task | Score |
|---|---|
| ArxivQA | 93.80 🥇 |
| DocVQA | 64.69 |
| InfoVQA | 94.51 |
| ShiftProject | 90.41 |
| SyntheticDocQA-AI | 98.65 |
| SyntheticDocQA-Energy | 96.52 |
| SyntheticDocQA-Gov | 96.79 |
| SyntheticDocQA-Health | 99.26 |
| TabFQuAD | 92.18 |
| TATDQA | 84.04 🥇 |
| Average | 91.08 🥇 |
📊 MTEB v1+v2 (NDCG@5) - Click to expand
ViDoRe v1 Tasks:
| Task | Score |
|---|---|
| ArxivQA | 93.80 🥇 |
| DocVQA | 64.69 |
| InfoVQA | 94.51 |
| ShiftProject | 90.41 |
| SyntheticDocQA-AI | 98.65 |
| SyntheticDocQA-Energy | 96.52 |
| SyntheticDocQA-Gov | 96.79 |
| SyntheticDocQA-Health | 99.26 |
| TabFQuAD | 92.18 |
| TATDQA | 84.04 🥇 |
ViDoRe v2 Tasks (Multilingual):
| Task | Score |
|---|---|
| ViDoRe-v2-2BioMed | 63.26 |
| ViDoRe-v2-2Econ | 57.98 |
| ViDoRe-v2-2ESG-HL | 70.77 |
| ViDoRe-v2-2ESG | 57.85 |
| Combined Average | 82.91 |
📊 ViDoRe v3 (NDCG@10) - Click to expand
| Task | Score |
|---|---|
| ViDoRe-v3-CS | 77.52 🥇 |
| ViDoRe-v3-Energy | 66.32 |
| ViDoRe-v3-FinanceEn | 55.79 |
| ViDoRe-v3-FinanceFr | 45.03 |
| ViDoRe-v3-HR | 59.96 |
| ViDoRe-v3-Industry | 50.39 |
| ViDoRe-v3-Pharma | 63.98 |
| ViDoRe-v3-Physics | 49.36 |
| Average | 58.55 |
Overall Summary (128-dim Models)
| Model | Params | ViDoRe v1 | MTEB v1+v2 | ViDoRe v3 |
|---|---|---|---|---|
| SauerkrautLM-ColQwen3-8b-v0.1 ⭐ | 8.0B | 91.08 (#1) | 82.91 (#2) | 58.55 (#1) |
| EvoQwen2.5-VL-Retriever-7B-v1 | 7.0B | 90.68 (#3) | 83.41 (#1) | - |
| SauerkrautLM-ColQwen3-4b-v0.1 | 4.0B | 90.80 (#2) | 81.97 (#4) | 56.03 (#4) |
| EvoQwen2.5-VL-Retriever-3B-v1 | 3.0B | 90.67 (#4) | 82.76 (#3) | - |
| SauerkrautLM-ColQwen3-2b-v0.1 | 2.2B | 90.24 (#5) | 81.02 (#7) | 54.32 (#5) |
| colnomic-embed-multimodal-7b | 7.0B | 89.72 (#7) | 81.30 (#5) | 57.64 (#2) |
| colnomic-embed-multimodal-3b | 3.0B | 89.86 (#6) | 80.09 (#8) | 56.40 (#3) |
| colqwen2.5-v0.2 | 3.0B | 89.54 (#8) | 81.12 (#6) | 52.44 (#6) |
| colqwen2-v1.0 | 2.2B | 89.23 (#9) | 79.74 (#9) | 44.18 (#8) |
📋 Summary Tables
128-dim Models Comparison
Comparison vs High-dim Models
✨ Key Features
- 🏆 #1 in 128-dim Class: Best ViDoRe v1 and v3 scores among all 128-dim models
- ⚡ Compact Embeddings: 128-dimensional (same as ColPali, 2.5x smaller than tomoro)
- 🌍 Multilingual: Trained on 6 languages (EN, DE, FR, ES, IT, PT)
- 📄 High Resolution: Supports up to 1540 visual tokens per image
- 🔧 MTEB Compatible: Standardized evaluation and easy integration
- 💻 Full Code: github.com/VAGOsolutions/sauerkrautlm-colpali
Model Details
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen3-VL-8B |
| Parameters | 8.0B |
| Embedding Dimension | 128 |
| VRAM (bfloat16) | ~16 GB |
| Max Context Length | 262,144 tokens |
| Image Resolution | Dynamic (up to 1540 visual tokens) |
| Supported Languages | EN, DE, FR, ES, IT, PT |
| License | Apache 2.0 |
Training
Hardware & Configuration
| Setting | Value |
|---|---|
| GPUs | 4x NVIDIA A100 SXM (80GB) |
| Effective Batch Size | 256 |
| Precision | bfloat16 |
| Optimizer | AdamW |
Datasets
| Dataset | Type | Description |
|---|---|---|
| vidore/colpali_train_set | Public | ColPali training data |
| openbmb/VisRAG-Ret-Train-In-domain-data | Public | Visual RAG training data |
| llamaindex/vdr-multilingual-train | Public | Multilingual document retrieval |
| VAGO Multilingual Dataset 1 | In-house | Proprietary multilingual document-query pairs |
| VAGO Multilingual Dataset 2 | In-house | Proprietary multilingual document-query pairs |
Installation & Usage
Sentence Transformers
This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:
pip install "sentence-transformers[image]>=6.0.0"
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1")
queries = [
"What is the variable represented on the y-axis of the graph?",
"Total outlay is maximum in which year?",
]
images = [
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]
query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([25, 128]) torch.Size([1251, 128])
# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[14.9121, 9.1426],
# [ 5.8672, 15.3125]], device='cuda:0')
SauerkrautLM ColPali
⚠️ Important: Install our package first before loading the model:
pip install git+https://github.com/VAGOsolutions/sauerkrautlm-colpali
import torch
from PIL import Image
from sauerkrautlm_colpali.models import ColQwen3, ColQwen3Processor
model_name = "VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1"
model = ColQwen3.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
device_map="cuda:0",
).eval()
processor = ColQwen3Processor.from_pretrained(model_name)
# Process inputs
images = [Image.open("document.png")]
queries = ["What is the main topic?"]
batch_images = processor.process_images(images).to(model.device)
batch_queries = processor.process_queries(queries).to(model.device)
with torch.no_grad():
image_embeddings = model(**batch_images)
query_embeddings = model(**batch_queries)
scores = processor.score(query_embeddings, image_embeddings)
📊 Additional Benchmark Visualizations
MTEB v1+v2 Benchmark (128-dim Models)
ViDoRe v3 Benchmark (128-dim Models)
Our Models vs High-dim Models
Citation
@misc{sauerkrautlm-colpali-2025,
title={SauerkrautLM-ColPali: Multi-Vector Vision Retrieval Models},
author={David Golchinfar},
organization={VAGO Solutions},
year={2025},
url={https://github.com/VAGOsolutions/sauerkrautlm-colpali}
}
Contact
- VAGO Solutions: https://vago-solutions.ai
- GitHub: https://github.com/VAGOsolutions
- Email: info@vago-solutions.ai
- Downloads last month
- 271
from sentence_transformers import SentenceTransformer model = SentenceTransformer("VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3]