Integrate with Sentence Transformers via MultiVectorEncoder
Hello!
Sister PR of https://huggingface.co/VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1/discussions/1, applying the same Sentence Transformers integration to VAGOsolutions/SauerkrautLM-ColQwen3-4b-v0.1. That PR carries the details and the verification against sauerkrautlm-colpali. Two differences: this checkpoint's config.json already declares qwen3_vl, so no model_type remap is needed, and since it declares float32 the snippet passes model_kwargs={"dtype": "bfloat16"} to match the model card.
On a 400-query NanoViDoRe v3 evaluation this integration scores 0.5656 nDCG@10, close to the 0.5649 reported for this model on the ViDoRe v3 board (our evaluation is a 400-query subset, so the two are not measured identically).
Heads up, this PR was AI-generated and human-reviewed.
pip install "sentence-transformers[image]>=6.0.0"
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder(
"VAGOsolutions/SauerkrautLM-ColQwen3-4b-v0.1",
model_kwargs={"dtype": "bfloat16"},
)
queries = [
"What is the variable represented on the y-axis of the graph?",
"Total outlay is maximum in which year?",
]
images = [
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]
query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([25, 128]) torch.Size([1251, 128])
# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[16.3877, 8.1367],
# [ 5.8350, 15.3848]], device='cuda:0')
To try this before merging, pass revision="refs/pr/1" to MultiVectorEncoder.
Happy to tweak anything you'd like changed. Please let me know if you have any questions or feedback!
- Tom Aarsen