Integrate with Sentence Transformers via MultiVectorEncoder
Hello!
Sister PR of https://huggingface.co/VAGOsolutions/SauerkrautLM-ColQwen3-8b-v0.1/discussions/1, applying the same Sentence Transformers integration to VAGOsolutions/SauerkrautLM-ColQwen3-1.7b-Turbo-v0.1. That PR carries the details and the verification against sauerkrautlm-colpali. The integration is identical: like the 8b, this checkpoint declares the custom colqwen3 model type, so sentence_bert_config.json remaps it to qwen3_vl at load time.
On a 400-query NanoViDoRe v3 evaluation this integration scores 0.5035 nDCG@10, close to the 0.4905 reported for this model on the ViDoRe v3 board (our evaluation is a 400-query subset, so the two are not measured identically).
Heads up, this PR was AI-generated and human-reviewed.
pip install "sentence-transformers[image]>=6.0.0"
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-ColQwen3-1.7b-Turbo-v0.1")
queries = [
"What is the variable represented on the y-axis of the graph?",
"Total outlay is maximum in which year?",
]
images = [
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]
query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([25, 128]) torch.Size([1251, 128])
# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[14.5811, 11.6055],
# [ 7.1665, 14.7393]], device='cuda:0')
To try this before merging, pass revision="refs/pr/1" to MultiVectorEncoder.
Happy to tweak anything you'd like changed. Please let me know if you have any questions or feedback!
- Tom Aarsen