VAGOsolutions

/

SauerkrautLM-ColMinistral3-3b-v0.1

like 3

Follow

VAGO solutions 189

Hello!

I recently released the new MultiVectorEncoder class in Sentence Transformers v6.0, and I think the SauerkrautLM Col* models would be great fits for it. You can read more about the release here: https://huggingface.co/blog/multi-vector-encoder. I would also love to feature this model in the accompanying documentation. I am opening matching PRs for the four ColQwen3 checkpoints and ColLFM2, which use different backbones and so have their own descriptions.

Heads up, this PR was AI-generated and human-reviewed. Here's a summary of the changes as reported by my agent:

Pull Request overview

• Integrate VAGOsolutions/SauerkrautLM-ColMinistral3-3b-v0.1 with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via MultiVectorEncoder.
• Add a Sentence Transformers usage section to the model card, plus the sentence-transformers tag.

Details

The integration is config-only: no new modeling code, and the weights are untouched. The checkpoint already uses the native mistral3 model type, so the base model loads directly, with a key mapping in sentence_bert_config.json stripping the model. prefix from the checkpoint keys. The module pipeline Transformer -> Dense -> Normalize -> MultiVectorMask mirrors the reference forward step for step, where 1_Dense/ carries the custom_text_proj projection (3072 -> 128, no bias) extracted unchanged from the checkpoint. A named chat template in additional_chat_templates/sentence_transformers.jinja reproduces the reference prompt formats faithfully: image documents render through the repository's own chat template, including the full Ministral reasoning system prompt the model was trained with and the doubled leading <s> your pipeline produces (the template text carries one and the processor call adds another), and text queries append the ten </s> augmentation tokens. Token ids from this path are bit-identical to ColMinistral3Processor.from_pretrained output on both queries and documents. processor_config.json now names the stock PixtralProcessor so that AutoProcessor resolves without colpali-engine installed, and tokenizer_config.json gains padding_side: left, matching the reference processor.

Verified against the reference model class on the exact batches Sentence Transformers produces: the per-token embeddings match with cosine similarity 1.0000 and identical MaxSim scores. One measurement note: Pixtral's vision tower concatenates the images of a batch, which makes bfloat16 outputs slightly sensitive to batch row order. Comparisons across differently ordered batches show up to 0.006 per-token cosine deviation on documents. That is a property of the architecture rather than of this integration, so the identical-batch comparison above is the meaningful check. The tokenizer also triggers the familiar fix_mistral_regex warning, an upstream tokenizer quirk that affects all loaders equally. For reference, on a 400-query NanoViDoRe v3 evaluation this integration scores 0.3978 nDCG@10, close to the 0.415 reported for this model on the ViDoRe v3 board (our evaluation is a 400-query subset, so the two are not measured identically).

pip install "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-ColMinistral3-3b-v0.1")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([24, 128]) torch.Size([427, 128])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[24.0039, 24.0000],
#         [19.2227, 19.5508]], device='cuda:0')

To try this before merging, pass revision="refs/pr/1" to MultiVectorEncoder.

Happy to tweak anything you'd like changed. Please let me know if you have any questions or feedback!

• Tom Aarsen