nomic-ai

/

nomic-embed-vision-v1.5

like 217

Follow

Nomic AI 1.08k

Hello!

Pull Request overview

• Integrate nomic-embed-vision-v1.5 with Sentence Transformers v5.4+

Details

The integration uses a Transformer -> Pooling(cls) -> Normalize pipeline with modality_config set to {"image": {"method": "forward", "method_output_name": "last_hidden_state"}} so the model accepts image inputs. A processor_config.json was added to ensure AutoProcessor loads the CLIPImageProcessor instead of falling back to a tokenizer (since model_type: nomic_bert would otherwise resolve to BertTokenizer).

Note: this model requires https://huggingface.co/nomic-ai/nomic-bert-2048/discussions/23 to fix three transformers v5 compatibility issues in the shared modeling code:

1. Adding self.post_init() to NomicVisionModel.__init__ (required for all_tied_weights_keys)
2. Lazy recomputation of rotary position embeddings in NomicVisionRotaryEmbeddingCat.get_embed (non-persistent buffers are not materialized when from_pretrained initializes on torch.device("meta") in v5)
3. Replacing self.norm_factor buffer with inline math.sqrt(self.head_dim) in NomicAttentionPooling and NomicBertAttention (same meta-device issue)

Added files:

• modules.json: Defines the Transformer -> Pooling -> Normalize pipeline
• config_sentence_transformers.json: ST model config with cosine similarity
• sentence_bert_config.json: Transformer config with image modality_config
• 1_Pooling/config.json: CLS pooling mode, 768-dim embeddings
• processor_config.json: Ensures AutoProcessor loads CLIPImageProcessor

Modified files:

• config.json: Fixed n_inner from 2048.0 (float) to 2048 (int) for transformers v5 strict validation
• README.md: Added sentence-transformers library tag, and a "Using Sentence Transformers" usage section

Here's a script that uses both this PR and the companion PR:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-vision-v1.5", revision="refs/pr/10", model_kwargs={"code_revision": "refs/pr/23"}, trust_remote_code=True)

embeddings = model.encode("http://images.cocodataset.org/val2017/000000039769.jpg")
print(embeddings.shape)
# (768,)

Once both are merged, then the revision/model_kwargs can be excluded.

Note that none of the old behaviour is affected/changed. It only adds an additional way to run this model in a familiar and common format.

• Tom Aarsen