MirrorNeuronLab/embeddinggemma-2-GGUF

EmbeddingGemma 2 Multimodal GGUF - Q4_K_M Text + Q8_0 Vision/Audio

Text, image, and audio embeddings in a shared 768-dimensional vector space. This MirrorNeuronLab release pairs a Q4_K_M text backbone with a Q8_0 companion containing both vision and audio encoders.

For image or audio inputs, load both GGUF files. Docker Model Runner automatically downloads the companion when you pull hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M. The Q4_K_M selection identifies the text backbone; the complete release supports text, images, and audio.

Image and audio embedding requests were verified through Docker Model Runner on NVIDIA DGX Spark. The model returns embeddings for search and similarity; it does not generate conversational answers or transcriptions.

Google DeepMind developed the original EmbeddingGemma 2 model. MirrorNeuronLab converted Google's original weights to GGUF and quantized them without additional training.

The text backbone uses standard llama.cpp Q4_K_M, quantized directly from a BF16 intermediate without an importance matrix. This is a MirrorNeuronLab GGUF conversion and quantization. The Q4_K_M preset mixes tensor types as defined by upstream llama.cpp, retaining small tensors and the embedding projection at higher precision where the preset requires it.

The separate Q8_0 vision/audio companion is required for image and audio inputs. It contains both encoders, including their projections. There is one text-model quantization and one multimodal companion in this repository. The Q8_0 companion supplements the Q4_K_M text model; it is not an alternative text model.

Files

File	Purpose	Exact bytes	MiB
embeddinggemma-2-Q4_K_M.gguf	Text backbone, standard Q4_K_M	181,694,688	173.278
mmproj-embeddinggemma-2-Q8_0.gguf	Combined vision/audio companion	554,821,056	529.119

Binary SHA-256 checksums:

d610afe26dd783739275646d51d4d85b4167820fd051600696d7a49e6f68609f  embeddinggemma-2-Q4_K_M.gguf
0531309472a6bc6f4b146c3736eee7795ec9db1903c25490d3c937490cef6dd8  mmproj-embeddinggemma-2-Q8_0.gguf

SHA256SUMS covers the release files. SOURCE_SHA256SUMS, PROVENANCE.json, GGUF_AUDIT.json, VALIDATION.json, and TIMINGS.json document the original checkpoint, conversion, resulting tensor inventory, and checks. The BF16 intermediate is kept outside the release directory.

Source and tools

• Google checkpoint: google/embeddinggemma-2 at f9972ac22e696a105c482202646979e33e4b5a1e.
• Original model.safetensors SHA-256: 197a32965d4b1105faf060417baa899e193fb73cd401f42ec9295234d5553d79.
• Conversion and quantization: ggml-org/llama.cpp at dde879bc4f954b809892fd0d660515f42536394a, build b12032. The upstream checkout is unmodified.
• Python 3.12.3; PyTorch 2.11.0+cpu; Transformers 4.57.6; Hugging Face Hub 0.36.2. Hub and Transformers were resolved together under llama.cpp's constraints; pip check passed. The full environment lock is included under reproduce/logs/.

Google's tokenizer config uses a newer Transformers schema. A small compatibility adapter maps its special-token configuration to the pinned Transformers 4 format in memory. It verifies that all 262,144 token strings and IDs match the original tokenizer.json. Original source files remain byte-for-byte unchanged. GGUF auditing confirms preserved tokenizer metadata, noncausal architecture, mean pooling, the 512-to-768 embedding projection, and matching tensor shapes between BF16 and Q4_K_M.

The AtomicChat model card was consulted for packaging and usage conventions. This release uses the standard Q4_K_M preset, with no AtomicChat weights, importance matrix, dynamic tensor layout, or benchmark results. It is not labeled AD-Q4_K_M.

Docker Model Runner

Use an up-to-date llama.cpp engine with gemma-embedding2 and Gemma4 vision/audio support. The Docker Model Runner controller version alone does not establish engine compatibility: older bundled engines report an unknown architecture. The files are portable to Apple Silicon Macs and Ubuntu/NVIDIA systems with a supporting backend. Those platforms were not physically tested in this run.

Pull the Q4_K_M text release; Docker Model Runner also selects the sole multimodal companion automatically:

docker model pull hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M
docker model configure --mode embedding --context-size 2048 \
  hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M \
  -- --batch-size 2048 --ubatch-size 2048 --pooling mean
docker model run --detach hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M

This is an embedding model. Use the embeddings API to obtain vectors after loading it:

curl http://localhost:12434/engines/llama.cpp/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"hf.co/MirrorNeuronLab/embeddinggemma-2-GGUF:Q4_K_M","input":["task: search result | query: What causes the northern lights?","title: none | text: Charged particles from the Sun interact with the atmosphere."],"encoding_format":"float"}'

For local files, explicitly package both GGUFs and the license. Run this from the downloaded release directory:

docker model package \
  --gguf "$PWD/embeddinggemma-2-Q4_K_M.gguf" \
  --mmproj "$PWD/mmproj-embeddinggemma-2-Q8_0.gguf" \
  --license "$PWD/LICENSE" --context-size 2048 \
  mirrorneuronlab/embeddinggemma-2:q4_k_m
docker model configure --mode embedding --context-size 2048 \
  mirrorneuronlab/embeddinggemma-2:q4_k_m \
  -- --batch-size 2048 --ubatch-size 2048 --pooling mean
docker model run --detach mirrorneuronlab/embeddinggemma-2:q4_k_m

Use mirrorneuronlab/embeddinggemma-2:q4_k_m as the API model name for a locally packaged artifact. Packaging keeps the original GGUF files on disk. See Docker packaging and the Docker Model Runner API.

llama.cpp and multimodal inputs

llama-server -m embeddinggemma-2-Q4_K_M.gguf \
  --mmproj mmproj-embeddinggemma-2-Q8_0.gguf \
  --embeddings --pooling mean -c 2048 -b 2048 -ub 2048 -ngl 99

For text-only use, omit --mmproj. GPU offloading uses the backend available on the machine, such as Metal on Apple Silicon or CUDA on NVIDIA systems.

The native output is a normalized 768-dimensional embedding. Use task prefixes such as task: search result | query: {query} and title: {title or none} | text: {document}. Index documents and query them with the same model file. For 512-, 256-, or 128-dimensional Matryoshka vectors, retain the first dimensions and normalize again.

For image/audio embeddings, send OpenAI-style content arrays as elements of input. Examples of the two element forms:

{"content":[{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}]}
{"content":[{"type":"input_audio","input_audio":{"data":"<base64 WAV>","format":"wav"}}]}

Set encoding_format to float. Apply task prefixes to text inputs. Images and audio do not need a text prefix.

EmbeddingGemma 2 uses bidirectional attention: the whole input must fit in one microbatch. Size context (-c), batch (-b), and microbatch (-ub) together for the longest input. The validation used 2048 for all three. Google's published context limit is 8192; the source config and preserved GGUF metadata expose a larger positional limit of 262144, which this release does not establish as a supported input length. For 8K inputs, raise all three settings to 8192 in a supporting engine. Small microbatches can also reduce the engine's image token budget.

Measured validation

Validated on NVIDIA DGX Spark, native Linux ARM64, NVIDIA GB10, CUDA 13.0, through Docker Model Runner v1.2.9 and the pinned upstream CUDA llama.cpp backend above. The official ARM64 CUDA runner image is pinned in PROVENANCE.json; its stock older engine was replaced for this isolated validation by a read-only mount of the newly compiled backend. The existing system runner was left intact. The controller reports the backend version as unknown because its parser does not recognize the new upstream version string; the executed binary and full commit are recorded.

docker model package and docker model run --detach succeeded. Text, image, audio, and mixed text/image/audio batches returned HTTP 200 with finite, L2-normalized 768-dimensional vectors. GPU detection and NVIDIA-SMI confirmed CUDA use. The four-text repeat check had maximum absolute difference 0.0.

On the small text sanity fixture, query-to-relevant-passage cosine was 0.865082, versus 0.548627 for an unrelated passage. Cosines to a BF16 CPU reference on four texts ranged from 0.988963 to 0.997227. These measurements include quantization and CPU/CUDA execution differences.

These are functional smoke checks on four texts, a solid-color image, and a synthetic two-second tone. They do not measure retrieval accuracy or image/audio semantic quality. Video was not tested. llama.cpp marks audio input experimental. No MTEB, MIEB, MMEB, or audio benchmark score is claimed for this conversion.

The published Hugging Face release was also pulled through Docker Model Runner using :Q4_K_M. docker model run --detach succeeded, and text, image, audio, and mixed-batch embedding checks passed using the automatically selected Q8_0 companion. See PUBLISHED_VALIDATION.json for the published artifact revision and results.

Reproduce

The conversion commands are:

python scripts/convert_compatible.py source --outtype bf16 \
  --model-name embeddinggemma-2 --outfile work/embeddinggemma-2-BF16.gguf
./llama.cpp/build/bin/llama-quantize work/embeddinggemma-2-BF16.gguf \
  release/embeddinggemma-2-Q4_K_M.gguf Q4_K_M
python llama.cpp/convert_hf_to_gguf.py source --mmproj --outtype q8_0 \
  --model-name embeddinggemma-2 --outfile release/mmproj-embeddinggemma-2-Q8_0.gguf

For a fresh Linux ARM64 workspace, run the bundled script with a new destination path:

bash reproduce/scripts/reproduce.sh /absolute/path/to/new-workspace

It downloads the pinned original source, installs the locked converter dependencies, checks out the full llama.cpp revision, builds native tools, converts, quantizes, and audits. Its dependency lock records the tested ARM64 CPU converter environment. For other operating systems, resolve platform wheels while retaining llama.cpp's package constraints. CUDA validation setup is supplied in reproduce/scripts/start_validation_runner.sh; build the backend first with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 for GB10. This GPU target applies to the tested backend build, not to the portable GGUF files.

Measured elapsed seconds for the successful stages:

Stage	Seconds
Original checkpoint download	31.993
Python dependencies	25.619
CPU build (quantizer and reference server)	59.664
CUDA build	222.716
BF16 text conversion	12.600
Q4_K_M quantization	2.439
Q8_0 vision/audio conversion	3.547
GGUF audit	21.716
BF16 reference validation	2.206
Docker packaging	2.361
Docker CUDA embedding validation	2.226
docker model run --detach	0.051

Stages ran partly in parallel, so these durations should not be added as an end-to-end runtime. TIMINGS.json records start/end times, commands, configure stages, and attempts. The first unadapted text-conversion attempt failed on the tokenizer schema before the successful compatible conversion.

License and attribution

Apache-2.0. Original model: Google DeepMind. GGUF conversion and quantization: MirrorNeuronLab. See LICENSE and the original model card for intended uses and limitations.