silv4souz4/qwen3-embedding-0.6b-coreai-w8a16

Model Card for Qwen3-Embedding-0.6B — Apple Core AI, int8 weights / fp16 activations (W8A16)

Format conversion of Qwen/Qwen3-Embedding-0.6B to Apple Core AI (.aimodel) with 8-bit integer weights and fp16 activations, for on-device text embeddings on Apple silicon. Tested on macOS 27 (Apple M5) with the Core AI Python runtime; not tested on iOS.

Model Details

Model Description

Qwen3-Embedding-0.6B turns a text (up to 32k tokens in the original) into a 1024-dimensional vector for retrieval, semantic search, clustering and classification, in 100+ languages. This repository contains it as one Core AI asset with two fixed-length functions (128 and 512 tokens) that share the weights; pooling (last token) and L2 normalization run inside the model, so the output can be compared with a dot product.

• Original model developed by: Qwen team, Alibaba (Zhang et al., 2025).
• Conversion by: silv4souz4 (independent; not affiliated with or endorsed by Alibaba/Qwen or Apple).
• Model type: text embedding model (Qwen3 decoder, 28 layers, 0.6 B parameters, 1024-dim output).
• Base model: Qwen/Qwen3-Embedding-0.6B.
• Relation to base: quantized (int8 weights); architecture and math unchanged.
• License: Apache License 2.0, inherited from Qwen3-Embedding.

Model Sources

• Original model: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
• Paper: https://arxiv.org/abs/2506.05176

Uses

Direct Use

On-device semantic search and retrieval: embed documents once, embed each query with its instruction, rank by dot product.

Downstream Use

Retrieval-augmented generation on device, duplicate detection, clustering, tagging, first stage before a reranker (for example qwen3-reranker-0.6b-coreai-w8a16).

Out-of-Scope Use

• Texts longer than 512 tokens without chunking (only 128 and 512 lengths are exported).
• Decisions about people (hiring, credit, law enforcement) based on similarity scores alone; surveillance or profiling.
• Any use that violates applicable law.

Bias, Risks, and Limitations

• Inherits the limitations and biases of Qwen3-Embedding and its training data; quality varies across languages and domains.
• Int8 weights shift the vectors slightly (cosine ≥ 0.998 to the fp32 original on our check); close ties in a ranking can swap.
• Our check is small (8 texts); validate on your own data before relying on it. fp16 numerics vary across compute units and OS releases. iOS is untested.
• Debug information embedded in the asset contains build-time file paths of the converting machine; it has no effect on inference.

Recommendations

Use the query instruction format below; chunk long documents; re-check retrieval metrics on your domain.

How to Get Started with the Model

Download the whole repository (asset, tokenizer, config.json and checksums):

hf download silv4souz4/qwen3-embedding-0.6b-coreai-w8a16 --local-dir qwen3-embedding-0.6b-coreai-w8a16

config.json is a machine-readable summary of the functions, tensor shapes, tokenization and quantization below (it is also the file the Hub uses to count downloads of this repository).

File	Size	Functions
qwen3_embedding_0.6b_w8a16.aimodel	599 MB	qwen3_emb_L128, qwen3_emb_L512
tokenizer.json (+ tokenizer_config.json, vocab.json, merges.txt)	16 MB	original Qwen3 tokenizer

Function	Inputs	Output
qwen3_emb_L128	input_ids int32 [1,128], attention_mask int32 [1,128]	embedding float16 [1,1024], L2-normalized
qwen3_emb_L512	input_ids int32 [1,512], attention_mask int32 [1,512]	embedding float16 [1,1024], L2-normalized

• Text format (as in the original card): queries are Instruct: {task}\nQuery:{query} (for example task = "Given a web search query, retrieve relevant passages that answer the query"); documents are plain text.
• Tokenization: tokenizer.json appends <|endoftext|> (151643). Pad on the left with 151643 up to 128 or 512, attention_mask 1 on real tokens and 0 on padding. Pick the smallest function that fits.
• Similarity: dot product of the two embeddings (they are already normalized).

import numpy as np
from tokenizers import Tokenizer
from coreai.runtime import AIModel, NDArray, SpecializationOptions
tok = Tokenizer.from_file("tokenizer.json")
m = await AIModel.load("qwen3_embedding_0.6b_w8a16.aimodel", SpecializationOptions.default())
f = m.load_function("qwen3_emb_L128")
def embed(text, L=128):
    ids = tok.encode(text).ids[-L:]
    x = np.full((1, L), 151643, np.int32); a = np.zeros((1, L), np.int32)
    x[0, L - len(ids):] = ids; a[0, L - len(ids):] = 1
    return (await f({"input_ids": NDArray(x), "attention_mask": NDArray(a)}))["embedding"].numpy()[0].astype(np.float32)

Ahead-of-time compilation (.aimodelc) — tutorial

The asset here is a portable .aimodel. Apple's documentation, Compiling Core AI models ahead of time, explains that every Core AI model must be specialized to the device before inference; the coreai-build tool moves the most expensive part of that specialization, model compilation, to your build machine, so the model loads faster in your app. It produces one .aimodelc per device architecture; you ship (or download) only the one that matches the device. According to Apple, AOT compilation targets devices that support Apple Intelligence (iPhone/iPad with A17 Pro or later, Mac with M1 or later, Apple Vision Pro); it is not supported on tvOS and watchOS.

1. Install the Metal Toolchain (Xcode > Settings > Components > Metal Toolchain, or):

xcodebuild -downloadComponent MetalToolchain

2. Compile (all architectures of a platform, or one with --architecture):

xcrun coreai-build compile qwen3_embedding_0.6b_w8a16.aimodel --platform macOS --min-deployment-version 27.0 --output compiled/
xcrun coreai-build compile qwen3_embedding_0.6b_w8a16.aimodel --platform iOS --min-deployment-version 27.0 --output compiled/

Output files are named <asset>.<arch>.aimodelc (for example .h17g for the Apple M5 used here); each works on any OS at or above --min-deployment-version. Run coreai-build compile --help for --preferred-compute and the other options.

3. Load the matching bundle — the same API as for .aimodel:

let arch = AIModel.deviceArchitectureName
let url = compiledDir.appending(path: "qwen3_embedding_0.6b_w8a16.\(arch).aimodelc")
let model = try await AIModel(contentsOf: url, options: .default)

Apple recommends hosting the per-architecture bundles and downloading only the matching one (for example with the BackgroundAssets framework). Some on-device specialization still remains after AOT, depending on the model and compute units.

What we measured (Apple M5, macOS 27.2, Swift runtime, qwen3_emb_L512, first load with an empty specialization cache, two rounds):

Step	Portable .aimodel	AOT .aimodelc
Compile on the Mac (coreai-build, h17g)	—	11.8 s
First load (model + function), cold cache	17–25 s	2.9–3.6 s
Later loads (cache warm)	0.9–1.2 s	1.2 s

AOT changes load time, not the math (on our check the M5 build gave the same cosines and rankings as the portable asset). It is not an inference speed-up. For this model the cold load of the portable asset is long, so shipping the .aimodelc is worth it; the cache is also rebuilt after every OS update.

Training Details

Training Data

No training or fine-tuning. Weights are the official Qwen3-Embedding-0.6B, trained by the Qwen team as described in the paper.

Training Procedure (conversion)

1. Qwen3 decoder re-authored in PyTorch for Core AI with last-token pooling and L2 normalization inside the graph and fixed lengths 128/512; the fp32 re-authoring reproduced transformers with cosine 1.0000.
2. Recipe: int8 symmetric weights per output channel on every nn.Linear (coreai-opt presets.w8), embedding table int8 per row; fp16 activations, with RMSNorm and the final L2 normalization scaled by the maximum so they cannot overflow in fp16.
3. Converted with coreai-torch 0.4.3 / coreai-opt 0.3.0 (PyTorch 2.13) to one portable .aimodel with two functions.

Evaluation

Testing Data, Factors & Metrics

8 texts: 4 queries with the instruction prefix (the two examples of the original card plus two in Portuguese) and 4 documents. Reference: the original model in transformers, fp32, same tokenization and pooling. Metrics: minimum cosine between our embedding and the reference over the 8 texts, and whether the query→document ranking is identical. Apple M5, Core AI runtime.

Results

Length	Min. cosine vs fp32	Ranking identical
128	0.9981	yes
512	0.9981	yes

Same numbers with the M5 AOT build. Descriptive timing (Python, uncontrolled machine): ~22 ms at 128 tokens, ~55 ms at 512.

Summary

Vectors within 0.002 cosine of the fp32 original and identical rankings on this small check, at 599 MB.

Environmental Impact

No training. Conversion and evaluation ran locally on one Apple M5 laptop; negligible.

Technical Specifications

Compute Infrastructure

• Hardware: Apple M5 (Mac17,3), 24 GB unified memory.
• Software: macOS 27.2, PyTorch 2.13, coreai-torch 0.4.3, coreai-opt 0.3.0, coreai-core 1.0.0b3, coreai-build 3605.5.4.

Citation

If you use these assets, please cite Qwen3 Embedding:

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}

License and Attribution

• License: Apache License 2.0 — full text in LICENSE, the license of Qwen3-Embedding-0.6B.
• Original model: © Alibaba Cloud / Qwen team.
• Modifications (Apache 2.0 §4(b)): re-authoring for Core AI with fixed lengths and in-graph pooling, int8 weight quantization, conversion to the Core AI format; tokenizer files unchanged; no retraining. See NOTICE.
• "Apple" and "Core AI" are trademarks of Apple Inc.; "Qwen" is a trademark of Alibaba; their use here is descriptive only.

Model Card Authors

silv4souz4

Model Card Contact

Please open a discussion in the Community tab of this repository.