nyaaorick/Qwen3.8-2B-q4f16_1-MLC

Qwen3.8-2B-Distill — MLC / WebGPU (q4f16_1)

An MLC compilation of empero-ai/Qwen3.8-2B-Distill for WebGPU in the browser, quantized to q4f16_1.

Text path only: the vision tower and the MTP head are stripped, since neither is used on the WebGPU path. 632 tensors → 320, 1,881,825,088 parameters, 1.06 GB at 4.503 bits/param.

⚠️ Read this before using the included .wasm on Firefox

Firefox's Metal backend caps maxStorageBuffersPerShaderStage at 9. This library contains four kernels that bind 10, which is normal for MLC builds — the stock Qwen3.5-0.8B-q4f16_1-MLC has exactly the same four. MLC registers them unconditionally as a fixed positional tuple handed to the C++ PagedKVCache, so they cannot be omitted without breaking the runtime's indexing.

Three are unreachable by configuration (sliding_window_size is -1, speculative decoding is never invoked). The fourth, batch_prefill_paged_kv_kernel, is reachable — it runs whenever the engine reuses a KV cache across turns.

A WebGPU pipeline that fails to create is silent. Its dispatches become no-ops, so the model emits garbage rather than raising an error. Measured on Firefox 154: with cross-turn KV reuse the same history and temperature: 0 produced a different, corrupted continuation each run.

If you run this on Firefox, force a full re-prefill instead of reusing the KV cache — for example call resetChat() before every prefill when adapter.limits.maxStorageBuffersPerShaderStage < 10. Chrome is unaffected and keeps KV reuse.

The cost of that workaround is measured at ~5 ms per history token re-prefilled.

Measured

M4 MacBook Air (16 GB), Firefox 154, macOS:

Decode	16.6-18.1 tok/s
Prefill	48 tok/s on a short prompt, 100-200 tok/s at length
Model load from cache	~51 s
Resident per engine	~2.4 GB

Configuration choices baked into mlc-chat-config.json

These are deliberate; change them if they do not suit you.

• Thinking is off — conv_template is qwen3_5_nothink, so the assistant turn opens with an already-closed <think></think> block. The base model is a reasoning distill and opens every reply with a think block otherwise, which costs seconds of output before the answer starts. Use qwen3_5 to turn it back on.
• context_window_size: 4096 — the checkpoint declares 262144, and the paged KV cache preallocates from this value.
• temperature: 0.6, top_p: 0.95 — the base model card's recommendation. It also warns that decoding too close to greedy is a repetition-loop failure mode for this class of model.
• Stop tokens [248046, 248044] (<|im_end|>, <|endoftext|>). Note that MLC's qwen2 template, which some Qwen3.5 builds are generated with, carries Qwen2 ids that do not exist in this 248320-token vocabulary.

Contents

Unusually for an MLC repo, the model library .wasm ships in this folder rather than in mlc-ai/binary-mlc-llm-libs, so the folder is self-contained and can be dropped straight into a browser cache.

mlc-chat-config.json
tensor-cache.json, tensor-cache-b16.json
params_shard_0.bin … params_shard_25.bin      (26 shards, 1.06 GB)
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt
Qwen3.8-2B-q4f16_1_cs1k-webgpu.wasm           (6.9 MB)

cs1k = prefill_chunk_size 1024.

Build provenance

Compiled with mlc-llm-nightly-cpu 0.26.dev6 against mlc-ai-nightly-cpu 0.26.dev246, with the TVM web runtime built from apache/tvm@837cb9de1 — the commit the wheel itself records.

Those two nightlies do not work together out of the box: mlc-llm 0.26.dev6 is written against an unreleased TVM in which PrimFunc.params holds Buffers, while every published mlc-ai wheel still uses params: Array<Var> plus a buffer_map. That, and four other breakages between pip install and a loadable .wasm, had to be worked around. If you are reproducing this, the import list of the produced .wasm must not exceed that of a build known to load — a runtime newer than the JS glue fails only at model load, after compiling and shipping cleanly.

License

Apache-2.0, inherited from the base model. Attribution: empero-ai/Qwen3.8-2B-Distill, itself a distillation into the Qwen/Qwen3.5-2B architecture.