Cyronius/Granite-4.2-3B-Uno-NPU2

Granite 4.2 3B + Uno drafter for the AMD XDNA 2 NPU

ibm-granite/granite-4.2-3b in OpenFlowLM's Q4NX format (q4_1), with its Uno adapter (uno.q4nx) and a prebuilt open kernel set. With all three, OpenFlowLM decodes greedy requests by Uno's two-pass cycle: one pass drafts 4 tokens, a second verifies them with the unmodified base. The output is identical to plain greedy decoding.

Needs an OpenFlowLM-Next build with PR #184 and PR #192 until they are merged. Without uno.q4nx, the same weights run as plain Granite.

Speed on the NPU

q4_1, 4 rows a pass, six prompts × 128 tokens, quiet machine (Ryzen AI HX 370).

prompt	tokens a cycle	Uno	plain decode	speedup
chat	2.42	14.77 tok/s	20.16 tok/s	0.73x
chat2	2.61	16.05	20.06	0.80x
code	3.20	19.60	19.72	0.99x
code2	3.20	19.62	20.24	0.97x
math	3.76	22.99	20.04	1.15x
reason	3.46	21.15	20.13	1.05x

All six are identical to plain decode. Geometric mean: 0.94x. A cycle costs about 3.2 decode steps and this adapter commits about 3 tokens a cycle at 4 rows, so Uno only wins on math and reasoning today. It should pay once a pass gets cheaper or 8-row passes run (3.53 tokens a cycle on the CPU reference).

Run

1. Download into a models directory:

   hf download Cyronius/Granite-4.2-3B-Uno-NPU2 --local-dir <root>/models/Granite-4.2-3B-Uno-NPU2
   

2. Add an entry under "models" in a copy of OpenFlowLM's model_list.json:

   "granite4.2": {"3b": {
     "name": "Granite-4.2-3B-Uno-NPU2", "url": "", "file_url": "", "ms_url": "",
     "modified_at": "2026-10-10T00:00:00Z", "oflm_min_version": "0.9.22",
     "files": ["config.json", "model.q4nx", "tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "uno.q4nx"],
     "default_context_length": 8192, "max_prefill_len": 4096,
     "details": {"family": "granite", "think": false, "think_toggleable": false,
                 "parameter_size": "3B", "quantization_level": "Q4_1"},
     "label": [], "footprint": 3.0}}
   

3. Point oflm at both and run:

   set OFLM_CONFIG_PATH=<your model_list.json>
   set OFLM_MODEL_PATH=<root>
   oflm run granite4.2:3b
   

   The kernels in open_kernels/ are found beside the model. Uno only runs greedy, and Granite samples by default, so type /set topk 1 and /set rep-pen 1 first. Over oflm serve, send "top_k": 1, "repetition_penalty": 1.

oflm add doesn't copy uno.q4nx or open_kernels/ yet, so use the steps above.

Files

File	Description
model.q4nx	Granite 4.2 3B, q4_1, q_proj pre-scaled by 0.125 (the fold config.json records)
uno.q4nx	The Uno adapter: rank-128 LoRA on all seven projections of the 40 layers, applied to draft rows only; noise ids [1, 100256)
open_kernels/	The NPU kernel set (decode, prefill, the 4-row pass and its LoRA variant, the LM heads), built from OpenFlowLM-Next's open kernel recipes
config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja	Runtime config, tokenizer and Granite's chat template

Reproduce

q4nx-build -i granite-4.2-3b-bf16.gguf -s <granite-4.2-3b dir> -f granite -t language --quant Q4_1 \
  --uno-adapter <granite-4.2-3b-uno adapter dir> --uno-noise-high 100256 -o Granite-4.2-3B-Uno-NPU2
python open_kernels/export_qwen36_kernels.py --model-dir Granite-4.2-3B-Uno-NPU2

The GGUF is ibm-granite/granite-4.2-3b-GGUF's bf16 file. The procedure and checks are requirement OPEN-FAMILY-GRANITE-UNO in OpenFlowLM-Next's specs/open-engine/spec.md.

Credits

• Base model: IBM Granite 4.2 3B (Apache-2.0).
• Method: Uno by IFM (Apache-2.0, arXiv 2609.04010). The adapter was trained on OpenThoughts3 and Tülu 3 with AMD Developer Cloud compute.
• Runtime: OpenFlowLM-Next.