README.md

metadata

license: apache-2.0
base_model: Qwen/Qwen3-4B
tags:
  - lora
  - gguf
  - tool-calling
  - function-calling
  - reproducible-training
  - xyntetik-runner

Qwen3-4B-Runner-ToolUse-Q4_K_M

Release: built to be downloaded and served with Xyntetik Runner.

• Measured status against the parent, limit first: off its training distribution this adapter is no better than the base. On 274 hand-written prompts over a 23-tool catalog, the base made 211 exact calls through the chat API and this adapter 200 (the shifted set). On its own task, held-out exact tool-call score 0.690 for the base Q4_K_M, 1.000 with this adapter (29 prompts, greedy); byte-reproducible training with a provenance record.
• Parent model: Qwen/Qwen3-4B at Q4_K_M
• Collection: Runner Releases, also in LoRA Training on Quantized Weights
• Evidence: training record, evals and the merge study (dataset)
• Runner compatibility: --train, --lora and --merge-lora in Runner; merge to F16 or Q8_0, never to 4-bit (measured erasure). Every adapter feature, from continued training to per-request serving and signing: the complete adapter guide.
• Validation status (Runner support tiers): each file's row in the Runner roster says which tier it is in, what is re-checked on a release and the date and version of the last run.
  • merged-Q8_0.gguf: roster row
  • Qwen3-4B-ToolUse-LoRA.gguf: roster row

Want this for your own data? The adapter was trained through the 4-bit file it is served with; the training walkthrough has the commands, and the complete adapter guide explains every adapter feature.

Request a model · Report your result · Community results · All files

If this file is useful to you, a contribution funds the hardware time behind the next one.

What this is, and is not. A LoRA adapter for one narrow, synthetic tool-calling task (four tools, 158 generated training examples, 29 held-out prompts), published to show the training mechanism and its reproducibility. It is not a general tool-use upgrade for Qwen3-4B, and the training recipe (316 steps at lr 1e-4 on 158 examples) is memorization-grade, right for a format habit and wrong for a knowledge task. Measured off its task, it does not help: on 274 hand-written prompts unlike its training, over 23 tools with overlapping names, the base makes more exact calls through the chat API than the adapter (211 to 200), and the adapter helps only where no tool fits (the shifted set).

This LoRA was trained directly through the quantized GGUF that serves it. No FP16 training copy. No Python training framework. The trainer's forward pass is the inference forward pass: Xyntetik Runner (a single-binary C11 inference engine) trained this adapter against the frozen Q4_K_M inference artifact on CPU, and the result is byte-reproducible: run the same command on the same inputs with the same build and you get this exact file, sha for sha. "Same build" is part of the claim: an independent tester reproduced the bytes on a Tesla T4 and also found that a binary rebuilt under a different instruction-set profile writes different adapter bytes, so the reproduction is a property of the build, the way Runner's receipt replay is.

What the mechanism covers, what it refuses and what it costs per step: the manual's adaptation section.

TL;DR

Off its task the adapter does not help: 274 hand-written prompts over 23 tools, exact calls 211 for the base and 200 with the adapter through the chat API (170 and 176 on the training template, inside noise; the shifted set). On its task:

	json parses	right tool	schema args	exact call
Qwen3-4B Q4_K_M, base	1.000	0.724	0.724	0.690
+ this adapter	1.000	1.000	1.000	1.000

Held-out tool-calling eval (29 prompts, greedy, temperature 0): pick the right tool among four schema-distinct tools, emit schema-shaped JSON args, refuse requests no available tool fits, no prose around the JSON.

Reproducibility record (measured, not asserted)

Everything below is from the machine-written provenance record (Qwen3-4B-ToolUse-LoRA.gguf.train.json, committed beside the adapter):

	sha256
base (Qwen_Qwen3-4B-Q4_K_M.gguf, bartowski's conversion)	fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3
dataset (data/train.jsonl, deterministic generator)	4c3feca6afa9d776b0e5d08afb6ac4b134bee93a94548dd934a74b2c944d6a5a
this adapter	ea38f80c33d381c9aa62c874920d2fa058dd307e7186e9b7c347d782f35b88e4

Config: rank 8 on every projection (attention q/k/v/output + FFN gate/up/down), alpha 16, AdamW (0.9/0.999, wd 0.01), lr 1e-4, 316 steps (2 epochs of 158 examples), seed 0, ctx 128. Loss 0.676 to 0.000006.

Determinism was verified on this exact base before training the release adapter: two independent 40-step runs with identical inputs produced byte-identical adapter files (35a19a2435fd717875929a098019250c7b7b5c3e1b669abec87e20b3fa1916f1, both). Runner gates this property in its test suite (same data + same seed → byte-identical adapter); the engine design notes are in docs/adaptation-engine.md.

Use it

# serve the base with the adapter (on the CPU here; on CUDA and Metal an
# offloaded block applies the adapter on the device: `runner -m ... --adapt-info`
# says which backend of your build carries adapter kernels)
runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --lora Qwen3-4B-ToolUse-LoRA.gguf --serve --gpu off

# or reproduce the adapter from scratch: the same sha256 on the same build
python3 scripts/make-tooluse-data.py data/
runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --train data/train.jsonl \
    --train-steps 316 --lr 1e-4 --train-out adapter.gguf

Any Runner release serves it (releases); this adapter was produced at commit 3f7cd50 (the v0.2.0 training code), and the adapter bytes are gated identical across later binaries, thread counts and the CUDA assist on the same instruction-set profile. The adapter GGUF uses the llama.cpp adapter naming convention.

More you can do with this adapter

Each item below is explained, with its limits, in the complete adapter guide.

• Serve it per request. runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --serve --gpu off --adapter tools=Qwen3-4B-ToolUse-LoRA.gguf loads it once; a request naming "model": "Qwen_Qwen3-4B-Q4_K_M.gguf:tools" gets the adapter, and one without the suffix gets the bare base, side by side on parallel slots.
• Evaluate it. --score -f heldout.txt, with and without --lora, gives teacher-forced logprobs, perplexity and top-1 rate for both.
• Train it further. --lora Qwen3-4B-ToolUse-LoRA.gguf --train more.jsonl --train-out next.gguf continues from this adapter. Since Runner 1.4.2 the new record names this file as from_adapter, so --lineage next.gguf walks back through it to the base and the data.
• Train on exact tokens, and see what a step trains on (Runner 1.4.2). A training line can give prompt_tokens and completion_tokens instead of text, and --train-logprobs FILE reports the logprob of every target token under the forward each step trains through.
• Preferences and rewards. --train-dpo trains from chosen and rejected answers against the frozen base, and scripts/train-grpo-lite.py runs a reinforcement loop whose every round checks that the trainer scores the sampled rollouts bit for bit as the server did (Runner 1.4.2).
• Sign it and require the signature. runner --sign-model Qwen3-4B-ToolUse-LoRA.gguf --model-key key.pem writes an OpenSSF Model Signing bundle; a server started with --model-pubkey key.pub --require-signed-model then refuses the adapter if its bytes or signature do not verify.
• Trace it. runner --lineage Qwen3-4B-ToolUse-LoRA.gguf checks the adapter against its .train.json record and names the base and the data it was trained from, by sha256 (each re-hashed where the file is on disk).

The three-precision study (study/)

The same task, same seed, same data, same config, trained through three precisions of the same base: does the quantization you train through change what you learn?

trained through	eval on own base	eval on Q4_K_M base	adapter sha256 (first 16)
bf16	1.000 exact	1.000 exact	24d5f02122bc7436
Q8_0	1.000 exact	1.000 exact	a2c70991d7421bab
Q4_K_M	1.000 exact	n/a	eda3c58491f9c654

Weight-space divergence between the learned adapters (all-parameter cosine and relative L2):

pair	cosine	rel L2
bf16 vs Q8_0	0.99983	0.019
bf16 vs Q4_K_M	0.99259	0.122
Q8_0 vs Q4_K_M	0.99276	0.120

Read: training through Q8_0 learns essentially the same adapter as training through bf16 (2% relative difference). Training through the 4-bit base learns a measurably different adapter (12% relative divergence), so "fine-tune FP16, then quantize for deployment" and "adapt the deployed quant directly" are not the same operation; yet on this task's supervised, wide-margin decisions the difference costs nothing: every adapter scores 1.000, and the bf16- and Q8-trained adapters drop onto the Q4_K_M serving base at 1.000 too. Read that scope literally: it is not free everywhere, and the two refinements below show where the cost actually shows up. All three study adapters and their provenance records are in study/.

Follow-up: the divergence is behavioral (evals/logprob-study/). Prompted by an independent reproduction of this repo's determinism claims (HF forum; the same tester verified adapter byte-equality on a Tesla T4 and correctly proposed logprobs as the next instrument): all 29 held-out prompts with gold completions, scored through the same Q4_K_M serving base under each study adapter, 3,195 positions.

pair	weight rel L2	mean |Δlogprob|	positions >1 nat
bf16-ad vs Q8-ad	0.019	0.020	0%
bf16-ad vs Q4-ad	0.122	0.146	2.2%
base vs any adapter	n/a	0.32 to 0.40	6 to 8%

The Q4-trained adapter differs from the bf16-trained one by ~40% of the entire adapter effect in logprob space; the 6x weight-space gap maps to a 7.4x logprob gap.

Refinement (2026-08-25, from the same independent reviewer who reproduced the determinism claims): splitting those positions by prompt vs gold completion changes what the aggregate measures. On the supervised completion region the three adapters nearly coincide (bf16-vs-Q4 mean |dlogprob| ~0.0003 nat, no position near 1 nat, verified from the published files); almost all of the 0.146 aggregate lives in the UNSUPERVISED prompt region. So the honest reading is: the precision path leaves a real function-level fingerprint, largely outside what the completion loss supervised, while the supervised decisions stay wide-margin. Those are two different observables. The same reviewer then supplied the missing piece: on deliberately ambiguous requests the fingerprint DOES cross a decision boundary. On "tell me what README.md says and translate it", the bf16- and Q8-trained adapters call read_file (margins ~0.49 nat) while the Q4-trained adapter confidently answers none (margin 2.17 nat), and the split survives full deterministic JSON generation. One constructed case, an existence proof rather than a prevalence estimate, and exactly the "where does weight divergence become behavioral" answer this study was reaching for. A small decision-boundary lane (top1/top2 margins over ambiguous tool choices) is the planned follow-up.

Systematized (2026-08-30, same reviewer). That constructed case is now a screened one. He built a 36-prompt tool-choice boundary bank across five ambiguity families (list_dir/search_files, read_file/search, read_file/write_file multi-intent, config browsing, and available-tool/none), kept the cases whose top1-top2 legal-choice margin sits at or under 3 nat, and ran the base plus all three study adapters on the survivors, generating the full JSON deterministically wherever the branches disagreed. Seven cases passed the screen, and the split above is the one that reproduced. Three constraints travel with it, all his, and they bound what this result may be used for:

• The screen selects on the Q4 adapter's own margin, so the seven selected cases cannot estimate any general margin distribution. One split out of seven out of 36 is an existence proof, not a rate.
• There is no monotonic precision law here. Several cases had smaller Q4 margins, not all did, and one case had Q8 as the tightest condition. What this supports is that some decision boundaries are far more sensitive to the training precision path than others, non-monotonically in bit width. It does not support "lower training precision degrades tool choice", which his own data refutes.
• The bank carries no gold labels, deliberately: labeling would convert a decision-sensitivity probe into a quality benchmark and quietly change the question.

Notebooks, clean and executed with reference outputs: John6666/forum3.

The merge study (merged/), or: why this repo ships an adapter

Runner's --merge-lora folds the adapter into the base weights (W' = W + (alpha/r)·B·A, deterministic: same inputs → byte-identical merged file) so the result runs in any GGUF runtime. We merged this adapter into three output precisions of the same Q4_K_M base and re-ran the held-out eval on each standalone file:

merged into	right tool	exact call	verdict
Q4_K_M (base's own type)	0.724	0.690	adapter erased: scores exactly the base
Q8_0	1.000	1.000	behavior fully preserved
F16	1.000	1.000	behavior fully preserved

The 4-bit merge is not slightly worse: it reproduces the base's numbers to the prompt. Byte-level comparison shows why: after requantizing to Q4_K_M, only 1.45% of weight bytes differ from the base; the LoRA delta is small relative to the 4-bit grid step, so quantization rounds it back to the original codes almost everywhere. Runner 1.2.1 measures this directly: --merge-lora projects the merged difference onto the intended delta and records delta_retained in the .merge.json; on this adapter the Q4_K_M merge kept 1.8% of the delta (3.0, 4.7 and 8.9% at 2, 4 and 8x scale) and the Q8_0 merge 99.5%. A merge that keeps under half is now refused with the destination untouched; --merge-allow-erased writes it anyway, which is what reproducing the erased file below requires. The visible symptom (via stock llama.cpp b10581, raw completion): the Q8_0 and F16 merges answer with JSON only, exactly as trained; the Q4_K_M merge answers, then drifts into conversational prose, the base's behavior.

Deployment guidance, measured: to serve 4-bit with this adaptation, serve base + --lora (exact, and how the 1.000 row above is produced). To ship a standalone merged file, merge into Q8_0 or wider and measure it: this repo's merged/merged-Q8_0.gguf is that artifact, verified 1.000 on the eval and runnable in stock llama.cpp. Merging into a 4-bit base is where fine-tunes silently disappear; if you do it anywhere, eval the merged file, not the adapter.

The survival threshold, measured (follow-up sweep): the same adapter merged at five --lora-scale values into the same Q4_K_M base, every merged file evaled standalone, served base+adapter as the control:

scale	merged-Q4 exact call	served exact call	bytes changed
0.5	0.690 (base)	1.000	1.15%
1	0.690 (base)	1.000	1.45%
2	0.690 (base)	1.000	2.05%
4	0.828	1.000	2.87%
8	1.000	0.138	4.60%

Survival is monotone in delta magnitude, as the rounding mechanism predicts, and note the inversion at 8x: the exact over-scaled adapter breaks the served model, while the 4-bit grid filters the same weights back to 1.000. Quantization acts as a filter on your fine-tune with a pass-band you don't control; since 1.2.1 that 8x merge is refused by default for the same reason. The sweep evals are in evals/sweep-*.json.

Interop, both directions: this adapter scores the same 1.000/1.000/1.000/1.000 on the full held-out eval when served by stock llama.cpp (b10581, llama-server --lora, raw completion); the eval output is evals/llamacpp-served-adapter.json. And as of runner main, --lora accepts the F16 adapters llama.cpp's convert_lora_to_gguf emits (found by loading a community adapter, which the old F32-only check refused).

The Q4_K_M and F16 merges are not uploaded (one is behaviorally the base, the other is 8 GB); their provenance records with sha256s are in merged/, and because the merge is deterministic, one command reproduces either file byte-for-byte:

runner -m Qwen_Qwen3-4B-Q4_K_M.gguf --lora Qwen3-4B-ToolUse-LoRA.gguf \
    --merge-lora merged-Q4_K_M.gguf --merge-allow-erased   # sha256 17443f0d55f5bb33...

(Runner 1.2.1 or later refuses this merge without --merge-allow-erased, because it keeps 1.8% of the delta; earlier releases wrote it silently.)

Honest limits

• Off its task the adapter does not help. On the shifted set (274 hand-written prompts over 23 tools with overlapping names, 2026-09-24) the base made 211 exact calls through the chat API and the adapter 200 (base favoured, z 2.2), and 170 and 176 on the training template (inside noise). On underspecified requests the adapter is more certain of the wrong tool. It helps only where no tool fits (30 of 34 against the base's 25). The record.
• The task is narrow and synthetic (four tools, template-generated requests, 29 held-out prompts). This demonstrates the training mechanism and its reproducibility, not general tool-use capability.
• 316 steps at lr 1e-4 on 158 examples is memorization-grade training, appropriate for a format task, not a recipe for knowledge tasks.
• The precision-study equivalence is measured on this task; harder tasks may separate the adapters. That experiment is now a one-command rerun for anyone, which is rather the point.
• Training is slow by Python-framework standards. This adapter took ~45 s/step at 4B on a 128-thread EPYC-class host with runner v0.2.0; current builds train the same config at ~20 s/step there, and a 7B Q4_K_M at 64.8 s/step on an RTX 3070 with the CUDA assist (RUNNER_TRAIN_GPU=1, 76.5 s on 8 CPU threads). Budget hours for a few hundred steps. The adapter bytes are gated identical across the old binary, the new one, any thread count and the CUDA assist on the same instruction-set profile, so the reproduce-the-sha command works unchanged on current builds; a different profile is a different build and can write different bytes.
• Coverage is dense families (Llama, Mistral, Qwen2.5, Granite 4.x, Gemma 3 and 4) plus attention-only adapters on sparse-MoE models with Qwen3-MoE or Mixtral routing; expert adapters, other routers, shared experts and recurrent blocks are refused by name.

Files

• Qwen3-4B-ToolUse-LoRA.gguf: the adapter (+ .train.json provenance)
• data/: the deterministic dataset (train + held-out eval)
• evals/: every eval JSON quoted above, unedited (incl. the three merged-artifact evals)
• study/: the three precision-study adapters, provenance, divergence
• merged/: the standalone merged Q8_0 GGUF (runs in any GGUF runtime, verified 1.000 on the eval and smoke-tested in llama.cpp b10581) + .merge.json provenance for all three merge targets

About Xyntetik

This repository is published by Xyntetik, the Zenova AB program for running capable language models on hardware you own and proving what they did. Its released engine is Xyntetik Runner: a free, single-binary GGUF engine in C for CPU, CUDA and Metal that serves, scores and trains through the quantized file it deploys and records every run as a receipt. The measurements behind the program's claims are collected at xyntetik.com/evidence. The program's organization on Hugging Face is Xyntetik.