Misakachain commited on
Commit
63e35ed
·
verified ·
1 Parent(s): 31ad728

LM Studio gateway backend: palw-gateway crate (sessions/receipts/settlement/search), plugin rev5, launchers + search sidecar, ModelGenesisManifest (mirrors misaka-proof-of-llm 749aabb)

Browse files
docs/evidence/qi35-lmstudio-palw-receipt.md CHANGED
@@ -1,9 +1,23 @@
1
  # Huihui Qwen3.6 PALW receipt model in LM Studio
2
 
3
  This integration exposes the repository's patched canonical-INTEGER runtime to
4
- LM Studio through a loopback OpenAI-compatible endpoint. LM Studio's stock
5
- llama.cpp runtime is not used for inference: the custom GGUF requires the PALW
6
- runtime patches, and `qi35_model` remains the receipt issuer.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
 
8
  ## Installed LM Studio generator
9
 
@@ -16,19 +30,114 @@ after changing the TypeScript adapter with:
16
 
17
  ## Start
18
 
19
- Start the endpoint before opening a chat in LM Studio:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
 
21
  ```sh
22
  cd /Users/wata/Downloads/qwen-8.0
23
- ./docs/evidence/qi35_lmstudio_server.sh
 
24
  ```
25
 
26
- The launcher binds only to `127.0.0.1:12345`, checks the audit key, starts the
27
- local SearXNG stack when needed, and enables canonical Metal offload. In LM
28
- Studio, select the installed generator and its only model:
 
 
 
 
29
 
30
  `Huihui-Qwen3.6-35B-A3B PALW receipt`
31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  ## Receipt and search semantics
33
 
34
  - Search routing is automatic in `smart` mode. `/search <question>` forces it.
 
1
  # Huihui Qwen3.6 PALW receipt model in LM Studio
2
 
3
  This integration exposes the repository's patched canonical-INTEGER runtime to
4
+ LM Studio through a loopback endpoint. LM Studio's stock llama.cpp runtime is
5
+ not used for inference: the custom GGUF requires the PALW runtime patches, and
6
+ `qi35_model` remains the receipt issuer.
7
+
8
+ Since plugin revision 5 there are two selectable backends (`Backend` in the
9
+ plugin config):
10
+
11
+ - **`palw-gateway` (default, 127.0.0.1:12346)** — the stateful successor. The
12
+ chat turn that answers the screen IS the PALW A-execution: persistent
13
+ sessions with id-stable prefix caching, per-turn engine receipts + runtime
14
+ ROOTS, privacy/mint mode fixed per message BEFORE sending, live web search
15
+ through the same sidecar stack, and the settlement pipeline (background
16
+ worker → replica → certified) whose status streams back into the chat. See
17
+ "Gateway backend" below.
18
+ - **`legacy` (127.0.0.1:12345)** — the stateless `qi35_lmstudio_server.py`,
19
+ kept as the reference implementation. Each turn re-prefills the whole
20
+ transcript.
21
 
22
  ## Installed LM Studio generator
23
 
 
30
 
31
  ## Start
32
 
33
+ Since plugin revision 4 there is nothing to start manually: on each chat the
34
+ generator health-checks the selected backend's `/healthz` and, when the
35
+ endpoint is down, spawns that backend's launcher itself — detached, so the
36
+ engine server outlives both the chat and LM Studio and is reused by later
37
+ sessions. Revision 5 selects the launcher by the `Backend` config:
38
+ `docs/evidence/qi35_lmstudio_gateway.sh` (gateway, port 12346) or
39
+ `docs/evidence/qi35_lmstudio_server.sh` (legacy, port 12345). Launch progress
40
+ streams into the same `receipt対象外` status block the backends use; launcher
41
+ output goes to `~/.config/misaka-palw/lmstudio-server.log` (override:
42
+ `QI35_LMSTUDIO_SERVER_LOG`). Concurrent chats share a single startup attempt,
43
+ and the plugin decides gateway-vs-legacy behavior by what actually answers
44
+ `/healthz` (the gateway reports `engine_resident`), not by config alone.
45
+
46
+ Plugin config: `Backend` (gateway/legacy), `PALW mode` (see below),
47
+ `Auto-start PALW engine server` (off restores manual-only behavior), and
48
+ `QI35 workspace` pointing at the repository root (default
49
+ `/Users/wata/Downloads/qwen-8.0`; env `QI35_WORKSPACE` wins). Receipt
50
+ semantics are unchanged on both paths: `qi35_model` behind the loopback bridge
51
+ remains the only issuer, and the spawned launcher is byte-identical to the
52
+ manual path.
53
+
54
+ Manual start remains supported (the plugin detects the healthy endpoint and
55
+ spawns nothing):
56
 
57
  ```sh
58
  cd /Users/wata/Downloads/qwen-8.0
59
+ ./docs/evidence/qi35_lmstudio_gateway.sh # gateway backend (default)
60
+ ./docs/evidence/qi35_lmstudio_server.sh # legacy backend (web search)
61
  ```
62
 
63
+ Stop a detached backend with `pkill -f palw-gateway` /
64
+ `pkill -f qi35_lmstudio_server.py`.
65
+
66
+ Both launchers bind loopback only, require the audit key, enable canonical
67
+ Metal offload, and start the SearXNG stack when search is enabled
68
+ (`QI35_SEARCH_AUTO=off` skips it). In LM Studio, select the installed
69
+ generator and its only model:
70
 
71
  `Huihui-Qwen3.6-35B-A3B PALW receipt`
72
 
73
+ ## Gateway backend (plugin revision 5)
74
+
75
+ One execution answers the screen AND is the PALW A-run — the design rule that
76
+ makes local-first economics work (no second inference for the receipt). Per
77
+ turn the plugin POSTs the LM Studio transcript to the gateway's
78
+ `/v1/lmstudio/turn`; the gateway resolves it onto its session store BY CONTENT
79
+ (`<think>`/status blocks stripped before comparison):
80
+
81
+ - matched prefix + same tail → `continued` on the existing branch — the
82
+ compiled prompt begins with the stored ids and the engine's prefix cache
83
+ hits (live: `cache_hit 206/223`, turn latency 23.1 s → 8.7 s);
84
+ - edit/regenerate in LM Studio → sibling branch under the last matching turn
85
+ (append-only history, exactly like the native session API);
86
+ - unknown transcript → new conversation, foreign history imported as
87
+ `mint_ineligible` turns (`stop_reason=imported`, ids re-encoded from text —
88
+ the one id-unstable step; every later turn is id-stable again).
89
+
90
+ `PALW mode` maps to the turn's privacy mode, fixed before sending (§16 of the
91
+ product architecture: deciding after generation invites grinding):
92
+ `PALW Mint` / `Verified, No Mint` enter the settlement pipeline,
93
+ `Local Only` never leaves the device. The `lmstudio_resolve` stream frame
94
+ reports the PREVIOUS turn's settlement state (that is how certification
95
+ results reach the chat without blocking it), and after the answer completes
96
+ the plugin polls `GET /v1/turns/{id}` briefly
97
+ (`QI35_SETTLEMENT_POLL_MS`, default 6 s) — settlement continues in the
98
+ background either way. LM Studio's automatic "2-5 word title" request is
99
+ detected and routed over the stateless `/v1/chat/completions` path so it
100
+ neither creates a session turn nor mints a receipt.
101
+
102
+ Live web search runs on the gateway path through
103
+ `docs/evidence/qi35_search_sidecar.py` (`--search-cmd`, wired by the
104
+ launcher): the ENTIRE legacy stack — smart/`/search` routing, SSRF-guarded
105
+ page fetch, canonical `live_search_bundle_v1`, typed failure bundles — reused
106
+ as a stdin/stdout subprocess, fail-open to a searchless turn only on
107
+ sidecar-level errors. The rendered evidence joins the turn's user block
108
+ (`user + "\n\n" + evidence`, the legacy `build_prompt_text` convention), so
109
+ the receipt's `prompt_commitment` covers `bundle_sha256=` transitively — no
110
+ receipt schema change. The bundle is persisted on the turn (schema v4
111
+ `search_bundle_json`), streamed to the plugin as a `search_status` frame, and
112
+ exported as the third sidecar `.search.json` (bundle + `receipt_binding`).
113
+ The B replica replays the committed prompt ids — A's snapshot, never a
114
+ re-search — so search turns settle exactly like plain turns.
115
+
116
+ Gateway receipts are the same engine `RCPT` v0 objects, persisted on the turn
117
+ row AND written as sidecars (`--receipt-dir`, default
118
+ `docs/evidence/lmstudio-receipts/`, files 0600):
119
+ `turn-<turn_id>-<output_commitment16>.receipt.json` plus a
120
+ `qi35-gateway-opening/v1` `.opening.json` carrying the exact prompt/output
121
+ ids, tokenizer digest, runtime roots and context meta. Offline verification =
122
+ `qi35_chat.verify_receipt` (MAC) + recomputing both commitments from the
123
+ opening's ids; verified live on 2026-07-31 for all four E2E turns, alongside
124
+ the full loop `chat turn → receipt → A-commit → self-replica re-execution on
125
+ the real 35B → route/kv/state roots matched → certified → certified_head
126
+ advanced`, with `verified_no_mint` and `local_only` behaving per spec. A
127
+ live `/search` turn additionally proved the binding chain offline: MAC +
128
+ both commitments recomputed, `bundle_sha256=a7f79b9b…` found inside the
129
+ committed 1340-token prompt, `.search.json` `receipt_binding` and the
130
+ opening's `context_meta.search` matching — then settled `certified` through
131
+ the same replica pipeline.
132
+
133
+ Not on the gateway path yet (stated honestly): model-emitted tool calls, and
134
+ everything consensus-side (`--palw-loopback` certifies plumbing, not
135
+ economics — point `QI35_PALW_COORDINATOR` at a node bridge for the real
136
+ counterparty). Verifier note: `palw-verify-search` replays LEGACY openings
137
+ (`external_input` form); gateway search turns verify today via the opening's
138
+ ids + `.search.json` as above, and extending the Rust verifier to the
139
+ `qi35-gateway-opening/v1` shape is follow-up work.
140
+
141
  ## Receipt and search semantics
142
 
143
  - Search routing is automatic in `smart` mode. `/search <question>` forces it.
docs/evidence/qi35_lmstudio_gateway.sh ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/zsh
2
+ # Launch palw-gateway as the LM Studio backend (the stateful successor of
3
+ # qi35_lmstudio_server.py): persistent sessions with id-stable prefix caching,
4
+ # per-turn engine receipts + runtime ROOTS, the PALW settlement pipeline
5
+ # (loopback coordinator by default; point QI35_PALW_COORDINATOR at a node-side
6
+ # bridge to replace it), and GPT-style live search via the sidecar
7
+ # (qi35_search_sidecar.py reusing the legacy stack; QI35_SEARCH_AUTO=off
8
+ # disables it).
9
+ set -e
10
+
11
+ SP="$(cd "$(dirname "$0")" && pwd)"
12
+ WS="${QI35_WORKSPACE:-$(cd "$SP/../.." && pwd)}"
13
+ export QI35_MODEL="${QI35_MODEL:-$WS/models/Qwen3.6-abliterated-35b-Claude-4.7/Qwen3.6-abliterated-35b-Claude-4.7-Q4_K_M.gguf}"
14
+ export QI35_TABLES="${QI35_TABLES:-$SP/qi35_tables.bin}"
15
+ export QI35_ENGINE="${QI35_ENGINE:-$SP/qi35_model}"
16
+ export QI35_TOKENIZER="${QI35_TOKENIZER:-$WS/models/Qwen3.6-35B-A3B-Claude-4.7-base-meta/tokenizer.json}"
17
+ export QI35_AUDIT_KEY="${QI35_AUDIT_KEY:-$HOME/.config/misaka-palw/audit-keys/local-audit.key}"
18
+ export QI35_RECEIPT_DIR="${QI35_RECEIPT_DIR:-$SP/lmstudio-receipts}"
19
+ QI35_GATEWAY_STATE="${QI35_GATEWAY_STATE:-$HOME/.config/misaka-palw/lmstudio-gateway}"
20
+ QI35_GATEWAY_PORT="${QI35_GATEWAY_PORT:-12346}"
21
+ QI35_MAX_NEW="${QI35_MAX_NEW:-2048}"
22
+ QI35_MAX_CONTEXT="${QI35_MAX_CONTEXT:-8192}"
23
+ QI35_THREADS="${QI35_THREADS:-8}"
24
+ QI35_LMSTUDIO_API_KEY="${QI35_LMSTUDIO_API_KEY:-palw-local}"
25
+ export QI35_SEARCH_AUTO="${QI35_SEARCH_AUTO:-smart}"
26
+ export QI35_SEARCH_BACKEND="${QI35_SEARCH_BACKEND:-searxng}"
27
+ export QI35_SEARXNG_URL="${QI35_SEARXNG_URL:-http://127.0.0.1:8080}"
28
+ export QI35_SEARCH_LANGUAGE="${QI35_SEARCH_LANGUAGE:-ja-JP}"
29
+ export QI35_SEARCH_FETCH_PAGES="${QI35_SEARCH_FETCH_PAGES:-2}"
30
+ QI35_SYSTEM="${QI35_SYSTEM:-You are a thoughtful and accurate assistant. The local date is $(date +%F). Write entirely in the user's language, including reasoning. For each non-trivial request, output exactly one concise <think>...</think> analysis, always close it, then give the direct answer with useful explanation, examples, and cautions. Do not invent current facts; say so when you would need a live source.}"
31
+
32
+ if [[ ! -f "$QI35_AUDIT_KEY" ]]; then
33
+ print -u2 "qi35_lmstudio_gateway: audit key not found: $QI35_AUDIT_KEY"
34
+ exit 1
35
+ fi
36
+
37
+ # SearXNG stack for the search sidecar — same fail-closed block as the legacy
38
+ # launcher, but only when search is enabled and uses the searxng backend.
39
+ if [[ "$QI35_SEARCH_AUTO" != "off" && "$QI35_SEARCH_BACKEND" == "searxng" ]]; then
40
+ if ! curl -fsS --max-time 2 "$QI35_SEARXNG_URL/healthz" >/dev/null 2>&1; then
41
+ command -v colima >/dev/null 2>&1 || { print -u2 "Colima is required for search (QI35_SEARCH_AUTO=off to disable)"; exit 1; }
42
+ colima status >/dev/null 2>&1 || colima start --cpu 2 --memory 3 --vm-type vz
43
+ command -v docker-compose >/dev/null 2>&1 || { print -u2 "docker-compose is required for search"; exit 1; }
44
+ docker-compose --project-directory "$WS/infra/searxng" -f "$WS/infra/searxng/compose.yaml" up -d
45
+ for _ in {1..30}; do
46
+ curl -fsS --max-time 2 "$QI35_SEARXNG_URL/healthz" >/dev/null 2>&1 && break
47
+ sleep 1
48
+ done
49
+ fi
50
+ curl -fsS --max-time 2 "$QI35_SEARXNG_URL/healthz" >/dev/null
51
+ fi
52
+
53
+ GW_BIN="${QI35_GATEWAY_BIN:-$WS/palw-gateway/target/release/palw-gateway}"
54
+ if [[ ! -x "$GW_BIN" ]]; then
55
+ command -v cargo >/dev/null 2>&1 || { print -u2 "palw-gateway binary missing and cargo unavailable: $GW_BIN"; exit 1; }
56
+ print -u2 "qi35_lmstudio_gateway: building palw-gateway (release)…"
57
+ (cd "$WS/palw-gateway" && cargo build --release)
58
+ fi
59
+
60
+ mkdir -p "$QI35_GATEWAY_STATE"
61
+ chmod 700 "$QI35_GATEWAY_STATE"
62
+
63
+ # The worker's counterparty: a node-side bridge when given, else the in-process
64
+ # loopback coordinator (dev harness — certifies plumbing, not consensus).
65
+ palw_args=(--palw-loopback)
66
+ if [[ -n "${QI35_PALW_COORDINATOR:-}" ]]; then
67
+ palw_args=(--palw-coordinator "$QI35_PALW_COORDINATOR")
68
+ [[ -n "${QI35_PALW_COORDINATOR_TOKEN:-}" ]] && palw_args+=(--palw-coordinator-token "$QI35_PALW_COORDINATOR_TOKEN")
69
+ fi
70
+ [[ "${QI35_METAL:-v3}" == "0" ]] && palw_args+=(--no-metal)
71
+ [[ "$QI35_SEARCH_AUTO" != "off" ]] && palw_args+=(--search-cmd "$SP/qi35_search_sidecar.py")
72
+
73
+ exec "$GW_BIN" \
74
+ --gguf "$QI35_MODEL" \
75
+ --tables "$QI35_TABLES" \
76
+ --tokenizer "$QI35_TOKENIZER" \
77
+ --engine "$QI35_ENGINE" \
78
+ --listen "127.0.0.1:$QI35_GATEWAY_PORT" \
79
+ --db "$QI35_GATEWAY_STATE/sessions.sqlite3" \
80
+ --artifacts-dir "$QI35_GATEWAY_STATE/artifacts" \
81
+ --workspace "$QI35_GATEWAY_STATE/workspace" \
82
+ --threads "$QI35_THREADS" \
83
+ --max-new "$QI35_MAX_NEW" \
84
+ --max-context "$QI35_MAX_CONTEXT" \
85
+ --system-prompt "$QI35_SYSTEM" \
86
+ --auth-token "$QI35_LMSTUDIO_API_KEY" \
87
+ --audit-key-file "$QI35_AUDIT_KEY" \
88
+ --receipt-dir "$QI35_RECEIPT_DIR" \
89
+ "${palw_args[@]}"
docs/evidence/qi35_lmstudio_install.sh CHANGED
@@ -12,5 +12,6 @@ command -v npm >/dev/null 2>&1 || { print -u2 "npm is required"; exit 1; }
12
  (cd "$PLUGIN" && npm install && "$LMS" dev --install -y)
13
 
14
  print "Installed misakachain/palw-receipt-adapter in LM Studio."
15
- print "Start the PALW endpoint with:"
16
- print " $SP/qi35_lmstudio_server.sh"
 
 
12
  (cd "$PLUGIN" && npm install && "$LMS" dev --install -y)
13
 
14
  print "Installed misakachain/palw-receipt-adapter in LM Studio."
15
+ print "Backends auto-start from the plugin; manual start:"
16
+ print " $SP/qi35_lmstudio_gateway.sh # gateway (sessions + settlement, default)"
17
+ print " $SP/qi35_lmstudio_server.sh # legacy (reference)"
docs/evidence/qi35_search_sidecar.py ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Search sidecar for palw-gateway: one JSON request on stdin, one JSON reply on stdout.
3
+
4
+ The gateway is Rust; the whole GPT-style search stack (routing, SSRF-guarded page
5
+ fetch, canonical live_search_bundle_v1) already exists in qi35_web_search.py. This
6
+ CLI re-exposes exactly the legacy qi35_chat.prepare_search() behavior — including
7
+ "failures are typed into a bundle, not thrown away" — without importing qi35_chat
8
+ (which requires the model/tokenizer environment). stdlib only.
9
+
10
+ stdin : {"query": str, "previous_user_query": str|null}
11
+ stdout: {"searched": false, "query": q} — routed away
12
+ {"searched": true, "query": q, "evidence_text": str,
13
+ "effective_compression": "off"|"quality_gated_v1",
14
+ "bundle": {...}, "bundle_sha256": str, "summary_lines": [str]} — searched
15
+ {"searched": false, "query": q, "error": str} — sidecar-level failure (fail-open to a searchless turn)
16
+ """
17
+
18
+ import json
19
+ import os
20
+ import sys
21
+ import unicodedata
22
+
23
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
24
+
25
+ from qi35_web_search import ( # noqa: E402
26
+ SearchError,
27
+ prompt_context_with_meta,
28
+ route_search_query,
29
+ search,
30
+ search_failure_bundle,
31
+ should_search,
32
+ )
33
+
34
+ SEARCH_MODE = os.environ.get("QI35_SEARCH_AUTO", "smart")
35
+
36
+
37
+ def summary_lines(bundle):
38
+ outcome = bundle.get("outcome", {"status": "ok"})
39
+ digest = bundle.get("bundle_sha256", "")[:16]
40
+ if outcome.get("status") != "ok":
41
+ return [f"検索失敗を証跡化 {bundle.get('provider', '?')} sha256={digest}…: {outcome.get('message', '')}"]
42
+ lines = [f"検索 {bundle.get('provider', '?')} sha256={digest}… hits={len(bundle.get('results', []))}"]
43
+ for result in bundle.get("results", []):
44
+ lines.append(f"[{result['rank']}] {result['title']} — {result['url']}")
45
+ return lines
46
+
47
+
48
+ def run(request):
49
+ raw_query = unicodedata.normalize("NFC", request.get("query", ""))
50
+ explicit = raw_query.startswith("/search ")
51
+ query = raw_query[len("/search "):].strip() if explicit else raw_query
52
+ if explicit and not query:
53
+ raise SearchError("検索語を入力してください")
54
+ if not explicit and not should_search(query, SEARCH_MODE):
55
+ return {"searched": False, "query": query}
56
+ previous = request.get("previous_user_query") or None
57
+ search_query, routing_reason, require_exact = route_search_query(query, previous)
58
+ try:
59
+ bundle = search(
60
+ search_query,
61
+ original_query=query,
62
+ routing_reason=routing_reason,
63
+ require_exact=require_exact,
64
+ )
65
+ except SearchError as error:
66
+ bundle = search_failure_bundle(
67
+ search_query,
68
+ error,
69
+ original_query=query,
70
+ routing_reason=routing_reason,
71
+ )
72
+ evidence_text, effective_mode = prompt_context_with_meta(bundle)
73
+ return {
74
+ "searched": True,
75
+ "query": query,
76
+ "evidence_text": evidence_text,
77
+ "effective_compression": effective_mode,
78
+ "bundle": bundle,
79
+ "bundle_sha256": bundle["bundle_sha256"],
80
+ "summary_lines": summary_lines(bundle),
81
+ }
82
+
83
+
84
+ def main():
85
+ request = json.load(sys.stdin)
86
+ try:
87
+ reply = run(request)
88
+ except Exception as error: # sidecar-level failure: the turn continues searchless
89
+ reply = {"searched": False, "query": request.get("query", ""), "error": str(error)}
90
+ json.dump(reply, sys.stdout, ensure_ascii=False)
91
+ sys.stdout.write("\n")
92
+
93
+
94
+ if __name__ == "__main__":
95
+ main()
docs/model-genesis-candidate.md CHANGED
@@ -43,3 +43,84 @@ d. Freeze tokenizer + runtime + quantization in the manifest; register the manif
43
  context in the LOCKED signature-domain table; sign with both parties' keys.
44
 
45
  Steps a–b are executable by one operator today; c is external by definition; d is blocked on c.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
  context in the LOCKED signature-domain table; sign with both parties' keys.
44
 
45
  Steps a–b are executable by one operator today; c is external by definition; d is blocked on c.
46
+
47
+ ## Prerequisites (in-repo) — status 2026-07-27
48
+
49
+ The flip path above was not executable, for a reason that had nothing to do with the external
50
+ gates: **the object the two parties are supposed to compare did not exist.** `ModelGenesisManifest`
51
+ had zero hits across both repositories, so two operators could each run a conversion and still have
52
+ nothing comparable at the end. That is now closed; the external items are untouched.
53
+
54
+ ### Built
55
+
56
+ | Prerequisite | Where | What it gives |
57
+ |---|---|---|
58
+ | `ModelGenesisManifest` type, canonical encoding, keyed-BLAKE2b-512 hash | `runtime-palw/src/model_genesis.rs` | The comparable artifact of ADR-0047 §1. Backend-independent and weights-derived only; big-endian integers, `u64`-length-framed variable fields (prefix-free ⇒ injective). Hash domain `misaka-palw-v1/model-genesis`, distinct from `palw-k1/file` and every `misaka-palw-v3/*` receipt domain. |
59
+ | Field-level diff | `ModelGenesisManifest::first_mismatch` | ADR-0047's acceptance test is a hash comparison, which tells a failing operator nothing. This names the differing field. |
60
+ | Mechanical backend-independence check | `ModelGenesisManifest::validate` | Rejects a conversion procedure naming a backend/host token (`metal`, `cuda`, `gpu`, `ngl`, `thread`, `arm64`, …), absolute or `..`-bearing paths (which carry an operator's home directory into the digest), non-ASCII text, and mutable revision pins (a branch name is not reproducible). |
61
+ | Honest readiness assessment | `model_genesis::assess_genesis` | Reproduces the four disqualifiers below as data. `eligible` can never be `true` — `ManifestUnsigned` is unconditional — and a unit test asserts that no input produces a genesis. Two stacks owned by one operator are reported as `TwoPartyForm::Weak`, per ADR-0047 §1. |
62
+ | Deterministic conversion + manifest tool | `scripts/model_genesis_manifest.py` (`selftest` / `convert` / `manifest` / `verify`) | The identical procedure both operators run. Fixed conversion environment (`LC_ALL=C`, `TZ=UTC`, `PYTHONHASHSEED=0`, `SOURCE_DATE_EPOCH=0`) and a required `--model-name`, because the converter otherwise derives `general.name` from the local directory name — a per-operator difference that would break the match for no weights-related reason. |
63
+ | Rust↔Python parity | golden vector `b747790722bdf626…f756d9b3`, pinned in both implementations | If the two encoders drifted, a cross-operator mismatch would be unattributable (weights, or tool?). `selftest` and `scripts/tests/test_model_genesis_manifest.py` bind them. |
64
+
65
+ Fail-closed behaviour is not aspirational — it is the current state of this checkout. Against the
66
+ real tree, `convert` and `manifest` both refuse:
67
+
68
+ ```
69
+ $ python3 scripts/model_genesis_manifest.py manifest --weights-dir models/Qwen3.6-35B-A3B-Claude-4.7-base-meta …
70
+ model-genesis: --weights-dir … contains metadata only — no *.safetensors/*.bin weight shards were
71
+ found. Download the full checkpoint; a manifest must never be built from absent weights.
72
+ ```
73
+
74
+ `models/Qwen3.6-35B-A3B-Claude-4.7-base-meta/` holds the seven pinned metadata files and no weight
75
+ shards, and `llama-quantize` is not among the built targets in `config/runtime-pins.sh`
76
+ (`PALW_BUILD_TARGETS`). Both are named explicitly by the tool rather than worked around.
77
+
78
+ ### The signature-domain row that would be required (BLOCKED on a human unlock decision)
79
+
80
+ Disqualifier (3) cannot be closed by writing code. The ML-DSA-87 signature-context table
81
+ `MisakaLLM-palw-shared/consensus/core/src/signature_domains.rs` was **LOCKED on 2026-07-25** and is
82
+ guarded by a golden test that duplicates every row. Adding a signing object is an explicit human
83
+ decision, so `runtime-palw/src/model_genesis.rs` contains **no signing function at all**. The exact
84
+ row that would be needed:
85
+
86
+ ```rust
87
+ SignatureDomain {
88
+ object: "PALW model genesis manifest",
89
+ context: crate::palw::PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT, // b"misaka-palw-v1/model-genesis/mldsa87"
90
+ defined_in: "misaka_palw::model_genesis::ModelGenesisManifest::manifest_hash",
91
+ },
92
+ ```
93
+
94
+ Three test sites change in the same commit, by design:
95
+
96
+ 1. `signature_domain_table_is_locked` — a new golden line (the lock's review surface).
97
+ 2. `palw_naming_divergence_is_pinned_not_forgotten` — asserts exactly 15 PALW rows and that every
98
+ PALW row outside a named allow-list contains no `/`. A slash-convention row must be added to
99
+ that allow-list, i.e. the naming convention is decided explicitly rather than inherited.
100
+ 3. `signature_domains_are_prefix_free` — already satisfied: the proposed context is distinct from
101
+ and not a prefix of `misaka-palw-v3/receipt/mldsa87` or `misaka-palw-v3/jobspec/mldsa87`
102
+ (checked by `model_genesis::tests::proposed_context_is_distinct_and_prefix_free`).
103
+
104
+ The context constant is recorded in this repo as
105
+ `PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED` — named so that using it before the unlock is
106
+ not a typo one can make silently. Signature *contexts* and keyed-*hash* domains are separate
107
+ namespaces (see the `signature_domains.rs` module note), so the hash domain above did not require
108
+ the table and does not pre-empt the decision.
109
+
110
+ ### Unchanged: what is still external
111
+
112
+ Disqualifiers (1), (2) and (4) are all still open, and none of them moved:
113
+
114
+ - **(1) provenance** — the pins in `config/runtime-pins.sh` still terminate at
115
+ `huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated`, a **third-party abliterated
116
+ derivative**, not official Alibaba weights. The manifest records this as a claim *inside* the
117
+ hashed encoding (`ProvenanceTier::ThirdPartyDerivative`), so one party cannot quietly record
118
+ "official" while the other records "derivative", and the tool prints `NOT a genesis candidate`
119
+ when that tier is used. Tooling makes the derivative reproducible; it cannot make it official.
120
+ - **(2) two-party reproduction** — no run has happened. Requires the official weights of (1), a
121
+ working RTX box, and preferably an organisationally independent second operator.
122
+ - **(4) compute-set** — receipts still bind the integer-canonical 0.5B calibration set; out of
123
+ ADR-0047's scope, tracked on the Receipt v3 / compute-set track.
124
+
125
+ `runtime-palw/src/mint.rs` `model.official_genesis` (`MainnetBlocker::ExternalModelGenesis`) is
126
+ therefore still unmet, and the mint test asserting that gate is never `met()` still holds.
docs/security-model.md CHANGED
@@ -59,6 +59,21 @@ R32は`In progress`だがCUDA Receiptは発行不可である。
59
  host OS、driver、GPU firmware まで敵対的とみなす場合は、TEE attestation または ZK/VC を追加
60
  しなければならない。この v1 はその主張を行わない。
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ## Threat controls
63
 
64
  | Threat | Primary controls | Fail condition |
@@ -81,7 +96,7 @@ host OS、driver、GPU firmware まで敵対的とみなす場合は、TEE attes
81
  | External double settlement | stable payment command、signed terminal confirmation、single durable terminal | command/distribution/pair/amount/epoch不一致、逆terminal、conflicting replay |
82
  | Canary false slash | scheduler-signed precommit、atomic receipt acceptance、fault-party timeout attribution | precommit/request/assignment/window不一致、opening未到達、scheduler opening timeout |
83
  | Bond funding/slash replay | signed funding event、typed primary proof、conservation reconciliation、appeal state | event/offense/canonical payload conflict、account/assignment/evidence不一致 |
84
- | Dependency/MSRV drift | exact crypto pins、lockfile、Rust 1.81 all-target CI | lock変更、MSRV manifest parse失敗、lint/test failure |
85
  | Premature Work Ticket | opaque maturity basis、required bond release links、WorkTicketV2、atomic one-shot issue | raw claim、epoch前倒し、bond link不足、grant不一致、source再消費 |
86
 
87
  ## Cryptographic rules
 
59
  host OS、driver、GPU firmware まで敵対的とみなす場合は、TEE attestation または ZK/VC を追加
60
  しなければならない。この v1 はその主張を行わない。
61
 
62
+ ### Consensus 側の三本柱(node ADR-0045 D2 の反映)
63
+
64
+ node 側 consensus が担う正しさの柱は次の三つで、役割は重複しない。本 crate の Receipt/audit
65
+ 機構は第 1 の柱への入力であり、支払いの最終性・遡及の扱いは第 3 の柱の規律に従う。
66
+
67
+ | 柱 | 検出するもの | 効果 | 時点 |
68
+ |---|---|---|---|
69
+ | Audit rounds | 標本化された leaf の再検証不一致 | certificate 拒否(事前)/ FraudEvidence(事後) | batch lifecycle 内 |
70
+ | PCPB | ticket と provider の束縛偽装(委譲・横流し) | ticket 無効 = mint 不能(事前) | mint 時 |
71
+ | Fraud + slash | 事後に発覚した不正(withhold、虚偽 receipt) | Revoked(**非遡及**)+ bond slash(遡及的担保) | いつでも |
72
+
73
+ Revoked の非遡及は node 側 SS-04 で確定済み(coinbase 済み provider 支払いは UTXO model 上
74
+ 巻き戻せない;事後抑止は bond slash が担う)。単一 self-reported Receipt が独立証明でないという
75
+ 本書の宣言を consensus 側も正とし、参照は双方向に固定される(node `docs/adr/0045`)。
76
+
77
  ## Threat controls
78
 
79
  | Threat | Primary controls | Fail condition |
 
96
  | External double settlement | stable payment command、signed terminal confirmation、single durable terminal | command/distribution/pair/amount/epoch不一致、逆terminal、conflicting replay |
97
  | Canary false slash | scheduler-signed precommit、atomic receipt acceptance、fault-party timeout attribution | precommit/request/assignment/window不一致、opening未到達、scheduler opening timeout |
98
  | Bond funding/slash replay | signed funding event、typed primary proof、conservation reconciliation、appeal state | event/offense/canonical payload conflict、account/assignment/evidence不一致 |
99
+ | Dependency/MSRV drift | exact crypto pins、lockfile、Rust 1.85 all-target CI | lock変更、MSRV manifest parse失敗、lint/test failure |
100
  | Premature Work Ticket | opaque maturity basis、required bond release links、WorkTicketV2、atomic one-shot issue | raw claim、epoch前倒し、bond link不足、grant不一致、source再消費 |
101
 
102
  ## Cryptographic rules
infra/lmstudio/palw-receipt-adapter/manifest.json CHANGED
@@ -3,5 +3,5 @@
3
  "runner": "node",
4
  "owner": "misakachain",
5
  "name": "palw-receipt-adapter",
6
- "revision": 3
7
  }
 
3
  "runner": "node",
4
  "owner": "misakachain",
5
  "name": "palw-receipt-adapter",
6
+ "revision": 5
7
  }
infra/lmstudio/palw-receipt-adapter/src/config.ts CHANGED
@@ -6,7 +6,7 @@ export const configSchematics = createConfigSchematics()
6
  "select",
7
  {
8
  displayName: "PALW receipt model",
9
- subtitle: "Canonical-integer Huihui Qwen3.6 with automatic live search",
10
  options: [
11
  {
12
  value: "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-PALW-receipt",
@@ -16,4 +16,52 @@ export const configSchematics = createConfigSchematics()
16
  },
17
  "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-PALW-receipt",
18
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  .build();
 
6
  "select",
7
  {
8
  displayName: "PALW receipt model",
9
+ subtitle: "Canonical-integer Huihui Qwen3.6 (integer engine issues the receipt)",
10
  options: [
11
  {
12
  value: "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-PALW-receipt",
 
16
  },
17
  "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-PALW-receipt",
18
  )
19
+ .field(
20
+ "privacyMode",
21
+ "select",
22
+ {
23
+ displayName: "PALW mode",
24
+ subtitle:
25
+ "Fixed per message BEFORE sending (deciding after generation invites grinding). Gateway backend only; the legacy server ignores it.",
26
+ options: [
27
+ { value: "palw_mint", displayName: "PALW Mint — replica + settlement, reward candidate" },
28
+ { value: "verified_no_mint", displayName: "Verified, No Mint — replica match only" },
29
+ { value: "local_only", displayName: "Local Only — never leaves this device" },
30
+ ],
31
+ },
32
+ "palw_mint",
33
+ )
34
+ .field(
35
+ "serverFlavor",
36
+ "select",
37
+ {
38
+ displayName: "Backend",
39
+ subtitle:
40
+ "Gateway (12346): persistent sessions, prefix-cache fast turns, settlement status, receipt sidecars. Legacy (12345): stateless server with live web search.",
41
+ options: [
42
+ { value: "gateway", displayName: "palw-gateway (sessions + settlement)" },
43
+ { value: "legacy", displayName: "legacy qi35_lmstudio_server.py (web search)" },
44
+ ],
45
+ },
46
+ "gateway",
47
+ )
48
+ .field(
49
+ "autoStartServer",
50
+ "boolean",
51
+ {
52
+ displayName: "Auto-start PALW engine server",
53
+ subtitle:
54
+ "Spawn the selected backend's launcher from docs/evidence when its port is down (log: ~/.config/misaka-palw/lmstudio-server.log)",
55
+ },
56
+ true,
57
+ )
58
+ .field(
59
+ "workspacePath",
60
+ "string",
61
+ {
62
+ displayName: "QI35 workspace",
63
+ subtitle: "Repository root containing docs/evidence launchers (QI35_WORKSPACE overrides)",
64
+ },
65
+ "/Users/wata/Downloads/qwen-8.0",
66
+ )
67
  .build();
infra/lmstudio/palw-receipt-adapter/src/gateway.ts ADDED
@@ -0,0 +1,212 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // palw-gateway client: the stateful LM Studio path. One POST to /v1/lmstudio/turn
2
+ // streams GenerationEventV1 frames (prefixed by a lmstudio_resolve frame); the turn's
3
+ // settlement status is then readable at GET /v1/turns/{id}. The receipt issuer is
4
+ // unchanged — qi35_model behind the gateway — this file only moves bytes.
5
+
6
+ export interface ResolveFrame {
7
+ resolution: string;
8
+ conversation_id: string;
9
+ parent_turn_id: string | null;
10
+ matched_pairs: number;
11
+ imported_turns: number;
12
+ parent: null | {
13
+ turn_id: string;
14
+ verification_status: string;
15
+ privacy_mode: string;
16
+ receipt_present: boolean;
17
+ };
18
+ certified_head: string | null;
19
+ }
20
+
21
+ export interface TurnStreamCallbacks {
22
+ onResolve: (frame: ResolveFrame) => void;
23
+ onSearchStatus: (lines: string[], bundleSha256: string) => void;
24
+ onStarted: (info: { job_id: string; turn_id: string; prompt_tokens: number }) => void;
25
+ onContextBudget: (droppedTurns: number) => void;
26
+ onDelta: (utf8: string) => void;
27
+ onCompleted: (info: {
28
+ stop_reason: string;
29
+ output_tokens: number;
30
+ cache_hit_tokens: number;
31
+ elapsed_ms: number;
32
+ }) => void;
33
+ }
34
+
35
+ export interface TurnInfo {
36
+ turn_id: string;
37
+ verification_status: string;
38
+ privacy_mode: string;
39
+ stop_reason: string | null;
40
+ certified_head: string | null;
41
+ receipt: {
42
+ present: boolean;
43
+ schema?: string;
44
+ output_commitment?: string;
45
+ prompt_commitment?: string;
46
+ signer_key_id?: string;
47
+ eos_reached?: boolean;
48
+ };
49
+ search?: {
50
+ present: boolean;
51
+ bundle_sha256?: string;
52
+ provider?: string;
53
+ };
54
+ }
55
+
56
+ export class GatewayError extends Error {}
57
+
58
+ interface ChatMessage {
59
+ role: string;
60
+ content: string;
61
+ }
62
+
63
+ function baseOrigin(baseUrl: string): string {
64
+ return baseUrl.replace(/\/v1\/?$/, "");
65
+ }
66
+
67
+ function authHeaders(apiKey: string): Record<string, string> {
68
+ return { "Content-Type": "application/json", Authorization: `Bearer ${apiKey}` };
69
+ }
70
+
71
+ // Detect which backend answers /healthz: the gateway reports {ok, engine_resident},
72
+ // the legacy python server reports {status:"ok", model}. Unreachable → null.
73
+ export async function detectFlavor(baseUrl: string): Promise<"gateway" | "legacy" | null> {
74
+ try {
75
+ const response = await fetch(`${baseOrigin(baseUrl)}/healthz`, {
76
+ signal: AbortSignal.timeout(1_500),
77
+ });
78
+ if (!response.ok) return null;
79
+ const body: unknown = await response.json();
80
+ if (typeof body === "object" && body !== null && "engine_resident" in body) return "gateway";
81
+ return "legacy";
82
+ } catch {
83
+ return null;
84
+ }
85
+ }
86
+
87
+ /// Stream one turn through POST /v1/lmstudio/turn. Resolves when the stream finishes
88
+ /// ([DONE] or EOF); throws GatewayError on HTTP errors or a `failed` frame. Abort by
89
+ /// firing `signal` — the gateway marks the turn user-cancelled via /v1/jobs/{id}/cancel,
90
+ /// which the caller is responsible for invoking (it needs the job id from onStarted).
91
+ export async function streamTurn(
92
+ baseUrl: string,
93
+ apiKey: string,
94
+ messages: ChatMessage[],
95
+ privacyMode: string,
96
+ callbacks: TurnStreamCallbacks,
97
+ signal: AbortSignal,
98
+ ): Promise<void> {
99
+ const response = await fetch(`${baseUrl.replace(/\/$/, "")}/lmstudio/turn`, {
100
+ method: "POST",
101
+ headers: authHeaders(apiKey),
102
+ body: JSON.stringify({ messages, privacy_mode: privacyMode }),
103
+ signal,
104
+ });
105
+ if (!response.ok || response.body === null) {
106
+ const text = await response.text().catch(() => "");
107
+ throw new GatewayError(`gateway ${response.status}: ${text.slice(0, 300)}`);
108
+ }
109
+ const reader = response.body.getReader();
110
+ const decoder = new TextDecoder();
111
+ let buffer = "";
112
+ const handleFrame = (payload: string) => {
113
+ if (payload === "[DONE]") return;
114
+ let event: Record<string, unknown>;
115
+ try {
116
+ event = JSON.parse(payload) as Record<string, unknown>;
117
+ } catch {
118
+ return;
119
+ }
120
+ switch (event.type) {
121
+ case "lmstudio_resolve":
122
+ callbacks.onResolve(event as unknown as ResolveFrame);
123
+ break;
124
+ case "search_status":
125
+ callbacks.onSearchStatus(
126
+ Array.isArray(event.lines) ? (event.lines as string[]) : [],
127
+ typeof event.bundle_sha256 === "string" ? event.bundle_sha256 : "",
128
+ );
129
+ break;
130
+ case "started":
131
+ callbacks.onStarted(event as unknown as { job_id: string; turn_id: string; prompt_tokens: number });
132
+ break;
133
+ case "context_budget":
134
+ callbacks.onContextBudget(Number(event.dropped_turns ?? 0));
135
+ break;
136
+ case "token":
137
+ if (typeof event.utf8_delta === "string" && event.utf8_delta.length > 0) {
138
+ callbacks.onDelta(event.utf8_delta);
139
+ }
140
+ break;
141
+ case "completed":
142
+ callbacks.onCompleted(
143
+ event as unknown as { stop_reason: string; output_tokens: number; cache_hit_tokens: number; elapsed_ms: number },
144
+ );
145
+ break;
146
+ case "failed":
147
+ throw new GatewayError(String(event.message ?? "generation failed"));
148
+ default:
149
+ break;
150
+ }
151
+ };
152
+ for (;;) {
153
+ const { done, value } = await reader.read();
154
+ if (done) break;
155
+ buffer += decoder.decode(value, { stream: true });
156
+ let split;
157
+ while ((split = buffer.indexOf("\n\n")) >= 0) {
158
+ const frame = buffer.slice(0, split);
159
+ buffer = buffer.slice(split + 2);
160
+ for (const line of frame.split("\n")) {
161
+ if (line.startsWith("data: ")) handleFrame(line.slice("data: ".length));
162
+ }
163
+ }
164
+ }
165
+ }
166
+
167
+ export async function cancelJob(baseUrl: string, apiKey: string, jobId: string): Promise<void> {
168
+ await fetch(`${baseUrl.replace(/\/$/, "")}/jobs/${jobId}/cancel`, {
169
+ method: "POST",
170
+ headers: authHeaders(apiKey),
171
+ signal: AbortSignal.timeout(3_000),
172
+ }).catch(() => {});
173
+ }
174
+
175
+ export async function fetchTurnInfo(baseUrl: string, apiKey: string, turnId: string): Promise<TurnInfo | null> {
176
+ try {
177
+ const response = await fetch(`${baseUrl.replace(/\/$/, "")}/turns/${turnId}`, {
178
+ headers: authHeaders(apiKey),
179
+ signal: AbortSignal.timeout(3_000),
180
+ });
181
+ if (!response.ok) return null;
182
+ return (await response.json()) as TurnInfo;
183
+ } catch {
184
+ return null;
185
+ }
186
+ }
187
+
188
+ const SETTLED = new Set(["replica_matched", "certified", "matured", "mismatch", "mint_ineligible"]);
189
+
190
+ /// Poll the turn's settlement for up to `budgetMs`, returning the last info seen. The
191
+ /// pipeline continues in the gateway regardless — this only decides how much the current
192
+ /// chat message can already show.
193
+ export async function pollSettlement(
194
+ baseUrl: string,
195
+ apiKey: string,
196
+ turnId: string,
197
+ budgetMs: number,
198
+ isAborted: () => boolean,
199
+ ): Promise<TurnInfo | null> {
200
+ const deadline = Date.now() + budgetMs;
201
+ let last: TurnInfo | null = null;
202
+ for (;;) {
203
+ if (isAborted()) return last;
204
+ const info = await fetchTurnInfo(baseUrl, apiKey, turnId);
205
+ if (info !== null) {
206
+ last = info;
207
+ if (SETTLED.has(info.verification_status)) return info;
208
+ }
209
+ if (Date.now() >= deadline) return last;
210
+ await new Promise((resolve) => setTimeout(resolve, 1_000));
211
+ }
212
+ }
infra/lmstudio/palw-receipt-adapter/src/generator.ts CHANGED
@@ -2,12 +2,27 @@ import { type Chat, type GeneratorController, type InferParsedConfig } from "@lm
2
  import OpenAI from "openai";
3
  import { type ChatCompletionMessageParam } from "openai/resources/index";
4
  import { configSchematics } from "./config";
 
 
5
 
6
- const BASE_URL = process.env.QI35_LMSTUDIO_BASE_URL ?? "http://127.0.0.1:12345/v1";
 
 
 
 
7
  const API_KEY = process.env.QI35_LMSTUDIO_API_KEY ?? "palw-local";
8
- // The PALW server now streams its own live status block (receipt対象外) covering search,
9
- // engine load, and prefill progress, so no static placeholder is emitted here. Both the
10
- // legacy placeholder and the live status block are stripped from history before resend.
 
 
 
 
 
 
 
 
 
11
  function withoutRuntimeStatus(content: string) {
12
  return content.replace(/<think>\s*PALW (?:runtime )?status \(receipt対象外\)[\s\S]*?<\/think>\s*/gi, "");
13
  }
@@ -24,17 +39,205 @@ function toMessages(history: Chat): ChatCompletionMessageParam[] {
24
  return messages;
25
  }
26
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  export async function generate(ctl: GeneratorController, history: Chat) {
28
  const config: InferParsedConfig<typeof configSchematics> = ctl.getPluginConfig(configSchematics);
29
- const client = new OpenAI({ apiKey: API_KEY, baseURL: BASE_URL, timeout: 30 * 60 * 1000 });
30
- const stream = await client.chat.completions.create({
31
- model: config.get("model"),
32
- messages: toMessages(history),
33
- stream: true,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
  });
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  ctl.onAborted(() => stream.controller.abort());
37
- const parser = new ReasoningStreamParser(ctl);
38
  for await (const chunk of stream) {
39
  const delta = chunk.choices[0]?.delta?.content;
40
  if (delta) parser.push(delta);
 
2
  import OpenAI from "openai";
3
  import { type ChatCompletionMessageParam } from "openai/resources/index";
4
  import { configSchematics } from "./config";
5
+ import { cancelJob, detectFlavor, fetchTurnInfo, pollSettlement, streamTurn, GatewayError } from "./gateway";
6
+ import { AbortedError, ensureServer, type ServerFlavor } from "./server";
7
 
8
+ // Per-backend default endpoints; QI35_LMSTUDIO_BASE_URL overrides both.
9
+ const DEFAULT_URLS: Record<ServerFlavor, string> = {
10
+ gateway: "http://127.0.0.1:12346/v1",
11
+ legacy: "http://127.0.0.1:12345/v1",
12
+ };
13
  const API_KEY = process.env.QI35_LMSTUDIO_API_KEY ?? "palw-local";
14
+ // How long the finished turn may wait for its replica verdict before deferring the
15
+ // answer to the next message's resolve frame. The replica is a real 35B re-execution;
16
+ // with the warm prefix cache it often lands within seconds, but never stall the chat.
17
+ const SETTLEMENT_POLL_MS = Number(process.env.QI35_SETTLEMENT_POLL_MS ?? 6_000);
18
+
19
+ function resolveBaseUrl(flavor: ServerFlavor): string {
20
+ return process.env.QI35_LMSTUDIO_BASE_URL ?? DEFAULT_URLS[flavor];
21
+ }
22
+
23
+ // The PALW backends stream their own live status blocks (receipt対象外) covering engine
24
+ // start, prefill, resolve and settlement. All of them — and the legacy placeholder — are
25
+ // stripped from history before resend so runtime decoration never re-enters a prompt.
26
  function withoutRuntimeStatus(content: string) {
27
  return content.replace(/<think>\s*PALW (?:runtime )?status \(receipt対象外\)[\s\S]*?<\/think>\s*/gi, "");
28
  }
 
39
  return messages;
40
  }
41
 
42
+ // LM Studio's automatic chat-title request ("…come up with a 2-5 word title…") arrives
43
+ // through the same generator. Detect it and answer over the STATELESS OpenAI path so it
44
+ // neither creates a session turn nor mints a receipt — a title is not a conversation.
45
+ function isTitleRequest(text: string): boolean {
46
+ return /\b\d+(?:\s*-\s*\d+)?\s*word title\b/i.test(text) || /\btitle for this conversation\b/i.test(text);
47
+ }
48
+
49
+ // One receipt対象外 status block: opens lazily on the first line, closes idempotently.
50
+ // Everything runtime-flavored goes through here so one history-strip regex covers it all.
51
+ class StatusBlock {
52
+ private opened = false;
53
+
54
+ public constructor(
55
+ private readonly parser: ReasoningStreamParser,
56
+ private readonly isAborted: () => boolean,
57
+ ) {}
58
+
59
+ public line(text: string) {
60
+ if (this.isAborted()) return;
61
+ if (!this.opened) {
62
+ this.opened = true;
63
+ this.parser.push("<think>PALW status (receipt対象外):\n");
64
+ }
65
+ this.parser.push(text + "\n");
66
+ }
67
+
68
+ public close() {
69
+ if (this.opened) {
70
+ this.opened = false;
71
+ this.parser.push("</think>\n");
72
+ }
73
+ }
74
+ }
75
+
76
  export async function generate(ctl: GeneratorController, history: Chat) {
77
  const config: InferParsedConfig<typeof configSchematics> = ctl.getPluginConfig(configSchematics);
78
+ let aborted = false;
79
+ ctl.onAborted(() => {
80
+ aborted = true;
81
+ });
82
+ const parser = new ReasoningStreamParser(ctl);
83
+ const status = new StatusBlock(parser, () => aborted);
84
+ const flavor = config.get("serverFlavor") as ServerFlavor;
85
+ const baseUrl = resolveBaseUrl(flavor);
86
+
87
+ if (config.get("autoStartServer")) {
88
+ try {
89
+ await ensureServer({
90
+ workspace: process.env.QI35_WORKSPACE ?? config.get("workspacePath").trim(),
91
+ baseUrl,
92
+ flavor,
93
+ emitStatus: (line) => status.line(line),
94
+ isAborted: () => aborted,
95
+ });
96
+ } catch (error) {
97
+ if (error instanceof AbortedError || aborted) return;
98
+ status.line(error instanceof Error ? error.message : String(error));
99
+ status.close();
100
+ parser.finish();
101
+ throw error;
102
+ }
103
+ }
104
+ if (aborted) return;
105
+
106
+ // Trust what actually answers the port over the configured flavor: an overridden
107
+ // BASE_URL may point either backend at either port.
108
+ const actual = (await detectFlavor(baseUrl)) ?? flavor;
109
+ const messages = toMessages(history);
110
+
111
+ if (actual === "legacy") {
112
+ status.close();
113
+ await openAiCompletion(ctl, parser, config.get("model"), baseUrl, messages, () => aborted);
114
+ return;
115
+ }
116
+
117
+ const trailingUser = [...messages].reverse().find((m) => m.role === "user");
118
+ const trailingText = typeof trailingUser?.content === "string" ? trailingUser.content : "";
119
+ if (isTitleRequest(trailingText)) {
120
+ status.close();
121
+ await openAiCompletion(ctl, parser, config.get("model"), baseUrl, messages, () => aborted);
122
+ return;
123
+ }
124
+
125
+ // ---- gateway path: one execution answers the screen AND becomes the PALW A-run ----
126
+ const privacyMode = config.get("privacyMode");
127
+ let jobId: string | null = null;
128
+ let turnId: string | null = null;
129
+ let promptTokens = 0;
130
+ let completed: { stop_reason: string; output_tokens: number; cache_hit_tokens: number; elapsed_ms: number } | null =
131
+ null;
132
+ const controller = new AbortController();
133
+ ctl.onAborted(() => {
134
+ controller.abort();
135
+ if (jobId !== null) void cancelJob(baseUrl, API_KEY, jobId);
136
  });
137
 
138
+ try {
139
+ await streamTurn(
140
+ baseUrl,
141
+ API_KEY,
142
+ messages as { role: string; content: string }[],
143
+ privacyMode,
144
+ {
145
+ onResolve: (frame) => {
146
+ const label =
147
+ frame.resolution === "continued"
148
+ ? `既存セッション継続 (一致 ${frame.matched_pairs} turn)`
149
+ : frame.resolution === "branched"
150
+ ? `分岐 (edit/regenerate、一致 ${frame.matched_pairs} turn)`
151
+ : frame.resolution === "imported"
152
+ ? `新規セッション (過去 ${frame.imported_turns} turn を import)`
153
+ : "新規セッション";
154
+ status.line(`会話復元: ${label} [${privacyMode}]`);
155
+ if (frame.parent !== null) {
156
+ const receipt = frame.parent.receipt_present ? " receipt有" : "";
157
+ status.line(`前turnのPALW検証: ${frame.parent.verification_status}${receipt}`);
158
+ }
159
+ },
160
+ onSearchStatus: (lines) => {
161
+ for (const line of lines) status.line(line);
162
+ },
163
+ onStarted: (info) => {
164
+ jobId = info.job_id;
165
+ turnId = info.turn_id;
166
+ promptTokens = info.prompt_tokens;
167
+ status.line(`prompt ${info.prompt_tokens} tokens — 生成開始`);
168
+ },
169
+ onContextBudget: (dropped) => status.line(`context予算: 古い ${dropped} turn を除外`),
170
+ onDelta: (utf8) => {
171
+ status.close();
172
+ parser.push(utf8);
173
+ },
174
+ onCompleted: (info) => {
175
+ completed = info;
176
+ },
177
+ },
178
+ controller.signal,
179
+ );
180
+ } catch (error) {
181
+ if (aborted) return;
182
+ const message = error instanceof GatewayError ? error.message : String(error);
183
+ status.line(message);
184
+ status.close();
185
+ parser.finish();
186
+ throw error;
187
+ }
188
+ if (aborted) return;
189
+
190
+ // Post-answer settlement block (receipt対象外): the answer clock is done, this shows
191
+ // how far the settlement clock got within its small budget. The pipeline continues in
192
+ // the gateway either way and the NEXT message's resolve frame reports the final state.
193
+ if (turnId !== null && completed !== null) {
194
+ const done: { stop_reason: string; output_tokens: number; cache_hit_tokens: number; elapsed_ms: number } = completed;
195
+ const wantsSettlement = privacyMode !== "local_only";
196
+ const info = wantsSettlement
197
+ ? await pollSettlement(baseUrl, API_KEY, turnId, SETTLEMENT_POLL_MS, () => aborted)
198
+ : await fetchTurnInfo(baseUrl, API_KEY, turnId);
199
+ const seconds = (done.elapsed_ms / 1000).toFixed(1);
200
+ status.line(
201
+ `生成完了: ${done.stop_reason} ${done.output_tokens} tok / cache_hit ${done.cache_hit_tokens}/${promptTokens} / ${seconds}s`,
202
+ );
203
+ if (info !== null) {
204
+ const pending = !["replica_matched", "certified", "matured", "mismatch", "mint_ineligible", "local_complete"].includes(
205
+ info.verification_status,
206
+ );
207
+ const suffix =
208
+ wantsSettlement && (pending || info.verification_status === "replica_pending")
209
+ ? "(バックグラウンド継続中 — 次メッセージで反映)"
210
+ : "";
211
+ status.line(`PALW検証: ${info.verification_status}${suffix}`);
212
+ if (info.receipt.present) {
213
+ const commitment = (info.receipt.output_commitment ?? "").slice(0, 16);
214
+ status.line(`receipt: ${info.receipt.schema ?? "?"} output_commitment=${commitment}…`);
215
+ }
216
+ if (info.search?.present) {
217
+ status.line(`検索bundle: sha256=${(info.search.bundle_sha256 ?? "").slice(0, 16)}… (${info.search.provider ?? "?"})`);
218
+ }
219
+ }
220
+ status.close();
221
+ }
222
+ parser.finish();
223
+ }
224
+
225
+ // Stateless OpenAI-compatible path: the legacy server, and gateway-side title requests.
226
+ async function openAiCompletion(
227
+ ctl: GeneratorController,
228
+ parser: ReasoningStreamParser,
229
+ model: string,
230
+ baseUrl: string,
231
+ messages: ChatCompletionMessageParam[],
232
+ isAborted: () => boolean,
233
+ ) {
234
+ const client = new OpenAI({ apiKey: API_KEY, baseURL: baseUrl, timeout: 30 * 60 * 1000 });
235
+ const stream = await client.chat.completions.create({ model, messages, stream: true });
236
+ if (isAborted()) {
237
+ stream.controller.abort();
238
+ return;
239
+ }
240
  ctl.onAborted(() => stream.controller.abort());
 
241
  for await (const chunk of stream) {
242
  const delta = chunk.choices[0]?.delta?.content;
243
  if (delta) parser.push(delta);
infra/lmstudio/palw-receipt-adapter/src/server.ts CHANGED
@@ -9,7 +9,12 @@ import * as path from "path";
9
  export const SERVER_LOG =
10
  process.env.QI35_LMSTUDIO_SERVER_LOG ??
11
  path.join(homedir(), ".config", "misaka-palw", "lmstudio-server.log");
12
- const LAUNCHER_REL = path.join("docs", "evidence", "qi35_lmstudio_server.sh");
 
 
 
 
 
13
  const HEALTH_TIMEOUT_MS = 1_500;
14
  const POLL_INTERVAL_MS = 1_000;
15
  // Cold start may include `colima start` plus the SearXNG compose stack.
@@ -22,6 +27,7 @@ const PROGRESS_EVERY_MS = 5_000;
22
  export interface EnsureServerOptions {
23
  workspace: string;
24
  baseUrl: string;
 
25
  emitStatus: (line: string) => void;
26
  isAborted: () => boolean;
27
  }
@@ -74,8 +80,8 @@ interface LauncherExit {
74
  message?: string;
75
  }
76
 
77
- function spawnLauncher(workspace: string, exitState: LauncherExit): void {
78
- const launcher = path.join(workspace, LAUNCHER_REL);
79
  if (!existsSync(launcher)) {
80
  throw new ServerStartError(`launcher not found: ${launcher} (workspace 設定を確認)`);
81
  }
@@ -113,11 +119,12 @@ function spawnLauncher(workspace: string, exitState: LauncherExit): void {
113
  async function startAndWait(
114
  workspace: string,
115
  baseUrl: string,
 
116
  emitStatus: (line: string) => void,
117
  ): Promise<void> {
118
- emitStatus(`エンジンサーバ自動起動中 (${LAUNCHER_REL})…`);
119
  const exitState: LauncherExit = { at: 0, code: null };
120
- spawnLauncher(workspace, exitState);
121
 
122
  const startedAt = Date.now();
123
  let lastProgressAt = 0;
@@ -157,7 +164,7 @@ let inflight: Promise<void> | null = null;
157
  export async function ensureServer(options: EnsureServerOptions): Promise<void> {
158
  if (await healthy(options.baseUrl)) return;
159
  if (inflight === null) {
160
- const attempt = startAndWait(options.workspace, options.baseUrl, options.emitStatus).finally(
161
  () => {
162
  inflight = null;
163
  },
 
9
  export const SERVER_LOG =
10
  process.env.QI35_LMSTUDIO_SERVER_LOG ??
11
  path.join(homedir(), ".config", "misaka-palw", "lmstudio-server.log");
12
+ /// Launchers per backend flavor, both under docs/evidence in the workspace.
13
+ export const LAUNCHERS = {
14
+ gateway: path.join("docs", "evidence", "qi35_lmstudio_gateway.sh"),
15
+ legacy: path.join("docs", "evidence", "qi35_lmstudio_server.sh"),
16
+ } as const;
17
+ export type ServerFlavor = keyof typeof LAUNCHERS;
18
  const HEALTH_TIMEOUT_MS = 1_500;
19
  const POLL_INTERVAL_MS = 1_000;
20
  // Cold start may include `colima start` plus the SearXNG compose stack.
 
27
  export interface EnsureServerOptions {
28
  workspace: string;
29
  baseUrl: string;
30
+ flavor: ServerFlavor;
31
  emitStatus: (line: string) => void;
32
  isAborted: () => boolean;
33
  }
 
80
  message?: string;
81
  }
82
 
83
+ function spawnLauncher(workspace: string, flavor: ServerFlavor, exitState: LauncherExit): void {
84
+ const launcher = path.join(workspace, LAUNCHERS[flavor]);
85
  if (!existsSync(launcher)) {
86
  throw new ServerStartError(`launcher not found: ${launcher} (workspace 設定を確認)`);
87
  }
 
119
  async function startAndWait(
120
  workspace: string,
121
  baseUrl: string,
122
+ flavor: ServerFlavor,
123
  emitStatus: (line: string) => void,
124
  ): Promise<void> {
125
+ emitStatus(`エンジンサーバ自動起動中 (${LAUNCHERS[flavor]})…`);
126
  const exitState: LauncherExit = { at: 0, code: null };
127
+ spawnLauncher(workspace, flavor, exitState);
128
 
129
  const startedAt = Date.now();
130
  let lastProgressAt = 0;
 
164
  export async function ensureServer(options: EnsureServerOptions): Promise<void> {
165
  if (await healthy(options.baseUrl)) return;
166
  if (inflight === null) {
167
+ const attempt = startAndWait(options.workspace, options.baseUrl, options.flavor, options.emitStatus).finally(
168
  () => {
169
  inflight = null;
170
  },
palw-gateway/Cargo.lock ADDED
@@ -0,0 +1,879 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # This file is automatically @generated by Cargo.
2
+ # It is not intended for manual editing.
3
+ version = 4
4
+
5
+ [[package]]
6
+ name = "ahash"
7
+ version = "0.8.12"
8
+ source = "registry+https://github.com/rust-lang/crates.io-index"
9
+ checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
10
+ dependencies = [
11
+ "cfg-if",
12
+ "getrandom",
13
+ "once_cell",
14
+ "serde",
15
+ "version_check",
16
+ "zerocopy",
17
+ ]
18
+
19
+ [[package]]
20
+ name = "aho-corasick"
21
+ version = "1.1.4"
22
+ source = "registry+https://github.com/rust-lang/crates.io-index"
23
+ checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
24
+ dependencies = [
25
+ "memchr",
26
+ ]
27
+
28
+ [[package]]
29
+ name = "ascii"
30
+ version = "1.1.0"
31
+ source = "registry+https://github.com/rust-lang/crates.io-index"
32
+ checksum = "d92bec98840b8f03a5ff5413de5293bfcd8bf96467cf5452609f939ec6f5de16"
33
+
34
+ [[package]]
35
+ name = "base64"
36
+ version = "0.13.1"
37
+ source = "registry+https://github.com/rust-lang/crates.io-index"
38
+ checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8"
39
+
40
+ [[package]]
41
+ name = "bitflags"
42
+ version = "2.13.1"
43
+ source = "registry+https://github.com/rust-lang/crates.io-index"
44
+ checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da"
45
+
46
+ [[package]]
47
+ name = "blake2"
48
+ version = "0.10.6"
49
+ source = "registry+https://github.com/rust-lang/crates.io-index"
50
+ checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe"
51
+ dependencies = [
52
+ "digest",
53
+ ]
54
+
55
+ [[package]]
56
+ name = "block-buffer"
57
+ version = "0.10.4"
58
+ source = "registry+https://github.com/rust-lang/crates.io-index"
59
+ checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
60
+ dependencies = [
61
+ "generic-array",
62
+ ]
63
+
64
+ [[package]]
65
+ name = "castaway"
66
+ version = "0.2.4"
67
+ source = "registry+https://github.com/rust-lang/crates.io-index"
68
+ checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a"
69
+ dependencies = [
70
+ "rustversion",
71
+ ]
72
+
73
+ [[package]]
74
+ name = "cc"
75
+ version = "1.4.0"
76
+ source = "registry+https://github.com/rust-lang/crates.io-index"
77
+ checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9"
78
+ dependencies = [
79
+ "find-msvc-tools",
80
+ "shlex",
81
+ ]
82
+
83
+ [[package]]
84
+ name = "cfg-if"
85
+ version = "1.0.4"
86
+ source = "registry+https://github.com/rust-lang/crates.io-index"
87
+ checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
88
+
89
+ [[package]]
90
+ name = "chunked_transfer"
91
+ version = "1.5.0"
92
+ source = "registry+https://github.com/rust-lang/crates.io-index"
93
+ checksum = "6e4de3bc4ea267985becf712dc6d9eed8b04c953b3fcfb339ebc87acd9804901"
94
+
95
+ [[package]]
96
+ name = "compact_str"
97
+ version = "0.9.1"
98
+ source = "registry+https://github.com/rust-lang/crates.io-index"
99
+ checksum = "9dfdd1c2274d9aa354115b09dc9a901d6c5576818cdf70d14cae2bdb47df00ab"
100
+ dependencies = [
101
+ "castaway",
102
+ "cfg-if",
103
+ "itoa",
104
+ "rustversion",
105
+ "ryu",
106
+ "serde",
107
+ "static_assertions",
108
+ ]
109
+
110
+ [[package]]
111
+ name = "crossbeam-deque"
112
+ version = "0.8.7"
113
+ source = "registry+https://github.com/rust-lang/crates.io-index"
114
+ checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb"
115
+ dependencies = [
116
+ "crossbeam-epoch",
117
+ "crossbeam-utils",
118
+ ]
119
+
120
+ [[package]]
121
+ name = "crossbeam-epoch"
122
+ version = "0.9.20"
123
+ source = "registry+https://github.com/rust-lang/crates.io-index"
124
+ checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
125
+ dependencies = [
126
+ "crossbeam-utils",
127
+ ]
128
+
129
+ [[package]]
130
+ name = "crossbeam-utils"
131
+ version = "0.8.22"
132
+ source = "registry+https://github.com/rust-lang/crates.io-index"
133
+ checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17"
134
+
135
+ [[package]]
136
+ name = "crypto-common"
137
+ version = "0.1.7"
138
+ source = "registry+https://github.com/rust-lang/crates.io-index"
139
+ checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
140
+ dependencies = [
141
+ "generic-array",
142
+ "typenum",
143
+ ]
144
+
145
+ [[package]]
146
+ name = "darling"
147
+ version = "0.20.11"
148
+ source = "registry+https://github.com/rust-lang/crates.io-index"
149
+ checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"
150
+ dependencies = [
151
+ "darling_core",
152
+ "darling_macro",
153
+ ]
154
+
155
+ [[package]]
156
+ name = "darling_core"
157
+ version = "0.20.11"
158
+ source = "registry+https://github.com/rust-lang/crates.io-index"
159
+ checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"
160
+ dependencies = [
161
+ "fnv",
162
+ "ident_case",
163
+ "proc-macro2",
164
+ "quote",
165
+ "strsim",
166
+ "syn 2.0.119",
167
+ ]
168
+
169
+ [[package]]
170
+ name = "darling_macro"
171
+ version = "0.20.11"
172
+ source = "registry+https://github.com/rust-lang/crates.io-index"
173
+ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"
174
+ dependencies = [
175
+ "darling_core",
176
+ "quote",
177
+ "syn 2.0.119",
178
+ ]
179
+
180
+ [[package]]
181
+ name = "dary_heap"
182
+ version = "0.3.9"
183
+ source = "registry+https://github.com/rust-lang/crates.io-index"
184
+ checksum = "8b1e3a325bc115f096c8b77bbf027a7c2592230e70be2d985be950d3d5e60ebe"
185
+ dependencies = [
186
+ "serde",
187
+ ]
188
+
189
+ [[package]]
190
+ name = "derive_builder"
191
+ version = "0.20.2"
192
+ source = "registry+https://github.com/rust-lang/crates.io-index"
193
+ checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"
194
+ dependencies = [
195
+ "derive_builder_macro",
196
+ ]
197
+
198
+ [[package]]
199
+ name = "derive_builder_core"
200
+ version = "0.20.2"
201
+ source = "registry+https://github.com/rust-lang/crates.io-index"
202
+ checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"
203
+ dependencies = [
204
+ "darling",
205
+ "proc-macro2",
206
+ "quote",
207
+ "syn 2.0.119",
208
+ ]
209
+
210
+ [[package]]
211
+ name = "derive_builder_macro"
212
+ version = "0.20.2"
213
+ source = "registry+https://github.com/rust-lang/crates.io-index"
214
+ checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"
215
+ dependencies = [
216
+ "derive_builder_core",
217
+ "syn 2.0.119",
218
+ ]
219
+
220
+ [[package]]
221
+ name = "digest"
222
+ version = "0.10.7"
223
+ source = "registry+https://github.com/rust-lang/crates.io-index"
224
+ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
225
+ dependencies = [
226
+ "block-buffer",
227
+ "crypto-common",
228
+ "subtle",
229
+ ]
230
+
231
+ [[package]]
232
+ name = "either"
233
+ version = "1.17.0"
234
+ source = "registry+https://github.com/rust-lang/crates.io-index"
235
+ checksum = "9e5e8f6c15a24b9a3ee5efec809ccd006d3b30e8b3bb63c39af737c7f87daa1d"
236
+
237
+ [[package]]
238
+ name = "esaxx-rs"
239
+ version = "0.1.10"
240
+ source = "registry+https://github.com/rust-lang/crates.io-index"
241
+ checksum = "d817e038c30374a4bcb22f94d0a8a0e216958d4c3dcde369b1439fec4bdda6e6"
242
+
243
+ [[package]]
244
+ name = "fallible-iterator"
245
+ version = "0.3.0"
246
+ source = "registry+https://github.com/rust-lang/crates.io-index"
247
+ checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649"
248
+
249
+ [[package]]
250
+ name = "fallible-streaming-iterator"
251
+ version = "0.1.9"
252
+ source = "registry+https://github.com/rust-lang/crates.io-index"
253
+ checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
254
+
255
+ [[package]]
256
+ name = "find-msvc-tools"
257
+ version = "0.1.9"
258
+ source = "registry+https://github.com/rust-lang/crates.io-index"
259
+ checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
260
+
261
+ [[package]]
262
+ name = "fnv"
263
+ version = "1.0.7"
264
+ source = "registry+https://github.com/rust-lang/crates.io-index"
265
+ checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
266
+
267
+ [[package]]
268
+ name = "generic-array"
269
+ version = "0.14.7"
270
+ source = "registry+https://github.com/rust-lang/crates.io-index"
271
+ checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
272
+ dependencies = [
273
+ "typenum",
274
+ "version_check",
275
+ ]
276
+
277
+ [[package]]
278
+ name = "getrandom"
279
+ version = "0.3.4"
280
+ source = "registry+https://github.com/rust-lang/crates.io-index"
281
+ checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
282
+ dependencies = [
283
+ "cfg-if",
284
+ "libc",
285
+ "r-efi",
286
+ "wasip2",
287
+ ]
288
+
289
+ [[package]]
290
+ name = "hashbrown"
291
+ version = "0.14.5"
292
+ source = "registry+https://github.com/rust-lang/crates.io-index"
293
+ checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
294
+ dependencies = [
295
+ "ahash",
296
+ ]
297
+
298
+ [[package]]
299
+ name = "hashlink"
300
+ version = "0.9.1"
301
+ source = "registry+https://github.com/rust-lang/crates.io-index"
302
+ checksum = "6ba4ff7128dee98c7dc9794b6a411377e1404dba1c97deb8d1a55297bd25d8af"
303
+ dependencies = [
304
+ "hashbrown",
305
+ ]
306
+
307
+ [[package]]
308
+ name = "httpdate"
309
+ version = "1.0.3"
310
+ source = "registry+https://github.com/rust-lang/crates.io-index"
311
+ checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
312
+
313
+ [[package]]
314
+ name = "ident_case"
315
+ version = "1.0.1"
316
+ source = "registry+https://github.com/rust-lang/crates.io-index"
317
+ checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
318
+
319
+ [[package]]
320
+ name = "itertools"
321
+ version = "0.14.0"
322
+ source = "registry+https://github.com/rust-lang/crates.io-index"
323
+ checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
324
+ dependencies = [
325
+ "either",
326
+ ]
327
+
328
+ [[package]]
329
+ name = "itoa"
330
+ version = "1.0.18"
331
+ source = "registry+https://github.com/rust-lang/crates.io-index"
332
+ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
333
+
334
+ [[package]]
335
+ name = "libc"
336
+ version = "0.2.189"
337
+ source = "registry+https://github.com/rust-lang/crates.io-index"
338
+ checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
339
+
340
+ [[package]]
341
+ name = "libsqlite3-sys"
342
+ version = "0.28.0"
343
+ source = "registry+https://github.com/rust-lang/crates.io-index"
344
+ checksum = "0c10584274047cb335c23d3e61bcef8e323adae7c5c8c760540f73610177fc3f"
345
+ dependencies = [
346
+ "cc",
347
+ "pkg-config",
348
+ "vcpkg",
349
+ ]
350
+
351
+ [[package]]
352
+ name = "log"
353
+ version = "0.4.33"
354
+ source = "registry+https://github.com/rust-lang/crates.io-index"
355
+ checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
356
+
357
+ [[package]]
358
+ name = "macro_rules_attribute"
359
+ version = "0.2.3"
360
+ source = "registry+https://github.com/rust-lang/crates.io-index"
361
+ checksum = "b3ae8f6d608c795738406608304d30a2dfbdc8e58e44f7ba43236da5208ded3c"
362
+ dependencies = [
363
+ "macro_rules_attribute-proc_macro",
364
+ "pastey",
365
+ ]
366
+
367
+ [[package]]
368
+ name = "macro_rules_attribute-proc_macro"
369
+ version = "0.2.3"
370
+ source = "registry+https://github.com/rust-lang/crates.io-index"
371
+ checksum = "fc04a4c58212d57930a24bf47d3fa87485264a3a054e9c10e042eb373573ad3c"
372
+
373
+ [[package]]
374
+ name = "memchr"
375
+ version = "2.8.3"
376
+ source = "registry+https://github.com/rust-lang/crates.io-index"
377
+ checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
378
+
379
+ [[package]]
380
+ name = "minimal-lexical"
381
+ version = "0.2.1"
382
+ source = "registry+https://github.com/rust-lang/crates.io-index"
383
+ checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
384
+
385
+ [[package]]
386
+ name = "monostate"
387
+ version = "0.1.18"
388
+ source = "registry+https://github.com/rust-lang/crates.io-index"
389
+ checksum = "3341a273f6c9d5bef1908f17b7267bbab0e95c9bf69a0d4dcf8e9e1b2c76ef67"
390
+ dependencies = [
391
+ "monostate-impl",
392
+ "serde",
393
+ "serde_core",
394
+ ]
395
+
396
+ [[package]]
397
+ name = "monostate-impl"
398
+ version = "0.1.18"
399
+ source = "registry+https://github.com/rust-lang/crates.io-index"
400
+ checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9"
401
+ dependencies = [
402
+ "proc-macro2",
403
+ "quote",
404
+ "syn 2.0.119",
405
+ ]
406
+
407
+ [[package]]
408
+ name = "nom"
409
+ version = "7.1.3"
410
+ source = "registry+https://github.com/rust-lang/crates.io-index"
411
+ checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
412
+ dependencies = [
413
+ "memchr",
414
+ "minimal-lexical",
415
+ ]
416
+
417
+ [[package]]
418
+ name = "once_cell"
419
+ version = "1.21.4"
420
+ source = "registry+https://github.com/rust-lang/crates.io-index"
421
+ checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
422
+
423
+ [[package]]
424
+ name = "onig"
425
+ version = "6.5.3"
426
+ source = "registry+https://github.com/rust-lang/crates.io-index"
427
+ checksum = "0cc3cbf698f9438986c11a880c90a6d04b9de27575afd28bbf45b154b6c709e2"
428
+ dependencies = [
429
+ "bitflags",
430
+ "libc",
431
+ "once_cell",
432
+ "onig_sys",
433
+ ]
434
+
435
+ [[package]]
436
+ name = "onig_sys"
437
+ version = "69.9.3"
438
+ source = "registry+https://github.com/rust-lang/crates.io-index"
439
+ checksum = "1e68317604e77e53b85896388e1a803c1d21b74c899ec9e5e1112db90735edd7"
440
+ dependencies = [
441
+ "cc",
442
+ "pkg-config",
443
+ ]
444
+
445
+ [[package]]
446
+ name = "palw-gateway"
447
+ version = "0.1.0"
448
+ dependencies = [
449
+ "blake2",
450
+ "rusqlite",
451
+ "serde",
452
+ "serde_json",
453
+ "tiny_http",
454
+ "tokenizers",
455
+ ]
456
+
457
+ [[package]]
458
+ name = "paste"
459
+ version = "1.0.15"
460
+ source = "registry+https://github.com/rust-lang/crates.io-index"
461
+ checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
462
+
463
+ [[package]]
464
+ name = "pastey"
465
+ version = "0.2.3"
466
+ source = "registry+https://github.com/rust-lang/crates.io-index"
467
+ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
468
+
469
+ [[package]]
470
+ name = "pkg-config"
471
+ version = "0.3.33"
472
+ source = "registry+https://github.com/rust-lang/crates.io-index"
473
+ checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
474
+
475
+ [[package]]
476
+ name = "ppv-lite86"
477
+ version = "0.2.21"
478
+ source = "registry+https://github.com/rust-lang/crates.io-index"
479
+ checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
480
+ dependencies = [
481
+ "zerocopy",
482
+ ]
483
+
484
+ [[package]]
485
+ name = "proc-macro2"
486
+ version = "1.0.107"
487
+ source = "registry+https://github.com/rust-lang/crates.io-index"
488
+ checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
489
+ dependencies = [
490
+ "unicode-ident",
491
+ ]
492
+
493
+ [[package]]
494
+ name = "quote"
495
+ version = "1.0.47"
496
+ source = "registry+https://github.com/rust-lang/crates.io-index"
497
+ checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
498
+ dependencies = [
499
+ "proc-macro2",
500
+ ]
501
+
502
+ [[package]]
503
+ name = "r-efi"
504
+ version = "5.3.0"
505
+ source = "registry+https://github.com/rust-lang/crates.io-index"
506
+ checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
507
+
508
+ [[package]]
509
+ name = "rand"
510
+ version = "0.9.5"
511
+ source = "registry+https://github.com/rust-lang/crates.io-index"
512
+ checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"
513
+ dependencies = [
514
+ "rand_chacha",
515
+ "rand_core",
516
+ ]
517
+
518
+ [[package]]
519
+ name = "rand_chacha"
520
+ version = "0.9.0"
521
+ source = "registry+https://github.com/rust-lang/crates.io-index"
522
+ checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
523
+ dependencies = [
524
+ "ppv-lite86",
525
+ "rand_core",
526
+ ]
527
+
528
+ [[package]]
529
+ name = "rand_core"
530
+ version = "0.9.5"
531
+ source = "registry+https://github.com/rust-lang/crates.io-index"
532
+ checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
533
+ dependencies = [
534
+ "getrandom",
535
+ ]
536
+
537
+ [[package]]
538
+ name = "rayon"
539
+ version = "1.12.0"
540
+ source = "registry+https://github.com/rust-lang/crates.io-index"
541
+ checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d"
542
+ dependencies = [
543
+ "either",
544
+ "rayon-core",
545
+ ]
546
+
547
+ [[package]]
548
+ name = "rayon-cond"
549
+ version = "0.4.0"
550
+ source = "registry+https://github.com/rust-lang/crates.io-index"
551
+ checksum = "2964d0cf57a3e7a06e8183d14a8b527195c706b7983549cd5462d5aa3747438f"
552
+ dependencies = [
553
+ "either",
554
+ "itertools",
555
+ "rayon",
556
+ ]
557
+
558
+ [[package]]
559
+ name = "rayon-core"
560
+ version = "1.13.0"
561
+ source = "registry+https://github.com/rust-lang/crates.io-index"
562
+ checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91"
563
+ dependencies = [
564
+ "crossbeam-deque",
565
+ "crossbeam-utils",
566
+ ]
567
+
568
+ [[package]]
569
+ name = "regex"
570
+ version = "1.13.1"
571
+ source = "registry+https://github.com/rust-lang/crates.io-index"
572
+ checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
573
+ dependencies = [
574
+ "aho-corasick",
575
+ "memchr",
576
+ "regex-automata",
577
+ "regex-syntax",
578
+ ]
579
+
580
+ [[package]]
581
+ name = "regex-automata"
582
+ version = "0.4.16"
583
+ source = "registry+https://github.com/rust-lang/crates.io-index"
584
+ checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad"
585
+ dependencies = [
586
+ "aho-corasick",
587
+ "memchr",
588
+ "regex-syntax",
589
+ ]
590
+
591
+ [[package]]
592
+ name = "regex-syntax"
593
+ version = "0.8.11"
594
+ source = "registry+https://github.com/rust-lang/crates.io-index"
595
+ checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
596
+
597
+ [[package]]
598
+ name = "rusqlite"
599
+ version = "0.31.0"
600
+ source = "registry+https://github.com/rust-lang/crates.io-index"
601
+ checksum = "b838eba278d213a8beaf485bd313fd580ca4505a00d5871caeb1457c55322cae"
602
+ dependencies = [
603
+ "bitflags",
604
+ "fallible-iterator",
605
+ "fallible-streaming-iterator",
606
+ "hashlink",
607
+ "libsqlite3-sys",
608
+ "smallvec",
609
+ ]
610
+
611
+ [[package]]
612
+ name = "rustversion"
613
+ version = "1.0.23"
614
+ source = "registry+https://github.com/rust-lang/crates.io-index"
615
+ checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
616
+
617
+ [[package]]
618
+ name = "ryu"
619
+ version = "1.0.23"
620
+ source = "registry+https://github.com/rust-lang/crates.io-index"
621
+ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
622
+
623
+ [[package]]
624
+ name = "serde"
625
+ version = "1.0.229"
626
+ source = "registry+https://github.com/rust-lang/crates.io-index"
627
+ checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
628
+ dependencies = [
629
+ "serde_core",
630
+ "serde_derive",
631
+ ]
632
+
633
+ [[package]]
634
+ name = "serde_core"
635
+ version = "1.0.229"
636
+ source = "registry+https://github.com/rust-lang/crates.io-index"
637
+ checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
638
+ dependencies = [
639
+ "serde_derive",
640
+ ]
641
+
642
+ [[package]]
643
+ name = "serde_derive"
644
+ version = "1.0.229"
645
+ source = "registry+https://github.com/rust-lang/crates.io-index"
646
+ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
647
+ dependencies = [
648
+ "proc-macro2",
649
+ "quote",
650
+ "syn 3.0.3",
651
+ ]
652
+
653
+ [[package]]
654
+ name = "serde_json"
655
+ version = "1.0.151"
656
+ source = "registry+https://github.com/rust-lang/crates.io-index"
657
+ checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
658
+ dependencies = [
659
+ "itoa",
660
+ "memchr",
661
+ "serde",
662
+ "serde_core",
663
+ "zmij",
664
+ ]
665
+
666
+ [[package]]
667
+ name = "shlex"
668
+ version = "2.0.1"
669
+ source = "registry+https://github.com/rust-lang/crates.io-index"
670
+ checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
671
+
672
+ [[package]]
673
+ name = "smallvec"
674
+ version = "1.15.2"
675
+ source = "registry+https://github.com/rust-lang/crates.io-index"
676
+ checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"
677
+
678
+ [[package]]
679
+ name = "spm_precompiled"
680
+ version = "0.1.4"
681
+ source = "registry+https://github.com/rust-lang/crates.io-index"
682
+ checksum = "5851699c4033c63636f7ea4cf7b7c1f1bf06d0cc03cfb42e711de5a5c46cf326"
683
+ dependencies = [
684
+ "base64",
685
+ "nom",
686
+ "serde",
687
+ "unicode-segmentation",
688
+ ]
689
+
690
+ [[package]]
691
+ name = "static_assertions"
692
+ version = "1.1.0"
693
+ source = "registry+https://github.com/rust-lang/crates.io-index"
694
+ checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
695
+
696
+ [[package]]
697
+ name = "strsim"
698
+ version = "0.11.1"
699
+ source = "registry+https://github.com/rust-lang/crates.io-index"
700
+ checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
701
+
702
+ [[package]]
703
+ name = "subtle"
704
+ version = "2.6.1"
705
+ source = "registry+https://github.com/rust-lang/crates.io-index"
706
+ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
707
+
708
+ [[package]]
709
+ name = "syn"
710
+ version = "2.0.119"
711
+ source = "registry+https://github.com/rust-lang/crates.io-index"
712
+ checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
713
+ dependencies = [
714
+ "proc-macro2",
715
+ "quote",
716
+ "unicode-ident",
717
+ ]
718
+
719
+ [[package]]
720
+ name = "syn"
721
+ version = "3.0.3"
722
+ source = "registry+https://github.com/rust-lang/crates.io-index"
723
+ checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3"
724
+ dependencies = [
725
+ "proc-macro2",
726
+ "quote",
727
+ "unicode-ident",
728
+ ]
729
+
730
+ [[package]]
731
+ name = "thiserror"
732
+ version = "2.0.19"
733
+ source = "registry+https://github.com/rust-lang/crates.io-index"
734
+ checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9"
735
+ dependencies = [
736
+ "thiserror-impl",
737
+ ]
738
+
739
+ [[package]]
740
+ name = "thiserror-impl"
741
+ version = "2.0.19"
742
+ source = "registry+https://github.com/rust-lang/crates.io-index"
743
+ checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd"
744
+ dependencies = [
745
+ "proc-macro2",
746
+ "quote",
747
+ "syn 3.0.3",
748
+ ]
749
+
750
+ [[package]]
751
+ name = "tiny_http"
752
+ version = "0.12.0"
753
+ source = "registry+https://github.com/rust-lang/crates.io-index"
754
+ checksum = "389915df6413a2e74fb181895f933386023c71110878cd0825588928e64cdc82"
755
+ dependencies = [
756
+ "ascii",
757
+ "chunked_transfer",
758
+ "httpdate",
759
+ "log",
760
+ ]
761
+
762
+ [[package]]
763
+ name = "tokenizers"
764
+ version = "0.21.4"
765
+ source = "registry+https://github.com/rust-lang/crates.io-index"
766
+ checksum = "a620b996116a59e184c2fa2dfd8251ea34a36d0a514758c6f966386bd2e03476"
767
+ dependencies = [
768
+ "ahash",
769
+ "aho-corasick",
770
+ "compact_str",
771
+ "dary_heap",
772
+ "derive_builder",
773
+ "esaxx-rs",
774
+ "getrandom",
775
+ "itertools",
776
+ "log",
777
+ "macro_rules_attribute",
778
+ "monostate",
779
+ "onig",
780
+ "paste",
781
+ "rand",
782
+ "rayon",
783
+ "rayon-cond",
784
+ "regex",
785
+ "regex-syntax",
786
+ "serde",
787
+ "serde_json",
788
+ "spm_precompiled",
789
+ "thiserror",
790
+ "unicode-normalization-alignments",
791
+ "unicode-segmentation",
792
+ "unicode_categories",
793
+ ]
794
+
795
+ [[package]]
796
+ name = "typenum"
797
+ version = "1.20.1"
798
+ source = "registry+https://github.com/rust-lang/crates.io-index"
799
+ checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
800
+
801
+ [[package]]
802
+ name = "unicode-ident"
803
+ version = "1.0.24"
804
+ source = "registry+https://github.com/rust-lang/crates.io-index"
805
+ checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
806
+
807
+ [[package]]
808
+ name = "unicode-normalization-alignments"
809
+ version = "0.1.12"
810
+ source = "registry+https://github.com/rust-lang/crates.io-index"
811
+ checksum = "43f613e4fa046e69818dd287fdc4bc78175ff20331479dab6e1b0f98d57062de"
812
+ dependencies = [
813
+ "smallvec",
814
+ ]
815
+
816
+ [[package]]
817
+ name = "unicode-segmentation"
818
+ version = "1.13.3"
819
+ source = "registry+https://github.com/rust-lang/crates.io-index"
820
+ checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8"
821
+
822
+ [[package]]
823
+ name = "unicode_categories"
824
+ version = "0.1.1"
825
+ source = "registry+https://github.com/rust-lang/crates.io-index"
826
+ checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e"
827
+
828
+ [[package]]
829
+ name = "vcpkg"
830
+ version = "0.2.15"
831
+ source = "registry+https://github.com/rust-lang/crates.io-index"
832
+ checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
833
+
834
+ [[package]]
835
+ name = "version_check"
836
+ version = "0.9.5"
837
+ source = "registry+https://github.com/rust-lang/crates.io-index"
838
+ checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
839
+
840
+ [[package]]
841
+ name = "wasip2"
842
+ version = "1.0.4+wasi-0.2.12"
843
+ source = "registry+https://github.com/rust-lang/crates.io-index"
844
+ checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487"
845
+ dependencies = [
846
+ "wit-bindgen",
847
+ ]
848
+
849
+ [[package]]
850
+ name = "wit-bindgen"
851
+ version = "0.57.1"
852
+ source = "registry+https://github.com/rust-lang/crates.io-index"
853
+ checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e"
854
+
855
+ [[package]]
856
+ name = "zerocopy"
857
+ version = "0.8.55"
858
+ source = "registry+https://github.com/rust-lang/crates.io-index"
859
+ checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb"
860
+ dependencies = [
861
+ "zerocopy-derive",
862
+ ]
863
+
864
+ [[package]]
865
+ name = "zerocopy-derive"
866
+ version = "0.8.55"
867
+ source = "registry+https://github.com/rust-lang/crates.io-index"
868
+ checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb"
869
+ dependencies = [
870
+ "proc-macro2",
871
+ "quote",
872
+ "syn 2.0.119",
873
+ ]
874
+
875
+ [[package]]
876
+ name = "zmij"
877
+ version = "1.0.23"
878
+ source = "registry+https://github.com/rust-lang/crates.io-index"
879
+ checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
palw-gateway/Cargo.toml ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [package]
2
+ name = "palw-gateway"
3
+ version = "0.1.0"
4
+ edition = "2021"
5
+ # Matches runtime-palw's honest floor (see its Cargo.toml note).
6
+ rust-version = "1.85"
7
+ description = "PALW Desktop UX Phase 0 — resident streaming inference gateway + conversation session layer"
8
+ license = "Apache-2.0"
9
+
10
+ [workspace]
11
+
12
+ [[bin]]
13
+ name = "palw-gateway"
14
+ path = "src/main.rs"
15
+
16
+ [dependencies]
17
+ # HTTP with hand-rolled SSE: tiny_http is synchronous, dependency-light, and lets one
18
+ # thread own one response stream — exactly the shape a token stream needs. No async runtime.
19
+ tiny_http = "0.12"
20
+ # The session layer database. Bundled, like runtime-palw's state_store.
21
+ rusqlite = { version = "0.31", features = ["bundled"] }
22
+ serde = { version = "1", features = ["derive"] }
23
+ serde_json = "1"
24
+ # The HF tokenizer (reads the pinned tokenizer.json). The repo has NO Rust encoder today —
25
+ # the Explore survey confirmed decode-only — so this is a required new dependency.
26
+ tokenizers = { version = "0.21", default-features = false, features = ["onig"] }
27
+ blake2 = "0.10"
28
+
29
+ [profile.release]
30
+ opt-level = 2
palw-gateway/README.md ADDED
@@ -0,0 +1,223 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # palw-gateway — PALW Desktop UX Phase 0 + Phase 1 + Phase 2 worker + LM Studio bridge
2
+
3
+ The resident streaming inference gateway + Conversation Session Layer (Phase 0), plus the
4
+ Phase-1 product layers of `docs/palw-desktop-product-architecture.md`: Context Compiler v1,
5
+ project workspace, long-term memory, content-addressed artifacts with version chains, the
6
+ three-mode tool runtime with sandbox, the five-priority resource scheduler, and the
7
+ settlement-clock writer (verification transitions / mismatch cascade / certified head).
8
+ Replaces the session-less, globally-locked `docs/evidence/qi35_lmstudio_server.py`; reuses
9
+ the `qi35_model --serve` integer engine untouched (forward pass + KV snapshot + speculative
10
+ decode — the survey's "reuse core, rewrite serving loop" verdict).
11
+
12
+ ## What it adds (none of which existed in the repo)
13
+
14
+ - **Conversation Session Layer** (`store.rs`): turns, branches, dual heads
15
+ (`local_head`/`certified_head`), `TurnVerificationStatus` + `TurnPrivacyMode` columns
16
+ from day one so UX Phase 2 (PALW settlement) writes existing columns, never migrates.
17
+ SQLite hardened with the `runtime-palw/state_store.rs` pattern (own `application_id`,
18
+ WAL, `synchronous=FULL`, foreign keys).
19
+ - **Token-id-stable history** (`context.rs`): the fix for the live bug the survey found —
20
+ the old stack rebuilt each prompt by `decode(gen)`→retokenize, which is NOT id-stable,
21
+ so the engine's longest-prefix cache missed on every multi-turn exchange. Here history
22
+ is assembled by concatenating STORED ids across ChatML fragment boundaries (each starts
23
+ with a special token the BPE never merges across), so turn N+1's prompt literally begins
24
+ with turn N's ids and the cache HITS. Verified live: turn 2 reported
25
+ `cache_hit_tokens=83` of a 101-token prompt (~1.9 s vs turn 1's 12.7 s).
26
+ - **Display/model separation**: `display_text` (UI) and `prompt_ids`/`output_ids` (model)
27
+ are different columns, so receipt/search footers can never leak into the next prompt
28
+ (the other live bug the survey found).
29
+ - **`GenerationEventV1` SSE** (`events.rs`, `http.rs`): the design doc's frozen wire type.
30
+ Two clocks kept separate — answer latency (streamed tokens) vs settlement latency
31
+ (`TurnVerificationStatus`, owned by the Phase-2 supervisor).
32
+ - **Cancel = v1 semantics**: partial answer kept, turn marked `MintIneligible`
33
+ (B cannot reproduce the user's stop instant).
34
+
35
+ ## Phase 1 (this build)
36
+
37
+ - **Context Compiler v1** (`context.rs`): system region = base prompt + project
38
+ instructions (§10) + memory snapshot (§12), stable-first for cacheability; attachments
39
+ and executed tool results render into the TURN's user block (like pasted text), so
40
+ attaching a file to turn N never invalidates turns 1..N-1's prefix. Deterministic
41
+ keyword chunk selection (`keyword-chunks-v1`) bounds each attachment; what was included
42
+ (memory snapshot root, memory ids, chunk selections, policies, dropped turns) lands in
43
+ `context_meta_json` on the turn — the future receipt commitment surface. The exact
44
+ rendered block is persisted (`user_block_text`) so later compiles reproduce identical
45
+ ids; legacy v1 rows fold back to the raw user text, byte-identical.
46
+ - **Artifact store** (`artifact.rs` + schema): content-addressed blobs (blake2b-256,
47
+ temp+fsync+rename), verified reads, version chains with DERIVED version numbers (§8/§9),
48
+ parser policy `utf8-text-v1` (the only v1 parser; recorded per artifact).
49
+ - **Tool runtime + sandbox** (`tools.rs`, §6/§17): `DeterministicReplay` (exact-i128
50
+ calculator) and `SnapshotReadOnly` (read_file/list_dir) execute immediately and are
51
+ snapshotted as artifacts; `ExactlyOnceSideEffect` (write_file/run_command) is stored as
52
+ a PROPOSAL and only runs through approve → the store's atomic `approved→executing`
53
+ claim, so a side effect can never fire twice. Sandbox: canonicalize-then-scope-check
54
+ paths (symlink escape refused), workspace-relative writes only, ABSOLUTE-path command
55
+ allowlist (no PATH lookup), cleared child env, wall-clock timeout, output caps. Honest
56
+ v1 limit: no kernel-level network isolation for spawned commands — the allowlist is the
57
+ network policy.
58
+ - **Resource scheduler** (`scheduler.rs`, §14): five-priority queue in front of the
59
+ engine (foreground chat overtakes any queued background work; FIFO within a class), no
60
+ mid-generation preemption (engine has no stop channel — worst case = one background
61
+ generation), decode-rate EMA, and the SLA admission probe the Phase-2 replica worker
62
+ must call BEFORE accepting an assignment (refusing beats no-showing).
63
+ - **Settlement-clock writer** (`store.rs`, §4): legal-transition machine
64
+ (`local_complete → replica_pending → replica_matched → [audit_pending →] certified →
65
+ matured`, mismatch reachable from the replica/audit states), LocalOnly turns barred from
66
+ the replica pipeline, mismatch cascades `MintIneligible` to every descendant (local
67
+ conversation survives; the reward dependency chain does not), `certified_head` = deepest
68
+ turn whose whole prefix is certified — out-of-order certifications do not advance it
69
+ past a gap.
70
+ - **Schema v2 migration**: additive, in-place, v1 databases keep serving byte-identical
71
+ history.
72
+ - **Streaming termination fix** (latent Phase-0 bug): SSE bodies are now
73
+ `Transfer-Encoding: chunked` with a terminating zero-chunk. tiny_http's `into_writer`
74
+ never closes the socket on drop (it parks the connection for keep-alive), so the old
75
+ bare-framed stream left generic clients (curl, fetch, EventSource) hanging for EOF after
76
+ `[DONE]`; the plugin never noticed because it aborts the connection itself. The zero-chunk
77
+ ends the response on the wire and the connection stays legitimately reusable.
78
+
79
+ ## Phase 2: background worker (§15)
80
+
81
+ `worker.rs` is a STATELESS RECONCILER — no durable worker state, the session store is the
82
+ single source of truth, and every cycle recomputes the work:
83
+
84
+ 1. **A-commits**: turns `local_complete` ∧ privacy ∈ {verified_no_mint, palw_mint} →
85
+ `submit_job` (idempotent by job id) → `replica_pending`. Crash between submit and the
86
+ store write heals on the next cycle's re-submit; §15's restart recovery is structural,
87
+ not a feature.
88
+ 2. **Verdicts**: turns in the pipeline → `fetch_verdicts` → legal transitions only
89
+ (`steps_to` walks intermediate states, e.g. a `certified` verdict against a
90
+ `replica_pending` turn applies matched→certified; stale verdicts are ignored).
91
+ 3. **B replicas**: `fetch_assignments` → §14 admission probe FIRST (refuse ⇒ decline with
92
+ reason — refusing beats no-showing) → §5-v1 Full Context replay on the engine at
93
+ `ReplicaJob` priority → result root.
94
+
95
+ The coordinator is a TRAIT (`coordinator.rs`) with two implementations, because the real
96
+ counterparty — the MISAKA node's PALW surface — is not finished:
97
+
98
+ - **`LoopbackCoordinator`** (enable with `--palw-loopback`): a complete in-process
99
+ coordinator (submit → assign → result → matched → certified; mismatch; deadline
100
+ requeue; decline requeue), also SERVED over HTTP at `/v1/palw/loopback/*` so a second
101
+ gateway instance can run a genuine two-instance A/B loop against it. It is a dev
102
+ harness, NOT consensus: no beacons, bonds, DA retention, or rewards, and its verdicts
103
+ certify plumbing, not economics. Single-instance self-replica (default on) shares the
104
+ engine and its prefix cache — it validates the pipeline, never replica independence.
105
+ `Matured` is chain state (coinbase maturity) and is never synthesized here.
106
+ - **`HttpCoordinator`** (`--palw-coordinator <base-url>`): client half of the same JSON
107
+ protocol — point it at another gateway's loopback, or at the node-side bridge once the
108
+ protocol features land there. Plain HTTP/1.1, `Connection: close`, no TLS: a
109
+ localhost/LAN dev transport that the production node binding replaces.
110
+
111
+ ### PALW coordinator protocol (JSON, v1)
112
+
113
+ ```
114
+ POST {base}/jobs {job_id, provider_id, prompt_ids, max_new, output_root,
115
+ receipt_json?, runtime_roots?: {route, kv, state}}
116
+ → {accepted:true} (idempotent by job_id)
117
+ POST {base}/verdicts {job_ids:[…]}
118
+ → {verdicts:[{job_id, verdict: replica_matched|certified|mismatch}]}
119
+ (pending jobs are absent; matched is reported exactly once,
120
+ then promotes to certified)
121
+ GET {base}/assignments?provider_id=X
122
+ → {assignments:[{job_id, prompt_ids, max_new, deadline_unix_ms}]}
123
+ (claimed on fetch; deadline lapse or decline requeues)
124
+ POST {base}/assignments/{job_id}/decline {provider_id, reason} → {declined:true}
125
+ POST {base}/replica-results {job_id, provider_id, output_root, runtime_roots?}
126
+ → {recorded:true}
127
+ (output_root = blake2b-256 over little-endian u32 output ids)
128
+ ```
129
+
130
+ `runtime_roots` are the engine's per-generation execution commitments (`ROOTS route= kv=
131
+ state=`, emitted under `QI35_SERVE_ROOTS`, which the engine host always sets): the MoE
132
+ routing-trace root, the KV-cache root, and the recurrent-state root. When a submitter
133
+ commits them, an honest replica must reproduce them — the match then covers execution
134
+ STRUCTURE, not just output ids. The loopback matches leniently when the submitter omitted
135
+ them; the node-side bridge (below) refuses rootless submissions outright.
136
+
137
+ **The node-side server of this protocol is `misaka-palw-bridge`** (MisakaLLM workspace,
138
+ `mil/bridge`): durable hash-chained journal, submitter ≠ replica enforced with no
139
+ opt-out, and the match decided by the node's real eight-field
140
+ `ReplicaMatchKey`/`run_replica_k2` (ADR-0039 §7.5) built over these roots. Point
141
+ `--palw-coordinator http://host:26621/palw/v1` at it. A permanent protocol rejection
142
+ (HTTP 400, e.g. a rootless pre-v3 turn against the class-strict bridge) marks the turn
143
+ `mint_ineligible` instead of retrying forever.
144
+
145
+ Verified end-to-end on real hardware: a `palw_mint` chat turn answered by the 35B engine
146
+ in 11.2 s (model load included), then in the background: A-commit → self-assigned replica
147
+ re-executed on the engine → roots matched → `certified` → `certified_head` advanced —
148
+ answer clock and settlement clock visibly separate. Cross-process: two instances over
149
+ real HTTP, with the replica root computed independently (Python blake2b) matching the
150
+ Rust root, drove `replica_pending → replica_matched → certified` on the A side.
151
+
152
+ ## Endpoints
153
+
154
+ - `GET /healthz`, `GET /v1/models` — supervisor probes (the LM Studio plugin's
155
+ `ensureServer()` polls `/healthz` unchanged; `engine_resident` in the body is also how the
156
+ plugin distinguishes this gateway from the legacy python server).
157
+ - `POST /v1/chat/completions` — OpenAI-compatible SSE, STATELESS (client owns transcript).
158
+ - `POST /v1/lmstudio/turn` — the LM Studio bridge (`lmstudio.rs`): body
159
+ `{messages, privacy_mode?}`; the client-owned transcript is resolved onto the session store
160
+ BY CONTENT (normalized text match, `<think>`/status blocks stripped) —
161
+ `continued` on the active branch (prefix cache hits), `branched` sibling on edit/regenerate,
162
+ or a new conversation with foreign history imported as `mint_ineligible` turns — then the
163
+ turn runs the NATIVE pipeline (id-stable prompt, receipt, ROOTS, settlement). The
164
+ `GenerationEventV1` stream is prefixed with one `lmstudio_resolve` frame that carries the
165
+ resolution and the parent turn's current settlement state.
166
+ - `GET /v1/turns/{id}` — settlement status + receipt summary + runtime roots (the plugin's
167
+ post-answer poll target); `GET /v1/turns/{id}/receipt` — the engine receipt JSON verbatim.
168
+ - `--receipt-dir <dir>`: every turn with a receipt also writes
169
+ `turn-<turn_id>-<oc16>.receipt.json` + `.opening.json` sidecars (0600; opening =
170
+ `qi35-gateway-opening/v1` with exact prompt/output ids + tokenizer digest + roots), the file
171
+ form offline verifiers consume — plus `.search.json` (bundle + `receipt_binding`) when the
172
+ turn searched.
173
+ - `--search-cmd <program>`: GPT-style live search on the bridge path via a stdin/stdout JSON
174
+ sidecar (`docs/evidence/qi35_search_sidecar.py` reuses the whole legacy stack — routing,
175
+ SSRF-guarded page fetch, canonical `live_search_bundle_v1`, typed failure bundles).
176
+ Fail-open: sidecar-level errors degrade to a searchless turn. The rendered evidence joins
177
+ the TURN's user block (`user + "\n\n" + evidence`), so `prompt_commitment` covers the
178
+ bundle digest transitively (no receipt schema change); the bundle persists on the turn
179
+ (schema v4 `turns.search_bundle_json`) and streams to the client as a `search_status`
180
+ frame. B replicas replay A's committed ids — the snapshot, never a re-search.
181
+ - `POST /v1/sessions` (`title`, `project_id`?), `GET /v1/sessions`,
182
+ `GET /v1/sessions/{id}/history` — session mgmt.
183
+ - `POST /v1/sessions/{id}/messages` — the NATIVE turn endpoint (id-stable prompt compile,
184
+ `GenerationEventV1` stream). `parent_turn_id` ≠ head → branch/edit/regenerate. Body also
185
+ takes `privacy_mode`, `artifact_ids` (attachments), `tool_call_ids` (executed calls).
186
+ - `POST /v1/sessions/{id}/head` — move `local_head` (branch switching).
187
+ - `POST /v1/jobs/{job}/cancel` — cancel an in-flight generation.
188
+ - `POST/GET /v1/projects`, `POST /v1/projects/{id}/instructions` — workspace layer.
189
+ - `POST/GET /v1/memory`, `POST /v1/memory/{id}/delete` — long-term memory (soft delete;
190
+ namespace = `global` or a project id; both compile into the system region).
191
+ - `POST /v1/artifacts?name=&project_id=&parent_artifact_id=` (raw body),
192
+ `GET /v1/artifacts`, `GET /v1/artifacts/{id}` (verified bytes),
193
+ `GET /v1/artifacts/{id}/meta` (meta + lineage + children).
194
+ - `GET /v1/tools`, `POST /v1/tools/execute`, `POST /v1/tools/calls/{id}/approve|deny`,
195
+ `GET /v1/tools/calls/{id}` — the three-mode tool surface.
196
+ - `GET /v1/scheduler/status`, `POST /v1/scheduler/admission` — §14 observability + SLA
197
+ admission probe.
198
+ - `POST /v1/turns/{id}/verification` — settlement-clock transitions (manual/supervisor
199
+ writes; returns cascade + new certified head). The background worker drives the same
200
+ store machinery internally.
201
+ - `GET /v1/palw/status` — worker counters + loopback job counts.
202
+ - `/v1/palw/loopback/*` — the coordinator protocol above (404 unless `--palw-loopback`).
203
+
204
+ ## Not in this build (stated honestly)
205
+
206
+ - Embedding-based RAG (§11's full form) — attachment retrieval is deterministic keyword
207
+ chunking; an embedding index is a separate subsystem.
208
+ - Model-EMITTED tool calls / agent loops (§7, Phase 3) — tools are client-invoked; wiring
209
+ tool schemas into the canonical ChatML template changes the Compute Set prompt format
210
+ and belongs with the consensus-side template freeze.
211
+ - The consensus half of settlement: beacons, provider bonds, DA retention, auditor
212
+ lottery, rewards/maturity, and the node RPC binding. The worker's coordinator trait is
213
+ the seam where the node-side bridge plugs in; the loopback coordinator stands in for it
214
+ and proves nothing about economics or replica independence.
215
+ - Kernel-level sandbox isolation (network namespaces, rlimits) for `run_command`.
216
+
217
+ ## Run
218
+
219
+ ```
220
+ palw-gateway --gguf <gguf> --tables docs/evidence/qi35_tables.bin \
221
+ --tokenizer models/.../tokenizer.json --engine docs/evidence/qi35_model \
222
+ --listen 127.0.0.1:12346 --db sessions.sqlite3
223
+ ```
palw-gateway/src/artifact.rs ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Content-addressed artifact blob store (design doc §8) — bytes on disk, addressed by
2
+ //! blake2b-256 of the content, metadata in the session store.
3
+ //!
4
+ //! Properties the doc requires and this layout provides:
5
+ //! * **Content-addressed**: the same bytes stored twice are one file; the hash IS the address,
6
+ //! so a stored artifact can always be re-verified against its own name.
7
+ //! * **Crash-safe writes** (§18): write to a temp file in the same directory, fsync, rename —
8
+ //! a torn write can leave garbage temp files, never a wrong blob under a valid hash.
9
+ //! * **Parser canonicalization** (§8 正準化): v1 ships exactly one parser policy,
10
+ //! `utf8-text-v1` — the parsed content of a text artifact is its bytes verbatim, so
11
+ //! `parsed_content_root == content_hash`. PDF/DOCX/OCR parsers are future policies; the field
12
+ //! exists so a receipt can commit WHICH parser produced the tokens the model saw.
13
+ //!
14
+ //! Layout: `<root>/<hash[0..2]>/<hash>` — one fan-out level keeps directories small without
15
+ //! nested churn.
16
+
17
+ use std::io::Write;
18
+ use std::path::{Path, PathBuf};
19
+
20
+ use blake2::{Blake2b, Digest, digest::consts::U32};
21
+
22
+ /// The one parser policy v1 ships (see module docs).
23
+ pub const PARSER_POLICY_UTF8_TEXT_V1: &str = "utf8-text-v1";
24
+
25
+ pub struct ArtifactBlobStore {
26
+ root: PathBuf,
27
+ }
28
+
29
+ pub fn content_hash_hex(bytes: &[u8]) -> String {
30
+ let mut h = Blake2b::<U32>::new();
31
+ h.update(bytes);
32
+ h.finalize().iter().map(|b| format!("{b:02x}")).collect()
33
+ }
34
+
35
+ impl ArtifactBlobStore {
36
+ pub fn open(root: &Path) -> Result<Self, String> {
37
+ std::fs::create_dir_all(root).map_err(|e| format!("create {}: {e}", root.display()))?;
38
+ Ok(Self { root: root.to_path_buf() })
39
+ }
40
+
41
+ fn blob_path(&self, hash: &str) -> Result<PathBuf, String> {
42
+ // The hash is used as a path component; refuse anything that is not plain lowercase hex
43
+ // so a crafted "hash" can never traverse.
44
+ if hash.len() != 64 || !hash.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) {
45
+ return Err(format!("not a blake2b-256 hex hash: {hash:?}"));
46
+ }
47
+ Ok(self.root.join(&hash[..2]).join(hash))
48
+ }
49
+
50
+ /// Store bytes; returns the content hash. Idempotent — an existing blob is left untouched
51
+ /// (same hash ⇒ same bytes, modulo verify-detectable disk tampering).
52
+ pub fn put(&self, bytes: &[u8]) -> Result<String, String> {
53
+ let hash = content_hash_hex(bytes);
54
+ let path = self.blob_path(&hash)?;
55
+ if path.exists() {
56
+ return Ok(hash);
57
+ }
58
+ let dir = path.parent().expect("blob path has a parent");
59
+ std::fs::create_dir_all(dir).map_err(|e| format!("create {}: {e}", dir.display()))?;
60
+ let tmp = dir.join(format!(".tmp-{}-{}", std::process::id(), &hash[..16]));
61
+ {
62
+ let mut f = std::fs::File::create(&tmp).map_err(|e| format!("create {}: {e}", tmp.display()))?;
63
+ f.write_all(bytes).map_err(|e| format!("write {}: {e}", tmp.display()))?;
64
+ f.sync_all().map_err(|e| format!("fsync {}: {e}", tmp.display()))?;
65
+ }
66
+ std::fs::rename(&tmp, &path).map_err(|e| format!("rename into {}: {e}", path.display()))?;
67
+ Ok(hash)
68
+ }
69
+
70
+ pub fn get(&self, hash: &str) -> Result<Vec<u8>, String> {
71
+ let path = self.blob_path(hash)?;
72
+ std::fs::read(&path).map_err(|e| format!("artifact blob {hash}: {e}"))
73
+ }
74
+
75
+ /// Read AND re-hash — the §19 "hash verification" primitive. A blob whose bytes no longer
76
+ /// match its address (disk corruption, tampering) is an error, never silently returned.
77
+ pub fn get_verified(&self, hash: &str) -> Result<Vec<u8>, String> {
78
+ let bytes = self.get(hash)?;
79
+ let actual = content_hash_hex(&bytes);
80
+ if actual != hash {
81
+ return Err(format!("artifact blob {hash} fails verification (bytes hash to {actual}) — refusing to serve"));
82
+ }
83
+ Ok(bytes)
84
+ }
85
+
86
+ /// The `utf8-text-v1` parse: the artifact's bytes as UTF-8 text, or an error explaining the
87
+ /// artifact cannot enter model context under v1's only parser policy.
88
+ pub fn parse_utf8_text(&self, hash: &str) -> Result<String, String> {
89
+ let bytes = self.get_verified(hash)?;
90
+ String::from_utf8(bytes)
91
+ .map_err(|_| format!("artifact {hash} is not valid UTF-8 — parser policy {PARSER_POLICY_UTF8_TEXT_V1} cannot place it in context"))
92
+ }
93
+ }
94
+
95
+ #[cfg(test)]
96
+ mod tests {
97
+ use super::*;
98
+
99
+ fn store_at(name: &str) -> ArtifactBlobStore {
100
+ let dir = std::env::temp_dir().join(format!("palw-gw-blob-{}-{name}", std::process::id()));
101
+ let _ = std::fs::remove_dir_all(&dir);
102
+ ArtifactBlobStore::open(&dir).unwrap()
103
+ }
104
+
105
+ #[test]
106
+ fn roundtrip_idempotent_and_verified() {
107
+ let s = store_at("roundtrip");
108
+ let h1 = s.put(b"hello artifact").unwrap();
109
+ let h2 = s.put(b"hello artifact").unwrap();
110
+ assert_eq!(h1, h2, "content-addressed: same bytes, same address");
111
+ assert_eq!(s.get_verified(&h1).unwrap(), b"hello artifact");
112
+ assert_eq!(s.parse_utf8_text(&h1).unwrap(), "hello artifact");
113
+ }
114
+
115
+ #[test]
116
+ fn tampered_blob_is_refused() {
117
+ let s = store_at("tamper");
118
+ let h = s.put(b"original bytes").unwrap();
119
+ let path = s.blob_path(&h).unwrap();
120
+ std::fs::write(&path, b"tampered bytes!").unwrap();
121
+ assert!(s.get_verified(&h).is_err(), "bytes no longer matching the address must not be served");
122
+ }
123
+
124
+ #[test]
125
+ fn hash_is_validated_as_path_component() {
126
+ let s = store_at("path");
127
+ assert!(s.get("../../etc/passwd").is_err());
128
+ assert!(s.get("ABCDEF").is_err());
129
+ }
130
+
131
+ #[test]
132
+ fn non_utf8_cannot_enter_context() {
133
+ let s = store_at("utf8");
134
+ let h = s.put(&[0xff, 0xfe, 0x00]).unwrap();
135
+ assert!(s.parse_utf8_text(&h).is_err());
136
+ // …but the blob itself is stored and retrievable (it can live as an attachment).
137
+ assert_eq!(s.get_verified(&h).unwrap(), vec![0xff, 0xfe, 0x00]);
138
+ }
139
+ }
palw-gateway/src/context.rs ADDED
@@ -0,0 +1,635 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Context Compiler v1 (design doc §2) + Token Budget Manager (§3) + the Phase-1 inputs:
2
+ //! project instructions (§10), memory snapshot (§12), attachments with deterministic chunk
3
+ //! selection (§8/§11), tool results (§6).
4
+ //!
5
+ //! THE invariant this module exists to enforce (from the Explore survey's root-cause): model
6
+ //! history is assembled from STORED TOKEN IDS, never from detokenized text. The current stack
7
+ //! re-tokenized `decode(gen).strip()` each turn, which is not id-stable, so the engine's prefix
8
+ //! cache missed on every multi-turn exchange and the whole transcript re-prefilled every time.
9
+ //!
10
+ //! Canonical fragment assembly:
11
+ //!
12
+ //! ```text
13
+ //! enc("<|im_start|>system\n{SYS}{project}{memory}<|im_end|>\n") — per conversation state
14
+ //! per past turn:
15
+ //! enc("<|im_start|>user\n{B}<|im_end|>\n<|im_start|>assistant\n") — B is the turn's STORED
16
+ //! user block (material +
17
+ //! user text), re-encoded
18
+ //! stored output_ids (EOS stripped) — never re-tokenized
19
+ //! enc("<|im_end|>\n") — constant, cached
20
+ //! current turn:
21
+ //! enc("<|im_start|>user\n{B_now}<|im_end|>\n<|im_start|>assistant\n")
22
+ //! ```
23
+ //!
24
+ //! Every fragment BOUNDARY starts with a special token (`<|im_start|>`/`<|im_end|>`), which the
25
+ //! tokenizer never merges across, so fragment-wise encoding is stable under concatenation. Turn
26
+ //! N+1's prompt literally starts with turn N's prompt+output ids, so the engine's
27
+ //! longest-prefix cache HITS across turns — as long as the system region (base + project
28
+ //! instructions + memory) is unchanged. Editing memory/instructions changes token 0 onward and
29
+ //! honestly costs one full re-prefill; that is inherent, not a bug.
30
+ //!
31
+ //! Attachments and tool results are rendered INTO the turn's user block (like a user pasting
32
+ //! text), not into the system region — so attaching a file to turn N leaves turns 1..N-1's
33
+ //! prefix intact. The exact rendered block is persisted per turn (`user_block_text`) and re-used
34
+ //! verbatim by later compiles; with no material the block IS the raw user text, byte-identical
35
+ //! to what the v1 schema produced.
36
+ //!
37
+ //! Budget v0 (§3, the doc's "初期版" verbatim): keep system, keep the current message, drop
38
+ //! OLDEST turns whole — never split a message mid-token, never auto-summarize. Attachments are
39
+ //! bounded per file by deterministic keyword chunk selection (§11's retrieval snapshot in
40
+ //! miniature: policy id, selected chunk indexes, and the roots all land in `context_meta_json`,
41
+ //! which is the future receipt commitment surface).
42
+
43
+ use blake2::{Blake2b, Digest, digest::consts::U32};
44
+ use serde::Serialize;
45
+
46
+ use crate::store::StoredTurn;
47
+ use crate::tokenizer::TokenizerHost;
48
+
49
+ pub const CONTEXT_POLICY_ID: &str = "compile-v1";
50
+ pub const MEMORY_POLICY_ID: &str = "memory-all-active-v1";
51
+ pub const RETRIEVAL_POLICY_ID: &str = "keyword-chunks-v1";
52
+
53
+ /// Chunk granularity for attachment selection (lines per chunk).
54
+ const CHUNK_LINES: usize = 30;
55
+
56
+ pub struct ContextCompiler {
57
+ system_prompt: String,
58
+ im_end_nl_ids: Vec<u32>,
59
+ pub max_context_tokens: usize,
60
+ pub reserved_output_tokens: usize,
61
+ /// Per-attachment character budget (chunk selection kicks in above it).
62
+ pub max_file_chars: usize,
63
+ /// Per-tool-result character budget (head-truncated above it).
64
+ pub max_tool_chars: usize,
65
+ }
66
+
67
+ pub struct MemoryForContext {
68
+ pub memory_id: String,
69
+ pub content: String,
70
+ }
71
+
72
+ pub struct AttachmentForContext {
73
+ pub artifact_id: String,
74
+ pub display_name: String,
75
+ pub content_hash: String,
76
+ pub text: String,
77
+ }
78
+
79
+ pub struct ToolResultForContext {
80
+ pub call_id: String,
81
+ pub tool_name: String,
82
+ pub text: String,
83
+ }
84
+
85
+ #[derive(Default)]
86
+ pub struct ContextInputs<'a> {
87
+ pub project_instructions: Option<&'a str>,
88
+ pub memory: &'a [MemoryForContext],
89
+ pub attachments: &'a [AttachmentForContext],
90
+ pub tool_results: &'a [ToolResultForContext],
91
+ pub history: &'a [StoredTurn],
92
+ pub user_text: &'a str,
93
+ /// Rendered live-search evidence for THIS turn. Appended after the user text inside the
94
+ /// user block (the legacy `build_prompt_text` convention: `user + "\n\n" + evidence`), so
95
+ /// the committed prompt ids — and therefore the receipt's prompt commitment — cover the
96
+ /// bundle digest line transitively.
97
+ pub search: Option<SearchForContext>,
98
+ }
99
+
100
+ #[derive(Clone)]
101
+ pub struct SearchForContext {
102
+ pub evidence_text: String,
103
+ pub bundle_sha256: String,
104
+ pub effective_compression: String,
105
+ }
106
+
107
+ #[derive(Serialize)]
108
+ struct AttachmentMeta {
109
+ artifact_id: String,
110
+ content_hash: String,
111
+ chunks_total: u32,
112
+ chunks_selected: Vec<u32>,
113
+ }
114
+
115
+ #[derive(Serialize)]
116
+ struct SearchMetaV1 {
117
+ bundle_sha256: String,
118
+ effective_compression: String,
119
+ }
120
+
121
+ /// What the compiler actually put in front of the model — stored on the turn row; this is the
122
+ /// surface a Phase-2 receipt commits (roots, policies, selections), never UI state.
123
+ #[derive(Serialize)]
124
+ struct ContextMetaV1 {
125
+ context_policy: &'static str,
126
+ memory_policy: &'static str,
127
+ retrieval_policy: &'static str,
128
+ #[serde(skip_serializing_if = "Option::is_none")]
129
+ memory_snapshot_root: Option<String>,
130
+ memory_ids: Vec<String>,
131
+ attachments: Vec<AttachmentMeta>,
132
+ tool_calls: Vec<String>,
133
+ dropped_turns: u32,
134
+ #[serde(skip_serializing_if = "Option::is_none")]
135
+ search: Option<SearchMetaV1>,
136
+ }
137
+
138
+ pub struct CompiledPrompt {
139
+ pub ids: Vec<u32>,
140
+ /// How many leading ids are the system region (the engine's `cache_prefix_len` convention —
141
+ /// this snapshot boundary is stable for the conversation until memory/instructions change).
142
+ pub system_prefix_len: usize,
143
+ /// Turns dropped by the budget, oldest-first (surfaced to the client, never silent).
144
+ pub dropped_turns: usize,
145
+ /// The EXACT text encoded into the current turn's user block — persist this on the turn so
146
+ /// later compiles reproduce the same ids.
147
+ pub user_block_text: String,
148
+ pub context_meta_json: String,
149
+ }
150
+
151
+ /// blake2b-256 over `id \n content \n` per entry, in snapshot order — the §12 memory snapshot
152
+ /// root (only what was USED this turn is committed, never the whole memory DB).
153
+ pub fn memory_snapshot_root(memory: &[MemoryForContext]) -> Option<String> {
154
+ if memory.is_empty() {
155
+ return None;
156
+ }
157
+ let mut h = Blake2b::<U32>::new();
158
+ for m in memory {
159
+ h.update(m.memory_id.as_bytes());
160
+ h.update(b"\n");
161
+ h.update(m.content.as_bytes());
162
+ h.update(b"\n");
163
+ }
164
+ Some(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
165
+ }
166
+
167
+ pub struct ChunkSelection {
168
+ pub chunks_total: u32,
169
+ pub chunks_selected: Vec<u32>,
170
+ pub rendered: String,
171
+ }
172
+
173
+ /// Deterministic keyword chunk selection (`keyword-chunks-v1`): split into fixed line-blocks,
174
+ /// score each by occurrences of the query's terms (lowercased, ≥3 chars, deduped), select
175
+ /// best-first until the char budget is spent, render in DOCUMENT order with `…` gap markers.
176
+ /// No embeddings, no randomness, no clock — B recomputes the identical selection from the same
177
+ /// (text, query, budget).
178
+ pub fn select_chunks(text: &str, query: &str, budget_chars: usize) -> ChunkSelection {
179
+ if text.chars().count() <= budget_chars {
180
+ return ChunkSelection { chunks_total: 1, chunks_selected: vec![0], rendered: text.to_string() };
181
+ }
182
+ let lines: Vec<&str> = text.lines().collect();
183
+ let chunks: Vec<String> = lines.chunks(CHUNK_LINES).map(|c| c.join("\n")).collect();
184
+
185
+ let mut terms: Vec<String> = query
186
+ .to_lowercase()
187
+ .split(|c: char| !c.is_alphanumeric())
188
+ .filter(|t| t.chars().count() >= 3)
189
+ .map(String::from)
190
+ .collect();
191
+ terms.sort();
192
+ terms.dedup();
193
+
194
+ let mut scored: Vec<(usize, u64)> = chunks
195
+ .iter()
196
+ .enumerate()
197
+ .map(|(i, chunk)| {
198
+ let lower = chunk.to_lowercase();
199
+ let score: u64 = terms.iter().map(|t| lower.matches(t.as_str()).count() as u64).sum();
200
+ (i, score)
201
+ })
202
+ .collect();
203
+ // Best score first; document order breaks ties — fully deterministic.
204
+ scored.sort_by(|a, b| b.1.cmp(&a.1).then(a.0.cmp(&b.0)));
205
+
206
+ let mut selected: Vec<usize> = Vec::new();
207
+ let mut remaining = budget_chars;
208
+ for (i, _score) in &scored {
209
+ let cost = chunks[*i].chars().count();
210
+ if cost <= remaining {
211
+ selected.push(*i);
212
+ remaining -= cost;
213
+ }
214
+ }
215
+ if selected.is_empty() {
216
+ // Even the best chunk alone is over budget: hard-truncate it at a char boundary rather
217
+ // than sending nothing.
218
+ let best = scored.first().map(|(i, _)| *i).unwrap_or(0);
219
+ let truncated: String = chunks[best].chars().take(budget_chars).collect();
220
+ return ChunkSelection { chunks_total: chunks.len() as u32, chunks_selected: vec![best as u32], rendered: truncated };
221
+ }
222
+ selected.sort();
223
+
224
+ let mut rendered = String::new();
225
+ let mut prev: Option<usize> = None;
226
+ for i in &selected {
227
+ if let Some(p) = prev {
228
+ if *i != p + 1 {
229
+ rendered.push_str("\n…\n");
230
+ } else {
231
+ rendered.push('\n');
232
+ }
233
+ }
234
+ rendered.push_str(&chunks[*i]);
235
+ prev = Some(*i);
236
+ }
237
+ ChunkSelection {
238
+ chunks_total: chunks.len() as u32,
239
+ chunks_selected: selected.iter().map(|i| *i as u32).collect(),
240
+ rendered,
241
+ }
242
+ }
243
+
244
+ fn truncate_chars(text: &str, budget: usize) -> (String, bool) {
245
+ if text.chars().count() <= budget {
246
+ (text.to_string(), false)
247
+ } else {
248
+ (text.chars().take(budget).collect(), true)
249
+ }
250
+ }
251
+
252
+ impl ContextCompiler {
253
+ pub fn new(
254
+ tokenizer: &TokenizerHost,
255
+ system_prompt: &str,
256
+ max_context_tokens: usize,
257
+ reserved_output_tokens: usize,
258
+ max_file_chars: usize,
259
+ max_tool_chars: usize,
260
+ ) -> Result<Self, String> {
261
+ // Validate the constant fragment once up front (encode_lossless refuses what it cannot
262
+ // commit); the system region itself is compiled per turn now that it carries state.
263
+ let im_end_nl_ids = tokenizer.encode_lossless("<|im_end|>\n")?;
264
+ Ok(Self {
265
+ system_prompt: system_prompt.to_string(),
266
+ im_end_nl_ids,
267
+ max_context_tokens,
268
+ reserved_output_tokens,
269
+ max_file_chars,
270
+ max_tool_chars,
271
+ })
272
+ }
273
+
274
+ fn user_block_ids(&self, tokenizer: &TokenizerHost, block_text: &str) -> Result<Vec<u32>, String> {
275
+ tokenizer.encode_lossless(&format!("<|im_start|>user\n{block_text}<|im_end|>\n<|im_start|>assistant\n"))
276
+ }
277
+
278
+ fn trimmed_output(output_ids: &[u32], eos_id: u32) -> &[u32] {
279
+ match output_ids.last() {
280
+ Some(&last) if last == eos_id => &output_ids[..output_ids.len() - 1],
281
+ _ => output_ids,
282
+ }
283
+ }
284
+
285
+ /// The system region text: base prompt, then project instructions, then the memory
286
+ /// snapshot — stable-first, so the least volatile content owns the earliest (most cacheable)
287
+ /// token positions.
288
+ fn system_text(&self, inputs: &ContextInputs) -> String {
289
+ let mut text = self.system_prompt.clone();
290
+ if let Some(instructions) = inputs.project_instructions {
291
+ if !instructions.is_empty() {
292
+ text.push_str("\n\n[project instructions]\n");
293
+ text.push_str(instructions);
294
+ }
295
+ }
296
+ if !inputs.memory.is_empty() {
297
+ text.push_str("\n\n[memory]");
298
+ for m in inputs.memory {
299
+ text.push_str("\n- ");
300
+ text.push_str(&m.content);
301
+ }
302
+ }
303
+ text
304
+ }
305
+
306
+ /// The current turn's user block: attachments (chunk-selected), then tool results, then the
307
+ /// user's text — material first, question last, and the whole block is what gets persisted.
308
+ fn build_user_block(&self, inputs: &ContextInputs) -> (String, Vec<AttachmentMeta>) {
309
+ let mut block = String::new();
310
+ let mut attachment_meta = Vec::new();
311
+ for a in inputs.attachments {
312
+ let selection = select_chunks(&a.text, inputs.user_text, self.max_file_chars);
313
+ block.push_str(&format!(
314
+ "[file {} blake2b={} chunks={}/{}]\n",
315
+ a.display_name,
316
+ &a.content_hash[..16.min(a.content_hash.len())],
317
+ selection.chunks_selected.len(),
318
+ selection.chunks_total
319
+ ));
320
+ block.push_str(&selection.rendered);
321
+ block.push_str("\n[/file]\n");
322
+ attachment_meta.push(AttachmentMeta {
323
+ artifact_id: a.artifact_id.clone(),
324
+ content_hash: a.content_hash.clone(),
325
+ chunks_total: selection.chunks_total,
326
+ chunks_selected: selection.chunks_selected,
327
+ });
328
+ }
329
+ for t in inputs.tool_results {
330
+ let (text, truncated) = truncate_chars(&t.text, self.max_tool_chars);
331
+ block.push_str(&format!("[tool_result {} call={}]\n", t.tool_name, t.call_id));
332
+ block.push_str(&text);
333
+ if truncated {
334
+ block.push_str("\n[truncated]");
335
+ }
336
+ block.push_str("\n[/tool_result]\n");
337
+ }
338
+ block.push_str(inputs.user_text);
339
+ if let Some(search) = &inputs.search {
340
+ block.push_str("\n\n");
341
+ block.push_str(&search.evidence_text);
342
+ }
343
+ (block, attachment_meta)
344
+ }
345
+
346
+ /// Compile the prompt for a new user message. `inputs.history` is the root-first branch;
347
+ /// applies budget v0 by dropping oldest turns whole.
348
+ pub fn compile(&self, tokenizer: &TokenizerHost, inputs: &ContextInputs) -> Result<CompiledPrompt, String> {
349
+ let system_ids =
350
+ tokenizer.encode_lossless(&format!("<|im_start|>system\n{}<|im_end|>\n", self.system_text(inputs)))?;
351
+ let (user_block_text, attachment_meta) = self.build_user_block(inputs);
352
+ let current_block = self.user_block_ids(tokenizer, &user_block_text)?;
353
+
354
+ let budget = self
355
+ .max_context_tokens
356
+ .checked_sub(self.reserved_output_tokens)
357
+ .ok_or("reserved_output_tokens exceeds max_context_tokens")?;
358
+ let fixed = system_ids.len() + current_block.len();
359
+ if fixed > budget {
360
+ return Err(format!(
361
+ "system region + current message need {fixed} tokens against a {budget}-token input budget — \
362
+ shorten the message/attachments or raise max_context"
363
+ ));
364
+ }
365
+
366
+ // Per-turn fragment cost, newest-last; then take the longest suffix that fits.
367
+ let mut turn_frags: Vec<(Vec<u32>, &[u32])> = Vec::with_capacity(inputs.history.len());
368
+ for turn in inputs.history {
369
+ let block = self.user_block_ids(tokenizer, turn.user_block())?;
370
+ let out = Self::trimmed_output(&turn.output_ids, tokenizer.eos_id);
371
+ turn_frags.push((block, out));
372
+ }
373
+ let mut kept_from = 0usize;
374
+ loop {
375
+ let hist_cost: usize = turn_frags[kept_from..]
376
+ .iter()
377
+ .map(|(b, o)| b.len() + o.len() + self.im_end_nl_ids.len())
378
+ .sum();
379
+ if fixed + hist_cost <= budget {
380
+ break;
381
+ }
382
+ kept_from += 1;
383
+ if kept_from > turn_frags.len() {
384
+ unreachable!("fixed-part fit was checked above");
385
+ }
386
+ }
387
+
388
+ let mut ids = system_ids.clone();
389
+ for (block, out) in &turn_frags[kept_from..] {
390
+ ids.extend_from_slice(block);
391
+ ids.extend_from_slice(out);
392
+ ids.extend_from_slice(&self.im_end_nl_ids);
393
+ }
394
+ ids.extend_from_slice(&current_block);
395
+
396
+ let meta = ContextMetaV1 {
397
+ context_policy: CONTEXT_POLICY_ID,
398
+ memory_policy: MEMORY_POLICY_ID,
399
+ retrieval_policy: RETRIEVAL_POLICY_ID,
400
+ memory_snapshot_root: memory_snapshot_root(inputs.memory),
401
+ memory_ids: inputs.memory.iter().map(|m| m.memory_id.clone()).collect(),
402
+ attachments: attachment_meta,
403
+ tool_calls: inputs.tool_results.iter().map(|t| t.call_id.clone()).collect(),
404
+ dropped_turns: kept_from as u32,
405
+ search: inputs.search.as_ref().map(|s| SearchMetaV1 {
406
+ bundle_sha256: s.bundle_sha256.clone(),
407
+ effective_compression: s.effective_compression.clone(),
408
+ }),
409
+ };
410
+ let context_meta_json = serde_json::to_string(&meta).map_err(|e| e.to_string())?;
411
+
412
+ Ok(CompiledPrompt {
413
+ ids,
414
+ system_prefix_len: system_ids.len(),
415
+ dropped_turns: kept_from,
416
+ user_block_text,
417
+ context_meta_json,
418
+ })
419
+ }
420
+ }
421
+
422
+ #[cfg(test)]
423
+ mod tests {
424
+ use super::*;
425
+ use crate::events::{TurnPrivacyMode, TurnVerificationStatus};
426
+
427
+ /// EOS-stripping is what lets a stored answer sit INSIDE a longer prompt: the `<|im_end|>\n`
428
+ /// fragment provides the turn boundary, so the answer's own EOS must not be duplicated. A
429
+ /// mistrimmed output would put two end markers back-to-back and desync every downstream id.
430
+ #[test]
431
+ fn trimmed_output_drops_exactly_one_trailing_eos() {
432
+ assert_eq!(ContextCompiler::trimmed_output(&[10, 20, 99], 99), &[10, 20]);
433
+ // No EOS present (e.g. a max_tokens stop) ⇒ untouched.
434
+ assert_eq!(ContextCompiler::trimmed_output(&[10, 20], 99), &[10, 20]);
435
+ // Only the LAST is stripped — an EOS mid-sequence is real content, never removed.
436
+ assert_eq!(ContextCompiler::trimmed_output(&[99, 10, 99], 99), &[99, 10]);
437
+ assert_eq!(ContextCompiler::trimmed_output(&[], 99), &[] as &[u32]);
438
+ }
439
+
440
+ #[test]
441
+ fn memory_root_is_order_sensitive_and_deterministic() {
442
+ let a = MemoryForContext { memory_id: "m1".into(), content: "likes rust".into() };
443
+ let b = MemoryForContext { memory_id: "m2".into(), content: "tabs are wrong".into() };
444
+ let r1 = memory_snapshot_root(&[a, b]).unwrap();
445
+ let a2 = MemoryForContext { memory_id: "m1".into(), content: "likes rust".into() };
446
+ let b2 = MemoryForContext { memory_id: "m2".into(), content: "tabs are wrong".into() };
447
+ assert_eq!(r1, memory_snapshot_root(&[a2, b2]).unwrap(), "same snapshot, same root");
448
+ let a3 = MemoryForContext { memory_id: "m1".into(), content: "likes rust".into() };
449
+ let b3 = MemoryForContext { memory_id: "m2".into(), content: "tabs are wrong".into() };
450
+ assert_ne!(r1, memory_snapshot_root(&[b3, a3]).unwrap(), "order is part of the snapshot");
451
+ assert_eq!(memory_snapshot_root(&[]), None);
452
+ }
453
+
454
+ #[test]
455
+ fn chunk_selection_finds_the_needle_deterministically() {
456
+ // 120 lines; the needle sits at lines 61-62 → chunk index 2 (30-line chunks).
457
+ let mut lines: Vec<String> = (0..120).map(|i| format!("filler line {i}")).collect();
458
+ lines[60] = "the needle sits here".into();
459
+ lines[61] = "another needle mention".into();
460
+ let text = lines.join("\n");
461
+ let budget = 700; // forces selection: whole text is ~2000 chars
462
+
463
+ let s1 = select_chunks(&text, "where is the needle?", budget);
464
+ let s2 = select_chunks(&text, "where is the needle?", budget);
465
+ assert_eq!(s1.chunks_selected, s2.chunks_selected, "deterministic");
466
+ assert_eq!(s1.chunks_total, 4);
467
+ assert!(s1.chunks_selected.contains(&2), "the scoring must pick the needle chunk, got {:?}", s1.chunks_selected);
468
+ assert!(s1.rendered.contains("the needle sits here"));
469
+ assert!(s1.rendered.chars().count() <= budget);
470
+
471
+ // Under budget ⇒ whole text, single logical chunk.
472
+ let all = select_chunks("short", "anything", 100);
473
+ assert_eq!(all.chunks_total, 1);
474
+ assert_eq!(all.rendered, "short");
475
+ }
476
+
477
+ // ---- tokenizer-backed tests (run when the local model checkout provides tokenizer.json;
478
+ // the file is not git-tracked, so CI without models/ skips them loudly). --------------
479
+
480
+ fn load_tokenizer() -> Option<TokenizerHost> {
481
+ let path = std::path::Path::new("../models/Qwen3.6-35B-A3B-Claude-4.7-base-meta/tokenizer.json");
482
+ if !path.exists() {
483
+ eprintln!("SKIP: {} not present — tokenizer-backed compiler tests skipped", path.display());
484
+ return None;
485
+ }
486
+ Some(TokenizerHost::load(path, crate::QWEN_EOS_ID).unwrap())
487
+ }
488
+
489
+ fn stored_turn(user_block: &str, output_ids: Vec<u32>) -> StoredTurn {
490
+ StoredTurn {
491
+ turn_id: "t".into(),
492
+ conversation_id: "c".into(),
493
+ parent_turn_id: None,
494
+ role_user_text: user_block.into(),
495
+ user_block_text: Some(user_block.into()),
496
+ context_meta_json: None,
497
+ display_text: String::new(),
498
+ prompt_ids: Vec::new(),
499
+ output_ids,
500
+ stop_reason: None,
501
+ verification_status: TurnVerificationStatus::LocalComplete,
502
+ privacy_mode: TurnPrivacyMode::LocalOnly,
503
+ receipt_json: None,
504
+ runtime_roots_json: None,
505
+ search_bundle_json: None,
506
+ created_ms: 0,
507
+ }
508
+ }
509
+
510
+ /// The legacy binding convention: rendered evidence joins the user block AFTER the user
511
+ /// text (`user + "\n\n" + evidence`), so committed prompt ids — and hence the receipt's
512
+ /// prompt commitment — cover the bundle digest line. The meta records what was committed.
513
+ #[test]
514
+ fn search_evidence_joins_the_user_block_and_meta() {
515
+ let Some(tok) = load_tokenizer() else { return };
516
+ let compiler = ContextCompiler::new(&tok, "S.", 8192, 512, 6000, 6000).unwrap();
517
+ let compiled = compiler
518
+ .compile(
519
+ &tok,
520
+ &ContextInputs {
521
+ user_text: "question?",
522
+ search: Some(SearchForContext {
523
+ evidence_text: "[UNTRUSTED LIVE SEARCH EVIDENCE]\nbundle_sha256=deadbeef\n[END]".into(),
524
+ bundle_sha256: "deadbeef".into(),
525
+ effective_compression: "off".into(),
526
+ }),
527
+ ..Default::default()
528
+ },
529
+ )
530
+ .unwrap();
531
+ assert!(compiled.user_block_text.starts_with("question?\n\n[UNTRUSTED"));
532
+ assert!(compiled.user_block_text.contains("bundle_sha256=deadbeef"));
533
+ let decoded = tok.decode(&compiled.ids).unwrap();
534
+ assert!(decoded.contains("bundle_sha256=deadbeef"), "digest line must be inside the committed ids");
535
+ let meta: serde_json::Value = serde_json::from_str(&compiled.context_meta_json).unwrap();
536
+ assert_eq!(meta["search"]["bundle_sha256"], "deadbeef");
537
+ assert_eq!(meta["search"]["effective_compression"], "off");
538
+ // A searchless turn's meta has no search key at all (schema-stable with pre-search turns).
539
+ let plain = compiler.compile(&tok, &ContextInputs { user_text: "hi", ..Default::default() }).unwrap();
540
+ let plain_meta: serde_json::Value = serde_json::from_str(&plain.context_meta_json).unwrap();
541
+ assert!(plain_meta.get("search").is_none());
542
+ }
543
+
544
+ #[test]
545
+ fn plain_turns_keep_the_cross_turn_prefix_invariant() {
546
+ let Some(tok) = load_tokenizer() else { return };
547
+ let compiler = ContextCompiler::new(&tok, "You are a helpful assistant.", 8192, 512, 6000, 6000).unwrap();
548
+
549
+ let c1 = compiler
550
+ .compile(&tok, &ContextInputs { user_text: "hello there", ..Default::default() })
551
+ .unwrap();
552
+ // No material ⇒ the persisted block IS the raw user text (v1-schema compatibility).
553
+ assert_eq!(c1.user_block_text, "hello there");
554
+
555
+ // Pretend the engine answered; next turn's prompt must extend the previous one.
556
+ let output = vec![c1.ids[5], c1.ids[6], crate::QWEN_EOS_ID];
557
+ let t1 = stored_turn(&c1.user_block_text, output.clone());
558
+ let c2 = compiler
559
+ .compile(&tok, &ContextInputs { history: std::slice::from_ref(&t1), user_text: "and again", ..Default::default() })
560
+ .unwrap();
561
+ assert!(c2.ids.starts_with(&c1.ids), "turn 2 prompt must extend turn 1's prompt (prefix-cache invariant)");
562
+ let continued = &c2.ids[c1.ids.len()..c1.ids.len() + 2];
563
+ assert_eq!(continued, &output[..2], "stored output ids are spliced in verbatim (EOS trimmed)");
564
+ }
565
+
566
+ #[test]
567
+ fn memory_lives_in_the_system_region_and_attachments_in_the_user_block() {
568
+ let Some(tok) = load_tokenizer() else { return };
569
+ let compiler = ContextCompiler::new(&tok, "Base.", 8192, 512, 6000, 6000).unwrap();
570
+
571
+ let memory = vec![MemoryForContext { memory_id: "m1".into(), content: "user prefers Japanese".into() }];
572
+ let attachments = vec![AttachmentForContext {
573
+ artifact_id: "a1".into(),
574
+ display_name: "notes.txt".into(),
575
+ content_hash: "ab".repeat(32),
576
+ text: "attachment body".into(),
577
+ }];
578
+ let compiled = compiler
579
+ .compile(
580
+ &tok,
581
+ &ContextInputs {
582
+ project_instructions: Some("Answer tersely."),
583
+ memory: &memory,
584
+ attachments: &attachments,
585
+ user_text: "question?",
586
+ ..Default::default()
587
+ },
588
+ )
589
+ .unwrap();
590
+
591
+ let system_text = tok.decode(&compiled.ids[..compiled.system_prefix_len]).unwrap();
592
+ assert!(system_text.contains("[project instructions]\nAnswer tersely."));
593
+ assert!(system_text.contains("[memory]\n- user prefers Japanese"));
594
+ assert!(!system_text.contains("attachment body"), "attachments must not pollute the system region");
595
+
596
+ assert!(compiled.user_block_text.starts_with("[file notes.txt blake2b=abababababababab chunks=1/1]\n"));
597
+ assert!(compiled.user_block_text.contains("attachment body"));
598
+ assert!(compiled.user_block_text.ends_with("question?"));
599
+
600
+ let meta: serde_json::Value = serde_json::from_str(&compiled.context_meta_json).unwrap();
601
+ assert_eq!(meta["memory_ids"][0], "m1");
602
+ assert!(meta["memory_snapshot_root"].is_string());
603
+ assert_eq!(meta["attachments"][0]["artifact_id"], "a1");
604
+ assert_eq!(meta["context_policy"], CONTEXT_POLICY_ID);
605
+ }
606
+
607
+ #[test]
608
+ fn budget_drops_oldest_turns_whole() {
609
+ let Some(tok) = load_tokenizer() else { return };
610
+ let filler = "some words that cost a handful of tokens each time";
611
+ let history = vec![
612
+ stored_turn(filler, vec![9906, 9906, 9906]),
613
+ stored_turn(filler, vec![9906, 9906, 9906]),
614
+ stored_turn(filler, vec![9906, 9906, 9906]),
615
+ ];
616
+ let inputs = ContextInputs { history: &history, user_text: "final question", ..Default::default() };
617
+
618
+ // Measure the unpressured prompt, then rebuild the compiler with an input budget 5
619
+ // tokens short of it — at least one whole turn must go, and dropping one frees far more
620
+ // than 5 tokens, so the rest fit. No token-count guessing.
621
+ let roomy = ContextCompiler::new(&tok, "S.", 8192, 64, 6000, 6000).unwrap();
622
+ let full = roomy.compile(&tok, &inputs).unwrap();
623
+ assert_eq!(full.dropped_turns, 0);
624
+
625
+ let tight = ContextCompiler::new(&tok, "S.", full.ids.len() + 64 - 5, 64, 6000, 6000).unwrap();
626
+ let compiled = tight.compile(&tok, &inputs).unwrap();
627
+ assert_eq!(compiled.dropped_turns, 1, "5 tokens over ⇒ exactly the oldest turn goes");
628
+ assert!(compiled.ids.len() <= full.ids.len() - 5, "the result honors the shrunken budget");
629
+ // The oldest turn was dropped, the newest survives: the compiled ids end with the same
630
+ // current block and keep the LAST history turn's fragment.
631
+ assert_eq!(&compiled.ids[compiled.system_prefix_len..], &full.ids[full.ids.len() - (compiled.ids.len() - compiled.system_prefix_len)..]);
632
+ let meta: serde_json::Value = serde_json::from_str(&compiled.context_meta_json).unwrap();
633
+ assert_eq!(meta["dropped_turns"], 1);
634
+ }
635
+ }
palw-gateway/src/coordinator.rs ADDED
@@ -0,0 +1,567 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! The PALW coordinator boundary (design doc §15).
2
+ //!
3
+ //! The background worker needs five verbs against "the network": submit an A-commit, poll
4
+ //! verdicts, poll B assignments, decline an assignment, submit a replica result. The REAL
5
+ //! counterparty is the MISAKA node's PALW surface (replica dispatch / receipt submission / DA —
6
+ //! the protocol-side features §15 lists), which lives in the consensus repo and is not finished.
7
+ //! This module therefore freezes the verbs as a TRAIT and ships two honest implementations:
8
+ //!
9
+ //! * `LoopbackCoordinator` — a complete in-process coordinator with the real state machine
10
+ //! (submit → assign → result → matched → certified, mismatch, deadline requeue). The gateway
11
+ //! exposes it over HTTP (`/v1/palw/loopback/*`), so a SECOND gateway instance can point its
12
+ //! `HttpCoordinator` here and run a genuine two-instance A/B loop on one desk. This is the
13
+ //! devnet harness for the desktop product — it is NOT consensus: no beacons, no bonds, no DA
14
+ //! retention, no chain. Its verdicts certify the plumbing, not the economics.
15
+ //! * `HttpCoordinator` — the client half of the same JSON protocol (documented in the README),
16
+ //! which a node-side bridge implements to make settlement real. Plain HTTP/1.1 over TCP,
17
+ //! `Connection: close`, no TLS — a localhost/LAN dev transport; the production node binding
18
+ //! replaces this, it does not extend it.
19
+ //!
20
+ //! Determinism note for self-replica dev mode: a single-instance loopback replica runs on the
21
+ //! SAME engine process, so the prefix cache makes B's "replay" nearly free and — more
22
+ //! importantly — not independent. Loopback validates the pipeline, never the security claim;
23
+ //! independence needs a second host by definition.
24
+
25
+ use std::collections::HashMap;
26
+ use std::io::{Read, Write};
27
+ use std::sync::Mutex;
28
+
29
+ use serde::{Deserialize, Serialize};
30
+ use serde_json::{Value, json};
31
+
32
+ use crate::events::RuntimeRootsV1;
33
+
34
+ /// A-commit: what the submitting side registers for replication. `prompt_ids` doubles as the
35
+ /// context DA in loopback (the doc's Conversation-DA in miniature).
36
+ #[derive(Clone, Debug, Serialize, Deserialize)]
37
+ pub struct JobSubmission {
38
+ pub job_id: String,
39
+ pub provider_id: String,
40
+ pub prompt_ids: Vec<u32>,
41
+ pub max_new: u32,
42
+ /// blake2b-256 over the little-endian output ids — A's claim, compared against B's.
43
+ pub output_root: String,
44
+ #[serde(skip_serializing_if = "Option::is_none")]
45
+ pub receipt_json: Option<String>,
46
+ /// Engine execution roots (route/kv/state). When A commits them, an honest replica must
47
+ /// reproduce them — the match covers execution structure, not just the output ids.
48
+ #[serde(skip_serializing_if = "Option::is_none", default)]
49
+ pub runtime_roots: Option<RuntimeRootsV1>,
50
+ }
51
+
52
+ /// B's answer for an assignment.
53
+ #[derive(Clone, Debug, Serialize, Deserialize)]
54
+ pub struct ReplicaResultV1 {
55
+ pub job_id: String,
56
+ pub provider_id: String,
57
+ pub output_root: String,
58
+ #[serde(skip_serializing_if = "Option::is_none", default)]
59
+ pub runtime_roots: Option<RuntimeRootsV1>,
60
+ }
61
+
62
+ #[derive(Clone, Debug, Serialize, Deserialize)]
63
+ pub struct ReplicaAssignment {
64
+ pub job_id: String,
65
+ pub prompt_ids: Vec<u32>,
66
+ pub max_new: u32,
67
+ /// Absolute unix-ms deadline; the worker computes remaining time for admission.
68
+ pub deadline_unix_ms: i64,
69
+ }
70
+
71
+ /// What the A-side learns when polling. Mirrors `TurnVerificationStatus` names on the wire.
72
+ #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
73
+ #[serde(rename_all = "snake_case")]
74
+ pub enum JobVerdict {
75
+ ReplicaMatched,
76
+ Certified,
77
+ Mismatch,
78
+ }
79
+
80
+ impl JobVerdict {
81
+ pub fn as_str(self) -> &'static str {
82
+ match self {
83
+ Self::ReplicaMatched => "replica_matched",
84
+ Self::Certified => "certified",
85
+ Self::Mismatch => "mismatch",
86
+ }
87
+ }
88
+ pub fn from_str(s: &str) -> Option<Self> {
89
+ Some(match s {
90
+ "replica_matched" => Self::ReplicaMatched,
91
+ "certified" => Self::Certified,
92
+ "mismatch" => Self::Mismatch,
93
+ _ => return None,
94
+ })
95
+ }
96
+ }
97
+
98
+ pub trait PalwCoordinator: Send + Sync {
99
+ /// Idempotent by `job_id` — re-submitting a known job is an Ok no-op (worker restart safety).
100
+ fn submit_job(&self, submission: &JobSubmission) -> Result<(), String>;
101
+ /// Verdicts for the given jobs; jobs still pending are simply absent from the map.
102
+ fn fetch_verdicts(&self, job_ids: &[String]) -> Result<HashMap<String, JobVerdict>, String>;
103
+ /// Assignments offered to `provider_id` (claimed on fetch; a re-poll does not re-offer).
104
+ fn fetch_assignments(&self, provider_id: &str) -> Result<Vec<ReplicaAssignment>, String>;
105
+ fn decline_assignment(&self, job_id: &str, provider_id: &str, reason: &str) -> Result<(), String>;
106
+ fn submit_replica_result(&self, result: &ReplicaResultV1) -> Result<(), String>;
107
+ }
108
+
109
+ /// The match rule shared by coordinators: output roots must agree, and IF the submitter
110
+ /// committed runtime roots the replica must reproduce them exactly. (A submitter that omits
111
+ /// roots gets output-only matching — a weaker class, visible in the job record.)
112
+ pub fn replica_matches(submission: &JobSubmission, result: &ReplicaResultV1) -> bool {
113
+ if submission.output_root != result.output_root {
114
+ return false;
115
+ }
116
+ match &submission.runtime_roots {
117
+ Some(expected) => result.runtime_roots.as_ref() == Some(expected),
118
+ None => true,
119
+ }
120
+ }
121
+
122
+ pub fn output_root(ids: &[u32]) -> String {
123
+ use blake2::{Blake2b, Digest, digest::consts::U32};
124
+ let mut h = Blake2b::<U32>::new();
125
+ for id in ids {
126
+ h.update(id.to_le_bytes());
127
+ }
128
+ h.finalize().iter().map(|b| format!("{b:02x}")).collect()
129
+ }
130
+
131
+ // ---- loopback ---------------------------------------------------------------------------
132
+
133
+ #[derive(Clone, Copy, Debug, PartialEq, Eq)]
134
+ enum JobPhase {
135
+ /// Waiting for a replica provider (or requeued after decline/deadline).
136
+ Unassigned,
137
+ Assigned,
138
+ /// Replica root matched; the NEXT verdict fetch promotes to Certified so the A-side sees
139
+ /// the intermediate state exactly once.
140
+ Matched,
141
+ Certified,
142
+ Mismatch,
143
+ }
144
+
145
+ struct LoopbackJob {
146
+ submission: JobSubmission,
147
+ phase: JobPhase,
148
+ assigned_to: Option<String>,
149
+ deadline_unix_ms: i64,
150
+ declines: u32,
151
+ }
152
+
153
+ pub struct LoopbackConfig {
154
+ /// Offer a job back to its own submitter when nobody else polls. TRUE is the single-desk
155
+ /// dev default; it validates plumbing while (see module docs) proving nothing about
156
+ /// independence.
157
+ pub allow_self_replica: bool,
158
+ pub assignment_deadline_ms: i64,
159
+ }
160
+
161
+ impl Default for LoopbackConfig {
162
+ fn default() -> Self {
163
+ Self { allow_self_replica: true, assignment_deadline_ms: 120_000 }
164
+ }
165
+ }
166
+
167
+ pub struct LoopbackCoordinator {
168
+ config: LoopbackConfig,
169
+ jobs: Mutex<HashMap<String, LoopbackJob>>,
170
+ }
171
+
172
+ fn now_unix_ms() -> i64 {
173
+ std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as i64).unwrap_or(0)
174
+ }
175
+
176
+ impl LoopbackCoordinator {
177
+ pub fn new(config: LoopbackConfig) -> Self {
178
+ Self { config, jobs: Mutex::new(HashMap::new()) }
179
+ }
180
+
181
+ /// Introspection for `/v1/palw/status`.
182
+ pub fn counts(&self) -> HashMap<&'static str, u64> {
183
+ let jobs = self.jobs.lock().unwrap();
184
+ let mut out: HashMap<&'static str, u64> =
185
+ [("unassigned", 0), ("assigned", 0), ("matched", 0), ("certified", 0), ("mismatch", 0)].into();
186
+ for j in jobs.values() {
187
+ let key = match j.phase {
188
+ JobPhase::Unassigned => "unassigned",
189
+ JobPhase::Assigned => "assigned",
190
+ JobPhase::Matched => "matched",
191
+ JobPhase::Certified => "certified",
192
+ JobPhase::Mismatch => "mismatch",
193
+ };
194
+ *out.get_mut(key).unwrap() += 1;
195
+ }
196
+ out
197
+ }
198
+ }
199
+
200
+ impl PalwCoordinator for LoopbackCoordinator {
201
+ fn submit_job(&self, submission: &JobSubmission) -> Result<(), String> {
202
+ let mut jobs = self.jobs.lock().unwrap();
203
+ if jobs.contains_key(&submission.job_id) {
204
+ return Ok(()); // idempotent re-submit
205
+ }
206
+ jobs.insert(
207
+ submission.job_id.clone(),
208
+ LoopbackJob {
209
+ submission: submission.clone(),
210
+ phase: JobPhase::Unassigned,
211
+ assigned_to: None,
212
+ deadline_unix_ms: 0,
213
+ declines: 0,
214
+ },
215
+ );
216
+ Ok(())
217
+ }
218
+
219
+ fn fetch_verdicts(&self, job_ids: &[String]) -> Result<HashMap<String, JobVerdict>, String> {
220
+ let mut jobs = self.jobs.lock().unwrap();
221
+ let mut out = HashMap::new();
222
+ for id in job_ids {
223
+ if let Some(job) = jobs.get_mut(id) {
224
+ match job.phase {
225
+ JobPhase::Matched => {
226
+ out.insert(id.clone(), JobVerdict::ReplicaMatched);
227
+ // Loopback has no auditor lottery: a matched replica certifies on the
228
+ // next observation.
229
+ job.phase = JobPhase::Certified;
230
+ }
231
+ JobPhase::Certified => {
232
+ out.insert(id.clone(), JobVerdict::Certified);
233
+ }
234
+ JobPhase::Mismatch => {
235
+ out.insert(id.clone(), JobVerdict::Mismatch);
236
+ }
237
+ JobPhase::Unassigned | JobPhase::Assigned => {}
238
+ }
239
+ }
240
+ }
241
+ Ok(out)
242
+ }
243
+
244
+ fn fetch_assignments(&self, provider_id: &str) -> Result<Vec<ReplicaAssignment>, String> {
245
+ let now = now_unix_ms();
246
+ let mut jobs = self.jobs.lock().unwrap();
247
+ let mut out = Vec::new();
248
+ for job in jobs.values_mut() {
249
+ // Deadline enforcement: an assigned-but-silent replica loses the claim.
250
+ if job.phase == JobPhase::Assigned && now > job.deadline_unix_ms {
251
+ job.phase = JobPhase::Unassigned;
252
+ job.assigned_to = None;
253
+ }
254
+ if job.phase != JobPhase::Unassigned {
255
+ continue;
256
+ }
257
+ if job.submission.provider_id == provider_id && !self.config.allow_self_replica {
258
+ continue;
259
+ }
260
+ job.phase = JobPhase::Assigned;
261
+ job.assigned_to = Some(provider_id.to_string());
262
+ job.deadline_unix_ms = now + self.config.assignment_deadline_ms;
263
+ out.push(ReplicaAssignment {
264
+ job_id: job.submission.job_id.clone(),
265
+ prompt_ids: job.submission.prompt_ids.clone(),
266
+ max_new: job.submission.max_new,
267
+ deadline_unix_ms: job.deadline_unix_ms,
268
+ });
269
+ }
270
+ Ok(out)
271
+ }
272
+
273
+ fn decline_assignment(&self, job_id: &str, provider_id: &str, _reason: &str) -> Result<(), String> {
274
+ let mut jobs = self.jobs.lock().unwrap();
275
+ let job = jobs.get_mut(job_id).ok_or_else(|| format!("unknown job {job_id}"))?;
276
+ if job.phase == JobPhase::Assigned && job.assigned_to.as_deref() == Some(provider_id) {
277
+ job.phase = JobPhase::Unassigned;
278
+ job.assigned_to = None;
279
+ job.declines += 1;
280
+ }
281
+ Ok(())
282
+ }
283
+
284
+ fn submit_replica_result(&self, result: &ReplicaResultV1) -> Result<(), String> {
285
+ let now = now_unix_ms();
286
+ let job_id = result.job_id.as_str();
287
+ let mut jobs = self.jobs.lock().unwrap();
288
+ let job = jobs.get_mut(job_id).ok_or_else(|| format!("unknown job {job_id}"))?;
289
+ if job.phase != JobPhase::Assigned || job.assigned_to.as_deref() != Some(result.provider_id.as_str()) {
290
+ return Err(format!("job {job_id} is not assigned to {}", result.provider_id));
291
+ }
292
+ if now > job.deadline_unix_ms {
293
+ // Too late: the claim is gone, the job goes back to the queue.
294
+ job.phase = JobPhase::Unassigned;
295
+ job.assigned_to = None;
296
+ return Err(format!("job {job_id} deadline passed — result ignored, job requeued"));
297
+ }
298
+ job.phase = if replica_matches(&job.submission, result) { JobPhase::Matched } else { JobPhase::Mismatch };
299
+ Ok(())
300
+ }
301
+ }
302
+
303
+ // ---- HTTP client ------------------------------------------------------------------------
304
+
305
+ /// Client half of the loopback JSON protocol (see README "PALW coordinator protocol"). The
306
+ /// server half is the gateway's `/v1/palw/loopback/*` routes — or, later, a node-side bridge.
307
+ pub struct HttpCoordinator {
308
+ /// e.g. `http://127.0.0.1:12346/v1/palw/loopback`
309
+ base: String,
310
+ bearer: Option<String>,
311
+ }
312
+
313
+ impl HttpCoordinator {
314
+ pub fn new(base: &str, bearer: Option<String>) -> Self {
315
+ Self { base: base.trim_end_matches('/').to_string(), bearer }
316
+ }
317
+
318
+ fn call(&self, method: &str, path_and_query: &str, body: Option<&Value>) -> Result<Value, String> {
319
+ let url = format!("{}{}", self.base, path_and_query);
320
+ let (status, bytes) = http_request(method, &url, self.bearer.as_deref(), body)?;
321
+ let value: Value = if bytes.is_empty() {
322
+ Value::Null
323
+ } else {
324
+ serde_json::from_slice(&bytes).map_err(|e| format!("{method} {url}: bad json: {e}"))?
325
+ };
326
+ if status != 200 {
327
+ let msg = value
328
+ .get("error")
329
+ .and_then(|e| e.get("message"))
330
+ .and_then(|m| m.as_str())
331
+ .unwrap_or("unexpected status");
332
+ return Err(format!("{method} {url}: {status} {msg}"));
333
+ }
334
+ Ok(value)
335
+ }
336
+ }
337
+
338
+ impl PalwCoordinator for HttpCoordinator {
339
+ fn submit_job(&self, submission: &JobSubmission) -> Result<(), String> {
340
+ let body = serde_json::to_value(submission).map_err(|e| e.to_string())?;
341
+ self.call("POST", "/jobs", Some(&body)).map(|_| ())
342
+ }
343
+
344
+ fn fetch_verdicts(&self, job_ids: &[String]) -> Result<HashMap<String, JobVerdict>, String> {
345
+ let response = self.call("POST", "/verdicts", Some(&json!({ "job_ids": job_ids })))?;
346
+ let mut out = HashMap::new();
347
+ if let Some(items) = response.get("verdicts").and_then(|v| v.as_array()) {
348
+ for item in items {
349
+ let (Some(id), Some(verdict)) = (
350
+ item.get("job_id").and_then(|j| j.as_str()),
351
+ item.get("verdict").and_then(|v| v.as_str()).and_then(JobVerdict::from_str),
352
+ ) else {
353
+ continue;
354
+ };
355
+ out.insert(id.to_string(), verdict);
356
+ }
357
+ }
358
+ Ok(out)
359
+ }
360
+
361
+ fn fetch_assignments(&self, provider_id: &str) -> Result<Vec<ReplicaAssignment>, String> {
362
+ let response = self.call("GET", &format!("/assignments?provider_id={provider_id}"), None)?;
363
+ let mut out = Vec::new();
364
+ if let Some(items) = response.get("assignments").and_then(|v| v.as_array()) {
365
+ for item in items {
366
+ out.push(serde_json::from_value(item.clone()).map_err(|e| format!("bad assignment: {e}"))?);
367
+ }
368
+ }
369
+ Ok(out)
370
+ }
371
+
372
+ fn decline_assignment(&self, job_id: &str, provider_id: &str, reason: &str) -> Result<(), String> {
373
+ self.call(
374
+ "POST",
375
+ &format!("/assignments/{job_id}/decline"),
376
+ Some(&json!({ "provider_id": provider_id, "reason": reason })),
377
+ )
378
+ .map(|_| ())
379
+ }
380
+
381
+ fn submit_replica_result(&self, result: &ReplicaResultV1) -> Result<(), String> {
382
+ let body = serde_json::to_value(result).map_err(|e| e.to_string())?;
383
+ self.call("POST", "/replica-results", Some(&body)).map(|_| ())
384
+ }
385
+ }
386
+
387
+ /// Minimal HTTP/1.1 request over plain TCP (`http://host[:port]/path` only). Deliberately
388
+ /// dependency-free: this is a localhost/LAN dev transport with `Connection: close` framing —
389
+ /// the response is everything until EOF, which tiny_http honors for close-marked requests.
390
+ fn http_request(method: &str, url: &str, bearer: Option<&str>, body: Option<&Value>) -> Result<(u16, Vec<u8>), String> {
391
+ let rest = url.strip_prefix("http://").ok_or_else(|| format!("only http:// urls are supported, got {url}"))?;
392
+ let (host_port, path) = match rest.split_once('/') {
393
+ Some((hp, p)) => (hp, format!("/{p}")),
394
+ None => (rest, "/".to_string()),
395
+ };
396
+ let authority = if host_port.contains(':') { host_port.to_string() } else { format!("{host_port}:80") };
397
+
398
+ let mut stream = std::net::TcpStream::connect(&authority).map_err(|e| format!("connect {authority}: {e}"))?;
399
+ stream.set_read_timeout(Some(std::time::Duration::from_secs(30))).map_err(|e| e.to_string())?;
400
+ stream.set_write_timeout(Some(std::time::Duration::from_secs(30))).map_err(|e| e.to_string())?;
401
+
402
+ let body_bytes = body.map(serde_json::to_vec).transpose().map_err(|e| e.to_string())?.unwrap_or_default();
403
+ let mut request = format!(
404
+ "{method} {path} HTTP/1.1\r\nHost: {host_port}\r\nConnection: close\r\nContent-Type: application/json\r\nContent-Length: {}\r\n",
405
+ body_bytes.len()
406
+ );
407
+ if let Some(token) = bearer {
408
+ request.push_str(&format!("Authorization: Bearer {token}\r\n"));
409
+ }
410
+ request.push_str("\r\n");
411
+ stream.write_all(request.as_bytes()).map_err(|e| format!("write {url}: {e}"))?;
412
+ stream.write_all(&body_bytes).map_err(|e| format!("write {url}: {e}"))?;
413
+
414
+ let mut response = Vec::new();
415
+ stream.read_to_end(&mut response).map_err(|e| format!("read {url}: {e}"))?;
416
+ let header_end = response
417
+ .windows(4)
418
+ .position(|w| w == b"\r\n\r\n")
419
+ .ok_or_else(|| format!("{url}: malformed response"))?;
420
+ let head = std::str::from_utf8(&response[..header_end]).map_err(|_| "non-utf8 response head".to_string())?;
421
+ let status: u16 = head
422
+ .lines()
423
+ .next()
424
+ .and_then(|l| l.split_whitespace().nth(1))
425
+ .and_then(|s| s.parse().ok())
426
+ .ok_or_else(|| format!("{url}: bad status line"))?;
427
+ let mut body = response[header_end + 4..].to_vec();
428
+
429
+ // tiny_http chunk-frames close-delimited responses too; unwrap if so.
430
+ if head.to_ascii_lowercase().contains("transfer-encoding: chunked") {
431
+ body = dechunk(&body)?;
432
+ }
433
+ Ok((status, body))
434
+ }
435
+
436
+ fn dechunk(mut data: &[u8]) -> Result<Vec<u8>, String> {
437
+ let mut out = Vec::new();
438
+ loop {
439
+ let line_end = data.windows(2).position(|w| w == b"\r\n").ok_or("chunked: missing size line")?;
440
+ let size_str = std::str::from_utf8(&data[..line_end]).map_err(|_| "chunked: bad size")?;
441
+ let size = usize::from_str_radix(size_str.trim().split(';').next().unwrap_or(""), 16)
442
+ .map_err(|_| format!("chunked: bad size {size_str:?}"))?;
443
+ data = &data[line_end + 2..];
444
+ if size == 0 {
445
+ return Ok(out);
446
+ }
447
+ if data.len() < size + 2 {
448
+ return Err("chunked: truncated body".into());
449
+ }
450
+ out.extend_from_slice(&data[..size]);
451
+ data = &data[size + 2..];
452
+ }
453
+ }
454
+
455
+ #[cfg(test)]
456
+ mod tests {
457
+ use super::*;
458
+
459
+ fn submission(job: &str, provider: &str, root: &str) -> JobSubmission {
460
+ JobSubmission {
461
+ job_id: job.into(),
462
+ provider_id: provider.into(),
463
+ prompt_ids: vec![1, 2, 3],
464
+ max_new: 16,
465
+ output_root: root.into(),
466
+ receipt_json: None,
467
+ runtime_roots: None,
468
+ }
469
+ }
470
+
471
+ fn result(job: &str, provider: &str, root: &str) -> ReplicaResultV1 {
472
+ ReplicaResultV1 { job_id: job.into(), provider_id: provider.into(), output_root: root.into(), runtime_roots: None }
473
+ }
474
+
475
+ #[test]
476
+ fn loopback_full_match_pipeline() {
477
+ let c = LoopbackCoordinator::new(LoopbackConfig::default());
478
+ let root = output_root(&[7, 8, 9]);
479
+ c.submit_job(&submission("j1", "prov-a", &root)).unwrap();
480
+ c.submit_job(&submission("j1", "prov-a", &root)).unwrap(); // idempotent
481
+
482
+ // No verdict while unassigned.
483
+ assert!(c.fetch_verdicts(&["j1".into()]).unwrap().is_empty());
484
+
485
+ // Another provider claims it; a re-poll does not double-offer.
486
+ let got = c.fetch_assignments("prov-b").unwrap();
487
+ assert_eq!(got.len(), 1);
488
+ assert_eq!(got[0].prompt_ids, vec![1, 2, 3]);
489
+ assert!(c.fetch_assignments("prov-b").unwrap().is_empty());
490
+
491
+ c.submit_replica_result(&result("j1", "prov-b", &output_root(&[7, 8, 9]))).unwrap();
492
+ // First observation: matched. Second: certified. (Intermediate state seen exactly once.)
493
+ assert_eq!(c.fetch_verdicts(&["j1".into()]).unwrap()["j1"], JobVerdict::ReplicaMatched);
494
+ assert_eq!(c.fetch_verdicts(&["j1".into()]).unwrap()["j1"], JobVerdict::Certified);
495
+ }
496
+
497
+ #[test]
498
+ fn loopback_mismatch_and_wrong_provider() {
499
+ let c = LoopbackCoordinator::new(LoopbackConfig::default());
500
+ c.submit_job(&submission("j1", "prov-a", &output_root(&[7]))).unwrap();
501
+ let _ = c.fetch_assignments("prov-b").unwrap();
502
+ // A provider that never claimed the job cannot answer for it.
503
+ assert!(c.submit_replica_result(&result("j1", "prov-c", &output_root(&[7]))).is_err());
504
+ c.submit_replica_result(&result("j1", "prov-b", &output_root(&[999]))).unwrap();
505
+ assert_eq!(c.fetch_verdicts(&["j1".into()]).unwrap()["j1"], JobVerdict::Mismatch);
506
+ // Mismatch is terminal.
507
+ assert_eq!(c.fetch_verdicts(&["j1".into()]).unwrap()["j1"], JobVerdict::Mismatch);
508
+ }
509
+
510
+ #[test]
511
+ fn loopback_decline_and_deadline_requeue() {
512
+ let c = LoopbackCoordinator::new(LoopbackConfig { allow_self_replica: false, assignment_deadline_ms: 0 });
513
+ c.submit_job(&submission("j1", "prov-a", "r")).unwrap();
514
+ // Self-replica disabled: the submitter is never offered its own job.
515
+ assert!(c.fetch_assignments("prov-a").unwrap().is_empty());
516
+
517
+ let got = c.fetch_assignments("prov-b").unwrap();
518
+ assert_eq!(got.len(), 1);
519
+ c.decline_assignment("j1", "prov-b", "over capacity").unwrap();
520
+ // Declined ⇒ requeued ⇒ another provider can claim it.
521
+ let got = c.fetch_assignments("prov-c").unwrap();
522
+ assert_eq!(got.len(), 1);
523
+ // deadline_ms=0 ⇒ instantly late: the result is refused and the job requeued.
524
+ std::thread::sleep(std::time::Duration::from_millis(5));
525
+ assert!(c.submit_replica_result(&result("j1", "prov-c", "r")).is_err());
526
+ assert_eq!(c.fetch_assignments("prov-d").unwrap().len(), 1);
527
+ }
528
+
529
+ #[test]
530
+ fn output_root_is_order_and_value_sensitive() {
531
+ assert_eq!(output_root(&[1, 2]), output_root(&[1, 2]));
532
+ assert_ne!(output_root(&[1, 2]), output_root(&[2, 1]));
533
+ assert_ne!(output_root(&[1]), output_root(&[]));
534
+ }
535
+
536
+ #[test]
537
+ fn committed_roots_bind_the_match() {
538
+ use crate::events::RuntimeRootsV1;
539
+ let roots = RuntimeRootsV1 { route: "aa".into(), kv: "bb".into(), state: "cc".into() };
540
+ let mut sub = submission("j1", "prov-a", "root");
541
+ sub.runtime_roots = Some(roots.clone());
542
+
543
+ let mut ok = result("j1", "prov-b", "root");
544
+ ok.runtime_roots = Some(roots.clone());
545
+ assert!(replica_matches(&sub, &ok));
546
+
547
+ // Replica omits or alters the committed roots ⇒ mismatch even with equal output.
548
+ assert!(!replica_matches(&sub, &result("j1", "prov-b", "root")));
549
+ let mut altered = result("j1", "prov-b", "root");
550
+ altered.runtime_roots = Some(RuntimeRootsV1 { route: "zz".into(), kv: "bb".into(), state: "cc".into() });
551
+ assert!(!replica_matches(&sub, &altered));
552
+
553
+ // Submitter without roots ⇒ output-only class; replica extras are ignored.
554
+ let plain = submission("j2", "prov-a", "root");
555
+ let mut extra = result("j2", "prov-b", "root");
556
+ extra.runtime_roots = Some(roots);
557
+ assert!(replica_matches(&plain, &extra));
558
+ assert!(!replica_matches(&plain, &result("j2", "prov-b", "other")));
559
+ }
560
+
561
+ #[test]
562
+ fn dechunk_unwraps_tiny_http_framing() {
563
+ let framed = b"5\r\nhello\r\n6\r\n world\r\n0\r\n\r\n";
564
+ assert_eq!(dechunk(framed).unwrap(), b"hello world");
565
+ assert!(dechunk(b"zz\r\n").is_err());
566
+ }
567
+ }
palw-gateway/src/engine.rs ADDED
@@ -0,0 +1,239 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! EngineHost — owns the resident `qi35_model --serve` child and speaks its line protocol
2
+ //! (`SERVE_READY` / `REQ n ids [DRAFT ids]` / `TOK` / `CACHE` / `SPEC` / `GEN` / `ROOTS` /
3
+ //! `RCPT` / `DONE` / `ERR`), exactly as read from `docs/evidence/qi35_model.rs:2432-2475`.
4
+ //!
5
+ //! Serving-loop facts this host designs around:
6
+ //! * One engine process = one generation at a time. The gateway serializes requests through a
7
+ //! queue (scheduler); multi-engine is a later phase.
8
+ //! * There is NO stop channel in the engine. Cancel therefore = stop forwarding tokens, drain
9
+ //! the stream to `DONE`, and mark the turn `MintIneligible` — the v1 cancel semantics the
10
+ //! product doc freezes (`--max-new` bounds the worst-case drain).
11
+ //! * Crash recovery: any read/parse failure poisons the child; the next request respawns it
12
+ //! (model load is expensive, so we never preemptively restart).
13
+
14
+ use std::io::{BufRead, BufReader, Write};
15
+ use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio};
16
+ use std::sync::atomic::{AtomicBool, Ordering};
17
+ use std::sync::Arc;
18
+
19
+ #[derive(Clone, Debug)]
20
+ pub struct EngineConfig {
21
+ pub binary: std::path::PathBuf,
22
+ pub gguf: std::path::PathBuf,
23
+ pub tables: std::path::PathBuf,
24
+ pub threads: u32,
25
+ pub max_new: u32,
26
+ pub eos_id: u32,
27
+ pub metal_v3: bool,
28
+ pub audit_key_file: Option<std::path::PathBuf>,
29
+ pub network_label: String,
30
+ }
31
+
32
+ pub struct EngineOutcome {
33
+ pub output_ids: Vec<u32>,
34
+ pub cache_hit_tokens: u32,
35
+ pub receipt_json: Option<String>,
36
+ /// The `ROOTS route= kv= state=` execution commitments (present since the host always sets
37
+ /// `QI35_SERVE_ROOTS`; None only against an older engine build that lacks the line).
38
+ pub runtime_roots: Option<crate::events::RuntimeRootsV1>,
39
+ pub eos_reached: bool,
40
+ pub elapsed_ms: u64,
41
+ }
42
+
43
+ pub struct EngineHost {
44
+ config: EngineConfig,
45
+ child: Option<EngineChild>,
46
+ }
47
+
48
+ struct EngineChild {
49
+ process: Child,
50
+ stdin: ChildStdin,
51
+ stdout: BufReader<ChildStdout>,
52
+ }
53
+
54
+ impl Drop for EngineHost {
55
+ fn drop(&mut self) {
56
+ if let Some(mut c) = self.child.take() {
57
+ let _ = writeln!(c.stdin, "/quit");
58
+ let _ = c.process.wait();
59
+ }
60
+ }
61
+ }
62
+
63
+ impl EngineHost {
64
+ pub fn new(config: EngineConfig) -> Self {
65
+ Self { config, child: None }
66
+ }
67
+
68
+ fn spawn(&mut self) -> Result<(), String> {
69
+ let cfg = &self.config;
70
+ let mut cmd = Command::new(&cfg.binary);
71
+ // The Metal build dlopens libqi35metal.dylib relative to the working directory; run the
72
+ // engine FROM its own directory (the launcher-script convention) so the dylib resolves.
73
+ if let Some(dir) = cfg.binary.parent() {
74
+ if !dir.as_os_str().is_empty() {
75
+ cmd.current_dir(dir);
76
+ }
77
+ }
78
+ // Phase D: 1 token = 1 command buffer — bit-exact, ~4x decode (the launcher's default).
79
+ cmd.env("QI35_PHASE_D", "1");
80
+ // Emit `ROOTS route= kv= state=` after every generation: the execution commitments the
81
+ // PALW replica match consumes (negligible cost — the roots are already computed).
82
+ cmd.env("QI35_SERVE_ROOTS", "1");
83
+ cmd.arg(&cfg.gguf)
84
+ .arg(&cfg.tables)
85
+ .arg("--serve")
86
+ .arg("--threads")
87
+ .arg(cfg.threads.to_string())
88
+ .arg("--max-new")
89
+ .arg(cfg.max_new.to_string())
90
+ .arg("--eos")
91
+ .arg(cfg.eos_id.to_string())
92
+ .stdin(Stdio::piped())
93
+ .stdout(Stdio::piped())
94
+ .stderr(Stdio::inherit());
95
+ if cfg.metal_v3 {
96
+ cmd.arg("--metal-v3");
97
+ }
98
+ if let Some(key) = &cfg.audit_key_file {
99
+ cmd.arg("--audit-key-file").arg(key);
100
+ cmd.arg("--network").arg(&cfg.network_label);
101
+ }
102
+ let mut process = cmd.spawn().map_err(|e| format!("spawn {}: {e}", cfg.binary.display()))?;
103
+ let stdin = process.stdin.take().ok_or("engine stdin unavailable")?;
104
+ let stdout = BufReader::new(process.stdout.take().ok_or("engine stdout unavailable")?);
105
+ let mut child = EngineChild { process, stdin, stdout };
106
+
107
+ // Model load can take minutes on first touch; read until SERVE_READY with no artificial
108
+ // deadline — a wedged load surfaces as EOF (child died), never a silent hang forever,
109
+ // because the child's death closes the pipe.
110
+ let mut line = String::new();
111
+ loop {
112
+ line.clear();
113
+ let n = child.stdout.read_line(&mut line).map_err(|e| format!("engine handshake read: {e}"))?;
114
+ if n == 0 {
115
+ let _ = child.process.wait();
116
+ return Err("engine exited before SERVE_READY (see its stderr above)".into());
117
+ }
118
+ if line.trim() == "SERVE_READY" {
119
+ break;
120
+ }
121
+ }
122
+ self.child = Some(child);
123
+ Ok(())
124
+ }
125
+
126
+ pub fn ensure_running(&mut self) -> Result<(), String> {
127
+ let dead = match &mut self.child {
128
+ None => true,
129
+ Some(c) => c.process.try_wait().map(|s| s.is_some()).unwrap_or(true),
130
+ };
131
+ if dead {
132
+ self.child = None;
133
+ self.spawn()?;
134
+ }
135
+ Ok(())
136
+ }
137
+
138
+ pub fn healthy(&mut self) -> bool {
139
+ if let Some(c) = &mut self.child {
140
+ c.process.try_wait().map(|s| s.is_none()).unwrap_or(false)
141
+ } else {
142
+ false
143
+ }
144
+ }
145
+
146
+ /// Run one generation. `on_token` receives each token id as it streams; `cancel` flips the
147
+ /// stream into drain mode (tokens no longer forwarded, but the protocol is consumed to
148
+ /// `DONE` so the child stays reusable). Any protocol violation poisons the child.
149
+ pub fn generate(
150
+ &mut self,
151
+ prompt_ids: &[u32],
152
+ cache_prefix_len: usize,
153
+ cancel: &Arc<AtomicBool>,
154
+ mut on_token: impl FnMut(u32, u32),
155
+ ) -> Result<EngineOutcome, String> {
156
+ self.ensure_running()?;
157
+ let started = std::time::Instant::now();
158
+
159
+ let idstr: String = prompt_ids.iter().map(|i| i.to_string()).collect::<Vec<_>>().join(",");
160
+ let write_result = {
161
+ let child = self.child.as_mut().expect("ensured above");
162
+ writeln!(child.stdin, "REQ {cache_prefix_len} {idstr}").and_then(|()| child.stdin.flush())
163
+ };
164
+ if let Err(e) = write_result {
165
+ self.child = None;
166
+ return Err(format!("engine write: {e}"));
167
+ }
168
+
169
+ let mut output_ids: Vec<u32> = Vec::new();
170
+ let mut cache_hit_tokens = 0u32;
171
+ let mut receipt_json: Option<String> = None;
172
+ let mut runtime_roots: Option<crate::events::RuntimeRootsV1> = None;
173
+ let mut token_index = 0u32;
174
+ let mut line = String::new();
175
+ loop {
176
+ line.clear();
177
+ let n = {
178
+ let child = self.child.as_mut().expect("held");
179
+ child.stdout.read_line(&mut line).map_err(|e| format!("engine read: {e}"))?
180
+ };
181
+ if n == 0 {
182
+ self.child = None;
183
+ return Err("engine died mid-generation".into());
184
+ }
185
+ let line = line.trim_end();
186
+ if let Some(tok) = line.strip_prefix("TOK ") {
187
+ let id: u32 = match tok.trim().parse() {
188
+ Ok(id) => id,
189
+ Err(_) => {
190
+ self.child = None;
191
+ return Err(format!("engine protocol: bad TOK line {line:?}"));
192
+ }
193
+ };
194
+ if !cancel.load(Ordering::Relaxed) {
195
+ on_token(token_index, id);
196
+ }
197
+ token_index += 1;
198
+ } else if let Some(rest) = line.strip_prefix("CACHE ") {
199
+ for part in rest.split_whitespace() {
200
+ if let Some(v) = part.strip_prefix("hit_tokens=") {
201
+ cache_hit_tokens = v.parse().unwrap_or(0);
202
+ }
203
+ }
204
+ } else if let Some(rest) = line.strip_prefix("GEN ") {
205
+ output_ids = match rest.split(',').filter(|s| !s.is_empty()).map(|s| s.trim().parse::<u32>()).collect::<Result<_, _>>()
206
+ {
207
+ Ok(ids) => ids,
208
+ Err(_) => {
209
+ self.child = None;
210
+ return Err("engine protocol: bad GEN line".into());
211
+ }
212
+ };
213
+ } else if let Some(json) = line.strip_prefix("RCPT ") {
214
+ receipt_json = Some(json.to_string());
215
+ } else if let Some(rest) = line.strip_prefix("ROOTS ") {
216
+ let field = |key: &str| {
217
+ rest.split_whitespace().find_map(|part| part.strip_prefix(key).map(str::to_string))
218
+ };
219
+ if let (Some(route), Some(kv), Some(state)) = (field("route="), field("kv="), field("state=")) {
220
+ runtime_roots = Some(crate::events::RuntimeRootsV1 { route, kv, state });
221
+ }
222
+ } else if line.starts_with("DONE ") {
223
+ let eos_reached = (output_ids.len() as u32) < self.config.max_new
224
+ || output_ids.last() == Some(&self.config.eos_id);
225
+ return Ok(EngineOutcome {
226
+ output_ids,
227
+ cache_hit_tokens,
228
+ receipt_json,
229
+ runtime_roots,
230
+ eos_reached,
231
+ elapsed_ms: started.elapsed().as_millis() as u64,
232
+ });
233
+ } else if let Some(err) = line.strip_prefix("ERR ") {
234
+ return Err(format!("engine rejected request: {err}"));
235
+ }
236
+ // SPEC and anything else: informational, skipped.
237
+ }
238
+ }
239
+ }
palw-gateway/src/events.rs ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Wire types frozen by `docs/palw-desktop-product-architecture.md`.
2
+ //!
3
+ //! Two clocks, never conflated: `GenerationEventV1` is the ANSWER clock (local A tokens,
4
+ //! streamed immediately); `TurnVerificationStatus` is the SETTLEMENT clock (B replica → match →
5
+ //! audit → certificate → maturity), advanced later by the PALW background supervisor (UX Phase
6
+ //! 2). Phase 0 persists the status column from day one so the schema never migrates for it, but
7
+ //! only ever writes `Streaming`/`LocalComplete`/`MintIneligible`.
8
+
9
+ use serde::{Deserialize, Serialize};
10
+
11
+ /// 64-byte identity, hex on the wire (the repo-wide Hash64 idiom, without pulling consensus
12
+ /// crates into the product runtime).
13
+ pub type HexHash = String;
14
+
15
+ #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
16
+ #[serde(rename_all = "snake_case", tag = "type")]
17
+ pub enum GenerationEventV1 {
18
+ Started {
19
+ job_id: HexHash,
20
+ turn_id: HexHash,
21
+ /// Engine-reported prefix-cache reuse is only known at the end; this is the compiled
22
+ /// prompt length so clients can show prefill progress honestly.
23
+ prompt_tokens: u32,
24
+ },
25
+ Token {
26
+ index: u32,
27
+ token_id: u32,
28
+ /// UTF-8 bytes that became SAFE to display at this token (the incremental decoder's
29
+ /// stable prefix delta — may be empty while a multi-byte/multi-token grapheme is open).
30
+ utf8_delta: String,
31
+ },
32
+ /// Reserved for the Phase-1 tool runtime; the Phase-0 engine never emits it, but freezing
33
+ /// the variant now means stream consumers are written against the final shape.
34
+ ToolCall {
35
+ call_id: HexHash,
36
+ tool_name: String,
37
+ arguments_json: String,
38
+ },
39
+ Completed {
40
+ stop_reason: CanonicalStopReason,
41
+ output_tokens: u32,
42
+ /// Engine-reported `CACHE hit_tokens=…` — how much prefill the id-stable history saved.
43
+ cache_hit_tokens: u32,
44
+ elapsed_ms: u64,
45
+ },
46
+ Failed {
47
+ error_code: u32,
48
+ message: String,
49
+ },
50
+ }
51
+
52
+ #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)]
53
+ #[serde(rename_all = "snake_case")]
54
+ pub enum CanonicalStopReason {
55
+ /// The model emitted EOS.
56
+ Eos,
57
+ /// The per-request output budget was reached.
58
+ MaxTokens,
59
+ /// The user stopped generation. v1 semantics (frozen in the design doc): the partial answer
60
+ /// is kept locally and the turn is `MintIneligible` — B cannot reproduce the user's stop
61
+ /// instant, so a cancelled turn never enters the settlement pipeline.
62
+ UserCancelled,
63
+ }
64
+
65
+ /// The settlement clock (design doc §"最重要"). Stored on every turn; Phase 0 writes only the
66
+ /// first two and `MintIneligible` (cancel), the PALW supervisor owns the rest in Phase 2.
67
+ #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)]
68
+ #[serde(rename_all = "snake_case")]
69
+ pub enum TurnVerificationStatus {
70
+ Streaming,
71
+ LocalComplete,
72
+ ReplicaPending,
73
+ ReplicaMatched,
74
+ AuditPending,
75
+ Certified,
76
+ Matured,
77
+ Mismatch,
78
+ MintIneligible,
79
+ }
80
+
81
+ impl TurnVerificationStatus {
82
+ pub fn as_str(self) -> &'static str {
83
+ match self {
84
+ Self::Streaming => "streaming",
85
+ Self::LocalComplete => "local_complete",
86
+ Self::ReplicaPending => "replica_pending",
87
+ Self::ReplicaMatched => "replica_matched",
88
+ Self::AuditPending => "audit_pending",
89
+ Self::Certified => "certified",
90
+ Self::Matured => "matured",
91
+ Self::Mismatch => "mismatch",
92
+ Self::MintIneligible => "mint_ineligible",
93
+ }
94
+ }
95
+
96
+ pub fn from_str(s: &str) -> Option<Self> {
97
+ Some(match s {
98
+ "streaming" => Self::Streaming,
99
+ "local_complete" => Self::LocalComplete,
100
+ "replica_pending" => Self::ReplicaPending,
101
+ "replica_matched" => Self::ReplicaMatched,
102
+ "audit_pending" => Self::AuditPending,
103
+ "certified" => Self::Certified,
104
+ "matured" => Self::Matured,
105
+ "mismatch" => Self::Mismatch,
106
+ "mint_ineligible" => Self::MintIneligible,
107
+ _ => return None,
108
+ })
109
+ }
110
+ }
111
+
112
+ /// The engine's per-generation execution roots (`ROOTS route= kv= state=`, emitted under
113
+ /// `QI35_SERVE_ROOTS`): the MoE routing-trace root, the KV-cache root, and the recurrent-state
114
+ /// root — real runtime commitments beyond the output ids. Carried on A-commits and replica
115
+ /// results so a coordinator can match replicas on execution structure, not just output.
116
+ #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
117
+ pub struct RuntimeRootsV1 {
118
+ pub route: String,
119
+ pub kv: String,
120
+ pub state: String,
121
+ }
122
+
123
+ /// Dual-head pointer pair (design doc §4). `certified_head` stays NULL until UX Phase 2 wires
124
+ /// the supervisor; the column existing from day one is the point — turn N+1 never waits on turn
125
+ /// N's verification, and the schema already knows the difference.
126
+ #[derive(Clone, Debug, Serialize, Deserialize)]
127
+ pub struct ConversationHeadsV1 {
128
+ pub local_head: Option<HexHash>,
129
+ pub certified_head: Option<HexHash>,
130
+ }
131
+
132
+ /// The privacy/mint mode, fixed BEFORE the message is sent (design doc §16 — deciding after
133
+ /// generation invites grinding). Phase 0 implements LocalOnly semantics for all three but
134
+ /// records the user's choice so turns created under a mint intent are marked from birth.
135
+ #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)]
136
+ #[serde(rename_all = "snake_case")]
137
+ pub enum TurnPrivacyMode {
138
+ LocalOnly,
139
+ VerifiedNoMint,
140
+ PalwMint,
141
+ }
142
+
143
+ impl TurnPrivacyMode {
144
+ pub fn as_str(self) -> &'static str {
145
+ match self {
146
+ Self::LocalOnly => "local_only",
147
+ Self::VerifiedNoMint => "verified_no_mint",
148
+ Self::PalwMint => "palw_mint",
149
+ }
150
+ }
151
+ pub fn from_str(s: &str) -> Option<Self> {
152
+ Some(match s {
153
+ "local_only" => Self::LocalOnly,
154
+ "verified_no_mint" => Self::VerifiedNoMint,
155
+ "palw_mint" => Self::PalwMint,
156
+ _ => return None,
157
+ })
158
+ }
159
+ }
palw-gateway/src/http.rs ADDED
@@ -0,0 +1,1682 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! The resident HTTP surface. Route families:
2
+ //!
3
+ //! * `/healthz`, `/v1/models` — supervisor probes (the existing LM Studio plugin's
4
+ //! `ensureServer()` polls `/healthz`, so the plugin points at this gateway unchanged).
5
+ //! * `/v1/chat/completions` — OpenAI-compatible, SSE streaming. STATELESS compatibility path:
6
+ //! the client owns the transcript (LM Studio's model). Kept so today's plugin works on day
7
+ //! one, but it cannot benefit from id-stable history.
8
+ //! * `/v1/sessions/*` — the NATIVE session API (the actual product surface): the gateway owns
9
+ //! turns/branches/heads, prompts are compiled from stored ids, and `GenerationEventV1` is the
10
+ //! stream format. This is the path where the prefix cache actually hits.
11
+ //! * `/v1/projects`, `/v1/memory`, `/v1/artifacts` — the Phase-1 workspace layers (§10/§12/§8):
12
+ //! project instructions and memory feed the compiler's system region; artifacts are
13
+ //! content-addressed blobs whose metadata (and version chains, §9) live in the session store.
14
+ //! * `/v1/tools/*` — the §6 tool runtime. Deterministic/read-only tools execute immediately and
15
+ //! are snapshotted; side-effect tools are stored as PROPOSALS and only run through the
16
+ //! approve → exactly-once-claim path.
17
+ //! * `/v1/turns/{id}/verification` — the Phase-2 settlement-clock writer (background supervisor
18
+ //! / tests): legal-transition enforcement, mismatch cascade, certified-head recompute all live
19
+ //! in the store.
20
+ //! * `/v1/scheduler/status` — §14 observability.
21
+ //!
22
+ //! Concurrency model: tiny_http hands each request to a worker thread; engine access serializes
23
+ //! through the §14 ResourceScheduler (priority queue) and then the EngineHost mutex (ownership).
24
+ //! Cancel is a per-job AtomicBool flipped by `POST /v1/jobs/{id}/cancel`.
25
+
26
+ use std::collections::HashMap;
27
+ use std::io::Write;
28
+ use std::sync::atomic::{AtomicBool, Ordering};
29
+ use std::sync::{Arc, Mutex};
30
+
31
+ use serde_json::{Value, json};
32
+ use tiny_http::{Header, Method, Request, Response, Server};
33
+
34
+ use crate::artifact::{ArtifactBlobStore, PARSER_POLICY_UTF8_TEXT_V1};
35
+ use crate::context::{
36
+ AttachmentForContext, ContextCompiler, ContextInputs, MemoryForContext, SearchForContext, ToolResultForContext,
37
+ };
38
+ use crate::coordinator::{JobSubmission, LoopbackCoordinator, PalwCoordinator};
39
+ use crate::engine::EngineHost;
40
+ use crate::events::{CanonicalStopReason, GenerationEventV1, TurnPrivacyMode, TurnVerificationStatus};
41
+ use crate::lmstudio::{self, ResolutionKind};
42
+ use crate::scheduler::{JobClass, ResourceScheduler};
43
+ use crate::store::{ArtifactMeta, NewTurn, SessionStore, ToolCallRow};
44
+ use crate::tokenizer::{IncrementalDecoder, TokenizerHost};
45
+ use crate::tools::{TOOLS, ToolExecutionMode, ToolRuntime, mode_of};
46
+ use crate::worker::WorkerStatus;
47
+
48
+ pub struct Gateway {
49
+ pub engine: Arc<Mutex<EngineHost>>,
50
+ pub store: Arc<Mutex<SessionStore>>,
51
+ pub tokenizer: TokenizerHost,
52
+ pub compiler: ContextCompiler,
53
+ pub blobs: ArtifactBlobStore,
54
+ pub tools: ToolRuntime,
55
+ pub scheduler: Arc<ResourceScheduler>,
56
+ pub cancels: Mutex<HashMap<String, Arc<AtomicBool>>>,
57
+ pub model_name: String,
58
+ pub auth_token: String,
59
+ /// When set, every turn that produced an engine receipt also lands on disk here as
60
+ /// `.receipt.json` + `.opening.json` sidecars (0600, dir 0700) — the file form the offline
61
+ /// verifiers consume, same convention as the legacy server's `lmstudio-receipts/`.
62
+ pub receipt_dir: Option<std::path::PathBuf>,
63
+ /// Search sidecar program (stdin/stdout JSON, `docs/evidence/qi35_search_sidecar.py`):
64
+ /// the whole legacy search stack — routing, SSRF-guarded page fetch, canonical
65
+ /// `live_search_bundle_v1` — reused as a subprocess. Absent ⇒ turns never search.
66
+ pub search_cmd: Option<std::path::PathBuf>,
67
+ /// Present when this instance SERVES the loopback coordinator protocol
68
+ /// (`/v1/palw/loopback/*`) — the dev-harness stand-in for the node-side surface.
69
+ pub palw_loopback: Option<Arc<LoopbackCoordinator>>,
70
+ /// Present when the Phase-2 background worker is running (its live counters).
71
+ pub palw_status: Option<Arc<WorkerStatus>>,
72
+ pub palw_provider: Option<String>,
73
+ }
74
+
75
+ fn now_ms() -> i64 {
76
+ std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as i64).unwrap_or(0)
77
+ }
78
+
79
+ fn fresh_id(prefix: &str) -> String {
80
+ // Uniqueness without a rand dependency: time + a process counter, hashed.
81
+ use blake2::{Blake2b, Digest, digest::consts::U16};
82
+ static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
83
+ let n = COUNTER.fetch_add(1, Ordering::Relaxed);
84
+ let mut h = Blake2b::<U16>::new();
85
+ h.update(now_ms().to_le_bytes());
86
+ h.update(n.to_le_bytes());
87
+ h.update(std::process::id().to_le_bytes());
88
+ format!("{prefix}_{}", hex16(&h.finalize()))
89
+ }
90
+
91
+ fn hex16(b: &[u8]) -> String {
92
+ b.iter().map(|x| format!("{x:02x}")).collect()
93
+ }
94
+
95
+ fn json_response(code: u16, body: &Value) -> Response<std::io::Cursor<Vec<u8>>> {
96
+ let bytes = serde_json::to_vec(body).unwrap_or_default();
97
+ Response::from_data(bytes)
98
+ .with_status_code(code)
99
+ .with_header(Header::from_bytes("Content-Type", "application/json").unwrap())
100
+ }
101
+
102
+ fn err_response(code: u16, message: &str) -> Response<std::io::Cursor<Vec<u8>>> {
103
+ json_response(code, &json!({ "error": { "message": message } }))
104
+ }
105
+
106
+ /// Minimal query-string access with %XX and '+' decoding — enough for names and ids.
107
+ fn query_param(url: &str, key: &str) -> Option<String> {
108
+ let query = url.split_once('?')?.1;
109
+ for pair in query.split('&') {
110
+ let (k, v) = pair.split_once('=').unwrap_or((pair, ""));
111
+ if k == key {
112
+ let mut out = Vec::new();
113
+ let bytes = v.as_bytes();
114
+ let mut i = 0;
115
+ while i < bytes.len() {
116
+ match bytes[i] {
117
+ b'+' => {
118
+ out.push(b' ');
119
+ i += 1;
120
+ }
121
+ b'%' if i + 2 < bytes.len() + 1 && i + 2 <= bytes.len() - 1 + 1 => {
122
+ let hex = bytes.get(i + 1..i + 3).and_then(|h| std::str::from_utf8(h).ok());
123
+ match hex.and_then(|h| u8::from_str_radix(h, 16).ok()) {
124
+ Some(b) => {
125
+ out.push(b);
126
+ i += 3;
127
+ }
128
+ None => {
129
+ out.push(b'%');
130
+ i += 1;
131
+ }
132
+ }
133
+ }
134
+ b => {
135
+ out.push(b);
136
+ i += 1;
137
+ }
138
+ }
139
+ }
140
+ return String::from_utf8(out).ok();
141
+ }
142
+ }
143
+ None
144
+ }
145
+
146
+ /// Extract `{id}` from `prefix{id}suffix`.
147
+ fn path_seg<'a>(path: &'a str, prefix: &str, suffix: &str) -> Option<&'a str> {
148
+ let rest = path.strip_prefix(prefix)?;
149
+ let id = rest.strip_suffix(suffix)?;
150
+ if id.is_empty() || id.contains('/') { None } else { Some(id) }
151
+ }
152
+
153
+ fn artifact_meta_json(meta: &ArtifactMeta) -> Value {
154
+ json!({
155
+ "artifact_id": meta.artifact_id,
156
+ "content_hash": meta.content_hash,
157
+ "mime_type": meta.mime_type,
158
+ "byte_length": meta.byte_length,
159
+ "display_name": meta.display_name,
160
+ "parent_artifact_id": meta.parent_artifact_id,
161
+ "version": meta.version,
162
+ "parser_policy": meta.parser_policy,
163
+ "project_id": meta.project_id,
164
+ "created_by_turn": meta.created_by_turn,
165
+ "created_ms": meta.created_ms,
166
+ })
167
+ }
168
+
169
+ /// One turn's search outcome, carried from the sidecar into the native turn path.
170
+ struct TurnSearch {
171
+ /// Normalized query (`/search ` prefix stripped) — becomes the turn's user text, exactly
172
+ /// like the legacy server.
173
+ query: String,
174
+ context: SearchForContext,
175
+ /// The full `live_search_bundle_v1` JSON, persisted on the turn and exported as the
176
+ /// `.search.json` sidecar.
177
+ bundle_json: String,
178
+ summary_lines: Vec<String>,
179
+ }
180
+
181
+ fn tool_call_json(row: &ToolCallRow) -> Value {
182
+ json!({
183
+ "call_id": row.call_id,
184
+ "tool_name": row.tool_name,
185
+ "mode": row.mode,
186
+ "arguments": serde_json::from_str::<Value>(&row.arguments_json).unwrap_or(Value::Null),
187
+ "status": row.status,
188
+ "result_artifact_id": row.result_artifact_id,
189
+ "result_preview": row.result_preview,
190
+ "error": row.error,
191
+ "conversation_id": row.conversation_id,
192
+ "created_ms": row.created_ms,
193
+ "executed_ms": row.executed_ms,
194
+ })
195
+ }
196
+
197
+ impl Gateway {
198
+ pub fn serve(self: Arc<Self>, addr: &str) -> Result<(), String> {
199
+ let server = Server::http(addr).map_err(|e| format!("bind {addr}: {e}"))?;
200
+ eprintln!("[palw-gateway] listening on http://{addr}");
201
+ loop {
202
+ let request = match server.recv() {
203
+ Ok(r) => r,
204
+ Err(e) => {
205
+ eprintln!("[palw-gateway] accept error: {e}");
206
+ continue;
207
+ }
208
+ };
209
+ let gw = Arc::clone(&self);
210
+ std::thread::spawn(move || gw.dispatch(request));
211
+ }
212
+ }
213
+
214
+ fn authorized(&self, request: &Request) -> bool {
215
+ if self.auth_token.is_empty() {
216
+ return true;
217
+ }
218
+ request
219
+ .headers()
220
+ .iter()
221
+ .find(|h| h.field.equiv("Authorization"))
222
+ .map(|h| h.value.as_str() == format!("Bearer {}", self.auth_token))
223
+ .unwrap_or(false)
224
+ }
225
+
226
+ fn dispatch(self: Arc<Self>, mut request: Request) {
227
+ let method = request.method().clone();
228
+ let url = request.url().to_string();
229
+ let path = url.split('?').next().unwrap_or("").to_string();
230
+
231
+ if path == "/healthz" {
232
+ let ok = self.engine.lock().map(|mut e| e.healthy()).unwrap_or(false);
233
+ let _ = request.respond(json_response(200, &json!({ "ok": true, "engine_resident": ok })));
234
+ return;
235
+ }
236
+ if !self.authorized(&request) {
237
+ let _ = request.respond(err_response(401, "missing or wrong bearer token"));
238
+ return;
239
+ }
240
+
241
+ // Bodies are BYTES (artifact upload); JSON handlers decode as needed.
242
+ let mut body = Vec::new();
243
+ if matches!(method, Method::Post) && request.as_reader().read_to_end(&mut body).is_err() {
244
+ let _ = request.respond(err_response(400, "unreadable body"));
245
+ return;
246
+ }
247
+ let body_json = || -> Result<Value, String> {
248
+ let text = std::str::from_utf8(&body).map_err(|_| "body is not UTF-8".to_string())?;
249
+ if text.trim().is_empty() {
250
+ return Ok(json!({}));
251
+ }
252
+ serde_json::from_str(text).map_err(|e| format!("bad json: {e}"))
253
+ };
254
+
255
+ let result: Result<(), String> = match (&method, path.as_str()) {
256
+ (Method::Get, "/v1/models") => {
257
+ let _ = request.respond(json_response(
258
+ 200,
259
+ &json!({ "object": "list", "data": [{ "id": self.model_name, "object": "model" }] }),
260
+ ));
261
+ Ok(())
262
+ }
263
+ (Method::Get, "/v1/scheduler/status") => {
264
+ let s = self.scheduler.status();
265
+ let mut waiting = serde_json::Map::new();
266
+ for class in JobClass::ALL {
267
+ waiting.insert(class.as_str().into(), json!(s.waiting_by_class[class.index()]));
268
+ }
269
+ let _ = request.respond(json_response(
270
+ 200,
271
+ &json!({
272
+ "busy": s.busy,
273
+ "waiting": waiting,
274
+ "ema_millitok_per_sec": s.ema_millitok_per_sec,
275
+ "completed_jobs": s.completed_jobs,
276
+ }),
277
+ ));
278
+ Ok(())
279
+ }
280
+ // §14 SLA admission probe — the Phase-2 background worker asks BEFORE accepting a
281
+ // replica/audit assignment; refusal here means "decline the assignment", which the
282
+ // doc prefers over accepting and no-showing.
283
+ (Method::Post, "/v1/scheduler/admission") => self.finish_json(request, body_json().and_then(|v| {
284
+ let class = v
285
+ .get("class")
286
+ .and_then(|c| c.as_str())
287
+ .and_then(JobClass::from_str)
288
+ .ok_or_else(|| "missing or unknown class".to_string())?;
289
+ let tokens = v.get("estimated_output_tokens").and_then(|t| t.as_u64()).unwrap_or(0);
290
+ let deadline = v.get("deadline_ms").and_then(|d| d.as_u64()).ok_or_else(|| "missing deadline_ms".to_string())?;
291
+ Ok(match self.scheduler.try_admit(class, tokens, deadline) {
292
+ Ok(()) => json!({ "admitted": true }),
293
+ Err(reason) => json!({ "admitted": false, "reason": reason }),
294
+ })
295
+ })),
296
+ (Method::Post, "/v1/chat/completions") => match body_json() {
297
+ Ok(v) => self.chat_completions(request, &v),
298
+ Err(e) => {
299
+ let _ = request.respond(err_response(400, &e));
300
+ Ok(())
301
+ }
302
+ },
303
+ // The LM Studio bridge: a client-owned transcript resolved onto the session store,
304
+ // then the NATIVE turn path (id-stable prompt, receipt, ROOTS, settlement).
305
+ (Method::Post, "/v1/lmstudio/turn") => match body_json() {
306
+ Ok(v) => self.lmstudio_turn(request, &v),
307
+ Err(e) => {
308
+ let _ = request.respond(err_response(400, &e));
309
+ Ok(())
310
+ }
311
+ },
312
+ (Method::Get, p) if path_seg(p, "/v1/turns/", "/receipt").is_some() => {
313
+ let turn = path_seg(p, "/v1/turns/", "/receipt").unwrap().to_string();
314
+ self.turn_receipt(request, &turn)
315
+ }
316
+ (Method::Get, p)
317
+ if p.starts_with("/v1/turns/") && !p.trim_start_matches("/v1/turns/").contains('/') =>
318
+ {
319
+ let turn = p.trim_start_matches("/v1/turns/").to_string();
320
+ self.turn_info(request, &turn)
321
+ }
322
+
323
+ // ---- sessions ----------------------------------------------------------------
324
+ (Method::Post, "/v1/sessions") => self.finish_json(request, body_json().and_then(|v| {
325
+ let title = v.get("title").and_then(|t| t.as_str()).unwrap_or_default().to_string();
326
+ let project_id = v.get("project_id").and_then(|p| p.as_str()).map(String::from);
327
+ let id = fresh_id("conv");
328
+ let store = self.store.lock().unwrap();
329
+ if let Some(p) = &project_id {
330
+ store.get_project(p)?; // referenced project must exist
331
+ }
332
+ store.create_conversation(&id, &title, project_id.as_deref(), now_ms())?;
333
+ Ok(json!({ "conversation_id": id, "title": title, "project_id": project_id }))
334
+ })),
335
+ (Method::Get, "/v1/sessions") => self.finish_json(request, (|| {
336
+ let rows = self.store.lock().unwrap().list_conversations()?;
337
+ let list: Vec<Value> = rows
338
+ .iter()
339
+ .map(|c| {
340
+ json!({
341
+ "conversation_id": c.conversation_id,
342
+ "title": c.title,
343
+ "project_id": c.project_id,
344
+ "local_head": c.heads.local_head,
345
+ "certified_head": c.heads.certified_head,
346
+ "updated_ms": c.updated_ms,
347
+ })
348
+ })
349
+ .collect();
350
+ Ok(json!({ "sessions": list }))
351
+ })()),
352
+ (Method::Post, p) if path_seg(p, "/v1/sessions/", "/messages").is_some() => {
353
+ let conv = path_seg(p, "/v1/sessions/", "/messages").unwrap().to_string();
354
+ match body_json() {
355
+ Ok(v) => self.session_message(request, &conv, &v, Vec::new(), None),
356
+ Err(e) => {
357
+ let _ = request.respond(err_response(400, &e));
358
+ Ok(())
359
+ }
360
+ }
361
+ }
362
+ (Method::Get, p) if path_seg(p, "/v1/sessions/", "/history").is_some() => {
363
+ let conv = path_seg(p, "/v1/sessions/", "/history").unwrap().to_string();
364
+ self.session_history(request, &conv)
365
+ }
366
+ (Method::Post, p) if path_seg(p, "/v1/sessions/", "/head").is_some() => {
367
+ // Branch switching: point local_head at any existing turn (the UI's "go back to
368
+ // this version" — history is immutable, only the head moves).
369
+ let conv = path_seg(p, "/v1/sessions/", "/head").unwrap().to_string();
370
+ self.finish_json(request, body_json().and_then(|v| {
371
+ let turn_id = v
372
+ .get("turn_id")
373
+ .and_then(|t| t.as_str())
374
+ .ok_or_else(|| "missing turn_id".to_string())?;
375
+ self.store.lock().unwrap().set_local_head(&conv, turn_id, now_ms())?;
376
+ Ok(json!({ "local_head": turn_id }))
377
+ }))
378
+ }
379
+ (Method::Post, p) if path_seg(p, "/v1/jobs/", "/cancel").is_some() => {
380
+ let job = path_seg(p, "/v1/jobs/", "/cancel").unwrap();
381
+ if let Some(flag) = self.cancels.lock().unwrap().get(job) {
382
+ flag.store(true, Ordering::Relaxed);
383
+ let _ = request.respond(json_response(200, &json!({ "cancelled": true })));
384
+ } else {
385
+ let _ = request.respond(err_response(404, "unknown or finished job"));
386
+ }
387
+ Ok(())
388
+ }
389
+
390
+ // ---- PALW worker + loopback coordinator (§15) --------------------------------
391
+ (Method::Get, "/v1/palw/status") => {
392
+ let worker = self.palw_status.as_ref().map(|s| {
393
+ json!({
394
+ "submitted": s.submitted.load(Ordering::Relaxed),
395
+ "verdicts_applied": s.verdicts_applied.load(Ordering::Relaxed),
396
+ "replicas_executed": s.replicas_executed.load(Ordering::Relaxed),
397
+ "replicas_declined": s.replicas_declined.load(Ordering::Relaxed),
398
+ "errors": s.errors.load(Ordering::Relaxed),
399
+ "last_cycle_unix_ms": s.last_cycle_unix_ms.load(Ordering::Relaxed),
400
+ "last_error": *s.last_error.lock().unwrap(),
401
+ })
402
+ });
403
+ let loopback = self.palw_loopback.as_ref().map(|l| json!(l.counts()));
404
+ let _ = request.respond(json_response(
405
+ 200,
406
+ &json!({
407
+ "worker_enabled": self.palw_status.is_some(),
408
+ "provider_id": self.palw_provider,
409
+ "worker": worker,
410
+ "loopback": loopback,
411
+ }),
412
+ ));
413
+ Ok(())
414
+ }
415
+ (Method::Post, "/v1/palw/loopback/jobs") => self.finish_json(request, body_json().and_then(|v| {
416
+ let loopback = self.loopback()?;
417
+ let submission: JobSubmission =
418
+ serde_json::from_value(v).map_err(|e| format!("bad submission: {e}"))?;
419
+ loopback.submit_job(&submission)?;
420
+ Ok(json!({ "accepted": true }))
421
+ })),
422
+ (Method::Post, "/v1/palw/loopback/verdicts") => self.finish_json(request, body_json().and_then(|v| {
423
+ let loopback = self.loopback()?;
424
+ let job_ids: Vec<String> = v
425
+ .get("job_ids")
426
+ .and_then(|j| j.as_array())
427
+ .map(|a| a.iter().filter_map(|x| x.as_str().map(String::from)).collect())
428
+ .unwrap_or_default();
429
+ let verdicts = loopback.fetch_verdicts(&job_ids)?;
430
+ let list: Vec<Value> =
431
+ verdicts.iter().map(|(id, v)| json!({ "job_id": id, "verdict": v.as_str() })).collect();
432
+ Ok(json!({ "verdicts": list }))
433
+ })),
434
+ (Method::Get, "/v1/palw/loopback/assignments") => self.finish_json(request, (|| {
435
+ let loopback = self.loopback()?;
436
+ let provider = query_param(&url, "provider_id").ok_or_else(|| "missing provider_id".to_string())?;
437
+ let assignments = loopback.fetch_assignments(&provider)?;
438
+ Ok(json!({ "assignments": assignments }))
439
+ })()),
440
+ (Method::Post, p) if path_seg(p, "/v1/palw/loopback/assignments/", "/decline").is_some() => {
441
+ let job = path_seg(p, "/v1/palw/loopback/assignments/", "/decline").unwrap().to_string();
442
+ self.finish_json(request, body_json().and_then(|v| {
443
+ let loopback = self.loopback()?;
444
+ let provider =
445
+ v.get("provider_id").and_then(|p| p.as_str()).ok_or_else(|| "missing provider_id".to_string())?;
446
+ let reason = v.get("reason").and_then(|r| r.as_str()).unwrap_or("");
447
+ loopback.decline_assignment(&job, provider, reason)?;
448
+ Ok(json!({ "declined": true }))
449
+ }))
450
+ }
451
+ (Method::Post, "/v1/palw/loopback/replica-results") => self.finish_json(request, body_json().and_then(|v| {
452
+ let loopback = self.loopback()?;
453
+ let result: crate::coordinator::ReplicaResultV1 =
454
+ serde_json::from_value(v).map_err(|e| format!("bad replica result: {e}"))?;
455
+ loopback.submit_replica_result(&result)?;
456
+ Ok(json!({ "recorded": true }))
457
+ })),
458
+
459
+ // ---- settlement clock (Phase-2 writer) ---------------------------------------
460
+ (Method::Post, p) if path_seg(p, "/v1/turns/", "/verification").is_some() => {
461
+ let turn = path_seg(p, "/v1/turns/", "/verification").unwrap().to_string();
462
+ self.finish_json(request, body_json().and_then(|v| {
463
+ let status = v
464
+ .get("status")
465
+ .and_then(|s| s.as_str())
466
+ .and_then(TurnVerificationStatus::from_str)
467
+ .ok_or_else(|| "missing or unknown status".to_string())?;
468
+ let outcome = self.store.lock().unwrap().set_verification_status(&turn, status, now_ms())?;
469
+ Ok(json!({
470
+ "turn_id": turn,
471
+ "status": outcome.status.as_str(),
472
+ "cascaded_mint_ineligible": outcome.cascaded,
473
+ "certified_head": outcome.certified_head,
474
+ }))
475
+ }))
476
+ }
477
+
478
+ // ---- projects (§10) ----------------------------------------------------------
479
+ (Method::Post, "/v1/projects") => self.finish_json(request, body_json().and_then(|v| {
480
+ let name = v.get("name").and_then(|n| n.as_str()).ok_or_else(|| "missing name".to_string())?;
481
+ let instructions = v.get("instructions").and_then(|i| i.as_str()).unwrap_or_default();
482
+ let id = fresh_id("proj");
483
+ self.store.lock().unwrap().create_project(&id, name, instructions, now_ms())?;
484
+ Ok(json!({ "project_id": id, "name": name }))
485
+ })),
486
+ (Method::Get, "/v1/projects") => self.finish_json(request, (|| {
487
+ let rows = self.store.lock().unwrap().list_projects()?;
488
+ let list: Vec<Value> = rows
489
+ .iter()
490
+ .map(|p| {
491
+ json!({
492
+ "project_id": p.project_id,
493
+ "name": p.name,
494
+ "instructions": p.instructions,
495
+ "updated_ms": p.updated_ms,
496
+ })
497
+ })
498
+ .collect();
499
+ Ok(json!({ "projects": list }))
500
+ })()),
501
+ (Method::Post, p) if path_seg(p, "/v1/projects/", "/instructions").is_some() => {
502
+ let id = path_seg(p, "/v1/projects/", "/instructions").unwrap().to_string();
503
+ self.finish_json(request, body_json().and_then(|v| {
504
+ let instructions = v
505
+ .get("instructions")
506
+ .and_then(|i| i.as_str())
507
+ .ok_or_else(|| "missing instructions".to_string())?;
508
+ self.store.lock().unwrap().set_project_instructions(&id, instructions, now_ms())?;
509
+ Ok(json!({ "project_id": id }))
510
+ }))
511
+ }
512
+
513
+ // ---- long-term memory (§12) --------------------------------------------------
514
+ (Method::Post, "/v1/memory") => self.finish_json(request, body_json().and_then(|v| {
515
+ let content =
516
+ v.get("content").and_then(|c| c.as_str()).ok_or_else(|| "missing content".to_string())?;
517
+ // Namespace = "global" or a project id (validated).
518
+ let namespace = match v.get("project_id").and_then(|p| p.as_str()) {
519
+ Some(p) => {
520
+ self.store.lock().unwrap().get_project(p)?;
521
+ p.to_string()
522
+ }
523
+ None => "global".to_string(),
524
+ };
525
+ let source_turn = v.get("source_turn_id").and_then(|t| t.as_str());
526
+ let id = fresh_id("mem");
527
+ let seq = self.store.lock().unwrap().add_memory(&id, &namespace, content, source_turn, now_ms())?;
528
+ Ok(json!({ "memory_id": id, "namespace": namespace, "created_sequence": seq }))
529
+ })),
530
+ (Method::Get, "/v1/memory") => self.finish_json(request, (|| {
531
+ let project = query_param(&url, "project_id");
532
+ let mut namespaces: Vec<&str> = vec!["global"];
533
+ if let Some(p) = &project {
534
+ namespaces.push(p.as_str());
535
+ }
536
+ let rows = self.store.lock().unwrap().active_memory(&namespaces)?;
537
+ let list: Vec<Value> = rows
538
+ .iter()
539
+ .map(|m| {
540
+ json!({
541
+ "memory_id": m.memory_id,
542
+ "namespace": m.namespace,
543
+ "content": m.content,
544
+ "source_turn_id": m.source_turn_id,
545
+ "created_sequence": m.created_sequence,
546
+ "created_ms": m.created_ms,
547
+ })
548
+ })
549
+ .collect();
550
+ Ok(json!({ "memory": list }))
551
+ })()),
552
+ (Method::Post, p) if path_seg(p, "/v1/memory/", "/delete").is_some() => {
553
+ let id = path_seg(p, "/v1/memory/", "/delete").unwrap().to_string();
554
+ self.finish_json(request, (|| {
555
+ self.store.lock().unwrap().delete_memory(&id)?;
556
+ Ok(json!({ "memory_id": id, "deleted": true }))
557
+ })())
558
+ }
559
+
560
+ // ---- artifacts (§8/§9) -------------------------------------------------------
561
+ (Method::Post, "/v1/artifacts") => {
562
+ let outcome = (|| -> Result<Value, String> {
563
+ let name = query_param(&url, "name").unwrap_or_default();
564
+ let project_id = query_param(&url, "project_id");
565
+ let parent = query_param(&url, "parent_artifact_id");
566
+ let created_by_turn = query_param(&url, "created_by_turn");
567
+ let mime = request
568
+ .headers()
569
+ .iter()
570
+ .find(|h| h.field.equiv("Content-Type"))
571
+ .map(|h| h.value.as_str().to_string())
572
+ .unwrap_or_else(|| "application/octet-stream".into());
573
+ if body.is_empty() {
574
+ return Err("empty artifact body".into());
575
+ }
576
+ let content_hash = self.blobs.put(&body)?;
577
+ let parser_policy = if std::str::from_utf8(&body).is_ok() {
578
+ PARSER_POLICY_UTF8_TEXT_V1
579
+ } else {
580
+ "" // stored, but no parser can place it in context yet
581
+ };
582
+ let store = self.store.lock().unwrap();
583
+ if let Some(p) = &project_id {
584
+ store.get_project(p)?;
585
+ }
586
+ let meta = ArtifactMeta {
587
+ artifact_id: fresh_id("art"),
588
+ content_hash: content_hash.clone(),
589
+ mime_type: mime,
590
+ byte_length: body.len() as i64,
591
+ display_name: name,
592
+ parent_artifact_id: parent,
593
+ version: 0, // derived by the store
594
+ parser_policy: parser_policy.into(),
595
+ project_id,
596
+ created_by_turn,
597
+ created_ms: now_ms(),
598
+ };
599
+ let version = store.insert_artifact(&meta)?;
600
+ Ok(json!({
601
+ "artifact_id": meta.artifact_id,
602
+ "content_hash": content_hash,
603
+ "byte_length": meta.byte_length,
604
+ "version": version,
605
+ "parser_policy": meta.parser_policy,
606
+ }))
607
+ })();
608
+ self.finish_json(request, outcome)
609
+ }
610
+ (Method::Get, "/v1/artifacts") => self.finish_json(request, (|| {
611
+ let project = query_param(&url, "project_id");
612
+ let rows = self.store.lock().unwrap().list_artifacts(project.as_deref())?;
613
+ Ok(json!({ "artifacts": rows.iter().map(artifact_meta_json).collect::<Vec<_>>() }))
614
+ })()),
615
+ (Method::Get, p) if path_seg(p, "/v1/artifacts/", "/meta").is_some() => {
616
+ let id = path_seg(p, "/v1/artifacts/", "/meta").unwrap().to_string();
617
+ self.finish_json(request, (|| {
618
+ let store = self.store.lock().unwrap();
619
+ let meta = store.get_artifact(&id)?;
620
+ let lineage = store.artifact_lineage(&id)?;
621
+ let children = store.artifact_children(&id)?;
622
+ Ok(json!({
623
+ "artifact": artifact_meta_json(&meta),
624
+ "lineage": lineage.iter().map(artifact_meta_json).collect::<Vec<_>>(),
625
+ "children": children.iter().map(artifact_meta_json).collect::<Vec<_>>(),
626
+ }))
627
+ })())
628
+ }
629
+ (Method::Get, p) if p.starts_with("/v1/artifacts/") && !p.trim_start_matches("/v1/artifacts/").contains('/') => {
630
+ let id = p.trim_start_matches("/v1/artifacts/").to_string();
631
+ let outcome = (|| -> Result<(Vec<u8>, String), String> {
632
+ let meta = self.store.lock().unwrap().get_artifact(&id)?;
633
+ let bytes = self.blobs.get_verified(&meta.content_hash)?;
634
+ Ok((bytes, meta.mime_type))
635
+ })();
636
+ match outcome {
637
+ Ok((bytes, mime)) => {
638
+ let response = Response::from_data(bytes).with_header(
639
+ Header::from_bytes("Content-Type", mime.as_bytes())
640
+ .unwrap_or_else(|_| Header::from_bytes("Content-Type", "application/octet-stream").unwrap()),
641
+ );
642
+ let _ = request.respond(response);
643
+ }
644
+ Err(e) => {
645
+ let _ = request.respond(err_response(404, &e));
646
+ }
647
+ }
648
+ Ok(())
649
+ }
650
+
651
+ // ---- tools (§6) --------------------------------------------------------------
652
+ (Method::Get, "/v1/tools") => {
653
+ let list: Vec<Value> = TOOLS
654
+ .iter()
655
+ .map(|(name, mode, help)| {
656
+ json!({
657
+ "tool": name,
658
+ "mode": mode.as_str(),
659
+ "requires_approval": *mode == ToolExecutionMode::ExactlyOnceSideEffect,
660
+ "description": help,
661
+ })
662
+ })
663
+ .collect();
664
+ let _ = request.respond(json_response(200, &json!({ "tools": list })));
665
+ Ok(())
666
+ }
667
+ (Method::Post, "/v1/tools/execute") => self.finish_json(request, body_json().and_then(|v| self.tool_execute(&v))),
668
+ (Method::Post, p) if path_seg(p, "/v1/tools/calls/", "/approve").is_some() => {
669
+ let id = path_seg(p, "/v1/tools/calls/", "/approve").unwrap().to_string();
670
+ self.finish_json(request, self.tool_approve(&id))
671
+ }
672
+ (Method::Post, p) if path_seg(p, "/v1/tools/calls/", "/deny").is_some() => {
673
+ let id = path_seg(p, "/v1/tools/calls/", "/deny").unwrap().to_string();
674
+ self.finish_json(request, (|| {
675
+ let store = self.store.lock().unwrap();
676
+ store.deny_tool_call(&id)?;
677
+ Ok(tool_call_json(&store.get_tool_call(&id)?))
678
+ })())
679
+ }
680
+ (Method::Get, p) if p.starts_with("/v1/tools/calls/") && !p.trim_start_matches("/v1/tools/calls/").contains('/') => {
681
+ let id = p.trim_start_matches("/v1/tools/calls/").to_string();
682
+ self.finish_json(request, self.store.lock().unwrap().get_tool_call(&id).map(|r| tool_call_json(&r)))
683
+ }
684
+
685
+ _ => {
686
+ let _ = request.respond(err_response(404, "no such route"));
687
+ Ok(())
688
+ }
689
+ };
690
+ if let Err(e) = result {
691
+ eprintln!("[palw-gateway] {method} {path}: {e}");
692
+ }
693
+ }
694
+
695
+ fn loopback(&self) -> Result<&Arc<LoopbackCoordinator>, String> {
696
+ self.palw_loopback
697
+ .as_ref()
698
+ .ok_or_else(|| "loopback coordinator not enabled on this instance (--palw-loopback)".to_string())
699
+ }
700
+
701
+ /// Respond 200 with the value, or 400 with the error — the uniform JSON endpoint shape.
702
+ fn finish_json(&self, request: Request, outcome: Result<Value, String>) -> Result<(), String> {
703
+ match outcome {
704
+ Ok(v) => {
705
+ let _ = request.respond(json_response(200, &v));
706
+ }
707
+ Err(e) => {
708
+ let code = if e.contains("not found") || e.contains("no such") { 404 } else { 400 };
709
+ let _ = request.respond(err_response(code, &e));
710
+ }
711
+ }
712
+ Ok(())
713
+ }
714
+
715
+ // ---- tool runtime glue --------------------------------------------------------------
716
+
717
+ /// POST /v1/tools/execute — deterministic/read-only tools run NOW (and are snapshotted);
718
+ /// side-effect tools are stored as proposals awaiting `/approve`.
719
+ fn tool_execute(&self, v: &Value) -> Result<Value, String> {
720
+ let tool = v.get("tool").and_then(|t| t.as_str()).ok_or_else(|| "missing tool".to_string())?;
721
+ let mode = mode_of(tool).ok_or_else(|| format!("unknown tool {tool:?}"))?;
722
+ let arguments = v.get("arguments").cloned().unwrap_or_else(|| json!({}));
723
+ let conversation_id = v.get("conversation_id").and_then(|c| c.as_str()).map(String::from);
724
+ let call_id = fresh_id("call");
725
+ let now = now_ms();
726
+
727
+ let initial_status = match mode {
728
+ ToolExecutionMode::ExactlyOnceSideEffect => "proposed",
729
+ _ => "executing",
730
+ };
731
+ {
732
+ let store = self.store.lock().unwrap();
733
+ store.insert_tool_call(&ToolCallRow {
734
+ call_id: call_id.clone(),
735
+ tool_name: tool.into(),
736
+ mode: mode.as_str().into(),
737
+ arguments_json: serde_json::to_string(&arguments).map_err(|e| e.to_string())?,
738
+ status: initial_status.into(),
739
+ result_artifact_id: None,
740
+ result_preview: String::new(),
741
+ error: None,
742
+ conversation_id,
743
+ created_ms: now,
744
+ executed_ms: None,
745
+ })?;
746
+ }
747
+ if mode == ToolExecutionMode::ExactlyOnceSideEffect {
748
+ let row = self.store.lock().unwrap().get_tool_call(&call_id)?;
749
+ return Ok(tool_call_json(&row));
750
+ }
751
+ self.run_claimed_tool(&call_id, tool, &arguments)
752
+ }
753
+
754
+ /// POST /v1/tools/calls/{id}/approve — the user's confirmation; claims the exactly-once
755
+ /// ticket and executes.
756
+ fn tool_approve(&self, call_id: &str) -> Result<Value, String> {
757
+ let (tool, arguments) = {
758
+ let store = self.store.lock().unwrap();
759
+ let row = store.get_tool_call(call_id)?;
760
+ store.approve_tool_call(call_id)?;
761
+ store.claim_tool_execution(call_id)?;
762
+ let args: Value = serde_json::from_str(&row.arguments_json).map_err(|e| e.to_string())?;
763
+ (row.tool_name, args)
764
+ };
765
+ self.run_claimed_tool(call_id, &tool, &arguments)
766
+ }
767
+
768
+ /// Shared tail: the call row is in `executing`; run the tool, snapshot the result as a
769
+ /// content-addressed artifact, finish the row either way.
770
+ fn run_claimed_tool(&self, call_id: &str, tool: &str, arguments: &Value) -> Result<Value, String> {
771
+ let executed = self.tools.execute(tool, arguments);
772
+ let now = now_ms();
773
+ let store = self.store.lock().unwrap();
774
+ match executed {
775
+ Ok(outcome) => {
776
+ let content_hash = self.blobs.put(&outcome.output)?;
777
+ let artifact_id = fresh_id("art");
778
+ store.insert_artifact(&ArtifactMeta {
779
+ artifact_id: artifact_id.clone(),
780
+ content_hash,
781
+ mime_type: outcome.mime.into(),
782
+ byte_length: outcome.output.len() as i64,
783
+ display_name: format!("tool:{tool}"),
784
+ parent_artifact_id: None,
785
+ version: 0,
786
+ parser_policy: if std::str::from_utf8(&outcome.output).is_ok() {
787
+ PARSER_POLICY_UTF8_TEXT_V1.into()
788
+ } else {
789
+ String::new()
790
+ },
791
+ project_id: None,
792
+ created_by_turn: None,
793
+ created_ms: now,
794
+ })?;
795
+ store.finish_tool_call(call_id, true, Some(&artifact_id), &outcome.preview, None, now)?;
796
+ }
797
+ Err(e) => {
798
+ store.finish_tool_call(call_id, false, None, "", Some(&e), now)?;
799
+ }
800
+ }
801
+ Ok(tool_call_json(&store.get_tool_call(call_id)?))
802
+ }
803
+
804
+ // ---- LM Studio bridge ---------------------------------------------------------------
805
+
806
+ /// Run the search sidecar for one query. Fail-open BY DESIGN: any sidecar-level failure
807
+ /// (spawn error, timeout, bad JSON) logs and returns `None`, and the turn proceeds
808
+ /// searchless — a broken search stack must degrade the answer, never block the chat.
809
+ /// (Search failures *inside* the stack are typed into a failure bundle by the sidecar and
810
+ /// still commit as evidence, exactly like the legacy server.)
811
+ fn run_search_sidecar(&self, query: &str, previous_user_query: Option<&str>) -> Option<TurnSearch> {
812
+ const SIDECAR_TIMEOUT_MS: u64 = 90_000;
813
+ let program = self.search_cmd.as_ref()?;
814
+ let attempt = (|| -> Result<Option<TurnSearch>, String> {
815
+ let mut child = std::process::Command::new(program)
816
+ .stdin(std::process::Stdio::piped())
817
+ .stdout(std::process::Stdio::piped())
818
+ .stderr(std::process::Stdio::piped())
819
+ .spawn()
820
+ .map_err(|e| format!("spawn {}: {e}", program.display()))?;
821
+ {
822
+ use std::io::Write as _;
823
+ let mut stdin = child.stdin.take().ok_or("sidecar stdin unavailable")?;
824
+ let request = json!({ "query": query, "previous_user_query": previous_user_query });
825
+ stdin.write_all(request.to_string().as_bytes()).map_err(|e| e.to_string())?;
826
+ } // drop closes stdin — the sidecar reads to EOF
827
+ let stdout = child.stdout.take().ok_or("sidecar stdout unavailable")?;
828
+ let reader = std::thread::spawn(move || {
829
+ use std::io::Read as _;
830
+ let mut out = String::new();
831
+ let mut handle = stdout;
832
+ let _ = handle.read_to_string(&mut out);
833
+ out
834
+ });
835
+ let deadline = std::time::Instant::now() + std::time::Duration::from_millis(SIDECAR_TIMEOUT_MS);
836
+ loop {
837
+ match child.try_wait().map_err(|e| e.to_string())? {
838
+ Some(status) => {
839
+ let out = reader.join().unwrap_or_default();
840
+ if !status.success() {
841
+ return Err(format!("sidecar exit {status}"));
842
+ }
843
+ let reply: Value = serde_json::from_str(out.trim()).map_err(|e| format!("sidecar json: {e}"))?;
844
+ if let Some(error) = reply.get("error").and_then(|e| e.as_str()) {
845
+ return Err(format!("sidecar error: {error}"));
846
+ }
847
+ if reply.get("searched").and_then(|s| s.as_bool()) != Some(true) {
848
+ return Ok(None);
849
+ }
850
+ let field = |k: &str| -> Result<String, String> {
851
+ reply
852
+ .get(k)
853
+ .and_then(|v| v.as_str())
854
+ .map(String::from)
855
+ .ok_or_else(|| format!("sidecar reply missing {k}"))
856
+ };
857
+ let bundle = reply.get("bundle").ok_or("sidecar reply missing bundle")?;
858
+ let summary_lines = reply
859
+ .get("summary_lines")
860
+ .and_then(|l| l.as_array())
861
+ .map(|a| a.iter().filter_map(|v| v.as_str().map(String::from)).collect())
862
+ .unwrap_or_default();
863
+ return Ok(Some(TurnSearch {
864
+ query: field("query")?,
865
+ context: SearchForContext {
866
+ evidence_text: field("evidence_text")?,
867
+ bundle_sha256: field("bundle_sha256")?,
868
+ effective_compression: field("effective_compression")?,
869
+ },
870
+ bundle_json: serde_json::to_string(bundle).map_err(|e| e.to_string())?,
871
+ summary_lines,
872
+ }));
873
+ }
874
+ None if std::time::Instant::now() >= deadline => {
875
+ let _ = child.kill();
876
+ let _ = child.wait();
877
+ return Err(format!("sidecar timeout after {SIDECAR_TIMEOUT_MS}ms"));
878
+ }
879
+ None => std::thread::sleep(std::time::Duration::from_millis(30)),
880
+ }
881
+ }
882
+ })();
883
+ match attempt {
884
+ Ok(result) => result,
885
+ Err(e) => {
886
+ eprintln!("[palw-gateway] search sidecar: {e} — continuing without search");
887
+ None
888
+ }
889
+ }
890
+ }
891
+
892
+ /// POST /v1/lmstudio/turn — body `{ "messages": [{role, content}…], "privacy_mode"? }`.
893
+ ///
894
+ /// The stateful LM Studio path: the transcript is resolved onto the session store by
895
+ /// content (`lmstudio::resolve` — continue / branch / new / import), then the turn runs
896
+ /// through the SAME native pipeline as `/v1/sessions/{id}/messages`: id-stable compiled
897
+ /// prompt (prefix cache hits), engine receipt + runtime ROOTS persisted on the turn,
898
+ /// settlement pipeline per `privacy_mode`. The stream is `GenerationEventV1` prefixed with
899
+ /// one `lmstudio_resolve` frame that also reports the parent turn's current settlement
900
+ /// status — that is how the plugin shows last turn's verification without polling mid-chat.
901
+ fn lmstudio_turn(self: &Arc<Self>, request: Request, parsed: &Value) -> Result<(), String> {
902
+ let empty = Vec::new();
903
+ let messages = parsed.get("messages").and_then(|m| m.as_array()).unwrap_or(&empty);
904
+ let (pairs, user_text) = match lmstudio::fold_messages(messages) {
905
+ Ok(f) => f,
906
+ Err(e) => {
907
+ let _ = request.respond(err_response(400, &e));
908
+ return Ok(());
909
+ }
910
+ };
911
+ let privacy_str = parsed
912
+ .get("privacy_mode")
913
+ .and_then(|p| p.as_str())
914
+ .unwrap_or(TurnPrivacyMode::LocalOnly.as_str())
915
+ .to_string();
916
+
917
+ let planned = (|| -> Result<_, String> {
918
+ let mut store = self.store.lock().unwrap();
919
+ let plan = lmstudio::resolve(&store, &pairs)?;
920
+ let (conversation_id, parent_turn_id, imported_turns) = match &plan.conversation_id {
921
+ Some(conv) => (conv.clone(), plan.parent_turn_id.clone(), 0usize),
922
+ None => {
923
+ let conv = fresh_id("conv");
924
+ let title: String = user_text.trim().chars().take(48).collect();
925
+ store.create_conversation(&conv, title.trim(), None, now_ms())?;
926
+ let leaf = if plan.kind == ResolutionKind::Imported {
927
+ lmstudio::import_pairs(&mut store, &self.tokenizer, &conv, &pairs, || fresh_id("turn"), now_ms())?
928
+ } else {
929
+ None
930
+ };
931
+ let imported = if leaf.is_some() { pairs.len() } else { 0 };
932
+ (conv, leaf, imported)
933
+ }
934
+ };
935
+ let parent_info = match &parent_turn_id {
936
+ Some(id) => {
937
+ let t = store.get_turn(id)?;
938
+ json!({
939
+ "turn_id": t.turn_id,
940
+ "verification_status": t.verification_status.as_str(),
941
+ "privacy_mode": t.privacy_mode.as_str(),
942
+ "receipt_present": t.receipt_json.is_some(),
943
+ })
944
+ }
945
+ None => Value::Null,
946
+ };
947
+ let certified_head = store.heads(&conversation_id)?.certified_head;
948
+ Ok((plan.kind, plan.matched_pairs, conversation_id, parent_turn_id, imported_turns, parent_info, certified_head))
949
+ })();
950
+ let (kind, matched_pairs, conversation_id, parent_turn_id, imported_turns, parent_info, certified_head) =
951
+ match planned {
952
+ Ok(p) => p,
953
+ Err(e) => {
954
+ let _ = request.respond(err_response(500, &e));
955
+ return Ok(());
956
+ }
957
+ };
958
+
959
+ let mut preambles = vec![json!({
960
+ "type": "lmstudio_resolve",
961
+ "resolution": kind.as_str(),
962
+ "conversation_id": conversation_id,
963
+ "parent_turn_id": parent_turn_id,
964
+ "matched_pairs": matched_pairs,
965
+ "imported_turns": imported_turns,
966
+ "parent": parent_info,
967
+ "certified_head": certified_head,
968
+ })];
969
+
970
+ // GPT-style live search, reusing the legacy stack via the sidecar. Runs after resolve
971
+ // (which only reads past pairs) and before compile; the rendered evidence joins the
972
+ // user block, so the receipt's prompt commitment covers the bundle digest.
973
+ let search = self.run_search_sidecar(&user_text, pairs.last().map(|p| p.user.as_str()));
974
+ let user_text = search.as_ref().map(|s| s.query.clone()).unwrap_or(user_text);
975
+ if let Some(s) = &search {
976
+ preambles.push(json!({
977
+ "type": "search_status",
978
+ "lines": s.summary_lines,
979
+ "bundle_sha256": s.context.bundle_sha256,
980
+ }));
981
+ }
982
+
983
+ let synth = json!({
984
+ "text": user_text,
985
+ "parent_turn_id": parent_turn_id,
986
+ "privacy_mode": privacy_str,
987
+ });
988
+ self.session_message(request, &conversation_id, &synth, preambles, search)
989
+ }
990
+
991
+ /// GET /v1/turns/{id} — settlement status + receipt summary for one turn (the plugin's
992
+ /// post-turn poll target; also generally useful observability).
993
+ fn turn_info(&self, request: Request, turn_id: &str) -> Result<(), String> {
994
+ let looked_up = (|| -> Result<_, String> {
995
+ let store = self.store.lock().unwrap();
996
+ let turn = store.get_turn(turn_id)?;
997
+ let certified_head = store.heads(&turn.conversation_id)?.certified_head;
998
+ Ok((turn, certified_head))
999
+ })();
1000
+ let (turn, certified_head) = match looked_up {
1001
+ Ok(t) => t,
1002
+ Err(e) => {
1003
+ let _ = request.respond(err_response(404, &e));
1004
+ return Ok(());
1005
+ }
1006
+ };
1007
+ let receipt = match turn.receipt_json.as_deref() {
1008
+ Some(raw) => {
1009
+ let v: Value = serde_json::from_str(raw).unwrap_or(Value::Null);
1010
+ let field = |k: &str| v.get(k).cloned().unwrap_or(Value::Null);
1011
+ json!({
1012
+ "present": true,
1013
+ "schema": field("schema"),
1014
+ "class": field("class"),
1015
+ "prompt_commitment": field("prompt_commitment"),
1016
+ "output_commitment": field("output_commitment"),
1017
+ "signer_key_id": field("signer_key_id"),
1018
+ "backend": field("backend"),
1019
+ "eos_reached": field("eos_reached"),
1020
+ })
1021
+ }
1022
+ None => json!({ "present": false }),
1023
+ };
1024
+ let _ = request.respond(json_response(
1025
+ 200,
1026
+ &json!({
1027
+ "turn_id": turn.turn_id,
1028
+ "conversation_id": turn.conversation_id,
1029
+ "parent_turn_id": turn.parent_turn_id,
1030
+ "verification_status": turn.verification_status.as_str(),
1031
+ "privacy_mode": turn.privacy_mode.as_str(),
1032
+ "stop_reason": turn.stop_reason,
1033
+ "output_tokens": turn.output_ids.len(),
1034
+ "created_ms": turn.created_ms,
1035
+ "certified_head": certified_head,
1036
+ "receipt": receipt,
1037
+ "runtime_roots": turn
1038
+ .runtime_roots_json
1039
+ .as_deref()
1040
+ .and_then(|r| serde_json::from_str::<Value>(r).ok()),
1041
+ "search": match turn.search_bundle_json.as_deref() {
1042
+ Some(raw) => {
1043
+ let bundle: Value = serde_json::from_str(raw).unwrap_or(Value::Null);
1044
+ json!({
1045
+ "present": true,
1046
+ "bundle_sha256": bundle.get("bundle_sha256").cloned().unwrap_or(Value::Null),
1047
+ "provider": bundle.get("provider").cloned().unwrap_or(Value::Null),
1048
+ })
1049
+ }
1050
+ None => json!({ "present": false }),
1051
+ },
1052
+ }),
1053
+ ));
1054
+ Ok(())
1055
+ }
1056
+
1057
+ /// GET /v1/turns/{id}/receipt — the engine receipt JSON verbatim (404 when the turn ran
1058
+ /// without an audit key or predates receipts).
1059
+ fn turn_receipt(&self, request: Request, turn_id: &str) -> Result<(), String> {
1060
+ let receipt = match self.store.lock().unwrap().get_turn(turn_id) {
1061
+ Ok(t) => t.receipt_json,
1062
+ Err(e) => {
1063
+ let _ = request.respond(err_response(404, &e));
1064
+ return Ok(());
1065
+ }
1066
+ };
1067
+ match receipt {
1068
+ Some(raw) => {
1069
+ let response = Response::from_data(raw.into_bytes())
1070
+ .with_status_code(200)
1071
+ .with_header(Header::from_bytes("Content-Type", "application/json").unwrap());
1072
+ let _ = request.respond(response);
1073
+ }
1074
+ None => {
1075
+ let _ = request.respond(err_response(404, "turn has no receipt"));
1076
+ }
1077
+ }
1078
+ Ok(())
1079
+ }
1080
+
1081
+ /// Write `.receipt.json` + `.opening.json` sidecars for a completed turn into
1082
+ /// `--receipt-dir` (dir 0700, files 0600 — the legacy server's convention). The receipt is
1083
+ /// the engine's JSON verbatim; the opening records what an offline verifier needs to replay
1084
+ /// the commitment: exact prompt/output ids, the tokenizer digest they decode under, and the
1085
+ /// turn's identity in the session store.
1086
+ fn write_receipt_sidecars(&self, turn_id: &str) -> Result<(), String> {
1087
+ let Some(dir) = &self.receipt_dir else { return Ok(()) };
1088
+ let turn = self.store.lock().unwrap().get_turn(turn_id)?;
1089
+ let receipt = turn.receipt_json.as_deref().ok_or("turn has no receipt")?;
1090
+ std::fs::create_dir_all(dir).map_err(|e| format!("create {}: {e}", dir.display()))?;
1091
+ #[cfg(unix)]
1092
+ {
1093
+ use std::os::unix::fs::PermissionsExt;
1094
+ let _ = std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700));
1095
+ }
1096
+ let commitment16: String = serde_json::from_str::<Value>(receipt)
1097
+ .ok()
1098
+ .and_then(|v| v.get("output_commitment").and_then(|c| c.as_str()).map(|s| s.chars().take(16).collect()))
1099
+ .unwrap_or_default();
1100
+ let base = if commitment16.is_empty() {
1101
+ format!("turn-{turn_id}")
1102
+ } else {
1103
+ format!("turn-{turn_id}-{commitment16}")
1104
+ };
1105
+ // Legacy convention: the .search.json sidecar is the full bundle plus the receipt
1106
+ // binding, so a verifier can tie the bundle to this exact generation.
1107
+ if let Some(bundle_raw) = turn.search_bundle_json.as_deref() {
1108
+ if let Ok(mut bundle) = serde_json::from_str::<Value>(bundle_raw) {
1109
+ if let Some(map) = bundle.as_object_mut() {
1110
+ let receipt_value: Value = serde_json::from_str(receipt).unwrap_or(Value::Null);
1111
+ map.insert(
1112
+ "receipt_binding".into(),
1113
+ json!({
1114
+ "prompt_commitment": receipt_value.get("prompt_commitment").cloned().unwrap_or(Value::Null),
1115
+ "output_commitment": receipt_value.get("output_commitment").cloned().unwrap_or(Value::Null),
1116
+ }),
1117
+ );
1118
+ }
1119
+ let path = dir.join(format!("{base}.search.json"));
1120
+ let bytes = serde_json::to_vec_pretty(&bundle).map_err(|e| e.to_string())?;
1121
+ std::fs::write(&path, &bytes).map_err(|e| format!("write {}: {e}", path.display()))?;
1122
+ #[cfg(unix)]
1123
+ {
1124
+ use std::os::unix::fs::PermissionsExt;
1125
+ let _ = std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600));
1126
+ }
1127
+ }
1128
+ }
1129
+ let opening = json!({
1130
+ "schema": "qi35-gateway-opening/v1",
1131
+ "turn_id": turn.turn_id,
1132
+ "conversation_id": turn.conversation_id,
1133
+ "parent_turn_id": turn.parent_turn_id,
1134
+ "privacy_mode": turn.privacy_mode.as_str(),
1135
+ "verification_status_at_write": turn.verification_status.as_str(),
1136
+ "stop_reason": turn.stop_reason,
1137
+ "user_text": turn.role_user_text,
1138
+ "user_block_text": turn.user_block(),
1139
+ "tokenizer_blake2b256": self.tokenizer.digest_hex,
1140
+ "prompt_ids": turn.prompt_ids,
1141
+ "output_ids": turn.output_ids,
1142
+ "runtime_roots": turn.runtime_roots_json.as_deref().and_then(|r| serde_json::from_str::<Value>(r).ok()),
1143
+ "context_meta": turn.context_meta_json.as_deref().and_then(|m| serde_json::from_str::<Value>(m).ok()),
1144
+ "created_ms": turn.created_ms,
1145
+ });
1146
+ for (suffix, bytes) in [
1147
+ (".receipt.json", receipt.as_bytes().to_vec()),
1148
+ (".opening.json", serde_json::to_vec_pretty(&opening).map_err(|e| e.to_string())?),
1149
+ ] {
1150
+ let path = dir.join(format!("{base}{suffix}"));
1151
+ std::fs::write(&path, &bytes).map_err(|e| format!("write {}: {e}", path.display()))?;
1152
+ #[cfg(unix)]
1153
+ {
1154
+ use std::os::unix::fs::PermissionsExt;
1155
+ let _ = std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600));
1156
+ }
1157
+ }
1158
+ Ok(())
1159
+ }
1160
+
1161
+ // ---- session surface ----------------------------------------------------------------
1162
+
1163
+ fn session_history(&self, request: Request, conversation_id: &str) -> Result<(), String> {
1164
+ let store = self.store.lock().unwrap();
1165
+ let heads = match store.heads(conversation_id) {
1166
+ Ok(h) => h,
1167
+ Err(e) => {
1168
+ let _ = request.respond(err_response(404, &e));
1169
+ return Ok(());
1170
+ }
1171
+ };
1172
+ let turns = match &heads.local_head {
1173
+ Some(leaf) => store.branch_history(leaf)?,
1174
+ None => Vec::new(),
1175
+ };
1176
+ let list: Vec<Value> = turns
1177
+ .iter()
1178
+ .map(|t| {
1179
+ json!({
1180
+ "turn_id": t.turn_id,
1181
+ "parent_turn_id": t.parent_turn_id,
1182
+ "user_text": t.role_user_text,
1183
+ "assistant_text": t.display_text,
1184
+ "verification_status": t.verification_status.as_str(),
1185
+ "privacy_mode": t.privacy_mode.as_str(),
1186
+ "stop_reason": t.stop_reason,
1187
+ "context_meta": t.context_meta_json.as_deref().and_then(|m| serde_json::from_str::<Value>(m).ok()),
1188
+ "created_ms": t.created_ms,
1189
+ })
1190
+ })
1191
+ .collect();
1192
+ let _ = request.respond(json_response(
1193
+ 200,
1194
+ &json!({ "local_head": heads.local_head, "certified_head": heads.certified_head, "turns": list }),
1195
+ ));
1196
+ Ok(())
1197
+ }
1198
+
1199
+ /// The native turn endpoint. Body:
1200
+ /// `{ "text": …, "parent_turn_id"?, "privacy_mode"?, "artifact_ids"?: […], "tool_call_ids"?: […] }`.
1201
+ /// `parent_turn_id` other than the current head IS the branch/edit/regenerate mechanism;
1202
+ /// attachments and executed tool results are compiled into THIS turn's user block.
1203
+ /// `preambles` (used by the LM Studio bridge) are streamed as raw SSE frames before
1204
+ /// `Started`, so the client learns how its transcript resolved (and what was searched)
1205
+ /// before tokens arrive. `search` joins the compiled user block and is persisted on the
1206
+ /// turn as the committed bundle.
1207
+ fn session_message(
1208
+ self: &Arc<Self>,
1209
+ request: Request,
1210
+ conversation_id: &str,
1211
+ parsed: &Value,
1212
+ preambles: Vec<Value>,
1213
+ search: Option<TurnSearch>,
1214
+ ) -> Result<(), String> {
1215
+ let user_text = match parsed.get("text").and_then(|t| t.as_str()) {
1216
+ Some(t) if !t.is_empty() => t.to_string(),
1217
+ _ => {
1218
+ let _ = request.respond(err_response(400, "missing text"));
1219
+ return Ok(());
1220
+ }
1221
+ };
1222
+ let privacy = parsed
1223
+ .get("privacy_mode")
1224
+ .and_then(|p| p.as_str())
1225
+ .and_then(TurnPrivacyMode::from_str)
1226
+ .unwrap_or(TurnPrivacyMode::LocalOnly);
1227
+ let id_list = |key: &str| -> Vec<String> {
1228
+ parsed
1229
+ .get(key)
1230
+ .and_then(|v| v.as_array())
1231
+ .map(|a| a.iter().filter_map(|x| x.as_str().map(String::from)).collect())
1232
+ .unwrap_or_default()
1233
+ };
1234
+ let artifact_ids = id_list("artifact_ids");
1235
+ let tool_call_ids = id_list("tool_call_ids");
1236
+
1237
+ // Gather everything the compiler needs under one store lock; blob reads happen outside.
1238
+ let gathered = (|| -> Result<_, String> {
1239
+ let store = self.store.lock().unwrap();
1240
+ let heads = store.heads(conversation_id)?;
1241
+ let parent = parsed
1242
+ .get("parent_turn_id")
1243
+ .and_then(|p| p.as_str())
1244
+ .map(String::from)
1245
+ .or(heads.local_head.clone());
1246
+ let history = match &parent {
1247
+ Some(leaf) => store.branch_history(leaf)?,
1248
+ None => Vec::new(),
1249
+ };
1250
+ let project = match store.conversation_project(conversation_id)? {
1251
+ Some(pid) => Some(store.get_project(&pid)?),
1252
+ None => None,
1253
+ };
1254
+ let mut namespaces = vec!["global".to_string()];
1255
+ if let Some(p) = &project {
1256
+ namespaces.push(p.project_id.clone());
1257
+ }
1258
+ let namespace_refs: Vec<&str> = namespaces.iter().map(String::as_str).collect();
1259
+ let memory = store.active_memory(&namespace_refs)?;
1260
+ let attachment_meta: Vec<ArtifactMeta> =
1261
+ artifact_ids.iter().map(|id| store.get_artifact(id)).collect::<Result<_, _>>()?;
1262
+ let tool_rows: Vec<ToolCallRow> =
1263
+ tool_call_ids.iter().map(|id| store.get_tool_call(id)).collect::<Result<_, _>>()?;
1264
+ Ok((parent, history, project, memory, attachment_meta, tool_rows))
1265
+ })();
1266
+ let (parent, history, project, memory, attachment_meta, tool_rows) = match gathered {
1267
+ Ok(g) => g,
1268
+ Err(e) => {
1269
+ let _ = request.respond(err_response(404, &e));
1270
+ return Ok(());
1271
+ }
1272
+ };
1273
+
1274
+ let prepared = (|| -> Result<_, String> {
1275
+ let mut attachments = Vec::new();
1276
+ for meta in &attachment_meta {
1277
+ if meta.parser_policy != PARSER_POLICY_UTF8_TEXT_V1 {
1278
+ return Err(format!(
1279
+ "artifact {} has no context parser (policy {:?}) — only UTF-8 text artifacts can attach in v1",
1280
+ meta.artifact_id, meta.parser_policy
1281
+ ));
1282
+ }
1283
+ attachments.push(AttachmentForContext {
1284
+ artifact_id: meta.artifact_id.clone(),
1285
+ display_name: if meta.display_name.is_empty() { meta.artifact_id.clone() } else { meta.display_name.clone() },
1286
+ content_hash: meta.content_hash.clone(),
1287
+ text: self.blobs.parse_utf8_text(&meta.content_hash)?,
1288
+ });
1289
+ }
1290
+ let mut tool_results = Vec::new();
1291
+ for row in &tool_rows {
1292
+ if row.status != "executed" {
1293
+ return Err(format!("tool call {} is {:?} — only executed calls can enter context", row.call_id, row.status));
1294
+ }
1295
+ let text = match &row.result_artifact_id {
1296
+ Some(artifact_id) => {
1297
+ let meta = self.store.lock().unwrap().get_artifact(artifact_id)?;
1298
+ self.blobs.parse_utf8_text(&meta.content_hash).unwrap_or_else(|_| row.result_preview.clone())
1299
+ }
1300
+ None => row.result_preview.clone(),
1301
+ };
1302
+ tool_results.push(ToolResultForContext { call_id: row.call_id.clone(), tool_name: row.tool_name.clone(), text });
1303
+ }
1304
+ Ok((attachments, tool_results))
1305
+ })();
1306
+ let (attachments, tool_results) = match prepared {
1307
+ Ok(p) => p,
1308
+ Err(e) => {
1309
+ let _ = request.respond(err_response(400, &e));
1310
+ return Ok(());
1311
+ }
1312
+ };
1313
+
1314
+ let memory_ctx: Vec<MemoryForContext> = memory
1315
+ .iter()
1316
+ .map(|m| MemoryForContext { memory_id: m.memory_id.clone(), content: m.content.clone() })
1317
+ .collect();
1318
+ let inputs = ContextInputs {
1319
+ project_instructions: project.as_ref().map(|p| p.instructions.as_str()),
1320
+ memory: &memory_ctx,
1321
+ attachments: &attachments,
1322
+ tool_results: &tool_results,
1323
+ history: &history,
1324
+ user_text: &user_text,
1325
+ search: search.as_ref().map(|s| s.context.clone()),
1326
+ };
1327
+ let compiled = match self.compiler.compile(&self.tokenizer, &inputs) {
1328
+ Ok(c) => c,
1329
+ Err(e) => {
1330
+ let _ = request.respond(err_response(400, &e));
1331
+ return Ok(());
1332
+ }
1333
+ };
1334
+
1335
+ let turn_id = fresh_id("turn");
1336
+ let job_id = fresh_id("job");
1337
+ {
1338
+ let mut store = self.store.lock().unwrap();
1339
+ if let Err(e) = store.begin_turn(&NewTurn {
1340
+ turn_id: &turn_id,
1341
+ conversation_id,
1342
+ parent_turn_id: parent.as_deref(),
1343
+ role_user_text: &user_text,
1344
+ user_block_text: &compiled.user_block_text,
1345
+ context_meta_json: Some(&compiled.context_meta_json),
1346
+ prompt_ids: &compiled.ids,
1347
+ privacy_mode: privacy,
1348
+ search_bundle_json: search.as_ref().map(|s| s.bundle_json.as_str()),
1349
+ now_ms: now_ms(),
1350
+ }) {
1351
+ let _ = request.respond(err_response(500, &e));
1352
+ return Ok(());
1353
+ }
1354
+ }
1355
+
1356
+ let cancel = Arc::new(AtomicBool::new(false));
1357
+ self.cancels.lock().unwrap().insert(job_id.clone(), Arc::clone(&cancel));
1358
+
1359
+ let stream_result = self.stream_generation(
1360
+ request,
1361
+ &job_id,
1362
+ &turn_id,
1363
+ &compiled.ids,
1364
+ compiled.system_prefix_len,
1365
+ compiled.dropped_turns,
1366
+ &cancel,
1367
+ preambles,
1368
+ );
1369
+ self.cancels.lock().unwrap().remove(&job_id);
1370
+ stream_result
1371
+ }
1372
+
1373
+ #[allow(clippy::too_many_arguments)]
1374
+ fn stream_generation(
1375
+ &self,
1376
+ request: Request,
1377
+ job_id: &str,
1378
+ turn_id: &str,
1379
+ prompt_ids: &[u32],
1380
+ cache_prefix_len: usize,
1381
+ dropped_turns: usize,
1382
+ cancel: &Arc<AtomicBool>,
1383
+ preambles: Vec<Value>,
1384
+ ) -> Result<(), String> {
1385
+ let mut writer = request.into_writer();
1386
+ let mut sse = SseWriter::new(&mut writer)?;
1387
+ for frame in &preambles {
1388
+ sse.raw(frame)?;
1389
+ }
1390
+ sse.event(&GenerationEventV1::Started {
1391
+ job_id: job_id.to_string(),
1392
+ turn_id: turn_id.to_string(),
1393
+ prompt_tokens: prompt_ids.len() as u32,
1394
+ })?;
1395
+ if dropped_turns > 0 {
1396
+ // No silent truncation (the doc's rule): surface the budget cut in-stream.
1397
+ sse.raw(&json!({ "type": "context_budget", "dropped_turns": dropped_turns }))?;
1398
+ }
1399
+
1400
+ let mut decoder = IncrementalDecoder::new();
1401
+ let outcome = {
1402
+ // §14: the user-facing turn goes through the scheduler as ForegroundChat — it
1403
+ // overtakes any queued background work, then owns the engine.
1404
+ let _lease = self.scheduler.acquire(JobClass::ForegroundChat);
1405
+ let mut engine = self.engine.lock().unwrap();
1406
+ let tokenizer = &self.tokenizer;
1407
+ let sse_cell = std::cell::RefCell::new(&mut sse);
1408
+ engine.generate(prompt_ids, cache_prefix_len, cancel, |index, token_id| {
1409
+ let delta = decoder.push(tokenizer, token_id).unwrap_or_default();
1410
+ let _ = sse_cell.borrow_mut().event(&GenerationEventV1::Token {
1411
+ index,
1412
+ token_id,
1413
+ utf8_delta: delta,
1414
+ });
1415
+ })
1416
+ };
1417
+
1418
+ match outcome {
1419
+ Ok(outcome) => {
1420
+ self.scheduler.record_generation(outcome.output_ids.len() as u64, outcome.elapsed_ms);
1421
+ let tail = decoder.finish(&self.tokenizer).unwrap_or_default();
1422
+ if !tail.is_empty() && !cancel.load(Ordering::Relaxed) {
1423
+ let last_index = outcome.output_ids.len().saturating_sub(1) as u32;
1424
+ sse.event(&GenerationEventV1::Token { index: last_index, token_id: 0, utf8_delta: tail })?;
1425
+ }
1426
+ let cancelled = cancel.load(Ordering::Relaxed);
1427
+ let stop = if cancelled {
1428
+ CanonicalStopReason::UserCancelled
1429
+ } else if outcome.eos_reached {
1430
+ CanonicalStopReason::Eos
1431
+ } else {
1432
+ CanonicalStopReason::MaxTokens
1433
+ };
1434
+ let status = if cancelled {
1435
+ TurnVerificationStatus::MintIneligible
1436
+ } else {
1437
+ TurnVerificationStatus::LocalComplete
1438
+ };
1439
+ let display = self.tokenizer.decode_display(&outcome.output_ids).unwrap_or_default();
1440
+ let roots_json = outcome.runtime_roots.as_ref().and_then(|r| serde_json::to_string(r).ok());
1441
+ self.store
1442
+ .lock()
1443
+ .unwrap()
1444
+ .complete_turn(&crate::store::CompletedTurn {
1445
+ turn_id,
1446
+ output_ids: &outcome.output_ids,
1447
+ display_text: display.trim(),
1448
+ stop_reason: match stop {
1449
+ CanonicalStopReason::Eos => "eos",
1450
+ CanonicalStopReason::MaxTokens => "max_tokens",
1451
+ CanonicalStopReason::UserCancelled => "user_cancelled",
1452
+ },
1453
+ status,
1454
+ receipt_json: outcome.receipt_json.as_deref(),
1455
+ runtime_roots_json: roots_json.as_deref(),
1456
+ })
1457
+ .map_err(|e| format!("persist turn: {e}"))?;
1458
+ if outcome.receipt_json.is_some() {
1459
+ // Sidecar failure never fails the turn — the receipt is durable in the store;
1460
+ // the files are the offline-verifier convenience form.
1461
+ if let Err(e) = self.write_receipt_sidecars(turn_id) {
1462
+ eprintln!("[palw-gateway] receipt sidecar for {turn_id}: {e}");
1463
+ }
1464
+ }
1465
+ sse.event(&GenerationEventV1::Completed {
1466
+ stop_reason: stop,
1467
+ output_tokens: outcome.output_ids.len() as u32,
1468
+ cache_hit_tokens: outcome.cache_hit_tokens,
1469
+ elapsed_ms: outcome.elapsed_ms,
1470
+ })?;
1471
+ sse.done()
1472
+ }
1473
+ Err(e) => {
1474
+ {
1475
+ let store = self.store.lock().unwrap();
1476
+ let _ = store.complete_turn(&crate::store::CompletedTurn {
1477
+ turn_id,
1478
+ output_ids: &[],
1479
+ display_text: "",
1480
+ stop_reason: "engine_error",
1481
+ status: TurnVerificationStatus::MintIneligible,
1482
+ receipt_json: None,
1483
+ runtime_roots_json: None,
1484
+ });
1485
+ // Failed turns do not anchor the branch: the next message retries from the
1486
+ // parent instead of building on a turn that produced nothing.
1487
+ if let Ok(turn) = store.get_turn(turn_id) {
1488
+ let _ = store.retract_failed_head(&turn.conversation_id, turn_id, now_ms());
1489
+ }
1490
+ }
1491
+ sse.event(&GenerationEventV1::Failed { error_code: 1, message: e.clone() })?;
1492
+ sse.done()?;
1493
+ Err(e)
1494
+ }
1495
+ }
1496
+ }
1497
+
1498
+ /// OpenAI-compatible stateless path (existing LM Studio plugin). Transcript comes from the
1499
+ /// client; we build ChatML text exactly like `qi35_chat.py` did, but through the compiler's
1500
+ /// fragment assembly so at least the system prefix caches.
1501
+ fn chat_completions(self: &Arc<Self>, request: Request, parsed: &Value) -> Result<(), String> {
1502
+ let empty = Vec::new();
1503
+ let messages = parsed.get("messages").and_then(|m| m.as_array()).unwrap_or(&empty);
1504
+ // Fold OpenAI messages[] into (history pairs, pending user) — the qi35_lmstudio_server
1505
+ // `conversation()` shape.
1506
+ let mut history: Vec<crate::store::StoredTurn> = Vec::new();
1507
+ let mut pending_user: Option<String> = None;
1508
+ for m in messages {
1509
+ let role = m.get("role").and_then(|r| r.as_str()).unwrap_or("");
1510
+ let content = m.get("content").and_then(|c| c.as_str()).unwrap_or("").to_string();
1511
+ match role {
1512
+ "user" => {
1513
+ if let Some(prev) = pending_user.take() {
1514
+ // Two users in a row: treat the earlier as a turn with empty answer.
1515
+ history.push(synthetic_turn(prev, Vec::new()));
1516
+ }
1517
+ pending_user = Some(content);
1518
+ }
1519
+ "assistant" => {
1520
+ let user = pending_user.take().unwrap_or_default();
1521
+ // Stateless path: the client only has TEXT, so we must encode the past
1522
+ // answer. This is exactly the id-instability the native path exists to
1523
+ // avoid — done once here for compatibility, flagged in the response.
1524
+ let ids = self.tokenizer.encode_lossless(&content).unwrap_or_default();
1525
+ history.push(synthetic_turn(user, ids));
1526
+ }
1527
+ _ => {}
1528
+ }
1529
+ }
1530
+ let Some(user_text) = pending_user else {
1531
+ let _ = request.respond(err_response(400, "no trailing user message"));
1532
+ return Ok(());
1533
+ };
1534
+ let inputs = ContextInputs { history: &history, user_text: &user_text, ..Default::default() };
1535
+ let compiled = match self.compiler.compile(&self.tokenizer, &inputs) {
1536
+ Ok(c) => c,
1537
+ Err(e) => {
1538
+ let _ = request.respond(err_response(400, &e));
1539
+ return Ok(());
1540
+ }
1541
+ };
1542
+
1543
+ let cancel = Arc::new(AtomicBool::new(false));
1544
+ let mut writer = request.into_writer();
1545
+ let mut sse = SseWriter::new(&mut writer)?;
1546
+ let created = now_ms() / 1000;
1547
+ let model = self.model_name.clone();
1548
+ let mut decoder = IncrementalDecoder::new();
1549
+ let outcome = {
1550
+ let _lease = self.scheduler.acquire(JobClass::ForegroundChat);
1551
+ let mut engine = self.engine.lock().unwrap();
1552
+ let tokenizer = &self.tokenizer;
1553
+ let sse_cell = std::cell::RefCell::new(&mut sse);
1554
+ engine.generate(&compiled.ids, compiled.system_prefix_len, &cancel, |_, token_id| {
1555
+ if let Ok(delta) = decoder.push(tokenizer, token_id) {
1556
+ if !delta.is_empty() {
1557
+ let chunk = json!({
1558
+ "object": "chat.completion.chunk",
1559
+ "created": created,
1560
+ "model": model,
1561
+ "choices": [{ "index": 0, "delta": { "content": delta }, "finish_reason": Value::Null }],
1562
+ });
1563
+ let _ = sse_cell.borrow_mut().raw(&chunk);
1564
+ }
1565
+ }
1566
+ })
1567
+ };
1568
+ match outcome {
1569
+ Ok(outcome) => {
1570
+ self.scheduler.record_generation(outcome.output_ids.len() as u64, outcome.elapsed_ms);
1571
+ let tail = decoder.finish(&self.tokenizer).unwrap_or_default();
1572
+ if !tail.is_empty() {
1573
+ sse.raw(&json!({
1574
+ "object": "chat.completion.chunk",
1575
+ "created": created,
1576
+ "model": self.model_name,
1577
+ "choices": [{ "index": 0, "delta": { "content": tail }, "finish_reason": Value::Null }],
1578
+ }))?;
1579
+ }
1580
+ sse.raw(&json!({
1581
+ "object": "chat.completion.chunk",
1582
+ "created": created,
1583
+ "model": self.model_name,
1584
+ "choices": [{ "index": 0, "delta": {}, "finish_reason": if outcome.eos_reached { "stop" } else { "length" } }],
1585
+ }))?;
1586
+ sse.done()
1587
+ }
1588
+ Err(e) => {
1589
+ sse.raw(&json!({ "error": { "message": e } }))?;
1590
+ sse.done()
1591
+ }
1592
+ }
1593
+ }
1594
+ }
1595
+
1596
+ fn synthetic_turn(user: String, output_ids: Vec<u32>) -> crate::store::StoredTurn {
1597
+ crate::store::StoredTurn {
1598
+ turn_id: String::new(),
1599
+ conversation_id: String::new(),
1600
+ parent_turn_id: None,
1601
+ role_user_text: user,
1602
+ user_block_text: None,
1603
+ context_meta_json: None,
1604
+ display_text: String::new(),
1605
+ prompt_ids: Vec::new(),
1606
+ output_ids,
1607
+ stop_reason: None,
1608
+ verification_status: TurnVerificationStatus::LocalComplete,
1609
+ privacy_mode: TurnPrivacyMode::LocalOnly,
1610
+ receipt_json: None,
1611
+ runtime_roots_json: None,
1612
+ search_bundle_json: None,
1613
+ created_ms: 0,
1614
+ }
1615
+ }
1616
+
1617
+ /// Minimal SSE writer over tiny_http's raw stream (headers hand-rolled; tiny_http's typed
1618
+ /// Response cannot stream a body incrementally).
1619
+ ///
1620
+ /// The body is CHUNKED (HTTP/1.1): tiny_http's `into_writer` hands us the socket but its drop
1621
+ /// does NOT close the connection — the library keeps it for the next keep-alive request. A
1622
+ /// bare-framed body therefore never signals its end and generic clients (curl, fetch,
1623
+ /// EventSource) hang waiting for EOF; the terminating zero-chunk is what tells them the
1624
+ /// response is over, and the connection stays legitimately reusable. (HTTP/1.0 clients are not
1625
+ /// supported on the streaming endpoints — every real consumer here speaks 1.1.)
1626
+ struct SseWriter<'a> {
1627
+ writer: &'a mut dyn Write,
1628
+ }
1629
+
1630
+ impl<'a> SseWriter<'a> {
1631
+ fn new(writer: &'a mut dyn Write) -> Result<Self, String> {
1632
+ write!(
1633
+ writer,
1634
+ "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nCache-Control: no-cache\r\nTransfer-Encoding: chunked\r\n\r\n"
1635
+ )
1636
+ .map_err(|e| format!("sse headers: {e}"))?;
1637
+ writer.flush().map_err(|e| e.to_string())?;
1638
+ Ok(Self { writer })
1639
+ }
1640
+
1641
+ fn chunk(&mut self, payload: &str) -> Result<(), String> {
1642
+ write!(self.writer, "{:x}\r\n{payload}\r\n", payload.len()).map_err(|e| format!("sse write: {e}"))?;
1643
+ self.writer.flush().map_err(|e| e.to_string())
1644
+ }
1645
+
1646
+ fn event(&mut self, event: &GenerationEventV1) -> Result<(), String> {
1647
+ let payload = serde_json::to_string(event).map_err(|e| e.to_string())?;
1648
+ self.chunk(&format!("data: {payload}\n\n"))
1649
+ }
1650
+
1651
+ fn raw(&mut self, value: &Value) -> Result<(), String> {
1652
+ self.chunk(&format!("data: {value}\n\n"))
1653
+ }
1654
+
1655
+ fn done(&mut self) -> Result<(), String> {
1656
+ self.chunk("data: [DONE]\n\n")?;
1657
+ // Terminating zero-chunk: end-of-response on the wire.
1658
+ write!(self.writer, "0\r\n\r\n").map_err(|e| format!("sse write: {e}"))?;
1659
+ self.writer.flush().map_err(|e| e.to_string())
1660
+ }
1661
+ }
1662
+
1663
+ #[cfg(test)]
1664
+ mod tests {
1665
+ use super::*;
1666
+
1667
+ #[test]
1668
+ fn query_param_decodes() {
1669
+ assert_eq!(query_param("/x?name=design.md", "name").as_deref(), Some("design.md"));
1670
+ assert_eq!(query_param("/x?name=a%20b+c&z=1", "name").as_deref(), Some("a b c"));
1671
+ assert_eq!(query_param("/x?a=1", "b"), None);
1672
+ assert_eq!(query_param("/x", "a"), None);
1673
+ }
1674
+
1675
+ #[test]
1676
+ fn path_seg_extracts_single_segment() {
1677
+ assert_eq!(path_seg("/v1/sessions/abc/messages", "/v1/sessions/", "/messages"), Some("abc"));
1678
+ assert_eq!(path_seg("/v1/sessions//messages", "/v1/sessions/", "/messages"), None);
1679
+ assert_eq!(path_seg("/v1/sessions/a/b/messages", "/v1/sessions/", "/messages"), None);
1680
+ assert_eq!(path_seg("/v1/other/abc/messages", "/v1/sessions/", "/messages"), None);
1681
+ }
1682
+ }
palw-gateway/src/lmstudio.rs ADDED
@@ -0,0 +1,393 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! LM Studio bridge: resolve a client-owned OpenAI transcript onto the session store.
2
+ //!
3
+ //! LM Studio's generator plugin owns the transcript and sends the WHOLE message list on every
4
+ //! turn — it has no gateway session ids. This module maps that stateless shape onto the native
5
+ //! id-stable session layer by CONTENT: the past `(user, assistant)` pairs are matched against
6
+ //! stored turns, so a continued chat lands on its existing branch (and the prefix cache hits),
7
+ //! an edited/regenerated message lands as a sibling branch of the turn it diverged from, and an
8
+ //! unknown transcript becomes a new conversation with the foreign history imported.
9
+ //!
10
+ //! Matching is on NORMALIZED text: `<think>…</think>` blocks are stripped from assistant text
11
+ //! before comparison, because the plugin re-sends history through LM Studio's own store, which
12
+ //! may or may not retain reasoning blocks (the legacy server's recorded openings show they
13
+ //! sometimes survive). The user side is compared verbatim (trimmed) — user text has no
14
+ //! runtime-injected decoration.
15
+ //!
16
+ //! Import keeps the conversation usable but honest: imported turns carry no engine execution
17
+ //! (`prompt_ids` empty, output ids re-encoded from text — exactly the id-instability the native
18
+ //! path avoids from then on) and are marked `MintIneligible` with stop reason `imported`, so
19
+ //! they can never enter the settlement pipeline.
20
+
21
+ use crate::events::{TurnPrivacyMode, TurnVerificationStatus};
22
+ use crate::store::{CompletedTurn, NewTurn, SessionStore, StoredTurn};
23
+ use crate::tokenizer::TokenizerHost;
24
+
25
+ /// One past exchange from the client transcript.
26
+ #[derive(Debug, Clone, PartialEq, Eq)]
27
+ pub struct MessagePair {
28
+ pub user: String,
29
+ /// Empty when the transcript had two user messages in a row (the earlier one is treated as
30
+ /// an exchange with an empty answer, mirroring `chat_completions`' fold).
31
+ pub assistant: String,
32
+ }
33
+
34
+ /// Fold an OpenAI `messages` array into `(past pairs, trailing user text)`.
35
+ ///
36
+ /// Same rules as the stateless `chat_completions` fold: `system` entries are dropped (the
37
+ /// system region is the gateway's), consecutive `user` messages become pairs with an empty
38
+ /// answer, and the list must end in a user message.
39
+ pub fn fold_messages(messages: &[serde_json::Value]) -> Result<(Vec<MessagePair>, String), String> {
40
+ let mut pairs = Vec::new();
41
+ let mut pending_user: Option<String> = None;
42
+ for m in messages {
43
+ let role = m.get("role").and_then(|r| r.as_str()).unwrap_or("");
44
+ let content = m.get("content").and_then(|c| c.as_str()).unwrap_or("").to_string();
45
+ match role {
46
+ "user" => {
47
+ if let Some(prev) = pending_user.take() {
48
+ pairs.push(MessagePair { user: prev, assistant: String::new() });
49
+ }
50
+ pending_user = Some(content);
51
+ }
52
+ "assistant" => {
53
+ let user = pending_user.take().unwrap_or_default();
54
+ pairs.push(MessagePair { user, assistant: content });
55
+ }
56
+ _ => {}
57
+ }
58
+ }
59
+ match pending_user {
60
+ Some(user) => Ok((pairs, user)),
61
+ None => Err("no trailing user message".into()),
62
+ }
63
+ }
64
+
65
+ /// Normalization applied to BOTH sides before text comparison: strip `<think>…</think>` blocks
66
+ /// (an unclosed `<think>` drops the rest of the string), collapse CRLF, trim.
67
+ ///
68
+ /// This intentionally covers the plugin's `PALW status (receipt対象外)` blocks too — they use
69
+ /// the same tag — so a transcript that still carries them matches the stored display text,
70
+ /// which never contains them.
71
+ pub fn normalize_for_match(text: &str) -> String {
72
+ const OPEN: &str = "<think>";
73
+ const CLOSE: &str = "</think>";
74
+ let mut out = String::with_capacity(text.len());
75
+ let mut rest = text;
76
+ loop {
77
+ match rest.find(OPEN) {
78
+ None => {
79
+ out.push_str(rest);
80
+ break;
81
+ }
82
+ Some(start) => {
83
+ out.push_str(&rest[..start]);
84
+ let after_open = &rest[start + OPEN.len()..];
85
+ match after_open.find(CLOSE) {
86
+ None => break, // unclosed block: everything after the tag is runtime noise
87
+ Some(end) => rest = &after_open[end + CLOSE.len()..],
88
+ }
89
+ }
90
+ }
91
+ }
92
+ out.replace("\r\n", "\n").trim().to_string()
93
+ }
94
+
95
+ fn pair_matches(turn: &StoredTurn, pair: &MessagePair) -> bool {
96
+ normalize_for_match(&turn.role_user_text) == normalize_for_match(&pair.user)
97
+ && normalize_for_match(&turn.display_text) == normalize_for_match(&pair.assistant)
98
+ }
99
+
100
+ /// How the transcript mapped onto the store.
101
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
102
+ pub enum ResolutionKind {
103
+ /// Every past pair matched the active branch's prefix — plain continuation.
104
+ Continued,
105
+ /// A prefix matched, then the transcript diverged (edit/regenerate in LM Studio) — the new
106
+ /// turn becomes a sibling branch under the last matching turn.
107
+ Branched,
108
+ /// Nothing matched and the transcript had no past pairs — a fresh conversation.
109
+ NewConversation,
110
+ /// Nothing matched but the transcript HAD past pairs — a fresh conversation with the
111
+ /// foreign history imported as mint-ineligible turns.
112
+ Imported,
113
+ }
114
+
115
+ impl ResolutionKind {
116
+ pub fn as_str(self) -> &'static str {
117
+ match self {
118
+ Self::Continued => "continued",
119
+ Self::Branched => "branched",
120
+ Self::NewConversation => "new_conversation",
121
+ Self::Imported => "imported",
122
+ }
123
+ }
124
+ }
125
+
126
+ /// Where the new turn should attach.
127
+ #[derive(Debug)]
128
+ pub struct ResolutionPlan {
129
+ pub kind: ResolutionKind,
130
+ /// Existing conversation to attach to (`None` ⇒ create one, importing `pairs` when
131
+ /// `kind == Imported`).
132
+ pub conversation_id: Option<String>,
133
+ /// Turn the new message continues from (`None` ⇒ the new turn is a root).
134
+ pub parent_turn_id: Option<String>,
135
+ /// How many leading pairs matched stored turns.
136
+ pub matched_pairs: usize,
137
+ }
138
+
139
+ /// Match the transcript's past pairs against stored turns.
140
+ ///
141
+ /// Candidates are root turns whose first exchange matches `pairs[0]`; each candidate is walked
142
+ /// down by matching children pair-by-pair (first match in creation order — deterministic under
143
+ /// duplicate regenerations). The deepest match wins; ties go to the oldest root. This is a
144
+ /// content walk over one branch path per candidate, not a search of the whole tree.
145
+ pub fn resolve(store: &SessionStore, pairs: &[MessagePair]) -> Result<ResolutionPlan, String> {
146
+ if pairs.is_empty() {
147
+ return Ok(ResolutionPlan {
148
+ kind: ResolutionKind::NewConversation,
149
+ conversation_id: None,
150
+ parent_turn_id: None,
151
+ matched_pairs: 0,
152
+ });
153
+ }
154
+ let mut best: Option<(usize, StoredTurn)> = None; // (matched pair count, last matched turn)
155
+ for root in store.root_turns()? {
156
+ if !pair_matches(&root, &pairs[0]) {
157
+ continue;
158
+ }
159
+ let mut last = root;
160
+ let mut matched = 1usize;
161
+ while matched < pairs.len() {
162
+ let next = store
163
+ .children_of(&last.turn_id)?
164
+ .into_iter()
165
+ .find(|child| pair_matches(child, &pairs[matched]));
166
+ match next {
167
+ Some(child) => {
168
+ last = child;
169
+ matched += 1;
170
+ }
171
+ None => break,
172
+ }
173
+ }
174
+ let deeper = best.as_ref().map(|(n, _)| matched > *n).unwrap_or(true);
175
+ if deeper {
176
+ let full = matched == pairs.len();
177
+ best = Some((matched, last));
178
+ if full {
179
+ break; // oldest fully-matching root wins; no deeper match exists
180
+ }
181
+ }
182
+ }
183
+ Ok(match best {
184
+ Some((matched, last)) => ResolutionPlan {
185
+ kind: if matched == pairs.len() { ResolutionKind::Continued } else { ResolutionKind::Branched },
186
+ conversation_id: Some(last.conversation_id.clone()),
187
+ parent_turn_id: Some(last.turn_id),
188
+ matched_pairs: matched,
189
+ },
190
+ None => ResolutionPlan {
191
+ kind: ResolutionKind::Imported,
192
+ conversation_id: None,
193
+ parent_turn_id: None,
194
+ matched_pairs: 0,
195
+ },
196
+ })
197
+ }
198
+
199
+ /// Import foreign past pairs as stored turns under `conversation_id`, returning the leaf id.
200
+ ///
201
+ /// Imported turns are honest about their provenance: no engine execution happened here, so
202
+ /// `prompt_ids` stays empty, `output_ids` is re-encoded from the normalized assistant TEXT (the
203
+ /// one unavoidable id-unstable step — every turn after this one is id-stable again), the status
204
+ /// is `MintIneligible` and the stop reason is `imported`. Display keeps the original text.
205
+ pub fn import_pairs(
206
+ store: &mut SessionStore,
207
+ tokenizer: &TokenizerHost,
208
+ conversation_id: &str,
209
+ pairs: &[MessagePair],
210
+ mut fresh_turn_id: impl FnMut() -> String,
211
+ now_ms: i64,
212
+ ) -> Result<Option<String>, String> {
213
+ let mut parent: Option<String> = None;
214
+ for pair in pairs {
215
+ let turn_id = fresh_turn_id();
216
+ store.begin_turn(&NewTurn {
217
+ turn_id: &turn_id,
218
+ conversation_id,
219
+ parent_turn_id: parent.as_deref(),
220
+ role_user_text: &pair.user,
221
+ user_block_text: &pair.user,
222
+ context_meta_json: None,
223
+ prompt_ids: &[],
224
+ privacy_mode: TurnPrivacyMode::LocalOnly,
225
+ search_bundle_json: None,
226
+ now_ms,
227
+ })?;
228
+ let clean = normalize_for_match(&pair.assistant);
229
+ let output_ids = if clean.is_empty() { Vec::new() } else { tokenizer.encode_lossless(&clean)? };
230
+ store.complete_turn(&CompletedTurn {
231
+ turn_id: &turn_id,
232
+ output_ids: &output_ids,
233
+ display_text: &pair.assistant,
234
+ stop_reason: "imported",
235
+ status: TurnVerificationStatus::MintIneligible,
236
+ receipt_json: None,
237
+ runtime_roots_json: None,
238
+ })?;
239
+ parent = Some(turn_id);
240
+ }
241
+ Ok(parent)
242
+ }
243
+
244
+ #[cfg(test)]
245
+ mod tests {
246
+ use super::*;
247
+ use serde_json::json;
248
+
249
+ #[test]
250
+ fn fold_matches_chat_completions_rules() {
251
+ let messages = vec![
252
+ json!({"role": "system", "content": "dropped"}),
253
+ json!({"role": "user", "content": "a"}),
254
+ json!({"role": "assistant", "content": "b"}),
255
+ json!({"role": "user", "content": "c"}),
256
+ json!({"role": "user", "content": "d"}),
257
+ ];
258
+ let (pairs, trailing) = fold_messages(&messages).unwrap();
259
+ assert_eq!(
260
+ pairs,
261
+ vec![
262
+ MessagePair { user: "a".into(), assistant: "b".into() },
263
+ MessagePair { user: "c".into(), assistant: String::new() },
264
+ ]
265
+ );
266
+ assert_eq!(trailing, "d");
267
+ assert!(fold_messages(&[json!({"role": "user", "content": "x"}), json!({"role": "assistant", "content": "y"})]).is_err());
268
+ }
269
+
270
+ #[test]
271
+ fn normalization_strips_think_and_status_blocks() {
272
+ assert_eq!(normalize_for_match(" answer "), "answer");
273
+ assert_eq!(normalize_for_match("<think>reasoning</think>\nanswer"), "answer");
274
+ assert_eq!(
275
+ normalize_for_match("<think>PALW status (receipt対象外):\nengine ready\n</think>\n<think>plan</think>body"),
276
+ "body"
277
+ );
278
+ // Unclosed block: everything after the tag is runtime noise, not answer text.
279
+ assert_eq!(normalize_for_match("head<think>never closed"), "head");
280
+ assert_eq!(normalize_for_match("a\r\nb"), "a\nb");
281
+ }
282
+
283
+ fn open_store() -> SessionStore {
284
+ let dir = std::env::temp_dir().join(format!("palw-lmstudio-test-{}-{}", std::process::id(), fastrand()));
285
+ std::fs::create_dir_all(&dir).unwrap();
286
+ SessionStore::open(&dir.join("s.sqlite3")).unwrap()
287
+ }
288
+
289
+ fn fastrand() -> u64 {
290
+ use std::time::{SystemTime, UNIX_EPOCH};
291
+ static C: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
292
+ SystemTime::now().duration_since(UNIX_EPOCH).unwrap().subsec_nanos() as u64
293
+ + C.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
294
+ }
295
+
296
+ fn seed_turn(
297
+ store: &mut SessionStore,
298
+ conversation: &str,
299
+ turn_id: &str,
300
+ parent: Option<&str>,
301
+ user: &str,
302
+ assistant: &str,
303
+ ) {
304
+ store
305
+ .begin_turn(&NewTurn {
306
+ turn_id,
307
+ conversation_id: conversation,
308
+ parent_turn_id: parent,
309
+ role_user_text: user,
310
+ user_block_text: user,
311
+ context_meta_json: None,
312
+ prompt_ids: &[1, 2, 3],
313
+ privacy_mode: TurnPrivacyMode::PalwMint,
314
+ search_bundle_json: None,
315
+ now_ms: 1,
316
+ })
317
+ .unwrap();
318
+ store
319
+ .complete_turn(&CompletedTurn {
320
+ turn_id,
321
+ output_ids: &[7, 8],
322
+ display_text: assistant,
323
+ stop_reason: "eos",
324
+ status: TurnVerificationStatus::LocalComplete,
325
+ receipt_json: None,
326
+ runtime_roots_json: None,
327
+ })
328
+ .unwrap();
329
+ }
330
+
331
+ #[test]
332
+ fn resolve_continues_branches_and_imports() {
333
+ let mut store = open_store();
334
+ store.create_conversation("conv1", "t", None, 1).unwrap();
335
+ seed_turn(&mut store, "conv1", "turn1", None, "hello", "<think>x</think>world");
336
+ seed_turn(&mut store, "conv1", "turn2", Some("turn1"), "next", "answer2");
337
+
338
+ // Continuation: both pairs match (client transcript lost the think block — still matches).
339
+ let pairs = vec![
340
+ MessagePair { user: "hello".into(), assistant: "world".into() },
341
+ MessagePair { user: "next".into(), assistant: "answer2".into() },
342
+ ];
343
+ let plan = resolve(&store, &pairs).unwrap();
344
+ assert_eq!(plan.kind, ResolutionKind::Continued);
345
+ assert_eq!(plan.conversation_id.as_deref(), Some("conv1"));
346
+ assert_eq!(plan.parent_turn_id.as_deref(), Some("turn2"));
347
+
348
+ // Edit of message 2 in LM Studio: first pair matches, second diverges → branch under turn1.
349
+ let pairs = vec![
350
+ MessagePair { user: "hello".into(), assistant: "world".into() },
351
+ MessagePair { user: "next".into(), assistant: "different answer".into() },
352
+ ];
353
+ let plan = resolve(&store, &pairs).unwrap();
354
+ assert_eq!(plan.kind, ResolutionKind::Branched);
355
+ assert_eq!(plan.parent_turn_id.as_deref(), Some("turn1"));
356
+ assert_eq!(plan.matched_pairs, 1);
357
+
358
+ // Regenerate (same user, no assistant yet): parent is the turn BEFORE the regenerated one.
359
+ let pairs = vec![MessagePair { user: "hello".into(), assistant: "world".into() }];
360
+ let plan = resolve(&store, &pairs).unwrap();
361
+ assert_eq!(plan.kind, ResolutionKind::Continued);
362
+ assert_eq!(plan.parent_turn_id.as_deref(), Some("turn1"));
363
+
364
+ // Unknown transcript with history → import; empty history → new conversation.
365
+ let pairs = vec![MessagePair { user: "elsewhere".into(), assistant: "said".into() }];
366
+ assert_eq!(resolve(&store, &pairs).unwrap().kind, ResolutionKind::Imported);
367
+ assert_eq!(resolve(&store, &[]).unwrap().kind, ResolutionKind::NewConversation);
368
+ }
369
+
370
+ #[test]
371
+ fn resolve_prefers_deepest_match_across_duplicate_roots() {
372
+ let mut store = open_store();
373
+ store.create_conversation("convA", "a", None, 1).unwrap();
374
+ store.create_conversation("convB", "b", None, 1).unwrap();
375
+ // Two conversations with the same first exchange; only convB has the second one.
376
+ seed_turn(&mut store, "convA", "a1", None, "hi", "yo");
377
+ seed_turn(&mut store, "convB", "b1", None, "hi", "yo");
378
+ seed_turn(&mut store, "convB", "b2", Some("b1"), "more", "deep");
379
+ let pairs = vec![
380
+ MessagePair { user: "hi".into(), assistant: "yo".into() },
381
+ MessagePair { user: "more".into(), assistant: "deep".into() },
382
+ ];
383
+ let plan = resolve(&store, &pairs).unwrap();
384
+ assert_eq!(plan.conversation_id.as_deref(), Some("convB"));
385
+ assert_eq!(plan.parent_turn_id.as_deref(), Some("b2"));
386
+ assert_eq!(plan.kind, ResolutionKind::Continued);
387
+ // The shallower duplicate root still wins when the transcript stops at pair 1
388
+ // (oldest root, deterministic).
389
+ let pairs = vec![MessagePair { user: "hi".into(), assistant: "yo".into() }];
390
+ let plan = resolve(&store, &pairs).unwrap();
391
+ assert_eq!(plan.conversation_id.as_deref(), Some("convA"));
392
+ }
393
+ }
palw-gateway/src/main.rs ADDED
@@ -0,0 +1,322 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! palw-gateway — PALW Desktop UX Phase 0+1: the resident streaming inference gateway, the
2
+ //! Conversation Session Layer, and the Phase-1 product layers from the frozen design doc
3
+ //! (`docs/palw-desktop-product-architecture.md`).
4
+ //!
5
+ //! Phase 0 (shipped first): resident engine host, token streaming, id-stable turn/branch store —
6
+ //! reuse the forward pass, replace the serving shell.
7
+ //!
8
+ //! Phase 1 (this build): Context Compiler v1 (project instructions §10 + memory snapshot §12 +
9
+ //! attachment chunk selection §11 in the system/user regions), content-addressed artifact store
10
+ //! with version chains (§8/§9), the three-mode tool runtime with sandbox and exactly-once
11
+ //! side-effect approval (§6/§17), the five-priority resource scheduler with SLA admission (§14),
12
+ //! and the settlement-clock writer (legal transitions, mismatch cascade, certified head — §4)
13
+ //! that UX Phase 2's background supervisor will drive.
14
+
15
+ mod artifact;
16
+ mod context;
17
+ mod coordinator;
18
+ mod engine;
19
+ mod events;
20
+ mod http;
21
+ mod lmstudio;
22
+ mod scheduler;
23
+ mod store;
24
+ mod tokenizer;
25
+ mod tools;
26
+ mod worker;
27
+
28
+ use std::path::PathBuf;
29
+ use std::sync::{Arc, Mutex};
30
+
31
+ use crate::artifact::ArtifactBlobStore;
32
+ use crate::context::ContextCompiler;
33
+ use crate::coordinator::{HttpCoordinator, LoopbackConfig, LoopbackCoordinator, PalwCoordinator};
34
+ use crate::engine::{EngineConfig, EngineHost};
35
+ use crate::http::Gateway;
36
+ use crate::scheduler::ResourceScheduler;
37
+ use crate::store::SessionStore;
38
+ use crate::tokenizer::TokenizerHost;
39
+ use crate::tools::{ToolRuntime, ToolSandboxPolicyV1};
40
+ use crate::worker::{EngineReplicaExecutor, WorkerConfig, WorkerStatus};
41
+
42
+ const DEFAULT_SYSTEM_PROMPT: &str = "You are a helpful assistant.";
43
+ pub(crate) const QWEN_EOS_ID: u32 = 248046; // <|im_end|> — the qi35_chat.py pin.
44
+
45
+ struct Args {
46
+ listen: String,
47
+ engine_binary: PathBuf,
48
+ gguf: PathBuf,
49
+ tables: PathBuf,
50
+ tokenizer_json: PathBuf,
51
+ db: PathBuf,
52
+ threads: u32,
53
+ max_new: u32,
54
+ max_context: usize,
55
+ metal_v3: bool,
56
+ system_prompt: String,
57
+ auth_token: String,
58
+ audit_key_file: Option<PathBuf>,
59
+ receipt_dir: Option<PathBuf>,
60
+ search_cmd: Option<PathBuf>,
61
+ // Phase 1
62
+ artifacts_dir: PathBuf,
63
+ workspace: PathBuf,
64
+ readable: Vec<PathBuf>,
65
+ allow_commands: Vec<PathBuf>,
66
+ tool_timeout_ms: u64,
67
+ max_file_chars: usize,
68
+ max_tool_chars: usize,
69
+ // Phase 2
70
+ palw_loopback: bool,
71
+ palw_coordinator: Option<String>,
72
+ palw_coordinator_token: Option<String>,
73
+ palw_provider: String,
74
+ palw_poll_ms: u64,
75
+ palw_no_self_replica: bool,
76
+ }
77
+
78
+ fn parse_args() -> Result<Args, String> {
79
+ let mut args = Args {
80
+ listen: "127.0.0.1:12346".into(),
81
+ engine_binary: PathBuf::from("docs/evidence/qi35_model"),
82
+ gguf: PathBuf::new(),
83
+ tables: PathBuf::new(),
84
+ tokenizer_json: PathBuf::new(),
85
+ db: PathBuf::from("palw-gateway-sessions.sqlite3"),
86
+ threads: 8,
87
+ max_new: 1024,
88
+ max_context: 8192,
89
+ metal_v3: true,
90
+ system_prompt: DEFAULT_SYSTEM_PROMPT.into(),
91
+ auth_token: std::env::var("PALW_GATEWAY_TOKEN").unwrap_or_default(),
92
+ audit_key_file: None,
93
+ receipt_dir: None,
94
+ search_cmd: None,
95
+ artifacts_dir: PathBuf::from("palw-artifacts"),
96
+ workspace: PathBuf::from("palw-workspace"),
97
+ readable: Vec::new(),
98
+ allow_commands: Vec::new(),
99
+ tool_timeout_ms: 30_000,
100
+ max_file_chars: 6_000,
101
+ max_tool_chars: 6_000,
102
+ palw_loopback: false,
103
+ palw_coordinator: None,
104
+ palw_coordinator_token: None,
105
+ palw_provider: "local-dev".into(),
106
+ palw_poll_ms: 2_000,
107
+ palw_no_self_replica: false,
108
+ };
109
+ let argv: Vec<String> = std::env::args().collect();
110
+ let mut i = 1;
111
+ while i < argv.len() {
112
+ let take = |i: &mut usize| -> Result<String, String> {
113
+ *i += 1;
114
+ argv.get(*i).cloned().ok_or_else(|| format!("{} needs a value", argv[*i - 1]))
115
+ };
116
+ match argv[i].as_str() {
117
+ "--listen" => args.listen = take(&mut i)?,
118
+ "--engine" => args.engine_binary = PathBuf::from(take(&mut i)?),
119
+ "--gguf" => args.gguf = PathBuf::from(take(&mut i)?),
120
+ "--tables" => args.tables = PathBuf::from(take(&mut i)?),
121
+ "--tokenizer" => args.tokenizer_json = PathBuf::from(take(&mut i)?),
122
+ "--db" => args.db = PathBuf::from(take(&mut i)?),
123
+ "--threads" => args.threads = take(&mut i)?.parse().map_err(|e| format!("--threads: {e}"))?,
124
+ "--max-new" => args.max_new = take(&mut i)?.parse().map_err(|e| format!("--max-new: {e}"))?,
125
+ "--max-context" => args.max_context = take(&mut i)?.parse().map_err(|e| format!("--max-context: {e}"))?,
126
+ "--no-metal" => args.metal_v3 = false,
127
+ "--system-prompt" => args.system_prompt = take(&mut i)?,
128
+ "--auth-token" => args.auth_token = take(&mut i)?,
129
+ "--audit-key-file" => args.audit_key_file = Some(PathBuf::from(take(&mut i)?)),
130
+ "--receipt-dir" => args.receipt_dir = Some(PathBuf::from(take(&mut i)?)),
131
+ "--search-cmd" => args.search_cmd = Some(PathBuf::from(take(&mut i)?)),
132
+ "--artifacts-dir" => args.artifacts_dir = PathBuf::from(take(&mut i)?),
133
+ "--workspace" => args.workspace = PathBuf::from(take(&mut i)?),
134
+ "--readable" => args.readable.push(PathBuf::from(take(&mut i)?)),
135
+ "--allow-command" => args.allow_commands.push(PathBuf::from(take(&mut i)?)),
136
+ "--tool-timeout-ms" => {
137
+ args.tool_timeout_ms = take(&mut i)?.parse().map_err(|e| format!("--tool-timeout-ms: {e}"))?
138
+ }
139
+ "--max-file-chars" => {
140
+ args.max_file_chars = take(&mut i)?.parse().map_err(|e| format!("--max-file-chars: {e}"))?
141
+ }
142
+ "--max-tool-chars" => {
143
+ args.max_tool_chars = take(&mut i)?.parse().map_err(|e| format!("--max-tool-chars: {e}"))?
144
+ }
145
+ "--palw-loopback" => args.palw_loopback = true,
146
+ "--palw-coordinator" => args.palw_coordinator = Some(take(&mut i)?),
147
+ "--palw-coordinator-token" => args.palw_coordinator_token = Some(take(&mut i)?),
148
+ "--palw-provider" => args.palw_provider = take(&mut i)?,
149
+ "--palw-poll-ms" => {
150
+ args.palw_poll_ms = take(&mut i)?.parse().map_err(|e| format!("--palw-poll-ms: {e}"))?
151
+ }
152
+ "--palw-no-self-replica" => args.palw_no_self_replica = true,
153
+ other => return Err(format!("unknown flag {other}")),
154
+ }
155
+ i += 1;
156
+ }
157
+ if args.gguf.as_os_str().is_empty() || args.tables.as_os_str().is_empty() || args.tokenizer_json.as_os_str().is_empty() {
158
+ return Err("required: --gguf <path> --tables <path> --tokenizer <tokenizer.json>".into());
159
+ }
160
+ Ok(args)
161
+ }
162
+
163
+ fn main() {
164
+ let args = match parse_args() {
165
+ Ok(a) => a,
166
+ Err(e) => {
167
+ eprintln!("palw-gateway: {e}");
168
+ eprintln!(
169
+ "usage: palw-gateway --gguf <gguf> --tables <tables> --tokenizer <tokenizer.json> \
170
+ [--engine <qi35_model>] [--listen 127.0.0.1:12346] [--db <sqlite>] [--threads N] \
171
+ [--max-new N] [--max-context N] [--no-metal] [--system-prompt S] [--auth-token T] \
172
+ [--audit-key-file <key>] [--receipt-dir <dir>] [--search-cmd <program>] \
173
+ [--artifacts-dir <dir>] [--workspace <dir>] \
174
+ [--readable <dir>]... [--allow-command </abs/path>]... [--tool-timeout-ms N] \
175
+ [--max-file-chars N] [--max-tool-chars N] \
176
+ [--palw-loopback] [--palw-coordinator http://host:port/v1/palw/loopback] \
177
+ [--palw-coordinator-token T] [--palw-provider ID] [--palw-poll-ms N] \
178
+ [--palw-no-self-replica]"
179
+ );
180
+ std::process::exit(2);
181
+ }
182
+ };
183
+
184
+ // Absolutize engine-facing paths: the engine child runs from ITS OWN directory (Metal dylib
185
+ // resolution), so relative gguf/tables paths would break.
186
+ let mut args = args;
187
+ for path in [&mut args.engine_binary, &mut args.gguf, &mut args.tables] {
188
+ if let Ok(canonical) = std::fs::canonicalize(&path) {
189
+ *path = canonical;
190
+ }
191
+ }
192
+
193
+ let tokenizer = match TokenizerHost::load(&args.tokenizer_json, QWEN_EOS_ID) {
194
+ Ok(t) => t,
195
+ Err(e) => {
196
+ eprintln!("palw-gateway: {e}");
197
+ std::process::exit(1);
198
+ }
199
+ };
200
+ eprintln!("[palw-gateway] tokenizer blake2b-256 {}", tokenizer.digest_hex);
201
+
202
+ let compiler = match ContextCompiler::new(
203
+ &tokenizer,
204
+ &args.system_prompt,
205
+ args.max_context,
206
+ args.max_new as usize,
207
+ args.max_file_chars,
208
+ args.max_tool_chars,
209
+ ) {
210
+ Ok(c) => c,
211
+ Err(e) => {
212
+ eprintln!("palw-gateway: context compiler: {e}");
213
+ std::process::exit(1);
214
+ }
215
+ };
216
+
217
+ let store = match SessionStore::open(&args.db) {
218
+ Ok(s) => s,
219
+ Err(e) => {
220
+ eprintln!("palw-gateway: session store: {e}");
221
+ std::process::exit(1);
222
+ }
223
+ };
224
+
225
+ let blobs = match ArtifactBlobStore::open(&args.artifacts_dir) {
226
+ Ok(b) => b,
227
+ Err(e) => {
228
+ eprintln!("palw-gateway: artifact store: {e}");
229
+ std::process::exit(1);
230
+ }
231
+ };
232
+
233
+ // Tool output cap 256 KiB, read cap 8 MiB — generous for logs/source, far below anything
234
+ // that could balloon the context compiler (which additionally clips per file/tool).
235
+ let sandbox = match ToolSandboxPolicyV1::new(
236
+ &args.workspace,
237
+ &args.readable,
238
+ &args.allow_commands,
239
+ args.tool_timeout_ms,
240
+ 256 * 1024,
241
+ 8 * 1024 * 1024,
242
+ ) {
243
+ Ok(p) => p,
244
+ Err(e) => {
245
+ eprintln!("palw-gateway: tool sandbox: {e}");
246
+ std::process::exit(1);
247
+ }
248
+ };
249
+ eprintln!(
250
+ "[palw-gateway] workspace {} | {} readable root(s) | {} allowlisted command(s)",
251
+ sandbox.workspace_root.display(),
252
+ sandbox.readable_roots.len(),
253
+ sandbox.allowed_commands.len()
254
+ );
255
+
256
+ let engine = Arc::new(Mutex::new(EngineHost::new(EngineConfig {
257
+ binary: args.engine_binary,
258
+ gguf: args.gguf,
259
+ tables: args.tables,
260
+ threads: args.threads,
261
+ max_new: args.max_new,
262
+ eos_id: QWEN_EOS_ID,
263
+ metal_v3: args.metal_v3,
264
+ audit_key_file: args.audit_key_file,
265
+ network_label: "MISAKA-Qwen-PALW-v1".into(),
266
+ })));
267
+ let store = Arc::new(Mutex::new(store));
268
+ let scheduler = Arc::new(ResourceScheduler::new());
269
+
270
+ // Phase 2: the loopback coordinator (served for other instances and/or used in-process)
271
+ // and the background worker. --palw-coordinator (a remote coordinator) takes precedence as
272
+ // the WORKER's counterparty; --palw-loopback additionally serves the protocol here.
273
+ let loopback = args.palw_loopback.then(|| {
274
+ Arc::new(LoopbackCoordinator::new(LoopbackConfig {
275
+ allow_self_replica: !args.palw_no_self_replica,
276
+ ..Default::default()
277
+ }))
278
+ });
279
+ let worker_coordinator: Option<Arc<dyn PalwCoordinator>> = match (&args.palw_coordinator, &loopback) {
280
+ (Some(url), _) => {
281
+ Some(Arc::new(HttpCoordinator::new(url, args.palw_coordinator_token.clone())) as Arc<dyn PalwCoordinator>)
282
+ }
283
+ (None, Some(l)) => Some(Arc::clone(l) as Arc<dyn PalwCoordinator>),
284
+ (None, None) => None,
285
+ };
286
+ let palw_status = worker_coordinator.as_ref().map(|_| Arc::new(WorkerStatus::default()));
287
+
288
+ let gateway = Arc::new(Gateway {
289
+ engine: Arc::clone(&engine),
290
+ store: Arc::clone(&store),
291
+ tokenizer,
292
+ compiler,
293
+ blobs,
294
+ tools: ToolRuntime::new(sandbox),
295
+ scheduler: Arc::clone(&scheduler),
296
+ cancels: Mutex::new(Default::default()),
297
+ model_name: "misaka-qi35".into(),
298
+ auth_token: args.auth_token,
299
+ receipt_dir: args.receipt_dir,
300
+ search_cmd: args.search_cmd,
301
+ palw_loopback: loopback,
302
+ palw_status: palw_status.clone(),
303
+ palw_provider: worker_coordinator.as_ref().map(|_| args.palw_provider.clone()),
304
+ });
305
+
306
+ let _worker = match (worker_coordinator, palw_status) {
307
+ (Some(coordinator), Some(status)) => Some(worker::spawn(
308
+ store,
309
+ Arc::clone(&scheduler),
310
+ coordinator,
311
+ Arc::new(EngineReplicaExecutor { engine, scheduler }),
312
+ WorkerConfig { provider_id: args.palw_provider, poll_ms: args.palw_poll_ms, max_new: args.max_new },
313
+ status,
314
+ )),
315
+ _ => None,
316
+ };
317
+
318
+ if let Err(e) = gateway.serve(&args.listen) {
319
+ eprintln!("palw-gateway: {e}");
320
+ std::process::exit(1);
321
+ }
322
+ }
palw-gateway/src/scheduler.rs ADDED
@@ -0,0 +1,311 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Local Resource Scheduler (design doc §14).
2
+ //!
3
+ //! One GPU, several masters: the user's own chat, background agents, other people's replica
4
+ //! jobs, auditor replays, maintenance. Phase 0 serialized engine access with a bare Mutex —
5
+ //! correct, but priority-blind. This module puts the doc's five-priority queue IN FRONT of the
6
+ //! engine: a foreground turn that arrives while a maintenance job is queued gets the next slot,
7
+ //! period.
8
+ //!
9
+ //! Two deliberate v1 properties, stated honestly:
10
+ //! * **No mid-generation preemption.** The engine has no stop channel (survey fact), so a
11
+ //! running job finishes; priority decides who goes NEXT. The worst-case foreground delay is
12
+ //! one background generation (bounded by `--max-new`).
13
+ //! * **Admission before acceptance** (the doc's no-show rule): `try_admit` answers "could this
14
+ //! job finish inside its SLA given the measured decode rate and the current queue?" BEFORE the
15
+ //! job is accepted. Refusing an assignment up front is healthier than taking it and
16
+ //! no-showing — the caller (Phase-2 replica worker) is expected to decline the assignment when
17
+ //! admission fails.
18
+ //!
19
+ //! The decode-rate estimate is a local EMA of observed generations (integer milli-tokens/sec —
20
+ //! telemetry, not consensus; nothing here is committed anywhere).
21
+
22
+ use std::sync::{Condvar, Mutex};
23
+
24
+ #[derive(Clone, Copy, Debug, PartialEq, Eq)]
25
+ pub enum JobClass {
26
+ /// Priority 1: the chat the user is looking at right now.
27
+ ForegroundChat,
28
+ /// Priority 2: agents the user started.
29
+ BackgroundAgent,
30
+ /// Priority 3: replica jobs with deadlines (Phase 2).
31
+ ReplicaJob,
32
+ /// Priority 4: auditor replays (Phase 2).
33
+ AuditorReplay,
34
+ /// Priority 5: conformance / indexing / artifact maintenance.
35
+ Maintenance,
36
+ }
37
+
38
+ impl JobClass {
39
+ pub const ALL: [JobClass; 5] =
40
+ [Self::ForegroundChat, Self::BackgroundAgent, Self::ReplicaJob, Self::AuditorReplay, Self::Maintenance];
41
+
42
+ fn priority(self) -> u8 {
43
+ match self {
44
+ Self::ForegroundChat => 1,
45
+ Self::BackgroundAgent => 2,
46
+ Self::ReplicaJob => 3,
47
+ Self::AuditorReplay => 4,
48
+ Self::Maintenance => 5,
49
+ }
50
+ }
51
+
52
+ pub fn index(self) -> usize {
53
+ self.priority() as usize - 1
54
+ }
55
+
56
+ pub fn as_str(self) -> &'static str {
57
+ match self {
58
+ Self::ForegroundChat => "foreground_chat",
59
+ Self::BackgroundAgent => "background_agent",
60
+ Self::ReplicaJob => "replica_job",
61
+ Self::AuditorReplay => "auditor_replay",
62
+ Self::Maintenance => "maintenance",
63
+ }
64
+ }
65
+
66
+ pub fn from_str(s: &str) -> Option<Self> {
67
+ Self::ALL.into_iter().find(|c| c.as_str() == s)
68
+ }
69
+ }
70
+
71
+ /// Conservative floor used until real generations teach the EMA — QI35 measures ~17 tok/s on
72
+ /// the M-series reference box; assuming 8 keeps early admission promises safe, not flattering.
73
+ const DEFAULT_MILLITOK_PER_SEC: u64 = 8_000;
74
+ /// EMA weight: new sample 1/8 — smooth enough to ignore one weird turn, fresh enough to track
75
+ /// thermal throttling within a few generations.
76
+ const EMA_SHIFT: u32 = 3;
77
+
78
+ struct SchedState {
79
+ busy: bool,
80
+ next_ticket: u64,
81
+ /// (priority, ticket) — the waiting set; the minimum is granted next (priority first, FIFO
82
+ /// inside a class via the monotone ticket).
83
+ waiting: Vec<(u8, u64)>,
84
+ waiting_by_class: [u32; 5],
85
+ ema_millitok_per_sec: u64,
86
+ completed_jobs: u64,
87
+ }
88
+
89
+ pub struct ResourceScheduler {
90
+ state: Mutex<SchedState>,
91
+ granted: Condvar,
92
+ /// Queue-depth caps per class (index by `JobClass::index`); 0 = unbounded (foreground).
93
+ queue_caps: [u32; 5],
94
+ }
95
+
96
+ /// RAII grant: hold it while talking to the engine; drop releases the slot to the best waiter.
97
+ pub struct EngineLease<'a> {
98
+ scheduler: &'a ResourceScheduler,
99
+ }
100
+
101
+ impl Drop for EngineLease<'_> {
102
+ fn drop(&mut self) {
103
+ let mut s = self.scheduler.state.lock().unwrap();
104
+ s.busy = false;
105
+ drop(s);
106
+ self.scheduler.granted.notify_all();
107
+ }
108
+ }
109
+
110
+ #[derive(Debug, Clone)]
111
+ pub struct SchedulerStatus {
112
+ pub busy: bool,
113
+ pub waiting_by_class: [u32; 5],
114
+ pub ema_millitok_per_sec: u64,
115
+ pub completed_jobs: u64,
116
+ }
117
+
118
+ impl ResourceScheduler {
119
+ pub fn new() -> Self {
120
+ Self {
121
+ state: Mutex::new(SchedState {
122
+ busy: false,
123
+ next_ticket: 0,
124
+ waiting: Vec::new(),
125
+ waiting_by_class: [0; 5],
126
+ ema_millitok_per_sec: DEFAULT_MILLITOK_PER_SEC,
127
+ completed_jobs: 0,
128
+ }),
129
+ granted: Condvar::new(),
130
+ // Foreground unbounded; modest bounded queues elsewhere — a deep background queue
131
+ // is exactly the "GPU busy for hours after the user left" failure §14 warns about.
132
+ queue_caps: [0, 4, 4, 4, 2],
133
+ }
134
+ }
135
+
136
+ /// Block until this job may use the engine. Grants strictly by (priority, arrival) order.
137
+ pub fn acquire(&self, class: JobClass) -> EngineLease<'_> {
138
+ let mut s = self.state.lock().unwrap();
139
+ let ticket = s.next_ticket;
140
+ s.next_ticket += 1;
141
+ let me = (class.priority(), ticket);
142
+ s.waiting.push(me);
143
+ s.waiting_by_class[class.index()] += 1;
144
+ loop {
145
+ let best = s.waiting.iter().copied().min().expect("self is waiting");
146
+ if !s.busy && best == me {
147
+ s.waiting.retain(|&w| w != me);
148
+ s.waiting_by_class[class.index()] -= 1;
149
+ s.busy = true;
150
+ return EngineLease { scheduler: self };
151
+ }
152
+ s = self.granted.wait(s).unwrap();
153
+ }
154
+ }
155
+
156
+ /// SLA admission (§14): would a job of `estimated_output_tokens` finish within
157
+ /// `deadline_ms`, given the measured decode rate and everything already ahead of it?
158
+ /// Foreground is always admitted (the user is waiting; refusing them helps nobody).
159
+ pub fn try_admit(&self, class: JobClass, estimated_output_tokens: u64, deadline_ms: u64) -> Result<(), String> {
160
+ if class == JobClass::ForegroundChat {
161
+ return Ok(());
162
+ }
163
+ let s = self.state.lock().unwrap();
164
+ let cap = self.queue_caps[class.index()];
165
+ if cap != 0 && s.waiting_by_class[class.index()] >= cap {
166
+ return Err(format!("{} queue is full ({cap})", class.as_str()));
167
+ }
168
+ // Own runtime at the measured rate, then scale by how many jobs sit in front (each may
169
+ // be a full generation) — a coarse but SAFE overestimate; refusing borderline work is
170
+ // the design's stated preference over no-shows.
171
+ let rate = s.ema_millitok_per_sec.max(1);
172
+ let own_ms = estimated_output_tokens.saturating_mul(1_000_000) / rate;
173
+ let ahead = s.waiting.len() as u64 + u64::from(s.busy);
174
+ let projected_ms = own_ms.saturating_mul(ahead + 1);
175
+ if projected_ms > deadline_ms {
176
+ return Err(format!(
177
+ "projected {projected_ms}ms (rate {}mtok/s, {ahead} ahead) exceeds the {deadline_ms}ms deadline",
178
+ rate
179
+ ));
180
+ }
181
+ Ok(())
182
+ }
183
+
184
+ /// Feed the decode-rate EMA from a finished generation.
185
+ pub fn record_generation(&self, output_tokens: u64, elapsed_ms: u64) {
186
+ if output_tokens == 0 || elapsed_ms == 0 {
187
+ return;
188
+ }
189
+ let sample = output_tokens.saturating_mul(1_000_000) / elapsed_ms;
190
+ let mut s = self.state.lock().unwrap();
191
+ let ema = s.ema_millitok_per_sec;
192
+ s.ema_millitok_per_sec = ema - (ema >> EMA_SHIFT) + (sample >> EMA_SHIFT);
193
+ s.completed_jobs += 1;
194
+ }
195
+
196
+ pub fn status(&self) -> SchedulerStatus {
197
+ let s = self.state.lock().unwrap();
198
+ SchedulerStatus {
199
+ busy: s.busy,
200
+ waiting_by_class: s.waiting_by_class,
201
+ ema_millitok_per_sec: s.ema_millitok_per_sec,
202
+ completed_jobs: s.completed_jobs,
203
+ }
204
+ }
205
+ }
206
+
207
+ #[cfg(test)]
208
+ mod tests {
209
+ use super::*;
210
+ use std::sync::Arc;
211
+ use std::sync::mpsc;
212
+ use std::time::{Duration, Instant};
213
+
214
+ fn wait_until(pred: impl Fn() -> bool) {
215
+ let deadline = Instant::now() + Duration::from_secs(5);
216
+ while !pred() {
217
+ assert!(Instant::now() < deadline, "condition not reached in 5s");
218
+ std::thread::sleep(Duration::from_millis(5));
219
+ }
220
+ }
221
+
222
+ #[test]
223
+ fn foreground_overtakes_earlier_background() {
224
+ let sched = Arc::new(ResourceScheduler::new());
225
+ let held = sched.acquire(JobClass::Maintenance);
226
+
227
+ let (tx, rx) = mpsc::channel::<&'static str>();
228
+ let s2 = Arc::clone(&sched);
229
+ let tx2 = tx.clone();
230
+ let h_maint = std::thread::spawn(move || {
231
+ let lease = s2.acquire(JobClass::Maintenance);
232
+ tx2.send("maintenance").unwrap();
233
+ drop(lease);
234
+ });
235
+ wait_until(|| sched.status().waiting_by_class[JobClass::Maintenance.index()] == 1);
236
+
237
+ let s3 = Arc::clone(&sched);
238
+ let h_fg = std::thread::spawn(move || {
239
+ let lease = s3.acquire(JobClass::ForegroundChat);
240
+ tx.send("foreground").unwrap();
241
+ drop(lease);
242
+ });
243
+ wait_until(|| sched.status().waiting_by_class[JobClass::ForegroundChat.index()] == 1);
244
+
245
+ drop(held);
246
+ // The maintenance job queued FIRST, but the foreground turn is granted first.
247
+ assert_eq!(rx.recv_timeout(Duration::from_secs(5)).unwrap(), "foreground");
248
+ assert_eq!(rx.recv_timeout(Duration::from_secs(5)).unwrap(), "maintenance");
249
+ h_maint.join().unwrap();
250
+ h_fg.join().unwrap();
251
+ }
252
+
253
+ #[test]
254
+ fn fifo_within_a_class() {
255
+ let sched = Arc::new(ResourceScheduler::new());
256
+ let held = sched.acquire(JobClass::ForegroundChat);
257
+ let (tx, rx) = mpsc::channel::<u32>();
258
+ let mut handles = Vec::new();
259
+ for i in 0..3u32 {
260
+ let s = Arc::clone(&sched);
261
+ let tx = tx.clone();
262
+ handles.push(std::thread::spawn(move || {
263
+ let lease = s.acquire(JobClass::BackgroundAgent);
264
+ tx.send(i).unwrap();
265
+ drop(lease);
266
+ }));
267
+ // Serialize arrival so tickets are ordered 0,1,2.
268
+ wait_until(|| s_waiting(&sched) == i + 1);
269
+ }
270
+ drop(held);
271
+ let order: Vec<u32> = (0..3).map(|_| rx.recv_timeout(Duration::from_secs(5)).unwrap()).collect();
272
+ assert_eq!(order, vec![0, 1, 2]);
273
+ for h in handles {
274
+ h.join().unwrap();
275
+ }
276
+
277
+ fn s_waiting(s: &ResourceScheduler) -> u32 {
278
+ s.status().waiting_by_class[JobClass::BackgroundAgent.index()]
279
+ }
280
+ }
281
+
282
+ #[test]
283
+ fn admission_enforces_deadline_and_caps() {
284
+ let sched = ResourceScheduler::new();
285
+ // Default EMA 8 tok/s ⇒ 800 tokens ≈ 100s; a 5s deadline must refuse, a 200s one admits.
286
+ assert!(sched.try_admit(JobClass::ReplicaJob, 800, 5_000).is_err());
287
+ assert!(sched.try_admit(JobClass::ReplicaJob, 800, 200_000).is_ok());
288
+ // Foreground is never refused.
289
+ assert!(sched.try_admit(JobClass::ForegroundChat, u64::MAX, 1).is_ok());
290
+
291
+ // Busy engine counts as one job ahead: same estimate now needs double the time.
292
+ let lease = sched.acquire(JobClass::ForegroundChat);
293
+ assert!(sched.try_admit(JobClass::ReplicaJob, 800, 150_000).is_err(), "one ahead doubles the projection");
294
+ assert!(sched.try_admit(JobClass::ReplicaJob, 800, 250_000).is_ok());
295
+ drop(lease);
296
+ }
297
+
298
+ #[test]
299
+ fn ema_tracks_observed_rate() {
300
+ let sched = ResourceScheduler::new();
301
+ let before = sched.status().ema_millitok_per_sec;
302
+ // 100 tokens in 1000ms = 100 tok/s — far above the 8 tok/s floor; EMA must move up.
303
+ sched.record_generation(100, 1_000);
304
+ let after = sched.status().ema_millitok_per_sec;
305
+ assert!(after > before, "{after} !> {before}");
306
+ // Degenerate samples are ignored.
307
+ sched.record_generation(0, 100);
308
+ sched.record_generation(100, 0);
309
+ assert_eq!(sched.status().completed_jobs, 1);
310
+ }
311
+ }
palw-gateway/src/store.rs ADDED
@@ -0,0 +1,1469 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! The Conversation Session Layer store — the thing the Explore survey confirmed does not exist
2
+ //! anywhere in the repo yet.
3
+ //!
4
+ //! Design constraints carried in from the survey and the frozen product doc:
5
+ //! * **Token-id-stable history is the schema's core invariant.** Every turn stores the exact
6
+ //! `prompt_ids` it was generated from and the exact `output_ids` the engine returned. The next
7
+ //! turn's context is built by CONCATENATING stored ids — never by detokenize→retokenize, which
8
+ //! is what silently defeated the engine's prefix cache in the current stack.
9
+ //! * **Display text is separate from model history.** The receipt footer / search-source block
10
+ //! polluting the next prompt was a live bug; here `display_text` (what the UI shows) and the
11
+ //! ids (what the model sees) are different columns, so decoration can never leak into context.
12
+ //! * **Dual-head from day one** (`local_head` / `certified_head`), and a `TurnVerificationStatus`
13
+ //! column, so UX Phase 2 is a writer of existing columns, not a migration.
14
+ //! * Branching = a turn's `parent_turn_id` pointing at any earlier turn; `local_head` names the
15
+ //! active leaf. "Edit message N" is: append a new turn whose parent is N-1's turn and move the
16
+ //! head — history is immutable, exactly like the design doc's message-branch model.
17
+ //!
18
+ //! Schema v2 (UX Phase 1) adds the product-doc layers that live NEXT TO the turn tree:
19
+ //! projects/workspaces (§10), long-term memory with provenance + soft delete (§12), artifact
20
+ //! versioning (§8/§9), the tool-call ledger with the exactly-once side-effect guard (§6), and
21
+ //! two turn columns — `user_block_text` (the EXACT text encoded into the turn's user block:
22
+ //! attachments + tool results + user text, so history recompilation is id-stable even for turns
23
+ //! that carried context material) and `context_meta_json` (what the compiler actually included:
24
+ //! memory snapshot root, chunk selections — the future receipt commitment surface).
25
+ //!
26
+ //! Verification pipeline writes (Phase 2's settlement clock) also live here:
27
+ //! `set_verification_status` enforces the legal transition machine, recomputes `certified_head`
28
+ //! (deepest turn with an unbroken certified prefix from the root), and on `Mismatch` cascades
29
+ //! `MintIneligible` to every descendant — the §4 semantics: local conversation survives, the
30
+ //! reward dependency chain does not.
31
+ //!
32
+ //! Hardening copies the `runtime-palw/state_store.rs` pattern where it is cheap and load-bearing
33
+ //! (application_id, WAL, synchronous=FULL, foreign keys). The consensus ledger itself is NOT
34
+ //! reused — the survey's verdict — because chat state does not belong in a protocol ledger.
35
+
36
+ use std::path::Path;
37
+
38
+ use rusqlite::{Connection, OpenFlags, params};
39
+
40
+ use crate::events::{ConversationHeadsV1, TurnPrivacyMode, TurnVerificationStatus};
41
+
42
+ /// "PALW" ^ 0x20202020 — distinct from the protocol ledger's 0x5041_4c57 so the two databases
43
+ /// can never be mistaken for one another by application_id.
44
+ const APPLICATION_ID: i64 = 0x7061_6c77; // "palw"
45
+ const SCHEMA_VERSION: i64 = 4;
46
+
47
+ pub struct SessionStore {
48
+ conn: Connection,
49
+ }
50
+
51
+ #[derive(Debug)]
52
+ #[allow(dead_code)] // read-model: fields mirror the row even where the gateway does not read them yet
53
+ pub struct StoredTurn {
54
+ pub turn_id: String,
55
+ pub conversation_id: String,
56
+ pub parent_turn_id: Option<String>,
57
+ pub role_user_text: String,
58
+ /// The exact text the compiler encoded into this turn's user block (user text + rendered
59
+ /// attachments/tool results). `None` on rows written by schema v1 ⇒ the block was the raw
60
+ /// user text; `user_block()` folds that fallback so history recompilation stays id-stable
61
+ /// across the migration.
62
+ pub user_block_text: Option<String>,
63
+ pub context_meta_json: Option<String>,
64
+ pub display_text: String,
65
+ pub prompt_ids: Vec<u32>,
66
+ pub output_ids: Vec<u32>,
67
+ pub stop_reason: Option<String>,
68
+ pub verification_status: TurnVerificationStatus,
69
+ pub privacy_mode: TurnPrivacyMode,
70
+ /// The engine's `RCPT` line (present when the engine ran with an audit key) — attached to
71
+ /// the A-commit by the Phase-2 worker.
72
+ pub receipt_json: Option<String>,
73
+ /// Engine execution roots (`RuntimeRootsV1` as JSON) captured at completion — the extra
74
+ /// commitments the A-commit carries for structural replica matching.
75
+ pub runtime_roots_json: Option<String>,
76
+ /// The committed `live_search_bundle_v1` when this turn ran a search. Fixed at
77
+ /// `begin_turn` because its rendered evidence is inside the compiled user block — the
78
+ /// prompt commitment covers it transitively, exactly like the legacy server.
79
+ pub search_bundle_json: Option<String>,
80
+ pub created_ms: i64,
81
+ }
82
+
83
+ impl StoredTurn {
84
+ pub fn user_block(&self) -> &str {
85
+ self.user_block_text.as_deref().unwrap_or(&self.role_user_text)
86
+ }
87
+ }
88
+
89
+ #[derive(Debug)]
90
+ #[allow(dead_code)]
91
+ pub struct ConversationRow {
92
+ pub conversation_id: String,
93
+ pub title: String,
94
+ pub project_id: Option<String>,
95
+ pub heads: ConversationHeadsV1,
96
+ pub created_ms: i64,
97
+ pub updated_ms: i64,
98
+ }
99
+
100
+ #[derive(Debug, Clone)]
101
+ #[allow(dead_code)]
102
+ pub struct ProjectRow {
103
+ pub project_id: String,
104
+ pub name: String,
105
+ pub instructions: String,
106
+ pub created_ms: i64,
107
+ pub updated_ms: i64,
108
+ }
109
+
110
+ #[derive(Debug, Clone)]
111
+ pub struct MemoryEntryRow {
112
+ pub memory_id: String,
113
+ pub namespace: String,
114
+ pub content: String,
115
+ pub source_turn_id: Option<String>,
116
+ pub created_sequence: i64,
117
+ pub created_ms: i64,
118
+ }
119
+
120
+ #[derive(Debug, Clone)]
121
+ pub struct ArtifactMeta {
122
+ pub artifact_id: String,
123
+ pub content_hash: String,
124
+ pub mime_type: String,
125
+ pub byte_length: i64,
126
+ pub display_name: String,
127
+ pub parent_artifact_id: Option<String>,
128
+ pub version: i64,
129
+ pub parser_policy: String,
130
+ pub project_id: Option<String>,
131
+ pub created_by_turn: Option<String>,
132
+ pub created_ms: i64,
133
+ }
134
+
135
+ #[derive(Debug, Clone)]
136
+ pub struct ToolCallRow {
137
+ pub call_id: String,
138
+ pub tool_name: String,
139
+ pub mode: String,
140
+ pub arguments_json: String,
141
+ pub status: String,
142
+ pub result_artifact_id: Option<String>,
143
+ pub result_preview: String,
144
+ pub error: Option<String>,
145
+ pub conversation_id: Option<String>,
146
+ pub created_ms: i64,
147
+ pub executed_ms: Option<i64>,
148
+ }
149
+
150
+ pub struct NewTurn<'a> {
151
+ pub turn_id: &'a str,
152
+ pub conversation_id: &'a str,
153
+ pub parent_turn_id: Option<&'a str>,
154
+ pub role_user_text: &'a str,
155
+ pub user_block_text: &'a str,
156
+ pub context_meta_json: Option<&'a str>,
157
+ pub prompt_ids: &'a [u32],
158
+ pub privacy_mode: TurnPrivacyMode,
159
+ pub search_bundle_json: Option<&'a str>,
160
+ pub now_ms: i64,
161
+ }
162
+
163
+ /// Completion payload for a turn (mirrors `NewTurn` — struct args, not positional).
164
+ pub struct CompletedTurn<'a> {
165
+ pub turn_id: &'a str,
166
+ pub output_ids: &'a [u32],
167
+ pub display_text: &'a str,
168
+ pub stop_reason: &'a str,
169
+ pub status: TurnVerificationStatus,
170
+ pub receipt_json: Option<&'a str>,
171
+ pub runtime_roots_json: Option<&'a str>,
172
+ }
173
+
174
+ /// What a settlement-clock write changed, so the supervisor/UI can react without re-reading.
175
+ #[derive(Debug)]
176
+ pub struct VerificationOutcome {
177
+ pub status: TurnVerificationStatus,
178
+ /// Descendant turns forced to `MintIneligible` by a mismatch cascade (empty otherwise).
179
+ pub cascaded: Vec<String>,
180
+ pub certified_head: Option<String>,
181
+ }
182
+
183
+ fn ids_to_blob(ids: &[u32]) -> Vec<u8> {
184
+ let mut out = Vec::with_capacity(ids.len() * 4);
185
+ for id in ids {
186
+ out.extend_from_slice(&id.to_le_bytes());
187
+ }
188
+ out
189
+ }
190
+
191
+ fn blob_to_ids(blob: &[u8]) -> Vec<u32> {
192
+ blob.chunks_exact(4).map(|c| u32::from_le_bytes([c[0], c[1], c[2], c[3]])).collect()
193
+ }
194
+
195
+ /// The settlement pipeline's legal moves (§"最重要" + §4). `ReplicaMatched → Certified` is legal
196
+ /// because auditing is SAMPLED — a job the auditor lottery skips certifies straight from the
197
+ /// match. `MintIneligible` is reachable from any live state (policy exit), never from a terminal
198
+ /// one. Nothing ever leaves `Mismatch`, `MintIneligible`, or `Matured`.
199
+ fn verification_transition_legal(from: TurnVerificationStatus, to: TurnVerificationStatus) -> bool {
200
+ use TurnVerificationStatus::*;
201
+ matches!(
202
+ (from, to),
203
+ (LocalComplete, ReplicaPending)
204
+ | (ReplicaPending, ReplicaMatched)
205
+ | (ReplicaPending, Mismatch)
206
+ | (ReplicaMatched, AuditPending)
207
+ | (ReplicaMatched, Certified)
208
+ | (ReplicaMatched, Mismatch)
209
+ | (AuditPending, Certified)
210
+ | (AuditPending, Mismatch)
211
+ | (Certified, Matured)
212
+ ) || (to == MintIneligible && !matches!(from, Mismatch | MintIneligible | Matured))
213
+ }
214
+
215
+ fn branch_history_conn(conn: &Connection, leaf: &str) -> Result<Vec<StoredTurn>, String> {
216
+ let mut chain = Vec::new();
217
+ let mut cursor = Some(leaf.to_string());
218
+ while let Some(id) = cursor {
219
+ let turn = get_turn_conn(conn, &id)?;
220
+ cursor = turn.parent_turn_id.clone();
221
+ chain.push(turn);
222
+ if chain.len() > 100_000 {
223
+ return Err("turn chain exceeds 100k — refusing (corrupt parent loop?)".into());
224
+ }
225
+ }
226
+ chain.reverse();
227
+ Ok(chain)
228
+ }
229
+
230
+ const TURN_COLUMNS: &str = "turn_id, conversation_id, parent_turn_id, role_user_text, display_text,
231
+ prompt_ids, output_ids, stop_reason, verification_status, privacy_mode, created_ms,
232
+ user_block_text, context_meta_json, receipt_json, runtime_roots_json, search_bundle_json";
233
+
234
+ fn turn_from_row(r: &rusqlite::Row) -> rusqlite::Result<StoredTurn> {
235
+ let prompt: Vec<u8> = r.get(5)?;
236
+ let output: Vec<u8> = r.get(6)?;
237
+ let status: String = r.get(8)?;
238
+ let privacy: String = r.get(9)?;
239
+ Ok(StoredTurn {
240
+ turn_id: r.get(0)?,
241
+ conversation_id: r.get(1)?,
242
+ parent_turn_id: r.get(2)?,
243
+ role_user_text: r.get(3)?,
244
+ display_text: r.get(4)?,
245
+ prompt_ids: blob_to_ids(&prompt),
246
+ output_ids: blob_to_ids(&output),
247
+ stop_reason: r.get(7)?,
248
+ verification_status: TurnVerificationStatus::from_str(&status)
249
+ .unwrap_or(TurnVerificationStatus::MintIneligible),
250
+ privacy_mode: TurnPrivacyMode::from_str(&privacy).unwrap_or(TurnPrivacyMode::LocalOnly),
251
+ created_ms: r.get(10)?,
252
+ user_block_text: r.get(11)?,
253
+ context_meta_json: r.get(12)?,
254
+ receipt_json: r.get(13)?,
255
+ runtime_roots_json: r.get(14)?,
256
+ search_bundle_json: r.get(15)?,
257
+ })
258
+ }
259
+
260
+ fn get_turn_conn(conn: &Connection, turn_id: &str) -> Result<StoredTurn, String> {
261
+ conn.query_row(&format!("SELECT {TURN_COLUMNS} FROM turns WHERE turn_id = ?1"), params![turn_id], turn_from_row)
262
+ .map_err(|e| format!("turn {turn_id}: {e}"))
263
+ }
264
+
265
+ /// Recompute `certified_head` = the DEEPEST turn on the local-head branch whose whole prefix
266
+ /// (root..=turn) is `Certified`/`Matured`. Replica/audit results land out of order, so a later
267
+ /// turn certifying before an earlier one must NOT advance the head past the gap — re-issue from
268
+ /// certified head (§4) only makes sense if everything before it is settled.
269
+ fn recompute_certified_head(conn: &Connection, conversation_id: &str, now_ms: i64) -> Result<Option<String>, String> {
270
+ let local_head: Option<String> = conn
271
+ .query_row("SELECT local_head FROM conversations WHERE conversation_id = ?1", params![conversation_id], |r| r.get(0))
272
+ .map_err(|e| format!("conversation {conversation_id}: {e}"))?;
273
+ let mut head: Option<String> = None;
274
+ if let Some(leaf) = local_head {
275
+ for turn in branch_history_conn(conn, &leaf)? {
276
+ match turn.verification_status {
277
+ TurnVerificationStatus::Certified | TurnVerificationStatus::Matured => head = Some(turn.turn_id),
278
+ _ => break,
279
+ }
280
+ }
281
+ }
282
+ conn.execute(
283
+ "UPDATE conversations SET certified_head = ?2, updated_ms = ?3 WHERE conversation_id = ?1",
284
+ params![conversation_id, head, now_ms],
285
+ )
286
+ .map_err(|e| e.to_string())?;
287
+ Ok(head)
288
+ }
289
+
290
+ impl SessionStore {
291
+ pub fn open(path: &Path) -> Result<Self, String> {
292
+ if let Some(parent) = path.parent() {
293
+ std::fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?;
294
+ }
295
+ let conn = Connection::open_with_flags(
296
+ path,
297
+ OpenFlags::SQLITE_OPEN_READ_WRITE | OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_NO_MUTEX,
298
+ )
299
+ .map_err(|e| format!("open {}: {e}", path.display()))?;
300
+
301
+ conn.pragma_update(None, "journal_mode", "WAL").map_err(|e| e.to_string())?;
302
+ conn.pragma_update(None, "synchronous", "FULL").map_err(|e| e.to_string())?;
303
+ conn.pragma_update(None, "foreign_keys", "ON").map_err(|e| e.to_string())?;
304
+
305
+ let existing_app_id: i64 = conn.query_row("PRAGMA application_id", [], |r| r.get(0)).map_err(|e| e.to_string())?;
306
+ if existing_app_id == 0 {
307
+ conn.pragma_update(None, "application_id", APPLICATION_ID).map_err(|e| e.to_string())?;
308
+ } else if existing_app_id != APPLICATION_ID {
309
+ return Err(format!(
310
+ "database {} has application_id {existing_app_id:#x}; refusing to open a non-session database",
311
+ path.display()
312
+ ));
313
+ }
314
+
315
+ let user_version: i64 = conn.query_row("PRAGMA user_version", [], |r| r.get(0)).map_err(|e| e.to_string())?;
316
+ match user_version {
317
+ 0 => {
318
+ Self::create_schema(&conn)?;
319
+ conn.pragma_update(None, "user_version", SCHEMA_VERSION).map_err(|e| e.to_string())?;
320
+ }
321
+ 1 => {
322
+ Self::migrate_v1_to_v2(&conn)?;
323
+ Self::migrate_v2_to_v3(&conn)?;
324
+ Self::migrate_v3_to_v4(&conn)?;
325
+ }
326
+ 2 => {
327
+ Self::migrate_v2_to_v3(&conn)?;
328
+ Self::migrate_v3_to_v4(&conn)?;
329
+ }
330
+ 3 => {
331
+ Self::migrate_v3_to_v4(&conn)?;
332
+ }
333
+ SCHEMA_VERSION => {}
334
+ v => return Err(format!("unsupported session schema version {v} (this build speaks {SCHEMA_VERSION})")),
335
+ }
336
+ Ok(Self { conn })
337
+ }
338
+
339
+ fn create_schema(conn: &Connection) -> Result<(), String> {
340
+ conn.execute_batch(
341
+ r#"
342
+ CREATE TABLE IF NOT EXISTS projects (
343
+ project_id TEXT PRIMARY KEY,
344
+ name TEXT NOT NULL,
345
+ instructions TEXT NOT NULL DEFAULT '',
346
+ created_ms INTEGER NOT NULL,
347
+ updated_ms INTEGER NOT NULL
348
+ );
349
+ CREATE TABLE IF NOT EXISTS conversations (
350
+ conversation_id TEXT PRIMARY KEY,
351
+ title TEXT NOT NULL DEFAULT '',
352
+ project_id TEXT REFERENCES projects(project_id),
353
+ local_head TEXT,
354
+ certified_head TEXT,
355
+ created_ms INTEGER NOT NULL,
356
+ updated_ms INTEGER NOT NULL
357
+ );
358
+ CREATE TABLE IF NOT EXISTS turns (
359
+ turn_id TEXT PRIMARY KEY,
360
+ conversation_id TEXT NOT NULL REFERENCES conversations(conversation_id),
361
+ parent_turn_id TEXT REFERENCES turns(turn_id),
362
+ role_user_text TEXT NOT NULL,
363
+ user_block_text TEXT,
364
+ context_meta_json TEXT,
365
+ display_text TEXT NOT NULL DEFAULT '',
366
+ prompt_ids BLOB NOT NULL,
367
+ output_ids BLOB NOT NULL DEFAULT x'',
368
+ stop_reason TEXT,
369
+ verification_status TEXT NOT NULL,
370
+ privacy_mode TEXT NOT NULL,
371
+ receipt_json TEXT,
372
+ runtime_roots_json TEXT,
373
+ search_bundle_json TEXT,
374
+ created_ms INTEGER NOT NULL
375
+ );
376
+ CREATE INDEX IF NOT EXISTS turns_by_conversation ON turns(conversation_id, created_ms);
377
+ CREATE INDEX IF NOT EXISTS turns_by_parent ON turns(parent_turn_id);
378
+ CREATE TABLE IF NOT EXISTS artifacts (
379
+ artifact_id TEXT PRIMARY KEY,
380
+ content_hash TEXT NOT NULL,
381
+ mime_type TEXT NOT NULL,
382
+ byte_length INTEGER NOT NULL,
383
+ display_name TEXT NOT NULL DEFAULT '',
384
+ parent_artifact_id TEXT REFERENCES artifacts(artifact_id),
385
+ version INTEGER NOT NULL DEFAULT 1,
386
+ parser_policy TEXT NOT NULL DEFAULT '',
387
+ project_id TEXT REFERENCES projects(project_id),
388
+ created_by_turn TEXT REFERENCES turns(turn_id),
389
+ created_ms INTEGER NOT NULL
390
+ );
391
+ CREATE INDEX IF NOT EXISTS artifacts_by_project ON artifacts(project_id, created_ms);
392
+ CREATE TABLE IF NOT EXISTS memory_entries (
393
+ memory_id TEXT PRIMARY KEY,
394
+ namespace TEXT NOT NULL,
395
+ content TEXT NOT NULL,
396
+ source_turn_id TEXT REFERENCES turns(turn_id),
397
+ created_sequence INTEGER NOT NULL,
398
+ deleted_sequence INTEGER,
399
+ created_ms INTEGER NOT NULL
400
+ );
401
+ CREATE INDEX IF NOT EXISTS memory_by_namespace ON memory_entries(namespace, created_sequence);
402
+ CREATE TABLE IF NOT EXISTS tool_calls (
403
+ call_id TEXT PRIMARY KEY,
404
+ tool_name TEXT NOT NULL,
405
+ mode TEXT NOT NULL,
406
+ arguments_json TEXT NOT NULL,
407
+ status TEXT NOT NULL,
408
+ result_artifact_id TEXT REFERENCES artifacts(artifact_id),
409
+ result_preview TEXT NOT NULL DEFAULT '',
410
+ error TEXT,
411
+ conversation_id TEXT REFERENCES conversations(conversation_id),
412
+ created_ms INTEGER NOT NULL,
413
+ executed_ms INTEGER
414
+ );
415
+ "#,
416
+ )
417
+ .map_err(|e| e.to_string())
418
+ }
419
+
420
+ /// v1 → v2: additive only (§18's migration rule — existing turns stay readable, their
421
+ /// `user_block_text` NULL means "the user block was the raw user text", which
422
+ /// `StoredTurn::user_block()` folds back so recompiled prompts are byte-identical to what
423
+ /// a v1 gateway produced for the same history.
424
+ fn migrate_v1_to_v2(conn: &Connection) -> Result<(), String> {
425
+ conn.execute_batch(
426
+ r#"
427
+ BEGIN;
428
+ ALTER TABLE conversations ADD COLUMN project_id TEXT REFERENCES projects(project_id);
429
+ ALTER TABLE turns ADD COLUMN user_block_text TEXT;
430
+ ALTER TABLE turns ADD COLUMN context_meta_json TEXT;
431
+ ALTER TABLE artifacts ADD COLUMN display_name TEXT NOT NULL DEFAULT '';
432
+ ALTER TABLE artifacts ADD COLUMN parent_artifact_id TEXT REFERENCES artifacts(artifact_id);
433
+ ALTER TABLE artifacts ADD COLUMN version INTEGER NOT NULL DEFAULT 1;
434
+ ALTER TABLE artifacts ADD COLUMN parser_policy TEXT NOT NULL DEFAULT '';
435
+ ALTER TABLE artifacts ADD COLUMN project_id TEXT REFERENCES projects(project_id);
436
+ CREATE TABLE projects (
437
+ project_id TEXT PRIMARY KEY,
438
+ name TEXT NOT NULL,
439
+ instructions TEXT NOT NULL DEFAULT '',
440
+ created_ms INTEGER NOT NULL,
441
+ updated_ms INTEGER NOT NULL
442
+ );
443
+ CREATE TABLE memory_entries (
444
+ memory_id TEXT PRIMARY KEY,
445
+ namespace TEXT NOT NULL,
446
+ content TEXT NOT NULL,
447
+ source_turn_id TEXT REFERENCES turns(turn_id),
448
+ created_sequence INTEGER NOT NULL,
449
+ deleted_sequence INTEGER,
450
+ created_ms INTEGER NOT NULL
451
+ );
452
+ CREATE INDEX memory_by_namespace ON memory_entries(namespace, created_sequence);
453
+ CREATE TABLE tool_calls (
454
+ call_id TEXT PRIMARY KEY,
455
+ tool_name TEXT NOT NULL,
456
+ mode TEXT NOT NULL,
457
+ arguments_json TEXT NOT NULL,
458
+ status TEXT NOT NULL,
459
+ result_artifact_id TEXT REFERENCES artifacts(artifact_id),
460
+ result_preview TEXT NOT NULL DEFAULT '',
461
+ error TEXT,
462
+ conversation_id TEXT REFERENCES conversations(conversation_id),
463
+ created_ms INTEGER NOT NULL,
464
+ executed_ms INTEGER
465
+ );
466
+ CREATE INDEX IF NOT EXISTS turns_by_parent ON turns(parent_turn_id);
467
+ CREATE INDEX IF NOT EXISTS artifacts_by_project ON artifacts(project_id, created_ms);
468
+ PRAGMA user_version = 2;
469
+ COMMIT;
470
+ "#,
471
+ )
472
+ .map_err(|e| format!("migrate session schema v1→v2: {e}"))
473
+ }
474
+
475
+ /// v2 → v3: one additive column — the engine execution roots captured per turn.
476
+ fn migrate_v2_to_v3(conn: &Connection) -> Result<(), String> {
477
+ conn.execute_batch(
478
+ r#"
479
+ BEGIN;
480
+ ALTER TABLE turns ADD COLUMN runtime_roots_json TEXT;
481
+ PRAGMA user_version = 3;
482
+ COMMIT;
483
+ "#,
484
+ )
485
+ .map_err(|e| format!("migrate session schema v2→v3: {e}"))
486
+ }
487
+
488
+ /// v3 → v4: one additive column — the committed live-search bundle. It is fixed BEFORE
489
+ /// generation (the evidence is part of the compiled user block, hence of the prompt
490
+ /// commitment), so it lands on `begin_turn`, not completion.
491
+ fn migrate_v3_to_v4(conn: &Connection) -> Result<(), String> {
492
+ conn.execute_batch(
493
+ r#"
494
+ BEGIN;
495
+ ALTER TABLE turns ADD COLUMN search_bundle_json TEXT;
496
+ PRAGMA user_version = 4;
497
+ COMMIT;
498
+ "#,
499
+ )
500
+ .map_err(|e| format!("migrate session schema v3→v4: {e}"))
501
+ }
502
+
503
+ // ---- conversations ------------------------------------------------------------------
504
+
505
+ pub fn create_conversation(
506
+ &self,
507
+ conversation_id: &str,
508
+ title: &str,
509
+ project_id: Option<&str>,
510
+ now_ms: i64,
511
+ ) -> Result<(), String> {
512
+ self.conn
513
+ .execute(
514
+ "INSERT INTO conversations (conversation_id, title, project_id, created_ms, updated_ms)
515
+ VALUES (?1, ?2, ?3, ?4, ?4)",
516
+ params![conversation_id, title, project_id, now_ms],
517
+ )
518
+ .map_err(|e| e.to_string())?;
519
+ Ok(())
520
+ }
521
+
522
+ pub fn list_conversations(&self) -> Result<Vec<ConversationRow>, String> {
523
+ let mut stmt = self
524
+ .conn
525
+ .prepare(
526
+ "SELECT conversation_id, title, project_id, local_head, certified_head, created_ms, updated_ms
527
+ FROM conversations ORDER BY updated_ms DESC",
528
+ )
529
+ .map_err(|e| e.to_string())?;
530
+ let rows = stmt
531
+ .query_map([], |r| {
532
+ Ok(ConversationRow {
533
+ conversation_id: r.get(0)?,
534
+ title: r.get(1)?,
535
+ project_id: r.get(2)?,
536
+ heads: ConversationHeadsV1 { local_head: r.get(3)?, certified_head: r.get(4)? },
537
+ created_ms: r.get(5)?,
538
+ updated_ms: r.get(6)?,
539
+ })
540
+ })
541
+ .map_err(|e| e.to_string())?;
542
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
543
+ }
544
+
545
+ pub fn heads(&self, conversation_id: &str) -> Result<ConversationHeadsV1, String> {
546
+ self.conn
547
+ .query_row(
548
+ "SELECT local_head, certified_head FROM conversations WHERE conversation_id = ?1",
549
+ params![conversation_id],
550
+ |r| Ok(ConversationHeadsV1 { local_head: r.get(0)?, certified_head: r.get(1)? }),
551
+ )
552
+ .map_err(|e| format!("conversation {conversation_id}: {e}"))
553
+ }
554
+
555
+ pub fn conversation_project(&self, conversation_id: &str) -> Result<Option<String>, String> {
556
+ self.conn
557
+ .query_row("SELECT project_id FROM conversations WHERE conversation_id = ?1", params![conversation_id], |r| {
558
+ r.get(0)
559
+ })
560
+ .map_err(|e| format!("conversation {conversation_id}: {e}"))
561
+ }
562
+
563
+ // ---- turns --------------------------------------------------------------------------
564
+
565
+ /// Append a turn under `parent_turn_id` (None ⇒ a root — either the first turn or a branch
566
+ /// from "before the first message") and move `local_head` to it. History is append-only:
567
+ /// edit/regenerate NEVER rewrites a row, it adds a sibling and moves the head.
568
+ pub fn begin_turn(&mut self, t: &NewTurn) -> Result<(), String> {
569
+ let tx = self.conn.transaction().map_err(|e| e.to_string())?;
570
+ if let Some(parent) = t.parent_turn_id {
571
+ let parent_conv: String = tx
572
+ .query_row("SELECT conversation_id FROM turns WHERE turn_id = ?1", params![parent], |r| r.get(0))
573
+ .map_err(|e| format!("parent turn {parent}: {e}"))?;
574
+ if parent_conv != t.conversation_id {
575
+ return Err(format!("parent turn {parent} belongs to another conversation"));
576
+ }
577
+ }
578
+ tx.execute(
579
+ "INSERT INTO turns (turn_id, conversation_id, parent_turn_id, role_user_text, user_block_text,
580
+ context_meta_json, prompt_ids, verification_status, privacy_mode,
581
+ search_bundle_json, created_ms)
582
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
583
+ params![
584
+ t.turn_id,
585
+ t.conversation_id,
586
+ t.parent_turn_id,
587
+ t.role_user_text,
588
+ t.user_block_text,
589
+ t.context_meta_json,
590
+ ids_to_blob(t.prompt_ids),
591
+ TurnVerificationStatus::Streaming.as_str(),
592
+ t.privacy_mode.as_str(),
593
+ t.search_bundle_json,
594
+ t.now_ms
595
+ ],
596
+ )
597
+ .map_err(|e| e.to_string())?;
598
+ tx.execute(
599
+ "UPDATE conversations SET local_head = ?2, updated_ms = ?3 WHERE conversation_id = ?1",
600
+ params![t.conversation_id, t.turn_id, t.now_ms],
601
+ )
602
+ .map_err(|e| e.to_string())?;
603
+ tx.commit().map_err(|e| e.to_string())
604
+ }
605
+
606
+ pub fn complete_turn(&self, t: &CompletedTurn) -> Result<(), String> {
607
+ let n = self
608
+ .conn
609
+ .execute(
610
+ "UPDATE turns SET output_ids = ?2, display_text = ?3, stop_reason = ?4,
611
+ verification_status = ?5, receipt_json = ?6, runtime_roots_json = ?7
612
+ WHERE turn_id = ?1",
613
+ params![
614
+ t.turn_id,
615
+ ids_to_blob(t.output_ids),
616
+ t.display_text,
617
+ t.stop_reason,
618
+ t.status.as_str(),
619
+ t.receipt_json,
620
+ t.runtime_roots_json
621
+ ],
622
+ )
623
+ .map_err(|e| e.to_string())?;
624
+ if n != 1 {
625
+ return Err(format!("turn {} not found at completion", t.turn_id));
626
+ }
627
+ Ok(())
628
+ }
629
+
630
+ pub fn get_turn(&self, turn_id: &str) -> Result<StoredTurn, String> {
631
+ get_turn_conn(&self.conn, turn_id)
632
+ }
633
+
634
+ /// All root turns (no parent) across every conversation, oldest first. The candidate set for
635
+ /// LM Studio transcript resolution: that client owns no gateway ids, so a conversation can
636
+ /// only be re-found by content, starting from its first exchange.
637
+ pub fn root_turns(&self) -> Result<Vec<StoredTurn>, String> {
638
+ let mut stmt = self
639
+ .conn
640
+ .prepare(&format!(
641
+ "SELECT {TURN_COLUMNS} FROM turns WHERE parent_turn_id IS NULL ORDER BY created_ms, turn_id"
642
+ ))
643
+ .map_err(|e| e.to_string())?;
644
+ let rows = stmt.query_map([], turn_from_row).map_err(|e| e.to_string())?;
645
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
646
+ }
647
+
648
+ /// Direct children of a turn, oldest first (edit/regenerate siblings in creation order).
649
+ pub fn children_of(&self, turn_id: &str) -> Result<Vec<StoredTurn>, String> {
650
+ let mut stmt = self
651
+ .conn
652
+ .prepare(&format!(
653
+ "SELECT {TURN_COLUMNS} FROM turns WHERE parent_turn_id = ?1 ORDER BY created_ms, turn_id"
654
+ ))
655
+ .map_err(|e| e.to_string())?;
656
+ let rows = stmt.query_map(params![turn_id], turn_from_row).map_err(|e| e.to_string())?;
657
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
658
+ }
659
+
660
+ /// Walk parent pointers from `leaf` back to the root, returning ROOT-FIRST. This is the
661
+ /// branch-aware history the context compiler consumes: exactly the turns on the active
662
+ /// branch, none from abandoned siblings.
663
+ pub fn branch_history(&self, leaf: &str) -> Result<Vec<StoredTurn>, String> {
664
+ branch_history_conn(&self.conn, leaf)
665
+ }
666
+
667
+ /// A turn that FAILED (engine error — no output, nothing a user would keep) must not stay
668
+ /// the branch head: retract the head to its parent so the next message retries as a sibling
669
+ /// instead of silently building on a broken turn. The failed row itself is kept (append-only
670
+ /// history; it remains visible under its own id).
671
+ pub fn retract_failed_head(&self, conversation_id: &str, failed_turn_id: &str, now_ms: i64) -> Result<(), String> {
672
+ let turn = self.get_turn(failed_turn_id)?;
673
+ if turn.conversation_id != conversation_id {
674
+ return Err(format!("turn {failed_turn_id} belongs to another conversation"));
675
+ }
676
+ let head: Option<String> = self
677
+ .conn
678
+ .query_row("SELECT local_head FROM conversations WHERE conversation_id = ?1", params![conversation_id], |r| {
679
+ r.get(0)
680
+ })
681
+ .map_err(|e| e.to_string())?;
682
+ if head.as_deref() == Some(failed_turn_id) {
683
+ self.conn
684
+ .execute(
685
+ "UPDATE conversations SET local_head = ?2, updated_ms = ?3 WHERE conversation_id = ?1",
686
+ params![conversation_id, turn.parent_turn_id, now_ms],
687
+ )
688
+ .map_err(|e| e.to_string())?;
689
+ }
690
+ Ok(())
691
+ }
692
+
693
+ pub fn set_local_head(&self, conversation_id: &str, turn_id: &str, now_ms: i64) -> Result<(), String> {
694
+ let turn = self.get_turn(turn_id)?;
695
+ if turn.conversation_id != conversation_id {
696
+ return Err(format!("turn {turn_id} belongs to another conversation"));
697
+ }
698
+ self.conn
699
+ .execute(
700
+ "UPDATE conversations SET local_head = ?2, updated_ms = ?3 WHERE conversation_id = ?1",
701
+ params![conversation_id, turn_id, now_ms],
702
+ )
703
+ .map_err(|e| e.to_string())?;
704
+ // The certified prefix is branch-relative: switching branches moves it too.
705
+ recompute_certified_head(&self.conn, conversation_id, now_ms)?;
706
+ Ok(())
707
+ }
708
+
709
+ // ---- settlement clock (Phase 2 writes) ----------------------------------------------
710
+
711
+ /// Advance a turn along the verification pipeline. Enforces the legal transition machine,
712
+ /// requires replica states to have a non-LocalOnly privacy mode (a LocalOnly turn never
713
+ /// leaves the device, so it can never be replica-pending), cascades mismatches, and
714
+ /// recomputes `certified_head`.
715
+ pub fn set_verification_status(
716
+ &mut self,
717
+ turn_id: &str,
718
+ to: TurnVerificationStatus,
719
+ now_ms: i64,
720
+ ) -> Result<VerificationOutcome, String> {
721
+ let tx = self.conn.transaction().map_err(|e| e.to_string())?;
722
+ let turn = get_turn_conn(&tx, turn_id)?;
723
+ let from = turn.verification_status;
724
+ if !verification_transition_legal(from, to) {
725
+ return Err(format!("illegal verification transition {} → {}", from.as_str(), to.as_str()));
726
+ }
727
+ if to == TurnVerificationStatus::ReplicaPending && turn.privacy_mode == TurnPrivacyMode::LocalOnly {
728
+ return Err(format!("turn {turn_id} is local_only — it can never enter the replica pipeline"));
729
+ }
730
+ tx.execute("UPDATE turns SET verification_status = ?2 WHERE turn_id = ?1", params![turn_id, to.as_str()])
731
+ .map_err(|e| e.to_string())?;
732
+
733
+ // §4: a mismatch keeps the local conversation but invalidates the REWARD dependency
734
+ // chain — every descendant (any branch below this turn) leaves the pipeline.
735
+ let mut cascaded = Vec::new();
736
+ if to == TurnVerificationStatus::Mismatch {
737
+ let mut stmt = tx
738
+ .prepare(
739
+ "WITH RECURSIVE below(id) AS (
740
+ SELECT turn_id FROM turns WHERE parent_turn_id = ?1
741
+ UNION ALL
742
+ SELECT t.turn_id FROM turns t JOIN below b ON t.parent_turn_id = b.id
743
+ )
744
+ SELECT id FROM below",
745
+ )
746
+ .map_err(|e| e.to_string())?;
747
+ let descendants: Vec<String> = stmt
748
+ .query_map(params![turn_id], |r| r.get(0))
749
+ .map_err(|e| e.to_string())?
750
+ .collect::<Result<_, _>>()
751
+ .map_err(|e| e.to_string())?;
752
+ drop(stmt);
753
+ for id in descendants {
754
+ let n = tx
755
+ .execute(
756
+ "UPDATE turns SET verification_status = ?2 WHERE turn_id = ?1
757
+ AND verification_status NOT IN ('mismatch', 'mint_ineligible')",
758
+ params![id, TurnVerificationStatus::MintIneligible.as_str()],
759
+ )
760
+ .map_err(|e| e.to_string())?;
761
+ if n == 1 {
762
+ cascaded.push(id);
763
+ }
764
+ }
765
+ }
766
+
767
+ let certified_head = recompute_certified_head(&tx, &turn.conversation_id, now_ms)?;
768
+ tx.commit().map_err(|e| e.to_string())?;
769
+ Ok(VerificationOutcome { status: to, cascaded, certified_head })
770
+ }
771
+
772
+ /// A-side scan (§15 "A receipt登録"): completed turns whose privacy mode wants replication
773
+ /// and that have not yet been submitted. Stateless-worker recovery falls out of this being
774
+ /// a pure store query — a worker restarted mid-flight simply finds them again and re-submits
775
+ /// idempotently.
776
+ pub fn turns_awaiting_palw_submission(&self) -> Result<Vec<StoredTurn>, String> {
777
+ let mut stmt = self
778
+ .conn
779
+ .prepare(&format!(
780
+ "SELECT {TURN_COLUMNS} FROM turns
781
+ WHERE verification_status = 'local_complete'
782
+ AND privacy_mode IN ('verified_no_mint', 'palw_mint')
783
+ ORDER BY created_ms, turn_id"
784
+ ))
785
+ .map_err(|e| e.to_string())?;
786
+ let rows = stmt.query_map([], turn_from_row).map_err(|e| e.to_string())?;
787
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
788
+ }
789
+
790
+ /// Turns whose settlement is in flight — the worker polls the coordinator for verdicts on
791
+ /// exactly these.
792
+ pub fn turns_in_palw_pipeline(&self) -> Result<Vec<StoredTurn>, String> {
793
+ let mut stmt = self
794
+ .conn
795
+ .prepare(&format!(
796
+ "SELECT {TURN_COLUMNS} FROM turns
797
+ WHERE verification_status IN ('replica_pending', 'replica_matched', 'audit_pending')
798
+ ORDER BY created_ms, turn_id"
799
+ ))
800
+ .map_err(|e| e.to_string())?;
801
+ let rows = stmt.query_map([], turn_from_row).map_err(|e| e.to_string())?;
802
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
803
+ }
804
+
805
+ // ---- projects (§10) -----------------------------------------------------------------
806
+
807
+ pub fn create_project(&self, project_id: &str, name: &str, instructions: &str, now_ms: i64) -> Result<(), String> {
808
+ self.conn
809
+ .execute(
810
+ "INSERT INTO projects (project_id, name, instructions, created_ms, updated_ms) VALUES (?1, ?2, ?3, ?4, ?4)",
811
+ params![project_id, name, instructions, now_ms],
812
+ )
813
+ .map_err(|e| e.to_string())?;
814
+ Ok(())
815
+ }
816
+
817
+ pub fn get_project(&self, project_id: &str) -> Result<ProjectRow, String> {
818
+ self.conn
819
+ .query_row(
820
+ "SELECT project_id, name, instructions, created_ms, updated_ms FROM projects WHERE project_id = ?1",
821
+ params![project_id],
822
+ |r| {
823
+ Ok(ProjectRow {
824
+ project_id: r.get(0)?,
825
+ name: r.get(1)?,
826
+ instructions: r.get(2)?,
827
+ created_ms: r.get(3)?,
828
+ updated_ms: r.get(4)?,
829
+ })
830
+ },
831
+ )
832
+ .map_err(|e| format!("project {project_id}: {e}"))
833
+ }
834
+
835
+ pub fn list_projects(&self) -> Result<Vec<ProjectRow>, String> {
836
+ let mut stmt = self
837
+ .conn
838
+ .prepare("SELECT project_id, name, instructions, created_ms, updated_ms FROM projects ORDER BY updated_ms DESC")
839
+ .map_err(|e| e.to_string())?;
840
+ let rows = stmt
841
+ .query_map([], |r| {
842
+ Ok(ProjectRow {
843
+ project_id: r.get(0)?,
844
+ name: r.get(1)?,
845
+ instructions: r.get(2)?,
846
+ created_ms: r.get(3)?,
847
+ updated_ms: r.get(4)?,
848
+ })
849
+ })
850
+ .map_err(|e| e.to_string())?;
851
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
852
+ }
853
+
854
+ pub fn set_project_instructions(&self, project_id: &str, instructions: &str, now_ms: i64) -> Result<(), String> {
855
+ let n = self
856
+ .conn
857
+ .execute(
858
+ "UPDATE projects SET instructions = ?2, updated_ms = ?3 WHERE project_id = ?1",
859
+ params![project_id, instructions, now_ms],
860
+ )
861
+ .map_err(|e| e.to_string())?;
862
+ if n != 1 {
863
+ return Err(format!("project {project_id} not found"));
864
+ }
865
+ Ok(())
866
+ }
867
+
868
+ // ---- long-term memory (§12) ---------------------------------------------------------
869
+
870
+ /// Namespaces: `"global"` or a project id. Provenance (`source_turn_id`) and soft delete
871
+ /// (`deleted_sequence`) are the §12 requirements — memory is never silently rewritten, and
872
+ /// "これは忘れて" is a tombstone, not a DELETE.
873
+ pub fn add_memory(
874
+ &self,
875
+ memory_id: &str,
876
+ namespace: &str,
877
+ content: &str,
878
+ source_turn_id: Option<&str>,
879
+ now_ms: i64,
880
+ ) -> Result<i64, String> {
881
+ let seq: i64 = self
882
+ .conn
883
+ .query_row("SELECT COALESCE(MAX(created_sequence), 0) + 1 FROM memory_entries", [], |r| r.get(0))
884
+ .map_err(|e| e.to_string())?;
885
+ self.conn
886
+ .execute(
887
+ "INSERT INTO memory_entries (memory_id, namespace, content, source_turn_id, created_sequence, created_ms)
888
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
889
+ params![memory_id, namespace, content, source_turn_id, seq, now_ms],
890
+ )
891
+ .map_err(|e| e.to_string())?;
892
+ Ok(seq)
893
+ }
894
+
895
+ pub fn delete_memory(&self, memory_id: &str) -> Result<(), String> {
896
+ let seq: i64 = self
897
+ .conn
898
+ .query_row("SELECT COALESCE(MAX(created_sequence), 0) + 1 FROM memory_entries", [], |r| r.get(0))
899
+ .map_err(|e| e.to_string())?;
900
+ let n = self
901
+ .conn
902
+ .execute(
903
+ "UPDATE memory_entries SET deleted_sequence = ?2 WHERE memory_id = ?1 AND deleted_sequence IS NULL",
904
+ params![memory_id, seq],
905
+ )
906
+ .map_err(|e| e.to_string())?;
907
+ if n != 1 {
908
+ return Err(format!("memory {memory_id} not found or already deleted"));
909
+ }
910
+ Ok(())
911
+ }
912
+
913
+ /// Active (non-tombstoned) entries across the given namespaces, oldest-first by global
914
+ /// sequence — the deterministic order the context compiler serializes into the snapshot.
915
+ pub fn active_memory(&self, namespaces: &[&str]) -> Result<Vec<MemoryEntryRow>, String> {
916
+ if namespaces.is_empty() {
917
+ return Ok(Vec::new());
918
+ }
919
+ let placeholders: Vec<String> = (1..=namespaces.len()).map(|i| format!("?{i}")).collect();
920
+ let sql = format!(
921
+ "SELECT memory_id, namespace, content, source_turn_id, created_sequence, created_ms
922
+ FROM memory_entries WHERE namespace IN ({}) AND deleted_sequence IS NULL
923
+ ORDER BY created_sequence",
924
+ placeholders.join(", ")
925
+ );
926
+ let mut stmt = self.conn.prepare(&sql).map_err(|e| e.to_string())?;
927
+ let rows = stmt
928
+ .query_map(rusqlite::params_from_iter(namespaces.iter()), |r| {
929
+ Ok(MemoryEntryRow {
930
+ memory_id: r.get(0)?,
931
+ namespace: r.get(1)?,
932
+ content: r.get(2)?,
933
+ source_turn_id: r.get(3)?,
934
+ created_sequence: r.get(4)?,
935
+ created_ms: r.get(5)?,
936
+ })
937
+ })
938
+ .map_err(|e| e.to_string())?;
939
+ rows.collect::<Result<Vec<_>, _>>().map_err(|e| e.to_string())
940
+ }
941
+
942
+ // ---- artifacts (§8/§9) --------------------------------------------------------------
943
+
944
+ /// Register artifact metadata. If `parent_artifact_id` is set this is a new VERSION: the
945
+ /// stored `version` is parent.version + 1 regardless of what the caller guessed — version
946
+ /// numbers are derived, never client-supplied.
947
+ pub fn insert_artifact(&self, meta: &ArtifactMeta) -> Result<i64, String> {
948
+ let version = match &meta.parent_artifact_id {
949
+ Some(parent) => self.get_artifact(parent)?.version + 1,
950
+ None => 1,
951
+ };
952
+ self.conn
953
+ .execute(
954
+ "INSERT INTO artifacts (artifact_id, content_hash, mime_type, byte_length, display_name,
955
+ parent_artifact_id, version, parser_policy, project_id, created_by_turn, created_ms)
956
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
957
+ params![
958
+ meta.artifact_id,
959
+ meta.content_hash,
960
+ meta.mime_type,
961
+ meta.byte_length,
962
+ meta.display_name,
963
+ meta.parent_artifact_id,
964
+ version,
965
+ meta.parser_policy,
966
+ meta.project_id,
967
+ meta.created_by_turn,
968
+ meta.created_ms
969
+ ],
970
+ )
971
+ .map_err(|e| e.to_string())?;
972
+ Ok(version)
973
+ }
974
+
975
+ pub fn get_artifact(&self, artifact_id: &str) -> Result<ArtifactMeta, String> {
976
+ self.conn
977
+ .query_row(
978
+ "SELECT artifact_id, content_hash, mime_type, byte_length, display_name, parent_artifact_id,
979
+ version, parser_policy, project_id, created_by_turn, created_ms
980
+ FROM artifacts WHERE artifact_id = ?1",
981
+ params![artifact_id],
982
+ |r| {
983
+ Ok(ArtifactMeta {
984
+ artifact_id: r.get(0)?,
985
+ content_hash: r.get(1)?,
986
+ mime_type: r.get(2)?,
987
+ byte_length: r.get(3)?,
988
+ display_name: r.get(4)?,
989
+ parent_artifact_id: r.get(5)?,
990
+ version: r.get(6)?,
991
+ parser_policy: r.get(7)?,
992
+ project_id: r.get(8)?,
993
+ created_by_turn: r.get(9)?,
994
+ created_ms: r.get(10)?,
995
+ })
996
+ },
997
+ )
998
+ .map_err(|e| format!("artifact {artifact_id}: {e}"))
999
+ }
1000
+
1001
+ /// The version chain containing `artifact_id`, root-first up to and including it (the §9
1002
+ /// "design.md v1 → v2 → v3" walk; forks below it are listed via `artifact_children`).
1003
+ pub fn artifact_lineage(&self, artifact_id: &str) -> Result<Vec<ArtifactMeta>, String> {
1004
+ let mut chain = Vec::new();
1005
+ let mut cursor = Some(artifact_id.to_string());
1006
+ while let Some(id) = cursor {
1007
+ let meta = self.get_artifact(&id)?;
1008
+ cursor = meta.parent_artifact_id.clone();
1009
+ chain.push(meta);
1010
+ if chain.len() > 100_000 {
1011
+ return Err("artifact version chain exceeds 100k — refusing (corrupt parent loop?)".into());
1012
+ }
1013
+ }
1014
+ chain.reverse();
1015
+ Ok(chain)
1016
+ }
1017
+
1018
+ pub fn artifact_children(&self, artifact_id: &str) -> Result<Vec<ArtifactMeta>, String> {
1019
+ let mut stmt = self
1020
+ .conn
1021
+ .prepare(
1022
+ "SELECT artifact_id FROM artifacts WHERE parent_artifact_id = ?1 ORDER BY created_ms, artifact_id",
1023
+ )
1024
+ .map_err(|e| e.to_string())?;
1025
+ let ids: Vec<String> = stmt
1026
+ .query_map(params![artifact_id], |r| r.get(0))
1027
+ .map_err(|e| e.to_string())?
1028
+ .collect::<Result<_, _>>()
1029
+ .map_err(|e| e.to_string())?;
1030
+ ids.iter().map(|id| self.get_artifact(id)).collect()
1031
+ }
1032
+
1033
+ pub fn list_artifacts(&self, project_id: Option<&str>) -> Result<Vec<ArtifactMeta>, String> {
1034
+ let sql = match project_id {
1035
+ Some(_) => {
1036
+ "SELECT artifact_id FROM artifacts WHERE project_id = ?1 ORDER BY created_ms DESC, artifact_id"
1037
+ }
1038
+ None => "SELECT artifact_id FROM artifacts WHERE ?1 IS NULL ORDER BY created_ms DESC, artifact_id",
1039
+ };
1040
+ let mut stmt = self.conn.prepare(sql).map_err(|e| e.to_string())?;
1041
+ let ids: Vec<String> = stmt
1042
+ .query_map(params![project_id], |r| r.get(0))
1043
+ .map_err(|e| e.to_string())?
1044
+ .collect::<Result<_, _>>()
1045
+ .map_err(|e| e.to_string())?;
1046
+ ids.iter().map(|id| self.get_artifact(id)).collect()
1047
+ }
1048
+
1049
+ // ---- tool-call ledger (§6) ----------------------------------------------------------
1050
+
1051
+ pub fn insert_tool_call(&self, call: &ToolCallRow) -> Result<(), String> {
1052
+ self.conn
1053
+ .execute(
1054
+ "INSERT INTO tool_calls (call_id, tool_name, mode, arguments_json, status, result_artifact_id,
1055
+ result_preview, error, conversation_id, created_ms, executed_ms)
1056
+ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)",
1057
+ params![
1058
+ call.call_id,
1059
+ call.tool_name,
1060
+ call.mode,
1061
+ call.arguments_json,
1062
+ call.status,
1063
+ call.result_artifact_id,
1064
+ call.result_preview,
1065
+ call.error,
1066
+ call.conversation_id,
1067
+ call.created_ms,
1068
+ call.executed_ms
1069
+ ],
1070
+ )
1071
+ .map_err(|e| e.to_string())?;
1072
+ Ok(())
1073
+ }
1074
+
1075
+ pub fn get_tool_call(&self, call_id: &str) -> Result<ToolCallRow, String> {
1076
+ self.conn
1077
+ .query_row(
1078
+ "SELECT call_id, tool_name, mode, arguments_json, status, result_artifact_id, result_preview,
1079
+ error, conversation_id, created_ms, executed_ms
1080
+ FROM tool_calls WHERE call_id = ?1",
1081
+ params![call_id],
1082
+ |r| {
1083
+ Ok(ToolCallRow {
1084
+ call_id: r.get(0)?,
1085
+ tool_name: r.get(1)?,
1086
+ mode: r.get(2)?,
1087
+ arguments_json: r.get(3)?,
1088
+ status: r.get(4)?,
1089
+ result_artifact_id: r.get(5)?,
1090
+ result_preview: r.get(6)?,
1091
+ error: r.get(7)?,
1092
+ conversation_id: r.get(8)?,
1093
+ created_ms: r.get(9)?,
1094
+ executed_ms: r.get(10)?,
1095
+ })
1096
+ },
1097
+ )
1098
+ .map_err(|e| format!("tool call {call_id}: {e}"))
1099
+ }
1100
+
1101
+ /// proposed → approved. The UI confirmation step (§6 step 2) — refuses anything not
1102
+ /// currently proposed, so a denied or already-run call cannot be re-approved.
1103
+ pub fn approve_tool_call(&self, call_id: &str) -> Result<(), String> {
1104
+ let n = self
1105
+ .conn
1106
+ .execute("UPDATE tool_calls SET status = 'approved' WHERE call_id = ?1 AND status = 'proposed'", params![call_id])
1107
+ .map_err(|e| e.to_string())?;
1108
+ if n != 1 {
1109
+ return Err(format!("tool call {call_id} is not awaiting approval"));
1110
+ }
1111
+ Ok(())
1112
+ }
1113
+
1114
+ pub fn deny_tool_call(&self, call_id: &str) -> Result<(), String> {
1115
+ let n = self
1116
+ .conn
1117
+ .execute("UPDATE tool_calls SET status = 'denied' WHERE call_id = ?1 AND status = 'proposed'", params![call_id])
1118
+ .map_err(|e| e.to_string())?;
1119
+ if n != 1 {
1120
+ return Err(format!("tool call {call_id} is not awaiting approval"));
1121
+ }
1122
+ Ok(())
1123
+ }
1124
+
1125
+ /// THE exactly-once guard (§6 step 3): flips approved → executing atomically; the single
1126
+ /// winning UPDATE is the execution ticket. A second caller — a double-click, a retried HTTP
1127
+ /// request, a concurrent thread — finds status ≠ 'approved' and is refused, so a side effect
1128
+ /// can never fire twice.
1129
+ pub fn claim_tool_execution(&self, call_id: &str) -> Result<(), String> {
1130
+ let n = self
1131
+ .conn
1132
+ .execute("UPDATE tool_calls SET status = 'executing' WHERE call_id = ?1 AND status = 'approved'", params![call_id])
1133
+ .map_err(|e| e.to_string())?;
1134
+ if n != 1 {
1135
+ return Err(format!("tool call {call_id} is not approved for execution"));
1136
+ }
1137
+ Ok(())
1138
+ }
1139
+
1140
+ pub fn finish_tool_call(
1141
+ &self,
1142
+ call_id: &str,
1143
+ ok: bool,
1144
+ result_artifact_id: Option<&str>,
1145
+ result_preview: &str,
1146
+ error: Option<&str>,
1147
+ now_ms: i64,
1148
+ ) -> Result<(), String> {
1149
+ let n = self
1150
+ .conn
1151
+ .execute(
1152
+ "UPDATE tool_calls SET status = ?2, result_artifact_id = ?3, result_preview = ?4, error = ?5,
1153
+ executed_ms = ?6
1154
+ WHERE call_id = ?1 AND status = 'executing'",
1155
+ params![call_id, if ok { "executed" } else { "failed" }, result_artifact_id, result_preview, error, now_ms],
1156
+ )
1157
+ .map_err(|e| e.to_string())?;
1158
+ if n != 1 {
1159
+ return Err(format!("tool call {call_id} was not executing"));
1160
+ }
1161
+ Ok(())
1162
+ }
1163
+ }
1164
+
1165
+ #[cfg(test)]
1166
+ mod tests {
1167
+ use super::*;
1168
+
1169
+ fn store_at(name: &str) -> SessionStore {
1170
+ let dir = std::env::temp_dir().join(format!("palw-gw-test-{}-{name}", std::process::id()));
1171
+ let _ = std::fs::remove_dir_all(&dir);
1172
+ SessionStore::open(&dir.join("s.sqlite3")).unwrap()
1173
+ }
1174
+
1175
+ fn complete(s: &SessionStore, turn_id: &str, output_ids: &[u32], status: TurnVerificationStatus) {
1176
+ s.complete_turn(&CompletedTurn {
1177
+ turn_id,
1178
+ output_ids,
1179
+ display_text: "ok",
1180
+ stop_reason: "eos",
1181
+ status,
1182
+ receipt_json: None,
1183
+ runtime_roots_json: None,
1184
+ })
1185
+ .unwrap();
1186
+ }
1187
+
1188
+ fn new_turn<'a>(
1189
+ turn_id: &'a str,
1190
+ conversation_id: &'a str,
1191
+ parent: Option<&'a str>,
1192
+ text: &'a str,
1193
+ prompt_ids: &'a [u32],
1194
+ privacy: TurnPrivacyMode,
1195
+ now: i64,
1196
+ ) -> NewTurn<'a> {
1197
+ NewTurn {
1198
+ turn_id,
1199
+ conversation_id,
1200
+ parent_turn_id: parent,
1201
+ role_user_text: text,
1202
+ user_block_text: text,
1203
+ context_meta_json: None,
1204
+ prompt_ids,
1205
+ privacy_mode: privacy,
1206
+ search_bundle_json: None,
1207
+ now_ms: now,
1208
+ }
1209
+ }
1210
+
1211
+ #[test]
1212
+ fn ids_roundtrip_and_branching() {
1213
+ let mut s = store_at("branch");
1214
+ s.create_conversation("c1", "t", None, 1).unwrap();
1215
+ s.begin_turn(&new_turn("t1", "c1", None, "hello", &[1, 2, 3], TurnPrivacyMode::LocalOnly, 2)).unwrap();
1216
+ complete(&s, "t1", &[9, 8], TurnVerificationStatus::LocalComplete);
1217
+ s.begin_turn(&new_turn("t2", "c1", Some("t1"), "next", &[1, 2, 3, 9, 8, 4], TurnPrivacyMode::LocalOnly, 3))
1218
+ .unwrap();
1219
+ complete(&s, "t2", &[7], TurnVerificationStatus::LocalComplete);
1220
+
1221
+ // Branch: edit turn 2 — sibling under t1, head moves, t2 stays intact.
1222
+ s.begin_turn(&new_turn("t2b", "c1", Some("t1"), "next-edited", &[1, 2, 3, 9, 8, 5], TurnPrivacyMode::LocalOnly, 4))
1223
+ .unwrap();
1224
+ let heads = s.heads("c1").unwrap();
1225
+ assert_eq!(heads.local_head.as_deref(), Some("t2b"));
1226
+ assert_eq!(heads.certified_head, None);
1227
+
1228
+ let hist = s.branch_history("t2b").unwrap();
1229
+ assert_eq!(hist.iter().map(|t| t.turn_id.as_str()).collect::<Vec<_>>(), vec!["t1", "t2b"]);
1230
+ // Token ids survive exactly — the id-stable-history invariant.
1231
+ assert_eq!(hist[0].output_ids, vec![9, 8]);
1232
+ let old = s.branch_history("t2").unwrap();
1233
+ assert_eq!(old.len(), 2);
1234
+ assert_eq!(old[1].prompt_ids, vec![1, 2, 3, 9, 8, 4]);
1235
+ }
1236
+
1237
+ /// A database created by the v1 gateway must open, migrate, and keep serving byte-identical
1238
+ /// history — `user_block_text` NULL folds back to the raw user text.
1239
+ #[test]
1240
+ fn migrates_v1_database_in_place() {
1241
+ let dir = std::env::temp_dir().join(format!("palw-gw-test-{}-migrate", std::process::id()));
1242
+ let _ = std::fs::remove_dir_all(&dir);
1243
+ std::fs::create_dir_all(&dir).unwrap();
1244
+ let path = dir.join("s.sqlite3");
1245
+ {
1246
+ // The v1 DDL, verbatim from the Phase-0 store.
1247
+ let conn = Connection::open(&path).unwrap();
1248
+ conn.pragma_update(None, "application_id", APPLICATION_ID).unwrap();
1249
+ conn.execute_batch(
1250
+ r#"
1251
+ CREATE TABLE conversations (
1252
+ conversation_id TEXT PRIMARY KEY,
1253
+ title TEXT NOT NULL DEFAULT '',
1254
+ local_head TEXT,
1255
+ certified_head TEXT,
1256
+ created_ms INTEGER NOT NULL,
1257
+ updated_ms INTEGER NOT NULL
1258
+ );
1259
+ CREATE TABLE turns (
1260
+ turn_id TEXT PRIMARY KEY,
1261
+ conversation_id TEXT NOT NULL REFERENCES conversations(conversation_id),
1262
+ parent_turn_id TEXT REFERENCES turns(turn_id),
1263
+ role_user_text TEXT NOT NULL,
1264
+ display_text TEXT NOT NULL DEFAULT '',
1265
+ prompt_ids BLOB NOT NULL,
1266
+ output_ids BLOB NOT NULL DEFAULT x'',
1267
+ stop_reason TEXT,
1268
+ verification_status TEXT NOT NULL,
1269
+ privacy_mode TEXT NOT NULL,
1270
+ receipt_json TEXT,
1271
+ created_ms INTEGER NOT NULL
1272
+ );
1273
+ CREATE INDEX turns_by_conversation ON turns(conversation_id, created_ms);
1274
+ CREATE TABLE artifacts (
1275
+ artifact_id TEXT PRIMARY KEY,
1276
+ content_hash TEXT NOT NULL,
1277
+ mime_type TEXT NOT NULL,
1278
+ byte_length INTEGER NOT NULL,
1279
+ created_by_turn TEXT REFERENCES turns(turn_id),
1280
+ created_ms INTEGER NOT NULL
1281
+ );
1282
+ PRAGMA user_version = 1;
1283
+ "#,
1284
+ )
1285
+ .unwrap();
1286
+ conn.execute(
1287
+ "INSERT INTO conversations (conversation_id, title, local_head, created_ms, updated_ms)
1288
+ VALUES ('c1', 'old', 't1', 1, 1)",
1289
+ [],
1290
+ )
1291
+ .unwrap();
1292
+ conn.execute(
1293
+ "INSERT INTO turns (turn_id, conversation_id, role_user_text, prompt_ids, output_ids,
1294
+ verification_status, privacy_mode, created_ms)
1295
+ VALUES ('t1', 'c1', 'hello', x'01000000', x'02000000', 'local_complete', 'local_only', 1)",
1296
+ [],
1297
+ )
1298
+ .unwrap();
1299
+ }
1300
+
1301
+ let s = SessionStore::open(&path).unwrap();
1302
+ let turn = s.get_turn("t1").unwrap();
1303
+ assert_eq!(turn.user_block_text, None);
1304
+ assert_eq!(turn.user_block(), "hello");
1305
+ assert_eq!(turn.prompt_ids, vec![1]);
1306
+ assert_eq!(turn.output_ids, vec![2]);
1307
+ assert_eq!(turn.search_bundle_json, None, "v4 column readable on migrated rows");
1308
+ // v2 tables exist and take writes after migration.
1309
+ s.create_project("p1", "proj", "", 2).unwrap();
1310
+ s.add_memory("m1", "global", "fact", None, 2).unwrap();
1311
+ let version: i64 = s
1312
+ .conn
1313
+ .query_row("PRAGMA user_version", [], |r| r.get(0))
1314
+ .unwrap();
1315
+ assert_eq!(version, SCHEMA_VERSION);
1316
+ }
1317
+
1318
+ #[test]
1319
+ fn search_bundle_persists_from_begin_turn() {
1320
+ let mut s = store_at("search-bundle");
1321
+ s.create_conversation("c1", "t", None, 1).unwrap();
1322
+ let mut turn = new_turn("t1", "c1", None, "q", &[1], TurnPrivacyMode::PalwMint, 1);
1323
+ turn.search_bundle_json = Some(r#"{"bundle_sha256":"abc"}"#);
1324
+ s.begin_turn(&turn).unwrap();
1325
+ assert_eq!(s.get_turn("t1").unwrap().search_bundle_json.as_deref(), Some(r#"{"bundle_sha256":"abc"}"#));
1326
+ }
1327
+
1328
+ #[test]
1329
+ fn memory_soft_delete_and_namespace_order() {
1330
+ let s = store_at("memory");
1331
+ s.add_memory("m1", "global", "a", None, 1).unwrap();
1332
+ s.add_memory("m2", "proj-1", "b", None, 2).unwrap();
1333
+ s.add_memory("m3", "global", "c", None, 3).unwrap();
1334
+ s.delete_memory("m1").unwrap();
1335
+ assert!(s.delete_memory("m1").is_err(), "double tombstone must be refused");
1336
+
1337
+ let active = s.active_memory(&["global", "proj-1"]).unwrap();
1338
+ assert_eq!(active.iter().map(|m| m.memory_id.as_str()).collect::<Vec<_>>(), vec!["m2", "m3"]);
1339
+ let global_only = s.active_memory(&["global"]).unwrap();
1340
+ assert_eq!(global_only.iter().map(|m| m.memory_id.as_str()).collect::<Vec<_>>(), vec!["m3"]);
1341
+ }
1342
+
1343
+ #[test]
1344
+ fn artifact_versions_are_derived() {
1345
+ let s = store_at("artifacts");
1346
+ let base = ArtifactMeta {
1347
+ artifact_id: "a1".into(),
1348
+ content_hash: "h1".into(),
1349
+ mime_type: "text/markdown".into(),
1350
+ byte_length: 3,
1351
+ display_name: "design.md".into(),
1352
+ parent_artifact_id: None,
1353
+ version: 99, // caller-supplied versions are ignored
1354
+ parser_policy: "utf8-text-v1".into(),
1355
+ project_id: None,
1356
+ created_by_turn: None,
1357
+ created_ms: 1,
1358
+ };
1359
+ assert_eq!(s.insert_artifact(&base).unwrap(), 1);
1360
+ let v2 = ArtifactMeta { artifact_id: "a2".into(), content_hash: "h2".into(), parent_artifact_id: Some("a1".into()), created_ms: 2, ..base.clone() };
1361
+ assert_eq!(s.insert_artifact(&v2).unwrap(), 2);
1362
+ let v3 = ArtifactMeta { artifact_id: "a3".into(), content_hash: "h3".into(), parent_artifact_id: Some("a2".into()), created_ms: 3, ..base.clone() };
1363
+ assert_eq!(s.insert_artifact(&v3).unwrap(), 3);
1364
+
1365
+ let lineage = s.artifact_lineage("a3").unwrap();
1366
+ assert_eq!(lineage.iter().map(|a| a.artifact_id.as_str()).collect::<Vec<_>>(), vec!["a1", "a2", "a3"]);
1367
+ assert_eq!(lineage.iter().map(|a| a.version).collect::<Vec<_>>(), vec![1, 2, 3]);
1368
+ let children = s.artifact_children("a1").unwrap();
1369
+ assert_eq!(children.len(), 1);
1370
+ assert_eq!(children[0].artifact_id, "a2");
1371
+ }
1372
+
1373
+ #[test]
1374
+ fn tool_call_exactly_once_guard() {
1375
+ let s = store_at("tools");
1376
+ let call = ToolCallRow {
1377
+ call_id: "call1".into(),
1378
+ tool_name: "write_file".into(),
1379
+ mode: "exactly_once_side_effect".into(),
1380
+ arguments_json: "{}".into(),
1381
+ status: "proposed".into(),
1382
+ result_artifact_id: None,
1383
+ result_preview: String::new(),
1384
+ error: None,
1385
+ conversation_id: None,
1386
+ created_ms: 1,
1387
+ executed_ms: None,
1388
+ };
1389
+ s.insert_tool_call(&call).unwrap();
1390
+ assert!(s.claim_tool_execution("call1").is_err(), "unapproved call must not execute");
1391
+ s.approve_tool_call("call1").unwrap();
1392
+ assert!(s.approve_tool_call("call1").is_err(), "double approval refused");
1393
+ s.claim_tool_execution("call1").unwrap();
1394
+ assert!(s.claim_tool_execution("call1").is_err(), "second claim refused — exactly once");
1395
+ s.finish_tool_call("call1", true, None, "ok", None, 2).unwrap();
1396
+ assert_eq!(s.get_tool_call("call1").unwrap().status, "executed");
1397
+
1398
+ // A denied proposal is terminal.
1399
+ s.insert_tool_call(&ToolCallRow { call_id: "call2".into(), ..call.clone() }).unwrap();
1400
+ s.deny_tool_call("call2").unwrap();
1401
+ assert!(s.approve_tool_call("call2").is_err());
1402
+ }
1403
+
1404
+ #[test]
1405
+ fn verification_pipeline_certifies_and_cascades() {
1406
+ use TurnVerificationStatus::*;
1407
+ let mut s = store_at("verify");
1408
+ s.create_conversation("c1", "t", None, 1).unwrap();
1409
+ for (id, parent, n) in [("t1", None, 2i64), ("t2", Some("t1"), 3), ("t3", Some("t2"), 4)] {
1410
+ s.begin_turn(&new_turn(id, "c1", parent, "m", &[1], TurnPrivacyMode::PalwMint, n)).unwrap();
1411
+ complete(&s, id, &[2], LocalComplete);
1412
+ }
1413
+
1414
+ // LocalOnly turns can never enter the pipeline.
1415
+ s.begin_turn(&new_turn("t4", "c1", Some("t3"), "m", &[1], TurnPrivacyMode::LocalOnly, 5)).unwrap();
1416
+ complete(&s, "t4", &[2], LocalComplete);
1417
+ assert!(s.set_verification_status("t4", ReplicaPending, 6).is_err());
1418
+ // Put the head back on t3's line for the cascade check below.
1419
+ s.set_local_head("c1", "t3", 6).unwrap();
1420
+
1421
+ // Out-of-order settlement: t2 certifies before t1 — certified_head must NOT move.
1422
+ for id in ["t1", "t2", "t3"] {
1423
+ s.set_verification_status(id, ReplicaPending, 7).unwrap();
1424
+ }
1425
+ s.set_verification_status("t2", ReplicaMatched, 8).unwrap();
1426
+ let out = s.set_verification_status("t2", Certified, 8).unwrap();
1427
+ assert_eq!(out.certified_head, None, "gap at t1 — no certified prefix yet");
1428
+
1429
+ s.set_verification_status("t1", ReplicaMatched, 9).unwrap();
1430
+ s.set_verification_status("t1", AuditPending, 9).unwrap();
1431
+ let out = s.set_verification_status("t1", Certified, 9).unwrap();
1432
+ assert_eq!(out.certified_head.as_deref(), Some("t2"), "prefix t1..t2 now contiguous");
1433
+
1434
+ // Illegal jumps are refused.
1435
+ assert!(s.set_verification_status("t3", Certified, 10).is_err());
1436
+
1437
+ // Mismatch at t3: its only descendant is the LocalOnly t4, which also leaves the reward
1438
+ // pipeline (it was never eligible; the cascade makes that explicit).
1439
+ let out = s.set_verification_status("t3", Mismatch, 11).unwrap();
1440
+ assert_eq!(out.cascaded, vec!["t4"]);
1441
+ assert_eq!(out.certified_head.as_deref(), Some("t2"), "certified prefix survives a later mismatch");
1442
+
1443
+ // A mismatched turn is terminal.
1444
+ assert!(s.set_verification_status("t3", ReplicaMatched, 12).is_err());
1445
+ assert!(s.set_verification_status("t3", MintIneligible, 12).is_err());
1446
+ }
1447
+
1448
+ #[test]
1449
+ fn mismatch_cascade_invalidates_descendants() {
1450
+ use TurnVerificationStatus::*;
1451
+ let mut s = store_at("cascade");
1452
+ s.create_conversation("c1", "t", None, 1).unwrap();
1453
+ // t1 ← t2 ← t3, plus a sibling branch t2b under t1.
1454
+ for (id, parent, n) in [("t1", None, 2i64), ("t2", Some("t1"), 3), ("t3", Some("t2"), 4), ("t2b", Some("t1"), 5)] {
1455
+ s.begin_turn(&new_turn(id, "c1", parent, "m", &[1], TurnPrivacyMode::PalwMint, n)).unwrap();
1456
+ complete(&s, id, &[2], LocalComplete);
1457
+ }
1458
+ s.set_verification_status("t1", ReplicaPending, 6).unwrap();
1459
+ let out = s.set_verification_status("t1", Mismatch, 7).unwrap();
1460
+ let mut cascaded = out.cascaded.clone();
1461
+ cascaded.sort();
1462
+ assert_eq!(cascaded, vec!["t2", "t2b", "t3"], "every descendant leaves the reward pipeline");
1463
+ assert_eq!(s.get_turn("t2").unwrap().verification_status, MintIneligible);
1464
+ assert_eq!(s.get_turn("t3").unwrap().verification_status, MintIneligible);
1465
+ assert_eq!(s.get_turn("t2b").unwrap().verification_status, MintIneligible);
1466
+ // Local conversation survives: rows still exist with their text.
1467
+ assert_eq!(s.get_turn("t3").unwrap().display_text, "ok");
1468
+ }
1469
+ }
palw-gateway/src/tokenizer.rs ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Tokenizer host: the pinned `tokenizer.json`, the `encode_lossless` invariant carried over
2
+ //! from `qi35_chat.py`, and the incremental stream decoder (stable-tail hold-back) so SSE
3
+ //! deltas never emit a half-open UTF-8/BPE boundary.
4
+
5
+ use tokenizers::Tokenizer;
6
+
7
+ pub struct TokenizerHost {
8
+ inner: Tokenizer,
9
+ pub eos_id: u32,
10
+ pub digest_hex: String,
11
+ }
12
+
13
+ impl TokenizerHost {
14
+ pub fn load(path: &std::path::Path, eos_id: u32) -> Result<Self, String> {
15
+ let bytes = std::fs::read(path).map_err(|e| format!("read {}: {e}", path.display()))?;
16
+ let digest_hex = {
17
+ use blake2::{Blake2b, Digest, digest::consts::U32};
18
+ let mut h = Blake2b::<U32>::new();
19
+ h.update(&bytes);
20
+ hex::encode(h.finalize())
21
+ };
22
+ let inner = Tokenizer::from_bytes(&bytes).map_err(|e| format!("tokenizer {}: {e}", path.display()))?;
23
+ Ok(Self { inner, eos_id, digest_hex })
24
+ }
25
+
26
+ /// Encode with the qi35 invariant: the encoding must decode back to the exact input string,
27
+ /// or we refuse — a prompt we cannot commit losslessly must never reach the engine.
28
+ pub fn encode_lossless(&self, text: &str) -> Result<Vec<u32>, String> {
29
+ let enc = self.inner.encode(text, false).map_err(|e| format!("encode: {e}"))?;
30
+ let ids: Vec<u32> = enc.get_ids().to_vec();
31
+ let back = self.decode(&ids)?;
32
+ if back != text {
33
+ return Err(format!("encode is not lossless for this input ({} chars in, {} chars back)", text.len(), back.len()));
34
+ }
35
+ Ok(ids)
36
+ }
37
+
38
+ pub fn decode(&self, ids: &[u32]) -> Result<String, String> {
39
+ // skip_special_tokens = false: ChatML markers must round-trip for encode_lossless.
40
+ self.inner.decode(ids, false).map_err(|e| format!("decode: {e}"))
41
+ }
42
+
43
+ /// Decode for DISPLAY: special tokens dropped (an EOS at the end of a stored answer must
44
+ /// not render as `<|im_end|>` in the UI).
45
+ pub fn decode_display(&self, ids: &[u32]) -> Result<String, String> {
46
+ self.inner.decode(ids, true).map_err(|e| format!("decode: {e}"))
47
+ }
48
+ }
49
+
50
+ /// Streaming detokenizer with the stable-tail trick from `qi35_chat.py`'s IncrementalDecoder:
51
+ /// re-decode the full generated id list each token, but only emit up to `HOLD_BACK` characters
52
+ /// before the end — the held tail may still change while a multi-token grapheme or BPE merge is
53
+ /// open, the earlier prefix cannot.
54
+ pub struct IncrementalDecoder {
55
+ ids: Vec<u32>,
56
+ emitted_chars: usize,
57
+ }
58
+
59
+ const HOLD_BACK: usize = 8;
60
+
61
+ impl IncrementalDecoder {
62
+ pub fn new() -> Self {
63
+ Self { ids: Vec::new(), emitted_chars: 0 }
64
+ }
65
+
66
+ /// Push one generated token; returns the newly-stable UTF-8 delta (possibly empty).
67
+ pub fn push(&mut self, host: &TokenizerHost, token_id: u32) -> Result<String, String> {
68
+ self.ids.push(token_id);
69
+ let full = host.decode_display(&self.ids)?;
70
+ let chars: Vec<char> = full.chars().collect();
71
+ let stable_end = chars.len().saturating_sub(HOLD_BACK);
72
+ if stable_end <= self.emitted_chars {
73
+ return Ok(String::new());
74
+ }
75
+ let delta: String = chars[self.emitted_chars..stable_end].iter().collect();
76
+ self.emitted_chars = stable_end;
77
+ Ok(delta)
78
+ }
79
+
80
+ /// Flush the held tail at end of generation.
81
+ pub fn finish(&mut self, host: &TokenizerHost) -> Result<String, String> {
82
+ let full = host.decode_display(&self.ids)?;
83
+ let chars: Vec<char> = full.chars().collect();
84
+ if chars.len() <= self.emitted_chars {
85
+ return Ok(String::new());
86
+ }
87
+ let delta: String = chars[self.emitted_chars..].iter().collect();
88
+ self.emitted_chars = chars.len();
89
+ Ok(delta)
90
+ }
91
+ }
92
+
93
+ // Minimal local hex (avoids another dependency).
94
+ mod hex {
95
+ pub fn encode(bytes: impl AsRef<[u8]>) -> String {
96
+ bytes.as_ref().iter().map(|b| format!("{b:02x}")).collect()
97
+ }
98
+ }
palw-gateway/src/tools.rs ADDED
@@ -0,0 +1,572 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! Tool Runtime + sandbox (design doc §6 / §17).
2
+ //!
3
+ //! The doc's three-way split is the load-bearing structure:
4
+ //! * `DeterministicReplay` — pure functions of their arguments (calculator). A and B can both
5
+ //! re-run them; no snapshot is strictly needed but we snapshot anyway for a uniform ledger.
6
+ //! * `SnapshotReadOnly` — reads of external state (filesystem here; web/db later). Executed by
7
+ //! A once, result content-addressed; B reads the snapshot, never re-reads the world. PALW then
8
+ //! attests "answered FROM this data", not "this data is true".
9
+ //! * `ExactlyOnceSideEffect` — writes to the world. Never auto-executed: the call is stored as
10
+ //! `proposed`, the user approves (§6 step 2 — over this gateway's API the approval is an
11
+ //! explicit second request), and the store's `claim_tool_execution` UPDATE is the exactly-once
12
+ //! ticket. B replicas read the proposal and the result snapshot; they never re-fire.
13
+ //!
14
+ //! Sandbox (§17): read scope = workspace + explicitly configured readable roots, write scope =
15
+ //! the workspace only, `run_command` = an ABSOLUTE-PATH allowlist (no PATH lookup, so what runs
16
+ //! is exactly what the operator listed), cleared child environment (secret 遮断 — the gateway's
17
+ //! env never leaks into tools), wall-clock timeout, output byte cap. Paths are canonicalized
18
+ //! before the scope check, so symlinks cannot escape. v1 limitation, stated honestly: there is
19
+ //! no kernel-level network isolation for spawned commands — the allowlist IS the network policy
20
+ //! (list only binaries that do not reach the network if that matters to you).
21
+
22
+ use std::path::{Component, Path, PathBuf};
23
+ use std::time::{Duration, Instant};
24
+
25
+ use serde_json::Value;
26
+
27
+ #[derive(Clone, Copy, Debug, PartialEq, Eq)]
28
+ pub enum ToolExecutionMode {
29
+ DeterministicReplay,
30
+ SnapshotReadOnly,
31
+ ExactlyOnceSideEffect,
32
+ }
33
+
34
+ impl ToolExecutionMode {
35
+ pub fn as_str(self) -> &'static str {
36
+ match self {
37
+ Self::DeterministicReplay => "deterministic_replay",
38
+ Self::SnapshotReadOnly => "snapshot_read_only",
39
+ Self::ExactlyOnceSideEffect => "exactly_once_side_effect",
40
+ }
41
+ }
42
+ }
43
+
44
+ #[derive(Clone, Debug)]
45
+ pub struct ToolSandboxPolicyV1 {
46
+ /// Canonicalized. The write scope, `run_command`'s cwd, and the base for relative reads.
47
+ pub workspace_root: PathBuf,
48
+ /// Canonicalized, read-only. The workspace is always readable in addition to these.
49
+ pub readable_roots: Vec<PathBuf>,
50
+ /// Absolute binary paths; `run_command` refuses anything not EXACTLY in this list.
51
+ pub allowed_commands: Vec<PathBuf>,
52
+ pub max_runtime_ms: u64,
53
+ pub max_output_bytes: usize,
54
+ pub max_read_bytes: u64,
55
+ }
56
+
57
+ impl ToolSandboxPolicyV1 {
58
+ pub fn new(
59
+ workspace: &Path,
60
+ readable: &[PathBuf],
61
+ allowed_commands: &[PathBuf],
62
+ max_runtime_ms: u64,
63
+ max_output_bytes: usize,
64
+ max_read_bytes: u64,
65
+ ) -> Result<Self, String> {
66
+ std::fs::create_dir_all(workspace).map_err(|e| format!("create workspace {}: {e}", workspace.display()))?;
67
+ let workspace_root =
68
+ std::fs::canonicalize(workspace).map_err(|e| format!("canonicalize {}: {e}", workspace.display()))?;
69
+ let mut readable_roots = Vec::new();
70
+ for r in readable {
71
+ readable_roots.push(std::fs::canonicalize(r).map_err(|e| format!("readable root {}: {e}", r.display()))?);
72
+ }
73
+ for c in allowed_commands {
74
+ if !c.is_absolute() {
75
+ return Err(format!("allowed command must be an absolute path: {}", c.display()));
76
+ }
77
+ if !c.is_file() {
78
+ return Err(format!("allowed command does not exist: {}", c.display()));
79
+ }
80
+ }
81
+ Ok(Self {
82
+ workspace_root,
83
+ readable_roots,
84
+ allowed_commands: allowed_commands.to_vec(),
85
+ max_runtime_ms,
86
+ max_output_bytes,
87
+ max_read_bytes,
88
+ })
89
+ }
90
+ }
91
+
92
+ pub struct ToolOutcome {
93
+ /// Canonical result bytes — what gets content-addressed and (for mint turns) handed to B.
94
+ pub output: Vec<u8>,
95
+ pub mime: &'static str,
96
+ /// Short human-readable form for the tool-call ledger row.
97
+ pub preview: String,
98
+ }
99
+
100
+ pub struct ToolRuntime {
101
+ pub policy: ToolSandboxPolicyV1,
102
+ }
103
+
104
+ pub const TOOLS: &[(&str, ToolExecutionMode, &str)] = &[
105
+ ("calculator", ToolExecutionMode::DeterministicReplay, "exact i128 integer arithmetic: + - * / % and parentheses; / must divide exactly"),
106
+ ("read_file", ToolExecutionMode::SnapshotReadOnly, "read a file inside the readable scope; result is snapshotted"),
107
+ ("list_dir", ToolExecutionMode::SnapshotReadOnly, "list a directory inside the readable scope, sorted"),
108
+ ("write_file", ToolExecutionMode::ExactlyOnceSideEffect, "write a workspace-relative file (requires approval)"),
109
+ ("run_command", ToolExecutionMode::ExactlyOnceSideEffect, "run an allowlisted binary in the workspace (requires approval)"),
110
+ ];
111
+
112
+ pub fn mode_of(tool: &str) -> Option<ToolExecutionMode> {
113
+ TOOLS.iter().find(|(name, _, _)| *name == tool).map(|(_, mode, _)| *mode)
114
+ }
115
+
116
+ fn preview_of(text: &str) -> String {
117
+ const MAX: usize = 400;
118
+ if text.chars().count() <= MAX {
119
+ text.to_string()
120
+ } else {
121
+ let cut: String = text.chars().take(MAX).collect();
122
+ format!("{cut}…")
123
+ }
124
+ }
125
+
126
+ fn arg_str<'a>(args: &'a Value, key: &str) -> Result<&'a str, String> {
127
+ args.get(key).and_then(|v| v.as_str()).ok_or_else(|| format!("missing string argument {key:?}"))
128
+ }
129
+
130
+ impl ToolRuntime {
131
+ pub fn new(policy: ToolSandboxPolicyV1) -> Self {
132
+ Self { policy }
133
+ }
134
+
135
+ /// Execute a tool NOW. The caller is responsible for the mode gate: side-effect tools reach
136
+ /// this only through the store's approved→executing claim.
137
+ pub fn execute(&self, tool: &str, args: &Value) -> Result<ToolOutcome, String> {
138
+ match tool {
139
+ "calculator" => self.calculator(args),
140
+ "read_file" => self.read_file(args),
141
+ "list_dir" => self.list_dir(args),
142
+ "write_file" => self.write_file(args),
143
+ "run_command" => self.run_command(args),
144
+ other => Err(format!("unknown tool {other:?}")),
145
+ }
146
+ }
147
+
148
+ // ---- path scope ---------------------------------------------------------------------
149
+
150
+ /// Resolve a read path: relative ⇒ workspace-joined; then canonicalize (symlinks resolve
151
+ /// HERE, before the scope check) and require the result inside workspace or a readable root.
152
+ fn resolve_readable(&self, path: &str) -> Result<PathBuf, String> {
153
+ let joined =
154
+ if Path::new(path).is_absolute() { PathBuf::from(path) } else { self.policy.workspace_root.join(path) };
155
+ let canon = std::fs::canonicalize(&joined).map_err(|e| format!("{path:?}: {e}"))?;
156
+ let in_scope = canon.starts_with(&self.policy.workspace_root)
157
+ || self.policy.readable_roots.iter().any(|r| canon.starts_with(r));
158
+ if !in_scope {
159
+ return Err(format!("{path:?} resolves outside the readable scope"));
160
+ }
161
+ Ok(canon)
162
+ }
163
+
164
+ /// Validate a WRITE path: must be relative, no `..`/root components; the (created) parent
165
+ /// must canonicalize back inside the workspace, so a symlinked subdirectory cannot redirect
166
+ /// the write outside.
167
+ fn resolve_writable(&self, path: &str) -> Result<PathBuf, String> {
168
+ let p = Path::new(path);
169
+ if p.is_absolute() {
170
+ return Err("write paths must be workspace-relative".into());
171
+ }
172
+ if p.components().any(|c| !matches!(c, Component::Normal(_) | Component::CurDir)) {
173
+ return Err("write paths may not contain '..' or root components".into());
174
+ }
175
+ if p.as_os_str().is_empty() {
176
+ return Err("empty write path".into());
177
+ }
178
+ let target = self.policy.workspace_root.join(p);
179
+ let parent = target.parent().ok_or("write path has no parent")?;
180
+ std::fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?;
181
+ let canon_parent = std::fs::canonicalize(parent).map_err(|e| format!("canonicalize {}: {e}", parent.display()))?;
182
+ if !canon_parent.starts_with(&self.policy.workspace_root) {
183
+ return Err(format!("{path:?} escapes the workspace through a symlinked directory"));
184
+ }
185
+ let file_name = target.file_name().ok_or("write path has no file name")?;
186
+ Ok(canon_parent.join(file_name))
187
+ }
188
+
189
+ // ---- tools --------------------------------------------------------------------------
190
+
191
+ fn calculator(&self, args: &Value) -> Result<ToolOutcome, String> {
192
+ let expression = arg_str(args, "expression")?;
193
+ let value = calc_eval(expression)?;
194
+ let text = value.to_string();
195
+ Ok(ToolOutcome { preview: preview_of(&text), output: text.into_bytes(), mime: "text/plain" })
196
+ }
197
+
198
+ fn read_file(&self, args: &Value) -> Result<ToolOutcome, String> {
199
+ let path = arg_str(args, "path")?;
200
+ let canon = self.resolve_readable(path)?;
201
+ let len = std::fs::metadata(&canon).map_err(|e| format!("{path:?}: {e}"))?.len();
202
+ if len > self.policy.max_read_bytes {
203
+ return Err(format!("{path:?} is {len} bytes — over the {}-byte read cap", self.policy.max_read_bytes));
204
+ }
205
+ let bytes = std::fs::read(&canon).map_err(|e| format!("{path:?}: {e}"))?;
206
+ let preview = match std::str::from_utf8(&bytes) {
207
+ Ok(text) => preview_of(text),
208
+ Err(_) => format!("({len} binary bytes)"),
209
+ };
210
+ Ok(ToolOutcome { output: bytes, mime: "application/octet-stream", preview })
211
+ }
212
+
213
+ fn list_dir(&self, args: &Value) -> Result<ToolOutcome, String> {
214
+ let path = arg_str(args, "path")?;
215
+ let canon = self.resolve_readable(path)?;
216
+ let mut lines = Vec::new();
217
+ for entry in std::fs::read_dir(&canon).map_err(|e| format!("{path:?}: {e}"))? {
218
+ let entry = entry.map_err(|e| e.to_string())?;
219
+ let name = entry.file_name().to_string_lossy().into_owned();
220
+ let meta = entry.metadata().map_err(|e| e.to_string())?;
221
+ let kind = if meta.is_dir() { "dir" } else { "file" };
222
+ lines.push(format!("{name}\t{kind}\t{}", if meta.is_dir() { 0 } else { meta.len() }));
223
+ }
224
+ lines.sort(); // deterministic listing — read_dir order is filesystem-dependent
225
+ let text = lines.join("\n");
226
+ Ok(ToolOutcome { preview: preview_of(&text), output: text.into_bytes(), mime: "text/plain" })
227
+ }
228
+
229
+ fn write_file(&self, args: &Value) -> Result<ToolOutcome, String> {
230
+ let path = arg_str(args, "path")?;
231
+ let content = arg_str(args, "content")?;
232
+ let target = self.resolve_writable(path)?;
233
+ std::fs::write(&target, content.as_bytes()).map_err(|e| format!("write {path:?}: {e}"))?;
234
+ let hash = crate::artifact::content_hash_hex(content.as_bytes());
235
+ let text = format!("wrote {} bytes to {path}\ncontent blake2b-256 {hash}", content.len());
236
+ Ok(ToolOutcome { preview: preview_of(&text), output: text.into_bytes(), mime: "text/plain" })
237
+ }
238
+
239
+ fn run_command(&self, args: &Value) -> Result<ToolOutcome, String> {
240
+ let command = arg_str(args, "command")?;
241
+ let argv: Vec<String> = match args.get("args") {
242
+ None => Vec::new(),
243
+ Some(Value::Array(items)) => items
244
+ .iter()
245
+ .map(|v| v.as_str().map(String::from).ok_or_else(|| "args must be strings".to_string()))
246
+ .collect::<Result<_, _>>()?,
247
+ Some(_) => return Err("args must be an array of strings".into()),
248
+ };
249
+ let command_path = Path::new(command);
250
+ if !self.policy.allowed_commands.iter().any(|allowed| allowed == command_path) {
251
+ return Err(format!("{command:?} is not on the command allowlist"));
252
+ }
253
+
254
+ // Cleared environment: the gateway's env (tokens, keys) never reaches a tool child.
255
+ // PATH is pinned so anything the child itself spawns is at least system-scoped, and
256
+ // LC_ALL=C keeps tool output locale-independent (deterministic snapshots).
257
+ let mut child = std::process::Command::new(command_path)
258
+ .args(&argv)
259
+ .current_dir(&self.policy.workspace_root)
260
+ .env_clear()
261
+ .env("PATH", "/usr/bin:/bin")
262
+ .env("LC_ALL", "C")
263
+ .stdin(std::process::Stdio::null())
264
+ .stdout(std::process::Stdio::piped())
265
+ .stderr(std::process::Stdio::piped())
266
+ .spawn()
267
+ .map_err(|e| format!("spawn {command}: {e}"))?;
268
+
269
+ let cap = self.policy.max_output_bytes;
270
+ let stdout = child.stdout.take().expect("piped");
271
+ let stderr = child.stderr.take().expect("piped");
272
+ let out_reader = std::thread::spawn(move || read_capped(stdout, cap));
273
+ let err_reader = std::thread::spawn(move || read_capped(stderr, cap));
274
+
275
+ let deadline = Instant::now() + Duration::from_millis(self.policy.max_runtime_ms);
276
+ let mut timed_out = false;
277
+ let status = loop {
278
+ match child.try_wait().map_err(|e| e.to_string())? {
279
+ Some(status) => break Some(status),
280
+ None if Instant::now() >= deadline => {
281
+ let _ = child.kill();
282
+ let _ = child.wait();
283
+ timed_out = true;
284
+ break None;
285
+ }
286
+ None => std::thread::sleep(Duration::from_millis(10)),
287
+ }
288
+ };
289
+ let (stdout_bytes, stdout_truncated) = out_reader.join().map_err(|_| "stdout reader panicked")?;
290
+ let (stderr_bytes, stderr_truncated) = err_reader.join().map_err(|_| "stderr reader panicked")?;
291
+
292
+ let exit = if timed_out {
293
+ format!("timeout after {}ms", self.policy.max_runtime_ms)
294
+ } else {
295
+ match status.and_then(|s| s.code()) {
296
+ Some(code) => format!("exit {code}"),
297
+ None => "killed by signal".into(),
298
+ }
299
+ };
300
+ let mut text = format!("{exit}\n--- stdout ---\n");
301
+ text.push_str(&String::from_utf8_lossy(&stdout_bytes));
302
+ if stdout_truncated {
303
+ text.push_str("\n[stdout truncated]");
304
+ }
305
+ text.push_str("\n--- stderr ---\n");
306
+ text.push_str(&String::from_utf8_lossy(&stderr_bytes));
307
+ if stderr_truncated {
308
+ text.push_str("\n[stderr truncated]");
309
+ }
310
+ Ok(ToolOutcome { preview: preview_of(&text), output: text.into_bytes(), mime: "text/plain" })
311
+ }
312
+ }
313
+
314
+ /// Read a pipe to EOF, keeping at most `cap` bytes. Draining past the cap (instead of stopping)
315
+ /// keeps the child from blocking on a full pipe after we stop caring.
316
+ fn read_capped(mut reader: impl std::io::Read, cap: usize) -> (Vec<u8>, bool) {
317
+ let mut kept = Vec::new();
318
+ let mut truncated = false;
319
+ let mut buf = [0u8; 8192];
320
+ loop {
321
+ match reader.read(&mut buf) {
322
+ Ok(0) | Err(_) => break,
323
+ Ok(n) => {
324
+ if kept.len() < cap {
325
+ let take = n.min(cap - kept.len());
326
+ kept.extend_from_slice(&buf[..take]);
327
+ if take < n {
328
+ truncated = true;
329
+ }
330
+ } else {
331
+ truncated = true;
332
+ }
333
+ }
334
+ }
335
+ }
336
+ (kept, truncated)
337
+ }
338
+
339
+ // ---- calculator (DeterministicReplay) ---------------------------------------------------
340
+ //
341
+ // Exact i128 integer arithmetic, checked everywhere: overflow, division by zero, and inexact
342
+ // division are ERRORS, never wrapped or rounded — a deterministic-replay tool must have exactly
343
+ // one right answer on every input, on every machine.
344
+
345
+ const CALC_MAX_LEN: usize = 4096;
346
+ const CALC_MAX_DEPTH: u32 = 64;
347
+
348
+ struct CalcParser<'a> {
349
+ bytes: &'a [u8],
350
+ pos: usize,
351
+ }
352
+
353
+ pub fn calc_eval(expression: &str) -> Result<i128, String> {
354
+ if expression.len() > CALC_MAX_LEN {
355
+ return Err(format!("expression longer than {CALC_MAX_LEN} bytes"));
356
+ }
357
+ let mut p = CalcParser { bytes: expression.as_bytes(), pos: 0 };
358
+ let v = p.expr(0)?;
359
+ p.skip_ws();
360
+ if p.pos != p.bytes.len() {
361
+ return Err(format!("unexpected trailing input at byte {}", p.pos));
362
+ }
363
+ Ok(v)
364
+ }
365
+
366
+ impl<'a> CalcParser<'a> {
367
+ fn skip_ws(&mut self) {
368
+ while self.bytes.get(self.pos) == Some(&b' ') {
369
+ self.pos += 1;
370
+ }
371
+ }
372
+
373
+ fn peek(&mut self) -> Option<u8> {
374
+ self.skip_ws();
375
+ self.bytes.get(self.pos).copied()
376
+ }
377
+
378
+ fn expr(&mut self, depth: u32) -> Result<i128, String> {
379
+ let mut acc = self.term(depth)?;
380
+ loop {
381
+ match self.peek() {
382
+ Some(b'+') => {
383
+ self.pos += 1;
384
+ let rhs = self.term(depth)?;
385
+ acc = acc.checked_add(rhs).ok_or("overflow in +")?;
386
+ }
387
+ Some(b'-') => {
388
+ self.pos += 1;
389
+ let rhs = self.term(depth)?;
390
+ acc = acc.checked_sub(rhs).ok_or("overflow in -")?;
391
+ }
392
+ _ => return Ok(acc),
393
+ }
394
+ }
395
+ }
396
+
397
+ fn term(&mut self, depth: u32) -> Result<i128, String> {
398
+ let mut acc = self.factor(depth)?;
399
+ loop {
400
+ match self.peek() {
401
+ Some(b'*') => {
402
+ self.pos += 1;
403
+ let rhs = self.factor(depth)?;
404
+ acc = acc.checked_mul(rhs).ok_or("overflow in *")?;
405
+ }
406
+ Some(b'/') => {
407
+ self.pos += 1;
408
+ let rhs = self.factor(depth)?;
409
+ if rhs == 0 {
410
+ return Err("division by zero".into());
411
+ }
412
+ let rem = acc.checked_rem(rhs).ok_or("overflow in /")?;
413
+ if rem != 0 {
414
+ return Err(format!("{acc}/{rhs} is not exact — integer calculator refuses to round"));
415
+ }
416
+ acc = acc.checked_div(rhs).ok_or("overflow in /")?;
417
+ }
418
+ Some(b'%') => {
419
+ self.pos += 1;
420
+ let rhs = self.factor(depth)?;
421
+ if rhs == 0 {
422
+ return Err("modulo by zero".into());
423
+ }
424
+ acc = acc.checked_rem(rhs).ok_or("overflow in %")?;
425
+ }
426
+ _ => return Ok(acc),
427
+ }
428
+ }
429
+ }
430
+
431
+ fn factor(&mut self, depth: u32) -> Result<i128, String> {
432
+ if depth > CALC_MAX_DEPTH {
433
+ return Err("expression nests too deep".into());
434
+ }
435
+ match self.peek() {
436
+ Some(b'-') => {
437
+ self.pos += 1;
438
+ let v = self.factor(depth + 1)?;
439
+ v.checked_neg().ok_or_else(|| "overflow in unary -".into())
440
+ }
441
+ Some(b'(') => {
442
+ self.pos += 1;
443
+ let v = self.expr(depth + 1)?;
444
+ if self.peek() != Some(b')') {
445
+ return Err(format!("expected ')' at byte {}", self.pos));
446
+ }
447
+ self.pos += 1;
448
+ Ok(v)
449
+ }
450
+ Some(c) if c.is_ascii_digit() => {
451
+ let start = self.pos;
452
+ while matches!(self.bytes.get(self.pos), Some(d) if d.is_ascii_digit()) {
453
+ self.pos += 1;
454
+ }
455
+ let text = std::str::from_utf8(&self.bytes[start..self.pos]).expect("digits are ascii");
456
+ text.parse::<i128>().map_err(|_| format!("number too large: {text}"))
457
+ }
458
+ other => Err(format!("unexpected input {:?} at byte {}", other.map(|c| c as char), self.pos)),
459
+ }
460
+ }
461
+ }
462
+
463
+ #[cfg(test)]
464
+ mod tests {
465
+ use super::*;
466
+ use serde_json::json;
467
+
468
+ fn runtime(name: &str, commands: &[PathBuf]) -> (ToolRuntime, PathBuf) {
469
+ let dir = std::env::temp_dir().join(format!("palw-gw-tools-{}-{name}", std::process::id()));
470
+ let _ = std::fs::remove_dir_all(&dir);
471
+ std::fs::create_dir_all(&dir).unwrap();
472
+ let policy = ToolSandboxPolicyV1::new(&dir, &[], commands, 2_000, 1_000, 1 << 20).unwrap();
473
+ let ws = policy.workspace_root.clone();
474
+ (ToolRuntime::new(policy), ws)
475
+ }
476
+
477
+ #[test]
478
+ fn calculator_is_exact_and_total() {
479
+ assert_eq!(calc_eval("2+3*4").unwrap(), 14);
480
+ assert_eq!(calc_eval("(2+3)*4").unwrap(), 20);
481
+ assert_eq!(calc_eval("-5+2").unwrap(), -3);
482
+ assert_eq!(calc_eval("10/2").unwrap(), 5);
483
+ assert_eq!(calc_eval("7%2").unwrap(), 1);
484
+ assert_eq!(calc_eval(" 1 + 2 ").unwrap(), 3);
485
+ assert!(calc_eval("7/2").is_err(), "inexact division must refuse, not round");
486
+ assert!(calc_eval("1/0").is_err());
487
+ assert!(calc_eval("170141183460469231731687303715884105727+1").is_err(), "overflow is an error");
488
+ assert!(calc_eval("2+").is_err());
489
+ assert!(calc_eval("").is_err());
490
+ let bomb = "(".repeat(200) + "1" + &")".repeat(200);
491
+ assert!(calc_eval(&bomb).is_err(), "depth-capped");
492
+ }
493
+
494
+ #[test]
495
+ fn read_scope_blocks_escape_and_symlinks() {
496
+ let (rt, ws) = runtime("readscope", &[]);
497
+ std::fs::write(ws.join("ok.txt"), b"fine").unwrap();
498
+ let out = rt.execute("read_file", &json!({"path": "ok.txt"})).unwrap();
499
+ assert_eq!(out.output, b"fine");
500
+
501
+ assert!(rt.execute("read_file", &json!({"path": "../outside.txt"})).is_err());
502
+ assert!(rt.execute("read_file", &json!({"path": "/etc/hosts"})).is_err(), "absolute outside scope refused");
503
+
504
+ // A symlink INSIDE the workspace pointing outside must not read outside.
505
+ std::os::unix::fs::symlink("/etc/hosts", ws.join("sneaky")).unwrap();
506
+ assert!(rt.execute("read_file", &json!({"path": "sneaky"})).is_err(), "symlink escape refused");
507
+ }
508
+
509
+ #[test]
510
+ fn write_scope_is_workspace_relative_only() {
511
+ let (rt, ws) = runtime("writescope", &[]);
512
+ let out = rt.execute("write_file", &json!({"path": "sub/dir/x.txt", "content": "hello"})).unwrap();
513
+ assert!(String::from_utf8_lossy(&out.output).contains("wrote 5 bytes"));
514
+ assert_eq!(std::fs::read_to_string(ws.join("sub/dir/x.txt")).unwrap(), "hello");
515
+
516
+ assert!(rt.execute("write_file", &json!({"path": "/tmp/abs.txt", "content": "x"})).is_err());
517
+ assert!(rt.execute("write_file", &json!({"path": "../up.txt", "content": "x"})).is_err());
518
+
519
+ // A symlinked subdirectory pointing outside must not redirect the write.
520
+ let outside = std::env::temp_dir().join(format!("palw-gw-tools-outside-{}", std::process::id()));
521
+ std::fs::create_dir_all(&outside).unwrap();
522
+ std::os::unix::fs::symlink(&outside, ws.join("link")).unwrap();
523
+ assert!(rt.execute("write_file", &json!({"path": "link/y.txt", "content": "x"})).is_err());
524
+ }
525
+
526
+ #[test]
527
+ fn list_dir_is_sorted() {
528
+ let (rt, ws) = runtime("listdir", &[]);
529
+ std::fs::write(ws.join("b.txt"), b"2").unwrap();
530
+ std::fs::write(ws.join("a.txt"), b"1").unwrap();
531
+ std::fs::create_dir(ws.join("c")).unwrap();
532
+ let out = rt.execute("list_dir", &json!({"path": "."})).unwrap();
533
+ let text = String::from_utf8(out.output).unwrap();
534
+ assert_eq!(text, "a.txt\tfile\t1\nb.txt\tfile\t1\nc\tdir\t0");
535
+ }
536
+
537
+ #[test]
538
+ fn run_command_allowlist_timeout_and_cap() {
539
+ let echo = PathBuf::from("/bin/echo");
540
+ let sleep = PathBuf::from("/bin/sleep");
541
+ let (rt, _ws) = runtime("runcmd", &[echo.clone(), sleep.clone()]);
542
+
543
+ assert!(rt.execute("run_command", &json!({"command": "/bin/ls"})).is_err(), "not allowlisted");
544
+ assert!(rt.execute("run_command", &json!({"command": "echo"})).is_err(), "bare names never match");
545
+
546
+ let out = rt.execute("run_command", &json!({"command": "/bin/echo", "args": ["hello", "world"]})).unwrap();
547
+ let text = String::from_utf8(out.output).unwrap();
548
+ assert!(text.starts_with("exit 0\n"), "got: {text}");
549
+ assert!(text.contains("hello world"));
550
+
551
+ // Timeout: sleep 5 against a 2s policy cap — expect the timeout marker well before 5s.
552
+ let started = Instant::now();
553
+ let out = rt.execute("run_command", &json!({"command": "/bin/sleep", "args": ["5"]})).unwrap();
554
+ assert!(started.elapsed() < Duration::from_secs(4));
555
+ assert!(String::from_utf8_lossy(&out.output).starts_with("timeout after"));
556
+
557
+ // Output cap: 1000-byte policy cap against ~2000 bytes of stdout.
558
+ let big = "x".repeat(2000);
559
+ let out = rt.execute("run_command", &json!({"command": "/bin/echo", "args": [big]})).unwrap();
560
+ let text = String::from_utf8_lossy(&out.output).into_owned();
561
+ assert!(text.contains("[stdout truncated]"));
562
+ }
563
+
564
+ #[test]
565
+ fn modes_match_the_frozen_split() {
566
+ assert_eq!(mode_of("calculator"), Some(ToolExecutionMode::DeterministicReplay));
567
+ assert_eq!(mode_of("read_file"), Some(ToolExecutionMode::SnapshotReadOnly));
568
+ assert_eq!(mode_of("write_file"), Some(ToolExecutionMode::ExactlyOnceSideEffect));
569
+ assert_eq!(mode_of("run_command"), Some(ToolExecutionMode::ExactlyOnceSideEffect));
570
+ assert_eq!(mode_of("nope"), None);
571
+ }
572
+ }
palw-gateway/src/worker.rs ADDED
@@ -0,0 +1,496 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! PALW Background Worker (design doc §15) — the settlement-clock driver.
2
+ //!
3
+ //! Deliberate shape: the worker is a STATELESS RECONCILER. It keeps no durable state of its
4
+ //! own — the session store is the single source of truth, and every cycle recomputes what needs
5
+ //! doing from it:
6
+ //!
7
+ //! ```text
8
+ //! 1. A-commit : turns local_complete ∧ mint-intent → submit_job (idempotent) → replica_pending
9
+ //! 2. verdicts : turns in the pipeline → fetch_verdicts → legal transitions
10
+ //! 3. B replicas : fetch_assignments → admission probe → execute full replay → result root
11
+ //! └─ refused/failed → decline (refusing beats no-showing)
12
+ //! ```
13
+ //!
14
+ //! §15's "timeout/retry, restart recovery" fall out of this construction instead of being
15
+ //! features: a crash between submit and the store write is healed by the next cycle's
16
+ //! idempotent re-submit; a coordinator outage just leaves rows where they are (each cycle
17
+ //! degrades to a logged error and tries again).
18
+ //!
19
+ //! What the worker does NOT do (honest boundary): no beacons, no bonds, no DA retention, no
20
+ //! reward settlement — those are consensus-side. `Matured` is chain state (coinbase maturity)
21
+ //! and is never synthesized here; the loopback coordinator stops at `certified`.
22
+
23
+ use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering};
24
+ use std::sync::{Arc, Mutex};
25
+
26
+ use crate::coordinator::{JobSubmission, JobVerdict, PalwCoordinator, ReplicaResultV1, output_root};
27
+ use crate::events::RuntimeRootsV1;
28
+ use crate::events::TurnVerificationStatus;
29
+ use crate::scheduler::{JobClass, ResourceScheduler};
30
+ use crate::store::SessionStore;
31
+
32
+ pub struct WorkerConfig {
33
+ pub provider_id: String,
34
+ pub poll_ms: u64,
35
+ /// Advertised on A-commits so the replica knows the generation bound. NOTE: the engine's
36
+ /// serve protocol has no per-request max_new — a replica honors its OWN engine's bound, so
37
+ /// matching deployments must run the same --max-new (loopback trivially does).
38
+ pub max_new: u32,
39
+ }
40
+
41
+ #[derive(Default)]
42
+ pub struct WorkerStatus {
43
+ pub submitted: AtomicU64,
44
+ pub verdicts_applied: AtomicU64,
45
+ pub replicas_executed: AtomicU64,
46
+ pub replicas_declined: AtomicU64,
47
+ pub errors: AtomicU64,
48
+ pub last_cycle_unix_ms: AtomicI64,
49
+ pub last_error: Mutex<Option<String>>,
50
+ }
51
+
52
+ impl WorkerStatus {
53
+ fn record_error(&self, context: &str, error: &str) {
54
+ self.errors.fetch_add(1, Ordering::Relaxed);
55
+ *self.last_error.lock().unwrap() = Some(format!("{context}: {error}"));
56
+ eprintln!("[palw-worker] {context}: {error}");
57
+ }
58
+ }
59
+
60
+ /// What a replica execution yields: the output ids plus (when the engine emits them) the
61
+ /// execution roots the match key covers.
62
+ pub struct ReplicaOutput {
63
+ pub output_ids: Vec<u32>,
64
+ pub runtime_roots: Option<RuntimeRootsV1>,
65
+ }
66
+
67
+ /// How a replica actually runs — trait so the reconciler is testable without a 35B engine.
68
+ pub trait ReplicaExecutor: Send + Sync {
69
+ fn execute(&self, prompt_ids: &[u32], max_new: u32) -> Result<ReplicaOutput, String>;
70
+ }
71
+
72
+ /// The real thing: a §5-v1 Full Context Job on the resident engine, behind the §14 scheduler
73
+ /// as `ReplicaJob` priority (a foreground turn arriving first goes first). `cache_prefix_len=0`
74
+ /// asks the engine to snapshot nothing extra; NOTE the engine still reuses any matching prefix
75
+ /// it happens to hold (deterministic, so the output is identical either way) — which is exactly
76
+ /// why single-process loopback replicas prove plumbing, not independence.
77
+ pub struct EngineReplicaExecutor {
78
+ pub engine: Arc<Mutex<crate::engine::EngineHost>>,
79
+ pub scheduler: Arc<ResourceScheduler>,
80
+ }
81
+
82
+ impl ReplicaExecutor for EngineReplicaExecutor {
83
+ fn execute(&self, prompt_ids: &[u32], _max_new: u32) -> Result<ReplicaOutput, String> {
84
+ // _max_new: the serve protocol has no per-request bound — the engine's own --max-new
85
+ // applies (see WorkerConfig::max_new note).
86
+ let _lease = self.scheduler.acquire(JobClass::ReplicaJob);
87
+ let mut engine = self.engine.lock().unwrap();
88
+ let cancel = Arc::new(AtomicBool::new(false));
89
+ let outcome = engine.generate(prompt_ids, 0, &cancel, |_, _| {})?;
90
+ self.scheduler.record_generation(outcome.output_ids.len() as u64, outcome.elapsed_ms);
91
+ Ok(ReplicaOutput { output_ids: outcome.output_ids, runtime_roots: outcome.runtime_roots })
92
+ }
93
+ }
94
+
95
+ /// The legal-transition path from the turn's current status to what a verdict implies. The
96
+ /// store enforces legality anyway; this keeps the worker from ever ASKING for an illegal jump
97
+ /// (e.g. a `certified` verdict against a still-replica_pending turn walks matched→certified).
98
+ pub fn steps_to(current: TurnVerificationStatus, verdict: JobVerdict) -> Vec<TurnVerificationStatus> {
99
+ use TurnVerificationStatus as S;
100
+ match verdict {
101
+ JobVerdict::ReplicaMatched => match current {
102
+ S::ReplicaPending => vec![S::ReplicaMatched],
103
+ _ => vec![],
104
+ },
105
+ JobVerdict::Certified => match current {
106
+ S::ReplicaPending => vec![S::ReplicaMatched, S::Certified],
107
+ S::ReplicaMatched | S::AuditPending => vec![S::Certified],
108
+ _ => vec![],
109
+ },
110
+ JobVerdict::Mismatch => match current {
111
+ S::ReplicaPending | S::ReplicaMatched | S::AuditPending => vec![S::Mismatch],
112
+ _ => vec![],
113
+ },
114
+ }
115
+ }
116
+
117
+ fn now_unix_ms() -> i64 {
118
+ std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as i64).unwrap_or(0)
119
+ }
120
+
121
+ /// One reconciliation cycle. Locks are held only around store reads/writes — never across a
122
+ /// coordinator call or a replica execution.
123
+ pub fn run_cycle(
124
+ store: &Mutex<SessionStore>,
125
+ scheduler: &ResourceScheduler,
126
+ coordinator: &dyn PalwCoordinator,
127
+ executor: &dyn ReplicaExecutor,
128
+ config: &WorkerConfig,
129
+ status: &WorkerStatus,
130
+ ) {
131
+ status.last_cycle_unix_ms.store(now_unix_ms(), Ordering::Relaxed);
132
+
133
+ // ---- 1. A-commits -------------------------------------------------------------------
134
+ let awaiting = match store.lock().unwrap().turns_awaiting_palw_submission() {
135
+ Ok(t) => t,
136
+ Err(e) => {
137
+ status.record_error("scan awaiting", &e);
138
+ Vec::new()
139
+ }
140
+ };
141
+ for turn in awaiting {
142
+ let submission = JobSubmission {
143
+ job_id: turn.turn_id.clone(),
144
+ provider_id: config.provider_id.clone(),
145
+ prompt_ids: turn.prompt_ids.clone(),
146
+ max_new: config.max_new,
147
+ output_root: output_root(&turn.output_ids),
148
+ receipt_json: turn.receipt_json.clone(),
149
+ runtime_roots: turn.runtime_roots_json.as_deref().and_then(|j| serde_json::from_str(j).ok()),
150
+ };
151
+ match coordinator.submit_job(&submission) {
152
+ Ok(()) => {
153
+ match store.lock().unwrap().set_verification_status(
154
+ &turn.turn_id,
155
+ TurnVerificationStatus::ReplicaPending,
156
+ now_unix_ms(),
157
+ ) {
158
+ Ok(_) => {
159
+ status.submitted.fetch_add(1, Ordering::Relaxed);
160
+ }
161
+ Err(e) => status.record_error("mark replica_pending", &e),
162
+ }
163
+ }
164
+ Err(e) => {
165
+ // A PROTOCOL rejection (our HttpCoordinator formats these as "...: 400 ...") is
166
+ // permanent — the coordinator will refuse this submission forever (e.g. a turn
167
+ // generated before ROOTS capture against a class-strict bridge). Retrying every
168
+ // cycle is noise; the turn leaves the mint path. Transport errors keep retrying.
169
+ if e.contains(": 400 ") {
170
+ let _ = store.lock().unwrap().set_verification_status(
171
+ &turn.turn_id,
172
+ TurnVerificationStatus::MintIneligible,
173
+ now_unix_ms(),
174
+ );
175
+ status.record_error("submit rejected permanently — turn marked mint_ineligible", &e);
176
+ } else {
177
+ status.record_error("submit job", &e);
178
+ }
179
+ }
180
+ }
181
+ }
182
+
183
+ // ---- 2. verdicts --------------------------------------------------------------------
184
+ let pending = match store.lock().unwrap().turns_in_palw_pipeline() {
185
+ Ok(t) => t,
186
+ Err(e) => {
187
+ status.record_error("scan pipeline", &e);
188
+ Vec::new()
189
+ }
190
+ };
191
+ if !pending.is_empty() {
192
+ let ids: Vec<String> = pending.iter().map(|t| t.turn_id.clone()).collect();
193
+ match coordinator.fetch_verdicts(&ids) {
194
+ Ok(verdicts) => {
195
+ for turn in &pending {
196
+ let Some(verdict) = verdicts.get(&turn.turn_id) else { continue };
197
+ for step in steps_to(turn.verification_status, *verdict) {
198
+ match store.lock().unwrap().set_verification_status(&turn.turn_id, step, now_unix_ms()) {
199
+ Ok(_) => {
200
+ status.verdicts_applied.fetch_add(1, Ordering::Relaxed);
201
+ }
202
+ Err(e) => {
203
+ status.record_error("apply verdict", &e);
204
+ break;
205
+ }
206
+ }
207
+ }
208
+ }
209
+ }
210
+ Err(e) => status.record_error("fetch verdicts", &e),
211
+ }
212
+ }
213
+
214
+ // ---- 3. B replicas ------------------------------------------------------------------
215
+ let assignments = match coordinator.fetch_assignments(&config.provider_id) {
216
+ Ok(a) => a,
217
+ Err(e) => {
218
+ status.record_error("fetch assignments", &e);
219
+ Vec::new()
220
+ }
221
+ };
222
+ for assignment in assignments {
223
+ let remaining_ms = assignment.deadline_unix_ms - now_unix_ms();
224
+ let admit = if remaining_ms <= 0 {
225
+ Err("deadline already passed".to_string())
226
+ } else {
227
+ scheduler.try_admit(JobClass::ReplicaJob, u64::from(assignment.max_new), remaining_ms as u64)
228
+ };
229
+ if let Err(reason) = admit {
230
+ status.replicas_declined.fetch_add(1, Ordering::Relaxed);
231
+ if let Err(e) = coordinator.decline_assignment(&assignment.job_id, &config.provider_id, &reason) {
232
+ status.record_error("decline", &e);
233
+ }
234
+ continue;
235
+ }
236
+ match executor.execute(&assignment.prompt_ids, assignment.max_new) {
237
+ Ok(output) => {
238
+ let result = ReplicaResultV1 {
239
+ job_id: assignment.job_id.clone(),
240
+ provider_id: config.provider_id.clone(),
241
+ output_root: output_root(&output.output_ids),
242
+ runtime_roots: output.runtime_roots,
243
+ };
244
+ match coordinator.submit_replica_result(&result) {
245
+ Ok(()) => {
246
+ status.replicas_executed.fetch_add(1, Ordering::Relaxed);
247
+ }
248
+ Err(e) => status.record_error("submit replica result", &e),
249
+ }
250
+ }
251
+ Err(e) => {
252
+ status.replicas_declined.fetch_add(1, Ordering::Relaxed);
253
+ let reason = format!("execution failed: {e}");
254
+ if let Err(decline_err) = coordinator.decline_assignment(&assignment.job_id, &config.provider_id, &reason)
255
+ {
256
+ status.record_error("decline after failure", &decline_err);
257
+ }
258
+ status.record_error("replica execution", &e);
259
+ }
260
+ }
261
+ }
262
+ }
263
+
264
+ pub struct WorkerHandle {
265
+ stop: Arc<AtomicBool>,
266
+ // The gateway binary never returns from serve(), so only tests and future embedders join
267
+ // the thread — the field is dead code from the bin's point of view, kept for them.
268
+ #[allow(dead_code)]
269
+ join: Option<std::thread::JoinHandle<()>>,
270
+ }
271
+
272
+ impl WorkerHandle {
273
+ #[allow(dead_code)]
274
+ pub fn shutdown(mut self) {
275
+ self.stop.store(true, Ordering::Relaxed);
276
+ if let Some(join) = self.join.take() {
277
+ let _ = join.join();
278
+ }
279
+ }
280
+ }
281
+
282
+ impl Drop for WorkerHandle {
283
+ fn drop(&mut self) {
284
+ self.stop.store(true, Ordering::Relaxed);
285
+ }
286
+ }
287
+
288
+ /// Spawn the reconciler thread. Dependencies come in as Arcs so the caller (main) can share
289
+ /// the gateway's own store/scheduler.
290
+ pub fn spawn(
291
+ store: Arc<Mutex<SessionStore>>,
292
+ scheduler: Arc<ResourceScheduler>,
293
+ coordinator: Arc<dyn PalwCoordinator>,
294
+ executor: Arc<dyn ReplicaExecutor>,
295
+ config: WorkerConfig,
296
+ status: Arc<WorkerStatus>,
297
+ ) -> WorkerHandle {
298
+ let stop = Arc::new(AtomicBool::new(false));
299
+ let stop_thread = Arc::clone(&stop);
300
+ let join = std::thread::Builder::new()
301
+ .name("palw-worker".into())
302
+ .spawn(move || {
303
+ eprintln!(
304
+ "[palw-worker] started: provider {} poll {}ms",
305
+ config.provider_id, config.poll_ms
306
+ );
307
+ while !stop_thread.load(Ordering::Relaxed) {
308
+ run_cycle(&store, &scheduler, coordinator.as_ref(), executor.as_ref(), &config, &status);
309
+ // Sleep in small slices so shutdown stays prompt.
310
+ let mut slept = 0u64;
311
+ while slept < config.poll_ms && !stop_thread.load(Ordering::Relaxed) {
312
+ let slice = (config.poll_ms - slept).min(100);
313
+ std::thread::sleep(std::time::Duration::from_millis(slice));
314
+ slept += slice;
315
+ }
316
+ }
317
+ })
318
+ .expect("spawn palw-worker");
319
+ WorkerHandle { stop, join: Some(join) }
320
+ }
321
+
322
+ #[cfg(test)]
323
+ mod tests {
324
+ use super::*;
325
+ use crate::coordinator::{LoopbackConfig, LoopbackCoordinator};
326
+ use crate::events::TurnPrivacyMode;
327
+ use crate::store::NewTurn;
328
+
329
+ struct FakeExecutor(Vec<u32>);
330
+ impl ReplicaExecutor for FakeExecutor {
331
+ fn execute(&self, _prompt_ids: &[u32], _max_new: u32) -> Result<ReplicaOutput, String> {
332
+ Ok(ReplicaOutput { output_ids: self.0.clone(), runtime_roots: None })
333
+ }
334
+ }
335
+
336
+ fn store_with_turn(name: &str, privacy: TurnPrivacyMode, output: &[u32]) -> Mutex<SessionStore> {
337
+ let dir = std::env::temp_dir().join(format!("palw-gw-worker-{}-{name}", std::process::id()));
338
+ let _ = std::fs::remove_dir_all(&dir);
339
+ let mut s = SessionStore::open(&dir.join("s.sqlite3")).unwrap();
340
+ s.create_conversation("c1", "t", None, 1).unwrap();
341
+ s.begin_turn(&NewTurn {
342
+ turn_id: "t1",
343
+ conversation_id: "c1",
344
+ parent_turn_id: None,
345
+ role_user_text: "q",
346
+ user_block_text: "q",
347
+ context_meta_json: None,
348
+ prompt_ids: &[1, 2, 3],
349
+ privacy_mode: privacy,
350
+ search_bundle_json: None,
351
+ now_ms: 2,
352
+ })
353
+ .unwrap();
354
+ s.complete_turn(&crate::store::CompletedTurn {
355
+ turn_id: "t1",
356
+ output_ids: output,
357
+ display_text: "ans",
358
+ stop_reason: "eos",
359
+ status: TurnVerificationStatus::LocalComplete,
360
+ receipt_json: None,
361
+ runtime_roots_json: None,
362
+ })
363
+ .unwrap();
364
+ Mutex::new(s)
365
+ }
366
+
367
+ fn worker_config() -> WorkerConfig {
368
+ WorkerConfig { provider_id: "prov-a".into(), poll_ms: 10, max_new: 16 }
369
+ }
370
+
371
+ #[test]
372
+ fn full_loopback_match_reaches_certified_head() {
373
+ let store = store_with_turn("match", TurnPrivacyMode::PalwMint, &[7, 8, 9]);
374
+ let scheduler = ResourceScheduler::new();
375
+ let coordinator = LoopbackCoordinator::new(LoopbackConfig::default());
376
+ let executor = FakeExecutor(vec![7, 8, 9]); // honest replica: same ids as A stored
377
+ let config = worker_config();
378
+ let status = WorkerStatus::default();
379
+
380
+ // Cycle 1: submit + self-assign + execute + result.
381
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status);
382
+ assert_eq!(status.submitted.load(Ordering::Relaxed), 1);
383
+ assert_eq!(status.replicas_executed.load(Ordering::Relaxed), 1);
384
+ assert_eq!(
385
+ store.lock().unwrap().get_turn("t1").unwrap().verification_status,
386
+ TurnVerificationStatus::ReplicaPending
387
+ );
388
+
389
+ // Cycle 2: verdict replica_matched.
390
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status);
391
+ assert_eq!(
392
+ store.lock().unwrap().get_turn("t1").unwrap().verification_status,
393
+ TurnVerificationStatus::ReplicaMatched
394
+ );
395
+
396
+ // Cycle 3: verdict certified — certified_head advances (the §4 dual-head payoff).
397
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status);
398
+ let s = store.lock().unwrap();
399
+ assert_eq!(s.get_turn("t1").unwrap().verification_status, TurnVerificationStatus::Certified);
400
+ assert_eq!(s.heads("c1").unwrap().certified_head.as_deref(), Some("t1"));
401
+ assert!(status.last_error.lock().unwrap().is_none(), "{:?}", status.last_error.lock().unwrap());
402
+ }
403
+
404
+ #[test]
405
+ fn dishonest_replica_yields_mismatch() {
406
+ let store = store_with_turn("mismatch", TurnPrivacyMode::PalwMint, &[7, 8, 9]);
407
+ let scheduler = ResourceScheduler::new();
408
+ let coordinator = LoopbackCoordinator::new(LoopbackConfig::default());
409
+ let executor = FakeExecutor(vec![666]); // wrong output
410
+ let config = worker_config();
411
+ let status = WorkerStatus::default();
412
+
413
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status); // submit+replica
414
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status); // verdict
415
+ assert_eq!(
416
+ store.lock().unwrap().get_turn("t1").unwrap().verification_status,
417
+ TurnVerificationStatus::Mismatch
418
+ );
419
+ }
420
+
421
+ #[test]
422
+ fn local_only_turns_never_leave_the_device() {
423
+ let store = store_with_turn("localonly", TurnPrivacyMode::LocalOnly, &[7]);
424
+ let scheduler = ResourceScheduler::new();
425
+ let coordinator = LoopbackCoordinator::new(LoopbackConfig::default());
426
+ let executor = FakeExecutor(vec![7]);
427
+ let status = WorkerStatus::default();
428
+ run_cycle(&store, &scheduler, &coordinator, &executor, &worker_config(), &status);
429
+ assert_eq!(status.submitted.load(Ordering::Relaxed), 0);
430
+ assert_eq!(coordinator.counts()["unassigned"], 0, "nothing was ever submitted");
431
+ assert_eq!(
432
+ store.lock().unwrap().get_turn("t1").unwrap().verification_status,
433
+ TurnVerificationStatus::LocalComplete
434
+ );
435
+ }
436
+
437
+ #[test]
438
+ fn admission_refusal_declines_and_requeues() {
439
+ let store = store_with_turn("decline", TurnPrivacyMode::VerifiedNoMint, &[7]);
440
+ let scheduler = ResourceScheduler::new();
441
+ let coordinator = LoopbackCoordinator::new(LoopbackConfig::default());
442
+ let executor = FakeExecutor(vec![7]);
443
+ // 10M tokens at the 8 tok/s default EMA ⇒ admission must refuse against a 120s deadline.
444
+ let config = WorkerConfig { provider_id: "prov-a".into(), poll_ms: 10, max_new: 10_000_000 };
445
+ let status = WorkerStatus::default();
446
+
447
+ run_cycle(&store, &scheduler, &coordinator, &executor, &config, &status);
448
+ assert_eq!(status.replicas_declined.load(Ordering::Relaxed), 1);
449
+ assert_eq!(status.replicas_executed.load(Ordering::Relaxed), 0);
450
+ // Declined ⇒ requeued at the coordinator; the turn itself stays replica_pending.
451
+ assert_eq!(coordinator.counts()["unassigned"], 1);
452
+ assert_eq!(
453
+ store.lock().unwrap().get_turn("t1").unwrap().verification_status,
454
+ TurnVerificationStatus::ReplicaPending
455
+ );
456
+ }
457
+
458
+ #[test]
459
+ fn spawned_worker_reconciles_and_shuts_down() {
460
+ let store = store_with_turn("spawned", TurnPrivacyMode::PalwMint, &[7, 8, 9]);
461
+ let store = Arc::new(store);
462
+ let scheduler = Arc::new(ResourceScheduler::new());
463
+ let coordinator: Arc<dyn crate::coordinator::PalwCoordinator> =
464
+ Arc::new(LoopbackCoordinator::new(LoopbackConfig::default()));
465
+ let executor: Arc<dyn ReplicaExecutor> = Arc::new(FakeExecutor(vec![7, 8, 9]));
466
+ let status = Arc::new(WorkerStatus::default());
467
+
468
+ let handle = spawn(
469
+ Arc::clone(&store),
470
+ scheduler,
471
+ coordinator,
472
+ executor,
473
+ WorkerConfig { provider_id: "prov-a".into(), poll_ms: 10, max_new: 16 },
474
+ Arc::clone(&status),
475
+ );
476
+ let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
477
+ while store.lock().unwrap().get_turn("t1").unwrap().verification_status != TurnVerificationStatus::Certified {
478
+ assert!(std::time::Instant::now() < deadline, "worker did not certify within 10s");
479
+ std::thread::sleep(std::time::Duration::from_millis(10));
480
+ }
481
+ handle.shutdown();
482
+ assert!(status.submitted.load(Ordering::Relaxed) >= 1);
483
+ assert_eq!(store.lock().unwrap().heads("c1").unwrap().certified_head.as_deref(), Some("t1"));
484
+ }
485
+
486
+ #[test]
487
+ fn steps_never_request_illegal_jumps() {
488
+ use TurnVerificationStatus as S;
489
+ assert_eq!(steps_to(S::ReplicaPending, JobVerdict::Certified), vec![S::ReplicaMatched, S::Certified]);
490
+ assert_eq!(steps_to(S::ReplicaMatched, JobVerdict::Certified), vec![S::Certified]);
491
+ assert_eq!(steps_to(S::ReplicaPending, JobVerdict::ReplicaMatched), vec![S::ReplicaMatched]);
492
+ assert!(steps_to(S::Certified, JobVerdict::ReplicaMatched).is_empty(), "stale verdicts are ignored");
493
+ assert!(steps_to(S::Mismatch, JobVerdict::Certified).is_empty());
494
+ assert_eq!(steps_to(S::AuditPending, JobVerdict::Mismatch), vec![S::Mismatch]);
495
+ }
496
+ }
runtime-palw/src/lib.rs CHANGED
@@ -27,6 +27,7 @@ pub mod lifecycle;
27
  pub mod manifest;
28
  pub mod matcher;
29
  pub mod mint;
 
30
  pub mod observer;
31
  pub mod protocol_v2;
32
  pub mod qwen_adapter;
 
27
  pub mod manifest;
28
  pub mod matcher;
29
  pub mod mint;
30
+ pub mod model_genesis;
31
  pub mod observer;
32
  pub mod protocol_v2;
33
  pub mod qwen_adapter;
runtime-palw/src/model_genesis.rs ADDED
@@ -0,0 +1,1152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ //! `ModelGenesisManifest` — the ADR-0047 in-repo prerequisite.
2
+ //!
3
+ //! ADR-0047 (`MisakaLLM-palw-shared/docs/adr/0047-model-genesis-dual-reproduction.md` §1) fixes the
4
+ //! two-party reproduction protocol as: weights → sha256 → deterministic conversion →
5
+ //! `ModelGenesisManifest` → keyed-BLAKE2b(palw-k1 convention) hash → ML-DSA signature, with
6
+ //! acceptance defined as a **bit-exact manifest-hash match between two parties**. Before this
7
+ //! module the type did not exist anywhere in either repository, so two operators running the
8
+ //! external procedure had nothing to compare: the ADR's acceptance criterion was unevaluable.
9
+ //!
10
+ //! # What this module is, and what it is not
11
+ //!
12
+ //! It is the *comparable artifact*: a canonical, backend-independent, weights-derived encoding and
13
+ //! its domain-separated digest, plus a field-level diff so a mismatch names a field instead of
14
+ //! saying "the hashes differ".
15
+ //!
16
+ //! It is **not** a genesis. Every external item of the ADR remains open: official-weights
17
+ //! provenance, the two real GPU runs, and the evidence bundle. [`assess_genesis`] can never return
18
+ //! an eligible assessment (see [`GenesisDisqualifier::ManifestUnsigned`]), and that is enforced by
19
+ //! a unit test, in the same shape as `crate::mint`'s `MainnetBlocker::ExternalModelGenesis` gate.
20
+ //!
21
+ //! # Backend independence (ADR §1)
22
+ //!
23
+ //! The proof-of-llm verifier's Metal/CUDA token asymmetry is the reason the ADR demands a manifest
24
+ //! that contains no execution trace: any host-, backend- or build-dependent field makes the
25
+ //! two-party comparison fail for reasons that have nothing to do with the weights. This module
26
+ //! enforces that mechanically rather than by convention — [`ModelGenesisManifest::validate`]
27
+ //! rejects a conversion procedure that mentions a backend or host token, and rejects absolute or
28
+ //! traversal-bearing paths (which carry the operator's home directory into the digest).
29
+ //!
30
+ //! # Domain separation
31
+ //!
32
+ //! The digest primitive here (keyed BLAKE2b-512) is the same one used for receipt bodies, match
33
+ //! projections and file hashes (`palw-k1/file`, see `crate::receipt_v3`). Keying by a per-object
34
+ //! domain is what keeps those digest spaces unrelated: without it, a byte string that is a valid
35
+ //! canonical manifest could simultaneously be a valid preimage of another object class, and a
36
+ //! commitment or signature over one would silently be a commitment over the other. The model
37
+ //! genesis therefore takes its own key, [`PALW_MODEL_GENESIS_V1_HASH_DOMAIN`], distinct from every
38
+ //! domain in `crate::receipt_v3`.
39
+ //!
40
+ //! # Signing is blocked, deliberately
41
+ //!
42
+ //! The ML-DSA-87 signature-context table (`consensus/core/src/signature_domains.rs`) was LOCKED on
43
+ //! 2026-07-25 and is guarded by a golden test. A model-genesis signature therefore cannot be
44
+ //! produced today, and this module contains **no signing function at all**.
45
+ //! [`PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED`] records the exact context bytes that
46
+ //! would be requested, so the unlock decision has something concrete to review; it is not usable
47
+ //! until that decision is taken.
48
+
49
+ use crate::receipt_v3::Hash64;
50
+ use thiserror::Error;
51
+
52
+ /// Schema version of the canonical model-genesis encoding.
53
+ pub const MODEL_GENESIS_SCHEMA_V1: u16 = 1;
54
+
55
+ /// Keyed-BLAKE2b-512 domain (used as the `BLAKE2b` *key*) for [`ModelGenesisManifest::manifest_hash`].
56
+ ///
57
+ /// Distinct from every `crate::receipt_v3` domain and from `palw-k1/file`; see the module note on
58
+ /// why domain separation is load-bearing rather than cosmetic.
59
+ pub const PALW_MODEL_GENESIS_V1_HASH_DOMAIN: &[u8] = b"misaka-palw-v1/model-genesis";
60
+
61
+ /// The ML-DSA-87 signature context a signed model genesis WOULD require.
62
+ ///
63
+ /// **UNREGISTERED — do not sign with this.** The signature-domain table is LOCKED (2026-07-25) and
64
+ /// adding a row is an explicit human unlock decision. The row that would be required is recorded in
65
+ /// ADR-0047 and in `docs/model-genesis-candidate.md`. This constant exists only so that the review
66
+ /// of that decision can check distinctness and prefix-freedom against the contexts this crate
67
+ /// already knows (see `tests::proposed_context_is_distinct_and_prefix_free`).
68
+ pub const PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED: &[u8] =
69
+ b"misaka-palw-v1/model-genesis/mldsa87";
70
+
71
+ /// A SHA-256 file digest, as published by the weights host and recomputed locally.
72
+ pub type Sha256Digest = [u8; 32];
73
+
74
+ /// Errors from [`ModelGenesisManifest::validate`].
75
+ #[derive(Debug, Clone, PartialEq, Eq, Error)]
76
+ pub enum ModelGenesisError {
77
+ /// A text field contained a byte outside printable ASCII (`0x20..=0x7e`).
78
+ ///
79
+ /// Non-ASCII text has multiple encodings of the same-looking string; permitting it would make
80
+ /// the "two parties typed the same thing" assumption unverifiable.
81
+ #[error("{field}: byte {index} is outside printable ASCII")]
82
+ NonPrintableAscii {
83
+ /// Dotted field path.
84
+ field: &'static str,
85
+ /// Index of the offending byte.
86
+ index: usize,
87
+ },
88
+ /// A field that must pin an immutable revision was not 40 or 64 lowercase hex characters.
89
+ #[error("{field}: {value:?} is not an immutable revision pin (40 or 64 lowercase hex)")]
90
+ NotARevisionPin {
91
+ /// Dotted field path.
92
+ field: &'static str,
93
+ /// The rejected value.
94
+ value: String,
95
+ },
96
+ /// A path field was empty, absolute, or contained a `..` / `~` component.
97
+ #[error("{field}: {value:?} must be a relative, traversal-free path")]
98
+ UnsafePath {
99
+ /// Dotted field path.
100
+ field: &'static str,
101
+ /// The rejected value.
102
+ value: String,
103
+ },
104
+ /// The weight-file list was not strictly ascending by path.
105
+ ///
106
+ /// Sorting is required rather than applied silently: two operators whose listings differ have a
107
+ /// real difference in what they hashed, and hiding it behind a sort would hide that.
108
+ #[error("weights.files: must be strictly ascending by path ({0:?} is not before {1:?})")]
109
+ UnsortedFiles(String, String),
110
+ /// The weight-file list was empty.
111
+ #[error("weights.files: at least one weight file must be recorded")]
112
+ NoWeightFiles,
113
+ /// A conversion-procedure field named a backend- or host-specific token (ADR §1).
114
+ #[error(
115
+ "conversion.{field}: contains {token:?} — the manifest must be weights-derived only, \
116
+ backend/host-specific inputs make the two-party comparison meaningless (ADR-0047 §1)"
117
+ )]
118
+ BackendSpecificField {
119
+ /// Dotted field path within the conversion procedure.
120
+ field: &'static str,
121
+ /// The denied token that matched.
122
+ token: &'static str,
123
+ },
124
+ /// An architecture or tokenizer field that must describe real geometry was zero.
125
+ #[error("{0}: must be non-zero")]
126
+ ZeroField(&'static str),
127
+ }
128
+
129
+ /// How far the recorded provenance chain reaches.
130
+ ///
131
+ /// This is a claim both parties must make identically; it is part of the hashed encoding precisely
132
+ /// so that one party cannot quietly record "official" while the other records "derivative".
133
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
134
+ pub enum ProvenanceTier {
135
+ /// The recorded source is the official upstream publisher's checkpoint.
136
+ OfficialUpstream,
137
+ /// The recorded source is a third-party derivative (for example an "abliterated" re-release).
138
+ ///
139
+ /// This is what the pins in `docs/model-genesis-candidate.md` currently are. A manifest with
140
+ /// this tier is reproducible, but it can never become the genesis — disqualifier (1).
141
+ ThirdPartyDerivative,
142
+ }
143
+
144
+ impl ProvenanceTier {
145
+ /// Canonical wire tag.
146
+ #[must_use]
147
+ pub const fn tag(self) -> u8 {
148
+ match self {
149
+ Self::OfficialUpstream => 0,
150
+ Self::ThirdPartyDerivative => 1,
151
+ }
152
+ }
153
+ }
154
+
155
+ /// A repository/revision pair the recorded weights claim to descend from.
156
+ #[derive(Debug, Clone, PartialEq, Eq)]
157
+ pub struct UpstreamRef {
158
+ /// Publisher-scoped repository identifier, for example `Qwen/Qwen3.6-35B-A3B`.
159
+ pub source_id: String,
160
+ /// Immutable revision pin (40 or 64 lowercase hex characters).
161
+ pub revision: String,
162
+ }
163
+
164
+ /// One downloaded weight-side file and its locally recomputed digest.
165
+ #[derive(Debug, Clone, PartialEq, Eq)]
166
+ pub struct WeightFile {
167
+ /// Repository-relative, traversal-free path.
168
+ pub path: String,
169
+ /// SHA-256 of the file bytes as they exist on disk.
170
+ pub sha256: Sha256Digest,
171
+ /// Exact byte length.
172
+ pub size: u64,
173
+ }
174
+
175
+ /// Where the weights came from and exactly which bytes were obtained.
176
+ #[derive(Debug, Clone, PartialEq, Eq)]
177
+ pub struct WeightsProvenance {
178
+ /// How far the chain reaches.
179
+ pub tier: ProvenanceTier,
180
+ /// Repository identifier actually downloaded from.
181
+ pub source_id: String,
182
+ /// Immutable revision pin actually downloaded.
183
+ pub source_revision: String,
184
+ /// The upstream this source declares itself derived from, when it is not itself upstream.
185
+ pub upstream_of_record: Option<UpstreamRef>,
186
+ /// Every file that participates in the conversion, strictly ascending by `path`.
187
+ pub files: Vec<WeightFile>,
188
+ }
189
+
190
+ /// Model geometry, read from the checkpoint's own configuration.
191
+ ///
192
+ /// Every field here is a property of the weights, not of any runtime: the same numbers are read by
193
+ /// a Metal host and a CUDA host from the same `config.json`.
194
+ #[derive(Debug, Clone, PartialEq, Eq)]
195
+ pub struct ModelArchitecture {
196
+ /// Architecture class string, for example `Qwen3ForCausalLM`.
197
+ pub architecture: String,
198
+ /// Hidden size.
199
+ pub hidden_size: u32,
200
+ /// Transformer layer count.
201
+ pub layers: u32,
202
+ /// Attention head count.
203
+ pub attention_heads: u32,
204
+ /// Key/value head count.
205
+ pub key_value_heads: u32,
206
+ /// Feed-forward intermediate size.
207
+ pub intermediate_size: u32,
208
+ /// Per-head dimension.
209
+ pub head_dim: u32,
210
+ /// `RoPE` base.
211
+ ///
212
+ /// Integral by construction: a non-integral base must extend the schema rather than be rounded,
213
+ /// because rounding is exactly the kind of silent per-implementation difference that breaks a
214
+ /// bit-exact comparison.
215
+ pub rope_theta: u64,
216
+ /// Vocabulary size declared by the model configuration.
217
+ pub vocab_size: u32,
218
+ /// Maximum position embeddings.
219
+ pub max_position_embeddings: u32,
220
+ /// Stored parameter dtype, for example `bfloat16`.
221
+ pub parameter_dtype: String,
222
+ }
223
+
224
+ /// Tokenizer identity.
225
+ ///
226
+ /// Redundant with [`WeightsProvenance::files`] on purpose: naming the semantically load-bearing
227
+ /// files individually means a diff reports `tokenizer.tokenizer_sha256` rather than an index into a
228
+ /// file list.
229
+ #[derive(Debug, Clone, PartialEq, Eq)]
230
+ pub struct TokenizerIdentity {
231
+ /// SHA-256 of `tokenizer.json`.
232
+ pub tokenizer_sha256: Sha256Digest,
233
+ /// SHA-256 of `tokenizer_config.json`.
234
+ pub tokenizer_config_sha256: Sha256Digest,
235
+ /// SHA-256 of the chat template shipped with the checkpoint.
236
+ pub chat_template_sha256: Sha256Digest,
237
+ /// Vocabulary size as the tokenizer itself reports it.
238
+ pub tokenizer_vocab_size: u32,
239
+ }
240
+
241
+ /// The deterministic derivation both parties must run identically.
242
+ ///
243
+ /// A conversion *procedure* is not backend state: `convert_hf_to_gguf.py` at a pinned commit
244
+ /// produces the same bytes on any host. That is a claim the two-party run TESTS; recording it here
245
+ /// is what makes the test meaningful. Anything host- or backend-flavoured is rejected by
246
+ /// [`ModelGenesisManifest::validate`].
247
+ #[derive(Debug, Clone, PartialEq, Eq)]
248
+ pub struct ConversionProcedure {
249
+ /// Converter source repository URL.
250
+ pub converter_repo: String,
251
+ /// Converter commit pin (40 or 64 lowercase hex characters).
252
+ pub converter_commit: String,
253
+ /// Repository-relative converter entrypoint, for example `convert_hf_to_gguf.py`.
254
+ pub converter_entrypoint: String,
255
+ /// Exact ordered argument tail passed to the converter, excluding input/output paths.
256
+ pub converter_args: Vec<String>,
257
+ /// Repository-relative quantizer entrypoint, for example `llama-quantize`.
258
+ pub quantizer_entrypoint: String,
259
+ /// Exact ordered argument tail passed to the quantizer, excluding input/output paths.
260
+ pub quantizer_args: Vec<String>,
261
+ /// Target quantization label, for example `Q4_K_M`.
262
+ pub quantization: String,
263
+ }
264
+
265
+ /// The artifact the conversion produced.
266
+ #[derive(Debug, Clone, PartialEq, Eq)]
267
+ pub struct DerivedArtifact {
268
+ /// SHA-256 of the produced `GGUF`.
269
+ pub gguf_sha256: Sha256Digest,
270
+ /// Exact byte length of the produced `GGUF`.
271
+ pub gguf_size: u64,
272
+ }
273
+
274
+ /// The comparable object of ADR-0047 §1.
275
+ ///
276
+ /// Canonical encoding: all integers big-endian, every variable-length field length-prefixed with a
277
+ /// `u64` big-endian length, fields in declaration order. Length prefixing everywhere makes the
278
+ /// encoding injective (prefix-free), so no two distinct manifests can share canonical bytes.
279
+ #[derive(Debug, Clone, PartialEq, Eq)]
280
+ pub struct ModelGenesisManifest {
281
+ /// Encoding schema version; [`MODEL_GENESIS_SCHEMA_V1`] today.
282
+ pub schema_version: u16,
283
+ /// Where the weights came from.
284
+ pub weights: WeightsProvenance,
285
+ /// Geometry read from the checkpoint configuration.
286
+ pub architecture: ModelArchitecture,
287
+ /// Tokenizer identity.
288
+ pub tokenizer: TokenizerIdentity,
289
+ /// The derivation both parties run.
290
+ pub conversion: ConversionProcedure,
291
+ /// What the derivation produced.
292
+ pub artifact: DerivedArtifact,
293
+ }
294
+
295
+ /// Tokens that must never appear in a conversion-procedure field.
296
+ ///
297
+ /// Short or ambiguous substrings are deliberately excluded (`hip` matches `chip`), and the list is
298
+ /// applied only to the conversion procedure, where a hit is unambiguous evidence that host state
299
+ /// leaked into a weights-derived object.
300
+ const BACKEND_TOKENS: &[&str] = &[
301
+ "metal", "cuda", "cublas", "rocm", "vulkan", "sycl", "opencl", "gpu", "ngl", "device",
302
+ "thread", "darwin", "arm64", "x86_64", "avx", "neon",
303
+ ];
304
+
305
+ fn push_framed(out: &mut Vec<u8>, bytes: &[u8]) {
306
+ let len = u64::try_from(bytes.len()).expect("field length fits in u64 on supported platforms");
307
+ out.extend_from_slice(&len.to_be_bytes());
308
+ out.extend_from_slice(bytes);
309
+ }
310
+
311
+ fn check_ascii(field: &'static str, value: &str) -> Result<(), ModelGenesisError> {
312
+ for (index, byte) in value.as_bytes().iter().enumerate() {
313
+ if !(0x20..=0x7e).contains(byte) {
314
+ return Err(ModelGenesisError::NonPrintableAscii { field, index });
315
+ }
316
+ }
317
+ Ok(())
318
+ }
319
+
320
+ fn check_revision_pin(field: &'static str, value: &str) -> Result<(), ModelGenesisError> {
321
+ let ok = (value.len() == 40 || value.len() == 64)
322
+ && value
323
+ .bytes()
324
+ .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b));
325
+ if ok {
326
+ Ok(())
327
+ } else {
328
+ Err(ModelGenesisError::NotARevisionPin {
329
+ field,
330
+ value: value.to_owned(),
331
+ })
332
+ }
333
+ }
334
+
335
+ fn check_relative_path(field: &'static str, value: &str) -> Result<(), ModelGenesisError> {
336
+ let bad = value.is_empty()
337
+ || value.starts_with('/')
338
+ || value.starts_with('~')
339
+ || value.split('/').any(|part| part == ".." || part.is_empty());
340
+ if bad {
341
+ return Err(ModelGenesisError::UnsafePath {
342
+ field,
343
+ value: value.to_owned(),
344
+ });
345
+ }
346
+ Ok(())
347
+ }
348
+
349
+ fn check_backend_free(field: &'static str, value: &str) -> Result<(), ModelGenesisError> {
350
+ let lowered = value.to_ascii_lowercase();
351
+ for token in BACKEND_TOKENS {
352
+ if lowered.contains(token) {
353
+ return Err(ModelGenesisError::BackendSpecificField { field, token });
354
+ }
355
+ }
356
+ Ok(())
357
+ }
358
+
359
+ impl WeightsProvenance {
360
+ fn append_canonical(&self, out: &mut Vec<u8>) {
361
+ out.push(self.tier.tag());
362
+ push_framed(out, self.source_id.as_bytes());
363
+ push_framed(out, self.source_revision.as_bytes());
364
+ match &self.upstream_of_record {
365
+ None => out.push(0),
366
+ Some(upstream) => {
367
+ out.push(1);
368
+ push_framed(out, upstream.source_id.as_bytes());
369
+ push_framed(out, upstream.revision.as_bytes());
370
+ }
371
+ }
372
+ let count =
373
+ u64::try_from(self.files.len()).expect("file count fits in u64 on supported platforms");
374
+ out.extend_from_slice(&count.to_be_bytes());
375
+ for file in &self.files {
376
+ push_framed(out, file.path.as_bytes());
377
+ out.extend_from_slice(&file.sha256);
378
+ out.extend_from_slice(&file.size.to_be_bytes());
379
+ }
380
+ }
381
+ }
382
+
383
+ impl ModelArchitecture {
384
+ fn append_canonical(&self, out: &mut Vec<u8>) {
385
+ push_framed(out, self.architecture.as_bytes());
386
+ for value in [
387
+ self.hidden_size,
388
+ self.layers,
389
+ self.attention_heads,
390
+ self.key_value_heads,
391
+ self.intermediate_size,
392
+ self.head_dim,
393
+ ] {
394
+ out.extend_from_slice(&value.to_be_bytes());
395
+ }
396
+ out.extend_from_slice(&self.rope_theta.to_be_bytes());
397
+ out.extend_from_slice(&self.vocab_size.to_be_bytes());
398
+ out.extend_from_slice(&self.max_position_embeddings.to_be_bytes());
399
+ push_framed(out, self.parameter_dtype.as_bytes());
400
+ }
401
+ }
402
+
403
+ impl TokenizerIdentity {
404
+ fn append_canonical(&self, out: &mut Vec<u8>) {
405
+ out.extend_from_slice(&self.tokenizer_sha256);
406
+ out.extend_from_slice(&self.tokenizer_config_sha256);
407
+ out.extend_from_slice(&self.chat_template_sha256);
408
+ out.extend_from_slice(&self.tokenizer_vocab_size.to_be_bytes());
409
+ }
410
+ }
411
+
412
+ impl ConversionProcedure {
413
+ fn append_canonical(&self, out: &mut Vec<u8>) {
414
+ push_framed(out, self.converter_repo.as_bytes());
415
+ push_framed(out, self.converter_commit.as_bytes());
416
+ push_framed(out, self.converter_entrypoint.as_bytes());
417
+ append_string_list(out, &self.converter_args);
418
+ push_framed(out, self.quantizer_entrypoint.as_bytes());
419
+ append_string_list(out, &self.quantizer_args);
420
+ push_framed(out, self.quantization.as_bytes());
421
+ }
422
+ }
423
+
424
+ fn append_string_list(out: &mut Vec<u8>, values: &[String]) {
425
+ let count =
426
+ u64::try_from(values.len()).expect("list length fits in u64 on supported platforms");
427
+ out.extend_from_slice(&count.to_be_bytes());
428
+ for value in values {
429
+ push_framed(out, value.as_bytes());
430
+ }
431
+ }
432
+
433
+ impl DerivedArtifact {
434
+ fn append_canonical(&self, out: &mut Vec<u8>) {
435
+ out.extend_from_slice(&self.gguf_sha256);
436
+ out.extend_from_slice(&self.gguf_size.to_be_bytes());
437
+ }
438
+ }
439
+
440
+ impl ModelGenesisManifest {
441
+ /// Canonical bytes (big-endian integers, `u64`-length-framed variable fields).
442
+ ///
443
+ /// Hashing an unvalidated manifest is meaningless for comparison — call [`Self::validate`]
444
+ /// first; the assessment path does.
445
+ #[must_use]
446
+ pub fn canonical_bytes(&self) -> Vec<u8> {
447
+ let mut out = Vec::new();
448
+ out.extend_from_slice(&self.schema_version.to_be_bytes());
449
+ self.weights.append_canonical(&mut out);
450
+ self.architecture.append_canonical(&mut out);
451
+ self.tokenizer.append_canonical(&mut out);
452
+ self.conversion.append_canonical(&mut out);
453
+ self.artifact.append_canonical(&mut out);
454
+ out
455
+ }
456
+
457
+ /// Keyed BLAKE2b-512 of [`Self::canonical_bytes`] under
458
+ /// [`PALW_MODEL_GENESIS_V1_HASH_DOMAIN`].
459
+ ///
460
+ /// This is the value ADR-0047 requires to be bit-exact between the two parties.
461
+ #[must_use]
462
+ pub fn manifest_hash(&self) -> Hash64 {
463
+ let mut out = [0u8; 64];
464
+ out.copy_from_slice(
465
+ blake2b_simd::Params::new()
466
+ .hash_length(64)
467
+ .key(PALW_MODEL_GENESIS_V1_HASH_DOMAIN)
468
+ .to_state()
469
+ .update(&self.canonical_bytes())
470
+ .finalize()
471
+ .as_bytes(),
472
+ );
473
+ out
474
+ }
475
+
476
+ /// Structural validation: printable ASCII, immutable revision pins, safe relative paths, a
477
+ /// strictly ascending file list, non-zero geometry, and a backend/host-free conversion.
478
+ pub fn validate(&self) -> Result<(), ModelGenesisError> {
479
+ self.validate_weights()?;
480
+ self.validate_architecture()?;
481
+ self.validate_conversion()?;
482
+ if self.tokenizer.tokenizer_vocab_size == 0 {
483
+ return Err(ModelGenesisError::ZeroField(
484
+ "tokenizer.tokenizer_vocab_size",
485
+ ));
486
+ }
487
+ if self.artifact.gguf_size == 0 {
488
+ return Err(ModelGenesisError::ZeroField("artifact.gguf_size"));
489
+ }
490
+ Ok(())
491
+ }
492
+
493
+ fn validate_weights(&self) -> Result<(), ModelGenesisError> {
494
+ let weights = &self.weights;
495
+ check_ascii("weights.source_id", &weights.source_id)?;
496
+ check_revision_pin("weights.source_revision", &weights.source_revision)?;
497
+ if let Some(upstream) = &weights.upstream_of_record {
498
+ check_ascii("weights.upstream_of_record.source_id", &upstream.source_id)?;
499
+ check_revision_pin("weights.upstream_of_record.revision", &upstream.revision)?;
500
+ }
501
+ if weights.files.is_empty() {
502
+ return Err(ModelGenesisError::NoWeightFiles);
503
+ }
504
+ for file in &weights.files {
505
+ check_ascii("weights.files.path", &file.path)?;
506
+ check_relative_path("weights.files.path", &file.path)?;
507
+ }
508
+ for pair in weights.files.windows(2) {
509
+ if pair[0].path >= pair[1].path {
510
+ return Err(ModelGenesisError::UnsortedFiles(
511
+ pair[0].path.clone(),
512
+ pair[1].path.clone(),
513
+ ));
514
+ }
515
+ }
516
+ Ok(())
517
+ }
518
+
519
+ fn validate_architecture(&self) -> Result<(), ModelGenesisError> {
520
+ let arch = &self.architecture;
521
+ check_ascii("architecture.architecture", &arch.architecture)?;
522
+ check_ascii("architecture.parameter_dtype", &arch.parameter_dtype)?;
523
+ let scalars: [(&'static str, u64); 9] = [
524
+ ("architecture.hidden_size", u64::from(arch.hidden_size)),
525
+ ("architecture.layers", u64::from(arch.layers)),
526
+ (
527
+ "architecture.attention_heads",
528
+ u64::from(arch.attention_heads),
529
+ ),
530
+ (
531
+ "architecture.key_value_heads",
532
+ u64::from(arch.key_value_heads),
533
+ ),
534
+ (
535
+ "architecture.intermediate_size",
536
+ u64::from(arch.intermediate_size),
537
+ ),
538
+ ("architecture.head_dim", u64::from(arch.head_dim)),
539
+ ("architecture.rope_theta", arch.rope_theta),
540
+ ("architecture.vocab_size", u64::from(arch.vocab_size)),
541
+ (
542
+ "architecture.max_position_embeddings",
543
+ u64::from(arch.max_position_embeddings),
544
+ ),
545
+ ];
546
+ for (name, value) in scalars {
547
+ if value == 0 {
548
+ return Err(ModelGenesisError::ZeroField(name));
549
+ }
550
+ }
551
+ Ok(())
552
+ }
553
+
554
+ fn validate_conversion(&self) -> Result<(), ModelGenesisError> {
555
+ let conv = &self.conversion;
556
+ let single: [(&'static str, &str); 4] = [
557
+ ("converter_repo", conv.converter_repo.as_str()),
558
+ ("converter_commit", conv.converter_commit.as_str()),
559
+ ("converter_entrypoint", conv.converter_entrypoint.as_str()),
560
+ ("quantization", conv.quantization.as_str()),
561
+ ];
562
+ for (name, value) in single {
563
+ check_ascii(name, value)?;
564
+ check_backend_free(name, value)?;
565
+ }
566
+ check_ascii("quantizer_entrypoint", &conv.quantizer_entrypoint)?;
567
+ check_revision_pin("conversion.converter_commit", &conv.converter_commit)?;
568
+ check_relative_path(
569
+ "conversion.converter_entrypoint",
570
+ &conv.converter_entrypoint,
571
+ )?;
572
+ check_relative_path(
573
+ "conversion.quantizer_entrypoint",
574
+ &conv.quantizer_entrypoint,
575
+ )?;
576
+ for arg in &conv.converter_args {
577
+ check_ascii("converter_args", arg)?;
578
+ check_backend_free("converter_args", arg)?;
579
+ }
580
+ for arg in &conv.quantizer_args {
581
+ check_ascii("quantizer_args", arg)?;
582
+ check_backend_free("quantizer_args", arg)?;
583
+ }
584
+ Ok(())
585
+ }
586
+
587
+ /// First field whose value differs, using dotted paths, or `None` when the two manifests are
588
+ /// identical.
589
+ ///
590
+ /// This is the whole point of the type: ADR-0047's acceptance test is a hash comparison, and a
591
+ /// hash comparison alone tells a failing operator nothing about what to fix.
592
+ #[must_use]
593
+ #[allow(clippy::too_many_lines)]
594
+ pub fn first_mismatch(&self, other: &Self) -> Option<&'static str> {
595
+ if self.schema_version != other.schema_version {
596
+ return Some("schema_version");
597
+ }
598
+ if self.weights.tier != other.weights.tier {
599
+ return Some("weights.tier");
600
+ }
601
+ if self.weights.source_id != other.weights.source_id {
602
+ return Some("weights.source_id");
603
+ }
604
+ if self.weights.source_revision != other.weights.source_revision {
605
+ return Some("weights.source_revision");
606
+ }
607
+ if self.weights.upstream_of_record != other.weights.upstream_of_record {
608
+ return Some("weights.upstream_of_record");
609
+ }
610
+ if self.weights.files != other.weights.files {
611
+ return Some("weights.files");
612
+ }
613
+ if self.architecture != other.architecture {
614
+ return Some("architecture");
615
+ }
616
+ if self.tokenizer.tokenizer_sha256 != other.tokenizer.tokenizer_sha256 {
617
+ return Some("tokenizer.tokenizer_sha256");
618
+ }
619
+ if self.tokenizer.tokenizer_config_sha256 != other.tokenizer.tokenizer_config_sha256 {
620
+ return Some("tokenizer.tokenizer_config_sha256");
621
+ }
622
+ if self.tokenizer.chat_template_sha256 != other.tokenizer.chat_template_sha256 {
623
+ return Some("tokenizer.chat_template_sha256");
624
+ }
625
+ if self.tokenizer.tokenizer_vocab_size != other.tokenizer.tokenizer_vocab_size {
626
+ return Some("tokenizer.tokenizer_vocab_size");
627
+ }
628
+ if self.conversion != other.conversion {
629
+ return Some("conversion");
630
+ }
631
+ if self.artifact.gguf_sha256 != other.artifact.gguf_sha256 {
632
+ return Some("artifact.gguf_sha256");
633
+ }
634
+ if self.artifact.gguf_size != other.artifact.gguf_size {
635
+ return Some("artifact.gguf_size");
636
+ }
637
+ None
638
+ }
639
+ }
640
+
641
+ /// One party's claim to have reproduced the manifest.
642
+ #[derive(Debug, Clone, PartialEq, Eq)]
643
+ pub struct ReproductionAttestation {
644
+ /// Free-form label for the reproducing stack, for example `mac-metal` or `rtx-cuda`.
645
+ pub party_id: String,
646
+ /// Stable identity of the *operator* running that stack.
647
+ ///
648
+ /// Two stacks owned by the same operator are the weak form of "two parties" that ADR-0047 §1
649
+ /// records honestly rather than hiding.
650
+ pub operator_id: String,
651
+ /// The manifest hash that party computed.
652
+ pub manifest_hash: Hash64,
653
+ }
654
+
655
+ /// The set of reproduction claims collected so far.
656
+ #[derive(Debug, Clone, Default, PartialEq, Eq)]
657
+ pub struct DualReproduction {
658
+ /// Every claim, in collection order.
659
+ pub attestations: Vec<ReproductionAttestation>,
660
+ }
661
+
662
+ /// Strength of the two-party evidence actually held.
663
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
664
+ pub enum TwoPartyForm {
665
+ /// Fewer than two matching attestations.
666
+ None,
667
+ /// Two or more matching attestations, all from the same operator (ADR-0047 §1 weak form).
668
+ Weak,
669
+ /// Two or more matching attestations from distinct operators.
670
+ Strong,
671
+ }
672
+
673
+ /// A reason the recorded candidate is not the genesis.
674
+ ///
675
+ /// Mirrors the four disqualifiers frozen in `docs/model-genesis-candidate.md`.
676
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
677
+ pub enum GenesisDisqualifier {
678
+ /// (1) Provenance does not reach official upstream weights.
679
+ ProvenanceNotOfficial,
680
+ /// (2) Fewer than two parties produced a matching manifest hash.
681
+ ConversionNotReproduced,
682
+ /// (3) The manifest is unsigned, and cannot currently be signed at all.
683
+ ///
684
+ /// Unconditional: the ML-DSA-87 signature-domain table is LOCKED, no model-genesis context is
685
+ /// registered, and this module deliberately exposes no signing function. Only an explicit human
686
+ /// unlock decision can change that — writing code cannot.
687
+ ManifestUnsigned,
688
+ /// (4) The committed compute-set is not the model this manifest describes.
689
+ ComputeSetNotCommitted,
690
+ }
691
+
692
+ /// Outcome of [`assess_genesis`].
693
+ #[derive(Debug, Clone, PartialEq, Eq)]
694
+ pub struct GenesisAssessment {
695
+ /// Always `false` today; see [`GenesisDisqualifier::ManifestUnsigned`].
696
+ pub eligible: bool,
697
+ /// Every disqualifier that applies, in stable order.
698
+ pub disqualifiers: Vec<GenesisDisqualifier>,
699
+ /// Strength of the reproduction evidence held.
700
+ pub two_party_form: TwoPartyForm,
701
+ /// Structural validation error, when the manifest itself does not validate.
702
+ pub validation_error: Option<ModelGenesisError>,
703
+ }
704
+
705
+ /// Assess how far a candidate is from being a model genesis.
706
+ ///
707
+ /// `compute_set_committed` is supplied by the caller because it is a Receipt-v3 / compute-set
708
+ /// question, out of ADR-0047's scope; it is disqualifier (4) of the candidate document.
709
+ #[must_use]
710
+ pub fn assess_genesis(
711
+ manifest: &ModelGenesisManifest,
712
+ reproduction: &DualReproduction,
713
+ compute_set_committed: bool,
714
+ ) -> GenesisAssessment {
715
+ let validation_error = manifest.validate().err();
716
+ let expected = manifest.manifest_hash();
717
+ let matching: Vec<&ReproductionAttestation> = reproduction
718
+ .attestations
719
+ .iter()
720
+ .filter(|attestation| attestation.manifest_hash == expected)
721
+ .collect();
722
+ let distinct_operators = {
723
+ let mut operators: Vec<&str> = matching
724
+ .iter()
725
+ .map(|attestation| attestation.operator_id.as_str())
726
+ .collect();
727
+ operators.sort_unstable();
728
+ operators.dedup();
729
+ operators.len()
730
+ };
731
+ let two_party_form = if matching.len() < 2 {
732
+ TwoPartyForm::None
733
+ } else if distinct_operators >= 2 {
734
+ TwoPartyForm::Strong
735
+ } else {
736
+ TwoPartyForm::Weak
737
+ };
738
+
739
+ let mut disqualifiers = Vec::new();
740
+ if manifest.weights.tier != ProvenanceTier::OfficialUpstream {
741
+ disqualifiers.push(GenesisDisqualifier::ProvenanceNotOfficial);
742
+ }
743
+ if matching.len() < 2 || validation_error.is_some() {
744
+ disqualifiers.push(GenesisDisqualifier::ConversionNotReproduced);
745
+ }
746
+ // Unconditional — no registered signature context exists, so no signature can exist.
747
+ disqualifiers.push(GenesisDisqualifier::ManifestUnsigned);
748
+ if !compute_set_committed {
749
+ disqualifiers.push(GenesisDisqualifier::ComputeSetNotCommitted);
750
+ }
751
+
752
+ GenesisAssessment {
753
+ eligible: disqualifiers.is_empty(),
754
+ disqualifiers,
755
+ two_party_form,
756
+ validation_error,
757
+ }
758
+ }
759
+
760
+ #[cfg(test)]
761
+ mod tests {
762
+ use super::*;
763
+
764
+ /// The golden manifest. The identical structure is rebuilt independently by
765
+ /// `scripts/model_genesis_manifest.py --selftest`; the two implementations agreeing on
766
+ /// [`GOLDEN_MANIFEST_HASH_HEX`] is the cross-implementation parity check.
767
+ fn golden_manifest() -> ModelGenesisManifest {
768
+ ModelGenesisManifest {
769
+ schema_version: MODEL_GENESIS_SCHEMA_V1,
770
+ weights: WeightsProvenance {
771
+ tier: ProvenanceTier::OfficialUpstream,
772
+ source_id: "Example/Model-Genesis-Golden".to_owned(),
773
+ source_revision: "0123456789abcdef0123456789abcdef01234567".to_owned(),
774
+ upstream_of_record: None,
775
+ files: vec![
776
+ WeightFile {
777
+ path: "config.json".to_owned(),
778
+ sha256: [0x11; 32],
779
+ size: 1024,
780
+ },
781
+ WeightFile {
782
+ path: "model-00001-of-00002.safetensors".to_owned(),
783
+ sha256: [0x22; 32],
784
+ size: 4_294_967_296,
785
+ },
786
+ WeightFile {
787
+ path: "model-00002-of-00002.safetensors".to_owned(),
788
+ sha256: [0x33; 32],
789
+ size: 2_147_483_648,
790
+ },
791
+ WeightFile {
792
+ path: "tokenizer.json".to_owned(),
793
+ sha256: [0x44; 32],
794
+ size: 11_534_336,
795
+ },
796
+ ],
797
+ },
798
+ architecture: ModelArchitecture {
799
+ architecture: "Qwen3ForCausalLM".to_owned(),
800
+ hidden_size: 4096,
801
+ layers: 36,
802
+ attention_heads: 32,
803
+ key_value_heads: 8,
804
+ intermediate_size: 12288,
805
+ head_dim: 128,
806
+ rope_theta: 1_000_000,
807
+ vocab_size: 151_936,
808
+ max_position_embeddings: 40960,
809
+ parameter_dtype: "bfloat16".to_owned(),
810
+ },
811
+ tokenizer: TokenizerIdentity {
812
+ tokenizer_sha256: [0x44; 32],
813
+ tokenizer_config_sha256: [0x55; 32],
814
+ chat_template_sha256: [0x66; 32],
815
+ tokenizer_vocab_size: 151_936,
816
+ },
817
+ conversion: ConversionProcedure {
818
+ converter_repo: "https://github.com/ggml-org/llama.cpp.git".to_owned(),
819
+ converter_commit: "12127defda4f41b7679cb2477a4b0d65ee6a0c8f".to_owned(),
820
+ converter_entrypoint: "convert_hf_to_gguf.py".to_owned(),
821
+ converter_args: vec!["--outtype".to_owned(), "bf16".to_owned()],
822
+ quantizer_entrypoint: "llama-quantize".to_owned(),
823
+ quantizer_args: Vec::new(),
824
+ quantization: "Q4_K_M".to_owned(),
825
+ },
826
+ artifact: DerivedArtifact {
827
+ gguf_sha256: [0x77; 32],
828
+ gguf_size: 23_938_321_728,
829
+ },
830
+ }
831
+ }
832
+
833
+ const GOLDEN_MANIFEST_HASH_HEX: &str = concat!(
834
+ "b747790722bdf626a263f222c794a02ac3d32a08436bbc11a5af1f7cfaecc1b8",
835
+ "262474f15a819b4858b14fe49643c2445c771d8c8a0919bc107361ecf756d9b3"
836
+ );
837
+
838
+ #[test]
839
+ fn golden_vector_is_pinned() {
840
+ let manifest = golden_manifest();
841
+ manifest.validate().expect("golden manifest validates");
842
+ assert_eq!(
843
+ hex::encode(manifest.manifest_hash()),
844
+ GOLDEN_MANIFEST_HASH_HEX
845
+ );
846
+ }
847
+
848
+ #[test]
849
+ fn encoding_is_deterministic() {
850
+ let a = golden_manifest();
851
+ let b = golden_manifest();
852
+ assert_eq!(a.canonical_bytes(), b.canonical_bytes());
853
+ assert_eq!(a.manifest_hash(), b.manifest_hash());
854
+ assert_eq!(a.first_mismatch(&b), None);
855
+ }
856
+
857
+ #[test]
858
+ fn domain_separation_from_other_palw_digest_spaces() {
859
+ let bytes = golden_manifest().canonical_bytes();
860
+ let under_file_domain = blake2b_simd::Params::new()
861
+ .hash_length(64)
862
+ .key(b"palw-k1/file")
863
+ .to_state()
864
+ .update(&bytes)
865
+ .finalize();
866
+ assert_ne!(
867
+ golden_manifest().manifest_hash().as_slice(),
868
+ under_file_domain.as_bytes(),
869
+ "identical bytes under a different domain must not produce the same digest"
870
+ );
871
+ }
872
+
873
+ #[test]
874
+ fn encoding_is_prefix_free_across_adjacent_fields() {
875
+ let mut a = golden_manifest();
876
+ let mut b = golden_manifest();
877
+ a.weights.source_id = "ab".to_owned();
878
+ a.architecture.architecture = "c".to_owned();
879
+ b.weights.source_id = "a".to_owned();
880
+ b.architecture.architecture = "bc".to_owned();
881
+ assert_ne!(a.canonical_bytes(), b.canonical_bytes());
882
+ assert_ne!(a.manifest_hash(), b.manifest_hash());
883
+ }
884
+
885
+ /// One single-field mutation applied to a copy of the golden manifest.
886
+ type Mutator = fn(&mut ModelGenesisManifest);
887
+
888
+ #[test]
889
+ #[allow(clippy::too_many_lines)]
890
+ fn every_field_changes_the_hash() {
891
+ let base = golden_manifest();
892
+ let baseline = base.manifest_hash();
893
+ let mutators: Vec<(&str, Mutator)> = vec![
894
+ ("schema_version", |m| m.schema_version += 1),
895
+ ("weights.tier", |m| {
896
+ m.weights.tier = ProvenanceTier::ThirdPartyDerivative;
897
+ }),
898
+ ("weights.source_id", |m| m.weights.source_id.push('x')),
899
+ ("weights.source_revision", |m| {
900
+ m.weights.source_revision = "0123456789abcdef0123456789abcdef01234568".to_owned();
901
+ }),
902
+ ("weights.upstream_of_record", |m| {
903
+ m.weights.upstream_of_record = Some(UpstreamRef {
904
+ source_id: "Example/Upstream".to_owned(),
905
+ revision: "89abcdef0123456789abcdef0123456789abcdef".to_owned(),
906
+ });
907
+ }),
908
+ ("weights.files.path", |m| {
909
+ m.weights.files[0].path = "confiig.json".to_owned();
910
+ }),
911
+ ("weights.files.sha256", |m| {
912
+ m.weights.files[0].sha256[31] ^= 1;
913
+ }),
914
+ ("weights.files.size", |m| m.weights.files[0].size += 1),
915
+ ("architecture.architecture", |m| {
916
+ m.architecture.architecture.push('X');
917
+ }),
918
+ ("architecture.hidden_size", |m| {
919
+ m.architecture.hidden_size += 1;
920
+ }),
921
+ ("architecture.layers", |m| m.architecture.layers += 1),
922
+ ("architecture.attention_heads", |m| {
923
+ m.architecture.attention_heads += 1;
924
+ }),
925
+ ("architecture.key_value_heads", |m| {
926
+ m.architecture.key_value_heads += 1;
927
+ }),
928
+ ("architecture.intermediate_size", |m| {
929
+ m.architecture.intermediate_size += 1;
930
+ }),
931
+ ("architecture.head_dim", |m| m.architecture.head_dim += 1),
932
+ ("architecture.rope_theta", |m| {
933
+ m.architecture.rope_theta += 1;
934
+ }),
935
+ ("architecture.vocab_size", |m| {
936
+ m.architecture.vocab_size += 1;
937
+ }),
938
+ ("architecture.max_position_embeddings", |m| {
939
+ m.architecture.max_position_embeddings += 1;
940
+ }),
941
+ ("architecture.parameter_dtype", |m| {
942
+ m.architecture.parameter_dtype = "float16".to_owned();
943
+ }),
944
+ ("tokenizer.tokenizer_sha256", |m| {
945
+ m.tokenizer.tokenizer_sha256[0] ^= 1;
946
+ }),
947
+ ("tokenizer.tokenizer_config_sha256", |m| {
948
+ m.tokenizer.tokenizer_config_sha256[0] ^= 1;
949
+ }),
950
+ ("tokenizer.chat_template_sha256", |m| {
951
+ m.tokenizer.chat_template_sha256[0] ^= 1;
952
+ }),
953
+ ("tokenizer.tokenizer_vocab_size", |m| {
954
+ m.tokenizer.tokenizer_vocab_size += 1;
955
+ }),
956
+ ("conversion.converter_repo", |m| {
957
+ m.conversion.converter_repo.push('/');
958
+ }),
959
+ ("conversion.converter_commit", |m| {
960
+ m.conversion.converter_commit =
961
+ "12127defda4f41b7679cb2477a4b0d65ee6a0c8e".to_owned();
962
+ }),
963
+ ("conversion.converter_entrypoint", |m| {
964
+ m.conversion.converter_entrypoint = "convert.py".to_owned();
965
+ }),
966
+ ("conversion.converter_args", |m| {
967
+ m.conversion.converter_args.push("f32".to_owned());
968
+ }),
969
+ ("conversion.quantizer_entrypoint", |m| {
970
+ m.conversion.quantizer_entrypoint = "quantize".to_owned();
971
+ }),
972
+ ("conversion.quantizer_args", |m| {
973
+ m.conversion.quantizer_args.push("--pure".to_owned());
974
+ }),
975
+ ("conversion.quantization", |m| {
976
+ m.conversion.quantization = "Q5_K_M".to_owned();
977
+ }),
978
+ ("artifact.gguf_sha256", |m| m.artifact.gguf_sha256[17] ^= 1),
979
+ ("artifact.gguf_size", |m| m.artifact.gguf_size -= 1),
980
+ ];
981
+ let mut seen = Vec::new();
982
+ for (name, mutate) in mutators {
983
+ let mut mutated = base.clone();
984
+ mutate(&mut mutated);
985
+ let hash = mutated.manifest_hash();
986
+ assert_ne!(
987
+ hash, baseline,
988
+ "{name}: mutation did not change the manifest hash"
989
+ );
990
+ assert!(
991
+ base.first_mismatch(&mutated).is_some(),
992
+ "{name}: diff did not report a field"
993
+ );
994
+ assert!(
995
+ !seen.contains(&hash),
996
+ "{name}: mutation collided with another mutation"
997
+ );
998
+ seen.push(hash);
999
+ }
1000
+ }
1001
+
1002
+ #[test]
1003
+ fn backend_specific_conversion_is_rejected() {
1004
+ for bad in ["--n-gpu-layers", "-ngl", "CUDA", "metal", "--threads"] {
1005
+ let mut manifest = golden_manifest();
1006
+ manifest.conversion.converter_args = vec![bad.to_owned()];
1007
+ assert!(
1008
+ matches!(
1009
+ manifest.validate(),
1010
+ Err(ModelGenesisError::BackendSpecificField { .. })
1011
+ ),
1012
+ "{bad}: backend/host-specific argument must be rejected"
1013
+ );
1014
+ }
1015
+ }
1016
+
1017
+ #[test]
1018
+ fn host_paths_and_mutable_pins_are_rejected() {
1019
+ let mut absolute = golden_manifest();
1020
+ absolute.weights.files[0].path = "/Users/operator/config.json".to_owned();
1021
+ assert!(matches!(
1022
+ absolute.validate(),
1023
+ Err(ModelGenesisError::UnsafePath { .. })
1024
+ ));
1025
+
1026
+ let mut traversal = golden_manifest();
1027
+ traversal.conversion.converter_entrypoint = "../convert_hf_to_gguf.py".to_owned();
1028
+ assert!(matches!(
1029
+ traversal.validate(),
1030
+ Err(ModelGenesisError::UnsafePath { .. })
1031
+ ));
1032
+
1033
+ let mut branch = golden_manifest();
1034
+ branch.weights.source_revision = "main".to_owned();
1035
+ assert!(matches!(
1036
+ branch.validate(),
1037
+ Err(ModelGenesisError::NotARevisionPin { .. })
1038
+ ));
1039
+
1040
+ let mut unsorted = golden_manifest();
1041
+ unsorted.weights.files.swap(0, 1);
1042
+ assert!(matches!(
1043
+ unsorted.validate(),
1044
+ Err(ModelGenesisError::UnsortedFiles(..))
1045
+ ));
1046
+
1047
+ let mut non_ascii = golden_manifest();
1048
+ non_ascii.weights.source_id = "Example/モデル".to_owned();
1049
+ assert!(matches!(
1050
+ non_ascii.validate(),
1051
+ Err(ModelGenesisError::NonPrintableAscii { .. })
1052
+ ));
1053
+ }
1054
+
1055
+ fn attestation(party: &str, operator: &str, hash: Hash64) -> ReproductionAttestation {
1056
+ ReproductionAttestation {
1057
+ party_id: party.to_owned(),
1058
+ operator_id: operator.to_owned(),
1059
+ manifest_hash: hash,
1060
+ }
1061
+ }
1062
+
1063
+ #[test]
1064
+ fn no_input_can_produce_an_eligible_genesis() {
1065
+ let manifest = golden_manifest();
1066
+ let hash = manifest.manifest_hash();
1067
+ // The most favourable input that can exist in-repo: official tier, two independent
1068
+ // operators matching bit-exactly, compute-set committed.
1069
+ let best = DualReproduction {
1070
+ attestations: vec![
1071
+ attestation("mac-metal", "operator-a", hash),
1072
+ attestation("rtx-cuda", "operator-b", hash),
1073
+ ],
1074
+ };
1075
+ let assessment = assess_genesis(&manifest, &best, true);
1076
+ assert!(
1077
+ !assessment.eligible,
1078
+ "genesis must never be reachable by writing code"
1079
+ );
1080
+ assert_eq!(
1081
+ assessment.disqualifiers,
1082
+ vec![GenesisDisqualifier::ManifestUnsigned]
1083
+ );
1084
+ assert_eq!(assessment.two_party_form, TwoPartyForm::Strong);
1085
+ assert!(assessment.validation_error.is_none());
1086
+ }
1087
+
1088
+ #[test]
1089
+ fn same_operator_two_stacks_is_only_the_weak_form() {
1090
+ let manifest = golden_manifest();
1091
+ let hash = manifest.manifest_hash();
1092
+ let weak = DualReproduction {
1093
+ attestations: vec![
1094
+ attestation("mac-metal", "operator-a", hash),
1095
+ attestation("rtx-cuda", "operator-a", hash),
1096
+ ],
1097
+ };
1098
+ assert_eq!(
1099
+ assess_genesis(&manifest, &weak, true).two_party_form,
1100
+ TwoPartyForm::Weak
1101
+ );
1102
+ }
1103
+
1104
+ #[test]
1105
+ fn current_pins_record_every_candidate_disqualifier() {
1106
+ // The shape of the pins actually in `config/runtime-pins.sh`: a third-party abliterated
1107
+ // derivative, one party, and a compute-set that is the integer-canonical 0.5B set.
1108
+ let mut manifest = golden_manifest();
1109
+ manifest.weights.tier = ProvenanceTier::ThirdPartyDerivative;
1110
+ manifest.weights.source_id =
1111
+ "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated".to_owned();
1112
+ manifest.weights.source_revision = "ac18882735d037f6074a7630eb68d85db8234c25".to_owned();
1113
+ let alone = DualReproduction {
1114
+ attestations: vec![attestation(
1115
+ "mac-metal",
1116
+ "operator-a",
1117
+ manifest.manifest_hash(),
1118
+ )],
1119
+ };
1120
+ let assessment = assess_genesis(&manifest, &alone, false);
1121
+ assert_eq!(
1122
+ assessment.disqualifiers,
1123
+ vec![
1124
+ GenesisDisqualifier::ProvenanceNotOfficial,
1125
+ GenesisDisqualifier::ConversionNotReproduced,
1126
+ GenesisDisqualifier::ManifestUnsigned,
1127
+ GenesisDisqualifier::ComputeSetNotCommitted,
1128
+ ]
1129
+ );
1130
+ assert_eq!(assessment.two_party_form, TwoPartyForm::None);
1131
+ }
1132
+
1133
+ #[test]
1134
+ fn proposed_context_is_distinct_and_prefix_free() {
1135
+ use crate::receipt_v3::{PALW_JOBSPEC_V2_MLDSA87_CONTEXT, PALW_RECEIPT_V3_MLDSA87_CONTEXT};
1136
+ let known: [&[u8]; 2] = [
1137
+ PALW_RECEIPT_V3_MLDSA87_CONTEXT,
1138
+ PALW_JOBSPEC_V2_MLDSA87_CONTEXT,
1139
+ ];
1140
+ for other in known {
1141
+ assert_ne!(PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED, other);
1142
+ assert!(!PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED.starts_with(other));
1143
+ assert!(!other.starts_with(PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED));
1144
+ }
1145
+ // The hash domain and the signature context live in different namespaces, but keeping them
1146
+ // textually distinct removes any chance of one being pasted where the other belongs.
1147
+ assert_ne!(
1148
+ PALW_MODEL_GENESIS_V1_HASH_DOMAIN,
1149
+ PALW_MODEL_GENESIS_V1_MLDSA87_CONTEXT_UNREGISTERED
1150
+ );
1151
+ }
1152
+ }
scripts/model_genesis_manifest.py ADDED
@@ -0,0 +1,716 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Deterministic model-genesis conversion and manifest tool (ADR-0047 prerequisite).
3
+
4
+ ADR-0047 fixes the two-party reproduction protocol but its acceptance criterion — a bit-exact
5
+ `ModelGenesisManifest` hash match between two operators — was unevaluable, because no tool built
6
+ the manifest. This script is the procedure both operators run, so that two runs are comparable.
7
+
8
+ It is an independent reimplementation of `runtime-palw/src/model_genesis.rs`. The two
9
+ implementations agreeing on the pinned golden vector (`selftest`) is the parity check; a silent
10
+ divergence between them would make a cross-operator mismatch unattributable.
11
+
12
+ Honesty boundary (do not paper over it):
13
+
14
+ * The weight pins currently shipped in ``config/runtime-pins.sh`` terminate at a THIRD-PARTY
15
+ "abliterated" derivative, not official Alibaba weights. A manifest built from them is
16
+ reproducible but can never be the genesis — the tier is recorded in the hash, and
17
+ ``--provenance-tier third-party-derivative`` is what those pins require.
18
+ * Nothing here signs anything. The ML-DSA-87 signature-domain table is LOCKED (2026-07-25);
19
+ the model-genesis context is not registered, so a signed manifest cannot be produced.
20
+ * Every command fails closed. Missing weights, a missing converter, a missing quantizer, or an
21
+ unpinned checkout is an error naming exactly what is absent — never a manifest built from
22
+ inputs that were not there.
23
+
24
+ Subcommands
25
+ -----------
26
+ selftest Re-derive the golden vector and compare with the Rust golden constant.
27
+ convert Check every precondition and emit the exact conversion argv (runs it with --execute).
28
+ manifest Hash real local inputs and emit manifest JSON + its keyed-BLAKE2b-512 hash.
29
+ verify Compare two or more manifest JSON files; report the first differing field.
30
+ """
31
+
32
+ from __future__ import annotations
33
+
34
+ import argparse
35
+ import hashlib
36
+ import json
37
+ import os
38
+ import shutil
39
+ import subprocess
40
+ import sys
41
+ from pathlib import Path
42
+ from typing import Any
43
+
44
+ SCHEMA_NAME = "misaka.palw.model-genesis.v1"
45
+ SCHEMA_VERSION = 1
46
+
47
+ # Keyed-BLAKE2b-512 domain. Distinct from every receipt/file domain (`palw-k1/file`,
48
+ # `misaka-palw-v3/...`): identical bytes under a different domain must not collide, or a commitment
49
+ # over one object class silently becomes a commitment over another.
50
+ HASH_DOMAIN = b"misaka-palw-v1/model-genesis"
51
+
52
+ # The ML-DSA-87 context a signed genesis WOULD need. UNREGISTERED — the signature-domain table is
53
+ # LOCKED and adding a row is an explicit human unlock decision. Recorded, never used.
54
+ MLDSA87_CONTEXT_UNREGISTERED = b"misaka-palw-v1/model-genesis/mldsa87"
55
+
56
+ # Pinned in runtime-palw/src/model_genesis.rs::tests::GOLDEN_MANIFEST_HASH_HEX.
57
+ GOLDEN_MANIFEST_HASH_HEX = (
58
+ "b747790722bdf626a263f222c794a02ac3d32a08436bbc11a5af1f7cfaecc1b8"
59
+ "262474f15a819b4858b14fe49643c2445c771d8c8a0919bc107361ecf756d9b3"
60
+ )
61
+
62
+ PROVENANCE_TIER_TAGS = {"official-upstream": 0, "third-party-derivative": 1}
63
+
64
+ # Mirrors BACKEND_TOKENS in model_genesis.rs. Short ambiguous substrings are excluded on purpose.
65
+ BACKEND_TOKENS = (
66
+ "metal", "cuda", "cublas", "rocm", "vulkan", "sycl", "opencl", "gpu", "ngl",
67
+ "device", "thread", "darwin", "arm64", "x86_64", "avx", "neon",
68
+ )
69
+
70
+ REQUIRED_METADATA_FILES = ("config.json", "tokenizer.json", "tokenizer_config.json")
71
+ HASH_CHUNK_BYTES = 1 << 22
72
+
73
+
74
+ class FailClosed(SystemExit):
75
+ """Abort with an explicit message naming what was missing or wrong."""
76
+
77
+ def __init__(self, message: str) -> None:
78
+ super().__init__(f"model-genesis: {message}")
79
+
80
+
81
+ # --------------------------------------------------------------------------------------------
82
+ # canonical encoding — byte-for-byte identical to runtime-palw/src/model_genesis.rs
83
+ # --------------------------------------------------------------------------------------------
84
+
85
+
86
+ def _framed(value: bytes) -> bytes:
87
+ return len(value).to_bytes(8, "big") + value
88
+
89
+
90
+ def _text(field: str, value: Any) -> bytes:
91
+ if not isinstance(value, str):
92
+ raise FailClosed(f"{field}: expected a string, got {type(value).__name__}")
93
+ for index, byte in enumerate(value.encode("utf-8")):
94
+ if not 0x20 <= byte <= 0x7E:
95
+ raise FailClosed(f"{field}: byte {index} is outside printable ASCII")
96
+ return value.encode("ascii")
97
+
98
+
99
+ def _u(field: str, value: Any, width: int) -> bytes:
100
+ if not isinstance(value, int) or isinstance(value, bool) or value < 0:
101
+ raise FailClosed(f"{field}: expected a non-negative integer, got {value!r}")
102
+ if value >> (width * 8):
103
+ raise FailClosed(f"{field}: {value} does not fit in u{width * 8}")
104
+ return value.to_bytes(width, "big")
105
+
106
+
107
+ def _digest(field: str, value: Any) -> bytes:
108
+ if not isinstance(value, str) or len(value) != 64 or any(c not in "0123456789abcdef" for c in value):
109
+ raise FailClosed(f"{field}: expected 64 lowercase hex characters, got {value!r}")
110
+ return bytes.fromhex(value)
111
+
112
+
113
+ def _string_list(field: str, values: Any) -> bytes:
114
+ if not isinstance(values, list):
115
+ raise FailClosed(f"{field}: expected a list")
116
+ out = len(values).to_bytes(8, "big")
117
+ for index, value in enumerate(values):
118
+ out += _framed(_text(f"{field}[{index}]", value))
119
+ return out
120
+
121
+
122
+ def canonical_bytes(manifest: dict[str, Any]) -> bytes:
123
+ """Canonical encoding: big-endian integers, u64-length-framed variable fields, declaration order."""
124
+ weights = manifest["weights"]
125
+ arch = manifest["architecture"]
126
+ tok = manifest["tokenizer"]
127
+ conv = manifest["conversion"]
128
+ art = manifest["artifact"]
129
+
130
+ tier = weights["tier"]
131
+ if tier not in PROVENANCE_TIER_TAGS:
132
+ raise FailClosed(f"weights.tier: unknown tier {tier!r}")
133
+
134
+ out = _u("schema_version", manifest["schema_version"], 2)
135
+ out += bytes([PROVENANCE_TIER_TAGS[tier]])
136
+ out += _framed(_text("weights.source_id", weights["source_id"]))
137
+ out += _framed(_text("weights.source_revision", weights["source_revision"]))
138
+ upstream = weights.get("upstream_of_record")
139
+ if upstream is None:
140
+ out += b"\x00"
141
+ else:
142
+ out += b"\x01"
143
+ out += _framed(_text("weights.upstream_of_record.source_id", upstream["source_id"]))
144
+ out += _framed(_text("weights.upstream_of_record.revision", upstream["revision"]))
145
+ files = weights["files"]
146
+ out += len(files).to_bytes(8, "big")
147
+ for entry in files:
148
+ out += _framed(_text("weights.files.path", entry["path"]))
149
+ out += _digest("weights.files.sha256", entry["sha256"])
150
+ out += _u("weights.files.size", entry["size"], 8)
151
+
152
+ out += _framed(_text("architecture.architecture", arch["architecture"]))
153
+ for name in (
154
+ "hidden_size",
155
+ "layers",
156
+ "attention_heads",
157
+ "key_value_heads",
158
+ "intermediate_size",
159
+ "head_dim",
160
+ ):
161
+ out += _u(f"architecture.{name}", arch[name], 4)
162
+ out += _u("architecture.rope_theta", arch["rope_theta"], 8)
163
+ out += _u("architecture.vocab_size", arch["vocab_size"], 4)
164
+ out += _u("architecture.max_position_embeddings", arch["max_position_embeddings"], 4)
165
+ out += _framed(_text("architecture.parameter_dtype", arch["parameter_dtype"]))
166
+
167
+ out += _digest("tokenizer.tokenizer_sha256", tok["tokenizer_sha256"])
168
+ out += _digest("tokenizer.tokenizer_config_sha256", tok["tokenizer_config_sha256"])
169
+ out += _digest("tokenizer.chat_template_sha256", tok["chat_template_sha256"])
170
+ out += _u("tokenizer.tokenizer_vocab_size", tok["tokenizer_vocab_size"], 4)
171
+
172
+ out += _framed(_text("conversion.converter_repo", conv["converter_repo"]))
173
+ out += _framed(_text("conversion.converter_commit", conv["converter_commit"]))
174
+ out += _framed(_text("conversion.converter_entrypoint", conv["converter_entrypoint"]))
175
+ out += _string_list("conversion.converter_args", conv["converter_args"])
176
+ out += _framed(_text("conversion.quantizer_entrypoint", conv["quantizer_entrypoint"]))
177
+ out += _string_list("conversion.quantizer_args", conv["quantizer_args"])
178
+ out += _framed(_text("conversion.quantization", conv["quantization"]))
179
+
180
+ out += _digest("artifact.gguf_sha256", art["gguf_sha256"])
181
+ out += _u("artifact.gguf_size", art["gguf_size"], 8)
182
+ return out
183
+
184
+
185
+ def manifest_hash(manifest: dict[str, Any]) -> str:
186
+ """Keyed BLAKE2b-512 of the canonical bytes, hex — the value ADR-0047 compares between parties."""
187
+ return hashlib.blake2b(canonical_bytes(manifest), key=HASH_DOMAIN, digest_size=64).hexdigest()
188
+
189
+
190
+ # --------------------------------------------------------------------------------------------
191
+ # validation — mirrors ModelGenesisManifest::validate
192
+ # --------------------------------------------------------------------------------------------
193
+
194
+
195
+ def _check_revision_pin(field: str, value: str) -> None:
196
+ if len(value) not in (40, 64) or any(c not in "0123456789abcdef" for c in value):
197
+ raise FailClosed(
198
+ f"{field}: {value!r} is not an immutable revision pin (40 or 64 lowercase hex). "
199
+ "A branch or tag name is not reproducible."
200
+ )
201
+
202
+
203
+ def _check_relative_path(field: str, value: str) -> None:
204
+ parts = value.split("/")
205
+ if not value or value.startswith("/") or value.startswith("~") or any(p in ("", "..") for p in parts):
206
+ raise FailClosed(f"{field}: {value!r} must be a relative, traversal-free path")
207
+
208
+
209
+ def _check_backend_free(field: str, value: str) -> None:
210
+ lowered = value.lower()
211
+ for token in BACKEND_TOKENS:
212
+ if token in lowered:
213
+ raise FailClosed(
214
+ f"conversion.{field}: contains {token!r} — the manifest must be weights-derived only; "
215
+ "a backend/host-specific input makes the two-party comparison meaningless (ADR-0047 §1)"
216
+ )
217
+
218
+
219
+ def validate(manifest: dict[str, Any]) -> None:
220
+ """Reject anything that would make two operators' manifests incomparable."""
221
+ weights = manifest["weights"]
222
+ _check_revision_pin("weights.source_revision", weights["source_revision"])
223
+ upstream = weights.get("upstream_of_record")
224
+ if upstream is not None:
225
+ _check_revision_pin("weights.upstream_of_record.revision", upstream["revision"])
226
+ files = weights["files"]
227
+ if not files:
228
+ raise FailClosed("weights.files: at least one weight file must be recorded")
229
+ for entry in files:
230
+ _check_relative_path("weights.files.path", entry["path"])
231
+ for previous, following in zip(files, files[1:]):
232
+ if previous["path"] >= following["path"]:
233
+ raise FailClosed(
234
+ f"weights.files: must be strictly ascending by path "
235
+ f"({previous['path']!r} is not before {following['path']!r})"
236
+ )
237
+
238
+ arch = manifest["architecture"]
239
+ for name in (
240
+ "hidden_size",
241
+ "layers",
242
+ "attention_heads",
243
+ "key_value_heads",
244
+ "intermediate_size",
245
+ "head_dim",
246
+ "rope_theta",
247
+ "vocab_size",
248
+ "max_position_embeddings",
249
+ ):
250
+ if arch[name] == 0:
251
+ raise FailClosed(f"architecture.{name}: must be non-zero")
252
+ if manifest["tokenizer"]["tokenizer_vocab_size"] == 0:
253
+ raise FailClosed("tokenizer.tokenizer_vocab_size: must be non-zero")
254
+ if manifest["artifact"]["gguf_size"] == 0:
255
+ raise FailClosed("artifact.gguf_size: must be non-zero")
256
+
257
+ conv = manifest["conversion"]
258
+ for name in ("converter_repo", "converter_commit", "converter_entrypoint", "quantization"):
259
+ _check_backend_free(name, conv[name])
260
+ for name in ("converter_args", "quantizer_args"):
261
+ for arg in conv[name]:
262
+ _check_backend_free(name, arg)
263
+ _check_revision_pin("conversion.converter_commit", conv["converter_commit"])
264
+ _check_relative_path("conversion.converter_entrypoint", conv["converter_entrypoint"])
265
+ _check_relative_path("conversion.quantizer_entrypoint", conv["quantizer_entrypoint"])
266
+ canonical_bytes(manifest)
267
+
268
+
269
+ FIELD_ORDER = (
270
+ ("schema_version", lambda m: m["schema_version"]),
271
+ ("weights.tier", lambda m: m["weights"]["tier"]),
272
+ ("weights.source_id", lambda m: m["weights"]["source_id"]),
273
+ ("weights.source_revision", lambda m: m["weights"]["source_revision"]),
274
+ ("weights.upstream_of_record", lambda m: m["weights"].get("upstream_of_record")),
275
+ ("weights.files", lambda m: m["weights"]["files"]),
276
+ ("architecture", lambda m: m["architecture"]),
277
+ ("tokenizer.tokenizer_sha256", lambda m: m["tokenizer"]["tokenizer_sha256"]),
278
+ ("tokenizer.tokenizer_config_sha256", lambda m: m["tokenizer"]["tokenizer_config_sha256"]),
279
+ ("tokenizer.chat_template_sha256", lambda m: m["tokenizer"]["chat_template_sha256"]),
280
+ ("tokenizer.tokenizer_vocab_size", lambda m: m["tokenizer"]["tokenizer_vocab_size"]),
281
+ ("conversion", lambda m: m["conversion"]),
282
+ ("artifact.gguf_sha256", lambda m: m["artifact"]["gguf_sha256"]),
283
+ ("artifact.gguf_size", lambda m: m["artifact"]["gguf_size"]),
284
+ )
285
+
286
+
287
+ def first_mismatch(left: dict[str, Any], right: dict[str, Any]) -> str | None:
288
+ """First differing field, dotted — mirrors ModelGenesisManifest::first_mismatch."""
289
+ for name, getter in FIELD_ORDER:
290
+ if getter(left) != getter(right):
291
+ return name
292
+ return None
293
+
294
+
295
+ # --------------------------------------------------------------------------------------------
296
+ # local input hashing
297
+ # --------------------------------------------------------------------------------------------
298
+
299
+
300
+ def sha256_file(path: Path) -> tuple[str, int]:
301
+ digest = hashlib.sha256()
302
+ size = 0
303
+ with path.open("rb") as handle:
304
+ while chunk := handle.read(HASH_CHUNK_BYTES):
305
+ digest.update(chunk)
306
+ size += len(chunk)
307
+ return digest.hexdigest(), size
308
+
309
+
310
+ def collect_weight_files(weights_dir: Path) -> list[dict[str, Any]]:
311
+ """Hash every file under ``weights_dir``, failing closed on an empty or weightless directory."""
312
+ if not weights_dir.is_dir():
313
+ raise FailClosed(f"--weights-dir {weights_dir} does not exist")
314
+ entries: list[dict[str, Any]] = []
315
+ for path in sorted(weights_dir.rglob("*")):
316
+ if not path.is_file() or path.is_symlink():
317
+ continue
318
+ relative = path.relative_to(weights_dir).as_posix()
319
+ if relative.startswith(".") or "/." in relative:
320
+ continue
321
+ sha256, size = sha256_file(path)
322
+ entries.append({"path": relative, "sha256": sha256, "size": size})
323
+ if not entries:
324
+ raise FailClosed(f"--weights-dir {weights_dir} contains no files")
325
+ names = {entry["path"] for entry in entries}
326
+ missing = [name for name in REQUIRED_METADATA_FILES if name not in names]
327
+ if missing:
328
+ raise FailClosed(
329
+ f"--weights-dir {weights_dir} is missing required checkpoint files: {', '.join(missing)}"
330
+ )
331
+ if not any(entry["path"].endswith((".safetensors", ".bin")) for entry in entries):
332
+ raise FailClosed(
333
+ f"--weights-dir {weights_dir} contains metadata only — no *.safetensors/*.bin weight "
334
+ "shards were found. Download the full checkpoint; a manifest must never be built from "
335
+ "absent weights."
336
+ )
337
+ entries.sort(key=lambda entry: entry["path"])
338
+ return entries
339
+
340
+
341
+ def architecture_from_config(config_path: Path) -> dict[str, Any]:
342
+ config = json.loads(config_path.read_text(encoding="utf-8"))
343
+ architectures = config.get("architectures")
344
+ if not isinstance(architectures, list) or not architectures:
345
+ raise FailClosed(f"{config_path}: 'architectures' is missing or empty")
346
+ rope_theta = config.get("rope_theta")
347
+ if isinstance(rope_theta, float):
348
+ if not rope_theta.is_integer():
349
+ raise FailClosed(
350
+ f"{config_path}: rope_theta={rope_theta} is non-integral. Extend the manifest schema "
351
+ "rather than rounding — rounding is a per-implementation difference and would break "
352
+ "the bit-exact comparison."
353
+ )
354
+ rope_theta = int(rope_theta)
355
+ mapping = {
356
+ "hidden_size": "hidden_size",
357
+ "layers": "num_hidden_layers",
358
+ "attention_heads": "num_attention_heads",
359
+ "key_value_heads": "num_key_value_heads",
360
+ "intermediate_size": "intermediate_size",
361
+ "head_dim": "head_dim",
362
+ "vocab_size": "vocab_size",
363
+ "max_position_embeddings": "max_position_embeddings",
364
+ }
365
+ architecture: dict[str, Any] = {"architecture": architectures[0]}
366
+ for field, key in mapping.items():
367
+ value = config.get(key)
368
+ if not isinstance(value, int) or isinstance(value, bool):
369
+ raise FailClosed(f"{config_path}: {key!r} is missing or not an integer")
370
+ architecture[field] = value
371
+ architecture["rope_theta"] = rope_theta
372
+ architecture["parameter_dtype"] = config.get("torch_dtype") or config.get("dtype")
373
+ if not isinstance(architecture["parameter_dtype"], str):
374
+ raise FailClosed(f"{config_path}: 'torch_dtype'/'dtype' is missing")
375
+ ordered = (
376
+ "architecture", "hidden_size", "layers", "attention_heads", "key_value_heads",
377
+ "intermediate_size", "head_dim", "rope_theta", "vocab_size",
378
+ "max_position_embeddings", "parameter_dtype",
379
+ )
380
+ return {key: architecture[key] for key in ordered}
381
+
382
+
383
+ def tokenizer_vocab_size(tokenizer_path: Path) -> int:
384
+ """Highest token id + 1 — a single unambiguous rule, so both operators compute the same number."""
385
+ tokenizer = json.loads(tokenizer_path.read_text(encoding="utf-8"))
386
+ ids: list[int] = []
387
+ vocab = tokenizer.get("model", {}).get("vocab")
388
+ if isinstance(vocab, dict):
389
+ ids.extend(int(value) for value in vocab.values())
390
+ elif isinstance(vocab, list):
391
+ ids.extend(range(len(vocab)))
392
+ else:
393
+ raise FailClosed(f"{tokenizer_path}: 'model.vocab' is missing or has an unsupported shape")
394
+ for token in tokenizer.get("added_tokens", []) or []:
395
+ if isinstance(token, dict) and isinstance(token.get("id"), int):
396
+ ids.append(token["id"])
397
+ if not ids:
398
+ raise FailClosed(f"{tokenizer_path}: no token ids found")
399
+ return max(ids) + 1
400
+
401
+
402
+ # --------------------------------------------------------------------------------------------
403
+ # subcommands
404
+ # --------------------------------------------------------------------------------------------
405
+
406
+
407
+ def golden_manifest() -> dict[str, Any]:
408
+ """The vector duplicated in runtime-palw/src/model_genesis.rs::tests::golden_manifest."""
409
+ return {
410
+ "schema": SCHEMA_NAME,
411
+ "schema_version": SCHEMA_VERSION,
412
+ "weights": {
413
+ "tier": "official-upstream",
414
+ "source_id": "Example/Model-Genesis-Golden",
415
+ "source_revision": "0123456789abcdef0123456789abcdef01234567",
416
+ "upstream_of_record": None,
417
+ "files": [
418
+ {"path": "config.json", "sha256": "11" * 32, "size": 1024},
419
+ {"path": "model-00001-of-00002.safetensors", "sha256": "22" * 32, "size": 4294967296},
420
+ {"path": "model-00002-of-00002.safetensors", "sha256": "33" * 32, "size": 2147483648},
421
+ {"path": "tokenizer.json", "sha256": "44" * 32, "size": 11534336},
422
+ ],
423
+ },
424
+ "architecture": {
425
+ "architecture": "Qwen3ForCausalLM",
426
+ "hidden_size": 4096,
427
+ "layers": 36,
428
+ "attention_heads": 32,
429
+ "key_value_heads": 8,
430
+ "intermediate_size": 12288,
431
+ "head_dim": 128,
432
+ "rope_theta": 1000000,
433
+ "vocab_size": 151936,
434
+ "max_position_embeddings": 40960,
435
+ "parameter_dtype": "bfloat16",
436
+ },
437
+ "tokenizer": {
438
+ "tokenizer_sha256": "44" * 32,
439
+ "tokenizer_config_sha256": "55" * 32,
440
+ "chat_template_sha256": "66" * 32,
441
+ "tokenizer_vocab_size": 151936,
442
+ },
443
+ "conversion": {
444
+ "converter_repo": "https://github.com/ggml-org/llama.cpp.git",
445
+ "converter_commit": "12127defda4f41b7679cb2477a4b0d65ee6a0c8f",
446
+ "converter_entrypoint": "convert_hf_to_gguf.py",
447
+ "converter_args": ["--outtype", "bf16"],
448
+ "quantizer_entrypoint": "llama-quantize",
449
+ "quantizer_args": [],
450
+ "quantization": "Q4_K_M",
451
+ },
452
+ "artifact": {"gguf_sha256": "77" * 32, "gguf_size": 23938321728},
453
+ }
454
+
455
+
456
+ def cmd_selftest(_args: argparse.Namespace) -> int:
457
+ manifest = golden_manifest()
458
+ validate(manifest)
459
+ computed = manifest_hash(manifest)
460
+ if computed != GOLDEN_MANIFEST_HASH_HEX:
461
+ raise FailClosed(
462
+ "PARITY BROKEN: this script and runtime-palw/src/model_genesis.rs disagree on the golden "
463
+ f"vector.\n python: {computed}\n rust: {GOLDEN_MANIFEST_HASH_HEX}\n"
464
+ "A cross-operator mismatch would be unattributable until this is fixed."
465
+ )
466
+ mutated = golden_manifest()
467
+ mutated["artifact"]["gguf_sha256"] = "78" * 32
468
+ if manifest_hash(mutated) == computed or first_mismatch(manifest, mutated) != "artifact.gguf_sha256":
469
+ raise FailClosed("field sensitivity check failed")
470
+ print(f"golden vector OK (Rust/Python parity): {computed}")
471
+ print(f"hash domain: {HASH_DOMAIN.decode()}")
472
+ print(f"ML-DSA context (UNREGISTERED, unusable): {MLDSA87_CONTEXT_UNREGISTERED.decode()}")
473
+ return 0
474
+
475
+
476
+ def _require_pinned_checkout(llama_dir: Path, commit: str) -> None:
477
+ if not llama_dir.is_dir():
478
+ raise FailClosed(f"--llama-dir {llama_dir} does not exist")
479
+ converter = llama_dir / "convert_hf_to_gguf.py"
480
+ if not converter.is_file():
481
+ raise FailClosed(f"missing converter: {converter} (clone llama.cpp at {commit})")
482
+ if shutil.which("git") is None:
483
+ raise FailClosed("missing tool: git — the converter commit pin cannot be verified")
484
+ result = subprocess.run(
485
+ ["git", "-C", str(llama_dir), "rev-parse", "HEAD"],
486
+ capture_output=True,
487
+ text=True,
488
+ check=False,
489
+ )
490
+ if result.returncode != 0:
491
+ raise FailClosed(f"{llama_dir} is not a git checkout: {result.stderr.strip()}")
492
+ head = result.stdout.strip()
493
+ if head != commit:
494
+ raise FailClosed(f"{llama_dir} is at {head}, not the pinned converter commit {commit}")
495
+
496
+
497
+ def cmd_convert(args: argparse.Namespace) -> int:
498
+ weights_dir: Path = args.weights_dir
499
+ if not weights_dir.is_dir():
500
+ raise FailClosed(f"--weights-dir {weights_dir} does not exist")
501
+ if not any(weights_dir.rglob("*.safetensors")) and not any(weights_dir.rglob("*.bin")):
502
+ raise FailClosed(
503
+ f"--weights-dir {weights_dir} holds no *.safetensors/*.bin shards — the conversion has "
504
+ "no input. Download the full checkpoint first."
505
+ )
506
+ _require_pinned_checkout(args.llama_dir, args.converter_commit)
507
+
508
+ quantizer = args.quantizer
509
+ if quantizer is None or not Path(quantizer).is_file():
510
+ raise FailClosed(
511
+ f"missing quantizer binary: {quantizer or '--quantizer not given'}. Build it from the "
512
+ f"pinned checkout: cmake --build {args.llama_dir}/build-palw --target llama-quantize"
513
+ )
514
+ if args.out_gguf.exists():
515
+ raise FailClosed(f"--out-gguf {args.out_gguf} already exists; refusing to overwrite")
516
+
517
+ intermediate = args.out_gguf.with_suffix(".bf16.gguf")
518
+ convert_argv = [
519
+ sys.executable,
520
+ str(args.llama_dir / "convert_hf_to_gguf.py"),
521
+ str(weights_dir),
522
+ "--outfile",
523
+ str(intermediate),
524
+ "--outtype",
525
+ "bf16",
526
+ "--no-lazy",
527
+ "--model-name",
528
+ args.model_name,
529
+ ]
530
+ quantize_argv = [str(quantizer), str(intermediate), str(args.out_gguf), args.quantization]
531
+ # A fixed environment: locale/timezone/hash-seed differences between two operators are exactly
532
+ # the kind of host state that must not reach the artifact.
533
+ env = {
534
+ "PATH": os.environ.get("PATH", ""),
535
+ "HOME": os.environ.get("HOME", ""),
536
+ "LC_ALL": "C",
537
+ "TZ": "UTC",
538
+ "PYTHONHASHSEED": "0",
539
+ "SOURCE_DATE_EPOCH": "0",
540
+ }
541
+ print("conversion procedure (both operators must run exactly this):")
542
+ print(" 1) " + " ".join(convert_argv))
543
+ print(" 2) " + " ".join(quantize_argv))
544
+ print(" env: " + " ".join(f"{k}={v}" for k, v in sorted(env.items()) if k not in ("PATH", "HOME")))
545
+ if not args.execute:
546
+ print("dry run: nothing was executed (pass --execute to run)")
547
+ return 0
548
+ for argv in (convert_argv, quantize_argv):
549
+ result = subprocess.run(argv, env=env, check=False)
550
+ if result.returncode != 0:
551
+ raise FailClosed(f"command failed with exit {result.returncode}: {' '.join(argv)}")
552
+ print(f"produced {args.out_gguf}")
553
+ return 0
554
+
555
+
556
+ def cmd_manifest(args: argparse.Namespace) -> int:
557
+ weights_dir: Path = args.weights_dir
558
+ files = collect_weight_files(weights_dir)
559
+ if not args.gguf.is_file():
560
+ raise FailClosed(
561
+ f"--gguf {args.gguf} does not exist. Run the 'convert' subcommand first; a manifest is "
562
+ "never emitted for an artifact that was not produced."
563
+ )
564
+ _require_pinned_checkout(args.llama_dir, args.converter_commit)
565
+
566
+ chat_template = weights_dir / args.chat_template
567
+ if not chat_template.is_file():
568
+ raise FailClosed(f"missing chat template: {chat_template}")
569
+ by_path = {entry["path"]: entry for entry in files}
570
+ gguf_sha256, gguf_size = sha256_file(args.gguf)
571
+
572
+ manifest = {
573
+ "schema": SCHEMA_NAME,
574
+ "schema_version": SCHEMA_VERSION,
575
+ "weights": {
576
+ "tier": args.provenance_tier,
577
+ "source_id": args.source_id,
578
+ "source_revision": args.source_revision,
579
+ "upstream_of_record": (
580
+ {"source_id": args.upstream_source_id, "revision": args.upstream_revision}
581
+ if args.upstream_source_id
582
+ else None
583
+ ),
584
+ "files": files,
585
+ },
586
+ "architecture": architecture_from_config(weights_dir / "config.json"),
587
+ "tokenizer": {
588
+ "tokenizer_sha256": by_path["tokenizer.json"]["sha256"],
589
+ "tokenizer_config_sha256": by_path["tokenizer_config.json"]["sha256"],
590
+ "chat_template_sha256": by_path[args.chat_template]["sha256"],
591
+ "tokenizer_vocab_size": tokenizer_vocab_size(weights_dir / "tokenizer.json"),
592
+ },
593
+ "conversion": {
594
+ "converter_repo": args.converter_repo,
595
+ "converter_commit": args.converter_commit,
596
+ "converter_entrypoint": "convert_hf_to_gguf.py",
597
+ "converter_args": ["--outtype", "bf16", "--no-lazy"],
598
+ "quantizer_entrypoint": "llama-quantize",
599
+ "quantizer_args": [],
600
+ "quantization": args.quantization,
601
+ },
602
+ "artifact": {"gguf_sha256": gguf_sha256, "gguf_size": gguf_size},
603
+ }
604
+ if (args.upstream_source_id is None) != (args.upstream_revision is None):
605
+ raise FailClosed("--upstream-source-id and --upstream-revision must be given together")
606
+ validate(manifest)
607
+ digest = manifest_hash(manifest)
608
+ document = dict(manifest)
609
+ document["manifest_hash"] = digest
610
+ document["notes"] = {
611
+ "signature": "UNSIGNED — the ML-DSA-87 signature-domain table is LOCKED; no model-genesis "
612
+ "context is registered, so no signature can exist (ADR-0047 prerequisites).",
613
+ "provenance": "tier is a claim recorded IN the hash; 'third-party-derivative' can never "
614
+ "become the genesis (candidate disqualifier 1).",
615
+ }
616
+ args.out.parent.mkdir(parents=True, exist_ok=True)
617
+ args.out.write_text(json.dumps(document, indent=2, sort_keys=False) + "\n", encoding="utf-8")
618
+ print(f"manifest: {args.out}")
619
+ print(f"manifest_hash: {digest}")
620
+ if args.provenance_tier != "official-upstream":
621
+ print(
622
+ "NOTE: provenance tier is 'third-party-derivative' — this manifest is comparable and "
623
+ "reproducible, but it is NOT a genesis candidate (disqualifier 1)."
624
+ )
625
+ return 0
626
+
627
+
628
+ def _load_manifest(path: Path) -> dict[str, Any]:
629
+ if not path.is_file():
630
+ raise FailClosed(f"manifest {path} does not exist")
631
+ document = json.loads(path.read_text(encoding="utf-8"))
632
+ if document.get("schema") != SCHEMA_NAME:
633
+ raise FailClosed(f"{path}: schema is {document.get('schema')!r}, expected {SCHEMA_NAME!r}")
634
+ recorded = document.pop("manifest_hash", None)
635
+ document.pop("notes", None)
636
+ validate(document)
637
+ computed = manifest_hash(document)
638
+ if recorded is not None and recorded != computed:
639
+ raise FailClosed(f"{path}: recorded manifest_hash {recorded} != recomputed {computed}")
640
+ return document
641
+
642
+
643
+ def cmd_verify(args: argparse.Namespace) -> int:
644
+ if len(args.manifest) < 2:
645
+ raise FailClosed("verify needs at least two --manifest paths (that is the whole point)")
646
+ loaded = [(path, _load_manifest(path)) for path in args.manifest]
647
+ reference_path, reference = loaded[0]
648
+ reference_hash = manifest_hash(reference)
649
+ print(f"{reference_path}: {reference_hash}")
650
+ mismatched = False
651
+ for path, other in loaded[1:]:
652
+ digest = manifest_hash(other)
653
+ print(f"{path}: {digest}")
654
+ if digest != reference_hash:
655
+ mismatched = True
656
+ field = first_mismatch(reference, other) or "<hash differs but no field diff — schema bug>"
657
+ print(f" MISMATCH vs {reference_path}: first differing field = {field}")
658
+ if mismatched:
659
+ raise FailClosed("manifests do not match — ADR-0047 acceptance requires a bit-exact match")
660
+ print(f"MATCH: {len(loaded)} manifests agree bit-exactly")
661
+ print(
662
+ "NOTE: a match is reproduction evidence only. It is not a genesis: the manifest is unsigned "
663
+ "(signature-domain table LOCKED), and two stacks owned by one operator are the weak form of "
664
+ "'two parties' (ADR-0047 §1)."
665
+ )
666
+ return 0
667
+
668
+
669
+ def build_parser() -> argparse.ArgumentParser:
670
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
671
+ sub = parser.add_subparsers(dest="command", required=True)
672
+
673
+ sub.add_parser("selftest", help="re-derive the golden vector and check Rust/Python parity").set_defaults(
674
+ func=cmd_selftest
675
+ )
676
+
677
+ convert = sub.add_parser("convert", help="check preconditions and emit/run the exact conversion")
678
+ convert.add_argument("--weights-dir", required=True, type=Path)
679
+ convert.add_argument("--llama-dir", required=True, type=Path)
680
+ convert.add_argument("--converter-commit", required=True)
681
+ convert.add_argument("--quantizer", type=Path)
682
+ convert.add_argument("--out-gguf", required=True, type=Path)
683
+ convert.add_argument("--quantization", default="Q4_K_M")
684
+ convert.add_argument("--model-name", required=True, help="pinned GGUF general.name; must match between operators")
685
+ convert.add_argument("--execute", action="store_true", help="actually run (default: dry run)")
686
+ convert.set_defaults(func=cmd_convert)
687
+
688
+ manifest = sub.add_parser("manifest", help="hash real inputs and emit the manifest + its hash")
689
+ manifest.add_argument("--weights-dir", required=True, type=Path)
690
+ manifest.add_argument("--gguf", required=True, type=Path)
691
+ manifest.add_argument("--llama-dir", required=True, type=Path)
692
+ manifest.add_argument("--converter-commit", required=True)
693
+ manifest.add_argument("--converter-repo", default="https://github.com/ggml-org/llama.cpp.git")
694
+ manifest.add_argument("--provenance-tier", required=True, choices=sorted(PROVENANCE_TIER_TAGS))
695
+ manifest.add_argument("--source-id", required=True)
696
+ manifest.add_argument("--source-revision", required=True)
697
+ manifest.add_argument("--upstream-source-id")
698
+ manifest.add_argument("--upstream-revision")
699
+ manifest.add_argument("--chat-template", default="chat_template.jinja")
700
+ manifest.add_argument("--quantization", default="Q4_K_M")
701
+ manifest.add_argument("--out", required=True, type=Path)
702
+ manifest.set_defaults(func=cmd_manifest)
703
+
704
+ verify = sub.add_parser("verify", help="compare two or more manifests field by field")
705
+ verify.add_argument("--manifest", action="append", required=True, type=Path, default=[])
706
+ verify.set_defaults(func=cmd_verify)
707
+ return parser
708
+
709
+
710
+ def main() -> int:
711
+ args = build_parser().parse_args()
712
+ return int(args.func(args))
713
+
714
+
715
+ if __name__ == "__main__":
716
+ sys.exit(main())
scripts/tests/test_model_genesis_manifest.py ADDED
@@ -0,0 +1,327 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Tests for the ADR-0047 model-genesis manifest tool.
3
+
4
+ Two properties matter here and nothing else does:
5
+
6
+ * **Parity** — the Python encoder and `runtime-palw/src/model_genesis.rs` must agree on the
7
+ golden vector. If they drift, a cross-operator hash mismatch becomes unattributable (is it the
8
+ weights, or the tool?), which defeats the purpose of the ADR's acceptance criterion.
9
+ * **Fail-closed** — a manifest must never be emitted from inputs that were not there. The repo's
10
+ current state exercises this for real: the checkpoint directory holds metadata only and
11
+ `llama-quantize` is not built.
12
+ """
13
+
14
+ import json
15
+ import subprocess
16
+ import sys
17
+ import tempfile
18
+ import unittest
19
+ from pathlib import Path
20
+
21
+ REPO = Path(__file__).resolve().parents[2]
22
+ SCRIPTS = REPO / "scripts"
23
+ TOOL = SCRIPTS / "model_genesis_manifest.py"
24
+ RUST_MODULE = REPO / "runtime-palw" / "src" / "model_genesis.rs"
25
+
26
+ sys.path.insert(0, str(SCRIPTS))
27
+
28
+ import model_genesis_manifest as mg # noqa: E402
29
+
30
+
31
+ def run_tool(*args: str) -> subprocess.CompletedProcess:
32
+ return subprocess.run(
33
+ [sys.executable, str(TOOL), *args], capture_output=True, text=True, check=False
34
+ )
35
+
36
+
37
+ def write_checkpoint(directory: Path, *, with_weights: bool = True) -> None:
38
+ directory.mkdir(parents=True, exist_ok=True)
39
+ (directory / "config.json").write_text(
40
+ json.dumps(
41
+ {
42
+ "architectures": ["Qwen3ForCausalLM"],
43
+ "model_type": "qwen3",
44
+ "hidden_size": 4096,
45
+ "num_hidden_layers": 36,
46
+ "num_attention_heads": 32,
47
+ "num_key_value_heads": 8,
48
+ "intermediate_size": 12288,
49
+ "head_dim": 128,
50
+ "rope_theta": 1000000,
51
+ "vocab_size": 151936,
52
+ "max_position_embeddings": 40960,
53
+ "torch_dtype": "bfloat16",
54
+ }
55
+ ),
56
+ encoding="utf-8",
57
+ )
58
+ (directory / "tokenizer.json").write_text(
59
+ json.dumps({"model": {"vocab": {"a": 0, "b": 1}}, "added_tokens": [{"id": 7}]}),
60
+ encoding="utf-8",
61
+ )
62
+ (directory / "tokenizer_config.json").write_text("{}", encoding="utf-8")
63
+ (directory / "chat_template.jinja").write_text("{{ messages }}", encoding="utf-8")
64
+ if with_weights:
65
+ (directory / "model-00001-of-00001.safetensors").write_bytes(b"synthetic-weights")
66
+
67
+
68
+ def make_pinned_checkout(directory: Path) -> str:
69
+ directory.mkdir(parents=True, exist_ok=True)
70
+ (directory / "convert_hf_to_gguf.py").write_text("# stub\n", encoding="utf-8")
71
+ subprocess.run(["git", "init", "-q", str(directory)], check=True)
72
+ subprocess.run(["git", "-C", str(directory), "add", "-A"], check=True)
73
+ subprocess.run(
74
+ [
75
+ "git",
76
+ "-c",
77
+ "user.email=test@example.com",
78
+ "-c",
79
+ "user.name=test",
80
+ "-C",
81
+ str(directory),
82
+ "commit",
83
+ "-q",
84
+ "-m",
85
+ "stub",
86
+ ],
87
+ check=True,
88
+ )
89
+ return subprocess.run(
90
+ ["git", "-C", str(directory), "rev-parse", "HEAD"],
91
+ capture_output=True,
92
+ text=True,
93
+ check=True,
94
+ ).stdout.strip()
95
+
96
+
97
+ class GoldenVectorParity(unittest.TestCase):
98
+ def test_selftest_passes(self):
99
+ result = run_tool("selftest")
100
+ self.assertEqual(result.returncode, 0, result.stderr)
101
+ self.assertIn(mg.GOLDEN_MANIFEST_HASH_HEX, result.stdout)
102
+
103
+ def test_rust_module_pins_the_same_golden(self):
104
+ """The parity binding: the Rust golden constant contains the same 128 hex characters."""
105
+ source = RUST_MODULE.read_text(encoding="utf-8")
106
+ halves = [mg.GOLDEN_MANIFEST_HASH_HEX[:64], mg.GOLDEN_MANIFEST_HASH_HEX[64:]]
107
+ self.assertTrue(
108
+ mg.GOLDEN_MANIFEST_HASH_HEX in source or all(half in source for half in halves),
109
+ "runtime-palw/src/model_genesis.rs no longer pins this golden vector",
110
+ )
111
+
112
+ def test_domain_separation(self):
113
+ import hashlib
114
+
115
+ payload = mg.canonical_bytes(mg.golden_manifest())
116
+ other = hashlib.blake2b(payload, key=b"palw-k1/file", digest_size=64).hexdigest()
117
+ self.assertNotEqual(other, mg.GOLDEN_MANIFEST_HASH_HEX)
118
+
119
+
120
+ class Validation(unittest.TestCase):
121
+ def test_backend_specific_conversion_rejected(self):
122
+ for bad in ("--n-gpu-layers", "-ngl", "metal", "CUDA", "--threads"):
123
+ manifest = mg.golden_manifest()
124
+ manifest["conversion"]["converter_args"] = [bad]
125
+ with self.assertRaises(SystemExit, msg=bad):
126
+ mg.validate(manifest)
127
+
128
+ def test_host_paths_and_mutable_pins_rejected(self):
129
+ absolute = mg.golden_manifest()
130
+ absolute["weights"]["files"][0]["path"] = "/Users/operator/config.json"
131
+ with self.assertRaises(SystemExit):
132
+ mg.validate(absolute)
133
+
134
+ branch = mg.golden_manifest()
135
+ branch["weights"]["source_revision"] = "main"
136
+ with self.assertRaises(SystemExit):
137
+ mg.validate(branch)
138
+
139
+ unsorted = mg.golden_manifest()
140
+ files = unsorted["weights"]["files"]
141
+ files[0], files[1] = files[1], files[0]
142
+ with self.assertRaises(SystemExit):
143
+ mg.validate(unsorted)
144
+
145
+ def test_field_sensitivity(self):
146
+ base = mg.golden_manifest()
147
+ baseline = mg.manifest_hash(base)
148
+ seen = {baseline}
149
+ for path in (
150
+ ("weights", "source_id"),
151
+ ("architecture", "layers"),
152
+ ("tokenizer", "tokenizer_vocab_size"),
153
+ ("conversion", "quantization"),
154
+ ("artifact", "gguf_size"),
155
+ ):
156
+ mutated = mg.golden_manifest()
157
+ section, field = path
158
+ value = mutated[section][field]
159
+ mutated[section][field] = value + 1 if isinstance(value, int) else value + "x"
160
+ digest = mg.manifest_hash(mutated)
161
+ self.assertNotIn(digest, seen, f"{section}.{field} did not change the hash")
162
+ seen.add(digest)
163
+ self.assertIsNotNone(mg.first_mismatch(base, mutated))
164
+
165
+
166
+ class FailClosed(unittest.TestCase):
167
+ def test_metadata_only_checkpoint_is_refused(self):
168
+ """The repo's real state: base metadata is present, the weight shards are not."""
169
+ with tempfile.TemporaryDirectory() as tmp:
170
+ weights = Path(tmp) / "weights"
171
+ write_checkpoint(weights, with_weights=False)
172
+ llama = Path(tmp) / "llama"
173
+ commit = make_pinned_checkout(llama)
174
+ result = run_tool(
175
+ "manifest",
176
+ "--weights-dir", str(weights),
177
+ "--gguf", str(Path(tmp) / "absent.gguf"),
178
+ "--llama-dir", str(llama),
179
+ "--converter-commit", commit,
180
+ "--provenance-tier", "official-upstream",
181
+ "--source-id", "Example/Model",
182
+ "--source-revision", "0" * 40,
183
+ "--out", str(Path(tmp) / "manifest.json"),
184
+ )
185
+ self.assertEqual(result.returncode, 1)
186
+ self.assertIn("no *.safetensors", result.stderr)
187
+ self.assertFalse((Path(tmp) / "manifest.json").exists())
188
+
189
+ def test_absent_gguf_is_refused(self):
190
+ with tempfile.TemporaryDirectory() as tmp:
191
+ weights = Path(tmp) / "weights"
192
+ write_checkpoint(weights)
193
+ llama = Path(tmp) / "llama"
194
+ commit = make_pinned_checkout(llama)
195
+ result = run_tool(
196
+ "manifest",
197
+ "--weights-dir", str(weights),
198
+ "--gguf", str(Path(tmp) / "absent.gguf"),
199
+ "--llama-dir", str(llama),
200
+ "--converter-commit", commit,
201
+ "--provenance-tier", "official-upstream",
202
+ "--source-id", "Example/Model",
203
+ "--source-revision", "0" * 40,
204
+ "--out", str(Path(tmp) / "manifest.json"),
205
+ )
206
+ self.assertEqual(result.returncode, 1)
207
+ self.assertIn("does not exist", result.stderr)
208
+ self.assertFalse((Path(tmp) / "manifest.json").exists())
209
+
210
+ def test_unpinned_checkout_is_refused(self):
211
+ with tempfile.TemporaryDirectory() as tmp:
212
+ weights = Path(tmp) / "weights"
213
+ write_checkpoint(weights)
214
+ llama = Path(tmp) / "llama"
215
+ make_pinned_checkout(llama)
216
+ gguf = Path(tmp) / "model.gguf"
217
+ gguf.write_bytes(b"gguf")
218
+ result = run_tool(
219
+ "manifest",
220
+ "--weights-dir", str(weights),
221
+ "--gguf", str(gguf),
222
+ "--llama-dir", str(llama),
223
+ "--converter-commit", "1" * 40,
224
+ "--provenance-tier", "official-upstream",
225
+ "--source-id", "Example/Model",
226
+ "--source-revision", "0" * 40,
227
+ "--out", str(Path(tmp) / "manifest.json"),
228
+ )
229
+ self.assertEqual(result.returncode, 1)
230
+ self.assertIn("not the pinned converter commit", result.stderr)
231
+
232
+ def test_missing_quantizer_is_named(self):
233
+ with tempfile.TemporaryDirectory() as tmp:
234
+ weights = Path(tmp) / "weights"
235
+ write_checkpoint(weights)
236
+ llama = Path(tmp) / "llama"
237
+ commit = make_pinned_checkout(llama)
238
+ result = run_tool(
239
+ "convert",
240
+ "--weights-dir", str(weights),
241
+ "--llama-dir", str(llama),
242
+ "--converter-commit", commit,
243
+ "--out-gguf", str(Path(tmp) / "out.gguf"),
244
+ "--model-name", "Example",
245
+ )
246
+ self.assertEqual(result.returncode, 1)
247
+ self.assertIn("missing quantizer binary", result.stderr)
248
+ self.assertIn("llama-quantize", result.stderr)
249
+
250
+
251
+ class EndToEnd(unittest.TestCase):
252
+ def _emit(self, tmp: Path, name: str, *, tier: str = "official-upstream") -> Path:
253
+ weights = tmp / "weights"
254
+ write_checkpoint(weights)
255
+ llama = tmp / "llama"
256
+ commit = make_pinned_checkout(llama) if not (llama / ".git").exists() else subprocess.run(
257
+ ["git", "-C", str(llama), "rev-parse", "HEAD"], capture_output=True, text=True, check=True
258
+ ).stdout.strip()
259
+ gguf = tmp / "model.gguf"
260
+ gguf.write_bytes(b"synthetic-gguf-bytes")
261
+ out = tmp / name
262
+ result = run_tool(
263
+ "manifest",
264
+ "--weights-dir", str(weights),
265
+ "--gguf", str(gguf),
266
+ "--llama-dir", str(llama),
267
+ "--converter-commit", commit,
268
+ "--provenance-tier", tier,
269
+ "--source-id", "Example/Model",
270
+ "--source-revision", "0" * 40,
271
+ "--out", str(out),
272
+ )
273
+ self.assertEqual(result.returncode, 0, result.stderr)
274
+ return out
275
+
276
+ def test_two_identical_runs_match_and_a_changed_input_does_not(self):
277
+ with tempfile.TemporaryDirectory() as raw:
278
+ tmp = Path(raw)
279
+ first = self._emit(tmp, "a.json")
280
+ second = self._emit(tmp, "b.json")
281
+ self.assertEqual(
282
+ json.loads(first.read_text())["manifest_hash"],
283
+ json.loads(second.read_text())["manifest_hash"],
284
+ )
285
+ match = run_tool("verify", "--manifest", str(first), "--manifest", str(second))
286
+ self.assertEqual(match.returncode, 0, match.stderr)
287
+ self.assertIn("MATCH", match.stdout)
288
+ # A match is reproduction evidence, never a genesis claim.
289
+ self.assertIn("unsigned", match.stdout)
290
+
291
+ divergent = json.loads(second.read_text())
292
+ divergent.pop("manifest_hash")
293
+ divergent["weights"]["files"][0]["sha256"] = "ab" * 32
294
+ second.write_text(json.dumps(divergent), encoding="utf-8")
295
+ mismatch = run_tool("verify", "--manifest", str(first), "--manifest", str(second))
296
+ self.assertEqual(mismatch.returncode, 1)
297
+ self.assertIn("first differing field = weights.files", mismatch.stdout)
298
+
299
+ def test_derivative_tier_is_flagged_in_output(self):
300
+ with tempfile.TemporaryDirectory() as raw:
301
+ tmp = Path(raw)
302
+ weights = tmp / "weights"
303
+ write_checkpoint(weights)
304
+ llama = tmp / "llama"
305
+ commit = make_pinned_checkout(llama)
306
+ gguf = tmp / "model.gguf"
307
+ gguf.write_bytes(b"synthetic-gguf-bytes")
308
+ result = run_tool(
309
+ "manifest",
310
+ "--weights-dir", str(weights),
311
+ "--gguf", str(gguf),
312
+ "--llama-dir", str(llama),
313
+ "--converter-commit", commit,
314
+ "--provenance-tier", "third-party-derivative",
315
+ "--source-id", "huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated",
316
+ "--source-revision", "ac18882735d037f6074a7630eb68d85db8234c25",
317
+ "--out", str(tmp / "m.json"),
318
+ )
319
+ self.assertEqual(result.returncode, 0, result.stderr)
320
+ self.assertIn("NOT a genesis candidate", result.stdout)
321
+ document = json.loads((tmp / "m.json").read_text())
322
+ self.assertEqual(document["weights"]["tier"], "third-party-derivative")
323
+ self.assertIn("UNSIGNED", document["notes"]["signature"])
324
+
325
+
326
+ if __name__ == "__main__":
327
+ unittest.main()