minicpm5-1b-arcsum (sft-dropv6)

A MiniCPM5-1B fine-tune that reads a zh-TW meeting transcript in ~2,500-token chunks and curates a small external memory, then writes a single flowing zh-TW prose summary.

Read this first: this checkpoint did NOT pass its own ship gates. It clears 6 of 7 — the revision probe (G1) fails. The project's spec is all-or-nothing, so the recorded decision is "ship the map-reduce baseline", and this model is published as a measured result with its failure attached, not as a winner.

A later checkpoint (sft-dropv7) does pass all seven, but only because the failing capability was deliberately trained with synthetic data; on an independent probe of scenarios it was not trained on it scored 2/10 against this model's 3/10. That gate pass measures the training set, not the capability, so the honest artifact to publish is this one. Nothing here is rounded in the model's favour.

What it does

The transcript is read chunk by chunk. The harness owns a two-slot external memory — ARC (a rolling one-sentence throughline, ≤80 tokens) and POINTS (≤16 short facts) — and re-renders it into every step's prompt. No conversation history crosses steps. The model emits only edit lines:

ARC: <replacement throughline>
ADD - <new point>
DROP «<prefix of an existing point>»
NOP

The harness applies them deterministically, including cap-overflow handling. A final SYNTHESIZE call turns the finished memory into prose.

The point of the external memory is the thing aggregate scores cannot show: a later chunk can overturn an earlier conclusion. Map-reduce structurally cannot do this — each window is summarised independently, so a decision reversed at minute 90 never reaches the summary of minute 10. That capability is what the G1 gate tests directly, and it passes.

Measured results (n = 40 held-out meetings, paired)

Measured on 40 MeetingBank-derived zh-TW meetings that appear in no training pool and in no earlier measurement, built through the full corpus pipeline specifically to retire an earlier eval set that had been read six times. Baseline is a map-reduce summariser using the same model, same chunk size — the fair opponent the spec requires.

gate	result
G1 revision probe	FAIL	states the reversal but drops the subject term; see Limitations
G2 faithfulness	PASS	18 inversions vs baseline's 109, 40/40 paired, 0 judge failures
G3 ROUGE-1	PASS	+0.077 (lower bound +0.053), p=0.006, wins 29/40
G3 ROUGE-2	PASS	+0.043 (lower bound +0.031), p=0.001, wins 31/40
G3 ROUGE-L	PASS	+0.053 (lower bound +0.045), p=0.000, wins 33/40
G4 on-device budget	PASS	19.58 min vs a 20 min ceiling — projected, never measured

On long meetings (≥400 lines) the margin is far larger: 9/1 wins, +0.202 ROUGE-1. The summaries are also ~8x shorter than the baseline's (320 vs 2,492 characters).

Three numbers that need their caveats

1. G4 was never measured on a phone. 19.58 min against a 20.00 ceiling is a 2.1% margin extrapolated from short bursts; sustained thermal throttling is unmodelled and pushes the wrong way. It is the only gate here with no measurement behind it.
2. G2's per-claim rate favours the baseline (7.3% vs 4.9%). The agent wins decisively on absolute inversions partly because it asserts far fewer claims (246 vs 2,236) — it says less, so it is wrong less often in absolute terms. The gate is the absolute count; both readings are given here.
3. Q4_K_M was withdrawn from this repo. Measured against Q8_0 on the same 40 meetings, its margin over the baseline more than halved on ROUGE-1 (+0.077 -> +0.034, wins 29/40 -> 22/40) and its summaries were ~30% shorter. Serving it behind these numbers would misrepresent the model, so only the evaluated artifact ships.

Diagnostics (deliberately not gates)

	agent	baseline
coverage	0.982	0.993
density	3.26	4.05

Both measure extractiveness — how much text is copied verbatim. The agent is less extractive, which is the design goal (flowing abstractive prose), so a lower number here is expected rather than bad. They were briefly gated by mistake, which made shipping unreachable by construction: passing would have required the agent to copy more than map-reduce. Read them as shape descriptors, not scores.

Mean summary length: agent 317 chars, baseline 691, human reference 471. The agent wins ROUGE while producing summaries under half the baseline's length.

Files

file	use
MiniCPM5-1B.Q8_0.gguf	the only artifact. Every number above was measured on this.

Q4_K_M was previously published and has been removed: measured on the same 40 held-out meetings it retains under half the ROUGE-1 margin (see caveat 3). Quantise from the Q8 if you need something smaller, and re-measure before trusting it.

Usage

The edit-line protocol is not self-describing; the model expects the harness's exact prompts and a memory rendered fresh each step. Use the harness rather than prompting it directly:

llama-server -m MiniCPM5-1B.Q8_0.gguf --ctx-size 4096 --jinja --temp 0

Two serving details, both measured the hard way:

• Pin cache_prompt: false for anything you intend to report. llama.cpp's prompt cache changes generation: same model, same seed, same prompt returned 700 characters with the cache on and 167 with it off, each internally deterministic across three repeats. Results are reproducible given a cache state, not across runs.
• Use a repetition penalty (~1.1) on the prose call only, never the reading steps. Greedy decoding degenerates: one synthesis emitted the same sentence eight times. Reading steps emit a fixed op vocabulary, so a penalty there punishes the literal ADD/DROP/ARC tokens the format requires.

Training

Full fine-tune (not LoRA) of openbmb/MiniCPM5-1B, 3 epochs, lr 5e-5, on ~4.3k steps derived from 200 MeetingBank meetings machine-translated to zh-TW, with per-step targets produced by a Qwen3.8-27B teacher and replayed through the real harness before use.

The pool balance is load-bearing and easy to get wrong. An earlier checkpoint pushed the NOP share to 25.7% against the teacher's natural 38.2% — two rebalancing knobs compounding — and the model stopped emitting NOP at all, instead churning: DROP followed by a near-identical re-ADD, burning up to 45 of one 53-step meeting's steps on a single topic. A later attempt to fix the long-meeting weakness by oversampling late steps also regressed it (ROUGE-1 14/20 → 12/20), even though it demonstrably improved the meeting it targeted. Stable label shares did not imply stable behaviour.

Limitations

• zh-TW only. English is source material, never a product language.
• G1 (revision) fails. Given a decision reversed later in the meeting, the summary states the reversal but drops the identifying detail — in the probe case it says the relocation was withdrawn without naming the building, which appears 19 times in the source. The cause is a step-0 salience effect: on the FIRST chunk the model records the tail and skips the head (measured trigram containment 0.134 head vs 0.413 tail; on later chunks it is head-favoured, 0.174 vs 0.102). Four fixes were tried and refuted — smaller chunks, an explicit coverage instruction, reordering the prompt, and labelling the empty memory. Do not assume any of them is the answer.
• Reversals are barely in the training distribution. MeetingBank contains almost no within-meeting decision reversals: 3.4% of its gold items match reversal language, and those are legislative boilerplate ("repealing Section 5.53.090") that reverses external ordinances. The model's DROP rate already matches its supervision; what it has never seen is "a point in memory was just contradicted".
• Long meetings were the previous open weakness and are now the strongest case — 9/1 wins, +0.202 ROUGE-1 at ≥400 lines. Fixing it took genuinely new long-meeting supervision plus a position signal in the prompt; reweighting the existing corpus was tried and rejected.
• Not a general chat model. It emits edit lines, and outside the harness's prompts its behaviour is undefined.
• Reference summaries are teacher-synthesised from machine-translated English meetings, so no score here is comparable to published MeetingBank numbers.