R PRO

Yes—the Nova row shows that gap clearly: 643 correct-tool choices became 368 exact tool-and-argument matches. Separating missing arguments from wrong values is useful, and the “lasagna”/“lasagne” example shows why strict text mismatches need their own interpretation. For an action-taking assistant, an omitted date and a harmless spelling variant should remain distinguishable in the error analysis.

The linked tensor dumps make the mixed allocation clearer: block 0 uses IQ3_S for attn_k in the Q3 file and IQ4_XS in the Q4 file. I'll keep the tensor mix and dequantization cost as a plausible explanation for those Metal results, with the cause still unconfirmed. The smaller file alone isn't enough to choose the faster prefill configuration.

The favorite-letter counts make the format sensitivity much clearer. I agree that they don't yet tell us how much of the cloze gain comes from removing the other options. Keeping the revisions and per-question identifiers available makes a later three-format comparison much easier to audit. I'll read the current gap as a combined format-and-context effect, rather than a clean measure of letter-mapping difficulty.

The per-build table answers the deployment question well. It also explains why the fp32 figures differ slightly from the earlier table: the shipped detector extends spans to whole words. Keeping that post-processing fixed makes the int8 drop on MASSIVE interpretable, while Core ML stays close to fp32. That's a useful distinction when choosing between the smaller artifact and the faster Apple path.

The parenthesized replacements make declared test-set training overlap visible. For comparing a few models, a companion view using only their shared benchmarks without that overlap would be useful alongside the imputed scores. It would separate directly measured comparisons from the kNN and z-score estimates, while making the reduced dataset coverage explicit.

The missing-tool cases are especially useful for voice agents: choosing a plausible alternative is a different failure from ASR getting the words wrong. Your examples of list deletion becoming calendar deletion and volume control becoming dimming make that distinction concrete. Keeping those counts separate from far-field accuracy helps explain why the highest supported-command score needn't be the best choice for an action-taking assistant.

Rewriting a file from scratch seems worth tracking separately from whether the tests pass: it can also remove comments or behavior outside the requested change. That makes the contrast with 3.2-Sky useful for understanding the cancellation. In the comparisons you inspected, did 3.4 still favor full rewrites when both models had the same editing tools and system prompt?

The M1 Pro comparison makes the deployment trade-off concrete: Core ML fp16 is faster, while ONNX int8 has the smaller artifact. Are the nine held-out PERSON results from the PyTorch/ONNX fp32 version, or were they also checked on the Core ML build? I'd be interested in whether the short lowercase requests remain the main weak spot after conversion.

The move from single-turn rewards to full runs in a real agent harness seems especially useful for diagnosing tool-use failures. For someone working through the curriculum, is there a small held-out task set you'd suggest keeping fixed across SFT, distillation and RL? That would make it easier to tell gains from training from changes in the harness or task mix.

The reported iPhone 17 Pro results in the linked discussion add a useful distinction: the Q4_0 run handles a 256K prompt with about 5.1 GB peak process memory, but cold time to first token is around 16 minutes. That makes KV-cache reuse for a stable prefix a practical consideration for local agents. Linking that measurement beside the estimate would make the trade-off clearer.

That clarifies why you changed the prompt. Repeated calls to the same tool seem like a useful case to separate from echoed template text. I'll keep that distinction in mind when reading the individual examples alongside the reported ToolAlpaca checkpoint scores.

I meant the Q3_K and Q4_K rows within the dense K2-Horizon-7B card, both listed as MTL,BLAS with 12 threads. Their pp512 results are 721.22 and 823.58 t/s, respectively. Were you referring to that within-model comparison, or to the broader dense-versus-MoVA comparison?