Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Gjusev 
posted an update 3 days ago
Post
1,174
I retracted my own launch claim in public commits, then shipped what
survived.

MK-SH-1 is a 140M decision model I built on an open encoder. It returns
calibrated probabilities, a choice, and an explicit abstention below a
frozen gate (tau = 0.79, written down before the curve existed).

ECE 0.016 · 90% coverage in-sample; held-out numbers in the card · 106 ms on CPU.

Gjusev/MK-SH-1

Demo Gjusev/MK-SH-1-demo
Full record, retraction included https://github.com/Gjusev/MK-SH-1
BARIK12

nice

dipankarsarkar

The retraction is the part of this I trust most. One thing in E36 I would pull apart.

tau = 0.79 and the numbers quoted next to it come from the same 480 rows. e36_report.py fits T on split, sweeps the grid on split, takes the first tau with risk <= 0.05, and reports coverage and risk at that tau, also on split. The frozen evals only get NLL and ECE.

So 4.86% is 21 errors in 432 kept, measured on the rows that picked the gate. You froze the rule in advance, which is the right move. The value still comes from those rows, and the first grid point under 5% is the one most likely to sit there by luck. A Clopper-Pearson 95% upper bound on 21/432 is 7.3%.

Your descriptive table points the same way: ECE 0.016 on the split, 0.076 on the fresh OOD set (n=202), 0.042 on known thresholds.

The analyzer already loads per-case logits for known, new, mono, e and task. So coverage and selective risk at tau = 0.79 on each set is one more loop over data you have.

Does the 5% budget hold on the fresh OOD extremes, where accuracy is 0.875?

·
Gjusev

Checked it, and you're right about how the script works. Here's the loop you described, on the frozen logits (T = 7.653, tau = 0.79, Clopper-Pearson 95% upper bound in brackets):

  • split (n=480): coverage 0.900, risk 21/432 = 4.86% [7.3%]
  • known (n=48): 0.979, 1/47 = 2.1% [11.3%]
  • new (n=54): 0.944, 1/51 = 2.0% [10.5%]
  • mono (n=60): 1.000, 2/60 = 3.3% [11.5%]
  • e, fresh OOD extremes (n=202): 0.960, 16/194 = 8.2% [13.1%]
  • task (n=300): 0.683, 15/205 = 7.3% [11.8%]

So no, the 5% budget doesn't hold on the OOD extremes. The gate keeps 96% of those rows and 8.2% of the kept ones are wrong. The ECE gap you mentioned was already showing this. Small correction: 0.875 is the E-extrap pair score; row accuracy on that set is 0.906.

The 4.86% is an in-sample number. Freezing the rule in advance doesn't fix that, since the value still comes from the 480 rows that chose tau. I've added the per-set check as scripts/e36_heldout_policy.py (it reads the shipped policy and changes nothing). The model card now has the per-set table with intervals and says the 5% budget is only validated in-sample on in-range numeric questions:

https://huggingface.co/Gjusev/MK-SH-1
https://github.com/Gjusev/MK-SH-1/commit/9e1a226e

Thanks for reading the code 🫥 . This is the kind of check I should have run before quoting the number.
savaaax

interesting

dipankarsarkar

That table is better than anything in my comment. And thanks for the 0.906 correction, I quoted the pair score as row accuracy.

The row I'd sit with is e. The gate keeps 96 percent there, more than the 90 it keeps on the split, and 8.2 percent of the kept rows are wrong. Under shift it keeps more and is wrong more often. Confidence is not seeing the shift.

So tightening tau may buy little back, if the wrong e rows are confident ones. That is one more loop over the logits you have.

Task goes the other way: 68 percent kept, 7.3 percent risk. Two kinds of failure, one threshold.

Is there anything in the 16 wrong e rows the logits don't carry, like distance outside the training range?
In this post