Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -70,9 +70,36 @@ Base and tuned generated in one process under identical greedy decoding, scored
|
|
| 70 |
|
| 71 |
| slice | rows | what it isolates | base | **CashSage-4B** |
|
| 72 |
|---|---|---|---|---|
|
| 73 |
-
| held-out |
|
| 74 |
-
| hard | 180 | worst cases across all seven families | 1.
|
| 75 |
-
| **2026 rules** | 300 | **statutory limits the corpus never saw** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
The 2026 slice is the one to read carefully. Contribution limits, bend points and brackets change
|
| 78 |
every year, so it separates a model that **applies the rule it was given** from one that recites a
|
|
|
|
| 70 |
|
| 71 |
| slice | rows | what it isolates | base | **CashSage-4B** |
|
| 72 |
|---|---|---|---|---|
|
| 73 |
+
| held-out | 600 | own distribution, the competition's yardstick | 1.3% | **100.0%** |
|
| 74 |
+
| hard | 180 | worst cases across all seven families | 1.1% | **85.0%** |
|
| 75 |
+
| **2026 rules** | 300 | **statutory limits the corpus never saw** | 1.7% | **100.0%** |
|
| 76 |
+
|
| 77 |
+
The held-out slice as released contains 1,500 rows and the evaluation was run on the first
|
| 78 |
+
600 of them. Every family sits within 1.5 points of its share in the full file, so the
|
| 79 |
+
sample is representative rather than convenient, and the row count above is the number
|
| 80 |
+
actually scored rather than the number available.
|
| 81 |
+
|
| 82 |
+
### Version history, stated plainly
|
| 83 |
+
|
| 84 |
+
The clarification a model should ask for depends on which ambiguity the scenario states,
|
| 85 |
+
and the two undecidable families state different ones: `card_payoff` quotes a rate without
|
| 86 |
+
saying whether it is monthly or annual, `savings_timeline` quotes one not marked APR or
|
| 87 |
+
APY. The generator was refined to emit the question **specific to each family**, and **v2
|
| 88 |
+
is a continuation from the v1 adapter** on that corrected shard plus a seeded replay of the
|
| 89 |
+
original corpus, not a fresh run.
|
| 90 |
+
|
| 91 |
+
Both sets of numbers are published so the effect of the continuation is visible rather than
|
| 92 |
+
absorbed:
|
| 93 |
+
|
| 94 |
+
| slice | v1 | **v2, ships** |
|
| 95 |
+
|---|---|---|
|
| 96 |
+
| held-out | 99.8% | **100.0%** |
|
| 97 |
+
| hard | 83.9% | **85.0%** |
|
| 98 |
+
| 2026 rules | 100.0% | **100.0%** |
|
| 99 |
+
|
| 100 |
+
The condition for shipping v2 was written down before it was measured: hold or improve on
|
| 101 |
+
every slice v1 had been measured on. It does. The released held-out slice carries the
|
| 102 |
+
family-specific questions, so the released corpus and the released generator agree.
|
| 103 |
|
| 104 |
The 2026 slice is the one to read carefully. Contribution limits, bend points and brackets change
|
| 105 |
every year, so it separates a model that **applies the rule it was given** from one that recites a
|