CashSage-4B: A Money Model That Asks Before It Assumes

CashSage-4B is a LoRA adaptation of google/gemma-3-4b-it for personal finance computation: card payoff schedules, multi-debt strategies, mortgage prepayment, Social Security benefit estimates, 401k match and vesting, and long-horizon fund cost.

Built for the Adaption AutoScientist Challenge, Part 2 (Personal Finance); the training corpus was co-optimized with Adaptive Data (Adaption Labs).

Read this first: what kind of number follows

Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy, and the scorer earns it the hard way: it executes the model's program and checks the result against a reference simulator validated on the official SSA and Regulation Z worked examples.

The two are not interchangeable, and this project has the receipt. A sibling entry in this portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported a 95.48% win rate. Our scorer, run on that same adapter, measured 62.0% exact-match accuracy. A win rate is a preference; an accuracy is a fact.

what is measured	the model's computation, re-executed and checked to the cent
a language model judged something	never, not the labels, not the scoring
held-out rows scored	1,080, across three independent slices
rows the base model wins	1 of 1,080
significance	p = 1.2e-178, exact McNemar, held-out
unseen-year test	300 rows on 2026 statutory limits the corpus never contained
cost to recompute every number	one laptop, no GPU, stdlib Python only

TL;DR

At Day-0 the base model scored 0.0% strict on six of seven families. The single family it could do, savings_timeline, is the only one with a closed-form solution: substitute into a formula and you are done. Every other family requires iterating period by period, and the base could not do any of them.

After training, held-out accuracy is 100.0%.

The base model's exact profile, measured before training

family	closed form?	base, strict
savings_timeline	yes	32.1%
card_payoff	no	0.0%
multi_debt	no	0.0%
amortization_extra	no	0.0%
ssa_pia	no	0.0%
retirement_401k	no	0.0%
fund_cost_n1a	no	0.0%

This measurement shaped the corpus rather than decorating it. Mass was shifted away from the one family the base could already do and toward the six it could not, leaving savings_timeline at an 8% floor and no-closed-form families at 51% of the corpus. Building the dataset before measuring the base would have spent a fifth of the budget teaching a skill already present.

Why this base, and how to read a large delta

google/gemma-3-4b-it is the base used by every entry in this portfolio and the size class this challenge targets, fixed before any Part 2 result existed. A near-zero base invites the reading that the delta was farmed from a conveniently weak model, so two things are stated plainly. First, the base is not uniformly weak and the per-family table above says exactly where it is not: 32.1% on the one closed-form family. Its zeros sit precisely where period-by-period iteration is required, which is a property of the task, not a curated embarrassment. Second, the claim on this card is the absolute number, 100.0% strict on computations the scorer re-executes, and the delta is a consequence of the starting point rather than the target of the build. The sibling Science entry ablated this same recipe across two unrelated 4B bases and both landed within a point of each other; the recipe's gains come from the corpus, and that corpus-first recipe is what was applied here.

Headline results

Base and tuned generated in one process under identical greedy decoding, scored by the released score_cashsage.py, which re-executes every computation.

slice	rows	what it isolates	base	CashSage-4B
held-out	600	own distribution, the competition's yardstick	1.3%	100.0%
hard	180	worst cases across all seven families	1.1%	85.0%
2026 rules	300	statutory limits the corpus never saw	1.7%	100.0%

The held-out slice as released contains 1,500 rows and the evaluation was run on the first 600 of them. Every family sits within 1.5 points of its share in the full file, so the sample is representative rather than convenient, and the row count above is the number actually scored rather than the number available.

Version history, stated plainly

The clarification a model should ask for depends on which ambiguity the scenario states, and the two undecidable families state different ones: card_payoff quotes a rate without saying whether it is monthly or annual, savings_timeline quotes one not marked APR or APY. The generator was refined to emit the question specific to each family, and v2 is a continuation from the v1 adapter on that corrected shard plus a seeded replay of the original corpus, not a fresh run.

Both sets of numbers are published so the effect of the continuation is visible rather than absorbed:

slice	v1	v2, ships
held-out	99.8%	100.0%
hard	83.9%	85.0%
2026 rules	100.0%	100.0%

The condition for shipping v2 was written down before it was measured: hold or improve on every slice v1 had been measured on. It does. The released held-out slice carries the family-specific questions, so the released corpus and the released generator agree.

The 2026 slice is the one to read carefully. Contribution limits, bend points and brackets change every year, so it separates a model that applies the rule it was given from one that recites a rule it memorized. A large gap between held-out and 2026 would mean memorization, and it is reported either way.

Significance, stated rather than assumed

Paired design, so significance is an exact McNemar test on the rows where exactly one model was right, with the null that the adapter is no better than the base.

slice	only tuned right	only base right	exact two-sided p
held-out	592	0	1.2e-178
hard	152	1	2.7e-44
2026 rules	295	0	3.1e-89

Recompute with the released significance.py from the released generation files. It uses exact integer binomial coefficients, so a p-value this small is a count of outcomes rather than an artifact of a normal approximation.

Knowing when not to answer

20,000 of the 100,000 training rows teach the model not to compute: 10,000 ask for a missing input, 5,000 state the question is undecidable as posed, and 5,000 decline outright because the question is regulated advice rather than arithmetic.

behaviour	base	CashSage-4B
asked for the missing input instead of inventing one	0.0%	100.0%
declined a regulated-advice question	0.0%	100.0%
invented an input it was never given	100.0%	0.0%

A model that silently assumes an interest rate to finish a payoff schedule produces a confident number that is wrong in a way the user cannot see. In this domain that is worse than refusing, so abstention is graded as its own outcome rather than folded into accuracy.

Honest limitations, with numbers attached

1. savings_timeline starts from 32.1%, not zero, and the improvement there is correspondingly smaller. Quoting a single blended figure would hide that, so the per-family table is published.
2. Scope. US conventions: IRS contribution limits, Social Security bend points, N-1A fund cost disclosure. Not tax preparation, not state-specific rules, not securities selection.
3. This is not financial advice. 5,000 training rows exist to make the model say so, and the refusal rate is measured rather than assumed.
4. Token budget was measured, and the spec was wrong. The p99 is 1,309 tokens against an assumed 1280, which would have truncated 13.4% of rows. Training uses 1536.

Training details

base	google/gemma-3-4b-it
method	LoRA r32 alpha64 on the language-model linears, completion-only masking
corpus	100,000 rows, 7 families, 4 shards
epochs	2
max_len	1536, measured (p99 1,309), not the 1280 the spec assumed
hardware	one A100-40GB, Spot

Two epochs rather than three is a disclosed choice: at three this run projected to 33 hours on a preemptible instance, which is an unreasonable exposure for a marginal gain. The sibling Science entry in this portfolio earned its results at two epochs on a corpus of comparable size, so this is precedented rather than a concession.

Evaluation protocol

Base and tuned generated in one process under identical greedy decoding (do_sample=False). Output length compared base against tuned, with a tuned median below 0.6x the base median failing the run regardless of accuracy. Every label is produced by executing the computation and checked against an independent implementation; rows whose implementations disagreed were dropped, not repaired.

Decontaminated against FinQA, ConvFinQA, TatQA and FinChain: 101,980 rows screened against 38,744 reference 8-grams, 0 flagged, with the flagged-id list published and empty.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
                                             attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/cashsage-4b").eval()

Try it, and everything that backs it

Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/cashsage-demo Enter your own input and watch the base model and CashSage-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.

	Hugging Face	Kaggle
model	https://huggingface.co/Jainamshahhh/cashsage-4b	https://www.kaggle.com/datasets/jainamshahhh/cashsage-4b
dataset	https://huggingface.co/datasets/Jainamshahhh/cashsage	https://www.kaggle.com/datasets/jainamshahhh/cashsage-personal-finance-computation

Released on both platforms, with the scorer, every eval slice, the per row verdicts and significance.py alongside, so every number on this page can be recomputed rather than trusted.

Reproducibility and license

The corpus, the scorer, every held-out slice and the decontamination report are published. Apache-2.0, matching the base model. Built with Adaptive Data by Adaption Labs, under a per-row merge rule pre-registered before the enhancement pass ran, because a merge rule authored after inspecting the returned rows is a rule fitted to those rows.