Qwen3.5-4B-d0-vtok101-lora-seeds

From the paper Elucidating the Design Space of LM Data Attribution.

LoRA replicates for a sweep over how wide a synthetic corpus is as well as how deep: 84 runs over 3 function counts (25, 50, 100), 7 document counts (1, 5, 10, 20, 30, 40, 50 per function) and 4 training-order seeds.

Each model is Qwen 3.5 4B taught a set of synthetic constant functions, where each function's answer is a single added vocabulary token rather than a digit string. An earlier suite varied only documents per function, which makes its redundancy results a claim about depth; holding documents per function fixed and varying how many functions compete separates "how many documents carry this fact" from "what share of the corpus do they hold".

The corpora nest exactly. A document's genre follows its function's index and its arguments are salted per function, so a given function's documents are the same documents in all three arms: the 25-function corpus is a strict subset of the 50-function one, which is a strict subset of the 100-function one. A difference between arms is the corpus composition and not a redraw.

Loading a run

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-base")
tok = AutoTokenizer.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-base")
model = PeftModel.from_pretrained(base, "catt-bench/Qwen3.5-4B-d0-vtok101-lora-seeds", subfolder="f100_10d_sd1001")

Subfolders are f{functions}_{docs}d_sd{seed}. The base must be catt-bench/Qwen3.5-4B-d0-vtok101-base and nothing else: these adapters train embed_tokens and lm_head in full for a 248,478-row vocabulary, so peft will refuse any base of a different shape. In particular they are not interchangeable with the earlier 53-answer-token suite, which is a separate repository on a separate base.

What each run contains

File	What
adapter_model.safetensors, adapter_config.json	the adapter
run_meta.json	headline numbers: accuracy, open-vocabulary accuracy, retention perplexity, plus the recipe and how long it took
eval_results.json	all 2,500 scored prompts, each with its prediction and confidence, and per-function accuracy
retention_results.json	the perplexity measurement and its generations
training_config.json	the exact config that produced the run

Accuracy is an argmax over the 101 answer tokens, so chance is 1/101 for every run in the sweep; open_vocab_accuracy is the stricter argmax over the whole vocabulary. Retention perplexity is measured against the untuned base, whose own value is published with it.

The recipe

LoRA r=64 / alpha=128 / dropout 0.05 on all-linear, with embed_tokens and lm_head in modules_to_save and gradients masked below row 248,077 -- so the pre-existing embeddings cannot drift while the 401 appended rows train at full rate. Effective batch 10, cosine to 1e-6 with 100 warmup steps, bf16, max_length 2048. The LoRA init seed is pinned across data seeds, so the four seeds isolate training order and nothing else, and epochs are chosen per cell to hold total optimizer steps near 1000 -- 2 epochs at 100 functions x 50 documents against 400 at 25 x 1 -- so no arm is trained harder than another.

Some runs were trained with the batch split into micro-batches and gradient accumulation holding the effective batch constant, which lets smaller cards contribute. That is not quite numerically identical to one batch of 10, since per-micro-batch loss averaging differs from averaging over a whole batch's tokens, so run_meta.json records the micro-batch each run actually used.