Document corrected adapter training and loading

corrected_image_adapter_step500/TRAINING_FINDINGS.md

@@ -0,0 +1,86 @@
+# Realistic-image tuning findings
+
+## Data audit
+
+The 721 Parquet shards contain 46,119 rows, but only 44,440 have
+`status=completed`; 1,679 are failed generations. After intersecting completed
+rows with the released particle splits there are 39,430 training samples and
+343 held-out samples. Rows without released particle targets cannot be used for
+supervised PGF training.
+
+## Why the previous run did not work
+
+Evaluation uses the exact same particle target, diffusion time, and noise for
+the aligned-image, batch-permuted-image, and zero-image conditions. A useful
+image-conditioned model must have both a low aligned loss and positive control
+penalties.
+
+| checkpoint (EMA) | aligned loss | permuted - aligned | zero - aligned |
+|---|---:|---:|---:|
+| released image PGF | 0.9996 | -0.0048 | +0.0167 |
+| previous 5k | 1.1676 | +0.0223 | -0.0015 |
+| previous 10k | 1.3878 | -0.0179 | +0.0142 |
+| previous 15k | 1.6136 | +0.0156 | +0.0003 |
+| previous 20k | 1.9356 | -0.0799 | -0.0361 |
+| previous 25k | 2.4031 | -0.0580 | -0.0038 |
+| previous 30k | 2.7410 | +0.0977 | +0.2163 |
+| previous 35k | 3.3768 | +0.1124 | +0.0307 |
+
+The previous run therefore caused catastrophic forgetting and did not learn a
+reliable image signal. Its DCP metadata contains all 911 non-encoder model
+tensors, including the particle backbone. Updating that whole 1.4B-parameter
+model was unnecessarily destructive for a 39k-example domain adaptation.
+
+Padding every target to 8,192 points does not explain the result: under that
+distribution the base scores 1.0181 and the old 35k checkpoint scores 3.2950.
+
+There is also an architecture-name mismatch: the old checkpoints store 28
+`blocks.*.norm4.weight` tensors, while the released/current model calls these
+`blocks.*.additional_norm.weight`. A normal `strict=False` load silently drops
+the learned tensors. The evaluator explicitly maps these names.
+
+Finally, Hydra's `${now:...}` run directory is evaluated independently in each
+`torchrun` worker. Ranks that cross a one-second boundary write shards into
+different checkpoint directories. `train_realistic_image.sh` now resolves one
+absolute run directory in the parent process and passes it to every worker.
+
+## Corrected recipe
+
+The launcher starts from the released image PGF, trains only the image
+projection/cross-attention adapters (about 150M of 1.4B parameters), uses
+variable-length targets, front images only, 10% image dropout for classifier-
+free guidance, unconditional text, and a `2e-5` learning rate. Checkpoints are
+adapter-only and must be overlaid on the released image PGF.
+
+The 1,000-step pilot produced three verified checkpoints (the old trainer's
+final-save race made step 1,000 incomplete; that race is now fixed). Full
+held-out results are:
+
+| checkpoint/state | aligned loss | permuted - aligned | zero - aligned |
+|---|---:|---:|---:|
+| 250 EMA | 0.9964 | +0.0056 | -0.0056 |
+| 250 raw | 1.0139 | +0.0035 | -0.0116 |
+| **500 EMA** | **0.9947** | **+0.0204** | -0.0032 |
+| 500 raw | 1.0018 | +0.0052 | -0.0124 |
+| 750 EMA | 1.0026 | +0.0126 | -0.0152 |
+| 750 raw | 1.0063 | +0.0179 | -0.0089 |
+
+Step 500 EMA is the pilot winner: it slightly improves aligned loss while
+giving the largest mismatched-image penalty. The negative zero penalty is not
+the primary conditioning diagnostic because blank images were intentionally
+used during classifier-free training; batch permutation is the clean control.
+
+Evaluate one with:
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,2,4,5,6,7 uv run torchrun --standalone \
+  --nproc_per_node=7 tools/evaluate_realistic_checkpoints.py \
+  --base-checkpoint checkpoints/pgf_image \
+  --checkpoint /path/to/adapter/checkpoints/0000250 \
+  --checkpoint-state both \
+  --particle-dir ../garment_data/extracted \
+  --image-dir ../realistic_gcdv2 \
+  --assets-dir ../realistic_assets \
+  --split ../realistic_assets/garment_particle_realistic_val.txt \
+  --output-dir outputs/adapter_eval
+```