HumanAIConvention/simsat-lfm25vl-450m-v3

SimSat LFM2.5-VL-450M v3 — LoRA adapter (canonical) for satellite-tasking observations

LoRA fine-tune of LiquidAI/LFM2.5-VL-450M trained on operator-reviewed Sentinel-2 tiles for the AI in Space Hackathon (DPhi Space x Liquid AI) — Liquid Track.

This is the canonical SimSat adapter. v1 (HumanAIConvention/simsat-lfm25vl-450m-v1) remains published for reference, but v3 supersedes it on every measured metric.

Holdout eval (matched-pair, 32 samples, 8 per action class)

Metric	Base	Tuned (v3)	Delta
exact_action_agreement	0.156	0.844	+0.688
score_mae (lower better)	0.365	0.055	-0.310
useful_agreement (JSON usable_observation)	0.312	0.688	+0.375
parse_rate	1.000	1.000	0.000

Per-class action accuracy (tuned): accept 1.000 / refine 1.000 / defer 0.625 / skip 0.750.

Comparison to v1

Metric	v1 Run 14 (no rep_penalty)	v1 + rep_penalty (Run A)	v3
parse_rate	0.906	1.000	1.000
exact_action_agreement	0.656	0.750	0.844
score_mae	0.102	0.080	0.055
skip per-class	0.375	0.750	0.750
refine per-class	0.625	0.750	1.000

The +18.8 pp action lift over Run 14 (+9.4 pp over Run A) came from two independent levers compounding cleanly:

1. Decode hardening at eval time (repetition_penalty=1.05, no_repeat_ngram_size=20): killed a numeric-field repetition loop that was randomizing skip-class outputs. Applied symmetrically to base and tuned so the matched-pair stays fair.
2. +57.8% training data (109 → 165 train rows): two operator review sessions on materialized encounters added 16/31/4/0 a/r/d/s and 15/0/0/0 a/r/d/s respectively. Refine moved from 27 → 60 train rows and the holdout refine accuracy jumped 0.625 → 1.000.

Training recipe

• Framework: TRL SFTTrainer + PEFT LoRA, transformers (main).
• Dataset: benhaslam/simsat-lfm-v2 on Kaggle — 165 train / 32 holdout (8 per class) / 4 legacy eval. Holdout is identical to v1's holdout for fair comparison; new reviews went only into train.
• LoRA: r=16, alpha=32, dropout=0.05, target_modules from Liquid4All/leap-finetune LFM2.5-VL recipe (q/k/v/o + gate/up/down + vision proj).
• Optimizer: AdamW, lr=2e-4, cosine schedule, weight_decay=0.01, max_grad_norm=1.0, warmup_ratio=0.03.
• Batching: per_device_batch=1, grad_accum=8, effective batch 8.
• Schedule: 5 epochs, save_strategy="steps", save_steps=8, bfloat16 on Tesla T4.
• Loss masking: assistant-only (prompt-length re-tokenize in collator).
• Decode-time generation: repetition_penalty=1.05, no_repeat_ngram_size=20, do_sample=False — applied to BOTH base and tuned in eval.
• Trainable params: 4.5 M / 453 M (0.98%).

Reproduction

Public Kaggle kernel: benhaslam/simsat-lfm2-5-vl-v3-training. Code, dataset prep, and eval harness in HumanAIConvention/SimSat.

Inference

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "LiquidAI/LFM2.5-VL-450M"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype="bfloat16")
processor = AutoProcessor.from_pretrained(base)
model = PeftModel.from_pretrained(model, "HumanAIConvention/simsat-lfm25vl-450m-v3")

# At inference, use:
out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=False,
    repetition_penalty=1.05,
    no_repeat_ngram_size=20,
)

License

Apache-2.0, matching the LFM2.5-VL-450M base model. Operator-reviewed Sentinel-2 tiles in the training set are derived from Copernicus Sentinel-2 imagery (c) European Union, Copernicus Sentinel-2 data 2024-2026, redistributable under the Sentinel data legal notice.