dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability

Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Verlet Instability at Full-Run Scale (OpenWebText, d=384)

This repository holds the two full-scale training attempts ( $\gamma =0.10\backslash gamma\; =\; 0.10$ and $\gamma =0.30\backslash gamma\; =\; 0.30$) that followed the d=384 gamma sweep, together with a complete SCAF causal-leak and numerical-stiffness audit of both. Neither run is a successfully completed model. Both were targeting 100,000 steps; both instead entered a persistent cycle of gradient spikes and watchdog reloads well short of that target, and neither improved further once the cycle began. This repository exists to document why, with checkpoints and audit evidence retained specifically so the failure mode can be studied.

The headline finding, from the SCAF stiffness audit (Phase 7/7b/7c):

The model's own explicit Störmer–Verlet integrator is running past its analytical stability boundary. The anisotropic-Gaussian $V_{\theta }V\_\backslash theta$'s off-diagonal (rank-4 low-rank) curvature pushes the effective per-token stiffness $\omega \mathrm{\Delta }t\backslash omega\; \backslash Delta\; t$ above the Verlet stability bound of 2 at every single audited checkpoint in both runs — while a naive diagonal-only proxy for that same curvature stays under 1 throughout and would have reported no problem at all. This is a structural property of the trained potential, not a training-recipe bug, and it is what motivated replacing Verlet with a CfC + BAOAB propagator (unconditionally stable with respect to this curvature by construction) as this architecture's default integrator going forward.

Separately, both checkpoint sets pass SCAF's Phase-1 causal-leak audit cleanly — the instability documented here is a numerical-integration problem, not a causal-leak regression.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Repository
• Architecture
• About the SCAF Library
• Training Runs: Two Damping Coefficients, Two Stalled Runs
• SCAF Causal-Leak Audit: Clean
• SCAF Stiffness Audit: The Instability Mechanism
• Why the Diagonal Proxy Misses It
• Trajectory-Fidelity Cross-Check (Phase 7c)
• What This Motivated: CfC + BAOAB
• How to Get Started
• Available Artifacts
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Repository

Use this repository if you want to:

• Study a real, reproducible instance of explicit-symplectic-integrator instability in a trained conservative language model, including the exact checkpoints, training logs, and audit artifacts involved.
• Reproduce the SCAF stiffness audit methodology (diagonal proxy, Weyl-inequality upper bound, native-trajectory cross-check) against these or similar checkpoints.
• Compare the anisotropic-Gaussian $V_{\theta }V\_\backslash theta$ family's instability signature across two damping coefficients ( $\gamma =0.10\backslash gamma\; =\; 0.10$ and $0.300.30$ ) trained on identical architecture and data.

Do not use this repository if you want a well-trained OpenWebText language model: both checkpoint sets stalled between 7% and 17% of their 100,000-step target, and neither reflects converged behaviour. For a well-trained checkpoint from an earlier, isotropic- $V_{\theta }V\_\backslash theta$ version of this architecture, see semsimula-fock-parflm-depthcond-vtheta-openwebtext (27.23 PPL, 250K steps). The anisotropic- $V_{\theta }V\_\backslash theta$ + CfC-BAOAB successor run that this repository's finding motivated is expected to supersede both once it completes.

Architecture

Identical Fock-PARFLM v2.1 scaffold to the d=384 gamma sweep that selected these damping coefficients — same width, depth, $V_{\theta }V\_\backslash theta$ family, and Fock register configuration, now run to (attempted) full length instead of a 3,000-step sweep candidate:

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped Störmer–Verlet integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [5 contexts, 40 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi(h_t, h_s)                [autograd, sparse]
       |
       +-- Fock register pool (v2, 32 registers):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
       |     prefix-causal (leak-free by construction)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out                              [UNTIED W_out]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

Parameter	Value
Hidden dim (d)	384
Layers (L)	16
Max sequence length	1024 (trained at block 512)
Vocab (GPT-2 BPE)	50257
V_theta kind	Depth-conditioned multi-context anisotropic Gaussian (bounded mixture)
V_theta contexts (n_ctx)	5
Wells per context (K)	8
Total attractors	40
Anisotropic rank (r)	4
Precision init / cap	a_k init at -log(384), capped at 2/384
Xi channels (K_xi)	5 (alpha inits 0.50, 0.75, 0.95, 0.99, 0.995)
Fock-coupling regulariser	log-barrier on alpha_k, lambda=0.005, eps=1e-6
V_phi kind	structural_competitive, 4 heads
Top-k (sparse routing)	16
Fock version	v2, 32 registers, d_k=64, LIFO stack
Reverse channel	Yes — per-layer, stabilised, pre-LN, soft-norm, 4,000-step warmup
Integrator	Störmer–Verlet, position form (dt=1.0) — see below for why this is the finding
Embeddings	Untied (separate W_out)
Mass model	logfreq (frozen OpenWebText surprisal lookup)
Prefix-causal registers	Yes — both checkpoint sets trained natively leak-free from step 0
Total parameters	77,082,733

About the SCAF Library

Every audit result quoted in this card comes from SCAF (SemSimula Causal Auditing Framework), a standalone, model-agnostic library that resolves onto a checkpoint by structural (duck) typing — no base class or Protocol required — and offers two independent audit families:

• Causal audit (on main): detects, sizes, and attributes causal leaks — any path by which a token at position s influences a prediction at an earlier position t < s, which would otherwise report a perplexity the model did not earn. Two probes ask deliberately different questions: future_perturbation (bit-exact — is there a leak at all?) and target_relocation (how many nats of perplexity were unearned?), backed by three controls (determinism, placebo, positive) that a verdict of CLEAN cannot be issued without. A mediation diagnostic then attributes a confirmed leak to the responsible component by knockout, and scaf.LeakMonitor runs the same probes continuously during training, since leak channels are often gated by a scale initialised at zero and open gradually as training proceeds. A build_leak_frame / estimate_leak bridge hands the same intervention to DoWhy/EconML for exact paired-inference ATEs, refutation tests, and CATE-by-distance heterogeneity profiles via a causal forest. This is what certified both this repository's checkpoint sets CLEAN in SCAF Causal-Leak Audit: Clean.
• Stiffness audit (scaf.StiffnessProbe, currently on the stiffness_audit branch, scheduled to merge into main alongside SCAF's other active branches): audits a trained model's numerical integration stability rather than its causal structure, via the diagonal-proxy / Weyl-inequality-upper-bound / native-trajectory-cross-check methodology used throughout this card's stiffness audit section. It is trajectory-faithful by construction (it drives the model through InterventableModel.batch_logits_with_trajectory rather than re-deriving a trajectory by hand), reports block-bootstrap 95% CIs on every fraction-unstable statistic, and skips loudly rather than silently if a model does not expose the harmonic_terms()/mass() hooks it needs.

Full design rationale and worked examples for both audit families: docs/Stiffness_Audit_Requirements_and_Design.md and docs/Example_Stiffness_Audit_OWT_g0.1_Anisotropic_Gaussian.md (both on the stiffness_audit branch as of this card's creation; once merged, the same paths resolve on main).

Training Runs: Two Damping Coefficients, Two Stalled Runs

Both runs used identical architecture, data, and optimisation recipe, differing only in ${\gamma }_{\text{train}}\backslash gamma\_\{\backslash text\{train\}\}$, the damping coefficient selected by the gamma sweep ( $\gamma =0.10\backslash gamma\; =\; 0.10$ was the sweep winner; $\gamma =0.30\backslash gamma\; =\; 0.30$ was audited as a second candidate on the same architecture to see whether heavier damping alone would suppress the instability). Neither did:

	$\gamma =0.10\backslash gamma\; =\; 0.10$	$\gamma =0.30\backslash gamma\; =\; 0.30$
Target steps	100,000	100,000
Max step actually logged	17,100	11,254
Watchdog reloads	15	5
First reload	step 8,925	step 7,124
Best checkpoint (canonical _best.pt)	step 10,000	step 10,000
Best val PPL	184.11	211.63
PPL at last logged step	worse than best (192.53 at step 15,000)	worse than best (221.35 at step 7,500)

Both runs show the same qualitative signature: PPL improves normally up to the first watchdog reload cluster, then stops improving — every checkpoint saved after the reloads begin is worse than the pre-cluster best, and the run never recovers a new best afterward within the logged window. Heavier damping ( $\gamma =0.30\backslash gamma\; =\; 0.30$ ) delayed the first reload (step 7,124 vs. 8,925 — comparable, if anything slightly earlier) and reduced the reload count (5 vs. 15) but did not eliminate the pattern, which is consistent with a structural integrator problem rather than an under-damping problem (see below).

SCAF Causal-Leak Audit: Clean

Both checkpoint sets were run through SCAF's Phase-1 causal-leak audit (FockAdapter, future_perturbation and target_relocation probes, plus determinism/placebo/positive controls). Verdict: CLEAN for both, at every audited checkpoint:

Probe / control	$\gamma =0.10\backslash gamma=0.10$	$\gamma =0.30\backslash gamma=0.30$
control_determinism	PASS (0.0 logit)	PASS (0.0 logit)
control_placebo	PASS (0.0 logit)	PASS (0.0 logit)
control_positive	PASS (9.31 logit — probe reaches the model)	PASS (8.80 logit)
future_perturbation	PASS (0.0 logit, bit-exact)	PASS (0.0 logit, bit-exact)
target_relocation	PASS (0.0 nats, PPL inflation 1.0x)	PASS (0.0 nats, PPL inflation 1.0x)

Both checkpoint sets were trained natively with prefix_causal_registers=True from step 0, so this is the expected result, not a new fix — it confirms the instability documented below is purely a numerical-integration problem, with no interaction with the register-leak fix from earlier in this model family. Full audit output: results/gamma_0.10/scaf_analysis_results/phase1_scaf_audit.json and results/gamma_0.30/scaf_analysis_results/phase1_scaf_audit.json.

SCAF Stiffness Audit: The Instability Mechanism

The anisotropic-Gaussian $V_{\theta }V\_\backslash theta$'s force is a sum of Gaussian bumps with non-diagonal curvature:

$$f(h)=-\sum _k{g}_k(h)P_k(h-{\mu }_k),g_k(h)=w_k\exp (-\frac{1}{2}{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}}_k^{\mathrm{\top }}{P}_k{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}}_k)>0,P_k=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(a_k)+B_k{B}_k^{\mathrm{\top }},f(h)\; =\; -\backslash sum\_k\; g\_k(h)\; P\_k\; (h\; -\; \backslash mu\_k),\; \backslash qquad\; g\_k(h)\; =\; w\_k\; \backslash exp\backslash left(-\backslash tfrac\{1\}\{2\}\; \backslash mathrm\{diff\}\_k^\backslash top\; P\_k\; \backslash mathrm\{diff\}\_k\backslash right)\; >\; 0,\; \backslash qquad\; P\_k\; =\; \backslash mathrm\{diag\}(a\_k)\; +\; B\_k\; B\_k^\backslash top,$$

where $B_k\in {\mathbb{R}}^{384\times 4}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{384\; \backslash times\; 4\}$ is the learned low-rank factor giving each well its off-axis curvature. Near a well, this is locally a spring with stiffness matrix ${\sum }_k{g}_k{P}_k\backslash sum\_k\; g\_k\; P\_k$, and the explicit Störmer–Verlet integrator that this model was trained with is a second-order symplectic method whose stability for a harmonic oscillator with frequency $\omega \backslash omega$ requires

Above this bound the discrete update's amplification factor leaves the unit circle and every subsequent step amplifies the deviation instead of oscillating around it — trading a bounded, physically faithful trajectory for one that diverges geometrically, which is exactly the gradient-spike-then-reload signature seen in both training logs.

SCAF's stiffness audit (Phase 7) samples the trained model's own hidden-state trajectory and computes $\omega \mathrm{\Delta }t\backslash omega\; \backslash Delta\; t$ at each sampled (token, layer) position, in two ways:

1. Diagonal proxy (k_diag): keeps only the diagonal of ${\sum }_k{g}_k{P}_k\backslash sum\_k\; g\_k\; P\_k$, the quantity the CfC+BAOAB propagator (see below) can integrate exactly in closed form.
2. Weyl-inequality upper bound (Phase 7b): applies Weyl's inequality twice — once per well ( ${\lambda }_{\max }(P_k)\le {\max }_i{a}_k[i]+{\sigma }_{\max }(B_k{)}^2\backslash lambda\_\{\backslash max\}(P\_k)\; \backslash le\; \backslash max\_i\; a\_k[i]\; +\; \backslash sigma\_\{\backslash max\}(B\_k)^2$ ) and once across wells — to certify an upper bound on the true, any-direction top eigenvalue of ${\sum }_k{g}_k{P}_k\backslash sum\_k\; g\_k\; P\_k$, not just its diagonal.

step	val PPL	diag max	diag frac(>2)	Weyl max	Weyl frac(>2)
gamma=0.10, 9,000	194.68	0.6319	0.000e+00	4.2452	5.28e-02
gamma=0.10, 9,500	191.84	0.3423	0.000e+00	2.7128	5.62e-02
gamma=0.10, 10,000	184.11	0.4711	0.000e+00	2.9264	5.93e-02
gamma=0.10, 15,000	192.53	0.4182	0.000e+00	2.6572	6.21e-02
gamma=0.30, 4,500	222.60	0.4131	0.000e+00	2.5318	5.61e-02
gamma=0.30, 5,000	218.08	0.5607	0.000e+00	3.0960	2.68e-02
gamma=0.30, 6,000	215.09	0.4105	0.000e+00	2.8634	2.84e-03
gamma=0.30, 7,500	221.35	0.5914	0.000e+00	3.6095	2.94e-03
gamma=0.30, 10,000	211.63	0.3835	0.000e+00	2.8538	3.76e-03

At every one of the nine audited checkpoints across both runs, Weyl max clears the stability bound of 2 (range 2.53-4.25), while diag max never even approaches 1. Statistically, Weyl frac(>2) — the fraction of all sampled (token, layer) positions where the Weyl bound itself exceeds 2 — is consistently 2.7%-6.2% of positions, not a rare tail event: a non-trivial share of the model's own forward pass is operating past this integrator's stability boundary at every checkpoint examined.

Why the Diagonal Proxy Misses It

A synthetic verification (500 trials of random 8-well anisotropic mixtures at d=16, rank=4, checked against the exact eigendecomposition of the true effective matrix) confirms this is a real property of the inequality, not a fluke of these two checkpoints: k_diag's per-dimension maximum underestimated the true top eigenvalue in all 500/500 trials (mean gap 3.77, on a typical eigenvalue scale of 5-18), while the Weyl bound never once fell below the true value. The rank-4 low-rank correction $B_k{B}_k^{\mathrm{\top }}B\_k\; B\_k^\backslash top$ can concentrate curvature along a direction that is not axis-aligned, and a purely diagonal statistic is structurally blind to exactly that direction — which is precisely the direction the model's own learned wells are using. Full derivation and the synthetic verification plot: Weyl-inequality upper bound section of the SCAF stiffness-audit design doc.

Trajectory-Fidelity Cross-Check (Phase 7c)

Phase 7 samples the audited trajectory under a forced baoab_cfc integrator regardless of a checkpoint's own native integrator, which could in principle change which curvature the audit samples. Phase 7c closes this caveat by re-running each checkpoint under its own native Verlet trajectory instead, for both runs:

step	Weyl max, forced	Weyl max, native	Weyl frac(>2), forced	Weyl frac(>2), native
gamma=0.10, 9,000	4.2452	3.0195	5.282e-02	5.276e-02
gamma=0.10, 9,500	2.7128	4.0838	5.619e-02	5.621e-02
gamma=0.10, 10,000	2.9264	2.7457	5.932e-02	5.932e-02
gamma=0.10, 15,000	2.6572	3.1421	6.206e-02	6.204e-02
gamma=0.30, 4,500	2.5318	2.4708	5.610e-02	5.608e-02
gamma=0.30, 5,000	3.0960	2.6971	2.683e-02	2.673e-02
gamma=0.30, 6,000	2.8634	2.7393	2.838e-03	2.777e-03
gamma=0.30, 7,500	3.6095	2.8911	2.945e-03	2.914e-03
gamma=0.30, 10,000	2.8538	2.8538	3.761e-03	3.754e-03

Weyl frac(>2) agrees between forced and native trajectories to within 0.02 percentage points at every single checkpoint in both runs, and Weyl max stays clear of the stability bound in both modes at every checkpoint too. The forced-vs-native discrepancy is negligible next to the trend it is being asked to corroborate — the trajectory-substitution caveat is closed for both damping coefficients, by their own checkpoints, without needing extra training.

What This Motivated: CfC + BAOAB

This evidence — a structural curvature violation of the Verlet stability bound present at every audited checkpoint, in both damping regimes, confirmed independent of the audit's own trajectory choice — is what motivated replacing Störmer–Verlet with a Closed-form Continuous-time (CfC) + BAOAB propagator for this architecture going forward. CfC integrates the harmonic component of the force in closed form (via torch.sinc-based exact-flow update rather than an explicit finite-difference step), which is unconditionally stable with respect to this curvature by construction rather than conditionally stable below $\omega \mathrm{\Delta }t=2\backslash omega\; \backslash Delta\; t\; =\; 2$.

This is not a free lunch: the CfC+BAOAB propagator costs roughly 2.4x Störmer–Verlet's per-step wall-clock time on the same GPU. An early CfC+BAOAB run also hit its own regression — a grad=nan traced to torch.sqrt's infinite derivative at exactly k_diag/m == 0 inside cfc_substep — which was fixed with an _OMEGA_SQ_FLOOR = 1e-12 clamp and confirmed to produce zero NaNs across a full resumed training run. The CfC+BAOAB run is in progress as of this card's creation and will be published as its own model card once complete; see the deep-dive writeup for the full implementation: PyTorch_Implementation_of_CfC_BAOAB_in_Fock-PARFLM.md.

How to Get Started

import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability"
GAMMA = "0.10"  # also available: "0.30"
STEP = 10000    # the canonical best checkpoint for either gamma

logfreq_path = hf_hub_download(repo_id=REPO, filename=f"results/gamma_{GAMMA}/logfreq_surprisal_openwebtext.npy")

config = FockMultiXiPARFConfig(
    vocab_size=50257, d=384, max_len=1024, L=16,
    mass_mode="logfreq", logfreq_path=logfreq_path,
    init_gamma=1.0, fixed_gamma=float(GAMMA),
    xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
    xi_learnable=True, xi_alpha_init_mode="explicit",
    fock_version="v2", n_registers=32,
    reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
    reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
    register_repulsion=True, register_repulsion_coeff=0.05,
    prefix_causal_registers=True,
    v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16, top_k=16,
    use_output_bias=True, tie_embeddings=False,
    per_register_tau=True, per_register_keys=True, ortho_register_init=True,
)
model = FockMultiXiPARFLM(config)

model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=384, K=8, n_ctx=5, n_layers=16, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(384),
    precision_max=2.0 / 384,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_step{STEP}_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")   # 77,082,733
print(f"gamma={state['gamma']}  step={state['step']:,}  val_ppl={state['val_ppl']:.2f}")

Available Artifacts

Path	Description
checkpoints/gamma_0.10/ckpt_step{9000,9500,10000}.pt (_best), ckpt_step15000.pt	The four gamma=0.10 checkpoints audited by SCAF, plus the last periodic checkpoint of the run
checkpoints/gamma_0.30/ckpt_step{4500,5000,6000}.pt (_best), ckpt_step{7500,10000}.pt	The five gamma=0.30 checkpoints audited by SCAF (step 10,000 is also _best)
checkpoints/gamma_*/training_log.jsonl	Full per-step training log, including every watchdog_reload and grad_spike event
results/gamma_*/logfreq_surprisal_openwebtext.npy	Frozen per-token log-frequency (surprisal) lookup used by the logfreq mass model, per run
results/gamma_*/scaf_analysis_results/phase1_scaf_audit.json	SCAF Phase-1 causal-leak audit (verdict, controls, probes)
results/gamma_*/scaf_analysis_results/phase7_stiffness_audit.json	Diagonal-proxy omega*dt statistics per checkpoint
results/gamma_*/scaf_analysis_results/phase7b_eig_bound_audit.json	Weyl-bound omega*dt statistics per checkpoint, with block-bootstrap 95% CIs
results/gamma_*/scaf_analysis_results/phase7c_native_trajectory_audit.json	Native-Verlet-trajectory cross-check of both the diagonal proxy and the Weyl bound
results/gamma_*/scaf_analysis_results/*.png	All audit figures (stiffness-vs-step, eig-bound-vs-step, phase7c native-vs-forced)
results/gamma_*/scaf_analysis_results/phase2*.json, phase2_5*.csv, phase3*.json, phase4*.json, phase5*.png, phase6*.{json,png}	Supporting SCAF causal-estimation phases (ATE, register diagnostics, CATE-by-distance), included for completeness
model_aniso_gaussian_vtheta.py	Anisotropic Gaussian V_theta classes + install_aniso_depth_routing (identical file to the gamma-sweep repo's)
config.json	Full run configuration, per-run summary statistics, and links to the successor CfC+BAOAB work

Training Details

Training Data

OpenWebText, tokenized with GPT-2 BPE (vocab 50257), evaluated on a held-out 2M-token validation slice with no train/val overlap. Both runs targeted 100,000 steps and were early-terminated (in the sense of "checkpoints stopped being retained as _best") by the instability documented above, at step 17,100 (gamma=0.10) and step 11,254 (gamma=0.30).

Training Procedure

Hyperparameter	Value
Optimizer	AdamW (betas 0.9/0.95, weight decay 0.01)
LR schedule	WSD
Peak learning rate	3e-4
Block size	512
Target steps	100,000
Gradient clipping	per-group (global 1.0; V_phi and Fock gates tighter)
Watchdog	reloads last-best checkpoint on gradient-norm-EMA or grad-spike triggers
lambda_V (V_theta regularisation)	0.01
lambda_fock (coupling regulariser)	0.005
Hardware	1x NVIDIA H100/A100 (Google Colab)

Causal-Leak Verification

See SCAF Causal-Leak Audit: Clean above — both runs pass with verdict CLEAN.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_aniso_gaussian_fockreg_openwebtext.ipynb (companion repo). SCAF audit performed with notebooks/conservative_arch/scaleup/debug/scaf_checkpoint_analysis.ipynb (same repo).

Evaluation Results

OpenWebText Validation Perplexity (best retained checkpoint per run)

$\gamma \backslash gamma$	Best step	PPL
0.10	10,000	184.11
0.30	10,000	211.63

Neither of these is comparable to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic flagship) — both runs stalled at 10-17% of their 100,000-step target due to the instability documented above.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	Corpus	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	TinyStories	11.51	semsimula-splm-multixi
Multi-Xi SPLM (SQ3)	Structured scalar potential	TinyStories	13.33	semsimula-splm-multixi-structured-vtheta
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	TinyStories	12.06	semsimula-parflm-multixi
Multi-Xi PARFLM (SQ3)	Structured scalar + pairwise	TinyStories	12.27	semsimula-parflm-multixi-structured-vtheta
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	TinyStories	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	TinyStories	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian)	Bounded multi-context + pairwise + Fock	TinyStories	16.33	semsimula-fock-parflm-depthcond-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg)	Bounded, ellipsoidal multi-context + pairwise + Fock	TinyStories	9.04	semsimula-fock-parflm-anisogaussian-vtheta
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1)	Same as above, gradient-flow integrator	TinyStories	8.95	semsimula-fock-parflm-anisogaussian-vtheta-fock-g1
Fock Attention (MLP V_theta)	Fock + attention	TinyStories	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	TinyStories	8.50	semsimula-hybrid-splm
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	278.27 (best of 8, 3K-step sweep)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	326.97 (best of 8, 3K-step sweep)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=1024)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	244.23 (best of 8, 3K-step sweep)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, Verlet instability, d=384)	Same architecture, two full-run attempts — SCAF stiffness audit identifies structural Verlet instability	OpenWebText	184.11 / 211.63 (both runs stalled, not final)	this repository

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Not a final trained model — this is a diagnostic repository. Both checkpoint sets stalled at 10-17% of their 100,000-step target and were never intended to produce fluent text; use them to study the instability, not for generation or downstream use.
• PPL not comparable to converged checkpoints elsewhere in this family. See the caveat under Evaluation Results.
• Structural Verlet instability, not a training-recipe bug. The Weyl-bound curvature violation is present at every audited checkpoint across two different damping coefficients and is confirmed independent of the audit's trajectory-substitution choice (Phase 7c); it should be expected to recur in any future Störmer–Verlet-integrated run of this exact $V_{\theta }V\_\backslash theta$ family unless the curvature itself is constrained (e.g. by capping the rank-4 factor's spectral norm) or the integrator is replaced, as this repository's finding motivated.
• Weyl bound is an upper bound, not the exact top eigenvalue. A synthetic check found a mean overshoot of 3.77 on eigenvalues typically in the 5-18 range — the reported Weyl frac(>2) values (2.7%-6.2%) are a ceiling on how much of each run crosses the stability line via the off-diagonal direction, not an exact measurement. The exact-eigenvalue extension (torch.linalg.eigvalsh on the small rank-4-derived matrix) is a documented follow-up, not yet run against these checkpoints.
• No causal-leak issue. Both checkpoint sets were trained natively with prefix_causal_registers=True; see SCAF Causal-Leak Audit: Clean.
• OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering.
• Single-seed runs. Each of the two damping coefficients is one run; no seed-variance estimate is available for either the instability onset step or its severity.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: 1x NVIDIA H100/A100 80GB (Google Colab)
• Training: 17,100 + 11,254 = 28,354 total logged training steps across both runs, both terminated by the instability documented here rather than reaching their 100,000-step target
• Audit: SCAF Phase 1/7/7b/7c analysis, inference-only, on 9 checkpoints total across both runs
• Carbon footprint: small; single-GPU research runs, estimated on the order of a few kg CO2

Correction (2026-09-27) — "velocity-Verlet" was the wrong name. This lineage's layer step carries no velocity state: the velocity is the position difference h - h_prev, and the update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which undamped is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m. That is Störmer–Verlet in position form. Velocity-Verlet is the variant that does carry an explicit velocity, through half-kick/drift/half-kick, and was never used here. The stability bound is the Störmer/leapfrog bound and is unaffected by the correction, as is every measurement on this card.