---
pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - fock-space
  - parflm
  - parf
  - multi-channel-xi
  - structured-vtheta
  - analytical-gradient
  - interpretable
  - gaussian-mixture-potential
  - virtual-registers
  - controlled-conservativity
  - sparse-routing
  - tinystories
  - non-transformer
  - attention-free
  - constant-memory-inference
  - riemannian-geometry
  - riemannian-geodesics
datasets:
  - roneneldan/TinyStories
metrics:
  - perplexity
model-index:
  - name: semsimula-fock-parflm-structured-vtheta
    results:
      - task:
          type: text-generation
        dataset:
          type: roneneldan/TinyStories
          name: TinyStories
          split: validation
        metrics:
          - type: perplexity
            value: 10.90
            name: Validation Perplexity (SQ3 K=8)
            verified: false
---

> **CAUSAL-LEAK NOTICE — RESOLVED, MODEL RE-TRAINED WITH THE FIX**
>
> A post-publication causal audit identified an **architectural causal leak** in the Fock reverse-channel mechanism shared by all models in the SPLM family that use the reverse channel: the reverse channel blended each token's content into a **global register state shared across all positions within the same integration step**, so future-token information leaked backward into past-token predictions.
>
> **The checkpoint on this page is the re-trained, leak-fixed model** (`prefix_causal_registers=True`). The fix was certified on this exact training run:
> - **Future-perturbation probe, bit-exact:** perturbing tokens after position *t* changed past logits by exactly **0.0** in float64, at steps 8,000 and 16,000, and the deterministic architectural probe (`eval_max_delta`) is bit-exact `0.0` at all four checkpoints (4k/8k/12k/16k).
> - **Honest vs. standard PPL agree** within ~1.4–2.0 standard errors (not statistically distinguishable from zero) — the expected signature once the channel is closed.
>
> **Training-time cost of the fix:** a like-for-like rerun of the *leaky* architecture (same seed, same 16k steps) reproduces the original **10.36 PPL** almost exactly; the fixed rerun reaches **10.90 PPL** — a real but modest **+0.54 PPL** tax, similar in kind (slightly larger in magnitude) to the MLP-\\(V_\theta\\) sibling's **+0.40 PPL** tax. Both are far smaller than the **+3.51 nats (~33× PPL)** the same leak pathway cost on the much larger OpenWebText-scale variant.
> - The "Fock paradox" gap analysis below is updated accordingly: **1.20 PPL / 12.4%** gap vs the honest MLP baseline (was reported as 1.06 PPL / 11.4% under the leaky architecture).
> - **A separate, unrelated error was also found and fixed while re-certifying this model:** the architecture table below previously reported **17,407,980 total parameters**, copy-pasted from the MLP-\\(V_\theta\\) sibling card. The true count for this SQ3 variant, verified by loading the checkpoint with zero missing/unexpected keys, is **18,194,420** (the SQ3 hypernetwork is larger than the 3-layer MLP it replaces).
> - The structured \\(V_\theta\\) contributions (analytical gradients, explicit attractor centres, landscape compression) are unaffected — only the absolute PPL and parameter-count numbers changed.
>
> For the full root-cause analysis, see [`Fock-PARFLM_Causal_Leak_Audit_Results.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock-PARFLM_Causal_Leak_Audit_Results.md). For the paired leaky-vs-fixed rerun on this exact configuration, see this repo's [`causal_leak_fix_verification.md`](causal_leak_fix_verification.md).

# Fock-PARFLM v2.1 with Structured V_theta (SQ3 Mixture of Quadratic Wells)

A **structured variant** of the [Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) language model in which the MLP scalar potential \\(V_\theta\\) is replaced by a **mixture of K=8 diagonal quadratic wells** (SQ3), while retaining the full MLP-based pairwise potential \\(V_\phi\\) and the Fock register mechanism (16 registers, LIFO stack discipline, reverse channel, now leak-free via `prefix_causal_registers=True`). This replacement yields:

- **Full analytical gradients for \\(V_\theta\\)** — no `torch.autograd.grad` needed for the scalar potential force, eliminating the second-order computation graph for the \\(V_\theta\\) component
- **Explicit attractor centres** — the 8 semantic attractors \\(\mu_k(\xi)\\) are readable directly from the model parameters, with no gradient-descent extraction required
- **Most compressed \\(V_\theta\\) landscape** — with both \\(V_\phi\\) and Fock registers carrying the force budget, \\(V_\theta\\) collapses to the flattest bias field observed across all three architectures (mean 0.017, range 16.0)

The trade-off is a **1.20 PPL gap**: **10.90 PPL** (honest, leak-free) vs the MLP baseline's **9.70 PPL** (12.4% excess cross-entropy). This is larger than the PARFLM gap (0.17 PPL) despite even greater landscape compression — the "Fock paradox" discussed below.

This model is from the [Semantic Simulation](https://doi.org/10.5281/zenodo.19712427) framework.

## Table of Contents

- [When to Use This Model](#when-to-use-this-model)
- [Architecture](#architecture)
- [SCAF Causal-Leak Audit (Formal)](#scaf-causal-leak-audit-formal)
- [About the SCAF Library](#about-the-scaf-library)
- [The Fock Paradox: Maximal Compression, Moderate Gap](#the-fock-paradox-maximal-compression-moderate-gap)
- [How to Get Started](#how-to-get-started)
- [Training Details](#training-details)
- [Evaluation Results](#evaluation-results)
- [SPLM Family Overview](#splm-family-overview)
- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
- [Citation](#citation)

## When to Use This Model

Choose this structured variant over the [MLP-based Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) when:

| Priority | Structured V_theta (this model) | MLP V_theta ([baseline](https://huggingface.co/dimitarpg13/semsimula-fock-parflm)) |
|---|---|---|
| **Interpretability** | 8 explicit attractor centres, zero-cost basin readout | Black-box; requires 1,500-step GD extraction per prompt |
| **Inference speed** | ~2x faster V_theta force computation (analytical gradient) | Standard (autograd for both V_theta and V_phi) |
| **Raw PPL** | 10.90 (honest, leak-free) | **9.70** (honest, leak-free) |
| **PPL gap** | 1.20 PPL (12.4%) | — |
| **Memory** | No second-order graph for V_theta (V_phi graph retained) | Full graph for both |

Both PPL numbers are from the leak-fixed architecture (`prefix_causal_registers=True`); see the causal-leak notice above for the fix and its small, honest cost.

**Bottom line:** for Fock-PARFLM, structured \\(V_\theta\\) is a **viable option when interpretability or analytical-gradient inference is valued** — the 1.20 PPL cost is meaningful but the attractor readout and speed gains may justify it. For maximum PPL, use the MLP baseline.

## Architecture

```
Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps:
       |
       +-- K-EMA channels: xi^(k)_t = causal_ema(h, alpha_k)   [K_xi=4 channels]
       |
       +-- Structured V_theta (SQ3):
       |     xi_flat = flatten(xi_1..xi_K)                      [K_xi * d = 1024]
       |     V = -tau * logsumexp_k(-E_k/tau + log pi_k)        [K_mix=8 wells]
       |     f_theta = -analytical_grad_h V                     [closed-form]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange)
       |     *** register state is now per-position (prefix-causal fix); no leak ***
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]
```

| Parameter | Value |
|---|---|
| Hidden dim (d) | 256 |
| Layers (L) | 8 |
| V_theta kind | SQ3 (mixture of K quadratic wells) |
| Mixture components (K_mix) | 8 |
| Temperature (tau) | 1.0 |
| Xi channels (K_xi) | 4 |
| V_phi kind | structural_competitive |
| V_phi hidden | 128 |
| Top-k (sparse routing) | 8 |
| Gumbel tau | 1.0 (init), 0.3 (min) |
| Fock version | v2.1 |
| Registers (M) | 16 |
| Register d_k | 64 |
| Stack discipline | LIFO |
| Reverse channel | Yes — fixed lifecycle, leak-free (was the causal leak source pre-fix) |
| Per-register tau/keys | Yes |
| Gathered V_phi | Yes |
| Per-layer V_phi scale | Yes |
| LN before distance | Yes |
| Layer checkpoint | Yes |
| Mass model | logfreq (frozen surprisal lookup) |
| Damping gamma | 0.30 nominal, ~0.033 effective (LayerNorm prevents compounding; see note below) |
| lambda_V (V_theta regularisation) | 0.01 |
| Prefix-causal registers | **Yes** (`prefix_causal_registers=True`; bit-exact 0.0 future-perturbation sensitivity verified at steps 8k/16k) |
| Total parameters | **18,194,420** (corrected — previously misreported as 17,407,980, copy-pasted from the MLP sibling) |
| \\(V_\theta\\) parameters | **4,207,625** |

**Effective damping.** The nominal gamma = 0.30 overstates the true dissipation. The LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics are therefore **heavily underdamped** even at this nominal value. Gamma-sweep experiments on the [OpenWebText-scale depth-conditioned variant](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext) confirm that the effective damping is much smaller than the nominal coefficient.

## SCAF Causal-Leak Audit (Formal)

The causal-leak notice above and `causal_leak_fix_verification.md` document a hand-built verification of the fix. This section adds a formal, tool-generated audit of this exact checkpoint (`checkpoint/ckpt_best.pt`) using [SCAF](https://github.com/dimitarpg13/semsimula-scaf)'s standardised probe battery. **Verdict: `CLEAN`**, with every control and probe passing:

| Control / probe | Result |
|---|---|
| `control_determinism` | PASS (0.0 logit) |
| `control_placebo` | PASS (0.0 logit) |
| `control_positive` | PASS (61.56 logit — probe reaches the model) |
| `future_perturbation` | PASS (0.0 logit, bit-exact) |
| `target_relocation` | PASS (-3.8e-06 nats, PPL inflation ~1.0x) |
| Tier-A hidden-state leak probe (Phase 1.5) | PASS (0.0 cosine deviation at every one of 9 sampled layers) |
| Phase-2 ATE (do-future intervention) | 0.0 nats (p=1.0, 10,000 permutations) |
| Phase-3 legacy architectural probe | CLEAN (bit-exact 0.0 `max_dlogit_past` / `mean_dnll_past`) |
| Phase-6 estimation-report cross-check | agrees with reference ATE (0.0), refutations pass |

> **On the raw PPL numbers inside these JSON files:** `target_relocation` and the legacy probe report `standard`/`honest` PPL in the billions (e.g. `detail_ppl_standard: 19,317,229,008`). This is **not** a measurement of the model's real quality — it is a byproduct of scoring a tiny, deliberately adversarial 64-target diagnostic slice with the probe's own scoring convention, and is expected to look nothing like the model's real **10.90 PPL** validation score reported elsewhere on this card. The number that matters here is the **ratio** between the standard and honest variants (0.999996, i.e. effectively 1.0x inflation) and the **paired difference** (-3.8e-06 nats, noise-floor), not either value in isolation.

Tier-B (well-parameter) geometric probes are skipped for this checkpoint (`has_vtheta_wells: false`) — the structured-quadratic SQ3 potential does not expose the Gaussian-mixture `well_parameters()` interface Tier-B needs, a known, expected gap for this \\(V_\theta\\) family rather than an audit failure. Phase-4's register-usage diagnostics (entropy/diversity/creation-gate-strength per layer, verdict `MIXED`) are a **separate, non-leak diagnostic** — SCAF's mediation/diagnostic outputs never affect the `CLEAN`/`LEAK` verdict — included here for completeness rather than as evidence either way.

<p align="center"><img src="scaf_analysis_results/phase5_dashboard.png" alt="Six panel SCAF causal leak analysis dashboard for this checkpoint. Top left: a green scorecard listing every control and probe as PASS with the standard and honest PPL both around 19.3 billion. Top right: within window NLL profile bar chart, roughly flat around 24 nats across the 512 token window. Middle left: leak profile CATE by distance to cut, completely flat at zero across all distances with no spike anywhere. Middle right: register attention entropy per layer, declining from about 1.0 at layer 0 to about 0.5 at layer 7. Bottom left: register content diversity per layer, fluctuating between about 0.3 and 0.7 across the eight layers. Bottom right: creation gate attention strength per layer, rising from near zero at layer 0 to about 0.3 at layer 7." width="820"></p>

The middle-left panel (CATE by distance to cut) is the clearest single signal: a real leak produces a spike concentrated near distance zero that decays with distance (the "shared-register leak" profile SCAF's own documentation describes); here the profile is **flat at exactly zero across the entire window**, the expected picture for a genuinely causal model. Full raw output: `scaf_analysis_results/phase1_scaf_audit.json` through `phase6_estimation_report.json` in this repository.

## About the SCAF Library

This audit was produced with [**SCAF**](https://github.com/dimitarpg13/semsimula-scaf) (SemSimula Causal Auditing Framework), a standalone, model-agnostic library that resolves onto a checkpoint by structural (duck) typing — no base class or `Protocol` required. Its **causal audit** (on `main`, used above) detects, sizes, and attributes causal leaks via two complementary probes — `future_perturbation` (bit-exact: is there a leak at all?) and `target_relocation` (how many nats of perplexity were unearned?) — backed by three controls (`determinism`, `placebo`, `positive`) that gate the `CLEAN` verdict, a `mediation` knockout diagnostic for locating a confirmed leak, `scaf.LeakMonitor` for continuous training-time monitoring, and a DoWhy/EconML estimand bridge for exact paired-inference ATEs and CATE-by-distance heterogeneity profiles.

SCAF also has a second, independent audit family — a **stiffness audit** (`scaf.StiffnessProbe`, on the `stiffness_audit` branch, scheduled to merge into `main`) that checks a trained model's *numerical* integration stability rather than its causal structure. It is not applicable to this checkpoint's causal-audit results above, but is directly relevant to this architecture family: see [dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability) for a worked example where it identified a structural Störmer–Verlet instability in a sibling (anisotropic-Gaussian) \\(V_\theta\\) family.

## The Fock Paradox: Maximal Compression, Moderate Gap

Across the three SPLM-family architectures tested with structured \\(V_\theta\\), the Fock-PARFLM exhibits a striking paradox: the **flattest \\(V_\theta\\) landscape** yet the **second-largest expressivity gap**.

| Architecture | Structured V_theta PPL | MLP baseline PPL | Gap | Gap (%) | Mean V_theta | Range |
|---|---|---|---|---|---|---|
| [Multi-Xi SPLM (SQ3)](https://huggingface.co/dimitarpg13/semsimula-splm-multixi-structured-vtheta) | 13.33 | 11.51 | 1.82 | 5.5% | 99.8 | 644.9 |
| **Fock-PARFLM v2.1 (SQ3, this model)** | **10.90** | **9.70** | **1.20** | **12.4%** | **0.017** | **16.0** |
| [Multi-Xi PARFLM (SQ3)](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi-structured-vtheta) | 12.27 | 12.10 | 0.17 | 0.6% | 0.02 | 31.4 |

> Both Fock-PARFLM PPL values (structured and MLP) are now leak-free, from re-trained checkpoints with `prefix_causal_registers=True`. The SPLM and PARFLM values were always leak-free (no reverse channel). All cross-architecture comparisons in this table are now valid on a like-for-like (honest PPL) basis.

The landscape compression is monotonic (SPLM > PARFLM > Fock-PARFLM), but the expressivity gap is **non-monotonic**: PARFLM achieves the smallest gap despite a less compressed landscape than Fock-PARFLM. This ranking is unchanged after the leak fix — only the absolute PPL and gap magnitudes moved.

**Why?** Operating close to the dataset's entropy floor (9.70 PPL for the honest MLP baseline), the Fock model is where the marginal value of each nat of \\(V_\theta\\) precision is higher. The Fock register mechanism (creation/destruction operators, stack discipline, reverse channel) creates a more structured dynamical regime where even a near-flat \\(V_\theta\\) must provide fine-grained corrections that the diagonal quadratic parameterisation cannot match. The relationship between landscape compression and expressivity gap is therefore **architecture-dependent**, modulated by proximity to the entropy floor.

The **force** from the structured \\(V_\theta\\) is computed in closed form:

$$f_\theta = -\nabla_h V_\theta = -\sum_{k=1}^{K} q_k(\xi, h) \cdot a_k(\xi) \odot (h - \mu_k(\xi))$$

where \\(q_k\\) are the softmax responsibilities over the 8 quadratic wells. The \\(V_\phi\\) force still uses autograd (sparse, over top-k=8 neighbours only), and the Fock forces use their own differentiable computation.

For full derivations, all four structured variants (SQ1--SQ4), landscape compression analysis, attractor basin decoding, and hyperparameter selection strategies, see the companion note: [`Structured_VTheta_Design_and_Theory.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Structured_VTheta_Design_and_Theory.md).

## How to Get Started

```python
import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_structured_vtheta import MixtureQuadraticVTheta
from parf.model_structured_vtheta_multixi import StructuredVThetaMultiXiAdapter

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, L=8,
    v_hidden=1024, v_depth=3,
    max_len=1024,
    fixed_gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16, d_k=64,
    tau_create_init=8.0,
    register_salience_decay=0.5,
    register_salience_threshold=0.01,
    stack_discipline=True,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,  # leak fix — must be True to reproduce this checkpoint
)
model = FockMultiXiPARFLM(config)

# -- Swap in structured V_theta --
K_xi, d = 4, 256
inner = MixtureQuadraticVTheta(d=d, K=8, tau=1.0, xi_d=K_xi * d)
model.V_theta = StructuredVThetaMultiXiAdapter(inner, K=K_xi, d=d)

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-structured-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                        # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)      # (1, 64, 8, 1024)
    print(f"Attractor centres shape: {centres.shape}")   # 8 basins per token
```

### Available Artifacts

| File | Description |
|---|---|
| `checkpoint/ckpt_best.pt` | Best checkpoint, leak-fixed (A2 arm, `prefix_causal_registers=True`, 10.90 PPL at step 14,400) |
| `causal_leak_fix_verification.md` | Paired leaky-vs-fixed rerun: bit-exact future-perturbation probe, honest-vs-standard PPL, parameter-count correction |
| `scaf_analysis_results/phase1_scaf_audit.json` | Formal SCAF Phase-1 audit: verdict, controls, `future_perturbation`/`target_relocation` probes |
| `scaf_analysis_results/phase1_5_geometric_probes.json` | Tier-A hidden-state leak probe (per-layer cosine deviation) |
| `scaf_analysis_results/phase2_ate_result.json`, `phase2_leak_frame.csv` | Phase-2 do-future ATE estimate and the underlying per-position leak frame |
| `scaf_analysis_results/phase2_5_geometric_leak_frame.csv` | Geometric-distance leak frame (Tier-A companion dataset) |
| `scaf_analysis_results/phase3_legacy_probe.json` | Legacy architectural causal probe (bit-exact `max_dlogit_past`/`mean_dnll_past`) |
| `scaf_analysis_results/phase4_register_diagnostics.json` | Per-layer register entropy/diversity/creation-gate diagnostics (not part of the leak verdict) |
| `scaf_analysis_results/phase5_ate_bar.png`, `phase5_dashboard.png` | SCAF scorecard dashboard and ATE bar chart (see [SCAF Causal-Leak Audit](#scaf-causal-leak-audit-formal)) |
| `scaf_analysis_results/phase6_cate_by_distance.png`, `phase6_estimation_report.json` | CATE-by-distance-to-cut profile and DoWhy/EconML estimation cross-check |
| `training_log.jsonl` | Per-step training metrics (40 eval points), including `causal_probe` and `trained_leak_probe` events |
| `training_curve_A2.png` | Training/validation loss curves |
| `v_theta_hist_A2.png` | V_theta output distribution histogram |
| `landscape_stats_A2.json` | V_theta landscape statistics (mean, std, range), recomputed on the leak-fixed checkpoint |
| `model_structured_vtheta.py` | Structured V_theta classes (SQ1--SQ4) |
| `model_structured_vtheta_multixi.py` | Multi-Xi adapter |
| `config.json` | Model configuration |

## Training Details

### Training Data

[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.

### Training Procedure

The base model architecture is identical to the [Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm). The only modification is the \\(V_\theta\\) replacement: the 3-layer MLP is swapped for the SQ3 mixture at model construction time, before training begins from scratch. The pairwise \\(V_\phi\\) (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged.

| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | 5e-4 (cosine decay) |
| Warmup steps | 400 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Batch size | 16 |
| Block size | 512 |
| Training steps | 16,000 |
| lambda_V (V_theta regularisation) | 0.01 |
| Hardware | NVIDIA A100 40GB (Google Colab) |

### Training Script

[`notebooks/conservative_arch/scaleup/colab_fock_multixi_structured_vtheta.ipynb`](https://github.com/dimitarpg13/semsimula-paper/blob/main/notebooks/conservative_arch/scaleup/colab_fock_multixi_structured_vtheta.ipynb) --- Colab notebook with structured V_theta arms, GDrive output, checkpointing, and live progress display.

## Evaluation Results

### TinyStories Validation Perplexity

| Model | PPL | Params | Analytical V_theta grad | V_theta--MLP gap | Causal leak |
|---|---|---|---|---|---|
| Matched Attention (baseline) | **7.81** | 19.5M | --- | --- | No |
| [Fock-PARFLM v2.1 (MLP)](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) | **9.70** | 17.4M | No | --- | Fixed & verified |
| **Fock-PARFLM v2.1 (SQ3, this model)** | **10.90** | 18.2M | **Yes** | **1.20 PPL** | Fixed & verified |
| [Multi-Xi SPLM (MLP)](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) | **11.51** | 16.5M | No | --- | No |
| [Multi-Xi PARFLM (MLP)](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) | **12.06** | 17.6M | No | --- | No |
| [Multi-Xi PARFLM (SQ3)](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi-structured-vtheta) | **12.27** | 17.3M | Yes | 0.17 PPL | No |
| [Multi-Xi SPLM (SQ3)](https://huggingface.co/dimitarpg13/semsimula-splm-multixi-structured-vtheta) | **13.33** | 17.3M | Yes | 1.82 PPL | No |

> Both Fock-PARFLM models (MLP and SQ3) were re-trained with `prefix_causal_registers=True` and independently certified leak-free (bit-exact 0.0 future-perturbation sensitivity). All PPL values in this table are now on a like-for-like, honest basis. Fock-PARFLM (SQ3) still beats both Multi-Xi models, but its edge over the honest MLP-\\(V_\theta\\) Fock sibling (9.70) is now a 1.20 PPL gap rather than the 1.06 PPL gap reported under the leaky architecture.

### V_theta Landscape Statistics

| Metric | This model (Fock-PARFLM) | PARFLM structured V_theta | SPLM structured V_theta |
|---|---|---|---|
| Mean V_theta | 0.017 | 0.02 | 99.8 |
| Std V_theta | 0.42 | 0.51 | 26.3 |
| Range | 16.0 | 31.4 | 644.9 |

*Recomputed on the leak-fixed checkpoint (previously: mean 0.008, std 0.26, range 19.1 on the leaky checkpoint). Both runs show the same near-flat landscape; exact values differ because the fixed run is an independently-trained model with different weights.*

### Learned Xi-Channel Decay Rates

The final learned alpha values \\([\alpha_1, \ldots, \alpha_4] = [0.14, 0.56, 0.79, 0.95]\\) are stable and consistent with the SPLM \\([0.12, 0.59, 0.84, 0.97]\\) and PARFLM \\([0.11, 0.55, 0.81, 0.97]\\) values, confirming that the causal EMA context structure is invariant to the \\(V_\theta\\) parameterisation, the presence of \\(V_\phi\\), and the Fock dynamics.

## SPLM Family Overview

This model is part of the **Semantic Simulation SPLM family**:

| Model | Design | Corpus | PPL | HuggingFace |
|---|---|---|---|---|
| Multi-Xi SPLM (MLP) | Pure scalar potential | TinyStories | 11.51 | [semsimula-splm-multixi](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) |
| Multi-Xi SPLM (SQ3) | Structured scalar potential | TinyStories | 13.33 | [semsimula-splm-multixi-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-splm-multixi-structured-vtheta) |
| Multi-Xi PARFLM (MLP) | Scalar + pairwise forces | TinyStories | 12.06 | [semsimula-parflm-multixi](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) |
| Multi-Xi PARFLM (SQ3) | Structured scalar + pairwise | TinyStories | 12.27 | [semsimula-parflm-multixi-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi-structured-vtheta) |
| Fock-PARFLM v2.1 (MLP) | PARFLM + Fock registers | TinyStories | 9.70 | [semsimula-fock-parflm](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) |
| **Fock-PARFLM v2.1 (SQ3)** | **Structured + pairwise + Fock** | **TinyStories** | **10.90** | **this model** |
| Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian) | Bounded multi-context + pairwise + Fock | TinyStories | 16.33 | [semsimula-fock-parflm-depthcond-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta) |
| Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg) | Bounded, ellipsoidal multi-context + pairwise + Fock | TinyStories | 9.04 | [semsimula-fock-parflm-anisogaussian-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1) | Same as above, gradient-flow integrator | TinyStories | 8.95 | [semsimula-fock-parflm-anisogaussian-vtheta-fock-g1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-fock-g1) |
| Fock Attention (MLP V_theta) | Fock + attention | TinyStories | 9.42 | [semsimula-fock-attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) |
| Hybrid SPLM+Attn | Attention + SPLM refinement | TinyStories | 8.50 | [semsimula-hybrid-splm](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 278.27 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 326.97 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=1024) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 244.23 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, Verlet instability, d=384) | Same architecture, two full-run attempts — SCAF stiffness audit identifies structural Verlet instability | OpenWebText | 184.11 / 211.63 (both runs stalled, not final) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability) |

**Collection:** [Semantic Simulation SPLM Model Family](https://huggingface.co/collections/dimitarpg13/semantic-simulation-splm-model-family)

## Bias, Risks, and Limitations

- **Research checkpoint only.** This model is a proof-of-concept for structured scalar potentials in Fock-augmented architectures, not a production system.
- **TinyStories only.** Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
- **English only.** No multilingual capability.
- **Small scale.** 18.2M parameters, 256-dim hidden states.
- **No safety training.** No RLHF, DPO, or safety filtering has been applied.
- **V_phi and Fock forces still use autograd.** Only the \\(V_\theta\\) gradient is analytical; the \\(V_\phi\\) pairwise force and Fock register forces still require `torch.autograd.grad`. The overall training speedup is therefore partial (\\(V_\phi\\) and Fock forces dominate the cost).
- **Moderate expressivity gap.** The 1.20 PPL gap (12.4% excess CE) is larger than the PARFLM structured V_theta gap (0.17 PPL), reflecting the higher precision demands of the Fock architecture near the dataset's entropy floor.
- **Causal leak in the reverse channel — resolved.** The Fock reverse channel previously introduced a causal leak by blending each token's content into a global register state shared across all sequence positions within the same integration step. On the OpenWebText-scale variant of the same architecture, a causal probe measured +3.51 nats of NLL inflation (~33× PPL factor) from this pathway. **This checkpoint was re-trained with the architectural fix (`prefix_causal_registers=True`)** and certified leak-free via a bit-exact (0.0, float64) future-perturbation probe and honest-vs-standard PPL agreement within noise. The fix cost a modest **+0.54 PPL** relative to a like-for-like leaky rerun (10.36 → 10.90). See the [full audit](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock-PARFLM_Causal_Leak_Audit_Results.md), this repo's [`causal_leak_fix_verification.md`](causal_leak_fix_verification.md), and the formal [SCAF Causal-Leak Audit](#scaf-causal-leak-audit-formal) section above (verdict `CLEAN`, all controls and probes passing).

## Citation

```bibtex
@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}
```

## Environmental Impact

- **Hardware:** NVIDIA A100 40GB (Google Colab)
- **Training time:** ≈3 hours for the original leaky run (16,000 steps, A2 arm). The re-trained, leak-fixed checkpoint on this page took noticeably longer — checkpoint timestamps for the final segment (steps 14,400→16,000) imply ≈4.7s/step, i.e. a full run on the order of ≈20 hours, a larger relative slowdown than the ≈3× seen on the MLP-\\(V_\theta\\) sibling (per-position register bookkeeping is a bigger fraction of a lighter, analytical-gradient step).
- **Carbon footprint:** Estimated less than 10 kg CO2 for the leak-fixed rerun (was <2 kg CO2 for the original leaky run).

> **Correction (2026-09-27) — integrator name.** This model's layer step was described here as a *damped Euler* step. It is not. The update is
> `h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma))`, which carries no velocity state at all: the velocity is the position difference `h - h_prev`. Undamped this is `h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m`, i.e. **Störmer–Verlet in position form**. It is also not *velocity*-Verlet, which carries an explicit velocity through half-kick/drift/half-kick.
> The distinction matters because the \\(\omega \cdot dt < 2\\) stability wall is the Störmer/leapfrog bound and applies to this lineage; the semi-implicit Euler models in the same collection were never subject to it. Only the name was wrong — no measurement on this card changes.

> **Correction (2026-09-28) — this model is NOT purely conservative.** Its `config.json` sets `use_reverse_channel: true`, and the reverse channel — the register-to-token path — is not the gradient of anything in \\(h\\). Later measurement on this architecture puts the deflection of the layer step from the damped \\(V_\theta\\) geodesic at **1.09 with the reverse channel on and 0.0003 with it off**, i.e. the non-gradient term carries most of the step. The conservative-by-construction claim, and the Riemannian capabilities that were derived from it, hold for the **conservative sector** \\(V_\theta + V_\varphi\\) and for siblings trained without the reverse channel — not for this checkpoint as shipped. Measurements on this card are unaffected; only the framing was wrong.