entity2260/cl-pncm-tiny-124m

C-LPNCM: Calibrated Looped-Pointer Next-Concept Model

C-LPNCM (Calibrated Looped-Pointer Next-Concept Model) is a 100% pure-Transformer Next-Concept Modeling architecture (zero diffusion layers, zero ODE/flow-matching samplers) that shifts sequence generation from flat surface-token autoregression to hierarchical discrete concept planning, weight-tied fixed-point equilibrium refinement, and dual-path pointer-concept surface decoding.

---

1. Architecture Overview (Four-Tower Pure Transformer)

Raw Surface Tokens w_{1:T}
   │
   ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ MODULE 1: Entropy-Gated Multiscale RVQ-FSQ Encoder                         │
│ • Causal Local-Window Token Transformer (W=128) → Unquantized (K_raw,V_raw)│
│ • Shannon Entropy Spike Detector → Variable-Length Concept Spans (S_m ∈ 1..8)│
│ • Intra-Patch Bidirectional Self-Attention + Query Pooler → u_m            │
│ • Level-1 Cosine VQ Codebook (z^(1)) + Cascaded FSQ Residuals (z^(2..L))   │
└──────────────┬─────────────────────────────────────────────────────────────┘
               │ Coarse Concepts z^(1)_{1:M}  (1/4x Sequence Compression)
               ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ MODULE 2: Calibrated Latent Next-Concept Core                              │
│ • Causal Concept Transformer with <BOS_CONCEPT> Shift                      │
│ • Policy Head π_θ(z^(1)_{m+1}) + Dynamic Span-Length Head P_len(S_{m+1})   │
│ • Brier-Calibrated Epistemic Confidence Head q_φ(p_conf ∈ [0, 1])          │
│ • Deliberative Value Verifier Head V_verify & 3-Way Epistemic Router:      │
│   ├─ Fast Path (p_conf ≥ τ_high)         → K* = 1..2 Refiner Loops         │
│   ├─ Deliberative Path (τ_low ≤ p < τ_h) → Top-B Beam Rollout + K_max Loops│
│   └─ Epistemic Abstention (p < τ_low)    → Emit <ABSTAIN_OR_RETRIEVE>      │
└──────────────┬─────────────────────────────────────────────────────────────┘
               │ Chosen Concept z^(1)_{m+1}, Span Length S_{m+1}, Context H_core
               ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ MODULE 3: Adaptive Weight-Tied Looped Transformer Refiner                  │
│ • Span-Query Upsampler U_0 = W_up(z^(1)) + Q_pos[1 : S_{m+1}]              │
│ • Anchor-Injected Weight-Tied Equilibrium Loop (k = 1..K_max):             │
│     H^(k) = f_θ(RMSNorm(H^(k-1) + W_inj U_0), CrossAttn(H_core, ≤ m))      │
│ • Guarded Exclusive Self-Attention (XSA) + Head-Wise Sigmoid Gate          │
│ • Strict Bottleneck Cascaded FSQ Projection → Quantized Pyramid E_quant    │
└──────────────┬─────────────────────────────────────────────────────────────┘
               │ Quantized E_quant (Path A) & Continuous H^(K*) (Path B Query)
               ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ MODULE 4: Dual-Path Transformer Surface Decoder & Boundary Refiner         │
│ • Path A: Parallel Concept Transformer Decoder over E_quant → P_concept    │
│ • Path B: Strictly Causal Prior-Span Pointer over (K_raw, V_raw) → P_copy  │
│ • Dynamic Copy Gate g_i ∈ [0, 1] with 50% Stage-1 Pointer Dropout          │
│ • Neuro-Symbolic Prefix-Tree (Trie) Grammar/Schema Bitmask                 │
│ • Confidence-Gated 1-Pass Transformer Boundary Refiner → Surface Tokens    │
└────────────────────────────────────────────────────────────────────────────┘

Key Architectural Innovations

1. Zero Diffusion / Pure Next-Concept Transformer: All continuous diffusion noise schedules, timestep embeddings, and masked diffusion steps are eliminated in favor of discrete multiscale quantization (CosineVQ + FSQ) and parallel Transformer decoding.
2. Leak-Free Dual-Path Surface Decoding:
   • Path A (Concept Vocabulary Decoder) receives only quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^L \hat{z}^{(\ell)}$, forcing the RVQ-FSQ hierarchy to carry complete phrase structure.
   • Path B (Causal Prior-Span Pointer) queries the unquantized $(K_{\text{raw}}, V_{\text{raw}})$ memory bank strictly at token indices $j < t_{\text{start}}(m)$ (with a learnable <NULL_PTR> key at index 0), enabling exact copy of rare names and entities (e.g., "Tom", "Mr. Cuddles") without self-copy identity leakage.
3. Guarded Exclusive Self-Attention (XSA) & Banach Contraction Loops: Module 3 iterates a weight-tied Transformer block with anchor re-injection ($U_0$), multi-exit FSQ deep supervision, and a detached-reference Banach contraction penalty $\max(0, \Delta^{(k)} - \rho \cdot \text{sg}[\Delta^{(k-1)}])$.
4. Strictly Proper Brier Epistemic Calibration: Module 2 detaches backbone features before the calibration head $q_\phi$ and optimizes the strictly proper Brier score $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$, ensuring honest epistemic uncertainty estimates.

---

2. Model Variants & Specifications

Model Variant	Config File	Total Params	$d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$	Layers (Enc / Core / Loop / Dec)	Coarse VQ Codes	FSQ Bins	Target Hardware
C-LPNCM-124M (Colab)	configs/cl_pncm_colab.yml	123.6M	512 / 768 / 512	4 / 8 / 3 (×6 loops) / 4	4,096	[8, 8, 8, 6, 5]	1× Tesla T4 (16 GB)
C-LPNCM-660M (Base)	CLPNCMConfig.base()	659.6M	1024 / 1536 / 1024	4 / 12 / 4 (×6 loops) / 4	4,096	[8, 8, 8, 6, 5]	1× L4 / A100
C-LPNCM-1B (A100)	configs/cl_pncm_1b_a100.yml	1.04B	1024 / 2048 / 1024	6 / 12 / 4 (×6 loops) / 6	16,384	[8, 8, 8, 6, 5]	1× A100 (40/80 GB)
C-LPNCM-8B (Prod)	configs/cl_pncm_8b.yml	7.85B	2048 / 4096 / 2048	6 / 24 / 4 (×8 loops) / 6	16,384	[8, 8, 8, 6, 5]	4×–8× A100/H100

---

3. Three-Stage Training Curriculum

1. Stage 1 — Multiscale RVQ-FSQ + Pointer Autoencoding Warmup: Trains Module 1 (Encoder + Quantizer), Module 3 (Looped Refiner), and Module 4 (Dual-Path Decoder) end-to-end with surface token cross-entropy, unit-sphere Cosine VQ commitment loss, copy-gate supervision, and causal entropy-head prediction.
2. Stage 2 — Coarse Latent Next-Concept Pre-Training + Equilibrium Contraction: Freezes Module 1 and Module 4 (vq_to_core coordinate space locked) and trains Module 2 (CalibratedLatentCore) and Module 3 (AdaptiveLoopedRefiner) with next-concept cross-entropy, span-length cross-entropy, multi-exit FSQ deep supervision, Banach contraction penalty, Brier calibration warmup, and surface decoder NLL anchoring.
3. Stage 3 — Reinforcement through Calibrated Decisions (Latent GRPO): Optimizes concept rollouts via Group-Relative Policy Optimization ($\mathcal{L}_{\text{Cal-GRPO}}$), combining verifiable task/continuation rewards, strictly proper Brier calibration loss, differentiable ACT ponder penalty, and KL regularization.

---

4. Training Metrics & Qualitative Example (124M TinyStories Warmup)

Trained on hazyresearch/ncm-datasets (tinystories/train_bpe_8192.bin, 542.7M tokens, custom BPE vocabulary $V=8{,}192$):

• Stage 1 Surface Token Reconstruction CE Loss (1,000 steps): 9.0726 → 1.2899
• Stage 1 Auxiliary Entropy Head Loss (1,000 steps): 9.1262 → 4.1266
• Stage 2 Coarse Next-Concept CE Loss: 8.5003 → 6.1182
• Stage 2 Multi-Exit FSQ Equilibrium Loss: 28.0830 → 2.7818
• Stage 2 Banach Contraction Penalty: 0.0105 → 0.0014

Qualitative Output (1,000-Step Warmup Checkpoint)

Stage	Text Output
Input Prompt (16 tokens)	Once upon a time, there was a boy named Tom and
Concept-Bottleneck Reconstruction	Once upon a time, there was a boy very Tom and (15/16 tokens exact, proper noun "Tom" preserved)
Generated Next-Concept Continuation	He was happy then found laughing and (6 concepts → variable span lengths [1, 1, 2, 1, 2, 1])

---

5. How to Load & Use

Option A: Load via Hugging Face AutoModel (trust_remote_code=True)

import torch
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained("your-username/cl-pncm-tinystories-124m", trust_remote_code=True).to(device)
model.eval()

# Sample prompt token IDs [B, T]
prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)

print("Predicted Coarse Concept IDs:", gen_out["route_out"]["chosen_z1"])
print("Predicted Span Lengths:", gen_out["route_out"]["pred_span_lens"])
print("Generated Token IDs:", gen_out["tokens"])

Option B: Load Merged Checkpoint Directly with CLPNCM

import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig

REPO_ID = "your-username/cl-pncm-tinystories-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"

ckpt_file = hf_hub_download(repo_id=REPO_ID, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=REPO_ID, filename="bpe_vocab_8192.pickle")

ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
    bpe_vocab = pickle.load(f)

cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()

---

6. Citation

@misc{clpncm2026,
  title   = {Calibrated Looped-Pointer Next-Concept Model (C-LPNCM): Pure-Transformer Hierarchical Concept Generation with Equilibrium Refinement},
  year    = {2026},
  note    = {Built on Next-Concept Modeling (Kumbong et al., arXiv:2610.14162) and Looped Transformer Equilibrium Refinement}
}