Papers
arxiv:2611.26450

A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights

Published on Nov 19
· Submitted by
Tanmay Sah
on Nov 20
Authors:

Abstract

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid (>250times dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension (d to d+K) with d/(d+K) norm compensation, Base-1000 canonical BPE emission, and differential (+254,-256) tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed {+,-,times} circuits in constant TC^0 depth alongside 3-literal Boolean (100%, N=1{,}728; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic (N=1{,}200, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, le 0.036%) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, le 0.057%) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered >2K-token Tango splits (N=4{,}392; 0/11,134 OOD triggers).

Community

Paper author Paper submitter

Why teach an LLM arithmetic when you can compile arithmetic directly into its weights?

TL;DR: From learning weights to programming weights. We introduce closed-form, zero-gradient compilation primitives that embed deterministic computation directly into pretrained LLMs, demonstrating signed multi-digit arithmetic (+, -, ×) and automated compilation of Boolean, Majority, and Parity circuits across Qwen 3.5 4B, IBM Granite 4.2 3B, and architectural audits on Gemma 4 E4B.

  • Zero gradient updates. Zero training examples. Zero external tools. Computation is encoded directly into model weights and executed through unmodified model.generate().
  • Exact logic across >250× RMSNorm swings. Two-layer Mellin-GLU quadrature transforms unknown per-token normalization scaling into translation along a log-spaced gate grid, with a calibration-free variant.
  • Attention-free circuit routing on Qwen 3.5. Zero of eight full-attention layers modified. Cross-token operands are routed entirely through GatedDeltaNet blocks.
  • Less than 0.06% additional active weights. Compatible with INT4/INT8 quantized backbones and a compact FP32 circuit sidecar.

Key results:

  • 100% evaluated ALU-prefix accuracy with FP32 circuit execution.
  • 100% evaluated Boolean, Majority, and Parity accuracy.
  • 98–100% fluent inline continuation, versus 0% for 12.8K-example LoRA fine-tuning.
  • Zero false circuit triggers across 11,134 out-of-domain inputs.
  • Natural-language noninferiority across all 14 preregistered evaluation splits.
  • INT4 deployment with just 4.6 GiB / 8.1 GiB VRAM for Granite / Qwen.

What if adding new capabilities to neural networks didn't always require training them?

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.26450
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.26450 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.26450 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.26450 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.