mchammerr commited on
Commit
c86d23c
Β·
verified Β·
1 Parent(s): bd954c5

Remove architecture mechanism disclosure; link Z-Next results page + live demo

Browse files

Two changes:

1. IP: the card described the mechanism (state formulation, gating, and a per-component parameter breakdown naming each functional part). Replaced with grouped, non-identifying wording that preserves every measurement and the SFT conclusion.

2. Added links to the current generation: Z-Next 1B results page and the live public demo.

Files changed (1) hide show
  1. README.md +21 -15
README.md CHANGED
@@ -13,6 +13,12 @@ extra_gated_prompt: "Weights for this model are not published. This repository c
13
 
14
  # Tetracta-zkas-1B β€” results card (no weights)
15
 
 
 
 
 
 
 
16
  **This repository publishes measurements, not weights.**
17
 
18
  `tetracta_zkas_1B` is the **twin** of [Tetracta VAN-1B](https://huggingface.co/tetracta/tetracta-van-1b):
@@ -20,7 +26,7 @@ it saw **bit-identical pre-training data** (same corpus, same order, verified) a
20
  instruction-tuning run** (same 11,419 examples, same recipe, same seed). The only difference is the architecture.
21
 
22
  That makes it a controlled single-variable experiment, and it is the reason our central claim about SFT is not
23
- a one-model anecdote. **All of those results are below.** The architecture's equations, pole parameterisation,
24
  initialisation recipe and stability rules are **Tetracta's closed research work and are not published.**
25
 
26
  - 🌐 **[tetracta.ai](https://tetracta.ai)** Β· πŸ“„ **Paper:** *"What does SFT do?"* β†’ [tetracta.ai](https://tetracta.ai)
@@ -35,14 +41,14 @@ initialisation recipe and stability rules are **Tetracta's closed research work
35
  | | |
36
  |---|---|
37
  | Type | **Not a transformer.** No attention. |
38
- | Mechanism | Ξ³-pole **recurrent state** with a learned decay spectrum + a **multiplicative saturation gate** |
39
  | Inference | State size is **independent of sequence length** (O(1) per step) |
40
  | Parameters | **953,466,904** |
41
  | Pre-training | **315,000 steps Β· 10,321,920,000 tokens Β· one epoch**, bit-identical to VAN-1B |
42
  | Instruction tuning | 11,419 examples Β· 3 epochs Β· lr 1e-5 Β· batch 16 Β· 2,142 steps Β· 1,247 s (1Γ—H200) |
43
  | Stability | **0 skipped steps out of 315,000** during pre-training; 0 during SFT |
44
 
45
- Not disclosed: equations, pole parameterisation, initialisation recipe, MFU/parallelisation
46
  recipe, stability laws, energy-distribution findings.
47
 
48
  ---
@@ -53,7 +59,7 @@ Our claim from the VAN paper: **SFT does not add knowledge; it tunes the output
53
  If that is true, two very different architectures given the same data and the same SFT should end up at the
54
  same *interface* state even from different starting points. They do:
55
 
56
- | Measurement | VAN-1B (transformer) | **zkas-1B** (Ξ³-pole) |
57
  |---|---|---|
58
  | Base output entropy | 5.951 | 7.882 β€” *very different* |
59
  | **Post-SFT output entropy** | 1.795 | **1.624** β€” same order |
@@ -74,19 +80,19 @@ This is the cleanest evidence we have, and it is only visible here. In a transfo
74
  "memory" from "processing" β€” attention weights carry both. In this architecture components separate by function,
75
  so we can read directly where SFT went:
76
 
77
- | Functional component | Ξ΄ |
78
  |---|---|
79
- | **position/phase channel β€” output projection** | **18.72 %** ← largest move, 3.5Γ— the embedding |
80
  | embedding / output head (tied) | 5.327 % |
81
- | saturation gain | 2.560 % |
82
- | mixing projections (in / out) | 1.299 % / 1.119 % |
83
- | position/phase channel β€” input | 0.551 % |
84
- | pole coefficients | 0.210 % / 0.070 % |
85
- | **β˜… decay spectrum (the time memory)** | **0.00009 %** ← **frozen** |
86
 
87
  ![Where SFT went](fig2-where-sft-went.png)
88
 
89
- **SFT moved the memory by roughly one part in a million.** It rewired the position/phase output instead.
90
  *"SFT changes how the model speaks, not what it knows"* is, in this architecture, a **structural** statement
91
  rather than an inference.
92
 
@@ -98,8 +104,8 @@ rather than an inference.
98
  | **Alignment tax** | **+11.5 %** | **+35.9 %** |
99
 
100
  Same data, same recipe, **3Γ— the cost**. Our pre-registered prediction was "~11 % again" β€” that was a **miss**,
101
- and it is recorded as one. Likely mechanism: the position/phase channel participates in processing *every* token,
102
- so moving its output by 18.7 % perturbs the plain-text distribution more. *(Untested hypothesis. Testable
103
  prediction: giving that projection its own lower learning rate should reduce the tax.)*
104
 
105
  ## Result 4 β€” blind judge panel
@@ -173,7 +179,7 @@ There are no weights here, so there is nothing to license as a model. What *is*
173
  quote it, replot it, argue with it β€” just cite it. Attribution is the whole point; these numbers exist so that
174
  someone else can check whether our claim about SFT holds up.
175
 
176
- **Not covered by this licence, because it is not published:** the architecture's equations, pole
177
  parameterisation, initialisation recipe, parallelisation recipe and stability rules.
178
 
179
  ## Citation
 
13
 
14
  # Tetracta-zkas-1B β€” results card (no weights)
15
 
16
+ > **Newer work:** this card documents our 2025 architecture. The current generation β€”
17
+ > **Z-Next 1B**, no attention and no KV cache, with a session state that does not grow
18
+ > with context β€” has its own results page and a **live public demo you can talk to**:
19
+ > Β· [Z-Next 1B results & benchmarks](https://www.tetracta.ai/znext.html)
20
+ > Β· [**Live demo β†’**](https://www.tetracta.ai/zchat)
21
+
22
  **This repository publishes measurements, not weights.**
23
 
24
  `tetracta_zkas_1B` is the **twin** of [Tetracta VAN-1B](https://huggingface.co/tetracta/tetracta-van-1b):
 
26
  instruction-tuning run** (same 11,419 examples, same recipe, same seed). The only difference is the architecture.
27
 
28
  That makes it a controlled single-variable experiment, and it is the reason our central claim about SFT is not
29
+ a one-model anecdote. **All of those results are below.** The architecture's equations, parameterisation,
30
  initialisation recipe and stability rules are **Tetracta's closed research work and are not published.**
31
 
32
  - 🌐 **[tetracta.ai](https://tetracta.ai)** Β· πŸ“„ **Paper:** *"What does SFT do?"* β†’ [tetracta.ai](https://tetracta.ai)
 
41
  | | |
42
  |---|---|
43
  | Type | **Not a transformer.** No attention. |
44
+ | Mechanism | **Recurrent, constant-size state.** The mechanism itself is not disclosed. |
45
  | Inference | State size is **independent of sequence length** (O(1) per step) |
46
  | Parameters | **953,466,904** |
47
  | Pre-training | **315,000 steps Β· 10,321,920,000 tokens Β· one epoch**, bit-identical to VAN-1B |
48
  | Instruction tuning | 11,419 examples Β· 3 epochs Β· lr 1e-5 Β· batch 16 Β· 2,142 steps Β· 1,247 s (1Γ—H200) |
49
  | Stability | **0 skipped steps out of 315,000** during pre-training; 0 during SFT |
50
 
51
+ Not disclosed: equations, parameterisation, initialisation recipe, MFU/parallelisation
52
  recipe, stability laws, energy-distribution findings.
53
 
54
  ---
 
59
  If that is true, two very different architectures given the same data and the same SFT should end up at the
60
  same *interface* state even from different starting points. They do:
61
 
62
+ | Measurement | VAN-1B (transformer) | **zkas-1B** |
63
  |---|---|---|
64
  | Base output entropy | 5.951 | 7.882 β€” *very different* |
65
  | **Post-SFT output entropy** | 1.795 | **1.624** β€” same order |
 
80
  "memory" from "processing" β€” attention weights carry both. In this architecture components separate by function,
81
  so we can read directly where SFT went:
82
 
83
+ | Group of parameters | relative change (Ξ΄) |
84
  |---|---|
85
+ | the part that shapes **how the model speaks** | **18.72 %** ← largest move, 3.5Γ— the embedding |
86
  | embedding / output head (tied) | 5.327 % |
87
+ | everything else in the processing path | 0.07 % – 2.56 % |
88
+ | **β˜… the part that holds the model's memory over time** | **0.00009 %** ← **frozen** |
89
+
90
+ *(Component names and the full per-part breakdown are not disclosed; the grouping above
91
+ is what the measurement supports without describing the architecture.)*
92
 
93
  ![Where SFT went](fig2-where-sft-went.png)
94
 
95
+ **SFT moved the memory by roughly one part in a million.** It rewired the output-shaping path instead.
96
  *"SFT changes how the model speaks, not what it knows"* is, in this architecture, a **structural** statement
97
  rather than an inference.
98
 
 
104
  | **Alignment tax** | **+11.5 %** | **+35.9 %** |
105
 
106
  Same data, same recipe, **3Γ— the cost**. Our pre-registered prediction was "~11 % again" β€” that was a **miss**,
107
+ and it is recorded as one. Likely reason: the part that moved most participates in processing *every* token,
108
+ so perturbing it by 18.7 % disturbs the plain-text distribution more. *(Untested hypothesis. Testable
109
  prediction: giving that projection its own lower learning rate should reduce the tax.)*
110
 
111
  ## Result 4 β€” blind judge panel
 
179
  quote it, replot it, argue with it β€” just cite it. Attribution is the whole point; these numbers exist so that
180
  someone else can check whether our claim about SFT holds up.
181
 
182
+ **Not covered by this licence, because it is not published:** the architecture's equations,
183
  parameterisation, initialisation recipe, parallelisation recipe and stability rules.
184
 
185
  ## Citation