Remove architecture mechanism disclosure; link Z-Next results page + live demo
Browse filesTwo changes:
1. IP: the card described the mechanism (state formulation, gating, and a per-component parameter breakdown naming each functional part). Replaced with grouped, non-identifying wording that preserves every measurement and the SFT conclusion.
2. Added links to the current generation: Z-Next 1B results page and the live public demo.
README.md
CHANGED
|
@@ -13,6 +13,12 @@ extra_gated_prompt: "Weights for this model are not published. This repository c
|
|
| 13 |
|
| 14 |
# Tetracta-zkas-1B β results card (no weights)
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
**This repository publishes measurements, not weights.**
|
| 17 |
|
| 18 |
`tetracta_zkas_1B` is the **twin** of [Tetracta VAN-1B](https://huggingface.co/tetracta/tetracta-van-1b):
|
|
@@ -20,7 +26,7 @@ it saw **bit-identical pre-training data** (same corpus, same order, verified) a
|
|
| 20 |
instruction-tuning run** (same 11,419 examples, same recipe, same seed). The only difference is the architecture.
|
| 21 |
|
| 22 |
That makes it a controlled single-variable experiment, and it is the reason our central claim about SFT is not
|
| 23 |
-
a one-model anecdote. **All of those results are below.** The architecture's equations,
|
| 24 |
initialisation recipe and stability rules are **Tetracta's closed research work and are not published.**
|
| 25 |
|
| 26 |
- π **[tetracta.ai](https://tetracta.ai)** Β· π **Paper:** *"What does SFT do?"* β [tetracta.ai](https://tetracta.ai)
|
|
@@ -35,14 +41,14 @@ initialisation recipe and stability rules are **Tetracta's closed research work
|
|
| 35 |
| | |
|
| 36 |
|---|---|
|
| 37 |
| Type | **Not a transformer.** No attention. |
|
| 38 |
-
| Mechanism |
|
| 39 |
| Inference | State size is **independent of sequence length** (O(1) per step) |
|
| 40 |
| Parameters | **953,466,904** |
|
| 41 |
| Pre-training | **315,000 steps Β· 10,321,920,000 tokens Β· one epoch**, bit-identical to VAN-1B |
|
| 42 |
| Instruction tuning | 11,419 examples Β· 3 epochs Β· lr 1e-5 Β· batch 16 Β· 2,142 steps Β· 1,247 s (1ΓH200) |
|
| 43 |
| Stability | **0 skipped steps out of 315,000** during pre-training; 0 during SFT |
|
| 44 |
|
| 45 |
-
Not disclosed: equations,
|
| 46 |
recipe, stability laws, energy-distribution findings.
|
| 47 |
|
| 48 |
---
|
|
@@ -53,7 +59,7 @@ Our claim from the VAN paper: **SFT does not add knowledge; it tunes the output
|
|
| 53 |
If that is true, two very different architectures given the same data and the same SFT should end up at the
|
| 54 |
same *interface* state even from different starting points. They do:
|
| 55 |
|
| 56 |
-
| Measurement | VAN-1B (transformer) | **zkas-1B**
|
| 57 |
|---|---|---|
|
| 58 |
| Base output entropy | 5.951 | 7.882 β *very different* |
|
| 59 |
| **Post-SFT output entropy** | 1.795 | **1.624** β same order |
|
|
@@ -74,19 +80,19 @@ This is the cleanest evidence we have, and it is only visible here. In a transfo
|
|
| 74 |
"memory" from "processing" β attention weights carry both. In this architecture components separate by function,
|
| 75 |
so we can read directly where SFT went:
|
| 76 |
|
| 77 |
-
|
|
| 78 |
|---|---|
|
| 79 |
-
| **
|
| 80 |
| embedding / output head (tied) | 5.327 % |
|
| 81 |
-
|
|
| 82 |
-
|
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
|
| 87 |

|
| 88 |
|
| 89 |
-
**SFT moved the memory by roughly one part in a million.** It rewired the
|
| 90 |
*"SFT changes how the model speaks, not what it knows"* is, in this architecture, a **structural** statement
|
| 91 |
rather than an inference.
|
| 92 |
|
|
@@ -98,8 +104,8 @@ rather than an inference.
|
|
| 98 |
| **Alignment tax** | **+11.5 %** | **+35.9 %** |
|
| 99 |
|
| 100 |
Same data, same recipe, **3Γ the cost**. Our pre-registered prediction was "~11 % again" β that was a **miss**,
|
| 101 |
-
and it is recorded as one. Likely
|
| 102 |
-
so
|
| 103 |
prediction: giving that projection its own lower learning rate should reduce the tax.)*
|
| 104 |
|
| 105 |
## Result 4 β blind judge panel
|
|
@@ -173,7 +179,7 @@ There are no weights here, so there is nothing to license as a model. What *is*
|
|
| 173 |
quote it, replot it, argue with it β just cite it. Attribution is the whole point; these numbers exist so that
|
| 174 |
someone else can check whether our claim about SFT holds up.
|
| 175 |
|
| 176 |
-
**Not covered by this licence, because it is not published:** the architecture's equations,
|
| 177 |
parameterisation, initialisation recipe, parallelisation recipe and stability rules.
|
| 178 |
|
| 179 |
## Citation
|
|
|
|
| 13 |
|
| 14 |
# Tetracta-zkas-1B β results card (no weights)
|
| 15 |
|
| 16 |
+
> **Newer work:** this card documents our 2025 architecture. The current generation β
|
| 17 |
+
> **Z-Next 1B**, no attention and no KV cache, with a session state that does not grow
|
| 18 |
+
> with context β has its own results page and a **live public demo you can talk to**:
|
| 19 |
+
> Β· [Z-Next 1B results & benchmarks](https://www.tetracta.ai/znext.html)
|
| 20 |
+
> Β· [**Live demo β**](https://www.tetracta.ai/zchat)
|
| 21 |
+
|
| 22 |
**This repository publishes measurements, not weights.**
|
| 23 |
|
| 24 |
`tetracta_zkas_1B` is the **twin** of [Tetracta VAN-1B](https://huggingface.co/tetracta/tetracta-van-1b):
|
|
|
|
| 26 |
instruction-tuning run** (same 11,419 examples, same recipe, same seed). The only difference is the architecture.
|
| 27 |
|
| 28 |
That makes it a controlled single-variable experiment, and it is the reason our central claim about SFT is not
|
| 29 |
+
a one-model anecdote. **All of those results are below.** The architecture's equations, parameterisation,
|
| 30 |
initialisation recipe and stability rules are **Tetracta's closed research work and are not published.**
|
| 31 |
|
| 32 |
- π **[tetracta.ai](https://tetracta.ai)** Β· π **Paper:** *"What does SFT do?"* β [tetracta.ai](https://tetracta.ai)
|
|
|
|
| 41 |
| | |
|
| 42 |
|---|---|
|
| 43 |
| Type | **Not a transformer.** No attention. |
|
| 44 |
+
| Mechanism | **Recurrent, constant-size state.** The mechanism itself is not disclosed. |
|
| 45 |
| Inference | State size is **independent of sequence length** (O(1) per step) |
|
| 46 |
| Parameters | **953,466,904** |
|
| 47 |
| Pre-training | **315,000 steps Β· 10,321,920,000 tokens Β· one epoch**, bit-identical to VAN-1B |
|
| 48 |
| Instruction tuning | 11,419 examples Β· 3 epochs Β· lr 1e-5 Β· batch 16 Β· 2,142 steps Β· 1,247 s (1ΓH200) |
|
| 49 |
| Stability | **0 skipped steps out of 315,000** during pre-training; 0 during SFT |
|
| 50 |
|
| 51 |
+
Not disclosed: equations, parameterisation, initialisation recipe, MFU/parallelisation
|
| 52 |
recipe, stability laws, energy-distribution findings.
|
| 53 |
|
| 54 |
---
|
|
|
|
| 59 |
If that is true, two very different architectures given the same data and the same SFT should end up at the
|
| 60 |
same *interface* state even from different starting points. They do:
|
| 61 |
|
| 62 |
+
| Measurement | VAN-1B (transformer) | **zkas-1B** |
|
| 63 |
|---|---|---|
|
| 64 |
| Base output entropy | 5.951 | 7.882 β *very different* |
|
| 65 |
| **Post-SFT output entropy** | 1.795 | **1.624** β same order |
|
|
|
|
| 80 |
"memory" from "processing" β attention weights carry both. In this architecture components separate by function,
|
| 81 |
so we can read directly where SFT went:
|
| 82 |
|
| 83 |
+
| Group of parameters | relative change (Ξ΄) |
|
| 84 |
|---|---|
|
| 85 |
+
| the part that shapes **how the model speaks** | **18.72 %** β largest move, 3.5Γ the embedding |
|
| 86 |
| embedding / output head (tied) | 5.327 % |
|
| 87 |
+
| everything else in the processing path | 0.07 % β 2.56 % |
|
| 88 |
+
| **β
the part that holds the model's memory over time** | **0.00009 %** β **frozen** |
|
| 89 |
+
|
| 90 |
+
*(Component names and the full per-part breakdown are not disclosed; the grouping above
|
| 91 |
+
is what the measurement supports without describing the architecture.)*
|
| 92 |
|
| 93 |

|
| 94 |
|
| 95 |
+
**SFT moved the memory by roughly one part in a million.** It rewired the output-shaping path instead.
|
| 96 |
*"SFT changes how the model speaks, not what it knows"* is, in this architecture, a **structural** statement
|
| 97 |
rather than an inference.
|
| 98 |
|
|
|
|
| 104 |
| **Alignment tax** | **+11.5 %** | **+35.9 %** |
|
| 105 |
|
| 106 |
Same data, same recipe, **3Γ the cost**. Our pre-registered prediction was "~11 % again" β that was a **miss**,
|
| 107 |
+
and it is recorded as one. Likely reason: the part that moved most participates in processing *every* token,
|
| 108 |
+
so perturbing it by 18.7 % disturbs the plain-text distribution more. *(Untested hypothesis. Testable
|
| 109 |
prediction: giving that projection its own lower learning rate should reduce the tax.)*
|
| 110 |
|
| 111 |
## Result 4 β blind judge panel
|
|
|
|
| 179 |
quote it, replot it, argue with it β just cite it. Attribution is the whole point; these numbers exist so that
|
| 180 |
someone else can check whether our claim about SFT holds up.
|
| 181 |
|
| 182 |
+
**Not covered by this licence, because it is not published:** the architecture's equations,
|
| 183 |
parameterisation, initialisation recipe, parallelisation recipe and stability rules.
|
| 184 |
|
| 185 |
## Citation
|