AbstractPhila PRO

Framed training going well, going to put in a selector system for it to test some CoT multi-chains and start up an embedding prototype. I'm thinking multi-thinking is a viable option for some tinkering.

Too much data. Needed to purge old trains, loras, and unnecessary checkpoints on the SD15, SDXL, and multiple other diffusion trains.

Mostly purged optimizers but some checkpoints went as well, mostly useless checkpoints.

Experiments are humming along. Ready by this evening is likely not going to be the case, however I believe I have a few solutions for the aleph system.

Historic experimentation had a few leads, mostly dead ends. A couple of options showed promise; such as treating the signal as amplitude, while other options such as the heavily vetted Cantor positioning system are sidelined for stability in this version.

The result is likely going to treat that particular Aleph signal as confidence, as removing it from the model completely caused high confidence with incorrect answers while present the confidence would dip. It very would could be correlation without causation, but the tests are worth running. I will continue to investigation to get the correct answer for the causal behavior exhibiting the result, but it's definitely been difficult to tack down.

In any case, the last block large direction problem has been solved.

The prototypes will be trained in AdamW and Muon. AdamW shows stronger frame curation and control than Adam, and the large final direction being given AdamW with weight decay. The full prototypes will likely be trained with weight decay, as the prototypes have shown strong results with it.

As long as the final block is given Adam in the right place, the model converges, and the Aleph head quiets. To cultivate this head, the aleph system will be given the job of confidence. It's not as powerful as the alternative in theory, but the pragma shows the likelihood of it working to be considerably higher and more useful.

I'll need some days to align the next nonlinear prototype head, hopefully by the end of next week. In any case, the model itself still uses aleph internally, but the global aleph system will be quiet on this version. The next byte framing and context window will be more dominant, along with the banks. This is fine as many of these prototypes can be trained.

Stability is key, as long as the model is stable the models will work. If they aren't stable or collapse then they won't be of any use for data, other than trying to prevent or balance around it.

Legitimate strengths the other does not possess. Pragma and proof with cooperation.

Unironically, the answer for the splat problem was... a form of mathematics derived from art. The Canon procedure, which a specific paper's work adapted and fitted to Pollux and the splat layer in a meaningful manner. This brought the frame system from nearly nothing on recall, to a direct comparator to a rotary softmax attention. This when applied with rotary positions allowed the splat layers to behave and frame more effectively than the tested softmax rotary scales tested on many fronts. The cost of train went up a little, but the prototype does have meanginful frame learning and context awareness.

I've predominantly considered the splat component to be an artist component in many ways. It's reasonable to find many answers to the artist's quirks for organization, in the theory of music itself. It'll be in the writeup.

https://arxiv.org/abs/2512.17351

Okay so it may or may not work the same way as intended right now, however I'm not willing to abandon the semi-functional systems in favor of a formula switch that requires multiple new components. I'm not certain, and I will need to be certain.

I'll need to sleep on it. The model is far from dead, just needs a proper optimizer to cultivate the rifts. Something that handles the correct intention and the correct implicit skips as per the necessary geometric framing tested. There will need to be a tweak or two to the formula most likely, hopefully topical tweaks only. I don't want to be at the blackboard for another month only to find a model that runs like a snail on a salty day.

If the MUON system simply wasn't cultivating the results, I will figure it out tomorrow when i wake up and then figure out the correct answer.

We have our core optimization fix. Now I need to enable the correct Cantor Aleph-Void behavior to shortcut with the aleph addressing with something similar to the applied Fractal Wormhole positions as experimented with in the past, aka matching numbers down the chain that usually lead to the classic Cantor Dust problems. Number systems that have given mathematicians headaches for many years. https://en.wikipedia.org/wiki/Cantor_set

After that the models need to learn to depend on them, which should fit nicely into the context window framing. I believe this should be stable enough under fp32 to round since I'm not using the Cantor structure for positioning directly but instead the Aleph mathematics for the actual prediction system. This should be stable enough, but if not I'll work something else out. As long as we don't hit something as complex as the Mandelbrot it shouldn't collapse or explode. FP64 isn't on the menu thank you.

A fully represented Cantor Dust byte prediction system would take a considerable amount longer per token. This is a cultivated behavioral response system meant to replicate the activity through linear mathematics, not a direct 1:1 clone.

Every fractal slice will cost huge amounts of compute, which is the entire point of avoiding using them directly and instead cultivating them as a cooperative element. Without the MUON being fully dominant they are not cultivating properly on their own, as the steps aren't allowed to form those extra potentially incorrect distances that allow nonlinear shortcuts to form. Adam and AdamW turn those into smooth curves, when we want some gaps and rifts in order to learn the nonlinear math in a way that doesn't form a billion unmeasurable infinities, or entirely matching cosine final layers.

The reason why there's a component named governor is to limit these leaps and bounds. They will eventually cause much worse problems if they are left completely unchained. So a couple hundred million per 60 billion are casualties, give or take, for those who reach too far or are pinched too tightly for rounding to correct, and are rewarded for the action.

Smooth TOO MUCH and the head never activates. Smooth TOO LITTLE and you get the big clump. Likely will need an optimizer for the behavior.

Both good news, and bad news.

The good news, I solved the final direction problem.

canary_acc 0.000 · doc n=70 bpb=4.22 reset=2.47 · bits by position (row) 0-64 1.84 64-256 1.67 256-1024 1.58 1024-2048 1.52 2048-4096 1.53
[lastfix] c-head-adam step 1,500: val 1.5488 bits a byte | share along the last block's direction 0.324 (after the norm 0.295) | last bank write 0.249 on its direction, size 328 | last block rank 35 | output layer size 12.6, content loudness 1.25


canary_acc 0.000 · doc n=70 bpb=4.17 reset=2.37 · bits by position (row) 0-64 1.77 64-256 1.60 256-1024 1.50 1024-2048 1.44 2048-4096 1.45
[lastfix] c-head-adam step 2,000: val 1.4719 bits a byte | share along the last block's direction 0.313 (after the norm 0.285) | last bank write 0.229 on its direction, size 455 | last block rank 39 | output layer size 13.1, content loudness 1.31

Probably.

THE BAD NEWS, I'll need to run all of yesterdays tests again with the new optimizer. It invalidates many of the framing experiment guidelines without running the full test suite. The models simply cannot TRAIN FOR WEEKS without running full barrage testing on the structure, in a day that could save a month of training.

It's both irresponsible and painful on the wallet if I don't run the correct tests for speed, structural architecture, and accuracy before I run.

All the fixes work. Doesn't even need weight decay, just Adam. Eliminate Muon from the final layer's head, that's all it really needs to have. The most dominant pieces the model would have directional bias formed from, solved.

Castor's early setup is clean. Very clean. I expect a strong feature by the end of training, rather than a single dominant fractal direction that would take me weeks to investigate.

As before the aleph head is getting buried. The model isn't going to be incentivized to learn nonlinear structure without a fuss. Muon made this more than viable but it formed the cascade knot that would crush the final layer. There's likely a cascade middle-ground between behavioral removal and cascade articulation. Something that will form reasonable amounts of utility without forcing the model to learn something that will simply be optimized away.

They must LEARN to cooperate without smothering each other. MUON makes this viable, but there are cascade failure points from this that cannot go unaddressed.

Got a little ahead of myself here. We have one more phase to go before we begin. Possibly ready tonight or tomorrow morning. possibly sooner. If it's put off I'll write another post. The final layer's huge direction pressure needs to be addressed, and we have some potential fixes but they are untested. I'll need to run those first.

The idea here is to capture the amplification throughout the model rather than allowing it to become a causal obstruction in the final layer. I'd like to call it a fractal sift but it's really a lot simpler than that. Essentially giving the model the capacity to internalize the structure so the larger behavior can cut through rather than the heavier elements causing a clog at the final block.

AH it's a Muon byproduct according to the paper. That makes more sense. I'll try enabling weight decay on AdamW for the final block, and for the rest Adam no decay with Muon. I have had a strict barre on AdamW for quite a while now, as it would mask the results of problem architecture. At this point I think AdamW's weight decay is the solution to the final layer due to the methodology the optimizer employs and the byproduct the optimization has for the huge directional accumulation, so I don't need architecture I just need a couple tweaks.

Lets test that theory.

If this theory holds, I'll run tests using AdamW on the overall model rather than just on the final layer. It's possible the AdamW and the model are noncompliant though, which has been historically shown on a multitude of trains related to the aleph structures. If this is the case and they are noncompliant I will likely create a compliant version of AdamW specifically curated for the aleph structure for testing. I doubt I can create something as potent as AdamW, I just need to tweak it a little at most, but it may need to be specifically tweaked for the more errant curves that are being optimized away rather than utilized. Adam captures these a bit more effectively and I'm concerned the decay will destroy them as it did before, but I'm optimistic.

I'm approaching this one with a scalpel and a microscope.

As this may solve the faults with the aleph pred in it's entirety with a little luck. Just enabling weight decay alone may solve the problem, or it may completely brick the model. Small experiments first.

We have our twins.

Capable of read/write/copy of shared information, both housing the capacity of 4096 context depth with nested stable frame teaching, extractions, resizing, and many more elements that simply weren't present or possible to control before.

Roughly 20m params total for the twins, and the dual hub paradigm trains at around 500k tok/s.

Both Pollux and Castor are learning a form of dynamic context window division experimentally currently.

Experiments show the models can support multiple dynamic sub-sizes to the primary context window of 4096 with each model supporting different sizes and forms dubbed golden framing, exactly as predicted from the larger structure's experimental outcome and training data's curves would signify.

These nano twins show the aleph nature of the model isn't just externalizing, but it can be directly internalized and curated as well. Experiments show the bpb per frame is stronger than others for multiple frames, tested around 400 million bytes per in roughly 6-7 minutes per train give or take.

The models train at roughly 890k tok/s on the rented colab rtx 9000 pro, and they only require about 5 gigs vram to fully saturate the card's calculation capacity.

These models aren't meant to be intelligent, they are meant to show the differentiated capacity of the hyperbolic system when correctly shaped. They don't need to remember everything, they need to show WHERE they are stable, WHY they are stable there, and provide A REASON as to why they are and aren't stable there.

Cantor sets and fractal numeric systems are fickle and unstable numeric calculations.

These systems are not unstable.

Assigning a special token to frames is already happening. Larger frames within the context window allow larger coverage within. Sub-frames allow sub-framing within. 1024 made of multiple sub-slices to fit the window space, 2048 sliced into multiple 1024's.

So far Pollux is the only one capable of this, standard softmax variation does not yet support this behavior at this size. However, the tests are showing the twins don't need to stay independent. They can be combined together into co-dependent structures, both trained on the same data meant to communicate and arbitrate in their own ways.

Castor can read very effectively. Stores documentation and understanding within reasonable lookup space and can recall this information.

Pollux can combine effectively. Creating dynamic window shapes, scaling paradigms, structural undertones, and systems of complexity that Castor cannot manifest at this scale.

Strange concept I know. The dynamic context frame slicing of Pollux is inherently hard to understand, but the concept is something that we as people share. We don't keep a thousand questions on the immediate tip of our mind to get one answer, we tug on the structure of understanding and follow the memory chain. With that we find the tapestry of logic, but only if we need this logic.

Both of these are capable of part of that whole, but they aren't complete yet.

The nano variants train at 900k tokens/s on an rtx 6000 pro.

Say hello to Castor and Pollux.

One is Castor, the rotary softmax structure with a fixed lookup table. Pollux is rotary splat/softmax alternating 3/3.

Each 6 blocks, between 8 and 9 million params.

All tests show they each have their own strengths that the other cannot encompass. The twins are meant to be combinatory students, meant to learn and recall more data together than they otherwise cannot handle independently.

I have also named the 2s variant twins, given them individual names.

Well I think we have our answer.

I'm not going to measure any verdicts yet. Softmax with learned positions is strong, but multiple forms of rotary are known to extend models to huge context windows.

rotary hubs + rotary softmax alternating is a very powerful craft. The outcome within noise matters, but I will test the paired rotary version as our hybrid at depth. This might just form the necessary continuum behavior I've been looking for, preserving and updating records rather than suffering at depth for a while until the model catches up.

There are definite causal relational behaviors related to framing data within a context window. I'm starting to see the causal outcomes of the aleph resizing system and the behavior expected from this window. I believe we need a couple more scaling mechanisms to cultivate a full geometric skip-continuum behavior in the future, that's another experiment though. I've experimented in the past, chopping up the window does weird things, however this entire model is modular, so when the window behaves correctly we ought to be able to slice off pieces of the window itself for necessary safety points, guarded sections, and so on.

Not enough data to be certain yet.

Context window is starting to behave though, which means the model will function. The rotary system is most definitely functioning for both the hub AND the softmax. This will saturate the 4096 context window with room to spare, and next 8192 may be possible on the nano. Oddly enough, I don't know the limit just yet. I'm thinking there is inherent hardware limits to this, rather than conceptualization limits at this point.

I'm thinking the naming should be; nano, micro, mini, small for now. The small versions being what I trained already and were the prelude to the required answers.

I will not make any promises about beginning v4 yet, as we have to answer many questions before we can reach a verdict of continuity. Then, and only then, we cook.

Historically, each book has been stable up to 2048 positions. The last train shows that a single book cannot house the entirety of 4096, and trying to do this causes the system collapse.

However, two books... is another story. This aligns with our structure and fits comfortably within the system.

With two small-sized books, scaled on tiny shallow tests from before, we have a fairly inaccurate recall shown by the 2s and worse by the v3.

The v1 model had k128 and this was was sliced for optimization to k64 for v2 and v3 due to shallow tests, data expressing that earlier train steps were not affected drastically BY this book's k. However, I never set canary checks for recon shifts as they historically showed durability, which showed they quite literally fell apart over time. I looked for many useful traits and forgot one of the most critical, depth recon.

Now, k256 on the other hand, seems to be doing quite well. Same dimensional shape, different format.

I'll be testing k128, k512, and k1024 as well.

Each book is structurally compliant with signed predicted structures fitted to 2048 slots, larger books were often never used in testing. Classifiers almost never cared about numbers beyond 512 or so, depending on the classifier's shape and size. However, that was an entirely different process. THESE books are being fully saturated with the billions of bytes being fed. They can handle the differences.

Tests show that k64 is nowhere NEAR good enough, like mounting a single home printer in an office for 500. K128 will likely show a similar curve between k64 and k256, and k512 will likely be the strongest and fastest to converge, so we'll likely be using k512 for the nanos.

More than likely we can handle 4-8 books of k1024 and create a collective delegation for 8192 to 16384. Each book specifically tuned to datasets and their learning mutations slowed or frozen. We'll see how k512 handles first though.

That's why this system exists.

These tests are all being done directly on fineweb using the upcoming nano prototypes. There will be hundreds as they surge train.

The structures of the next variant will include a micro-sized version, entirely dedicated to attaching for experimentation. Nano sized, smaller than even a single arm operating on the 3s model. The smallest version will not be intelligent, but she will definitely be modular like the others. This nano version will likely be as intelligent as the best v3 model to date, containing a 4096 context window within the microscopic forms.

This model will train very very quickly. Lets see how much context length we can pack into this little piece of equipment, more akin to a cybernetic implant meant to augment and route than create decisions.

Nano will likely be a hybrid of a higher k splat combined with softmax as a dual brain, as they seem to function highly efficiently in pairs. Softmax learns quickly but often has problems for simpler and slower alternations, while splat takes longer to converge at length but has many benefits to it's scaling and sizes that softmax simply does not capture. I'm currently studying the different effects and alternating outcomes as based on the structures to determine the best methodology of coordinative cooperation.

I, honestly didn't expect the two to be cooperative without A LITTLE procrustes. They... work really well together, the twins do even without procrustes whitening rotation. WITH IT they teach the little nano versions rapidly.

Softmax Twin ETA: 18:00 GMT-8

Nano prelims are showing some genuine promise.

Rotary + Hub is actually showing some real recon weakness. That's... unexpected considering the prelims. That's okay though, the outcome is pushing towards two rather than a single anyway.

Looks like we're getting close now. The depth experiments are aligning to the expected dynamic framing.

You can kind of think of this structure here as a big fat memory blob that turned into soup, being given database lookup now thanks to the last three Beatrix models trained. We have the correct and necessary information needed to align her.

The modularity allows the twins to coexist as well. Multiple experiments were performed with the Beatrix 2s twins cooperating for recall accuracy tests. This concept had multiple unaddressed problems but it does work. The final 2s softmax twin isn't ready quite yet, she'll be ready this evening. I got a bit of a head-start on it from an earlier checkpoint and the results are aligning.

The only way to correctly align the v3 is for me to spend a week training and babysitting the softmax v3 model. For now I'm working with the 2s models until I'm satisfied the odds are in my favor. After that, we can start pushing much further for depth accuracy.

A successful full objective memory container of this scope ought to represent a context window larger than the traditional limitations. The rotary behavior mixed with addressing is showing promise, so a full rotary addressed softmax splat prototype is on the queue.

Larger addressing gives much more consistent recall with the byte structures, however the splat attention will need to be curated and correctly aligned to the necessary scale and scope at depth. Correctly aligning the addressing scale to the necessary depth will provide the most useful lookup structure in conjunction with the shared byte knowledge.

We have yet to decide a K structure size. Too large and the system takes too long to train with the structure, too few and the model simply does not rationalize the utilizable relations at depth. There are multiple models with examples but there are many potentials here that aren't present in most models.

I had Claude run many methods of positional encoding overnight. Some are more effective than others.

There are multiple bigger address splats with rotary that can support the recon behavior shallow and when at depth the model simply falls apart. The chosen address schema simply wasn't strong enough. When paired with rotary the model learns at depth either way. So, the address wasn't big enough. The v4 target is making itself.

The v4 variant will have multiple upcoming finetune targets. The v4 will inherit the v3 weights and be refitted to be predominantly softmax attention curated with a permanent softmax attention behavior at the core.

This has proven to be without a doubt the best methodology for delta attention structures as of today, tomorrow may be different but so far the softmax inclusion is one of the best courses of action.

V4 will be considerably more thought out, carefully curated into a behavior that makes the model have an internal perspective of the world around herself.

One of the experimental tests meant for the better recall version of her will include color checking.

I have a series of tests meant to test her pragmatism, understanding of normality, relational behavior to decision, and more. The more of these I develop, the better I'll be able to curate the model towards a behavior that is considered acceptable.

Alongside I'll be running her through arc tests as well for baseline behavioral utility and so on. There's quite a bit of information you can gain from pragmatic studies.

To create a reasonably curated and useful v4, the structure must be aligned to be a useful and understandable system. The bulk pretraining will be based on a curated system of articulated behavior, specifically designed to enhance the model's capacity to understand and perceive science, mathematics, literature, multiple languages, history, and more. The structure of this information curated appropriately to the various general tasks people might want to experiment with, and specifically for art and diffusion in my particular interests.

This will include a full background database understanding of art, photography, 3d spatial understanding, 2d representative imaging, and more. Essentially a world view around building information and using that information meaningfully for an art-driven diffusion conditioning. The entire process will revolve around her perspective of the world around a curated world space to perceive and rationalize a pragmatic objective reality.

Using a type of planted value for utility, the process is yielding depth. Another 8k step arm will determine if this process can be made permanent, or if the 3s model is simply too deviated and requires an additional finetune pass.

This doesn't solve the FP instability of a fractal rotary, fibo rotary, standard rope, or a few others in conjunction with codebook trigrams; it just makes the problem harder for the optimizer to solve, leaving bread crumbs.

I'll figure something out. So far it's looking promising, but it's still falling quite short of standard models. With this format of dynamic modularization, there is a heavy burden to sharing the space.

I have a few ideas and I won't give up.