Smilyai-labs-beta-testers/Smilyai-3rd-gen

SmilyAI 3rd Gen Assistant — CascadeFormer

Beta Release — This model is under active development. You are testing a pre-release architecture. Expect rough edges, routing instability in edge cases, and evolving quality across sessions. Your feedback directly shapes the next generation.

---

Table of Contents

• Overview
• Architecture
• Expert System
• Hardware Requirements
• Edge Deployment
• Usage
• Limitations
• Beta Tester Guide
• Reporting Issues
• Roadmap
• License

---

Overview

CascadeFormer is the third-generation assistant model from SmilyAI. It is a small language model designed to punch significantly above its weight class through a core architectural innovation: a Heterogeneous Mixture-of-Experts (HMoE) feed-forward system where experts are not equal in size — they are deliberately varied to match the natural complexity distribution of language itself.

The model is named CascadeFormer because knowledge flows through it in cascading tiers — from fast, cheap, surface-level pattern recognition at the bottom, through domain knowledge in the middle, up to deep reasoning at the top — much like a cascade of filters, each catching what the last could not.

Property	Value
Model family	CascadeFormer (SmilyAI 3rd Gen)
Total parameters	~5.0B
Active parameters per token	~350–500M
Architecture	Heterogeneous Sparse MoE Transformer
Attention	Grouped Query Attention (GQA)
Expert tiers	3 (NANO / MIDI / MACRO)
Total experts per layer	96
Layers	24
Model dimension	1024
Context length	4096 tokens
Vocabulary	32,000 (SentencePiece BPE)
Precision (training)	BF16
Generation	3rd
Status	Beta

---

Architecture

CascadeFormer is a decoder-only causal language model built on a heavily modified transformer backbone. The key architectural decisions are described below.

High-Level Structure

Input Tokens
     │
     ▼
Token Embedding (32,000 × 1024)
     │
     ▼
┌─────────────────────────────────────┐
│       CascadeBlock × 24             │
│                                     │
│  ┌─────────────────────────────┐    │
│  │  RMSNorm                    │    │
│  │  Grouped Query Attention    │    │
│  │  (local on even layers,     │    │
│  │   global on odd layers)     │    │
│  │  Residual connection        │    │
│  ├─────────────────────────────┤    │
│  │  RMSNorm                    │    │
│  │  Heterogeneous MoE FFN      │    │
│  │  Residual connection        │    │
│  └─────────────────────────────┘    │
└─────────────────────────────────────┘
     │
     ▼
Final RMSNorm
     │
     ▼
Output Head → Logits (32,000)

Attention

CascadeFormer uses Grouped Query Attention (GQA) with an alternating local/global pattern across layers.

Property	Value
Query heads	16
Key/Value heads	4 (4:1 GQA ratio)
Head dimension	64
Even layers	Sliding window attention (window = 512)
Odd layers	Full global attention
Position encoding	RoPE (base frequency 500,000)
Attention implementation	Flash Attention 2
QK normalization	Yes (RMSNorm on Q and K)

Why GQA: The 4:1 key-value reduction shrinks the KV cache by 4× at inference time. This is critical for edge deployment where memory is the bottleneck, not compute.

Why alternating local/global: Local attention on even layers captures syntactic and proximity patterns cheaply. Global attention on odd layers maintains long-range coherence. Combined this saves approximately 40% of attention compute versus full global attention on every layer.

Normalization

• RMSNorm everywhere — no LayerNorm used anywhere in the model
• Pre-norm placement (normalize before attention/FFN, not after)
• QK-Norm applied inside attention to prevent logit explosion at long sequences
• No dropout anywhere in the base model (modern practice at this scale)

Position Embeddings

• Rotary Position Embeddings (RoPE)
• Base frequency: 500,000 (versus the original 10,000)
• Enables context extrapolation beyond training length via YaRN scaling at inference without fine-tuning

---

Expert System

This is the core innovation of CascadeFormer. Unlike standard MoE models where every expert is identical in size, CascadeFormer uses three tiers of experts with fundamentally different parameter budgets matched to the natural complexity distribution of language.

The Philosophy

Not all language processing demands the same capacity. Handling a comma requires almost nothing. Tracking a logical implication across a paragraph requires significantly more. Forcing every expert to be the same size either wastes capacity on simple patterns or starves complex reasoning of the parameters it needs.

CascadeFormer solves this by letting the router decide not just which expert handles a token, but how much model capacity that token deserves.

Expert Tiers

NANO Experts — Fast, cheap, high-frequency patterns
MIDI Experts — Domain knowledge, balanced cost
MACRO Experts — Deep reasoning, rare but powerful

Tier	Count per layer	Hidden dim	Params per expert	Total per layer	Typical tokens routed
NANO	32	256	~0.39M	~12.6M	Articles, punctuation, common conjunctions, whitespace
MIDI	48	512	~1.57M	~75.5M	Domain terms, named entities, idioms, technical language
MACRO	16	1536	~4.72M	~75.5M	Logical connectives, mathematical symbols, causal reasoning tokens
Total	96	mixed	—	~163.6M	—

All experts use the SwiGLU activation function:

Expert(x) = Down( SiLU(Gate(x)) ⊙ Up(x) )

Where:
  Gate : Linear(d_model → d_expert_hidden)
  Up   : Linear(d_model → d_expert_hidden)
  Down : Linear(d_expert_hidden → d_model)

Routing Architecture

CascadeFormer uses a three-level hierarchical router:

Token hidden state x
        │
        ▼
┌───────────────────┐
│   Tier Router     │  Linear(1024 → 3) → Softmax
│  NANO/MIDI/MACRO  │  Selects which tier(s) to activate
└───────────────────┘
        │
   ┌────┴────┐
   ▼         ▼
NANO       MIDI        MACRO
Router     Router      Router
(top-2)    (top-2)     (top-1)
   │         │            │
   └────┬────┘            │
        └────────┬─────────┘
                 ▼
        Weighted combination
        back to d_model

Experts activated per token:

• Typical token: 2 NANO + 2 MIDI + 0 MACRO
• Complex token: 0–1 NANO + 2 MIDI + 1 MACRO
• Simple token: 2 NANO + 0–1 MIDI + 0 MACRO

Load Balancing

Standard MoE load balancing equalizes token counts per expert. CascadeFormer uses compute-normalized load balancing because a MACRO expert should receive fewer tokens than a NANO expert by design — they cost more per token.

L_balance = α × Σ_i ( f_i × cost_i × p_i )

cost_NANO  = 0.25
cost_MIDI  = 1.00  (reference)
cost_MACRO = 3.00

α = 0.01

Additional router Z-loss (prevents logit explosion):
L_z = β × (1/B) × Σ log²(Σ exp(router_logits))
β = 0.001

Target compute distribution at equilibrium:

• NANO: ~25% of total MoE compute
• MIDI: ~50% of total MoE compute
• MACRO: ~25% of total MoE compute

---

Hardware Requirements

Training Hardware

Component	Spec
GPU	NVIDIA RTX 6000 Pro Blackwell
VRAM	96 GB
Precision	BF16 (weights + activations), FP32 (optimizer states)
Estimated VRAM usage	~75 GB during training

Inference Hardware (Full Model)

Device	VRAM/RAM needed	Notes
NVIDIA GPU (server)	12 GB+ (INT4) / 20 GB+ (BF16)	Full quality
Apple M-series (M2 Ultra+)	24 GB+ unified memory	Via llama.cpp / MLX
CPU (high-end)	32 GB+ RAM	Slow but functional in INT4

---

Edge Deployment

A primary design goal of CascadeFormer is graceful degradation on edge hardware through selective expert loading. Because experts are organized in tiers, you can drop the most expensive tier first and lose capability in a predictable, controlled way.

Edge Deployment Profiles

Device	Expert Loading Strategy	Quantization	Quality Impact
Full server (96GB VRAM)	All 96 experts, all 24 layers	BF16	Full quality
Jetson AGX Orin (32GB)	All NANO + All MIDI + 4 top MACRO experts	INT8	Minor — deep reasoning slightly weaker
Jetson Nano (8GB)	All NANO + top 24 MIDI by activation frequency	INT8	Moderate — reduced domain depth
Raspberry Pi 5 (8GB RAM)	All NANO + top 8 MIDI	INT4	Noticeable — narrow domain only
Raspberry Pi 4 (4GB RAM)	All NANO only	INT4	Significant — surface patterns only

Expert Selection for Edge Devices

To determine which experts to load for your specific use case:

1. Run a calibration pass on 500–1000 samples from your target domain
2. Record the expert activation frequency histogram
3. Rank experts within each tier by activation count
4. Load only the top-K experts for your target device budget

This gives you a domain-optimized edge model rather than a generic pruned one.

Quantization Support

Format	Status	Notes
BF16	✅ Native	Full fidelity
FP16	✅ Supported	Minimal quality loss
INT8	✅ Supported	Recommended for Jetson class
INT4 (GPTQ)	🔄 In progress	Required for Pi class devices
INT4 (AWQ)	🔄 In progress	Preferred over GPTQ for MoE
GGUF	🔄 In progress	For llama.cpp compatibility

---

Usage

⚠️ Beta Note: The inference API is not yet finalized. The interface below reflects the intended API and may change before stable release.

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "smilyai/cascadeformer-3rd-gen"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True   # required for custom MoE architecture
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain the difference between induction and deduction."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        input_ids,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        repetition_penalty=1.1
    )

response = tokenizer.decode(
    output[0][input_ids.shape[-1]:],
    skip_special_tokens=True
)
print(response)

Edge Inference (Expert Subset Loading)

from transformers import AutoTokenizer
from cascadeformer import CascadeFormerForCausalLM, EdgeConfig

# Define edge profile
edge_config = EdgeConfig(
    nano_experts="all",          # load all 32 NANO experts
    midi_experts="top_24",       # load top 24 MIDI by calibration
    macro_experts="none",        # drop MACRO entirely
    quantization="int8",
    calibration_dataset="your_domain_samples.jsonl"
)

tokenizer = AutoTokenizer.from_pretrained("smilyai/cascadeformer-3rd-gen")
model = CascadeFormerForCausalLM.from_edge_config(
    "smilyai/cascadeformer-3rd-gen",
    edge_config=edge_config
)

Chat Template

CascadeFormer uses the following chat format:

<|system|>
You are a helpful assistant.
<|user|>
Hello, who are you?
<|assistant|>

---

Limitations

Beta testers should be aware of the following known limitations:

Architectural Limitations

• Context length: Hard limit of 4096 tokens during this beta. Context extension via YaRN is planned for stable release.
• MACRO expert warm-up: In the early checkpoints of each training session, MACRO experts may produce lower quality outputs as routing stabilizes. This improves significantly after ~5000 training steps.
• Routing sensitivity: The tier router can occasionally misclassify token complexity in ambiguous contexts, sending complex reasoning tokens to NANO experts. This is being actively monitored.

Quality Limitations

• Mathematical reasoning: MACRO experts handle this well but the model is still in early training. Multi-step arithmetic and algebra are unreliable.
• Long document coherence: At sequences approaching 4096 tokens, coherence may degrade in narrative or technical document generation.
• Rare language patterns: Very unusual syntax, code in obscure languages, and highly domain-specific jargon may produce unexpected outputs.
• Factual accuracy: As with all language models, factual claims should be independently verified. The model has a training data cutoff and may hallucinate.

Edge Deployment Limitations

• INT4 quality on NANO-only profile: The Raspberry Pi 4 profile (NANO-only) is an extreme degradation mode. Outputs are grammatically coherent but may lack semantic depth.
• Custom kernels required: Heterogeneous expert batching requires custom CUDA/Triton grouped GEMM kernels. Out-of-the-box Transformers batching may not be optimal.

---

Beta Tester Guide

Thank you for being part of the CascadeFormer beta. Here is what we need from you specifically.

What We Are Trying To Learn

1. Expert routing quality — Does the model feel like it "gets" your prompts correctly? Or does it feel shallow/confused on complex questions and over-elaborate on simple ones?
2. Edge performance — If you have Jetson or Pi hardware, please test the edge profiles and report both latency and perceived quality loss.
3. Failure mode taxonomy — We want to catalogue how the model fails, not just that it fails. Does it repeat itself? Does it go off-topic? Does it fabricate confidently?
4. Routing collapse signals — If you observe the model giving very similar responses to very different prompts, please flag this. It may indicate expert routing collapse.

Testing Checklist

Please try to cover these categories in your testing:

□ Simple factual questions (short answer)
□ Complex multi-step reasoning (math, logic puzzles)
□ Creative writing (story, poem, dialogue)
□ Code generation (any language)
□ Instruction following (step-by-step tasks)
□ Long context (paste a long document, ask questions about it)
□ Edge case prompts (very short, very long, unusual formatting)
□ Adversarial prompts (try to confuse or break the model)
□ Your specific domain / use case
□ Conversation over multiple turns (10+ turns)

What Good Feedback Looks Like

Less useful:

"The model gave a bad answer to my question about history."

More useful:

"I asked the model to explain the causes of WWI across 3 turns of conversation. By turn 2 it had forgotten the context of turn 1. By turn 3 it contradicted itself on the date of the Archduke's assassination. The response style was confident throughout which made the errors harder to spot."

---

Reporting Issues

Please report issues via the community tab or the SmilyAI beta feedback form.

When reporting include:

## Bug Report

**Checkpoint version:**
**Hardware:**
**Inference mode:** [Full / Jetson / Pi / other]
**Quantization:** [BF16 / INT8 / INT4 / none]

**Prompt:**
[paste exact prompt]

**Output:**
[paste exact output]

**Expected behaviour:**
[what should have happened]

**Failure category:**
[ ] Factual error
[ ] Coherence/repetition
[ ] Routing feels wrong (too shallow / too elaborate)
[ ] Instruction not followed
[ ] Context forgotten
[ ] Harmful/unexpected content
[ ] Performance/latency issue
[ ] Other: ___________

**Additional notes:**

---

Roadmap

Milestone	Target	Status
Architecture finalized	Session 1	✅ Done
Foundation training (sessions 1–5)	Sessions 1–5	🔄 In progress
Scale-up training (sessions 6–15)	Sessions 6–15	⏳ Planned
Expert specialization phase	Sessions 16–25	⏳ Planned
Instruction tuning (SFT)	Sessions 26–30	⏳ Planned
Alignment (DPO)	Sessions 31+	⏳ Planned
INT4 quantization (GPTQ/AWQ)	After SFT	⏳ Planned
GGUF / llama.cpp support	After INT4	⏳ Planned
Edge calibration tooling	After GGUF	⏳ Planned
Context extension to 8192 (YaRN)	Stable release	⏳ Planned
Public stable release	TBD	⏳ Planned

---

Citation

If you use CascadeFormer in your research or products during the beta period, please cite:

@misc{smilyai2025cascadeformer,
  title         = {CascadeFormer: Heterogeneous Mixture-of-Experts for Efficient Small Language Models},
  author        = {SmilyAI},
  year          = {2025},
  note          = {Beta release. Model available at https://huggingface.co/smilyai/cascadeformer-3rd-gen},
  howpublished  = {\url{https://huggingface.co/smilyai/cascadeformer-3rd-gen}}
}

---

License

This model is released under the Apache 2.0 License.

You are free to use, modify, and distribute this model for commercial and non-commercial purposes with attribution. See LICENSE for full terms.

---

Acknowledgements

CascadeFormer draws on insights from the following works:

• Shazeer et al. (2017) — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Jiang et al. (2024) — Mixtral of Experts
• Su et al. (2021) — RoFormer: Enhanced Transformer with Rotary Position Embedding
• Dao et al. (2022) — FlashAttention: Fast and Memory-Efficient Exact Attention
• Ainslie et al. (2023) — GQA: Training Generalized Multi-Query Transformer Models
• Zoph et al. (2022) — ST-MoE: Designing Stable and Transferable Sparse Expert Models
• Touvron et al. (2023) — LLaMA: Open and Efficient Foundation Language Models

---

CascadeFormer is a SmilyAI project. This model card describes a beta release and will be updated as training progresses and the architecture matures.

Last updated: 2025 — Beta Session Active