RecursiveSI

/

flint

Like 1.31k

Follow

Recursive Superintelligence, Inc. 692

Hi — I ported Flint to a custom C++/CUDA runtime (single RTX 5090, 32 GB) from the
published BF16 checkpoint, reading config.json, chat_template.jinja,
tokenizer.json and the flint_sglang wheel as the spec. The model is genuinely
interesting to work with; below are the observations I'd have wanted before
starting, plus a couple of documentation suggestions. Everything here is
reproducible from the checkpoint alone.

What was unusually easy (thank you)

• The learned-sink semantics are documented by code, not folklore. The
  fa4-learned-sink-fix.patch makes the intent unambiguous: the sink is an extra
  softmax column (max and denominator include it, numerator gets no value), and
  sink_rescale is exactly the streaming-softmax correction when the sink becomes
  the running max. I implemented it directly from that patch and it matched.
• norm_topk_prob: true + "weights are renormalized from the plain sigmoid
  scores" left no ambiguity about whether the routing weights are pre- or
  post-renormalization.
• Per-head q_norm/k_norm are plain RMSNorm (weight form), not the 1+w
  variant — one less thing to guess.
• The 2-token-per-expert ceiling (num_experts_per_tok: 6 of 256) and the
  ungated shared expert are simple and fast to get right.
• attention_bias: false everywhere makes fused QKV kernels trivial.

Numerics: Flint has massive activations — worth documenting

This is the single most important thing for anyone quantizing or writing kernels:

• On a 160-token tool prompt, the residual stream reaches ~1.4e4 at layer 48,
  and a single layer (46) amplifies its input by ~300× (in ≈ 29 → out ≈ 9.5e3).
• In the same layer the expert gate projection reaches ~5.3e4 and the
  SwiGLU product ~1.5e8.
• All of that is fine in BF16 (which is why SGLang is happy), but it does not fit
  FP16: my first FP16-activation port produced inf in the expert FFN at layer 47,
  and NaN in the residual two layers later.

What worked for me: keep the residual stream (and the combined MoE branch) in
FP32, keep the FFN intermediates in range with power-of-two scaling
(÷64 into the gate/up GEMV, ÷8192 after the SwiGLU, restored on the
accumulate/combine). A note in the model card ("residual magnitudes up to ~1e4;
BF16 or FP32 required, FP16 overflows") would save the next integrator a day.

Related: --enable-fp32-lm-head is not cosmetic here. The head multiplies every
logit, and with these activation magnitudes it is the cheapest precision you can
buy back.

Configuration oddities

1. no_rope_layer_indices is exactly the full_attention set. Two lists that
   must stay identical are a footgun: an edit to one and not the other silently
   rotates (or stops rotating) 14 of 49 layers. A single list, or a converter-time
   assertion that no_rope ⊆ full_attention, would make the invariant explicit.
2. There is no expert_bias tensor in the checkpoint although the plugin
   creates gate.expert_bias as zeros. Fine (selection == plain sigmoid top-k),
   but a reader of the config can't tell that from config.json alone.
3. decoder_sparse_step: 1 and router_aux_loss_coef are train-time knobs that
   don't affect inference; mlp_only_layers/layer_types/no_rope_layer_indices
   are the three that actually define the graph. Worth calling out which is which.
4. sliding_window: 513 includes the current token (512 preceding). This is
   documented in configuration_flint.py, but note that SGLang's
   sliding_window_size excludes it — an easy off-by-one if you pass the config
   value straight through.
5. The sink is applied in natural logit units, not scaled by
   1/sqrt(head_dim). I verified this against the patch, but it is the kind of
   detail that reads as a bug the first time you see it.
6. Sink magnitudes are not small: per layer means ≈ 2–4, with layer 0 spanning
   [-9.0, +9.9]. Quantizing or dropping the sinks changes attention noticeably
   (and flashinfer doesn't support sinks at all, which you already note).

Tokenizer: a custom o200k variant, not o200k

The pre-tokenizer is a 3-split Sequence:

1. \p{N}{1,3} isolated,
2. CJK [\u4e00-\u9fa5\u3040-\u309f\u30a0-\u30ff]+ isolated,
3. a case-run (o200k-style) 6-alternative pattern, but with
   ?[\p{P}\p{S}]+[\r\n]* as the punctuation alternative.

That last point matters: real o200k uses ?[^\s\p{L}\p{N}]+[\r\n/]*. For Flint,
"HTTPRequest" → one token (the greedy [Upper]*[Lower]+ wins), and
"</tool_call>" → "</" "tool" "_call" ">" (the optional prefix of the letter
alternative absorbs the _). Anyone reusing a stock o200k pre-tokenizer will be
subtly off on camelCase and on punctuation runs. I validated my implementation on
a 27-case adversarial corpus against tokenizer.json (digits 1..7, CJK, case
runs, contractions, multi-space lookahead, the markers) — all 27 exact.

Also: the 12 chat markers (<think>, <tool_call>, <arg_key>, …) are
special: false, i.e. they are vocabulary pieces, not special tokens. That is
the right call (they can appear inside reasoning as single tokens), but note that
decode(..., skip_special_tokens=True) will still print them, while
skip_special_tokens=False will also print the 132 special: true role markers.
Choosing the decode policy by hand is necessary.

pad_token_id being <|end_of_text|> (150001) — the same id as the first eos —
is fine for batching but means "pad" and "stop" are the same token; worth a line
in the card.

eos_token_id as a list

[150001, 150003, 150005, 150018] = <|end_of_text|>, <|user|>,
<|observation|>, <|developer|>. This is a nice design (the turn control tokens
double as stop tokens), but many runtimes read eos_token_id as a single int
and will happily run past </think>-then-<|user|>. Documenting "four stop ids,
all required" would prevent truncated/overrun generations in non-SGLang stacks.

Chat template

• Role markers are inline: <|system|>…<|user|>…<|assistant|>, no
  separators, and the generation prompt is exactly <|assistant|> with no
  trailing newline. I reproduced it byte-for-byte against apply_chat_template
  on five cases (including nested JSON arguments and parallel calls), but it is
  easy to insert a newline "for symmetry" and change the model's behaviour.
• With tools, the preamble is emitted as a second <|system|> turn after the
  (optional) system message. Two consecutive system turns are unusual — some
  servers/clients assume a single leading system message, and some templates
  collapse them. Worth flagging in the card.
• flint_text() strips literal <think>/</think> from user content and
  HTML-escapes the role markers. Good hygiene; also means you cannot pass a
  literal </think> through history.
• Tool calls: <tool_call>{name}<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>,
  non-string values as JSON, string values raw. I render non-string values with
  json.dumps(..., ensure_ascii=False) and it matches the template.
• reasoning_content is a first-class history field (the template replays it as
  <think>…</think>). If a client drops it, multi-turn quality degrades; if a
  client keeps it, it must not also re-send it in content.

Quantization note (if you publish NVFP4 variants)

I converted to NVFP4 (4-bit body, per-16 E4M3 scales + global scale,
FP16 edges/router/norms/sinks). Two measurements against a BF16 CPU reference of
the same checkpoint (greedy, same ids):

• Dense side needed a residual plane. With attention + dense MLP + shared
  expert carrying a second NVFP4 plane (w − dequant(base), accumulated), the
  first-step top-1 logit at a 160-token prefix moved from −2.10 to −1.06 relative
  to the source; at a 56-token prefix, −3.70 → −0.99.
• Routed experts at 4 bits remained the dominant error. Top-1 agreed with the
  source at prefixes 8/16/32 and flipped at ~64 tokens. Doubling the expert bytes
  (+21 GB) does not fit a 32 GB card, so on this hardware the honest statement is
  "4-bit experts change long-prompt greedy continuations". Anyone shipping a
  quantized build may want ~8 bits on the routed experts, or to be explicit that
  long-context agentic behaviour is precision-sensitive.

(For reference, the control prompt "The capital of France is" matches the source
token-for-token for 32 greedy steps even at 4-bit experts, and the model reasons
first: <think>…The capital of France is Paris…</think>The capital of France is **Paris**.)

Minor

• max_position_embeddings: 262144 with 35 sliding layers at 513 means a hybrid
  KV cache: per-layer window sizes must be honoured, and hybrid_layer_pattern /
  is_hybrid_swa in the config are exactly what a runtime needs — thanks for
  shipping those.
• architectures: ["FlintForCausalLM"] with transformers_version: 5.12.1 but no
  modeling code in the repo: AutoTokenizer warns "model type flint → model
  type ''" on load. Harmless, but a one-line note ("transformers modeling code is
  not included; use the SGLang kit") is already in the README — maybe repeat it in
  the model card's "Getting started".

None of the above is a complaint about the model itself — it is a well-specified
checkpoint with unusually complete semantics (the FA4 sink patch in particular is
a gift). It is mostly documentation-level: the BF16-scale activations, the two
identical layer lists, the custom pre-tokenizer, and the four stop ids are the
four things that cost the most time to discover.

Happy to share the details of the reference implementation / the parity corpus if
that's useful.