gatilin/longcat_next_visual_codec

LongCat-Next: complete standalone visual codec and flow refiner

This bundle retains the native dNaViT visual encoder, bridge, RVQ, visual embedding interface, base pixel decoder, full flow-refinement Transformer, refiner VAE and conditioning projection. The language/audio backbone is excluded. It can reconstruct images from pixels or compatible RVQ IDs; it cannot synthesize images from text by itself.

Load the base codec

from huggingface_hub import snapshot_download
import sys, torch
folder=snapshot_download("gatilin/longcat_next_visual_codec")
sys.path.insert(0,folder)
from component import load_model,encode_pixels,decode_codes
codec=load_model(folder)
pixels=torch.linspace(0,1,3*112*112).reshape(1,3,112,112)
with torch.inference_mode():
    ids,grid=encode_pixels(codec,pixels)
    reconstructed=decode_codes(codec,ids,grid)

patchify_pixels accepts NCHW RGB in [0,1], applies native CLIP normalization, duplicates the temporal frame, and packs the original Qwen2.5-VL patch order. Spatial dimensions must be divisible by 28; no implicit resize is performed. decode_codes returns native CLIP-normalized RGB tensors. Decode values back to RGB with the same mean/std when displaying.

RVQ uses 8 residual steps with one shared learned codebook, 16384 valid codes per step and one padding entry. IDs are [positions,8] in native residual order. Do not add the language vocabulary offset before passing IDs to this codec. The original global visual_offset=150581 is preserved in source_config.json for integration with the full LongCat backbone.

Full flow refinement

from component import load_refiner,refine_codes
refiner=load_refiner(folder,device="cuda")
with torch.inference_mode():
    refined=refine_codes(codec,refiner,ids,grid,seed=1019,num_inference_steps=28)

The refiner is approximately 7.934 GB of native BF16 Transformer weights, plus the shared VAE and condition projection. Base reconstruction and flow refinement are separate stages; noise and sampling settings affect the latter. Pixel dimensions divisible by both 28 and 16 (e.g. 112,224,448) suit the two-stage image path. CPU compatibility is included; CUDA performance and numerical equivalence are not benchmarked.

Exact extraction and alias preservation

The source main-model visual subtree has 425 serialized tensor entries and 5,314,251,776 payload bytes. Its 21 duplicate codebook/EMA-buffer entries were individually downloaded and verified byte-exact by SHA-256 against their canonical shared values. We store 404 entries, 2,025,358,336 payload bytes, and restore the native shared object/storage aliases in the loader. No quantization or dtype conversion is performed during extraction.

decoder/model.safetensors contains all base-decoder parameters, the refiner VAE and condition projection. refiner/model.safetensors contains every flow Transformer parameter. The additional outer visual_model copy in the upstream image-decoder container is unused by the native inference decode path and is excluded; the actual input visual encoder from the main checkpoint is retained. Tensor mapping and per-tensor source hashes for every stage are included.

Validation

Run python validate.py and python validate_refiner.py after installing requirements. The reports cover strict native loading; shared codebook object/storage; exact patch-layout inversion; 112x112 and rectangular56x84 pixels→8-stage RVQ→base reconstruction; visual embedding interface; refiner VAE and condition projection; full flow-weight loading; conditioned BF16 Transformer forward with reference-image latents; finite Euler update; and conditioning sensitivity. Tests use deterministic synthetic inputs and small spatial grids. They do not establish full-resolution perceptual quality, VQA or text-to-image benchmark performance.

CPU uses the preserved Qwen2.5-VL window-attention implementation because newer transformers removed the original get_window_index entry point. FlashAttention calls have a segment-preserving PyTorch fallback; native data layout, attention boundaries and weights remain explicit. The input visual model retains BF16 weights and F32 codebooks; base decoder/VAE are cast to F32 at runtime on CPU.

Sources: official weights, official code, paper. Source revisions, modifications, separate MIT/Apache notices and exact extraction records accompany the release.

The combined CPU refine_codes integration smoke run was stopped on the test host because of memory/swap pressure while the encoder and full refiner were resident together. No end-to-end pipeline pass is claimed. Base-codec and complete refiner component tests above passed independently.