gatilin/bitdance_binary_tokenizer_d32c128

BitDance d32c128 binary image tokenizer

Standalone native encoder, sign quantizer and GAN decoder from the official BitDance tokenizer release. The safetensors file is byte-for-byte identical to ae_d32c128.safetensors; its original SHA-256 and pinned source revision are recorded in extraction.json. No generation backbone is included.

This is the published BitDance codec. It is not asserted to be the semantic-distilled UniWeTok checkpoint merely because both describe 128-bit representations.

Load and reconstruct

from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download("gatilin/bitdance_binary_tokenizer_d32c128")
sys.path.insert(0, folder)
from component import load_model
codec = load_model(folder)
# RGB pixels: NCHW, [-1, 1], spatial dimensions divisible by 32.
pixels = torch.linspace(-1, 1, 3*256*256).reshape(1,3,256,256)
with torch.inference_mode():
    signs = codec.encode(pixels)  # [1,128,8,8], each value -1 or +1
    reconstructed = codec.decode(signs)

Install requirements.txt and run python validate.py in the downloaded directory. component.py also provides explicitly documented byte-packing helpers for storing 128 sign bits. Their byte convention is a portable bundle helper, not an inferred mapping to upstream AR vocabulary IDs.

Validation

Strict native state-dict loading, official-file SHA-256 equality, 256px encode/decode, binary alphabet, exact bit roundtrip, fixed-seed bit-decoding parity, native stochastic decoding, and rectangular 256x320 encode/decode passed on CPU in float32. Exact results and measured fixture MSE are in validation.json. These are component correctness tests on deterministic synthetic inputs, not claims about perceptual quality or text-to-image benchmark scores.

The GAN decoder injects fresh Gaussian noise even in eval mode. Set the same torch RNG seed before decoding if you need reproducible output; this native behavior is preserved.

The 2^128 space is represented implicitly by 128 binary channels, not a materialized embedding table. The published model uses the original convolutional implementation in autoencoder.py, not the separate attention-augmented autoencoder_uni.py. The only native-file edit changes an absolute filesystem-helper import into a package-relative import; source and code revisions are in the provenance files.

Source and license

• Weights: https://huggingface.co/shallowdream204/BitDance-Tokenizer
• Code: https://github.com/shallowdream204/BitDance
• Paper: https://arxiv.org/abs/2602.14041

Apache-2.0; original notices are retained. Extraction, tests, loader and documentation added for this standalone distribution.