latent-context/0.6b-4b-LCLM-16x

0.6b-4b LCLM, 16× compression

Latent Context Language Model: an encoder–decoder compressor described in End-to-End Context Compression at Scale (arXiv).

The text to compress should be wrapped between <|memory_start|> and <|memory_end|>.

Running these checkpoints requires the LCLM codebase: https://github.com/LeonLixyz/LCLM. Standard transformers.AutoModel / vllm.LLM will not load this format on its own.

Versions

• v1.1 (current, main): improves the agent with latent context and online latent compaction. The SFT mixture adds tool-use trajectories and synthetic trajectories that call an expand tool on compressed documents. This is the checkpoint evaluated in the current version of the paper.
• v1.0: the first release (May 2026). Load it with LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-16x", revision="v1.0").

Quick load

from latent_context import LCLM

model = LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-16x")

prompt = (
    "<|memory_start|>"
    "<long document, code, or text to compress>"
    "<|memory_end|> "
    "Summarize the document above."
)
# model.generate(...): see inference/hf.py in the codebase

For faster encoding, inference/optimized_encoder.py adds FlashAttention, torch.compile and CUDA graphs to the encoder (inference only).

vLLM serving (two-stage CLI)

The vLLM path runs the encoder and the decoder in separate processes that hand off via a .pt file on disk. Running both in one process OOMs: vLLM grabs all GPU memory at init, leaving none for the HF encoder.

# Step 1: HF encoder over a jsonl of prompts → embeds.pt
python -m inference.vllm_inference.encode \
    --checkpoint latent-context/0.6b-4b-LCLM-16x \
    --prompts-jsonl prompts.jsonl --out embeds.pt

# Step 2: vLLM decoder reads embeds.pt → completions.jsonl
python -m inference.vllm_inference.decode \
    --checkpoint latent-context/0.6b-4b-LCLM-16x \
    --embeds-pt embeds.pt --out completions.jsonl

See inference/examples/README.md in the codebase for the prompts.jsonl schema and an end-to-end RULER NIAH eval driver.

Configuration

field	value
encoder	Qwen/Qwen3-Embedding-0.6B
decoder	Qwen/Qwen3-4B-Instruct-2507
compression_ratio	16
encoder_window_size	1024
pooling	mean
encoder_mask_type	causal
boundary_overlap	0
adapter_type	mlp

Code: https://github.com/LeonLixyz/LCLM