athrael-soju commited on
Commit
c209084
·
1 Parent(s): 2dfb251

Declare causal retrieval attention contract

Browse files
Files changed (2) hide show
  1. README.md +10 -9
  2. config.json +1 -0
README.md CHANGED
@@ -170,8 +170,9 @@ pip install causal-conv1d flash-linear-attention
170
  ```
171
 
172
  - `causal-conv1d` + `flash-linear-attention` are required (the hybrid layers import them at runtime).
173
- - Attention must be **SDPA**. Retrieval runs bidirectional attention on the full-attention layers;
174
- `flash_attention_2` silently ignores the 2-D mask and scores as if causal.
 
175
 
176
  ## Usage
177
 
@@ -206,20 +207,20 @@ print(scores.shape) # torch.Size([2, 2])
206
  ```
207
 
208
  `config.json` carries `dim=320`, so `custom_text_proj` is sized correctly at load, with no manual
209
- config edits needed.
 
210
 
211
  ## Serving with vLLM
212
 
213
  vLLM serves this model natively through its pooling runner (the `ColQwen3_5` architecture), returning
214
  the per-token multi-vectors for late-interaction scoring. It requires a vLLM build that includes the
215
- ColQwen3.5 retrieval-correctness fix ([vllm-project/vllm#46108](https://github.com/vllm-project/vllm/pull/46108),
216
- merged 2026-06-22): build from `main`, or use a release tagged after that date. The fix runs the
217
- backbone bidirectionally and restores the projection bias, so vLLM reproduces the `transformers`
218
- reference within run-to-run noise. The server uses the stock chat/image processor, so the ColQwen3.5
219
  prompt contract is applied **client-side**: wrap each page image in the instruction template, append
220
  the query-augmentation tokens to each query, and set the visual-token budget through
221
- `mm-processor-kwargs`. Prefix caching and chunked prefill must be **off** (bidirectional attention and
222
- the GatedDeltaNet hybrid both break the causal-prefix invariant).
223
 
224
  ```python
225
  import torch
 
170
  ```
171
 
172
  - `causal-conv1d` + `flash-linear-attention` are required (the hybrid layers import them at runtime).
173
+ - This merged checkpoint was selected and evaluated with **causal** full-attention layers. Load its
174
+ `retrieval_attention_contract="causal"` metadata without overriding it.
175
+ - Use **SDPA**, matching the evaluation runtime of record.
176
 
177
  ## Usage
178
 
 
207
  ```
208
 
209
  `config.json` carries `dim=320`, so `custom_text_proj` is sized correctly at load, with no manual
210
+ config edits needed. It also carries `retrieval_attention_contract="causal"`; a conflicting runtime
211
+ override is an error.
212
 
213
  ## Serving with vLLM
214
 
215
  vLLM serves this model natively through its pooling runner (the `ColQwen3_5` architecture), returning
216
  the per-token multi-vectors for late-interaction scoring. It requires a vLLM build that includes the
217
+ projection-bias fix and explicit retrieval-attention metadata support. The handler reads
218
+ `retrieval_attention_contract="causal"` from this checkpoint and constructs all eight full-attention
219
+ layers as `AttentionType.DECODER`; it must reject missing, conflicting, or unsupported contract
220
+ metadata. The server uses the stock chat/image processor, so the ColQwen3.5
221
  prompt contract is applied **client-side**: wrap each page image in the instruction template, append
222
  the query-augmentation tokens to each query, and set the visual-token budget through
223
+ `mm-processor-kwargs`. Prefix caching and chunked prefill remain off for the GatedDeltaNet hybrid.
 
224
 
225
  ```python
226
  import torch
config.json CHANGED
@@ -4,6 +4,7 @@
4
  ],
5
  "image_token_id": 248056,
6
  "model_type": "qwen3_5",
 
7
  "text_config": {
8
  "attention_bias": false,
9
  "attention_dropout": 0.0,
 
4
  ],
5
  "image_token_id": 248056,
6
  "model_type": "qwen3_5",
7
+ "retrieval_attention_contract": "causal",
8
  "text_config": {
9
  "attention_bias": false,
10
  "attention_dropout": 0.0,