TheCacheIsALie/RidgeFill-Projector-Qwen3.8-Flash-Next-i8
Updated
Approximate Science
community
AI & ML interests
We do what we must because we can
Recent Activity
View all activity
Organization Card
Project page · OpenReview · Paper (DOI) · llama.cpp fork · Data
Long prompts spend most of their time to first token in prefill. RidgeFill skips most of a model's upper layers there: a ridge regression, fitted once in closed form, predicts each late layer's input from one middle layer, and the layer writes its own cache from that prediction. A chosen set of rows, the last few thousand tokens and every decoding step still run the full model. Nothing is trained and no token is dropped.
Results
| Deployment | Time to first token | Quality |
|---|---|---|
| Qwen3.8-Flash-Next · vLLM | 1.50–1.60× faster at 9K–32K; a whole 64K request in 47.5 s instead of 82.6 s | RULER needle tasks 99.8 vs 100.0 |
| Qwen3.8-27B · llama.cpp · one A100 | 1.25–1.31× faster at 16K–120K | RULER −1.2 / −1.4 points, LongBench v2 −1.4 (95 % intervals include zero) |
On three 8–9B reference models, the fitted map retrieves 49.1 and 49.6 more filled-region RULER points than the identity fill on the two dense models. Every number comes from the paper, and the data repository recomputes each one from the raw results.
Maps
| Model | Maps | Runs in |
|---|---|---|
| Qwen3.8-27B | RidgeFill-Projector-Qwen3.8-27B-i8 | llama.cpp fork |
| Qwen3.8-Flash-Next | RidgeFill-Projector-Qwen3.8-Flash-Next-i8 | vLLM fork |
| Llama-3.1-8B-Instruct | Llama-3.1-8B-Instruct-RidgeFill-Projector | reference study |
| Qwen3-8B | RidgeFill-Projector-Qwen3-8B | reference study |
| Qwen3.5-9B | RidgeFill-Projector-Qwen3.5-9B | reference study |
Cite
@misc{johnston2026ridgefill,
author = {Johnston, Dylan and Smith, Robert P},
title = {{RidgeFill}: Training-Free Late-Layer Cache Fill for Faster Prefill},
year = {2026},
publisher = {Zenodo},
note = {Preprint},
doi = {10.5281/zenodo.23890247},
url = {https://doi.org/10.5281/zenodo.23890247}
}
Dylan Johnston and Robert P Smith, independent researchers. The design follows YOCO and DeepSeek-V4.1-Flash; kishida's KVA projector was the first public retrofit.
datasets 0
None public yet