fraQtl

Inference efficiency for transformer LLMs — end to end

KV-cache compression. Weight quantization. Runtime memory optimization. And the diagnostics to measure if it actually helps your stack — before you commit.

Run larger models, longer contexts, more concurrency, at the same quality.

---

The stack

Most "compression" projects ship one of the above. fraQtl ships all four, with reproducible receipts at every layer.

---

What we ship

🧩 Weight quantization — fraQtl calibration

Higher-fidelity GGUF quants. Same file size as a standard Q4_K_M; measurably closer to the full-precision teacher across code, math, chat, tool calling, and long-form text.

Measured on Qwen 3.6 35B-A3B (symmetric top-20 KLD vs the Q8 teacher, 400-record held-out slices):

At the same file size as a standard Q4_K_M, ~30% lower KLD across both slices. Reproducibility drift across three independent runs: 0.00000.

→ Qwen 3.6 35B-A3B (Q4_K_M) — Q4_K_M · 21.4 GB · drop-in for llama.cpp / Ollama / LM Studio / koboldcpp / Jan

🛠 KV-cache compression — llama.cpp runtime

Measured live-VRAM reduction on Mistral-7B-Instruct-v0.3 Q4_K_M vs a true fp16-KV baseline:

The benefit grows with context — exactly where fp16 KV becomes the bottleneck. Reproducibility drift across independent runs: ≤ 0.004 PPL.

🔍 fraQtl Diagnostic — free + open (Apache 2.0)

pip install fraqtl-diagnostic — three tools in one package:

• KV Savings Estimate — drop in any HF model id → projected memory freed, GPU-tier impact, max-context extension, relative cost-per-token. Instant, no GPU.
• Inference Readiness Scan — config-level: KV memory, YaRN status, backend support, benchmark checklist. Instant, no GPU.
• Compression Fingerprint — per-layer spectral analysis (γ, k95, regime tags, Shannon ceiling).

→ Run in your browser · PyPI · GitHub

---

Live demos

• 🪶 fraqtl-diagnostic — measure your model's compression headroom + projected KV savings in seconds
• 🔥 fraQtl-demo — fraQtl-compressed Mistral-7B running live with KV-cache compression

---

Approach

• Per-tensor protection policy — same total bit budget, smarter allocation
• Calibration tuned to measured optima (not "more is better")
• Standard llama.cpp kernel path — no patched runtime, no custom flags
• Deterministic builds — reproducibility drift 0.00000 across independent runs

The thesis: compression should be calibration-aware and workload-aware.

---

Links

• 🌐 Website: fraqtl.ai
• 📦 PyPI: pypi.org/project/fraqtl-diagnostic
• 📄 Paper: arxiv.org/abs/2604.11501
• 📬 Contact: contact@fraqtl.ai

Patent pending.