CapyCTL

Run more models than your GPU can hold.

CapyCTL deploys, parks, wakes and switches models on vLLM, SGLang, TensorFold, llama.cpp or any server you describe, on GPU machines you own, behind one OpenAI-compatible endpoint.

Get started Docs GitHub Recipes Benchmarks

v0.1.10 · Apache-2.0 · one binary, written in Rust · Linux x86-64 and ARM64 · NVIDIA GPUs

$ capyctl list deployments
NAME           STATE    READY   REVISION   HOSTS
coding-small   ready    1/1     3          gpu-box
chat-large     ready    1/1     1          workstation
vision-small   parked   0/1     2          gpu-box

Example output. A parked model releases its GPU memory and wakes on the next request for it.

• More models than memoryMost GPU machines hold one or two models at a time. CapyCTL parks the idle ones and wakes the one a request asks for, making room by parking or stopping an idle model.
• One endpointOpenAI-compatible chat completions with streaming and tool calls, and decision models answering /v1/systemone with probabilities, behind an API key. Your apps keep one base URL and pick a model by name.
• Your engines, your weightsRegister the vLLM, SGLang, TensorFold or llama.cpp installation you already have, or any other server from a manifest. Models come from a directory or from the Hub, pinned to a commit when you deploy.
• Bigger than one GPUSpread one model across the identical GPUs of a machine, or across two machines over a direct link, with each card's memory accounted for.

Quick start

On Linux with an NVIDIA GPU and a vLLM, SGLang or TensorFold environment, or a llama.cpp build:

curl -fsSL https://edurdias.github.io/capyctl/install.sh | sh
capyctl engine add ~/venvs/vllm   # or SGLang, TensorFold, or ~/llama.cpp/build/bin/llama-server
capyctl start standalone

In another shell, deploy a model from the Hub. Three fields are enough:

# qwen3-4b.yaml
name: qwen3-4b
engine: vllm
model: {hf: Qwen/Qwen3-4B-Instruct-2507}

capyctl deploy model --file qwen3-4b.yaml --activate --wait

Any OpenAI client can now send qwen3-4b as the model name (Make a request). To free its GPU memory, run capyctl park deployment qwen3-4b; the next request wakes it. One server can also drive several GPU machines (Run on several machines).

Measured recipes

capyctl-recipes has 38 deployment files, each run through CapyCTL on the hardware it names. Every recipe gives the engine version, the pinned model revision, the commands from capyctl engine add to a first answer, ready times, memory and a capyctl-bench report. Some of them:

Decision models answer POST /v1/systemone with probabilities instead of generating tokens. Clef-Flash on llama.cpp v0.7.0 decides in 203 ms, and basal-2.0-4.5B, run as a custom engine, in 90 ms (one question about a 200-token text, on a GB10).

Rows that serve different checkpoints are not a one-to-one engine comparison; the comparisons below are.

Comparisons

The same model and machine on several engines, on two engine versions, or through CapyCTL against the engine launched directly. Each one states its method and publishes its evidence.

Every report, with its charts, is in the benchmarks Space, and every measured point is in the bench-results dataset.

In this organization

• benchmarks: the capyctl-bench report of every recipe and comparison, filterable by engine, model and GPU.
• bench-results: every point of those reports in one table: decode speed, time to first token, prompt processing and memory by prompt length and streams, and decision-model latency and quality.
• Models with CapyCTL recipes: 27 Hub checkpoints the recipes and comparisons serve, each with a note linking its numbers.
• FrogNano-4B-2609-MLX-4bit: MLX 4-bit, text only, for TensorFold on NVIDIA GPUs; 319 tokens/s at 8 streams on an RTX 4090 Laptop GPU.
• FrogNano-4B-2609-NVFP4: NVFP4 with FP8 attention, for vLLM, SGLang and TensorFold; up to 416 tokens/s at 8 streams on a GB10.

The converted checkpoints' cards say what changed from the original and how it was made. capyctl-bench, the benchmark behind all of the above, is a standard-library Python script: point it at any CapyCTL endpoint to measure your own models.

Status

0.1.10 is early software, and breaking changes are possible between minor versions while the version starts with 0. One model can span the identical GPUs of one machine (qualified on two) or two machines over a direct link; larger groups are not qualified yet. CapyCTL is driven by its CLI and a management API, on Linux with NVIDIA GPUs. Parking relies on engine development controls, kept on loopback behind a per-launch key; they are not production-hardened, and a host can opt out. Questions and bug reports go to GitHub. Licensed under Apache-2.0.