Instructions to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="bottlecapai/ThinkingCap-Qwen3.6-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bottlecapai/ThinkingCap-Qwen3.6-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("bottlecapai/ThinkingCap-Qwen3.6-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

SGLang

How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8
```

Multi-turn instability: FP8-quantized linear_attn recurrent projections cause agent "forgetting"

by galmsx - opened Jul 17

Discussion

galmsx

Jul 17

Thanks for the Ampere fix — it loads perfectly now on 2× RTX 3090, and MTP works great (mean accept length 3.36, matching the advertised 3.35).

However, I've noticed a stability issue in multi-turn agent workflows that doesn't occur with other quants of the same model (even the 4-bit ThinkingCap INT4 behaves correctly):

Observed behavior (within the first few turns, not long-context):

The model creates a file the user asked for, then in the next turn denies creating it and recreates it from scratch
It generates files, then gets "surprised" by them and removes them as if they were artifacts from a previous task
It sometimes reverts to working on an earlier subtask as if the last turn didn't happen

I don't see this all the time, but I've never encountered it on quants where these layers are kept in bf16 — including the official Qwen FP8 and the 4-bit ThinkingCap AutoRound.

Suspected cause: linear_attn.in_proj_qkvz and linear_attn.conv1d are quantized to FP8. These are the recurrent state projections of the Gated DeltaNet linear attention — unlike regular attention, errors
here don't just affect one token; they corrupt the hidden state that the model relies on to "remember" what it did across turns.

The official Qwen FP8 keeps these layers in bf16 for exactly this reason (336 linear_attn entries in the ignore list vs ThinkingCap FP8 only keeping in_proj_a/b).

Suggested fix: keep in_proj_qkvz and conv1d in bf16 as well. This is about ~100 MB of additional weight memory for a dramatic stability improvement in multi-turn agent use.

Happy to help test if a patched checkpoint is released.

d-roper

Jul 17

•

edited Jul 17

I have noticed the third behaviour as well. I currently have the model running analysis on multiple .sav files and sometimes it will ignore my prompt and instead work again on a prompt I sent previously.

*edit to provide more context

.sav or SPSS Statistics data files are commonly used for collection medical statistics
The harness I am using to run the model is opencode v1.18.3
I was running the model on a directory filled with 100+ of these files comparing specific values between them.

huginnfork

Jul 29

•

edited Jul 29

I have noticed the same issues with regards to model dementia.

I have created a FP8 quant that retains linear_attn in BF16, huginnfork/ThinkingCap-Qwen3.6-27B-FP8, as well as a full attnbf16 version huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16, which further leaves attention weights unquantized at bf16.

Happy to take in feedback to see if it fixes/alleviates multi-turn stability.

galmsx

Aug 3

Update: Solved — the issue was vLLM, not the model!

After days of debugging I found the problems are caused by vLLM bugs with hybrid GDN/Mamba models, not the finetune or quantization. ThinkingCap just makes them more visible. The key issues: MTP breaks prefix
caching (vllm#38182), Mamba state misalignment with MTP (vllm#45614, still unmerged), and empty blocks causing prompt drift (Qwen3.6#131).

Fix: run through Genesis vLLM patches (Sandermage/sndr_core_engine). I use a stable subset — PN346 (the key Mamba cache fix), PN345, PN125, PN133, PN402, PN290, P66, P60/P60B. Full GDN kernel patches (PN340/PN350) cause crashes with
compressed-tensors FP8 so I had to exclude them.

This completely fixed the "forgets which stage it's on" bug and allows MTP K=4 + prefix caching + max-num-seqs=2 to work simultaneously. On 2× RTX 3090: 1 agent 65-86 t/s, 2 agents 116-145 t/s, prefix cache 80-93%.

My docker launch command:

  docker run -d --name genesis-thinkingcap-fp8 \
    --entrypoint /bin/bash --gpus all --ipc=host --network host --shm-size=8g \
    -v ~/llm_models:/models:ro -v ~/.sndr:/plugin:ro \
    -v ~/.cache/triton:/root/.triton/cache \
    -v ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache \
    -v ~/.cache/huggingface:/root/.cache/huggingface:ro \
    -e SNDR_DEV_INSTALL_PLUGIN=1 \
    -e NCCL_P2P_DISABLE=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_DEBUG=WARN \
    -e OMP_NUM_THREADS=1 -e CUDA_DEVICE_MAX_CONNECTIONS=8 -e CUDA_MODULE_LOADING=LAZY \
    -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 \
    -e VLLM_USE_FLASHINFER_SAMPLER=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
    -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 -e VLLM_FLOAT32_MATMUL_PRECISION=high \
    -e VLLM_LOGGING_LEVEL=INFO -e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_STANDALONE_COMPILE=1 \
    -e VLLM_ENABLE_PREGRAD_PASSES=1 -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
    -e GENESIS_ENFORCE_VERSION_RANGE=1 -e GENESIS_OBSERVABILITY=1 -e GENESIS_BUFFER_MODE=shared \
    -e GENESIS_ENABLE_PN345=1 -e GENESIS_ENABLE_PN346=1 -e GENESIS_ENABLE_PN378_VOCAB_PAD_MASK=1 \
    -e GENESIS_ENABLE_PN133_MTP_EMPTY_OUTPUT_FIX=1 \
    -e GENESIS_ENABLE_PN402_SANITIZE_INVALID_DRAFT_TOKENS=1 \
    -e GENESIS_ENABLE_PN290_NUM_ACCEPTED_TOKENS_RACE=1 \
    -e GENESIS_ENABLE_PN125_HYBRID_FULL_AND_PIECEWISE=1 \
    -e GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1 \
    -e GENESIS_ENABLE_P72_PROFILE_RUN_CAP=1 -e GENESIS_PROFILE_RUN_CAP_M=4096 \
    -e GENESIS_ENABLE_P74_CHUNK_CLAMP=1 -e GENESIS_PREALLOC_TOKEN_BUDGET=4096 \
    -e GENESIS_ENABLE_P58_ASYNC_PLACEHOLDER_FIX=1 \
    -e GENESIS_ENABLE_P60_GDN_NGRAM_FIX=1 -e GENESIS_ENABLE_P60B_TRITON_KERNEL=1 \
    -e GENESIS_ENABLE_P61B_STREAMING_OVERLAP=1 \
    vllm/vllm-openai@sha256:6a93ae4316826f3dd8a92bee5442cbed50184a9cbd688d310f9e56ecad1eabeb \
    -c '
  set -euo pipefail
  cp -r /plugin /tmp/sndr_plugin_src
  pip install --quiet --no-deps -e /tmp/sndr_plugin_src
  python3 -m sndr.apply
  exec vllm serve /models/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 \
    --tensor-parallel-size 2 --disable-custom-all-reduce \
    --dtype auto --kv-cache-dtype fp8 \
    --enable-chunked-prefill --enable-prefix-caching \
    --gpu-memory-utilization 0.94 --max-model-len 250000 \
    --max-num-seqs 2 --max-num-batched-tokens 4096 \
    --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":4}" \
    --reasoning-parser qwen3 --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder --served-model-name current \
    --host 0.0.0.0 --port 8082
  '

Prerequisites: clone Sandermage/sndr_core_engine to ~/.sndr, install with pip install --no-deps -e ., pull the vLLM Docker image, create cache dirs.

Adamiros

BottleCapAI org Aug 3

•

edited Aug 3

Thank you for the update!

I've been working on reproducing your issue, and with my synthetic opencode benchmarks, the failure rates of our FP8 checkpoint and the one with linear attention weights kept in BF16 converged to the same values.

I'm glad you found a reliable solution in the meantime and hope you get a lot of value out of our model.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment