Instructions to use bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4

SGLang

How to use bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 with Docker Model Runner:
```
docker model run hf.co/bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4
```

Interested in running this with SGLang on one DGX Spark

by codyknowscode - opened 17 days ago

Discussion

codyknowscode

17 days ago

I tried running this model on one DGX Spark, as SGLang 0.5.22 allows me to do PLE offloading to SSD while vLLM does not support that yet. Here is the command:

uv run sglang serve 
  --model-path ~/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-Flash-Next-NVFP4/snapshots/df3922b21fc28a8604cc9b31451495690d155194 \
  --mem-fraction-static 0.90 \
  --chat-template templates/Qwen-Froggeric.jinja \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3 \
  --default-chat-template-kwargs '{ "preserve_thinking": true, "enable_thinking": true }' \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass \
  --kv-cache-dtype fp8_e4m3 \
  --ple-offload-embedding \
  --ple-offload-backend file \
  --context-length 262000 \
  --max-running-requests 3 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --sleep-on-idle \
  --tp 1 \
  --trust-remote-code \
  --chunked-prefill-size 2048 \
  --mamba-radix-cache-strategy extra_buffer \
  --mamba-ssm-dtype float32 \
  --max-mamba-cache-size 40 \
  --page-size 64 \
  --host 0.0.0.0 \
  --port 19000

This setup worked for me with models like ukisai/Swift-1.5-Qwen3.8-Flash-Next-NVFP4 so I was expecting this one to work as well but I encountered a few issues:

The following error:

ValueError: Unable to find matching target for model.language_model.layers.0.linear_attn.in_proj_qkvz in the compressed-tensors config.

I managed to get past this step by replicating what other models have done with their ignore list in config.json:

@@ -180,7 +180,20 @@
         "format": "nvfp4-pack-quantized"
       }
     },
-    "ignore": [],
+    "ignore": [
+      "lm_head",
+      "re:^model\\.language_model\\.embed_tokens$",
+      "re:^model\\.language_model\\.hyper_connection_mixer.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.attn_hyper_connection.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.mlp_hyper_connection.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.linear_attn.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.self_attn.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.mlp\\.gate$",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.mlp\\.shared_expert.*",
+      "re:^model\\.language_model\\.layers\\.\\d+\\.ple\\..*",
+      "re:.*\\bmtp\\.(?!layers\\.\\d+\\.mlp\\.experts\\.).*",
+      "re:^model\\.visual.*"
+    ],
     "kv_cache_scheme": null,
     "sparsity_config": {},
     "transform_config": {},

OOM at this point:

[2026-10-24 08:12:15] PLE table: WILLNEED prefetch on for gathers of >= 2048 rows (row = 160 B)
[2026-10-24 08:12:15] PLE table: resident set capped at 8.0 GiB, checked every 30 s
[2026-10-24 08:12:15] using attn output gate!
[2026-10-24 08:12:18] Peak GPU memory before loading weights: 73.514 GiB
Multi-thread loading shards:  36% Completed | 5/14 [01:59<02:51, 19.02s/it][2026-10-24 08:14:46] PLE table: trimmed resident set 16.1 -> 0.0 GiB (budget 8.0 GiB)
Multi-thread loading shards: 100% Completed | 14/14 [09:01<00:00, 38.67s/it]
[2026-10-24 08:21:20] Memory increase during load_weights: -2.536 GiB

That's basically it, all the information SGLang prints, then it gets a 137 error. Memory looks like this:

while true; do echo "$(date +%T) $(free -m | sed -n 2p)"; sleep 1; done 

07:24:51 Mem:          124608       83139        5893          95       36766       41468
07:24:52 Mem:          124608       83336        5542          95       36920       41271
07:24:53 Mem:          124608       83533        5190          95       37075       41074
07:24:54 Mem:          124608       83733        4831          95       37233       40874
07:24:55 Mem:          124608       83965        4416          95       37417       40642
07:24:56 Mem:          124608       84197        3993          95       37608       40410
07:24:57 Mem:          124608       84434        3569          95       37796       40174
07:24:58 Mem:          124608       84668        3148          95       37982       39940
07:24:59 Mem:          124608       84246        4099          95       37453       40361
07:25:00 Mem:          124608       84592        4442          95       36778       40015
07:25:01 Mem:          124608       90512        4099          95       31169       34095
07:25:02 Mem:          124608       97278        2752          95       25750       27330
07:25:03 Mem:          124608      104175        2030          95       19574       20433
07:25:04 Mem:          124608      110970        2437          95       12371       13637
07:25:05 Mem:          124608      117460        2621          95        5733        7147
07:25:06 Mem:          124608      122531        2667          95         476        2077
07:25:07 Mem:          124608      124605         632          95         396           3
07:25:09 Mem:          124608      123976         632          94         222         632
07:25:11 Mem:          124608      123965         642          94         204         642
07:25:12 Mem:          124608      123956         651          94         196         651
07:25:13 Mem:          124608      123969         638          94         230         638
07:25:16 Mem:          124608      124384         925          52         211         223
07:25:17 Mem:          124608       65295       59975           0         233       59312
07:25:18 Mem:          124608        2224      123022           2         287      122383
07:25:19 Mem:          124608        2470      122720           3         397      122138
07:25:20 Mem:          124608        2477      122711           3         402      122131
07:25:21 Mem:          124608        2478      122710           3         402      122130
07:25:22 Mem:          124608        2505      122662           3         443      122102
07:25:23 Mem:          124608        2507      122659           3         444      122100
07:25:24 Mem:          124608        2510      122654           3         450      122097
07:25:25 Mem:          124608        2510      122653           3         450      122097
07:25:26 Mem:          124608        2509      122654           3         450      122098

AI debugger claims it's: CompressedTensorsW4A16Nvfp4MoE exhausts memory after weight loading.

I've tried different combinations of --mem-fraction-static (0.7..0.95), --max-mamba-cache-size (8..40), --context-length and --max-running-requests set to minimum values, --disable-cuda-graph, these had no impact.

Anything you can do to the model to help out with running it on one DGX Spark?

Adamiros

BottleCapAI org 11 days ago

Hi, thank you for the report!

The ignore list is now fixed in config.json (revision 47236b5). The reason why other checkpoints work is they are W4A4 quants. Ours is W4A16, which triggers an SGLang bug (sgl-project/sglang#38074) that keeps the expert weights in VRAM twice while the model loads. I'll look into the option to release a W4A4 checkpoint, but that will require some time for proper validation, since it is quite prone to quality loss. In the meantime, you can try using the workaround described below:

Save this as ~/sglang_38074_workaround/sitecustomize.py. The folder name is up to you; the file name must be sitecustomize.py, because Python loads a file of that name at startup from any folder on PYTHONPATH.

# Workaround for https://github.com/sgl-project/sglang/issues/38074 (SGLang <= 0.5.22, Qwen3.8-Flash-Next NVFP4).
# qwen4_exp's load_weights leaves a reference cycle that keeps every original expert weight alive while SGLang
# repacks the experts for Marlin, so loading peaks at ~137 GiB instead of ~75 GiB. Collecting it right after
# load_weights lets each original be freed as soon as its layer is repacked.
# Use: put this file in its own folder and start SGLang with PYTHONPATH=<that folder>.
import importlib.abc, importlib.util, sys


class _PatchQwen4ExpLoader(importlib.abc.MetaPathFinder):
    def find_spec(self, name, path, target=None):
        if name != "sglang.srt.models.qwen4_exp":
            return None
        sys.meta_path.remove(self)
        spec = importlib.util.find_spec(name)
        exec_module = spec.loader.exec_module

        def patched_exec_module(module):
            exec_module(module)
            cls = module.Qwen4ExpForConditionalGeneration
            load_weights = cls.load_weights

            def load_weights_then_collect(self, *args, **kwargs):
                import gc
                result = load_weights(self, *args, **kwargs)
                gc.collect()
                print("[sglang-38074-workaround] collected the loader's reference cycle", flush=True)
                return result
            cls.load_weights = load_weights_then_collect
        spec.loader.exec_module = patched_exec_module
        return spec


sys.meta_path.insert(0, _PatchQwen4ExpLoader())

Download the fixed revision. Only config.json changed, so the weights you already have are reused:

hf download bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-NVFP4 --revision 47236b56a42cfc923eb48616d234f8fab5016c2e

Start SGLang with PYTHONPATH pointing at the folder. This is your command with that prefix and the new snapshot path; everything else is unchanged:

PYTHONPATH="$HOME/sglang_38074_workaround" uv run sglang serve \
  --model-path ~/.cache/huggingface/hub/models--bottlecapai--ThinkingCap-Qwen3.8-Flash-Next-NVFP4/snapshots/47236b56a42cfc923eb48616d234f8fab5016c2e \
  --mem-fraction-static 0.90 \
  --chat-template templates/Qwen-Froggeric.jinja \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 3 \
  --default-chat-template-kwargs '{ "preserve_thinking": true, "enable_thinking": true }' \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass \
  --kv-cache-dtype fp8_e4m3 \
  --ple-offload-embedding \
  --ple-offload-backend file \
  --context-length 262000 \
  --max-running-requests 3 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --sleep-on-idle \
  --tp 1 \
  --trust-remote-code \
  --chunked-prefill-size 2048 \
  --mamba-radix-cache-strategy extra_buffer \
  --mamba-ssm-dtype float32 \
  --max-mamba-cache-size 40 \
  --page-size 64 \
  --host 0.0.0.0 \
  --port 19000

Check the server log for [sglang-38074-workaround] collected the loader's reference cycle. It confirms the workaround is active. Load weight end should then report about 74 GB for the model instead of the out-of-memory kill.

We validated the workaround on a DGX Spark ourselves with this command on SGLang 0.5.22, using the model's built-in chat template in place of your Qwen-Froggeric.jinja.

codyknowscode

11 days ago

Thanks, this worked perfectly!

I'll end up reducing --mem-fraction-static to 0.84-0.88 as 0.90 gives me 1.5M token budget and I only need around 800k (3 concurrent agents at 262k tokens) before I usually see diminishing returns. But it's great that it fits so well.

As for W4A4, I don't think you need to bother to be honest, since this is a SGLang bug I think it's fine to just wait out until it's fixed upstream. Or wait out until vLLM imlpements PLE offloading to disk.

codyknowscode changed discussion status to closed 11 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment