Solstice-AI

/

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-NVFP4

Like 2

Follow

Solstice-AI 24

I will double check, this was an automated upload via agents, sorry for the inconvenience

Yo, you're right — the README's vLLM command doesn't match this repo. Checked everything on the actual repo + upstream docs. Here's the full breakdown:

1. The --quantization modelopt_fp4 flag is wrong for this repo.

This checkpoint is compressed-tensors format (that's what config.json says: "quant_method": "compressed-tensors", "format": "nvfp4-pack-quantized"). It's NOT a ModelOpt checkpoint.

If someone runs the README command as-is with --quantization modelopt_fp4, vLLM 0.20+ throws:

ValueError: Quantization method specified in the model config (compressed-tensors)
does not match the quantization method specified in the `quantization` argument (modelopt).

Multiple independent sources confirm this:

• vLLM docs for --quantization: "If None, we first check the quantization_config attribute in the model config file."
• NVIDIA DGX Spark + vLLM playbook (Flaviu Vlaicu, Jun 2026): "For compressed-tensors NVFP4 (Unsloth, llm-compressor community builds), vLLM auto-detects — do not pass --quantization. Getting this wrong is the first boot failure."
• Unsloth's own Qwen3.6-27B-NVFP4 HF discussion + Vadi Taslim's measured benchmark (Aug 17, vLLM 0.27.1, RTX PRO 6000, 150 tok/s raw): both use --quantization compressed-tensors with compressed-tensors checkpoints.
• Reddit Qwen3.8-27B release day megathread (Aug 15): community examples use --quantization compressed-tensors for compressed-tensors checkpoints.

The correct vLLM command for THIS repo (removing the Anvil section — see point 2):

vllm serve Solstice-AI/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-NVFP4 \
  --max-model-len 262144 \
  --kv-cache-dtype turboquant_4bit_nc \
  --gpu-memory-utilization 0.90 \
  --dtype auto \
  --attention-backend flashinfer \
  --enable-prefix-caching \
  --trust-remote-code \
  --port 8000

Notes on the flags:

• No --quantization flag needed — vLLM auto-detects compressed-tensors from config.json. You CAN add --quantization compressed-tensors explicitly (that's what the DGX Spark recipe thread + Unsloth/Vadi benchmarks do), but omitting it also works.
• --kv-cache-dtype turboquant_4bit_nc — this is the actual community standard for 262K context on a 24-32 GB card with Qwen3.8-27B NVFP4. It's a stock vLLM 0.20+ native preset (not Anvil-only): 3.8× KV compression, +2.71% PPL (measured, near-lossless in practice for chat/reasoning). The vLLM TurboQuant docs list it as one of four native presets. The README's --kv-cache-dtype nvfp4 is wrong for vLLM — native NVFP4 KV on Blackwell SM12.x went through a long landmine chain (flashinfer JIT, cu13 symlinks, libcudart linking, etc.) per the DGX Spark + vLLM playbook.
• --attention-backend flashinfer — needed for NVFP4 weight kernels on Blackwell; the Unsloth Qwen3.8-27B-NVFP4 recipes use this.
• --trust-remote-code — this model has a custom tokenizer/chat template, same as the base.

2. The Anvil section is misleading and needs to come out.

I checked the actual Anvil repo (Solstice-Labs/anvil on GitHub). Anvil is a llama.cpp/ggml-based engine — src/anvil.cpp includes llama.h/ggml.h, loads .gguf files exclusively, validates them with validate_gguf(), and the HF pull path downloads .gguf files. The error for non-GGUF repos literally says: "safetensors-only repos need the HF converter: backends/llama-turbo/convert_hf_to_gguf.py".

The submodule backends/llama-turbo (gondaliyashreyan1/anvil-llama-turbo, a ggml-org/llama.cpp fork) does have convert_hf_to_gguf.py which supports NVFP4 compressed-tensors → GGUF conversion, and conversion/qwen.py has a Qwen3_5TextModel converter with MTP + NVFP4 weight transform support. But: that's a conversion to a NEW GGUF from the weights — NOT running this checkpoint directly, and definitely NOT re-quantizing an already-quantized model (that defeats the point).

So the README's "Option 1: Anvil Engine (Recommended)" reads like you can anvil run hf:... this NVFP4 safetensors checkpoint directly. You can't. Anvil's native format is GGUF. The thread guy's problem might be exactly this — he tried Anvil on the safetensors repo and hit the .gguf-only error.

The Anvil badge + mention should either be removed, or rewritten as a footnote: "Anvil loads GGUF only. To run with Anvil, convert this NVFP4 safetensors checkpoint to GGUF via convert_hf_to_gguf.py first (produces a fresh GGUF from the weights — not a re-quantization)."

3. Also fixing the config.json.

The config is missing text_config entirely (it has quantization_config, chat_template_jinja, tokenizer_config — but no text_config). The base model (DavidAU/...-NM-DAU) has a full text_config with hidden_size: 5120, num_hidden_layers: 64, num_attention_heads: 24, num_key_value_heads: 4, vocab_size: 248320, max_position_embeddings: 262144, mtp_num_hidden_layers: 1, the full 64-entry layer_types array, rope_parameters, etc. The official Qwen/Qwen3.8-27B config has the same structure. vLLM and transformers both use text_config to construct the model — without it, loaders fall back to guessing, which is fragile (tokenizer/chat-template can silently break, and some vLLM quantized paths validate against the config).

Also model_type: "qwen3_5_text" at top level is unusual — the base model and official Qwen3.8-27B both use model_type: "qwen3_5" at top level, with "model_type": "qwen3_5_text" only INSIDE text_config. So top-level should be "qwen3_5".

What I'm pushing as a PR on this repo:

• Add full text_config block to config.json (mirror from DavidAU base + Qwen3.8-27B official)
• Change top-level model_type from "qwen3_5_text" → "qwen3_5"
• Remove the Anvil "Option 1" section from README (it doesn't apply to this safetensors repo)
• Fix the vLLM command: drop --quantization modelopt_fp4, use no --quantization (auto-detect) + --kv-cache-dtype turboquant_4bit_nc, keep --trust-remote-code
• Update the "Anvil execution engine" attribution line in the citations to not claim this repo runs on Anvil natively

The quant weights + index + MTP file are all already correct on main. Just the metadata + docs were borked.