Qwen3.5-122B-A10B-NVFP4-GGUF

Qwen3.5-122B-A10B is an Apache-2.0 natively multimodal MoE model with 122B total and 10B active parameters, combining a vision encoder with a hybrid Gated DeltaNet and Gated Attention architecture. It supports 262K context, extensible to approximately 1M tokens with YaRN, and offers 201 languages and dialects, multi-step prediction, and default thinking mode. Key benchmarks include MMLU-Pro 86.7, GPQA Diamond 86.6, IFEval 93.4, SWE-bench Verified 72.0, Terminal-Bench 2.0 49.4, BFCL-V4 72.2, BrowseComp 63.8, and HLE with tools 47.5. Vision results include MMMU 83.9, MathVision 86.2, OmniDocBench 1.5 89.8, V* 93.2 with code interpreter, ScreenSpot Pro 70.4, and VideoMME 87.3. OSWorld-Verified reaches 58.0, while GPT-5-mini leads on CodeForces and the 27B sibling scores higher on selected benchmarks. Supports SGLang, vLLM, KTransformers, and Transformers, with MTP and tool-calling capabilities.

MTP — Speculative Decoding: Yes

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

File Name	Quant Type	File Size	File Link	Description
Qwen3.5-122B-A10B-NVFP4.gguf	NVFP4	82.7 GB	Link	NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 122B parameter model.
Qwen3.5-122B-A10B-NVFP4.mmproj.gguf	mmproj	912 MB	Link	Multimodal projection file. Used for vision/language models.

NVFP4 Test Run

Tested Qwen3.5-122B-A10B-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.

Performance Metrics

Metric	Value
Average Prompt Processing Speed	122.63 tokens/s
Average Generation Speed	128.58 tokens/s
Total Wall Time	183.7 seconds
Context Length	16,384 tokens
KV Cache (K)	q8_0
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU VRAM Capacity	96 GB
Peak GPU VRAM Usage	77.46 GiB
Peak GPU Utilization	94%
GPU Layers Offloaded	99 (-ngl 99)
Random Seed	42
Speculative Decoding	MTP (draft_n_max=2)
Test Runs	5

Test Shots

Shot	Test
Shot 1	C: Insert Node at End of Singly Linked List
Shot 2	Jogger and Train Passing Time
Shot 3	Two Pipes Filling a Tank
Shot 4	Students in Examination Rooms
Shot 5	Father and Son Ages

Run Inference

./build/bin/llama-cli \
  -m /path/to/models/Qwen3.5-122B-A10B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  -ngl 99 \
  -c 16384 \
  --cache-type-k q8_0 \
  --seed 42 \
  -n 8192 \
  -st \
  -p "Your prompt here"

Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.

Snapshots (5)

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods