DhruvalLabs/Qwen-AgentWorld-35B-A3B-GGUF

Qwen-AgentWorld-35B-A3B — GGUF Quantizations (VLM)

Quantized GGUF versions of Qwen/Qwen-AgentWorld-35B-A3B

This is a Vision-Language Model (VLM) — it can understand both text and images.

Works with llama.cpp · LM Studio · Jan · Ollama

Quantized by Dhptl on June 24, 2026 using quant-kit

---

This VLM requires TWO files — a text backbone GGUF and the mmproj vision encoder GGUF. Download one text backbone (e.g. Q4_K_M) and the mmproj file. Both must be in the same folder.

---

📦 Available Files

🔤 Text Backbone (quantized — pick ONE)

Filename	Size	RAM Required	Quant	Quality	Best For
Qwen-AgentWorld-35B-A3B-Q2_K.gguf	12.05 GB	~13.6 GB	Q2_K	⭐	Extreme compression, significant quality loss.
Qwen-AgentWorld-35B-A3B-Q3_K_L.gguf	16.87 GB	~18.4 GB	Q3_K_L	⭐⭐⭐	Slightly better than Q3_K_M, still a compromise.
Qwen-AgentWorld-35B-A3B-Q3_K_M.gguf	15.61 GB	~17.1 GB	Q3_K_M	⭐⭐⭐	Very small file. Quality drop noticeable.
Qwen-AgentWorld-35B-A3B-Q3_K_S.gguf	14.14 GB	~15.6 GB	Q3_K_S	⭐⭐	Very high compression, high quality loss.
Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf	19.71 GB	~21.2 GB	Q4_K_M ✅ Recommended	⭐⭐⭐⭐	Best balance of size and quality. Recommended for most users.
Qwen-AgentWorld-35B-A3B-Q4_K_S.gguf	18.52 GB	~20.0 GB	Q4_K_S	⭐⭐⭐½	Good speed/size balance, slight quality loss.
Qwen-AgentWorld-35B-A3B-Q5_K_M.gguf	23.03 GB	~24.5 GB	Q5_K_M	⭐⭐⭐⭐½	Better quality than Q4, slightly larger. Great if you have the RAM.
Qwen-AgentWorld-35B-A3B-Q5_K_S.gguf	22.33 GB	~23.8 GB	Q5_K_S	⭐⭐⭐⭐	Large but accurate.
Qwen-AgentWorld-35B-A3B-Q6_K.gguf	26.56 GB	~28.1 GB	Q6_K	⭐⭐⭐⭐⭐	Near-perfect quality, very large.
Qwen-AgentWorld-35B-A3B-Q8_0.gguf	34.37 GB	~35.9 GB	Q8_0	⭐⭐⭐⭐⭐	Closest to original quality. Use when RAM is not a concern.

🖼️ Vision Encoder — mmproj (always required, always F16)

Filename	Size	Notes
Qwen-AgentWorld-35B-A3B-mmproj-f16.gguf	0.00 GB	Always F16 — vision encoder is not quantized

⚠️ You need BOTH files — one text backbone + the mmproj — to run this VLM.

---

⚡ Speed Benchmarks

Run python benchmark.py --model Qwen-AgentWorld-35B-A3B to generate results.

---

🚀 How to Use

LM Studio (Easiest — GUI)

1. Search for Dhptl/Qwen-AgentWorld-35B-A3B in LM Studio
2. Download the Q4_K_M text file and the mmproj file
3. Load the model — LM Studio automatically uses both files

Ollama

ollama run dhptl/qwen-agentworld-35b-a3b

llama.cpp CLI — Text + Image

# Download both files to the same directory, then:
./llama-llava-cli \
  -m Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf \
  --mmproj Qwen-AgentWorld-35B-A3B-mmproj-f16.gguf \
  --image /path/to/your/image.jpg \
  -p "Describe this image in detail." \
  -n 512

llama.cpp CLI — Text only (no image)

./llama-cli \
  -m Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf \
  -p "You are a helpful assistant." \
  --conversation

Python — llama-cpp-python

from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava16ChatHandler

# Load VLM with mmproj
chat_handler = Llava16ChatHandler(clip_model_path="./Qwen-AgentWorld-35B-A3B-mmproj-f16.gguf")
llm = Llama(
    model_path="./Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf",
    chat_handler=chat_handler,
    n_gpu_layers=-1,
    n_ctx=4096,
    logits_all=True,
)

# Text + image inference
response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                {"type": "text",      "text":      "What do you see in this image?"}
            ]
        }
    ]
)
print(response["choices"][0]["message"]["content"])

---

🔍 VLM Architecture

This model uses a two-component architecture:

Component	File	Purpose
Text Backbone	Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf	Language understanding & generation
Vision Encoder (mmproj)	Qwen-AgentWorld-35B-A3B-mmproj-f16.gguf	Image feature extraction (always F16)

Why is mmproj always F16? The vision encoder maps image pixels to token embeddings. Quantizing it causes visible visual artifacts and degraded image understanding. It stays at F16 (half precision) which is already very efficient at ~1-2GB for most models.

---

🔍 About GGUF Quantization

Format	Bits/weight	Quality
Q3_K_M	~3.3	⭐⭐⭐
Q4_K_M	~4.5	⭐⭐⭐⭐ ← recommended
Q5_K_M	~5.6	⭐⭐⭐⭐½
Q8_0	~8.5	⭐⭐⭐⭐⭐

---

💬 Community & Feedback

Found an issue? Open a Discussion in the Community tab.

If useful, please:

• ⭐ Star quant-kit on GitHub
• 👍 Like this model on HuggingFace