Instructions to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H
# Run inference directly in the terminal:
llama cli -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H
# Run inference directly in the terminal:
llama cli -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H
# Run inference directly in the terminal:
./build/bin/llama cli -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Use Docker

docker model run hf.co/steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Llama new
LM Studio
Jan
Ollama
How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with Ollama:
```
ollama run hf.co/steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H
```
Unsloth Desktop

How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with Docker Model Runner:
```
docker model run hf.co/steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H
```

Lemonade

How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Run and chat with the model

lemonade run user.Qwen3-4B-Instruct-2507-MP-GGUF-Q6_K_H

List all available models

lemonade list

Hermes Agent

How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use steampunque/Qwen3-4B-Instruct-2507-MP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "steampunque/Qwen3-4B-Instruct-2507-MP-GGUF:Q6_K_H" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Mixed Precision GGUF layer quantization of Qwen3-4B-Instruct-2507 by Qwen

Original model: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

The hybrid quant employs different quantization levels on a per layer basis to increase flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simultaneously optimize quantized size and model performance. These quants were specifically optimized for the Qwen3 4B Instruct 2507 edge model for same file size as Q6_K quant with improved performance.

The layer quants are as follows:

   Q5_K_L : Q5_K_M + attn_o = Q6_K
   Q6_K_S : Q6_K
   Q6_K_M : Q6_K_S + attn_v = Q8_0, ffn_d = Q8_0
   Q6_K_L : Q6_K_M + attn_o = Q8_0

   LAYER_TYPES='[
   [0 ,"Q6_K_M"],[1 ,"Q6_K_S"],[2 ,"Q6_K_S"],[3 ,"Q6_K_S"],[4 ,"Q5_K_L"],[5 ,"Q5_K_M"],
   [6 ,"Q5_K_M"],[7 ,"Q5_K_M"],[8, "Q5_K_M"],[9, "Q5_K_M"],[10,"Q5_K_M"],[11,"Q5_K_M"],
   [12,"Q5_K_L"],[13,"Q5_K_L"],[14,"Q5_K_L"],[15,"Q5_K_L"],[16,"Q5_K_L"],[17,"Q5_K_L"],
   [18,"Q6_K_S"],[19,"Q6_K_S"],[20,"Q6_K_S"],[21,"Q6_K_S"],[22,"Q6_K_S"],[23,"Q6_K_S"],
   [24,"Q6_K_M"],[25,"Q6_K_M"],[26,"Q6_K_M"],[27,"Q6_K_M"],[28,"Q6_K_M"],[29,"Q6_K_M"],
   [30,"Q6_K_L"],[31,"Q6_K_M"],[32,"Q6_K_L"],[33,"Q6_K_L"],[34,"Q6_K_L"],[35,"Q8_0"  ]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

These quants were optimized for accurate reasoning performance across a small set of curated test prompts.

Long context test:

The model fails to solve the long context problem on a 4070 with Q8_0 KV quant : https://huggingface.co/datasets/steampunque/benchlm/blob/main/Qwen3_Runescape_Massive_Prompt_85k.txt

The 4B thinking model https://huggingface.co/steampunque/Qwen3-4B-Thinking-2507-Hybrid-GGUF does solve the problem correctly with solid reasoning in its think block.

Comparison:

Quant	size	PPL	Comment
Q6_K	3.3e9	10.6	default embed and output
Q6_K_H	3.3e9	10.6	improved reasoning over Q6_K

Evals of the model are available at https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3-4B-Instruct-2507.Q6_K_H.gguf	Q6_K_H	3.3e9 B	Q6_K size with improved reasoning

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040

Downloads last month: 92

GGUF

Model size

4B params

Architecture

qwen3

Hardware compatibility

6-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for steampunque/Qwen3-4B-Instruct-2507-MP-GGUF

Base model

Qwen/Qwen3-4B-Instruct-2507

Quantized

(376)

this model