Instructions to use mradermacher/Qwen3-Reranker-0.6B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/Qwen3-Reranker-0.6B-GGUF", device_map="auto")

sentence-transformers

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("mradermacher/Qwen3-Reranker-0.6B-GGUF")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Ollama:
```
ollama run hf.co/mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-Reranker-0.6B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use mradermacher/Qwen3-Reranker-0.6B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "mradermacher/Qwen3-Reranker-0.6B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Usage instruction

by Bearsaerker - opened Jun 6, 2025

Discussion

Bearsaerker

Jun 6, 2025

I tried to run this model with --pooling rerank in llama-server but it tells me that it does not find the sep_token and therefore I can't use it. How can I use this model as a reranker with llama cpp server?

mradermacher

Owner Jun 6, 2025

according to the help output, there is no "reranker" type, only rank? and shouldn't the default be taken from the model?

if it's not that, llama.cpp might simply not yet support the model (i.e. either conversion produced a broken gguf, or llama-server does have no support).

Dargor381

Jun 9, 2025

See this issue https://github.com/ggml-org/llama.cpp/issues/13820 and related merge request https://github.com/ggml-org/llama.cpp/pull/14029

mradermacher

Owner Jun 9, 2025

Thanks a lot for finding those issues. That's a lot of changes, but it's not clear to me if the model needs to be requanted. Once this is merged and it doesn't work, you (anybody) can drop us a note and we will redo the model.

CHE0NAI

Jul 10, 2025

See this issue https://github.com/ggml-org/llama.cpp/issues/13820 and related merge request https://github.com/ggml-org/llama.cpp/pull/14029
It can indeed start normally, but it cannot return the correct answer.

mradermacher

Owner Jul 10, 2025

•

edited Jul 10, 2025

did you manually merge them and this is with the merged patches? from the issues, it seems they are not resolved.

however, if you want, i can delete this repo and simply requantise just in case to see if it helps, as it's not much effort (this will make this discussion go away as well). just drop me a note here, and i will do it. when the discussion is gone you will know that it's been redone.

CHE0NAI

Jul 10, 2025

did you manually merge them and this is with the merged patches? from the issues, it seems they are not resolved.

however, if you want, i can delete this repo and simply requantise just in case to see if it helps, as it's not much effort (this will make this discussion go away as well). just drop me a note here, and i will do it. when the discussion is gone you will know that it's been redone.

Yes with no doubt, I manually merged them, and this is what exactly the result after merge. I don't know if I am the only one who encountered this issue because I did not find many relevant comments under your post. Therefore, I think it is more meaningful to keep this repository, as it will bring together more people with related issues to discuss and solve this phenomenon.

At the same time, thank you for your continuous support and providing solutions.

mradermacher

Owner Jul 10, 2025

It does seem to require redoing the quantization, but presumably, only after the changes have been merged. @nicoboss do you have a take on this?

nicoboss

Jul 10, 2025

It unfortionately needs to be requantized once the PR is marged. Even more unfortionate is how hacky the rerank detection proposed in this PR is. They have to look at the README.md and check for certain text to detect if it is a rerank model which is so ugly. With the author of this PR on sick leave It is uncertain if and when this is getting merged.

mradermacher

Owner Jul 11, 2025

holy shit

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment