Instructions to use celertor-ai/QWQ-32B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use celertor-ai/QWQ-32B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="celertor-ai/QWQ-32B-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("celertor-ai/QWQ-32B-FP8")
model = AutoModelForCausalLM.from_pretrained("celertor-ai/QWQ-32B-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use celertor-ai/QWQ-32B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "celertor-ai/QWQ-32B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "celertor-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/celertor-ai/QWQ-32B-FP8

SGLang

How to use celertor-ai/QWQ-32B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "celertor-ai/QWQ-32B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "celertor-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "celertor-ai/QWQ-32B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "celertor-ai/QWQ-32B-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use celertor-ai/QWQ-32B-FP8 with Docker Model Runner:
```
docker model run hf.co/celertor-ai/QWQ-32B-FP8
```

QWQ-32B-FP8 / README.md

kai-celertorai

Upload 17 files

dbe8dbe verified 10 months ago

preview code

Raw

History Blame

1.44 kB

	---
	license: apache-2.0
	base_model:
	- Qwen/QwQ-32B
	base_model_relation: quantized
	library_name: transformers
	tags:
	- qwq
	- fp8
	---
	# Model Overview

	## Description

	FP8 Quantized QwQ-32B.

	## Evaluation

	The test results in the following table are based on the MMLU benchmark.

	In order to speed up the test, we prevent the model from generating too long thought chains, so the score may be different from that with longer thought chain.

	In our experiment, the accuracy of the FP8 quantized version is almost the same as the BF16 version, and it can be used for faster inference.

	\| Data Format \| MMLU Score \|
	\|:---\|:---\|
	\| BF16 Official \| 61.2 \|
	\| FP8 Quantized \| 61.2 \|
	\| Q8_0 (INT8) \| 59.1 \|
	\| AWQ (INT4) \| 53.4 \|

	## Quickstart
	We recommend using the Chitu inference framework(https://github.com/thu-pacman/chitu) to run this model.

	Here provides a simple command to show you how to run QwQ-32B-FP8.

	```bash
	torchrun --nproc_per_node 1 \
	--master_port=22525 \
	-m chitu \
	serve.port=21002 \
	infer.cache_type=paged \
	infer.pp_size=1 \
	infer.tp_size=1 \
	models=QwQ-32B-FP8 \
	models.ckpt_dir="your model path" \
	models.tokenizer_path="your model path" \
	dtype=float16 \
	infer.do_load=True \
	infer.max_reqs=1 \
	scheduler.prefill_first.num_tasks=100 \
	infer.max_seq_len=4096 \
	request.max_new_tokens=100 \
	infer.use_cuda_graph=True
	```

	## Contact

	solution@qingcheng.ai