Instructions to use groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit")
model = AutoModelForCausalLM.from_pretrained("groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit

SGLang

How to use groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit with Docker Model Runner:
```
docker model run hf.co/groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit
```

Hivemind 32B Preview — AutoRound AWQ 4-bit

Overview

hivemind-32b-preview-AutoRound-AWQ-4bit is a weight-quantized checkpoint intended for efficient GPU inference, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

Field	Details
Format	AWQ / AutoRound
Source / base	the source checkpoint identified in the repository metadata
Intended task	`text-generation`
License	the license declared in the repository files

What is included

*.safetensors (4 files)
config.json
generation_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja
quantization_config.json
Additional configuration, tokenizer, processor, or shard files (11 visible artifacts total)

Quick start

vLLM (AWQ-compatible runtimes)

vllm serve groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit \
  --quantization awq_marlin \
  --dtype float16 \
  --trust-remote-code

The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.

Compatibility and responsible use

Use a runtime that explicitly supports this format, architecture, and modality.
Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
Review the source model card and license before redistribution or deployment.
Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

AutoRound AWQ 4-bit quantization of Hivemind 32B Preview, packaged as a Transformers/vLLM-compatible sharded safetensors checkpoint.

Quantization

Format: AWQ (quant_method: awq)
Quantizer: AutoRound 0.9.2
Weight bits: 4
Group size: 128
Symmetric: yes
Zero point: no
Weight dtype: integer
Runtime dtype: float16

Generation parameters

The checkpoint's generation configuration uses:

max_new_tokens = 2048
temperature = 0.6
top_p = 0.95
top_k = 20

Context and tested serving profile

Native maximum context: 40,960 tokens
Tested with vLLM 0.19.1
Tensor parallel size: 2
GPU memory utilization: 0.92
KV cache dtype: float16 / auto (not FP8)

Example:

vllm serve groxaxo/hivemind-32b-preview-AutoRound-AWQ-4bit \
  --served-model-name hivemind-32b-preview-autoround \
  --tensor-parallel-size 2 \
  --max-model-len 40960 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype auto \
  --dtype float16 \
  --quantization awq

On 24 GB GPUs, the tested 40,960-token profile requires two GPUs. A single 24 GB GPU loaded the weights but did not leave enough KV-cache memory for the full native context.

Notes

Review the upstream model's terms and intended-use guidance before deployment. This repository contains a quantized derivative and does not change the capabilities, limitations, or risks of the source checkpoint.

Downloads last month: 24

Safetensors

Model size

2B params

Tensor type

I32

BF16