Instructions to use truespirit/Ornith-1.5-397B-MXFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use truespirit/Ornith-1.5-397B-MXFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="truespirit/Ornith-1.5-397B-MXFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("truespirit/Ornith-1.5-397B-MXFP4")
model = AutoModelForMultimodalLM.from_pretrained("truespirit/Ornith-1.5-397B-MXFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use truespirit/Ornith-1.5-397B-MXFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "truespirit/Ornith-1.5-397B-MXFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "truespirit/Ornith-1.5-397B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/truespirit/Ornith-1.5-397B-MXFP4

SGLang

How to use truespirit/Ornith-1.5-397B-MXFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "truespirit/Ornith-1.5-397B-MXFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "truespirit/Ornith-1.5-397B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "truespirit/Ornith-1.5-397B-MXFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "truespirit/Ornith-1.5-397B-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use truespirit/Ornith-1.5-397B-MXFP4 with Docker Model Runner:
```
docker model run hf.co/truespirit/Ornith-1.5-397B-MXFP4
```

Ornith-1.5-397B-MXFP4

MXFP4-quantized build of ornith-ai/Ornith-1.5-397B — the same model at roughly half the disk and memory footprint, targeting single-GPU serving on 288 GB AMD Instinct MI355X.

Checkpoint size: ~239 GB
Quantization: OCP MXFP4, block size 32
Quantizer: AMD Quark 0.12.post1
GPU tested: 1× AMD Instinct MI355X, 288 GB HBM
Tensor parallelism: TP=1
Architecture: Qwen3.5 MoE / Ornith 1.5
Routed MoE expert weights are MXFP4; attention, shared experts, gates, MTP, visual components and lm_head remain BF16.

Recommended serving

The checkpoint is tested with SGLang ROCm 7.2.4 on MI355X. AITER and native NEXTN/MTP are supported.

docker run --rm \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --security-opt seccomp=unconfined \
  --ipc=host \
  --shm-size=32g \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e SGLANG_USE_AITER=1 \
  -e SGLANG_USE_AITER_UNIFIED_ATTN=1 \
  -e AITER_FLYDSL_FORCE=1 \
  -e SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
  lmsysorg/sglang-rocm:v0.5.18-rocm724-mi35x-20260830 \
  python3 -m sglang.launch_server \
    --model-path truespirit/Ornith-1.5-397B-MXFP4 \
    --tp 1 \
    --attention-backend aiter \
    --page-size 16 \
    --reasoning-parser deepseek-r1 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --mem-fraction-static 0.95 \
    --max-running-requests 4 \
    --context-length 81920 \
    --host 0.0.0.0 \
    --port 8000

The server exposes an OpenAI-compatible API.

Measured on 1× MI355X

With the configuration above:

~180–200 output tok/s single-stream decode
~290 output tok/s aggregate at concurrency 4
~0.1 s TTFT on the tested 16K-prompt workload
~86K tokens measured single-request input ceiling
two concurrent ~62K-token inputs verified

Results are serving benchmarks for this specific hardware/runtime configuration and are not model-quality benchmarks.

For general reasoning, temperature=0.6, top_p=0.95 is a useful starting point. For coding/agent workloads, temperature=1.0, top_p=0.95 is also suitable.

Quantization

Only routed MoE expert weights (*.mlp.experts.*) were quantized to MXFP4. Components kept at higher precision follow the AMD Qwen3.5 MXFP4 recipe.

Quantization can cause numerical differences from the source FP8 checkpoint.

License

MIT, inherited from the base model. See the original ornith-ai/Ornith-1.5-397B model card for model details, intended use and limitations.

Downloads last month: 941

Safetensors

Model size

210B params

Tensor type

BF16

Model tree for truespirit/Ornith-1.5-397B-MXFP4

Base model

ornith-ai/Ornith-1.5-397B

Quantized

(12)

this model