Instructions to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
# Run inference directly in the terminal:
llama cli -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
# Run inference directly in the terminal:
llama cli -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
# Run inference directly in the terminal:
./llama-cli -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Use Docker

docker model run hf.co/gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

LM Studio
Jan

vLLM

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Ollama
How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with Ollama:
```
ollama run hf.co/gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
```
Unsloth Desktop

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with Docker Model Runner:
```
docker model run hf.co/gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF
```

Lemonade

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Run and chat with the model

lemonade run user.Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "gsrunion/Ornith-1.0-35B-ROCmFP4-STRIX_LEAN-DFLASH-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

DFlash-graft slower than plain LEAN on Beelink GTR9 Pro (Ryzen AI Max+ 395)

by rusl-dautov - opened 23 days ago

Discussion

rusl-dautov

23 days ago

Hi @gsrunion !

I'm running Ornith-1.0-35B-STRIX_LEAN-DFLASH on Beelink GTR9 Pro (Ryzen AI Max+ 395, 128GB LPDDR5x-8000, gfx1151).

Problem: DFlash is SLOWING DOWN my inference on my hardware!

My results:

Without DFlash (plain LEAN): 63.1 tok/s
With DFlash (any config): 30-45 tok/s (slower!)
Your claimed speed: 111 tok/s

I tried many spec-* parameter combinations, but DFlash always makes things worse on my system. The draft model either:

Generates fast (277 tok/s) but acceptance rate is only 35%
Generates slow (37 tok/s) with good acceptance rate (97%)

Questions:

What are your EXACT spec-* parameters for DFlash?
What's your commit hash for rocmfp4-llama (dflash-graft branch)?
Are you using any special environment variables?
Is your TDP unlocked to 120W+ in BIOS?
Do you use ROCm or Vulkan for DFlash?

My system seems incompatible with DFlash. Can you share your exact working configuration?

Thanks!

gsrunion

Owner 23 days ago

Full-Stack Build & Version Report

 Host: Proxmox node us (192.168.1.11) — pve-manager 9.2.2 (b9984c6d90a4bd80), kernel 7.0.2-6-pve
 GPU: AMD Strix Halo [Radeon 8060S/8050S] 1002:1586 (rev c1) → ISA gfx1151, 126 GB VRAM
 CPU: AMD Ryzen AI MAX+ 395 (32 threads)

 ────────────────────────────────────────────────────────────────────────────────

 1. Inference engine — llama.cpp fork (rocmfp4 / DFlash)

 The core dependency is gsrunion/rocmfp4-llama, branch refs/heads/dflash-graft. Two distinct things to note:

 Canonical BUILD-INFO manifests (in the ROCm package tarballs on this box):

 ┌────────────────────┬──────────────────────────────────────────┬───────┬────────────┬────────────────┐
 │ Build              │ Commit                                   │ ROCm  │ GPU target │ Version        │
 ├────────────────────┼──────────────────────────────────────────┼───────┼────────────┼────────────────┤
 │ llama-v010 (older) │ adcdf4f39d2aead93ce87e68dde36293b0aeb0d2 │ 7.2.1 │ gfx1151    │ libllama 0.0.5 │
 ├────────────────────┼──────────────────────────────────────────┼───────┼────────────┼────────────────┤
 │ llama-new          │ be6c7614a7d71675d4fbf8bfccae7349200fb856 │ 7.2.4 │ gfx1151    │ libllama 0.0.6 │
 └────────────────────┴──────────────────────────────────────────┴───────┴────────────┴────────────────┘

 Actually-deployed binaries (what's running now, higher version):

 ┌────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
 │ Component          │ Version                                                                                           │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ llama-server       │ version 9316 (5c493ec38)                                                                          │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ libllama           │ 0.0.9316 (also ships 0.0.9315)                                                                    │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ libggml            │ 0.11.1                                                                                            │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ Compiler           │ GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04)                                                         │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ Build path         │ /src/build-dflash                                                                                 │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ System fingerprint │ b9316-5c493ec38                                                                                   │
 ├────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
 │ DFlash impl        │ draft-dflash — n_max=16 (clamped to 15), n_min=0, p_min=0.75, block_size=16, mask_token_id=248077 │
 └────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

 2. ROCm runtime stack (loaded in-process)

 Installed at /opt/rocm-7.2.4 (.info/version = 7.2.4). Runtime-loaded libraries (/proc/*/maps):

 ┌─────────────┬──────────────────────────────────┐
 │ Library     │ Loaded soname                    │
 ├─────────────┼──────────────────────────────────┤
 │ HIP         │ libamdhip64.so.7.2.70204         │
 ├─────────────┼──────────────────────────────────┤
 │ hipBLAS     │ libhipblas.so.3.2.70204          │
 ├─────────────┼──────────────────────────────────┤
 │ hipBLASLt   │ libhipblaslt.so.1.2.70204        │
 ├─────────────┼──────────────────────────────────┤
 │ rocBLAS     │ librocblas.so.5.2.70204          │
 ├─────────────┼──────────────────────────────────┤
 │ rocSOLVER   │ librocsolver.so.0.7.70204        │
 ├─────────────┼──────────────────────────────────┤
 │ HSA         │ libhsa-runtime64.so.1.18.70204   │
 ├─────────────┼──────────────────────────────────┤
 │ comgr       │ libamd_comgr.so.3.0.0            │
 ├─────────────┼──────────────────────────────────┤
 │ rocRoller   │ librocroller.so.1.0.0            │
 ├─────────────┼──────────────────────────────────┤
 │ rocprofiler │ librocprofiler-register.so.0.6.0 │
 ├─────────────┼──────────────────────────────────┤
 │ ROCtX       │ libroctx64.so.4.1.70204          │
 └─────────────┴──────────────────────────────────┘

 3. Models

 ┌────────────────┬──────────────────────────────────┬───────────────────────────────────┐
 │                │ Ornith-1.0-35B-STRIX_LEAN-DFLASH │ Qwen3.6-35B-A3B-STRIX_LEAN-DFLASH │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ GGUF           │ v3                               │ v3                                │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ Arch           │ qwen35moe                        │ qwen35moe                         │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ n_params       │ 35,046,516,864 (~35.0B)          │ 35,891,157,632 (~35.9B)           │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ n_embd         │ 2048                             │ 2048                              │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ n_vocab        │ 248,320                          │ 248,320                           │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ n_ctx_train    │ 262,144                          │ 262,144                           │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ Active         │ ~3B (A3B MoE)                    │ ~3B (A3B MoE)                     │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ File size      │ 19,369,215,520 B (19.4 GB)       │ 19,818,808,704 B (19.8 GB)        │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ Embedded draft │ 782,818,560 B                    │ 782,818,560 B                     │
 ├────────────────┼──────────────────────────────────┼───────────────────────────────────┤
 │ Base/quant     │ Unsloth/rocmfp4, qwen35moe       │ Unsloth/rocmfp4, qwen35moe        │
 └────────────────┴──────────────────────────────────┴───────────────────────────────────┘

 4. Container runtime config (both containers identical except model path)

 - Ubuntu/Debian 13 (trixie), kernel shared with host
 - LD_LIBRARY_PATH=/opt/llama-dflash:/opt/rocm-7.2.4/lib
 - --ctx-size 262144 --n-gpu-layers 99 --no-mmap --parallel 4 --no-kv-unified
 - qwen adds --reasoning-format deepseek
 - GGML_HIP_ENABLE_UNIFIED_MEMORY=1 deliberately NOT set
 - GTT memory guard: FOOTPRINT_GIB=32 + 12 margin

 ────────────────────────────────────────────────────────────────────────────────

 Direct answer to your question — should ornith and qwen be closer?

 Architecturally: yes. In practice: no — and I verified why.

 ### The decisive finding

 Their raw (non-DFlash) decode speed is IDENTICAL: both = 45.7 tok/s (isolated, draft off). That's exactly what you'd expect from near-identical architecture.

 But under DFlash speculative decoding, they diverge massively by task:

 ┌─────────────────────────┬─────────────────┬─────────────────┐
 │ Task @ code             │ ornith          │ qwen            │
 ├─────────────────────────┼─────────────────┼─────────────────┤
 │ plain-LEAN              │ 45.7            │ 45.7            │
 ├─────────────────────────┼─────────────────┼─────────────────┤
 │ DFlash @512             │ 72              │ 117             │
 ├─────────────────────────┼─────────────────┼─────────────────┤
 │ DFlash draft acceptance │ 364/383 (95.0%) │ 438/443 (98.9%) │
 └─────────────────────────┴─────────────────┴─────────────────┘

 ### Why

 Since raw decode is identical and the whole gap appears only under DFlash, the difference is the quality of each model's embedded draft at predicting its own target — the acceptance rate. Qwen's draft verifies ~98.9% of proposed tokens on code, ornith's only ~95%. More validations
 per draft block → more tokens finalized per GPU step → higher tok/s.

 The relationship also reverses by task (prose: ornith 48.6 > qwen 39.5; math: ornith 63.6 > qwen 58.2), confirming it's per-model draft-matching quality, not a fixed compute-speed difference. The slight file-size difference (~2%) is too small to explain a ~60% swing — the swing is
 spec-decoding efficiency, which tracks the ~3.9% acceptance delta multiplied across every block.

 Bottom line: ornith is a Qwen3.6-35B-A3B build (same arch, ~same params, identical vocab — so same FLOPs and same 45.7 plain floor). But its embedded draft is a worse predictor for code than qwen's, so DFlash accelerates it less (72 vs 117). Both are on the canonical dflash-graft
 commit stack above; the 111 tok/s claim only reproduces on qwen, and specifically on draft-friendly code workloads.

 State now: qwen (303) serving on 192.168.1.49:8080; ornith (301) and ds4 (302) stopped.

gsrunion

Owner 23 days ago

Seems to me ornith and qwens speeds got mixed up.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment