Instructions to use sakamakismile/Nex-N2.5-Max-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use sakamakismile/Nex-N2.5-Max-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Use Docker

docker model run hf.co/sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use sakamakismile/Nex-N2.5-Max-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sakamakismile/Nex-N2.5-Max-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sakamakismile/Nex-N2.5-Max-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Ollama
How to use sakamakismile/Nex-N2.5-Max-GGUF with Ollama:
```
ollama run hf.co/sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
```
Unsloth Desktop

How to use sakamakismile/Nex-N2.5-Max-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use sakamakismile/Nex-N2.5-Max-GGUF with Docker Model Runner:
```
docker model run hf.co/sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M
```

Lemonade

How to use sakamakismile/Nex-N2.5-Max-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Nex-N2.5-Max-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use sakamakismile/Nex-N2.5-Max-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use sakamakismile/Nex-N2.5-Max-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "sakamakismile/Nex-N2.5-Max-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Nex-N2.5-Max GGUF (llama.cpp)

GGUF quantizations of nex-agi/Nex-N2.5-Max (1.6T-parameter text-only MoE, 384 routed experts / top-6, 61 layers, 1M context, MTP head), made at Lna-Lab on 2026-09-09 for pure-CPU inference on a 1 TiB DDR5 workstation.

Nex-N2.5-Max uses the DeepSeek-V4-Pro architecture unchanged (DeepseekV4ForCausalLM, hidden 7168, MoE inter 3072, indexer 64 heads / top-1024, hyper-connections x4, 3 hash-routed layers). Any llama.cpp build that runs DeepSeek-V4 runs these files.

Files

folder	recipe	size	BPW	notes
`Q4K-exps-Q8dense/`	routed experts Q4_K (pure), everything else (attention, shared expert, indexer, router, embeddings, output) Q8_0	898 GB (836 GiB)	4.57	fastest on CPU; needs ~880 GiB of free RAM
`Q3_K_M/`	stock llama.cpp `Q3_K_M` mixture	748 GB (697 GiB)	3.81	fits alongside other residents; slower on CPU (no repacked kernel for Q3_K)

Each folder is a standard llama.cpp split (-00001-of-000NN.gguf); point -m at the first shard.

Measured (not estimated)

Threadripper PRO 9985WX (64 cores), 1 TiB DDR5-8ch, no GPU (-ngl 0 -t 64), llama.cpp master 465e49b, warm page cache, single stream:

quant	prompt tok/s	generation tok/s
Q4K-exps-Q8dense	27.8	5.35
Q3_K_M	25.6	3.04

Q3_K_M is slower although smaller: llama.cpp has an AVX-512 repacked kernel for Q4_K but not for Q3_K. The repack also copies the Q4_K experts into anonymous memory (RssAnon ≈ 811 GiB for the Q4 file), so mmap does not save you RAM with that file. Cold first run (page faults) is ~0.5 tok/s; warm the file first (cat file > /dev/null in parallel chunks, ~50 s on a fast RAID) or just wait for the first generation.

Quality spot check (Japanese haiku with one-line commentary, thinking off): both quants produce clean Japanese with sensible kigo; we could not tell them apart on that task. The model thinks in English when enable_thinking is on (gpt-oss style), and answers in the user's language.

Running

llama-server -m Q4K-exps-Q8dense/Nex-N2.5-Max-Q4K-exps-Q8dense-00001-of-00021.gguf \
  -ngl 0 -t 64 -c 16384 -fa on --jinja --port 8030

The chat template is embedded (from the original repo) and supports chat_template_kwargs: enable_thinking (default true) and thinking_mode (interleaved | full | drop).

Known issue (llama.cpp 465e49b): /v1/chat/completions can return HTTP 500 "The model produced output that does not match the expected peg-native format" after a complete answer. The model is fine; the server-side output parser is not. Work-around: render with POST /apply-template and generate with POST /completion.

How these were made

The original checkpoint stores routed experts as FP8 e4m3 with 128x128 block scales stored as F32 (expert_dtype: fp8, scale_fmt: ue8m0), whereas the official DeepSeek-V4 release packs experts as MXFP4. llama.cpp's convert_hf_to_gguf.py (DeepSeek-V4 path) dequantizes the FP8 tensors, drops the scales, and then tries to repack experts as MXFP4 -> KeyError: Missing routed expert tensors. It also reads F32 scales as e8m0 bytes.

llama.cpp-convert-fp8-experts.patch (against conversion/deepseek.py, master 465e49b) fixes both:

float-typed block scales are used as values;
FP8-dequantized experts are stacked (lazily, 384 per projection) and emitted as ordinary expert tensors (they land as Q8_0 in the intermediate, per the converter's existing FP8 policy).

Pipeline: convert_hf_to_gguf.py --outtype bf16 (intermediate: experts Q8_0 + rest BF16, 1.70 TB, 2h18m) -> llama-quantize --allow-requantize:

# Q4K-exps-Q8dense
llama-quantize --allow-requantize --pure --output-tensor-type q8_0 --token-embedding-type q8_0 \
  --tensor-type 'attn_.*=q8_0' --tensor-type 'ffn_.*_shexp=q8_0' --tensor-type 'indexer.*=q8_0' --tensor-type 'ffn_gate_inp=q8_0' \
  intermediate.gguf out.gguf Q4_K_M 48
# Q3_K_M
llama-quantize --allow-requantize intermediate.gguf out.gguf Q3_K_M 48

No importance matrix was used. A stock Q4_K_M was also baked (951 GB, 4.84 BPW) but is not published: it does not fit in 1 TiB RAM next to anything else, and its attention weights are 4-bit anyway.

MTP layer is not included (the converter's main-model path skips mtp.*; those tensors are BF16 in the source).

Credits

Model: Nex-AGI, Apache-2.0. Architecture: DeepSeek.
Quantization and converter patch: YUKI (Claude Fable 5.1), researching together with Ken at Lna-Lab — @Tono_Ken3

Downloads last month: -

GGUF

Model size

1.6T params

Architecture

deepseek4

Hardware compatibility

3-bit

View +1 variant

Model tree for sakamakismile/Nex-N2.5-Max-GGUF

Base model

nex-agi/Nex-N2.5-Max

Quantized

(1)

this model