Instructions to use ngquocvinh/Nex-N2.5-mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ngquocvinh/Nex-N2.5-mini-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ngquocvinh/Nex-N2.5-mini-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ngquocvinh/Nex-N2.5-mini-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ngquocvinh/Nex-N2.5-mini-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Ollama
How to use ngquocvinh/Nex-N2.5-mini-GGUF with Ollama:
```
ollama run hf.co/ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
```
Unsloth Desktop

How to use ngquocvinh/Nex-N2.5-mini-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ngquocvinh/Nex-N2.5-mini-GGUF with Docker Model Runner:
```
docker model run hf.co/ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M
```

Lemonade

How to use ngquocvinh/Nex-N2.5-mini-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nex-N2.5-mini-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use ngquocvinh/Nex-N2.5-mini-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ngquocvinh/Nex-N2.5-mini-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ngquocvinh/Nex-N2.5-mini-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Nex-N2.5-mini GGUF

Community GGUF quantizations of nex-agi/Nex-N2.5-mini.

☕ If this GGUF made your day easier, a coffee would make mine.
Send a coffee ☕
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

About Nex-N2.5-mini

Nex-N2.5-mini is Nex-AGI's multimodal, agent-oriented model for long-horizon tasks. The upstream card describes the Nex-N2.5 family as supporting computer use, web browsing, visual grounding, coding, reasoning, and tool calling. It also documents image and video inputs through the official multimodal processor and chat template.

The upstream configuration identifies a Qwen3.5 Mixture-of-Experts model with 256 experts and 8 active experts per token, 40 text layers, and a 262,144-token text context configuration. The upstream repository presents the mini variant as a 35B-parameter BF16 model. The upstream deployment and benchmark details are available in the official model card.

This release contains text GGUF files plus a separate mmproj-Nex-N2.5-mini-F16.gguf vision projector. The local validation below uses a 4,096-token context and does not claim that the full configured context has been validated by this package.

Upstream Nex-N2.5 benchmark overview; the image and scores belong to the official model card.

This is a quantization-only release. No training, fine-tuning, merging, or weight modification other than GGUF conversion and quantization was performed. The Q8_0 file was quantized directly from the converted BF16 GGUF; the other published files used the same BF16 source and a model-specific importance matrix. No GGUF file was used as the source for another quantization.

Fidelity measurements

The table below compares every published text GGUF with the BF16 reference on a held-out WikiText pilot: eight chunks from wiki.test.raw and eight chunks from wiki.valid.raw, with a 4,096-token context, 512 batch/ubatch, 64 CPU threads, and the same Qwen3.5-compatible llama.cpp runtime. Values are averaged across the two splits. Lower Mean KLD, ΔPPL, and RMS Δp, and higher Top-1 agreement, indicate closer next-token behavior to BF16. The BF16 reference mean PPL was 6.684743 in this pilot.

File	Mean KLD ↓	Top-1 vs BF16 ↑	ΔPPL	RMS Δp
Nex-N2.5-mini-Q8_0.gguf	0.023277	94.523%	+0.986%	4.132%
Nex-N2.5-mini-Q6_K.gguf	0.027186	93.811%	-0.858%	4.676%
Nex-N2.5-mini-Q5_K_M.gguf	0.038910	92.636%	+0.036%	5.262%
Nex-N2.5-mini-Q4_K_M.gguf	0.063583	90.315%	+3.230%	6.732%
Nex-N2.5-mini-IQ4_NL.gguf	0.062792	90.273%	+1.378%	6.774%
Nex-N2.5-mini-IQ4_XS.gguf	0.065216	90.071%	+0.801%	6.854%
Nex-N2.5-mini-Q3_K_L.gguf	0.122060	86.334%	+3.083%	9.158%
Nex-N2.5-mini-Q3_K_M.gguf	0.129316	85.906%	+4.044%	9.427%
Nex-N2.5-mini-IQ3_M.gguf	0.172758	83.735%	+13.676%	11.555%
Nex-N2.5-mini-IQ3_S.gguf	0.154186	84.600%	+10.201%	10.701%
Nex-N2.5-mini-Q2_K.gguf	0.233133	80.709%	+10.839%	12.770%
Nex-N2.5-mini-Q2_K_S.gguf	0.278980	78.709%	+15.232%	14.023%
Nex-N2.5-mini-IQ2_XS.gguf	0.516971	71.590%	+51.851%	19.371%
Nex-N2.5-mini-IQ1_M.gguf	0.695877	66.064%	+69.116%	24.275%
Nex-N2.5-mini-Q1_0.gguf	8.616836	4.879%	+445003.665%	61.752%

For a general local profile, Q4_K_M is the practical starting point in this pilot. IQ4_NL and IQ4_XS are compact Q4-region alternatives. Q5_K_M and Q6_K are stronger quality/size choices, while Q8_0 is the highest-bit option. Q3_K_L, Q3_K_M, IQ3_M, and IQ3_S are lower-memory Q3-region compromises. Q2_K, Q2_K_S, IQ2_XS, IQ1_M, and Q1_0 are memory-constrained experimental profiles and should be checked against the intended workload.

These measurements describe next-token fidelity relative to BF16; they are not a direct percentage of capabilities retained. Instruction following, reasoning, multilingual behavior, formatting, vision, and tool-calling quality can vary by workload and should be evaluated separately when they matter.

The compact machine-readable results are available in reproducibility/quality-summary.tsv. Corpus hashes, conversion details, evaluation settings, runtime provenance, and artifact hashes are recorded in reproducibility/manifest.md.

Quick start

./llama-cli \
  -m Nex-N2.5-mini-Q4_K_M.gguf \
  --chat-template-file chat_template.jinja \
  --jinja \
  --reasoning off \
  -p 'Answer briefly in English: What is GGUF and why is it useful for running language models locally?' \
  -n 128 -c 4096 -ngl 99 --cpu-moe --fit on --fit-target 1024

--cpu-moe keeps the MoE weights on the CPU and is useful when the available GPU memory is smaller than the model working set. Omit it when the target machine has enough memory and the runtime configuration has been tested for that setup.

For the multimodal path, keep the text GGUF and the separate projector beside the executable:

./llama-mtmd-cli \
  -m Nex-N2.5-mini-Q4_K_M.gguf \
  --mmproj mmproj-Nex-N2.5-mini-F16.gguf \
  --image path/to/image.jpg \
  --jinja \
  -p 'Answer briefly in English: What is the main subject of this image?' \
  -n 64 -c 4096 -ngl 99 --cpu-moe --fit on --fit-target 1024

Reproducibility and validation

The source was locked to upstream revision 87420286149d9cce9bd46cd335ef9bda33c37c1b and converted directly from the upstream BF16 safetensors. The text converter used --no-mtp because this revision advertises MTP configuration but does not contain MTP tensors. The vision projector was converted separately to F16.

All fifteen published text GGUF files passed llama.cpp tensor checks, load, and English generation smoke tests. The BF16 reference also passed the same text smoke profile. The Q4_K_M text file and the included F16 projector passed a multimodal image smoke test. Runtime throughput is supplementary and is available as the compact reproducibility/runtime-summary.tsv; it is not a quality score or a replacement for the fidelity table.

Raw conversion, calibration, quantization, smoke-test, fidelity, and benchmark logs remain local and are intentionally not uploaded. Published artifact checksums are in SHA256SUMS.txt.

License and attribution

The upstream model metadata specifies Apache License 2.0. Preserve the upstream attribution and license when redistributing these derivative GGUF artifacts. These are community GGUF quantizations, not an official nex-agi/Nex-N2.5-mini release or endorsement.

Downloads last month: 3,325

GGUF

Model size

35B params

Architecture

qwen35moe

Hardware compatibility

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for ngquocvinh/Nex-N2.5-mini-GGUF

Base model

nex-agi/Nex-N2.5-mini

Quantized

(32)

this model