Instructions to use EryriLabs/phonellm-alpha-1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use EryriLabs/phonellm-alpha-1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use EryriLabs/phonellm-alpha-1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EryriLabs/phonellm-alpha-1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EryriLabs/phonellm-alpha-1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Ollama
How to use EryriLabs/phonellm-alpha-1-GGUF with Ollama:
```
ollama run hf.co/EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
```
Unsloth Desktop

How to use EryriLabs/phonellm-alpha-1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use EryriLabs/phonellm-alpha-1-GGUF with Docker Model Runner:
```
docker model run hf.co/EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M
```

Lemonade

How to use EryriLabs/phonellm-alpha-1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.phonellm-alpha-1-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use EryriLabs/phonellm-alpha-1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use EryriLabs/phonellm-alpha-1-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "EryriLabs/phonellm-alpha-1-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

phonellm-alpha-1 — GGUF

Unofficial GGUF quantizations of pipecat-ai/phonellm-alpha-1.

⚠️ Disclaimer: This is a community conversion. This repository is not affiliated with, endorsed by, or maintained by Pipecat AI or NVIDIA. All credit for the model goes to the original authors — please see the official model card for full details, intended use, and limitations.

About the model

PhoneLLM Alpha 1 is a voice-agent / phone-conversation model with tool-use and function-calling support, fine-tuned by Pipecat AI from NVIDIA-Nemotron-3-Nano-30B-A3B (a Nemotron-H hybrid Mamba–attention mixture-of-experts model, 30B total / ~3B active parameters).

Conversion details

Converted with llama.cpp release b10672 (convert_hf_to_gguf.py → F16, then llama-quantize)
Source weights: BF16 safetensors from the official repository (revision 8e76aaa)

Files

File	Quant	Size
phonellm-alpha-1-F16.gguf	F16	63.2 GB
phonellm-alpha-1-Q8_0.gguf	Q8_0	33.6 GB
phonellm-alpha-1-Q4_K_M.gguf	Q4_K_M	24.5 GB
phonellm-alpha-1-Q3_K_M.gguf	Q3_K_M	19.8 GB

⚠️ Important: these quants are larger than their names suggest

This architecture's weight tensors have column counts of 1856, 2688 and 3712 — none of which are divisible by 256, the superblock size that llama.cpp's K-quants require. As a result llama-quantize reports 134 of 401 tensor(s) required fallback quantization and substitutes the legacy block-32 types defined in its fallback table (src/llama-quant.cpp): Q3_K → Q4_0, Q4_K → Q5_0, Q6_K → Q8_0.

Only the 29 tensors with a 4096 column count keep their requested K-quant type. The actual composition by size is:

File	Real tensor mix
Q8_0	99.9% Q8_0
Q4_K_M	66.7% Q5_0, 32.5% Q8_0, 0.7% Q4_K
Q3_K_M	49.4% Q5_0, 45.3% Q4_0, 2.5% Q5_1, 1.9% Q8_0, 0.5% Q3_K

Practical implications:

The files are larger than you would expect from these quant names on a conventional dense model. Q3_K_M is ~19.8 GB, not the ~13 GB a 30B Q3_K_M would normally be.
Quality should be better than the names imply, since the fallbacks are higher-precision than the requested types.
No Q6_K is published. Its fallback type is Q8_0, so a Q6_K build came out 99.1% Q8_0 tensors at 33.5 GB — within 0.1 GB of the Q8_0 file and marginally worse. Use Q8_0 instead.

Usage

Requires a recent llama.cpp build (b10672 or newer) with Nemotron-H support:

llama-cli -m phonellm-alpha-1-Q4_K_M.gguf -cnv

Or serve an OpenAI-compatible API:

llama-server -m phonellm-alpha-1-Q4_K_M.gguf

License

PhoneLLM Alpha 1 is released under the BSD 2-Clause License (see LICENSE).

PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License (see LICENSE_NVIDIA.txt). Per Section 3 (Redistribution) of that license, this repository includes a copy of the NVIDIA Nemotron Open Model License and retains the NVIDIA copyright and attribution notices. The BSD 2-Clause terms apply to Pipecat AI's modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.

Downloads last month: 192

GGUF

Model size

32B params

Architecture

nemotron_h_moe

Hardware compatibility

3-bit

4-bit

8-bit

16-bit

Model tree for EryriLabs/phonellm-alpha-1-GGUF

Base model

nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Finetuned

pipecat-ai/phonellm-alpha-1

Quantized

(6)

this model