Instructions to use leeroy-jankins/buddy with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use leeroy-jankins/buddy with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("leeroy-jankins/buddy", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use leeroy-jankins/buddy with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf leeroy-jankins/buddy:Q4_K_M # Run inference directly in the terminal: llama cli -hf leeroy-jankins/buddy:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf leeroy-jankins/buddy:Q4_K_M # Run inference directly in the terminal: llama cli -hf leeroy-jankins/buddy:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf leeroy-jankins/buddy:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf leeroy-jankins/buddy:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf leeroy-jankins/buddy:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf leeroy-jankins/buddy:Q4_K_M
Use Docker
docker model run hf.co/leeroy-jankins/buddy:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use leeroy-jankins/buddy with Ollama:
ollama run hf.co/leeroy-jankins/buddy:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use leeroy-jankins/buddy with Docker Model Runner:
docker model run hf.co/leeroy-jankins/buddy:Q4_K_M
- Lemonade
How to use leeroy-jankins/buddy with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull leeroy-jankins/buddy:Q4_K_M
Run and chat with the model
lemonade run user.buddy-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Update README.md
Browse files
README.md
CHANGED
|
@@ -57,7 +57,7 @@ used.
|
|
| 57 |
The application configuration points to the following default local model path:
|
| 58 |
|
| 59 |
```text
|
| 60 |
-
|
| 61 |
```
|
| 62 |
|
| 63 |
The runtime loads the model through `Llama(...)` from `llama_cpp`, and the application defaults to a
|
|
@@ -120,7 +120,7 @@ Typical usage scenarios include:
|
|
| 120 |
from llama_cpp import Llama
|
| 121 |
|
| 122 |
llm = Llama(
|
| 123 |
-
model_path="
|
| 124 |
n_ctx=4096,
|
| 125 |
)
|
| 126 |
|
|
@@ -138,7 +138,7 @@ print(response["choices"][0]["message"]["content"])
|
|
| 138 |
|
| 139 |
```bash
|
| 140 |
./llama-cli \
|
| 141 |
-
-m
|
| 142 |
-c 4096 \
|
| 143 |
-p "Write a short description of buddy."
|
| 144 |
```
|
|
@@ -169,7 +169,7 @@ consumer hardware, but it may also reduce generation quality relative to higher-
|
|
| 169 |
The following points are supported by the buddy source files:
|
| 170 |
|
| 171 |
* the application expects a model file named `llm/gemma-3-270m-it-Q4_K_M.gguf`
|
| 172 |
-
* the configured model path is `
|
| 173 |
* inference is performed through `llama-cpp-python`
|
| 174 |
* the application uses a 4096-token context window by default
|
| 175 |
* the application combines local text generation with embedding-based retrieval components
|
|
@@ -264,7 +264,7 @@ This repository is expected to contain:
|
|
| 264 |
|
| 265 |
```text
|
| 266 |
README.md
|
| 267 |
-
|
| 268 |
```
|
| 269 |
|
| 270 |
## Relationship to the Main buddy Repository
|
|
|
|
| 57 |
The application configuration points to the following default local model path:
|
| 58 |
|
| 59 |
```text
|
| 60 |
+
buddy/gemma-3-270m-it-Q4_K_M.gguf
|
| 61 |
```
|
| 62 |
|
| 63 |
The runtime loads the model through `Llama(...)` from `llama_cpp`, and the application defaults to a
|
|
|
|
| 120 |
from llama_cpp import Llama
|
| 121 |
|
| 122 |
llm = Llama(
|
| 123 |
+
model_path="buddy/gemma-3-270m-it-Q4_K_M.gguf",
|
| 124 |
n_ctx=4096,
|
| 125 |
)
|
| 126 |
|
|
|
|
| 138 |
|
| 139 |
```bash
|
| 140 |
./llama-cli \
|
| 141 |
+
-m buddy\gemma-3-270m-it-Q4_K_M.gguf\
|
| 142 |
-c 4096 \
|
| 143 |
-p "Write a short description of buddy."
|
| 144 |
```
|
|
|
|
| 169 |
The following points are supported by the buddy source files:
|
| 170 |
|
| 171 |
* the application expects a model file named `llm/gemma-3-270m-it-Q4_K_M.gguf`
|
| 172 |
+
* the configured model path is `buddy/gemma-3-270m-it-Q4_K_M.gguf`
|
| 173 |
* inference is performed through `llama-cpp-python`
|
| 174 |
* the application uses a 4096-token context window by default
|
| 175 |
* the application combines local text generation with embedding-based retrieval components
|
|
|
|
| 264 |
|
| 265 |
```text
|
| 266 |
README.md
|
| 267 |
+
buddy/gemma-3-270m-it-Q4_K_M.gguf
|
| 268 |
```
|
| 269 |
|
| 270 |
## Relationship to the Main buddy Repository
|