create README.md

README.md

@@ -0,0 +1,167 @@
+---
+language:
+- en
+base_model: Nanbeige/Nanbeige4.2-3B
+pipeline_tag: text-generation
+tags:
+- gguf
+- llama.cpp
+- nanbeige
+- reasoning
+- instruct
+- local-llm
+license: other
+---
+
+# Nanbeige4.2-3B GGUF
+
+This repository provides the **GGUF conversions** of the original **Nanbeige/Nanbeige4.2-3B** model. All credit for the model architecture and weights belongs to the original Nanbeige team.
+
+> **Note:** This repository contains only GGUF conversions. The original Hugging Face model is **Nanbeige/Nanbeige4.2-3B**.
+
+---
+
+# Base Model
+
+**Original Hugging Face Model**
+
+**Nanbeige/Nanbeige4.2-3B**
+
+This repository does **not** modify or fine-tune the original model. It simply provides GGUF conversions for running the model locally with llama.cpp and other GGUF-compatible applications.
+
+---
+
+# Available Files
+
+This repository contains the following GGUF files:
+
+| File | Description |
+|------|-------------|
+| **Nanbeige4.2-3B-BF16.gguf** | Full-precision BF16 GGUF. Highest quality but requires significantly more RAM/VRAM. |
+| **Nanbeige4.2-3B-Q4_K_M.gguf** | 4-bit quantized GGUF. Recommended for most users due to its excellent balance of quality, speed, and memory usage. |
+
+> **You only need to download ONE of these files.**
+
+- Download **BF16** if you want the highest possible quality and have sufficient GPU memory.
+- Download **Q4_K_M** if you want lower memory usage while maintaining excellent performance.
+
+---
+
+# Important
+
+At the time of publishing, support for the **Nanbeige** architecture has **not yet been merged into the main llama.cpp repository**.
+
+Please use the official Nanbeige fork of llama.cpp:
+
+https://github.com/Nanbeige/llama.cpp/tree/nanbeige42
+
+If you build the upstream `ggml-org/llama.cpp`, you may encounter an error similar to:
+
+```text
+unknown model architecture: 'nanbeige'
+```
+
+The Nanbeige fork contains the required architecture support.
+
+---
+
+# Building llama.cpp
+
+Clone the repository:
+
+```bash
+git clone --recursive -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git
+cd llama.cpp
+```
+
+Build:
+
+```bash
+cmake -B build
+cmake --build build -j
+```
+
+---
+
+# Running with llama-server
+
+Example:
+
+```bash
+./build/bin/llama-server \
+    -m Nanbeige4.2-3B-Q4_K_M.gguf \
+    --host 0.0.0.0 \
+    --port 8080 \
+    -ngl 999 \
+    -c 65536
+```
+
+After starting the server, the OpenAI-compatible API will be available at:
+
+```text
+http://localhost:8080
+```
+
+---
+
+# Using Hermes
+
+Hermes works well with this model.
+
+1. Start `llama-server`.
+2. Open Hermes.
+3. Go to **Model Selection**.
+4. Choose **Custom URL**.
+5. Enter your llama-server endpoint, for example:
+
+```text
+http://localhost:8080
+```
+
+Hermes will communicate directly with your local Nanbeige model using the OpenAI-compatible API.
+
+---
+
+# Using a Web UI
+
+This model can also be used with web interfaces that support OpenAI-compatible endpoints, including:
+
+- Open WebUI
+- Hermes
+- Any application compatible with the OpenAI Chat Completions API
+
+Simply configure the application to connect to your running `llama-server` instance.
+
+---
+
+# Recommended Model
+
+For most users, the recommended file is:
+
+**✅ Nanbeige4.2-3B-Q4_K_M.gguf**
+
+It provides an excellent balance of:
+
+- Quality
+- Speed
+- Memory usage
+
+---
+
+# Credits
+
+- **Original model:** Nanbeige/Nanbeige4.2-3B
+- GGUF conversion provided by this repository.
+- llama.cpp support is currently available through the Nanbeige `nanbeige42` branch.
+
+All credit for the model architecture, tokenizer, training, and original model weights belongs entirely to the original Nanbeige team.
+
+---
+
+# License
+
+This repository distributes GGUF conversions of the original model.
+
+The license for these GGUF files is **the same as the license of the original Nanbeige/Nanbeige4.2-3B project**.
+
+Please refer to the original model repository for the official license terms, usage conditions, and any applicable restrictions.