westwaters commited on
Commit
bea528e
Β·
verified Β·
1 Parent(s): d3eed3f

Regenerate on the Pollard master card template

Browse files
Files changed (1) hide show
  1. README.md +54 -9
README.md CHANGED
@@ -23,21 +23,39 @@ tags:
23
 
24
  # MiniCPM5-2B β€” Pollard
25
 
26
- > ### Pollard shrank this model: **5.0 GB (f16) β†’ 1.41 GB** β€” **72% smaller, 3.6Γ— down**.
27
  > The smallest rung here; larger, higher-fidelity rungs are listed below.
28
  >
29
  > | format | this model's size |
30
  > |---|---:|
31
- > | f16 | 5.0 GB |
32
- > | Q8_0 | ~2.7 GB |
33
- > | Q6_K | ~2.0 GB |
34
- > | Q4_K_M | ~1.4 GB |
35
  > | **PollardMix (this repo's IQ4_XS)** | **1.41 GB** |
36
 
37
  Pollard builds of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) made with [Pollard Weights](https://github.com/WestWaters/pollard-weights) β€” a ladder of **measured-allocation** quants (bits placed by per-layer sensitivity, not a uniform crush).
38
 
39
  **Standard GGUF β€” runs in stock llama.cpp / ik_llama.cpp, Ollama, LM Studio.** Trellis (`IQ*_KT`) files need [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp); the K-quants run anywhere.
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  ## Available files
42
 
43
  | file | PPL | size | Mean KLD | notes |
@@ -46,19 +64,46 @@ Pollard builds of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-
46
  | `MiniCPM5-2B-Pollard-Q5_K_M.gguf` | β€” | 1.74 GB | β€” | Q5_K_M |
47
  | `MiniCPM5-2B-Pollard-Q6_K.gguf` | β€” | 2.07 GB | β€” | Q6_K |
48
 
49
- _PPL / Mean-KLD benchmarking pending β€” sizes and allocation are final._
 
 
 
 
 
 
 
 
50
 
51
- ## Usage
52
 
53
  ```bash
54
- llama-cli -m MiniCPM5-2B-Pollard-IQ4_XS.gguf -p "Explain why the sky is blue." --temp 0.7
55
- ollama run hf.co/PollardWeights/MiniCPM5-2B-Pollard
56
  ```
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  ## Errata
59
 
60
  - Trellis (`IQ*_KT`) quants need ik_llama.cpp to build/run; K-quants run in any recent llama.cpp.
61
  - Measured allocation places bits by per-layer sensitivity under a size budget.
62
  - Single machine; replication invited.
63
 
 
 
 
 
 
 
 
64
  *Built with [Pollard Weights](https://github.com/WestWaters/pollard-weights) β€” frontier models, small hardware, no compromise.*
 
23
 
24
  # MiniCPM5-2B β€” Pollard
25
 
26
+ > ### Pollard shrank this model: **5.04 GB (f16) β†’ 1.41 GB** β€” **72% smaller, 3.6Γ— down**.
27
  > The smallest rung here; larger, higher-fidelity rungs are listed below.
28
  >
29
  > | format | this model's size |
30
  > |---|---:|
31
+ > | f16 | 5.04 GB |
32
+ > | Q8_0 | ~2.67 GB |
33
+ > | Q6_K | ~2.07 GB |
34
+ > | Q4_K_M | ~1.46 GB |
35
  > | **PollardMix (this repo's IQ4_XS)** | **1.41 GB** |
36
 
37
  Pollard builds of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) made with [Pollard Weights](https://github.com/WestWaters/pollard-weights) β€” a ladder of **measured-allocation** quants (bits placed by per-layer sensitivity, not a uniform crush).
38
 
39
  **Standard GGUF β€” runs in stock llama.cpp / ik_llama.cpp, Ollama, LM Studio.** Trellis (`IQ*_KT`) files need [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp); the K-quants run anywhere.
40
 
41
+ ## Model details
42
+
43
+ | | |
44
+ |---|---|
45
+ | Parameter count | ~2.5B |
46
+ | Architecture | `llama` |
47
+ | Input support | text |
48
+ | imatrix | no |
49
+ | Perplexity measured | **yes** β€” table below |
50
+
51
+ ## Which file should I choose?
52
+
53
+ Every rung is the **same weights**, sized to a different RAM budget by the measured allocation. Pick the largest one that fits your machine with room for context:
54
+
55
+ - **~4 GB RAM / VRAM** β†’ **`Q6_K`** (2.07 GB). Q6_K
56
+ - **~4 GB RAM / VRAM** β†’ **`Q5_K_M`** (1.74 GB). Q5_K_M
57
+ - **~3 GB RAM / VRAM** β†’ **`IQ4_XS`** (1.41 GB). IQ4_XS
58
+
59
  ## Available files
60
 
61
  | file | PPL | size | Mean KLD | notes |
 
64
  | `MiniCPM5-2B-Pollard-Q5_K_M.gguf` | β€” | 1.74 GB | β€” | Q5_K_M |
65
  | `MiniCPM5-2B-Pollard-Q6_K.gguf` | β€” | 2.07 GB | β€” | Q6_K |
66
 
67
+ ## Download a specific file
68
+
69
+ ```bash
70
+ pip install -U "huggingface_hub[cli]"
71
+ hf download PollardWeights/MiniCPM5-2B-Pollard \
72
+ --include "MiniCPM5-2B-Pollard-IQ4_XS.gguf" --local-dir ./
73
+ ```
74
+
75
+ ## How to run
76
 
77
+ These are standard GGUF and run with **llama.cpp**:
78
 
79
  ```bash
80
+ llama-server -hf PollardWeights/MiniCPM5-2B-Pollard:IQ4_XS
 
81
  ```
82
 
83
+ or from a local file:
84
+
85
+ ```bash
86
+ llama-cli -m MiniCPM5-2B-Pollard-IQ4_XS.gguf -ngl 99 -p "Explain why the sky is blue."
87
+ llama-server -m MiniCPM5-2B-Pollard-IQ4_XS.gguf -ngl 99 # OpenAI-compatible API + web UI at :8080
88
+ ```
89
+
90
+ They also work in anything built on llama.cpp β€” **LM Studio, koboldcpp, Jan, ramalama, Ollama** (`ollama run hf.co/PollardWeights/MiniCPM5-2B-Pollard`).
91
+
92
+ ## ARM / AVX
93
+
94
+ llama.cpp repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines β€” no special file needed, online repacking covers these quants. The old `Q4_0_4_4/4_8/8_8` variants are not required.
95
+
96
  ## Errata
97
 
98
  - Trellis (`IQ*_KT`) quants need ik_llama.cpp to build/run; K-quants run in any recent llama.cpp.
99
  - Measured allocation places bits by per-layer sensitivity under a size budget.
100
  - Single machine; replication invited.
101
 
102
+ ## Credits & license
103
+
104
+ - Base model: [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B)
105
+ - Quantization tooling: [llama.cpp](https://github.com/ggml-org/llama.cpp) (ggml-org)
106
+ - Method + tooling: [Pollard Weights](https://github.com/WestWaters/pollard-weights) β€” *measure first, no claim before a number.*
107
+ - License: `apache-2.0`, inherited from the base model.
108
+
109
  *Built with [Pollard Weights](https://github.com/WestWaters/pollard-weights) β€” frontier models, small hardware, no compromise.*