Model card: numbers of 6 November, lens placement, SSD wear note
Browse files
README.md
CHANGED
|
@@ -45,24 +45,25 @@ These are the weights we run DeepSeek V4.1 Flash 1017 with on one home PC. The m
|
|
| 45 |
|
| 46 |
1. Build the fork from the patches in the [GitHub repository](https://github.com/MasherDon/Siege-Ladder). It is tested on Windows with CUDA 13 on an RTX 50 card (sm_120).
|
| 47 |
2. Pass `-m .../DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf`. The other parts load by themselves.
|
| 48 |
-
3. Set up the lens (G44 / G40, on by default).
|
| 49 |
|
| 50 |
Our PC: RTX 5070 Ti 16 GB on PCIe 4.0 x16, Ryzen 5 7600, 64 GB DDR5, one PCIe 5.0 NVMe (plus an optional expert mirror on a second NVMe).
|
| 51 |
|
| 52 |
| | tok/s |
|
| 53 |
|---|---|
|
| 54 |
-
| decode, short context |
|
| 55 |
-
| reading a 246K-token document (256K context) |
|
| 56 |
-
| reading a 479K-token document (512K context) |
|
| 57 |
-
| reading a 947K-token document (1M context) |
|
| 58 |
|
| 59 |
-
|
| 60 |
|
| 61 |
## Hardware notes
|
| 62 |
|
| 63 |
- **PCIe 5.0.** With a PCIe 5.0 x16 card *and* slot, both prompt reading and decode get faster with no code change. Copying experts over the bus is a large part of every decode token.
|
| 64 |
- Estimate: in an older profile, copies alone took 27.1 ms of a 54 ms token. Halving them gives about 24.7 instead of 18.5 tok/s.
|
| 65 |
- **A second NVMe** (expert mirror) helps prompt reading a little: +14–22% on cold prompts in an older test. It is optional.
|
|
|
|
| 66 |
- **Training.** Speed will also improve separately through training (expert prediction), independently of these weights.
|
| 67 |
|
| 68 |
## Which build
|
|
|
|
| 45 |
|
| 46 |
1. Build the fork from the patches in the [GitHub repository](https://github.com/MasherDon/Siege-Ladder). It is tested on Windows with CUDA 13 on an RTX 50 card (sm_120).
|
| 47 |
2. Pass `-m .../DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf`. The other parts load by themselves.
|
| 48 |
+
3. Set up the lens (G44 / G40, on by default). The engine looks for it in a folder named `g44` *next to* the model's folder, not inside it. After downloading this repository, move its `g44` folder one level up, or set `LLAMA_MOE_STREAM_G44` to the path of `1017-lens-d2.gguf`. Without the lens the engine still runs, but decodes slower.
|
| 49 |
|
| 50 |
Our PC: RTX 5070 Ti 16 GB on PCIe 4.0 x16, Ryzen 5 7600, 64 GB DDR5, one PCIe 5.0 NVMe (plus an optional expert mirror on a second NVMe).
|
| 51 |
|
| 52 |
| | tok/s |
|
| 53 |
|---|---|
|
| 54 |
+
| decode, short context | 18.44–18.80 through the server (6 November); 21.30 teacher-forced with G44 + G40 (4 November) |
|
| 55 |
+
| reading a 246K-token document (256K context) | 1128 (6 November, mean of two runs) |
|
| 56 |
+
| reading a 479K-token document (512K context) | 1043 (6 November) |
|
| 57 |
+
| reading a 947K-token document (1M context) | 888 (6 November) |
|
| 58 |
|
| 59 |
+
Single runs unless stated. From 256K context the engine reads the prompt layer-major (on by default).
|
| 60 |
|
| 61 |
## Hardware notes
|
| 62 |
|
| 63 |
- **PCIe 5.0.** With a PCIe 5.0 x16 card *and* slot, both prompt reading and decode get faster with no code change. Copying experts over the bus is a large part of every decode token.
|
| 64 |
- Estimate: in an older profile, copies alone took 27.1 ms of a 54 ms token. Halving them gives about 24.7 instead of 18.5 tok/s.
|
| 65 |
- **A second NVMe** (expert mirror) helps prompt reading a little: +14–22% on cold prompts in an older test. It is optional.
|
| 66 |
+
- **SSD wear.** The Engram tables get billions of small reads. On our Gen5 drive, after a few weeks their random reads fell from ~400K to ~40K IOPS, and the 512K document slowed from ~1050 to ~685 tok/s. Copying the two Engram files anew restored the speed. If long documents slow down over time, check the drive first.
|
| 67 |
- **Training.** Speed will also improve separately through training (expert prediction), independently of these weights.
|
| 68 |
|
| 69 |
## Which build
|