MasherDon commited on
Commit
4e1b024
·
verified ·
1 Parent(s): 85fdd50

Model card: numbers of 6 November, lens placement, SSD wear note

Browse files
Files changed (1) hide show
  1. README.md +7 -6
README.md CHANGED
@@ -45,24 +45,25 @@ These are the weights we run DeepSeek V4.1 Flash 1017 with on one home PC. The m
45
 
46
  1. Build the fork from the patches in the [GitHub repository](https://github.com/MasherDon/Siege-Ladder). It is tested on Windows with CUDA 13 on an RTX 50 card (sm_120).
47
  2. Pass `-m .../DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf`. The other parts load by themselves.
48
- 3. Set up the lens (G44 / G40, on by default). Either put `1017-lens-d2.gguf` into a folder named `g44` next to the model's folder, or point `LLAMA_MOE_STREAM_G44` at the file. Without the lens the engine still runs, but decodes slower.
49
 
50
  Our PC: RTX 5070 Ti 16 GB on PCIe 4.0 x16, Ryzen 5 7600, 64 GB DDR5, one PCIe 5.0 NVMe (plus an optional expert mirror on a second NVMe).
51
 
52
  | | tok/s |
53
  |---|---|
54
- | decode, short context | ~20 (21.30 teacher-forced with G44 + G40, 4 November) |
55
- | reading a 246K-token document (256K context) | 930 (5 November) |
56
- | reading a 479K-token document (512K context) | 867–900 (5 November) |
57
- | reading a 947K-token document (1M context) | 778 (5 November) |
58
 
59
- These are single runs, not averages.
60
 
61
  ## Hardware notes
62
 
63
  - **PCIe 5.0.** With a PCIe 5.0 x16 card *and* slot, both prompt reading and decode get faster with no code change. Copying experts over the bus is a large part of every decode token.
64
  - Estimate: in an older profile, copies alone took 27.1 ms of a 54 ms token. Halving them gives about 24.7 instead of 18.5 tok/s.
65
  - **A second NVMe** (expert mirror) helps prompt reading a little: +14–22% on cold prompts in an older test. It is optional.
 
66
  - **Training.** Speed will also improve separately through training (expert prediction), independently of these weights.
67
 
68
  ## Which build
 
45
 
46
  1. Build the fork from the patches in the [GitHub repository](https://github.com/MasherDon/Siege-Ladder). It is tested on Windows with CUDA 13 on an RTX 50 card (sm_120).
47
  2. Pass `-m .../DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf`. The other parts load by themselves.
48
+ 3. Set up the lens (G44 / G40, on by default). The engine looks for it in a folder named `g44` *next to* the model's folder, not inside it. After downloading this repository, move its `g44` folder one level up, or set `LLAMA_MOE_STREAM_G44` to the path of `1017-lens-d2.gguf`. Without the lens the engine still runs, but decodes slower.
49
 
50
  Our PC: RTX 5070 Ti 16 GB on PCIe 4.0 x16, Ryzen 5 7600, 64 GB DDR5, one PCIe 5.0 NVMe (plus an optional expert mirror on a second NVMe).
51
 
52
  | | tok/s |
53
  |---|---|
54
+ | decode, short context | 18.44–18.80 through the server (6 November); 21.30 teacher-forced with G44 + G40 (4 November) |
55
+ | reading a 246K-token document (256K context) | 1128 (6 November, mean of two runs) |
56
+ | reading a 479K-token document (512K context) | 1043 (6 November) |
57
+ | reading a 947K-token document (1M context) | 888 (6 November) |
58
 
59
+ Single runs unless stated. From 256K context the engine reads the prompt layer-major (on by default).
60
 
61
  ## Hardware notes
62
 
63
  - **PCIe 5.0.** With a PCIe 5.0 x16 card *and* slot, both prompt reading and decode get faster with no code change. Copying experts over the bus is a large part of every decode token.
64
  - Estimate: in an older profile, copies alone took 27.1 ms of a 54 ms token. Halving them gives about 24.7 instead of 18.5 tok/s.
65
  - **A second NVMe** (expert mirror) helps prompt reading a little: +14–22% on cold prompts in an older test. It is optional.
66
+ - **SSD wear.** The Engram tables get billions of small reads. On our Gen5 drive, after a few weeks their random reads fell from ~400K to ~40K IOPS, and the 512K document slowed from ~1050 to ~685 tok/s. Copying the two Engram files anew restored the speed. If long documents slow down over time, check the drive first.
67
  - **Training.** Speed will also improve separately through training (expert prediction), independently of these weights.
68
 
69
  ## Which build