Koopah commited on
Commit
56e044f
·
verified ·
1 Parent(s): 70eaa14

sglang chart: add MTP k=4 arm; restore continuous-serving (closed-loop) section with its chart

Browse files
Files changed (3) hide show
  1. README.md +22 -0
  2. frontier_gsm8k.png +3 -0
  3. frontier_sglang.png +2 -2
README.md CHANGED
@@ -63,6 +63,7 @@ numbers are a floor.
63
  | DSpark k=6 | 387 | 189 · 3,021 | 58 · 5,606 |
64
  | DSpark k=4 | 337 | 173 · 2,765 | **59 · 5,702** |
65
  | MTP (steps 4, k=6) | 290 | 158 · 2,531 | 57 · 5,453 |
 
66
  | no speculation | 156 | 96 · 1,539 | 43 · 4,128 |
67
 
68
  ### vLLM 0.26
@@ -88,6 +89,27 @@ confidence head would pick the verify budget per request per step, so
88
  the best-k envelope is a floor on what it would deliver (see known
89
  issues — no engine wires the head yet; all numbers verify blindly).
90
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
91
  ## Evaluation detail
92
 
93
  Serving-faithful rollout on **held-out prompts never seen in training**
 
63
  | DSpark k=6 | 387 | 189 · 3,021 | 58 · 5,606 |
64
  | DSpark k=4 | 337 | 173 · 2,765 | **59 · 5,702** |
65
  | MTP (steps 4, k=6) | 290 | 158 · 2,531 | 57 · 5,453 |
66
+ | MTP (steps 3, k=4) | 273 | 161 · 2,579 | 57 · 5,472 |
67
  | no speculation | 156 | 96 · 1,539 | 43 · 4,128 |
68
 
69
  ### vLLM 0.26
 
89
  the best-k envelope is a floor on what it would deliver (see known
90
  issues — no engine wires the head yet; all numbers verify blindly).
91
 
92
+ ### Continuous serving (closed loop)
93
+
94
+ The same GSM8K workload served as a continuous request stream
95
+ (EOS-terminated generations, prompts sampled per request — closer to
96
+ production traffic than the one-batch protocol above; sglang fork,
97
+ radix cache on as in production):
98
+
99
+ ![sglang continuous serving](./frontier_gsm8k.png)
100
+
101
+ | config | solo (bs1) | bs16 | bs96 |
102
+ | --- | ---: | ---: | ---: |
103
+ | **DSpark k=8** | **443 tok/s** | 128/user · 2,046 agg | 54/user · **5,205 agg** |
104
+ | DSpark k=5 | 369 | 115 · 1,845 | 53 · 5,057 |
105
+ | MTP (steps 4, k=6) | 284 | 103 · 1,647 | 34 · 3,226 |
106
+ | no speculation | 152 | 70 · 1,114 | 28 · 2,681 |
107
+
108
+ Under sustained load with request churn, DSpark's margin over MTP is
109
+ larger than in the one-batch numbers (+61% aggregate at capacity):
110
+ continuous serving mixes prefill into every decode window, and DSpark's
111
+ faster turnarounds keep queues shorter.
112
+
113
  ## Evaluation detail
114
 
115
  Serving-faithful rollout on **held-out prompts never seen in training**
frontier_gsm8k.png ADDED

Git LFS Details

  • SHA256: dc201d98cb9640ad5e54b9429163845609719045ebd43a8988a45f9eb2cb3cef
  • Pointer size: 131 Bytes
  • Size of remote file: 149 kB
frontier_sglang.png CHANGED

Git LFS Details

  • SHA256: bdc3d2f00ef82d9d0cdd8d552ae47268099679f3f3eae1b1f1d3d6ddd3544e7d
  • Pointer size: 131 Bytes
  • Size of remote file: 189 kB

Git LFS Details

  • SHA256: ee64a3ff5e067514a4e857e3031bfea3b9bfa0e06e6b56640341ac00b69f350b
  • Pointer size: 131 Bytes
  • Size of remote file: 206 kB