sglang chart: add MTP k=4 arm; restore continuous-serving (closed-loop) section with its chart
Browse files- README.md +22 -0
- frontier_gsm8k.png +3 -0
- frontier_sglang.png +2 -2
README.md
CHANGED
|
@@ -63,6 +63,7 @@ numbers are a floor.
|
|
| 63 |
| DSpark k=6 | 387 | 189 · 3,021 | 58 · 5,606 |
|
| 64 |
| DSpark k=4 | 337 | 173 · 2,765 | **59 · 5,702** |
|
| 65 |
| MTP (steps 4, k=6) | 290 | 158 · 2,531 | 57 · 5,453 |
|
|
|
|
| 66 |
| no speculation | 156 | 96 · 1,539 | 43 · 4,128 |
|
| 67 |
|
| 68 |
### vLLM 0.26
|
|
@@ -88,6 +89,27 @@ confidence head would pick the verify budget per request per step, so
|
|
| 88 |
the best-k envelope is a floor on what it would deliver (see known
|
| 89 |
issues — no engine wires the head yet; all numbers verify blindly).
|
| 90 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
## Evaluation detail
|
| 92 |
|
| 93 |
Serving-faithful rollout on **held-out prompts never seen in training**
|
|
|
|
| 63 |
| DSpark k=6 | 387 | 189 · 3,021 | 58 · 5,606 |
|
| 64 |
| DSpark k=4 | 337 | 173 · 2,765 | **59 · 5,702** |
|
| 65 |
| MTP (steps 4, k=6) | 290 | 158 · 2,531 | 57 · 5,453 |
|
| 66 |
+
| MTP (steps 3, k=4) | 273 | 161 · 2,579 | 57 · 5,472 |
|
| 67 |
| no speculation | 156 | 96 · 1,539 | 43 · 4,128 |
|
| 68 |
|
| 69 |
### vLLM 0.26
|
|
|
|
| 89 |
the best-k envelope is a floor on what it would deliver (see known
|
| 90 |
issues — no engine wires the head yet; all numbers verify blindly).
|
| 91 |
|
| 92 |
+
### Continuous serving (closed loop)
|
| 93 |
+
|
| 94 |
+
The same GSM8K workload served as a continuous request stream
|
| 95 |
+
(EOS-terminated generations, prompts sampled per request — closer to
|
| 96 |
+
production traffic than the one-batch protocol above; sglang fork,
|
| 97 |
+
radix cache on as in production):
|
| 98 |
+
|
| 99 |
+

|
| 100 |
+
|
| 101 |
+
| config | solo (bs1) | bs16 | bs96 |
|
| 102 |
+
| --- | ---: | ---: | ---: |
|
| 103 |
+
| **DSpark k=8** | **443 tok/s** | 128/user · 2,046 agg | 54/user · **5,205 agg** |
|
| 104 |
+
| DSpark k=5 | 369 | 115 · 1,845 | 53 · 5,057 |
|
| 105 |
+
| MTP (steps 4, k=6) | 284 | 103 · 1,647 | 34 · 3,226 |
|
| 106 |
+
| no speculation | 152 | 70 · 1,114 | 28 · 2,681 |
|
| 107 |
+
|
| 108 |
+
Under sustained load with request churn, DSpark's margin over MTP is
|
| 109 |
+
larger than in the one-batch numbers (+61% aggregate at capacity):
|
| 110 |
+
continuous serving mixes prefill into every decode window, and DSpark's
|
| 111 |
+
faster turnarounds keep queues shorter.
|
| 112 |
+
|
| 113 |
## Evaluation detail
|
| 114 |
|
| 115 |
Serving-faithful rollout on **held-out prompts never seen in training**
|
frontier_gsm8k.png
ADDED
|
Git LFS Details
|
frontier_sglang.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|