It looks like vllm and others also get this error https://github.com/vllm-project/vllm/pull/54686

Bug report: DeepSeek-V4-Flash-Vision-Exp emits malformed tool calls (Hermes-style envelope inside DSML) after any in-context tool-call round

Date: 2026-09-01 (all times UTC)
Affected model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
HF snapshot: 31ea11185e11ccafad1c385104188a9e3b648ad6
Severity: high for agentic/tool-use workloads — tool calls become invalid after the first round-trip, breaking agent loops

TL;DR

When the conversation history contains at least one prior assistant tool-call round, the
model frequently (≈80–100% in our measurements) stops emitting tool arguments in the format
its own chat template prescribes (one DSML parameter tag per argument) and instead wraps the
entire argument object into a single DSML parameter literally named arguments (occasionally
args), sometimes double-encoded as a JSON string, sometimes nested twice, and — after a client
rejects the call — progressively more corrupted. Fresh conversations (no prior tool rounds)
never malform.

We worked around it server-side (defensive unwrap in SGLang's parse paths; details below) and
verified the workaround eliminates the failures (0/13 post-fix vs 6/6 pre-fix on the same
replay), but the model-side format regression remains.

Environment

Symptom

The template instructs (and in-context history demonstrates) the format:

<｜DSML｜tool_calls>
<｜DSML｜invoke name="bash">
<｜DSML｜parameter name="command" string="true">ls -la dir0</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

After ≥1 prior tool-call round in context, the model instead emits (raw generation, captured by
bypassing the server parser):

<｜DSML｜tool_calls>
<｜DSML｜invoke name="bash">
<｜DSML｜parameter name="arguments" string="false">{"arguments": {"command": "pwd; ls -la; ls -la /home/jovyan | head -50"}}</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

i.e. a single parameter named arguments whose value is the whole argument object. After the
server parser does the faithful translation, the OpenAI response carries
function.arguments == '{"arguments": {"command": "…"}}', which fails any strict schema
validation (missing required property "command"), so the tool call is rejected by the client.

Observed variants

1. dominant: {"arguments": {…}} (single envelope, object value; both string="true" and string="false" flags seen)
2. double-encoded: {"arguments": "{\"command\": …}"} (value is a JSON string)
3. alternate key: {"args": {…}}
4. escalation under error feedback (model retries after the client's validation error):
   • {"arguments":{"arguments":{"command":"…"}}} (double wrap)
   • {"arguments":{"arguments":{"arguments":{"arguments":{"file_path":"path\": \"x\"}}}}}"}}}}} (quadruple wrap, corrupted)
   • mid-stream duplication/corruption of the inner command text

The escalation loop is notable: the client error message does not teach the model the correct
format; it retries with more wrapping, suggesting the model is pattern-matching the envelope
shape rather than representing the tool-call schema.

Trigger condition and measured rates

Reproduction: serve the checkpoint as above; send a chat completion with tools=[bash(command)]
and a history containing one prior assistant turn with tool_calls plus its tool result; ask
the model to continue ("continue" suffices; no explicit tool instruction needed).

Key diagnostic: the first tool call of a fresh conversation is always well-formed; the format
collapse begins specifically when the model can see its own prior DSML tool calls in context —
consistent with in-context pattern completion of a mis-learned serialization.

What we ruled out

• Client/harness: the failure is visible in the raw model text (captured with server-side
  tool parsing disabled), before any client processing.
• SGLang parser correctness: the deepseekv4 detector translates the model text faithfully;
  the malformed output is a faithful parse of malformed generation. (We additionally found the
  parsers pass the envelope through verbatim — see mitigation.)
• Streaming vs non-streaming: both reproduce.
• Prompt construction: reproduced with a two-tool minimal setup and with a 25-tool setup;
  reproduced with and without reasoning effort parameter; system prompt is plain prose, tool
  docs come from the model's chat template.

Interim mitigation (server-side, for awareness)

We applied a defensive normalization in SGLang at three parse points (one-shot
parse_base_json, streaming _parse_parameters_from_xml with prefix-consistent handling for
partial parses, and the encoding layer's decode_dsml_to_arguments): if the parsed parameters
are a single-key {"arguments": …} / {"args": …} dict (or a JSON-string containing one), unwrap
to the inner object; nested envelopes are unwrapped iteratively. Verified end-to-end: the same
replays that produced 6/6 malformed calls now produce 0/13, with legitimate calls (including tools
that genuinely declare an arguments property) untouched. We are happy to contribute this as an
upstream SGLang patch, but we want to stress it is a workaround — the model should emit the
per-parameter format its own template specifies.

Hypotheses

1. SFT data conflation: the checkpoint's tool-calling training data appears to mix
   Hermes/OpenAI-style serializations (where the complete call is {"name": …, "arguments": {…}})
   with the DSML parameter-tag format. Given prior DSML calls in context, the model maps the
   envelope key (arguments) into the DSML parameter-name position.
2. Under-specified "direct JSON body" alternative: the template documents a second accepted
   body form (direct JSON after <｜DSML｜invoke name="…">). The model may be filling that
   alternative with the envelope shape it knows from OpenAI-style data instead of a bare
   parameter object.
3. No schema-level representation of the requirement: the error-feedback escalation (more
   wrapping on retry) indicates the model does not model the target schema, only surface
   patterns — consistent with insufficient multi-turn tool-call-format consistency training in
   this experimental checkpoint.

Requests

1. Confirm whether this is a known regression in the -Exp checkpoint, and whether a refreshed
   snapshot addresses it.
2. Add multi-turn tool-call format-consistency coverage (SFT/RL) — specifically: correct format
   maintenance after N prior tool rounds, and correct recovery after client-side validation
   errors.
3. Clarify the canonical wire format for tool arguments (per-parameter DSML tags vs direct JSON
   body), so serving-side validators/normalizers can be tightened accordingly.
4. Optionally accept the SGLang-side defensive unwrap as an upstream patch (we can open a PR with
   tests).

Reproduction recipe (minimal)

# 1. serve
sglang serve --model-path deepseek-ai/DeepSeek-V4-Flash-Vision-Exp --tp 4 \
  --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --port 30000

# 2. chat completion with one prior tool round; repeat N times
curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
  "messages": [
    {"role":"user","content":"review this dir"},
    {"role":"assistant","content":"","tool_calls":[{"id":"c0","type":"function",
      "function":{"name":"bash","arguments":"{\"command\": \"ls -la dir0\"}"}}]},
    {"role":"tool","tool_call_id":"c0","content":"out 0"},
    {"role":"user","content":"continue"}
  ],
  "tools":[{"type":"function","function":{"name":"bash","description":"Run a shell command",
    "parameters":{"type":"object","properties":{"command":{"type":"string"}},"required":["command"]}}}]
}'
# observe tool_calls[0].function.arguments == '{"arguments": {"command": …}}' in a large
# fraction of samples; with no prior tool round in the messages, arguments are always correct.