Safetensors
French
English
nemotron
openllm-france
Jeronymous commited on
Commit
e99d852
·
verified ·
1 Parent(s): 5c46f0f

Add preserve_thinking option to the chat template

Browse files

This adds a `preserve_thinking` parameter to the chat template, controlling whether `<think>…</think>` reasoning blocks from previous assistant turns are kept when formatting a multi-turn conversation.

- **Default: `preserve_thinking = true`** — reasoning is kept in every assistant turn, exactly as before. The rendered output is unchanged for existing callers (verified byte-identical on multi-turn conversations, with and without a generation prompt).
- **`preserve_thinking = false`** — reasoning is removed from all assistant turns preceding the last user message; only the most recent turn keeps its `<think>` block. This shortens the prompt and reduces the KV-cache footprint in multi-turn settings. The mechanism reuses the template's existing `last_query_index` logic, matching the behavior introduced in Qwen3.6.

Usage:
```python
tokenizer.apply_chat_template(messages, preserve_thinking=False, ...)
```
or, when serving (e.g. vLLM), pass it through `chat_template_kwargs={"preserve_thinking": false}`.

Files changed (2) hide show
  1. README.md +2 -0
  2. chat_template.jinja +8 -1
README.md CHANGED
@@ -109,6 +109,8 @@ Jean-Baptiste Poquelin, known as Molière, was a 17th-century French playwright,
109
  Thinking can be turned off by using ```tokenizer.apply_chat_template(…, enable_thinking=False)```. This will append empty thinking tags (```<think>\n\n</think>\n\n```) in the prompt, immediately after "`assistant\n`".
110
  However, for testing a non-thinking model, it is better to conduct experiments on [Luciole-8B-Instruct-1.1](https://huggingface.co/OpenLLM-France/Luciole-8B-Instruct-1.1).
111
 
 
 
112
  ### Training procedure
113
 
114
  Luciole-8B-KD-Math-Thinking-1.1 was trained in two phases. The model was first finetuned on thinking data, followed by DPO on preference data with thinking traces.
 
109
  Thinking can be turned off by using ```tokenizer.apply_chat_template(…, enable_thinking=False)```. This will append empty thinking tags (```<think>\n\n</think>\n\n```) in the prompt, immediately after "`assistant\n`".
110
  However, for testing a non-thinking model, it is better to conduct experiments on [Luciole-8B-Instruct-1.1](https://huggingface.co/OpenLLM-France/Luciole-8B-Instruct-1.1).
111
 
112
+ By default, the `<think>…</think>` reasoning of all previous assistant turns is kept when formatting a multi-turn conversation. Setting ```tokenizer.apply_chat_template(…, preserve_thinking=False)``` drops it from every assistant turn preceding the last user message (only the most recent turn keeps its reasoning), which shortens the prompt and reduces the KV-cache footprint. With vLLM, set it per request via ```extra_body={"chat_template_kwargs": {"preserve_thinking": false}}```. Ollama does not use this parameter, as it formats prompts with its own built-in renderer.
113
+
114
  ### Training procedure
115
 
116
  Luciole-8B-KD-Math-Thinking-1.1 was trained in two phases. The model was first finetuned on thinking data, followed by DPO on preference data with thinking traces.
chat_template.jinja CHANGED
@@ -1,3 +1,6 @@
 
 
 
1
  {%- if messages[0].role == 'system' and messages[0].content.strip() != "" %}
2
  {%- set system_message = messages[0].content %}
3
  {%- else %}
@@ -42,7 +45,11 @@
42
  {%- set content = content.split('</think>')[-1].lstrip('\n') %}
43
  {%- endif %}
44
  {%- endif %}
45
- {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
 
 
 
 
46
  {%- if message.tool_calls %}
47
  {%- for tool_call in message.tool_calls %}
48
  {%- if (loop.first and content) or (not loop.first) %}
 
1
+ {%- if preserve_thinking is not defined %}
2
+ {%- set preserve_thinking = true %}
3
+ {%- endif %}
4
  {%- if messages[0].role == 'system' and messages[0].content.strip() != "" %}
5
  {%- set system_message = messages[0].content %}
6
  {%- else %}
 
45
  {%- set content = content.split('</think>')[-1].lstrip('\n') %}
46
  {%- endif %}
47
  {%- endif %}
48
+ {%- if preserve_thinking or (loop.index0 > ns.last_query_index and (loop.last or reasoning_content)) %}
49
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
50
+ {%- else %}
51
+ {{- '<|im_start|>' + message.role + '\n' + content }}
52
+ {%- endif %}
53
  {%- if message.tool_calls %}
54
  {%- for tool_call in message.tool_calls %}
55
  {%- if (loop.first and content) or (not loop.first) %}