Add preserve_thinking option to the chat template
Browse filesThis adds a `preserve_thinking` parameter to the chat template, controlling whether `<think>…</think>` reasoning blocks from previous assistant turns are kept when formatting a multi-turn conversation.
- **Default: `preserve_thinking = true`** — reasoning is kept in every assistant turn, exactly as before. The rendered output is unchanged for existing callers (verified byte-identical on multi-turn conversations, with and without a generation prompt).
- **`preserve_thinking = false`** — reasoning is removed from all assistant turns preceding the last user message; only the most recent turn keeps its `<think>` block. This shortens the prompt and reduces the KV-cache footprint in multi-turn settings. The mechanism reuses the template's existing `last_query_index` logic, matching the behavior introduced in Qwen3.6.
Usage:
```python
tokenizer.apply_chat_template(messages, preserve_thinking=False, ...)
```
or, when serving (e.g. vLLM), pass it through `chat_template_kwargs={"preserve_thinking": false}`.
- README.md +2 -0
- chat_template.jinja +8 -1
|
@@ -109,6 +109,8 @@ Jean-Baptiste Poquelin, known as Molière, was a 17th-century French playwright,
|
|
| 109 |
Thinking can be turned off by using ```tokenizer.apply_chat_template(…, enable_thinking=False)```. This will append empty thinking tags (```<think>\n\n</think>\n\n```) in the prompt, immediately after "`assistant\n`".
|
| 110 |
However, for testing a non-thinking model, it is better to conduct experiments on [Luciole-8B-Instruct-1.1](https://huggingface.co/OpenLLM-France/Luciole-8B-Instruct-1.1).
|
| 111 |
|
|
|
|
|
|
|
| 112 |
### Training procedure
|
| 113 |
|
| 114 |
Luciole-8B-KD-Math-Thinking-1.1 was trained in two phases. The model was first finetuned on thinking data, followed by DPO on preference data with thinking traces.
|
|
|
|
| 109 |
Thinking can be turned off by using ```tokenizer.apply_chat_template(…, enable_thinking=False)```. This will append empty thinking tags (```<think>\n\n</think>\n\n```) in the prompt, immediately after "`assistant\n`".
|
| 110 |
However, for testing a non-thinking model, it is better to conduct experiments on [Luciole-8B-Instruct-1.1](https://huggingface.co/OpenLLM-France/Luciole-8B-Instruct-1.1).
|
| 111 |
|
| 112 |
+
By default, the `<think>…</think>` reasoning of all previous assistant turns is kept when formatting a multi-turn conversation. Setting ```tokenizer.apply_chat_template(…, preserve_thinking=False)``` drops it from every assistant turn preceding the last user message (only the most recent turn keeps its reasoning), which shortens the prompt and reduces the KV-cache footprint. With vLLM, set it per request via ```extra_body={"chat_template_kwargs": {"preserve_thinking": false}}```. Ollama does not use this parameter, as it formats prompts with its own built-in renderer.
|
| 113 |
+
|
| 114 |
### Training procedure
|
| 115 |
|
| 116 |
Luciole-8B-KD-Math-Thinking-1.1 was trained in two phases. The model was first finetuned on thinking data, followed by DPO on preference data with thinking traces.
|
|
@@ -1,3 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
| 1 |
{%- if messages[0].role == 'system' and messages[0].content.strip() != "" %}
|
| 2 |
{%- set system_message = messages[0].content %}
|
| 3 |
{%- else %}
|
|
@@ -42,7 +45,11 @@
|
|
| 42 |
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
| 43 |
{%- endif %}
|
| 44 |
{%- endif %}
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
{%- if message.tool_calls %}
|
| 47 |
{%- for tool_call in message.tool_calls %}
|
| 48 |
{%- if (loop.first and content) or (not loop.first) %}
|
|
|
|
| 1 |
+
{%- if preserve_thinking is not defined %}
|
| 2 |
+
{%- set preserve_thinking = true %}
|
| 3 |
+
{%- endif %}
|
| 4 |
{%- if messages[0].role == 'system' and messages[0].content.strip() != "" %}
|
| 5 |
{%- set system_message = messages[0].content %}
|
| 6 |
{%- else %}
|
|
|
|
| 45 |
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
|
| 46 |
{%- endif %}
|
| 47 |
{%- endif %}
|
| 48 |
+
{%- if preserve_thinking or (loop.index0 > ns.last_query_index and (loop.last or reasoning_content)) %}
|
| 49 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
|
| 50 |
+
{%- else %}
|
| 51 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 52 |
+
{%- endif %}
|
| 53 |
{%- if message.tool_calls %}
|
| 54 |
{%- for tool_call in message.tool_calls %}
|
| 55 |
{%- if (loop.first and content) or (not loop.first) %}
|