SeanWang0027/Qwen3-4B-Base-NemotronIF-Reasoning-SFT-10k

Qwen3-4B-Base · Nemotron-IF reasoning-on SFT (10k)

Full-parameter SFT of Qwen3-4B-Base on 10,000 reasoning-on instruction-following conversations from nvidia/Nemotron-Instruction-Following-Chat-v1 (reasoning == "on" and capability_target == "instruction_following", random subset, seed 0). The model answers in the Qwen3 thinking format:

<|im_start|>assistant
<think>
...short GPT-OSS-style reasoning, with self-checks against the constraints...
</think>

...answer...<|im_end|>

Why full fine-tuning

Qwen3-4B-Base ships <|im_start|>, <|im_end|>, <think>, </think> as one shared, untrained embedding vector (norm 0.357 vs 1.13 for real tokens; 944 vocabulary rows are byte-identical to it). With tied embeddings their logits can never differ, so a LoRA on attention/MLP cannot learn to emit them: a LoRA run on the same data produced a random rare token where <think> belonged, never closed the think block, and never stopped. Here the four rows were re-initialised (mean of real-token rows + noise) and every parameter trained. After training, 4/4 held-out IF prompts were well-formed (<think>…</think>, answer, <|im_end|>).

Training

data	10,000 conversations → 23,588 examples (one per assistant turn; think-free history, current turn trained with its trace)
loss	completion-only on `…\n\n{answer}<
method	FSDP2 (fully_shard) over 2× A100-80GB, fp32 master + AdamW, bf16 compute, flash-attention-2, activation checkpointing
optimizer	AdamW β=(0.9, 0.95), wd 0, lr 2e-5 cosine, warmup 10 steps, grad clip 1.0
batch	global 128 (micro 2 × 2 GPUs × accum 32), max_len 8192, 1 epoch = 184 steps
tokens	~20M target tokens; 2.1 h wall clock
loss	1.18 (steps 1–20) → 0.67 (steps 81–100) → ~0.6 at the end

Scripts, report.json and the per-step train_log.jsonl are in training/.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = "SeanWang0027/Qwen3-4B-Base-NemotronIF-Reasoning-SFT-10k"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype="bfloat16", device_map="cuda")
msgs = [{"role": "user", "content": "Explain why the sky is blue in exactly two sentences, all lowercase."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")
out = model.generate(ids, max_new_tokens=1024, temperature=0.6, top_p=0.95, top_k=20, do_sample=True)
print(tok.decode(out[0, ids.shape[1]:]))

generation_config.eos_token_id = [<|im_end|>, <|endoftext|>]. Reasoning parsers that split on </think> (vLLM/sglang qwen3) work as-is. Trained only on thinking-on data; enable_thinking=False is untested.

Notes

• Base model licence: Apache-2.0 (Qwen3). Training data: ODC-BY 1.0 (Nemotron chat subset).
• 10k-conversation pilot; the full slice is 76,314 conversations.