tianzl66/Qwen3-8B-MetaMathQA-50K-LoRA

Qwen3-8B + MetaMathQA — LoRA

This repository contains the Qwen3-8B LoRA adapter trained on MetaMathQA-50K.

Base Model

Qwen/Qwen3-8B

Training

• Dataset: MetaMathQA
• Samples: 50K
• Epochs: 3
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Learning rate: 1e-4 (cosine schedule, warmup ratio 0.05)
• Global batch size: 32
• Max sequence length: 4096
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Chat template: non-thinking (enable_thinking=False)

Evaluation

Evaluation on GSM8K (test split, 1319 examples).

Model	GSM8K
Base	86.66% (1143/1319)
LoRA	84.00% (1108/1319)
HNS 4+1, all modules	88.40% (1166/1319)
HNS 8+2, all modules	88.02% (1161/1319)

Settings: Qwen3 non-thinking chat template (enable_thinking=False), greedy decoding, max_new_tokens=2048, vLLM backend, FLASH_ATTENTION, seed 42. The base score is unchanged at max_new_tokens=4096 (86.66%, 1143/1319), confirming the comparison is not truncation-limited.

Short-generation budget

With max_new_tokens=256, the base model's longer reasoning chains are truncated:

Model	GSM8K
Base	68.01% (897/1319)
LoRA	83.32% (1099/1319)
HNS 4+1, all modules	86.20% (1137/1319)
HNS 8+2, all modules	86.05% (1135/1319)

Files

• adapter_model.safetensors: PEFT LoRA weights
• adapter_config.json: PEFT configuration
• eval-gsm8k-maxnew2048/metrics.json: aggregate GSM8K metrics
• eval-gsm8k-maxnew2048/predictions.jsonl: per-example predictions
• run_config.json / run_args.json: training configuration