morningstarxcdcode/adaption-medical-reasoning-sft-120b-model

Medical Reasoning SFT 120B

Fine-tuned Llama 3.3 70B for medical reasoning. Trained on 20,000 synthetic medical conversations generated through Adaption's Adaptive Data platform. Part 1 Healthcare submission for the AutoScientist Challenge.

Model Details

• Base model: Meta-Llama-3.3-70B-Instruct-Reference
• Fine-tuning method: LoRA (Low-Rank Adaptation)
• LoRA rank: 64
• LoRA alpha: 128
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training epochs: 4
• Training steps: 224
• Final eval loss: 1.07
• License: Apache 2.0

Training Data

20,000 rows of synthetic medical conversations covering:

• Clinical reasoning and differential diagnosis
• Multiple-choice medical questions
• Treatment analysis and drug interactions
• Step-by-step diagnostic reasoning
• Biology and healthcare scenarios

Data was generated using gpt-oss-120b through Adaption's Adaptive Data platform with reasoning traces recipe.

Results

Measured against baseline on Adaption's held-out test set:

Metric	Base	Adapted	Change
Win Rate	31	70	+125.8%
Science Win Rate	22	78	+254.5%
Quality Score	7.0	7.6	+8.6%

How to Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference",
    device_map="auto",
    torch_dtype="bfloat16"
)

model = PeftModel.from_pretrained(
    base_model,
    "morningstarxcdcode/adaption-medical-reasoning-sft-120b-model"
)

tokenizer = AutoTokenizer.from_pretrained(
    "morningstarxcdcode/adaption-medical-reasoning-sft-120b-model"
)

messages = [
    {"role": "system", "content": "You are a medical reasoning assistant."},
    {"role": "user", "content": "Explain the differential diagnosis for chest pain."}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=1024, temperature=0.7)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Bias, Risks, and Limitations

This model was trained on synthetic medical data. It should not be used for real clinical decisions. The model may produce plausible-sounding but incorrect medical information. Always consult qualified healthcare professionals for medical advice.

Technical Specifications

• Architecture: LlamaForCausalLM (80 layers, 64 attention heads, 8 KV heads)
• Hidden size: 8192
• Vocab size: 128,256
• Max position embeddings: 131,072
• Precision: bfloat16
• PEFT version: 0.15.1
• Transformers version: 5.10.1

Links

• Dataset: morningstarxcdcode/adaption-medical-reasoning-sft-120b
• Demo: morningstarxcdcode/adaption-medical-reasoning-demo
• GitHub: LusterSourav/adaption-autoscientist-challenge

Team

Sourav Rajak, Priyanshu Tomar, Roshan G, Vivek Rajput

Acknowledgments

Built using Adaption Labs' AutoScientist and Adaptive Data platforms for the AutoScientist Challenge.