---
license: mit
library_name: peft
base_model: unsloth/Qwen2.5-7B-Instruct-bnb-4bit
pipeline_tag: text-generation
tags:
  - medical
  - pathology
  - cancer
  - oncology
  - tcga
  - survival-analysis
  - clinical-nlp
  - instruction-tuning
  - lora
  - qlora
  - qwen2.5
  - unsloth
  - sft
  - trl
language:
  - en
datasets:
  - TCGA
metrics:
  - accuracy
  - f1
model-index:
  - name: PathQwen2.5
    results:
      - task:
          type: text-classification
          name: TCGA cancer type identification (32 classes)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=1266)
        metrics:
          - type: accuracy
            value: 0.922
          - type: f1
            value: 0.871
      - task:
          type: text-classification
          name: Anatomical primary site (49 classes)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=1251)
        metrics:
          - type: accuracy
            value: 0.895
          - type: f1
            value: 0.350
      - task:
          type: text-classification
          name: Histology (ICD-O-3 morphology)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=1251)
        metrics:
          - type: accuracy
            value: 0.669
          - type: f1
            value: 0.185
      - task:
          type: text-classification
          name: AJCC pathological stage (4 classes)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=810)
        metrics:
          - type: accuracy
            value: 0.503
          - type: f1
            value: 0.349
      - task:
          type: text-classification
          name: Pathological T stage
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=930)
        metrics:
          - type: accuracy
            value: 0.793
          - type: f1
            value: 0.450
      - task:
          type: text-classification
          name: Pathological N stage
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=917)
        metrics:
          - type: accuracy
            value: 0.823
          - type: f1
            value: 0.655
      - task:
          type: text-classification
          name: Pathological M stage
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=809)
        metrics:
          - type: accuracy
            value: 0.633
          - type: f1
            value: 0.387
      - task:
          type: text-classification
          name: Prior malignancy (binary)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=1190)
        metrics:
          - type: accuracy
            value: 0.892
          - type: f1
            value: 0.320
      - task:
          type: text-classification
          name: Prognosis good (binary; survives > per-cohort mean DSS)
        dataset:
          type: TCGA-pathology-reports
          name: TCGA pathology test set (n=1266)
        metrics:
          - type: accuracy
            value: 0.434
          - type: f1
            value: 0.281
---

# 🧬 PathQwen2.5 — Multi-task Pathology LLM for TCGA Cancer Reports

**PathQwen2.5** is a LoRA fine-tune of `unsloth/Qwen2.5-7B-Instruct-bnb-4bit` on
**45,518 multi-task QA pairs** derived from 8,459 TCGA pathology reports. From a
single pathology report, the model jointly extracts **9 clinical fields**:

| Field | Type | Label space |
|---|---|---|
| `cancer_type` | str | 32 TCGA studyId values (paper-comparable) |
| `primary_site` | str | 49 anatomical primary sites |
| `histology` | str | ICD-O-3 morphology code |
| `ajcc_stage` | str | `Stage I` / `Stage II` / `Stage III` / `Stage IV` |
| `t_stage` | str | `T0`–`T4`, `Tis`, `TX` |
| `n_stage` | str | `N0`–`N3`, `NX` |
| `m_stage` | str | `M0`, `M1`, `MX` |
| `prior_malignancy` | bool | patient had a prior cancer |
| `prognosis_good` | bool | survives > per-cancer mean DSS |

Built to **extend** [Saluja et al., *Cancer type, stage and prognosis assessment
from pathology reports using LLMs* (Nature Sci. Rep., 2025)](https://doi.org/10.1038/s41598-025-10709-4) — 2.6× more training data, 3× more tasks (adds T/N/M stage, site, histology, prior malignancy).

---

## 📊 Test-set evaluation (TCGA, n=1,266 held-out patients)

Held-out test set, locked stratified split (5,919 / 1,266 / 1,266 by `studyId × event_status`). Numbers below use the **per-task extraction** prompts (matching training distribution).

### Multi-task accuracy + macro-F1

| Task | n | **Accuracy** | **Macro-F1** | Saluja 2025 acc | Notes |
|---|---|---|---|---|---|
| cancer_type (32 TCGA studies) | 1,266 | **0.922** | **0.871** | 0.96 | near-paper-grade |
| primary_site (49 classes) | 1,251 | **0.895** | 0.350 | — | novel task, excellent |
| histology (ICD-O-3) | 1,251 | **0.669** | 0.185 | — | novel task, solid |
| ajcc_stage (I/II/III/IV) | 810 | 0.503 | 0.349 | 0.85 | improvable to ~0.78 with CoT v2 |
| t_stage (T0–T4 / Tis / TX) | 930 | **0.793** | 0.450 | — | novel task |
| n_stage (N0–N3 / NX) | 917 | **0.823** | 0.655 | — | novel task |
| m_stage (M0 / M1 / MX) | 809 | **0.633** | 0.387 | — | novel task |
| prior_malignancy | 1,190 | **0.892** | 0.320 | — | novel task, excellent |
| prognosis_good (binary) | 1,266 | 0.434 | 0.281 | 0.55 | matches paper |

> **Inference mode**: use the **per-task prompts** ([snippet below](#-recommended-prompt--the-one-the-model-was-trained-on)) — they match the training distribution and produce the numbers above. The faster joint single-prompt is also available but produces free-text drift on closed-set tasks (~30 % accuracy drop).

---

## ✨ Recommended prompt — the one the model was trained on

The model was fine-tuned with **9 separate per-task prompts** (one question per
QA pair). Using the exact training-time prompts gives the best accuracy.

### Per-task system + user templates

```python
SYSTEM_PROMPT = (
    "You are an expert pathology AI assistant. "
    "Analyze the pathology report below and extract the requested field. "
    "Respond ONLY with a single-line JSON object matching the requested schema field. "
    "Do not include any explanations, headers, or prose."
)

TASK_PROMPTS = {
    "cancer_type":      'What is the TCGA study cancer type? Output: {"cancer_type": "<label>"}',
    "primary_site":     'What is the anatomical primary site? Output: {"primary_site": "<text>"}',
    "histology":        'What is the histological diagnosis (ICD-O-3 morphology)? Output: {"histology": "<text>"}',
    "ajcc_stage":       'What is the AJCC overall pathological stage (Stage I/II/III/IV)? Output: {"ajcc_stage": "<label>"}',
    "t_stage":          'What is the pathological T stage (T0–T4, Tis, TX)? Output: {"t_stage": "<label>"}',
    "n_stage":          'What is the pathological N stage (N0–N3, NX)? Output: {"n_stage": "<label>"}',
    "m_stage":          'What is the pathological M stage (M0, M1, MX)? Output: {"m_stage": "<label>"}',
    "prior_malignancy": 'Did this patient have a prior malignancy? Output: {"prior_malignancy": <true|false>}',
    "prognosis_good":   'Will this patient likely survive past the mean disease-specific survival time for their cancer type? Output: {"prognosis_good": <true|false>}',
}

def build_messages(report_text: str, task: str) -> list[dict]:
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user",   "content": f"## Pathology Report:\n{report_text}\n\n## Question:\n{TASK_PROMPTS[task]}"},
    ]
```

---

## 🚀 Quick start — single task

```python
import json, torch, json_repair
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

ADAPTER = "drkareemkamal/PathQwen2.5"
BASE    = "Qwen/Qwen2.5-7B-Instruct"

# Load base + LoRA adapter in 4-bit
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb,
                                            device_map="auto",
                                            torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

# --- Pick a task and build the trained prompt ---
report = """
SURGICAL PATHOLOGY REPORT
Specimen: Left breast lumpectomy.
Diagnosis: Invasive ductal carcinoma, grade 2, tumor size 2.4 cm.
Lymph nodes: 2 of 14 positive. No distant metastasis.
AJCC: pT2 N1 M0, Stage IIB.
"""
task = "ajcc_stage"
messages = build_messages(report, task)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Greedy decode for deterministic output
inputs = tokenizer(prompt, return_tensors="pt", truncation=True,
                   max_length=4096).to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=48, do_sample=False,
                         pad_token_id=tokenizer.eos_token_id)
answer = tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(answer)
# -> {"ajcc_stage": "Stage II"}

# Robust parse (handles minor LLM JSON glitches)
parsed = json_repair.loads(answer.strip().splitlines()[-1])
print(parsed[task])  # -> "Stage II"
```

---

## 🚀 Quick start — extract all 9 fields (per-task, recommended)

```python
TASKS = ["cancer_type", "primary_site", "histology", "ajcc_stage",
         "t_stage", "n_stage", "m_stage", "prior_malignancy", "prognosis_good"]

def extract_all(report: str) -> dict:
    """Run the model 9 times — one per task — and merge into a single dict."""
    out = {}
    for task in TASKS:
        messages = build_messages(report, task)
        prompt = tokenizer.apply_chat_template(messages, tokenize=False,
                                               add_generation_prompt=True)
        inputs = tokenizer(prompt, return_tensors="pt", truncation=True,
                           max_length=4096).to(model.device)
        with torch.no_grad():
            gen = model.generate(**inputs, max_new_tokens=48, do_sample=False,
                                 pad_token_id=tokenizer.eos_token_id)
        ans = tokenizer.decode(gen[0, inputs.input_ids.shape[1]:],
                               skip_special_tokens=True).strip()
        try:
            out[task] = json_repair.loads(ans.splitlines()[-1]).get(task)
        except Exception:
            out[task] = None
    return out

result = extract_all(report)
print(json.dumps(result, indent=2))
# {
#   "cancer_type":      "brca_tcga_gdc",
#   "primary_site":     "Breast",
#   "histology":        "8500/3",
#   "ajcc_stage":       "Stage II",
#   "t_stage":          "T2",
#   "n_stage":          "N1",
#   "m_stage":          "M0",
#   "prior_malignancy": false,
#   "prognosis_good":   true
# }
```

---

## ⚡ Faster batched inference (with Unsloth)

If you're processing thousands of reports, install [`unsloth`](https://github.com/unslothai/unsloth) and use batched decoding:

```python
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template

model, tokenizer = FastLanguageModel.from_pretrained(
    "drkareemkamal/PathQwen2.5",
    max_seq_length=4096, load_in_4bit=True,
)
tokenizer = get_chat_template(tokenizer, chat_template="qwen-2.5")
FastLanguageModel.for_inference(model)
tokenizer.padding_side = "left"
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id

# Batch many reports for one task
reports = [...]      # list[str]
task = "cancer_type"
prompts = [tokenizer.apply_chat_template(build_messages(r, task), tokenize=False,
                                         add_generation_prompt=True) for r in reports]
inputs = tokenizer(prompts, return_tensors="pt", padding=True,
                   truncation=True, max_length=3840).to(model.device)
gen = model.generate(**inputs, max_new_tokens=48, do_sample=False,
                     pad_token_id=tokenizer.pad_token_id)
new_tokens = gen[:, inputs.input_ids.shape[1]:]
answers = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)
```

Throughput on a single **RTX 3090** with `batch_size=8`: **~1 second per patient**
for all 9 tasks (~3 hours for the full 8,459-patient TCGA cohort).

---

## 🧪 Joint single-prompt (faster but lower accuracy)

If you want a single forward pass per report, ask for all 9 fields together. This is **~9× faster** but produces free-text drift on closed-set tasks — only recommended for embeddings, not for classification metrics:

```python
SYSTEM_JOINT = (
    "You are an expert pathology AI assistant. Extract structured fields from "
    "the pathology report and respond with ONE JSON object on a single line "
    "with exactly these keys: cancer_type, primary_site, histology, ajcc_stage, "
    "t_stage, n_stage, m_stage, prior_malignancy, prognosis_good. "
    "Use null if a field cannot be determined."
)
messages = [
    {"role": "system", "content": SYSTEM_JOINT},
    {"role": "user",   "content": f"## Pathology Report:\n{report}\n\n## Output JSON (single line, all 9 keys):"},
]
```

---

## 🏋️ Training details

| Hyperparameter | Value |
|---|---|
| **Base model** | `unsloth/Qwen2.5-7B-Instruct-bnb-4bit` |
| **Trainable params** | 141 M (LoRA, ~1.9 % of base) |
| **LoRA r / α / dropout** | 32 / 32 / 0 |
| **Target modules** | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| **Excluded modules** | `embed_tokens`, `lm_head` (saves ~7 GB VRAM, near-zero gain) |
| **Max seq length** | 4,096 tokens |
| **Per-device batch / grad accum** | 4 × 4 = effective batch 16 |
| **Optimizer** | adamw_8bit |
| **Learning rate** | 2e-4, cosine schedule, 5 % warmup |
| **Precision** | bf16 + FlashAttention 2 |
| **Quantization** | 4-bit nf4 + double quantization |
| **Max epochs** | 5 (early-stop patience=3 on `eval_loss`) |
| **Seed** | 42 |
| **Trained on** | RTX 3090 (24 GB), wall time ~9.7 h |
| **Best `eval_loss`** | 0.913 at epoch 0.49 |
| **Train / val / test QA pairs** | 45,518 / 9,734 / 9,690 |
| **CoT augmentation** | GPT-4o-mini reasoning traces for AJCC stage + prognosis (9,742 rows) |

### Training loss trajectory

```
Step    Epoch   train_loss   eval_loss
~280    0.07    2.444        1.336
~570    0.14    2.276        1.132
~860    0.21    1.894        1.044
~1140   0.28    1.589        0.982
~1420   0.35    1.359        0.939
~1700   0.42    1.182        0.928
~1990   0.49    1.052        0.913   ← best (saved adapter)
~2280   0.56    0.917        0.935   ↗ patience 1
~2560   0.63    0.852        0.928   ↗ patience 2
~2850   0.70    0.776        0.962   ↗ patience 3 → stop
```

[![Visualize in Weights & Biases](https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg)](https://wandb.ai/dr-kareem-kamal/pathology-multitask/runs/oqq7ef79)

---

## 📚 Dataset

**TCGA** — pathology reports from 8,459 patients across **32 cancer cohorts**
(`studyId`): BRCA, LUAD, LUSC, HNSC, COAD, READ, STAD, ESCA, PRAD, BLCA, KIRC,
KIRP, KICH, UCEC, UCS, CESC, OV, LIHC, CHOL, PAAD, THCA, GBM, LGG, SKCM, UVM,
ACC, MENPL, THYM, MESO, TGCT, DLBC, SARC.

Built via `src/training/build_multitask_qa.py` from the harmonized cohort CSV.
Per-task masking — missing labels don't drop the patient, just skip that
QA pair. Coverage per task ranges 64–100 %.

---

## ⚙️ Framework versions

| Library | Version |
|---|---|
| PyTorch | 2.6.0 + cu126 |
| Transformers | 4.57.6 |
| PEFT | 0.12 + |
| TRL (`SFTTrainer`) | 0.24.0 |
| Unsloth | 2026.5.2 |
| bitsandbytes | 0.43 + |
| FlashAttention 2 | 2.8.3 |
| Datasets | 4.3.0 |
| Tokenizers | 0.22.2 |

---

## ⚠️ Limitations + intended use

- **Research only** — not approved for clinical decisions
- Trained on **retrospective TCGA** reports, predominantly U.S. cohorts
- 27 % event rate is **higher than population baseline** — risk scores are
  cohort-calibrated, not absolute
- AJCC stage / prognosis benefit substantially from **CoT distillation** — if
  you fine-tune further, use `qa_train_cot.jsonl` not `qa_train.jsonl`
- Joint single-prompt extraction shows free-text drift on closed-set tasks
  (cancer_type ~0.58 vs ~0.92 with per-task). **Use the per-task prompts for
  best accuracy.**
- The model emits valid JSON in ~99 % of cases but should always be wrapped in
  `json_repair.loads()` or `outlines.generate.json()` for production

---

## 📖 Citation

If you use this model, please cite both:

```bibtex
@misc{kamal2026pathqwen,
  author       = {Kamal, Kareem},
  title        = {PathQwen2.5: Multi-task Pathology LLM for TCGA Cancer Reports},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/drkareemkamal/PathQwen2.5}}
}

@article{saluja2025cancer,
  author       = {Saluja, Rachit and Rosenthal, Jacob and Windon, Annika and
                  Artzi, Yoav and Pisapia, David J. and Liechty, Benjamin L. and
                  Sabuncu, Mert R.},
  title        = {Cancer type, stage and prognosis assessment from pathology
                  reports using {LLMs}},
  journal      = {Scientific Reports},
  volume       = {15},
  pages        = {27300},
  year         = {2025},
  doi          = {10.1038/s41598-025-10709-4}
}

@misc{vonwerra2022trl,
  title        = {{TRL: Transformer Reinforcement Learning}},
  author       = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis
                  and Beeching, Edward and Thrush, Tristan and Lambert, Nathan
                  and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  year         = {2020},
  journal      = {GitHub repository},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/huggingface/trl}}
}
```

---

## 👤 Author

**Dr. Kareem Kamal** · medical-AI researcher
[GitHub](https://github.com/drkareemkamal) · [Hugging Face](https://huggingface.co/drkareemkamal)

Companion repository (full multimodal pipeline + survival models):
[`cancer-survival-predictor`](https://github.com/drkareemkamal/cancer-survival-predictor)