PathoSynVLM: Case-Level Pathology Synoptic Report Generation

PathoSynVLM is a token-efficient vision-language model for generating case-level pathology synoptic reports from one or more whole-slide images represented as precomputed CONCHv1.5 patch embeddings.

Code and complete documentation: https://github.com/AtlasAnalyticsLab/PathoSynVLM

This repository provides the trained model package. Use it with the PathoSynVLM code repository for embedding preparation, case-level inference, evaluation, and training.

What This Repository Contains

Path	Purpose
llm/model.safetensors	Merged language-model weights for the selected Stage 2 checkpoint.
vlm_state.pt	Vision-language aligner, WSI marker, and WSI index tensors.
tokenizer/	Tokenizer configuration and chat template used for inference.
config.json	PathoSynVLM architecture and inference settings.
labels.json	Input contract and generated report-field schema.
best_checkpoint_summary.json	Selected checkpoint and validation summary.
model_index.json	Machine-readable task, dataset, and metric metadata.
examples/	Example case-level input manifest.

The paper run used unfreeze_llm_base=true, so the release package includes the merged/full language-model weights under llm/, not only a LoRA adapter.

Quick Start

The release is loaded through the PathoSynVLM inference code rather than directly through transformers.AutoModel. A CUDA-capable GPU is recommended for normal use; CPU execution is intended for smoke tests.

1. Install the code

git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
cd PathoSynVLM
conda create -n pathosynvlm python=3.11 -y
conda activate pathosynvlm
export PYTHONNOUSERSITE=1
pip install -e .

2. Download the model

source configs/paths.example.env
hf download AtlasAnalyticsLab/PathoSynVLM \
  --local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"

3. Generate a report for one case

Pass every WSI embedding file belonging to the case in the desired slide order:

# Optional when the paths below are relative.
export PATHOSYNVLM_EMBEDDINGS_ROOT=/path/to/conch_v15/embeddings

python scripts/generate_case_report.py \
  --embeddings case_001/slide_1.h5 case_001/slide_2.h5 \
  --output_json report.json

Relative --embeddings paths are resolved under PATHOSYNVLM_EMBEDDINGS_ROOT; absolute .h5 paths work without setting that variable. The JSON output records the generated report, resolved slide paths, per-WSI patch counts, and feature key.

Generated report text follows:

Diagnosis: ...
Certainty: ...
Conclusion: ...

Input Format

PathoSynVLM runs on precomputed WSI patch embeddings, not raw WSIs. Each .h5 file should contain:

/features/conch_v15  # shape: (num_patches, 768)

See the GitHub embedding guide for patch extraction, feature generation, H5 validation, and configurable storage paths. Dataset placement and access requirements are documented in the data guide.

From Precomputed H5 Feature Files

This is the fastest path. Put one or more WSI embedding files for a case into the --embeddings argument:

python scripts/generate_case_report.py \
  --embeddings case_slide_1.h5 case_slide_2.h5 \
  --output_json report.json

From Raw Whole-Slide Images

First extract tissue patches and CONCHv1.5 patch embeddings using a WSI preprocessing pipeline that writes the H5 layout above. Then pass the resulting H5 files to scripts/generate_case_report.py. PathoSynVLM does not send raw WSI pixels directly to the language model.

Running the Paper Pipeline

Follow the GitHub paper pipeline for the complete sequence: dataset setup, CONCHv1.5 embedding generation, metadata preparation, Stage 1 alignment, Stage 2 case-level fine-tuning, and evaluation. Machine-readable paper configurations and reported values are maintained in the same code repository.

Training Data

The released Stage 2 checkpoint was fine-tuned on case-report pairs from HISTAI. The official metadata repository is the starting point for dataset access and links to the organ-specific WSI repositories used by HISTAI. See the HISTAI source documentation for the dataset structure, subsets, citation, and access instructions.

HISTAI data remain subject to the dataset's CC BY-NC 4.0 license and current access requirements.

Training Recipe

• Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while keeping the CONCHv1.5 patch encoder and LLM frozen.
• Stage 2: fine-tune on HISTAI case-report pairs with WSI marker tokens.

Checkpoint selected for release:

• checkpoint step: 30400
• checkpoint epoch: 7
• validation loss: 1.010892
• prompt style: double
• patch level: 5x_512
• max vision tokens: 4096

Reported Metrics

Stage 1 aligner-only training:

ROUGE-L	METEOR	BLEU-4	BERTScore F1
0.4743	0.4810	0.1247	0.4253

Stage 2 HISTAI main result:

ROUGE-L	METEOR	BLEU-4	BERTScore F1	Diagnosis Exact	Diagnosis Relaxed	Certainty
0.2495	0.1988	0.0525	0.3018	0.1667	0.3333	0.9000

Intended Use

This model is intended for research on pathology report generation from precomputed WSI patch embeddings.

It is not a clinical diagnostic device and should not be used for patient care without appropriate validation, regulatory review, and expert oversight.

License and Commercial Use

This repository uses CC BY-NC-SA 4.0. Research and non-commercial use only. Dataset access, pretrained third-party models, and any externally hosted model weights remain subject to their own terms.

Citation

@inproceedings{yang2026simpletokenvlm,
  title     = {Simple Token-Efficient Vision-Language Model for Case-Level Pathology Synoptic Report Generation},
  author    = {Yang, Zhiyuan and Cheng, Jiahao and Trinh, Vincent Quoc-Huy and Hosseini, Mahdi S.},
  booktitle = {Proceedings of the 7th International Conference on Deep Learning Theory and Applications},
  pages     = {514--537},
  year      = {2026},
  issn      = {2184-9277}
}