README.md

---
license: apache-2.0
library_name: pytorch
tags:
- computer-vision
- scene-graph-generation
- open-vocabulary
- dynamic-scene-graph-generation
- action-genome
datasets:
- action-genome
pipeline_tag: object-detection
---

# OvDSGG Readme

[![arXiv](https://img.shields.io/badge/arXiv-2608.14835-b31b1b.svg)](https://arxiv.org/abs/2608.14835)

**Accepted at the ECCV 2026 Contextus Workshop.** _[Read the paper on arXiv](https://arxiv.org/abs/2608.14835)._

An open-vocabulary dynamic scene graph generation model (DSGG) that integrates the closed-set DSGG [OED](https://github.com/guanw-pku/OED) with the open-vocabulary scene graph generation model (SGG) [OvSGTR](https://github.com/gpt4vision/OvSGTR/). We use OvSGTR as the spatial feature extractor and feed its outputs into OED's temporal routing module.

The open-vocabulary setting was benchmarked against [an open-vocabulary adaptation of OED](https://github.com/jhelsby/oed) and [a version of OvSGTR adapted for DSGG](https://github.com/jhelsby/OvSGTR).

## Contents
* [Setup](#setup)
* [Prepare Data](#prepare-data)
* [Train](#train)
* [Evaluate](#evaluate)
* [Checkpoints](#checkpoints)
* [Open-Vocabulary Data Split](#open-vocabulary-data-split)
* [Key Losses and Metrics](#key-losses-and-metrics)

## Setup

You must use Python 3.9 for this repo to work, due to the dependencies of OvSGTR, which this repo is built upon.

```bash
conda create -n myenv python=3.9
conda activate myenv
./install.sh
```

## Prepare Data

We use the dataset Action Genome to train and evaluate OvDSGG. Please process the downloaded dataset with the [Toolkit](https://github.com/JingweiJ/ActionGenome) and put the [processed annotation files](https://drive.google.com/drive/folders/1tdfAyYm8GGXtO2okAoH1WgVHVOTl1QYe) with COCO style into annotations folder. The directories of the dataset should look like:
```
|-- action_genome
    |-- annotations   # gt annotations
        |-- ag_train_coco_style.json
        |-- ag_test_coco_style.json
        |-- ...
    |-- frames        # sampled frames
    |-- videos        # original videos
```

[Zero-Shot Recall (zR@K)](https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/METRICS.md) for Action Genome is calculated using the file [`datasets/ov_zeroshot_triplet.pytorch`](./datasets/ov_zeroshot_triplet.pytorch), in this repo. This file was generated using the following script:

```bash
python tools/generate_ag_zeroshot.py --ann_file data/action_genome/annotations/ag_train_coco_style.json
```

## Train

We train the spatial module first, then the temporal module on the best spatial checkpoint.

### Closed-set training

To run closed-set training, run these commands in order:

**1. Spatial Training**

Prior to training, download the [pretrained OvSGTR closed-set SGG Swin-T checkpoint](https://github.com/gpt4vision/OvSGTR/?tab=readme-ov-file#-checkpoints-closed-set-sgg) from [HuggingFace](https://huggingface.co/JosephZ/OvSGTR/blob/main/vg-swint-full.pth) and save it to `./checkpoints/vg-swint-full.pth`.

```bash
python scripts/train_spatial_sgdet_ovdsgg_closed_set.py
```

**2. Temporal Training**
```bash
python scripts/train_temporal_sgdet_ovdsgg_closed_set.py
```

### Open-vocabulary training

To run open-vocabulary training, run these commands in order:

**1. Spatial Training**

Prior to training, download the [pretrained OvSGTR OvD+R-SGG Swin-T checkpoint](https://github.com/gpt4vision/OvSGTR/?tab=readme-ov-file#-checkpoints-ovdr-sgg) from [HuggingFace](https://huggingface.co/JosephZ/OvSGTR/blob/main/vg-ovdr-swint.pth) and save it to `./checkpoints/vg-ovdr-swint.pth`.

```bash
python scripts/train_spatial_sgdet_ovdsgg_ovdr.py
```

**2. Temporal Training**
```bash
python scripts/train_temporal_sgdet_ovdsgg_ovdr.py
```

### Ablations

Our baseline derives object pair predictions and predicate classifications directly from isolated query representations, without interaction modules. To train the baseline, run these commands:

**Closed-set Baseline**
```bash
python scripts/train_ablation_baseline_sgdet_ovdsgg_closed_set.py
```

**Open-vocabulary Baseline**
```bash
python scripts/train_ablation_baseline_sgdet_ovdsgg_ovdr.py
```

## Evaluate

### Closed-set evaluation

To evaluate closed-set models, run these commands:

**1. Spatial Evaluation**
```bash
python scripts/eval_spatial_sgdet_ovdsgg_closed_set.py
```

**2. Temporal Evaluation**
```bash
python scripts/eval_temporal_sgdet_ovdsgg_closed_set.py
```

### Open-vocabulary evaluation

To evaluate open-vocabulary models, run these commands:

**1. Spatial Evaluation**
```bash
python scripts/eval_spatial_sgdet_ovdsgg_ovdr.py
```

**2. Temporal Evaluation**
```bash
python scripts/eval_temporal_sgdet_ovdsgg_ovdr.py
```

### Ablation evaluation

To evaluate the ablation baseline models, run these commands:

**Closed-set Baseline**
```bash
python scripts/eval_ablation_baseline_sgdet_ovdsgg_closed_set.py
```

**Open-vocabulary Baseline**
```bash
python scripts/eval_ablation_baseline_sgdet_ovdsgg_ovdr.py
```

## Checkpoints

The trained OvDSGG checkpoints are available from Hugging Face and Google Drive:

* **Hugging Face**: [jhelsby/OvDSGG](https://huggingface.co/jhelsby/OvDSGG) - includes checkpoints and matching config files
* **Google Drive**: [link](https://drive.google.com/drive/folders/1koo9E85QpemNhS7V1teUg8LJDUXM7RNV?usp=drive_link)

To download the Hugging Face checkpoints directly into the expected local paths, run:

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="jhelsby/OvDSGG",
    local_dir=".",
    allow_patterns=["exps/*/checkpoint.pth", "config/*.py"],
)
```

If downloading from Google Drive manually, place each checkpoint file at the corresponding default path:
* **Spatial Open-Vocabulary**: `exps/spatial_sgdet_ovdsgg_ovdr/checkpoint.pth`
* **Temporal Open-Vocabulary**: `exps/temporal_sgdet_ovdsgg_ovdr/checkpoint.pth`
* **Spatial Closed-Set**: `exps/spatial_sgdet_ovdsgg_closed_set/checkpoint.pth`
* **Temporal Closed-Set**: `exps/temporal_sgdet_ovdsgg_closed_set/checkpoint.pth`
* **Ablation Open-Vocabulary**: `exps/ablation_baseline_ovdr/checkpoint.pth`
* **Ablation Closed-Set**: `exps/ablation_baseline_closed_set/checkpoint.pth`


## Open-Vocabulary Data Split

To test open-vocabulary capabilities of OvDSGG, we train on only 70% of the categories in Action Genome. The rest are only seen during evaluation. To do this, we split the categories into Base (seen during training) and Novel (unseen during training)

The open-vocabulary training split is hardcoded in [datasets/ag.py](./datasets/ag.py), and has been chosen such that:

* ~70% of objects are in Base (25 out of 36, 69.4%).
* ~70% of predicates are in Base (18 out of 26, 69.2%).
* Common and rare categories are balanced across Base and Novel sets.
* None of the Novel categories were seen by the pretrained open-vocabulary OvSGTR/GroundingDINO checkpoint.

See PRs [#12](https://github.com/jhelsby/OvDSGG/pull/12) and [#13](https://github.com/jhelsby/OvDSGG/pull/13) for details. To generate the split yourself, run:

```bash
python tools/propose_ag_split.py
```

## Key Losses and Metrics

| Metric / Loss | Type | Description |
| :--- | :--- | :--- |
| **`loss`** | **Total Loss** | Weighted sum of all scaled losses minimized by the optimizer. |
| **`loss_obj_ce`** | Object Loss | Cross Entropy loss for object classification. |
| **`loss_obj_bbox`** | Object Loss | L1 error for object bounding box center and size. |
| **`loss_obj_giou`** | Object Loss | Generalized IoU loss for object bounding box overlap. |
| **`loss_sub_bbox`** | Subject Loss | L1 error for subject bounding box center and size. |
| **`loss_sub_giou`** | Subject Loss | Generalized IoU loss for subject bounding box overlap. |
| **`loss_attn_ce`** | Relation Loss | Loss for classifying the "attention" or main interaction. |
| **`loss_spatial_ce`** | Relation Loss | Loss for classifying spatial relationships (e.g., "in front of"). |
| **`loss_contacting_ce`** | Relation Loss | Loss for classifying contacting relationships (e.g., "holding"). |
| **`obj_class_error_unscaled`** | **Key Metric** | The raw percentage of objects misclassified (Error Rate %). |

## References

Please note that most of the code in this repository was adapted from [OED](https://github.com/guanw-pku/OED) and [OvSGTR](https://github.com/gpt4vision/OvSGTR/). We thank the authors for their excellent work.