Upload README.md with huggingface_hub

README.md

@@ -1,3 +1,200 @@
----
-license: apache-2.0
----
+# OvDSGG Readme
+
+[![arXiv](https://img.shields.io/badge/arXiv-2608.14835-b31b1b.svg)](https://arxiv.org/abs/2608.14835)
+
+**Accepted at the ECCV 2026 Contextus Workshop.** _[Read the paper on arXiv](https://arxiv.org/abs/2608.14835)._
+
+An open-vocabulary dynamic scene graph generation model (DSGG) that integrates the closed-set DSGG [OED](https://github.com/guanw-pku/OED) with the open-vocabulary scene graph generation model (SGG) [OvSGTR](https://github.com/gpt4vision/OvSGTR/). We use OvSGTR as the spatial feature extractor and feed its outputs into OED's temporal routing module.
+
+The open-vocabulary setting was benchmarked against [an open-vocabulary adaptation of OED](https://github.com/jhelsby/oed) and [a version of OvSGTR adapted for DSGG](https://github.com/jhelsby/OvSGTR).
+
+## Contents
+* [Setup](#setup)
+* [Prepare Data](#prepare-data)
+* [Train](#train)
+* [Evaluate](#evaluate)
+* [Checkpoints](#checkpoints)
+* [Open-Vocabulary Data Split](#open-vocabulary-data-split)
+* [Key Losses and Metrics](#key-losses-and-metrics)
+
+## Setup
+
+You must use Python 3.9 for this repo to work, due to the dependencies of OvSGTR, which this repo is built upon.
+
+```bash
+conda create -n myenv python=3.9
+conda activate myenv
+./install.sh
+```
+
+Sometimes GroundingDINO gives a `ModuleNotFoundError` - I think this is because some HexGPU nodes are using different Python versions. You can try rebuilding it with:
+```bash
+pip install -e ./GroundingDINO --no-build-isolation
+```
+
+## Prepare Data
+
+We use the dataset Action Genome to train and evaluate OvDSGG. Please process the downloaded dataset with the [Toolkit](https://github.com/JingweiJ/ActionGenome) and put the [processed annotation files](https://drive.google.com/drive/folders/1tdfAyYm8GGXtO2okAoH1WgVHVOTl1QYe) with COCO style into annotations folder. The directories of the dataset should look like:
+```
+|-- action_genome
+    |-- annotations   # gt annotations
+        |-- ag_train_coco_style.json
+        |-- ag_test_coco_style.json
+        |-- ...
+    |-- frames        # sampled frames
+    |-- videos        # original videos
+```
+
+[Zero-Shot Recall (zR@K)](https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch/blob/master/METRICS.md) for Action Genome is calculated using the file [`datasets/ov_zeroshot_triplet.pytorch`](./datasets/ov_zeroshot_triplet.pytorch), in this repo. This file was generated using the following script:
+
+```bash
+python tools/generate_ag_zeroshot.py --ann_file data/action_genome/annotations/ag_train_coco_style.json
+```
+
+## Train
+
+We train the spatial module first, then the temporal module on the best spatial checkpoint.
+
+### Closed-set training
+
+To run closed-set training, run these commands in order:
+
+**1. Spatial Training**
+
+Prior to training, download the [pretrained OvSGTR closed-set SGG Swin-T checkpoint](https://github.com/gpt4vision/OvSGTR/?tab=readme-ov-file#-checkpoints-closed-set-sgg) from [HuggingFace](https://huggingface.co/JosephZ/OvSGTR/blob/main/vg-swint-full.pth) and save it to `./checkpoints/vg-swint-full.pth`.
+
+```bash
+python scripts/train_spatial_sgdet_ovdsgg_closed_set.py
+```
+
+**2. Temporal Training**
+```bash
+python scripts/train_temporal_sgdet_ovdsgg_closed_set.py
+```
+
+### Open-vocabulary training
+
+To run open-vocabulary training, run these commands in order:
+
+**1. Spatial Training**
+
+Prior to training, download the [pretrained OvSGTR OvD+R-SGG Swin-T checkpoint](https://github.com/gpt4vision/OvSGTR/?tab=readme-ov-file#-checkpoints-ovdr-sgg) from [HuggingFace](https://huggingface.co/JosephZ/OvSGTR/blob/main/vg-ovdr-swint.pth) and save it to `./checkpoints/vg-ovdr-swint.pth`.
+
+```bash
+python scripts/train_spatial_sgdet_ovdsgg_ovdr.py
+```
+
+**2. Temporal Training**
+```bash
+python scripts/train_temporal_sgdet_ovdsgg_ovdr.py
+```
+
+### Ablations
+
+Our baseline derives object pair predictions and predicate classifications directly from isolated query representations, without interaction modules. To train the baseline, run these commands:
+
+**Closed-set Baseline**
+```bash
+python scripts/train_ablation_baseline_sgdet_ovdsgg_closed_set.py
+```
+
+**Open-vocabulary Baseline**
+```bash
+python scripts/train_ablation_baseline_sgdet_ovdsgg_ovdr.py
+```
+
+## Evaluate
+
+### Closed-set evaluation
+
+To evaluate closed-set models, run these commands:
+
+**1. Spatial Evaluation**
+```bash
+python scripts/eval_spatial_sgdet_ovdsgg_closed_set.py
+```
+
+**2. Temporal Evaluation**
+```bash
+python scripts/eval_temporal_sgdet_ovdsgg_closed_set.py
+```
+
+### Open-vocabulary evaluation
+
+To evaluate open-vocabulary models, run these commands:
+
+**1. Spatial Evaluation**
+```bash
+python scripts/eval_spatial_sgdet_ovdsgg_ovdr.py
+```
+
+**2. Temporal Evaluation**
+```bash
+python scripts/eval_temporal_sgdet_ovdsgg_ovdr.py
+```
+
+### Ablation evaluation
+
+To evaluate the ablation baseline models, run these commands:
+
+**Closed-set Baseline**
+```bash
+python scripts/eval_ablation_baseline_sgdet_ovdsgg_closed_set.py
+```
+
+**Open-vocabulary Baseline**
+```bash
+python scripts/eval_ablation_baseline_sgdet_ovdsgg_ovdr.py
+```
+
+## Checkpoints
+
+To run the evaluation scripts using our pre-trained models, download the necessary checkpoint here:
+
+* **Download**: [link](https://drive.google.com/drive/folders/1koo9E85QpemNhS7V1teUg8LJDUXM7RNV?usp=drive_link)
+
+Once downloaded, place each checkpoint file in its respective experiment directory so the evaluation scripts can find them automatically by default:
+* **Spatial Open-Vocabulary**: `exps/spatial_sgdet_ovdsgg_ovdr/checkpoint.pth`
+* **Temporal Open-Vocabulary**: `exps/temporal_sgdet_ovdsgg_ovdr/checkpoint.pth`
+* **Spatial Closed-Set**: `exps/spatial_sgdet_ovdsgg_closed_set/checkpoint.pth`
+* **Temporal Closed-Set**: `exps/temporal_sgdet_ovdsgg_closed_set/checkpoint.pth`
+* **Ablation Open-Vocabulary**: `exps/ablation_baseline_ovdr/checkpoint.pth`
+* **Ablation Closed-Set**: `exps/ablation_baseline_closed_set/checkpoint.pth`
+
+
+## Open-Vocabulary Data Split
+
+To test open-vocabulary capabilities of OvDSGG, we train on only 70% of the categories in Action Genome. The rest are only seen during evaluation. To do this, we split the categories into Base (seen during training) and Novel (unseen during training)
+
+The open-vocabulary training split is hardcoded in [datasets/ag.py](./datasets/ag.py), and has been chosen such that:
+
+* ~70% of objects are in Base (25 out of 36, 69.4%).
+* ~70% of predicates are in Base (18 out of 26, 69.2%).
+* Common and rare categories are balanced across Base and Novel sets.
+* None of the Novel categories were seen by the pretrained open-vocabulary OvSGTR/GroundingDINO checkpoint.
+
+See PRs [#12](https://github.com/jhelsby/OvDSGG/pull/12) and [#13](https://github.com/jhelsby/OvDSGG/pull/13) for details. To generate the split yourself, run:
+
+```bash
+python tools/propose_ag_split.py
+```
+
+## Key Losses and Metrics
+
+| Metric / Loss | Type | Description |
+| :--- | :--- | :--- |
+| **`loss`** | **Total Loss** | Weighted sum of all scaled losses minimized by the optimizer. |
+| **`loss_obj_ce`** | Object Loss | Cross Entropy loss for object classification. |
+| **`loss_obj_bbox`** | Object Loss | L1 error for object bounding box center and size. |
+| **`loss_obj_giou`** | Object Loss | Generalized IoU loss for object bounding box overlap. |
+| **`loss_sub_bbox`** | Subject Loss | L1 error for subject bounding box center and size. |
+| **`loss_sub_giou`** | Subject Loss | Generalized IoU loss for subject bounding box overlap. |
+| **`loss_attn_ce`** | Relation Loss | Loss for classifying the "attention" or main interaction. |
+| **`loss_spatial_ce`** | Relation Loss | Loss for classifying spatial relationships (e.g., "in front of"). |
+| **`loss_contacting_ce`** | Relation Loss | Loss for classifying contacting relationships (e.g., "holding"). |
+| **`obj_class_error_unscaled`** | **Key Metric** | The raw percentage of objects misclassified (Error Rate %). |
+
+## References
+
+Please note that most of the code in this repository was adapted from [OED](https://github.com/guanw-pku/OED) and [OvSGTR](https://github.com/gpt4vision/OvSGTR/). We thank the authors for their excellent work.
+