CoTracker360

The baseline model from TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025, Datasets & Benchmarks Track).

[Paper (arXiv:2511.21946)] [Project page] [Code] [Benchmark dataset]

What it does

Given a narrow field-of-view video and query points in the first frame, CoTracker360 predicts the 3D direction to each point in every frame — including when the point has left the camera's view.

It is built on CoTracker3 (offline). Instead of predicting 2D point positions, the last decoder layer is replaced with a linear layer whose 9 outputs form a 3×3 matrix, projected to the nearest rotation. Each point's initial direction (from its pixel position and the camera intrinsics) is rotated by these per-frame rotations to give its direction in each frame.

It was trained on 5k TAPVid-360 clips that are separate from the TAPVid360-10k benchmark.

Results on TAPVid360-10k

	δavg all ↑	δavg in-frame ↑	δavg out-of-frame ↑	AD all ↓	AD in-frame ↓	AD out-of-frame ↓
CoTracker360	0.2386	0.4060	0.1160	8.27	3.95	10.98

See the paper for the full comparison with TAP and TAPVid-3D methods.

Usage

The weights (cotracker360.pth, ~100 MB) are used through the TAPVid360 code:

git clone https://github.com/finlay-hudson/TAPVid360 && cd TAPVid360
pip install -r requirements.txt -r requirements-cotracker360.txt
mkdir -p checkpoints
wget -O checkpoints/cotracker360.pth https://huggingface.co/fhudson96/cotracker360/resolve/main/cotracker360.pth
PYTHONPATH=. python src/tapvid360/evaluation/model_runner.py

The checkpoint path and dataset location are set in src/tapvid360/evaluation/eval_configs.py. See the repository README for the evaluation workflow.

Licence

CC BY-NC 4.0 (non-commercial use only), as the model is derived from CoTracker3, whose weights are licensed CC BY-NC 4.0.

Citation

@inproceedings{hudson2025tapvid,
    title         = {{TAPVid-360}: Tracking Any Point in 360 from Narrow Field of View Video},
    author        = {Hudson, Finlay G. C. and Gardner, James A. D. and Smith, William A. P.},
    booktitle     = {The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year          = {2025},
    eprint        = {2511.21946},
    archivePrefix = {arXiv}
}

@article{karaev2024cotracker3,
    title   = {{CoTracker3}: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos},
    author  = {Karaev, Nikita and Makarov, Iurii and Wang, Jianyuan and Neverova, Natalia and Vedaldi, Andrea and Rupprecht, Christian},
    journal = {arXiv preprint arXiv:2410.11831},
    year    = {2024}
}