DFD-HR, re-implemented and trained under the MARS protocol

arXiv Code MARS Original DFD-HR code

This is the DFD-HR baseline from the MARS paper: our re-implementation of DFD-HR (Sun et al., CVPR 2026), trained and evaluated under the same protocol as MARS. It is not the original authors' model or weights; training data, inputs, optimization and checkpoint selection follow the MARS protocol, so its numbers differ from those in the DFD-HR paper.

DFD-HR scores each frame with a frozen CLIP ViT-L/14 carrying trainable mixture-of-experts adapters. Every frame is seen twice, as one 224 x 224 global view and as 2 x 2 tiles of 224 x 224 cut from the 448 x 448 frame. In the last blocks, a token router passes only the top-scoring tokens and a layer router lets a view stop early. The clip score is the mean of the frame scores.

Files

file contents size
model.safetensors trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use 415 MB
config.yaml model and data settings, read by the mars package 1 KB
model.onnx complete graph, frozen encoder included; runs with ONNX Runtime alone, float32 1.64 GB
model.fp16.onnx the same graph in float16, inputs and outputs still float32; less GPU memory 0.83 GB
usual_ranges.json percentiles of the ONNX video score on the authentic and manipulated test videos, with model.onnx 3 KB
usual_ranges.fp16.json the same with model.fp16.onnx 3 KB
serving.json measured GPU memory and time per clip with ONNX Runtime (both files, batch 1 and 2), opset, outputs, encoder chunk, ONNX Runtime version 1 KB

This is the checkpoint behind the DFD-HR results in the MARS paper.

Input

  1. Decode the video as RGB and keep every second frame (0, 2, 4, ...).
  2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is padded by repeating its last frame; n_valid is the number of real frames.
  3. Resize each frame to 448 x 448 (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1]. The model applies CLIP's normalization itself.
  4. The video score is the mean P(manipulated) over its clips.

Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting.

Use with safetensors (PyTorch)

Requires the code repository:

git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector

folder = snapshot_download("vector-institute/mars-dfd-hr", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network()  # downloads CLIP ViT-L/14 on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing)  # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data


def score(path: str) -> float:
    """Mean P(manipulated) over the clips of one video."""
    probs = []
    for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
        frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
        valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
        with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
            logits = model(frames, valid).logits
        probs.append(logits.float().softmax(-1)[0, 1].item())
    return sum(probs) / len(probs)


print(score("video.mp4"))

Use with ONNX (no PyTorch, no code repository)

pip install onnxruntime-gpu av numpy pillow huggingface_hub   # or onnxruntime for CPU
name shape, type meaning
input frames (B, 64, 3, 448, 448), float32 B clips in [-1, 1]
input n_valid (B,), int64 number of real frames in each clip, 1 to 64
output fake_prob (B,), float32 P(manipulated) for each clip
output logits (B, 2), float32 authentic and manipulated logits
output frame_probs (B, 64), float32 P(manipulated) of each frame on its own, 0 on padded frames

The batch size B is free. Clip length, frame stride and image size are also stored in the file's metadata.

Both files take the full 64-frame clip. Inside the graph the encoder processes 16 frames of each clip at a time; frames are encoded independently, so this lowers memory without changing the outputs. On an A100, one clip peaks at about 3.5 GiB of GPU memory with model.onnx and 2.6 GiB with model.fp16.onnx (details in serving.json). The float16 video scores are within 0.007 of float32 on UADFV and Celeb-DF. Use model.onnx on CPU, or as the source for a TensorRT engine.

The clip logits are the mean of the per-frame logits, so fake_prob is the sigmoid of the mean logit of frame_probs over the real frames, not the mean of frame_probs. For the clip score, use fake_prob; do not average frame_probs.

usual_ranges.json gives the percentiles (0 to 100) of the video score on the authentic and on the manipulated test videos listed under Results, computed with model.onnx; usual_ranges.fp16.json is the same for model.fp16.onnx. Use them to place a score in context, not as a threshold.

import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image

session = ort.InferenceSession(
    hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"),  # or "model.fp16.onnx" for less GPU memory
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))


def to_input(rgb: np.ndarray) -> np.ndarray:
    """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
    x = rgb.astype(np.float32) / 255.0
    if x.shape[:2] != (S, S):
        channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
        x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
    return ((x - 0.5) / 0.5).transpose(2, 0, 1)


def score(path: str) -> float:
    """Mean P(manipulated) over the clips of one video."""
    with av.open(path) as container:
        kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
    probs = []
    for start in range(0, len(kept), T):
        clip = kept[start : start + T]
        n_valid = len(clip)
        clip = clip + [clip[-1]] * (T - n_valid)
        inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
        probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
    return float(np.mean(probs))


print(score("video.mp4"))

Neither ONNX file runs in bfloat16 like the evaluation behind Results, so scores differ slightly from that table. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with CUDA is installed, import torch before creating the session also works.

Training

training data FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic)
validation FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3)
clips 64 frames, every second frame, full frames at 448 x 448
encoder OpenAI CLIP ViT-L/14 at 224, frozen; MoE adapters (4 experts) after every attention and MLP, token and layer routing in blocks 21 to 23
loss cross-entropy plus the ranking term (weight 0.1)
optimization AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0
hardware 4 x NVIDIA A40

Results

Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips.

test set videos AUROC AP
FaceForensics++ (test split) 700 99.80 99.95
Celeb-DF v2 518 93.89 96.70
DeepfakeTIMIT 960 99.59 99.82
UADFV 98 96.25 96.96
HiFiFace (paired with FF++ targets) 280 86.26 88.91
GenVideo (validation split, 10 generators) 18,286 79.48 79.10
DF26 (10 generators) 2,691 74.95 95.83
average 90.03

License

The weights in this repository are released under Apache 2.0. The frozen encoder is OpenAI's CLIP ViT-L/14, downloaded from OpenAI under its own license.

Citation

Please cite the MARS paper for these weights and the protocol, and the DFD-HR paper for the method.

@misc{radwan2026marsmultiscaleappearanceresidualstreams,
      title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
      author={Ahmed Y. Radwan and Shaina Raza},
      year={2026},
      eprint={2611.12508},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2611.12508},
}

@InProceedings{Sun_2026_CVPR,
    author    = {Sun, Jiamu and Yan, Zhiyuan and Zhang, Ke-Yue and Yao, Taiping and Ding, Shouhong},
    title     = {DFD-HR: Generalizable Deepfake Detection via Hierarchical Routing Learning},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {13984-13995}
}
Downloads last month
25
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for vector-institute/mars-dfd-hr

Adapter
(6)
this model

Paper for vector-institute/mars-dfd-hr