Sudabaka commited on
Commit
65effbe
·
verified ·
1 Parent(s): e853528

feat: 上传完整实现代码(切片/教师打标/标签/训练/评测/ONNX 导出)与 FP16 版本

Browse files
README.md CHANGED
@@ -20,6 +20,16 @@ datasets:
20
  (像素差、dHash、QR 分解、Farnebäck 光流)的判定行为,可在 NVIDIA / AMD / Intel 显卡上
21
  通过 DirectML 统一推理。
22
 
 
 
 
 
 
 
 
 
 
 
23
  ## 模型详情
24
 
25
  | 项目 | 说明 |
@@ -31,7 +41,8 @@ datasets:
31
  | 输入 | `frame_a`、`frame_b`:灰度 128×128,float32,范围 [0, 1],NCHW |
32
  | 输出 | `score`:形状 [batch, 1],Sigmoid 后位于 [0, 1],越接近 1 越可能重复 |
33
  | 导出格式 | ONNX,opset 17,动态 batch |
34
- | 文件大小 | 9.95 MB |
 
35
  | 运行后端 | onnxruntime:`DmlExecutionProvider`(DirectML)/ `CPUExecutionProvider` |
36
  | 建议阈值 | 0.5;需要更低误判时可用 0.6~0.7 |
37
  | 许可证 | CC BY-NC 4.0(仅限非商业用途) |
@@ -112,6 +123,37 @@ print(float(score)) # 1.0 表示两帧重复
112
  | Epoch | 8 |
113
  | 硬件 | NVIDIA P106-100(6 GB),CUDA 12.4,PyTorch 2.6.0 |
114
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  ## 评测结果
116
 
117
  ### 与教师的一致性(验证集 78147 样本,按视频隔离)
 
20
  (像素差、dHash、QR 分解、Farnebäck 光流)的判定行为,可在 NVIDIA / AMD / Intel 显卡上
21
  通过 DirectML 统一推理。
22
 
23
+ ## 训练数据声明
24
+
25
+ 本模型在训练过程中**未分发、未再发布任何原始视频、音频或视频帧**。训练仅使用从公开网络获取的视频中抽取的灰度切片,且全部监督信号由 Easy4K 自研的 SFDP 四阶判重判决器(像素差 / dHash / QR 码 / Farnebäck 光流)自动生成,不含人工标注。
26
+
27
+ 视频著作权归原作者所有。所用素材清单见 `training-materials.txt`(含标题、BV 号、UP 主、原始链接),仅用于说明数据来源,不构成对原作品的授权或再分发。
28
+
29
+ 若权利人认为其作品不应被用于本研究,请提交 issue,我们将在 72 小时内从清单中移除,并在后续版本训练中排除该来源。
30
+
31
+ 模型权重以 **CC BY-NC 4.0** 发布,仅限研究与非商业用途。模型输出为 0~1 的相似度分数,不具备生成、重建或复现原视频内容的能力。
32
+
33
  ## 模型详情
34
 
35
  | 项目 | 说明 |
 
41
  | 输入 | `frame_a`、`frame_b`:灰度 128×128,float32,范围 [0, 1],NCHW |
42
  | 输出 | `score`:形状 [batch, 1],Sigmoid 后位于 [0, 1],越接近 1 越可能重复 |
43
  | 导出格式 | ONNX,opset 17,动态 batch |
44
+ | FP32 版本 | `model.onnx`,约 9.95 MB,对应 Easy4K「完美模式」 |
45
+ | FP16 版本 | `model_fp16.onnx`,约 5 MB(内部半精度、输入输出保持 float32),对应 Easy4K「性能模式」 |
46
  | 运行后端 | onnxruntime:`DmlExecutionProvider`(DirectML)/ `CPUExecutionProvider` |
47
  | 建议阈值 | 0.5;需要更低误判时可用 0.6~0.7 |
48
  | 许可证 | CC BY-NC 4.0(仅限非商业用途) |
 
123
  | Epoch | 8 |
124
  | 硬件 | NVIDIA P106-100(6 GB),CUDA 12.4,PyTorch 2.6.0 |
125
 
126
+ ## 完整实现代码
127
+
128
+ 模型训练、导出与数据来源记录的全部实现代码随仓库提供,位于 [`code/`](./code):
129
+
130
+ | 文件 | 作用 |
131
+ |---|---|
132
+ | `code/siai_all.py` | 一体化流水线:视频切片(`tiles`)、教师打标(`teacher`)、软标签生成(`label`)、训练(`train`)、评测(`eval`)、ONNX 导出(`pack`) |
133
+ | `code/append_sources.py` | 训练素材清单的生成与去重追加,产出可溯源的 `training-materials.txt` |
134
+
135
+ 主要命令:
136
+
137
+ ```bash
138
+ # 1) 从视频抽取灰度切片
139
+ python code/siai_all.py tiles --raw <视频目录> --segments 4 --out <切片目录>
140
+
141
+ # 2) 用四阶判决器教师生成逐片段标签 CSV
142
+ python code/siai_all.py teacher --tiles <切片目录> --labels <标签目录>
143
+
144
+ # 3) 生成软标签训练集与验证集
145
+ python code/siai_all.py label
146
+
147
+ # 4) 训练(结束后自动导出 FP32 与 FP16 两份 ONNX)
148
+ python code/siai_all.py train --out <输出目录> --epochs 1 --workers 8 --lr 1e-4
149
+
150
+ # 5) 评测 / 打包
151
+ python code/siai_all.py eval --out <输出目录>
152
+ python code/siai_all.py pack --onnx <输出目录>/easy4k_sfdp_siamese_fp32.onnx --out <发布目录>
153
+ ```
154
+
155
+ 依赖见 `requirements.txt`(PyTorch、OpenCV、onnx、onnxruntime、onnxconverter-common)。教师端二进制 `SfdpTeacher` 属于 Easy4K 应用程序的一部分,不随本仓库分发。
156
+
157
  ## 评测结果
158
 
159
  ### 与教师的一致性(验证集 78147 样本,按视频隔离)
code/append_sources.py ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # -*- coding: utf-8 -*-
2
+ """把新下载的 B 站素材出处追加到 D:\\SFDP-Viedo\\训练素材清单.txt(去重,可反复运行)。"""
3
+ import json
4
+ import os
5
+
6
+ ROOT = r"D:\SFDP-Viedo"
7
+ META = os.path.join(ROOT, "音乐MV", "sources_bili.json")
8
+ TXT = os.path.join(ROOT, "训练素材清单.txt")
9
+ MARK = os.path.join(ROOT, ".listed_bvids.json")
10
+ EXTRA_DIRS = [os.path.join(ROOT, "二次元MV"), os.path.join(ROOT, "真人MV")]
11
+ VIDEO_EXT = (".mp4", ".mkv", ".webm", ".mov", ".flv")
12
+
13
+
14
+ def load(p, d):
15
+ try:
16
+ with open(p, "r", encoding="utf-8") as f:
17
+ return json.load(f)
18
+ except Exception:
19
+ return d
20
+
21
+
22
+ HEADER = """训练数据声明
23
+ ----------------
24
+ 本模型在训练过程中未分发、未再发布任何原始视频、音频或视频帧。训练仅使用从公开网络
25
+ 获取的视频中抽取的灰度切片,且全部监督信号由 Easy4K 自研的 SFDP 四阶判重判决器
26
+ (像素差 / dHash / QR 码 / Farneback 光流)自动生成,不含人工标注。
27
+ 视频著作权归原作者所有。本清单仅用于说明数据来源,不构成对原作品的授权或再分发。
28
+ 若权利人认为其作品不应被用于本研究,请提交 issue,我们将在 72 小时内从清单中移除,
29
+ 并在后续版本训练中排除该来源。
30
+ 模型权重以 CC BY-NC 4.0 发布,仅限研究与非商业用途。
31
+
32
+ """
33
+
34
+
35
+ def ensure_header():
36
+ old = ""
37
+ if os.path.exists(TXT):
38
+ with open(TXT, "r", encoding="utf-8") as f:
39
+ old = f.read()
40
+ if "训练数据声明" in old:
41
+ return
42
+ with open(TXT, "w", encoding="utf-8") as f:
43
+ f.write(HEADER + old)
44
+
45
+
46
+ ensure_header()
47
+
48
+ listed = set(load(MARK, []))
49
+ meta = load(META, [])
50
+ new = [m for m in meta if m.get("bvid") and m["bvid"] not in listed]
51
+ if new:
52
+ with open(TXT, "a", encoding="utf-8") as f:
53
+ f.write("\n== 新增 B 站素材(自动采集,出处可溯) ==\n")
54
+ for i, m in enumerate(new, 1):
55
+ f.write("%3d. %s\n" % (i, m.get("file") or m["bvid"]))
56
+ f.write(" 标题:%s\n" % m.get("title", ""))
57
+ f.write(" 作者:%s | 时长:%ss | 清晰度:%s | 切片:待统计\n" % (
58
+ m.get("author", ""), m.get("duration", ""), m.get("qn", "")))
59
+ f.write(" 来源:https://www.bilibili.com/video/%s\n" % m["bvid"])
60
+ listed |= {m["bvid"] for m in new}
61
+ with open(MARK, "w", encoding="utf-8") as f:
62
+ json.dump(sorted(listed), f, ensure_ascii=False)
63
+ print("APPENDED", len(new), "entries", flush=True)
64
+
65
+ for d in EXTRA_DIRS:
66
+ if not os.path.isdir(d):
67
+ continue
68
+ changed = False
69
+ for n in sorted(os.listdir(d)):
70
+ if os.path.splitext(n)[1].lower() not in VIDEO_EXT:
71
+ continue
72
+ key = "manual:" + os.path.basename(d) + "/" + n
73
+ if key in listed:
74
+ continue
75
+ with open(TXT, "a", encoding="utf-8") as f:
76
+ f.write("\n[手动素材] %s\\%s\n 来源:用户投放(待补充出处)\n" % (os.path.basename(d), n))
77
+ listed.add(key)
78
+ changed = True
79
+ if changed:
80
+ with open(MARK, "w", encoding="utf-8") as f:
81
+ json.dump(sorted(listed), f, ensure_ascii=False)
code/siai_all.py ADDED
@@ -0,0 +1,810 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+ """Easy4K SFDP Siamese NET AI pipeline (compact)."""
3
+ import argparse, csv, hashlib, http.cookiejar, json, math, os, random, re, subprocess, sys, time
4
+ import urllib.error, urllib.parse, urllib.request
5
+ from functools import lru_cache
6
+
7
+ import cv2, numpy as np, torch, torch.nn as nn
8
+ from torch.utils.data import DataLoader, Dataset, Sampler
9
+
10
+ DATA = r"D:\sfdp-ai\siamese\data"
11
+ TILES = os.environ.get("SFDP_TILES") or os.path.join(DATA, "tiles")
12
+ LABELS = os.path.join(DATA, "labels")
13
+ TEACHER = r"D:\sfdp-ai\bin\SfdpTeacher.exe"
14
+ RAW = os.path.join(DATA, "raw")
15
+ BILI_RAW = os.path.join(DATA, "raw_bili")
16
+ TEST_TILES = os.path.join(DATA, "tiles_test")
17
+ TEST_LABELS = os.path.join(DATA, "labels_test")
18
+
19
+
20
+ class DSConv(nn.Module):
21
+ def __init__(self, cin, cout, stride=1):
22
+ super().__init__()
23
+ self.dw = nn.Conv2d(cin, cin, 3, stride, 1, groups=cin, bias=False)
24
+ self.bn1 = nn.BatchNorm2d(cin)
25
+ self.pw = nn.Conv2d(cin, cout, 1, bias=False)
26
+ self.bn2 = nn.BatchNorm2d(cout)
27
+ self.act = nn.ReLU6(inplace=True)
28
+
29
+ def forward(self, x):
30
+ x = self.act(self.bn1(self.dw(x)))
31
+ return self.act(self.bn2(self.pw(x)))
32
+
33
+
34
+ class Encoder(nn.Module):
35
+ def __init__(self, emb=512):
36
+ super().__init__()
37
+ self.stem = nn.Sequential(nn.Conv2d(1, 32, 3, 2, 1, bias=False),
38
+ nn.BatchNorm2d(32), nn.ReLU6(inplace=True))
39
+ self.blocks = nn.Sequential(
40
+ DSConv(32, 48, 2), DSConv(48, 48), DSConv(48, 96, 2), DSConv(96, 96),
41
+ DSConv(96, 160, 2), DSConv(160, 160), DSConv(160, 224, 2), DSConv(224, 224),
42
+ DSConv(224, 320, 2), DSConv(320, 320))
43
+ self.proj = nn.Conv2d(320, emb, 1, bias=False)
44
+ self.bn = nn.BatchNorm2d(emb)
45
+ self.act = nn.ReLU6(inplace=True)
46
+
47
+ def forward(self, x):
48
+ x = self.act(self.bn(self.proj(self.blocks(self.stem(x)))))
49
+ return x.mean(dim=(2, 3))
50
+
51
+
52
+ class SiameseSFDP(nn.Module):
53
+ def __init__(self, emb=512, hidden=1024):
54
+ super().__init__()
55
+ self.encoder = Encoder(emb)
56
+ self.head = nn.Sequential(nn.Linear(emb * 4, hidden), nn.ReLU(inplace=True),
57
+ nn.Dropout(0.1), nn.Linear(hidden, 1), nn.Sigmoid())
58
+
59
+ def forward(self, a, b):
60
+ ea = self.encoder(a); eb = self.encoder(b)
61
+ return self.head(torch.cat([ea, eb, (ea - eb).abs(), ea * eb], dim=1))
62
+
63
+
64
+ def count_params(m):
65
+ return sum(p.numel() for p in m.parameters())
66
+
67
+
68
+ @lru_cache(maxsize=4096)
69
+ def load_gray(p):
70
+ img = cv2.imread(p, cv2.IMREAD_GRAYSCALE)
71
+ if img is None:
72
+ raise FileNotFoundError(p)
73
+ return img
74
+
75
+
76
+ _GRAY_CACHE = {}
77
+
78
+
79
+ def load_gray_cached(path):
80
+ """同一切片会被同片段的多个样本反复读取:按 worker 进程做一个小 LRU,
81
+ 省掉重复 PNG 解码(320x320 约 100KB,512 条 ≈ 50MB/worker)。"""
82
+ a = _GRAY_CACHE.get(path)
83
+ if a is not None:
84
+ return a
85
+ try:
86
+ a = load_gray(path)
87
+ except FileNotFoundError:
88
+ # 切片可能正被搬运/重写:短暂重试,实在没有就返回黑图,绝不让整轮训练崩掉
89
+ a = None
90
+ for _ in range(3):
91
+ time.sleep(0.2)
92
+ try:
93
+ a = load_gray(path)
94
+ break
95
+ except FileNotFoundError:
96
+ continue
97
+ if a is None:
98
+ a = np.zeros((320, 320), np.uint8)
99
+ if len(_GRAY_CACHE) >= 512:
100
+ _GRAY_CACHE.clear()
101
+ _GRAY_CACHE[path] = a
102
+ return a
103
+
104
+
105
+ FALLBACK_TILES = os.path.join(DATA, "tiles")
106
+
107
+
108
+ def _resolve_tile(root, rel):
109
+ """Hot tiles (SSD) may still be copying: fall back to the original tiles dir."""
110
+ p = os.path.join(root, rel)
111
+ if os.path.exists(p):
112
+ return p
113
+ fb = os.path.join(FALLBACK_TILES, rel)
114
+ return fb if os.path.exists(fb) else p
115
+
116
+
117
+ class PairDataset(Dataset):
118
+ def __init__(self, manifest, tiles_root, crop=128, augment=False):
119
+ self.items = [json.loads(l) for l in open(manifest, "r", encoding="utf-8") if l.strip()]
120
+ self.tiles_root = tiles_root; self.crop = crop; self.augment = augment
121
+
122
+ def __len__(self):
123
+ return len(self.items)
124
+
125
+ def __getitem__(self, i):
126
+ it = self.items[i]
127
+ a = load_gray_cached(_resolve_tile(self.tiles_root, it["a"]))
128
+ b = load_gray_cached(_resolve_tile(self.tiles_root, it["b"]))
129
+ c = self.crop; h, w = a.shape
130
+ x = min(max(0, it["x"]), w - c); y = min(max(0, it["y"]), h - c)
131
+ pa, pb = a[y:y + c, x:x + c], b[y:y + c, x:x + c]
132
+ if self.augment:
133
+ if np.random.rand() < 0.5:
134
+ pa, pb = pa[:, ::-1], pb[:, ::-1]
135
+ if np.random.rand() < 0.3:
136
+ g = 0.8 + np.random.rand() * 0.45
137
+ pa = np.clip((pa / 255.0) ** g * 255.0, 0, 255).astype(np.uint8)
138
+ pb = np.clip((pb / 255.0) ** g * 255.0, 0, 255).astype(np.uint8)
139
+ ta = torch.from_numpy(np.ascontiguousarray(pa)).float().div_(255.0).unsqueeze(0)
140
+ tb = torch.from_numpy(np.ascontiguousarray(pb)).float().div_(255.0).unsqueeze(0)
141
+ return (ta, tb, torch.tensor([it["soft"]], dtype=torch.float32),
142
+ torch.tensor([it["hard"]], dtype=torch.float32))
143
+
144
+
145
+ # ---------------- bilibili ----------------
146
+ BUA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
147
+ "(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
148
+ BILI_META = os.path.join(BILI_RAW, "sources_bili.json")
149
+ _jar = http.cookiejar.CookieJar()
150
+ _op = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(_jar))
151
+ _op.addheaders = [("User-Agent", BUA), ("Referer", "https://www.bilibili.com/")]
152
+
153
+
154
+ def bjson(url, params=None):
155
+ full = url + ("?" + urllib.parse.urlencode(params) if params else "")
156
+ last = None
157
+ for i in range(3):
158
+ try:
159
+ with _op.open(full, timeout=25) as r:
160
+ return json.loads(r.read().decode("utf-8"))
161
+ except urllib.error.HTTPError as e:
162
+ if e.code == 412:
163
+ raise
164
+ last = e; time.sleep(2 + i * 2)
165
+ except Exception as e:
166
+ last = e; time.sleep(2 + i * 2)
167
+ raise last
168
+
169
+
170
+ def bview(bvid):
171
+ d = bjson("https://api.bilibili.com/x/web-interface/view", {"bvid": bvid})
172
+ return (d.get("data") or None) if d.get("code") == 0 else None
173
+
174
+
175
+ def bplay(bvid, cid, qn):
176
+ d = bjson("https://api.bilibili.com/x/player/playurl",
177
+ {"bvid": bvid, "cid": cid, "qn": qn, "fnval": 0, "fourk": 0})
178
+ if d.get("code") != 0:
179
+ return None
180
+ dd = d.get("data") or {}; durl = dd.get("durl") or []
181
+ return {"quality": dd.get("quality"), "url": durl[0].get("url")} if durl else None
182
+
183
+
184
+ BILI_MAX_BPS = 6_000_000 # 下载限速 ~48 Mbps(千兆宽带留给家里其他人)
185
+
186
+
187
+ def bdownload(url, dst):
188
+ tmp = dst + ".part"
189
+ req = urllib.request.Request(url, headers={"User-Agent": BUA, "Referer": "https://www.bilibili.com/"})
190
+ for _ in range(3):
191
+ try:
192
+ with urllib.request.urlopen(req, timeout=30) as r, open(tmp, "wb") as f:
193
+ t0 = time.time()
194
+ got = 0
195
+ while True:
196
+ ch = r.read(262144)
197
+ if not ch:
198
+ break
199
+ f.write(ch)
200
+ got += len(ch)
201
+ # 令牌桶式限速:按已下载字节应花的时间补 sleep
202
+ expected = got / BILI_MAX_BPS
203
+ elapsed = time.time() - t0
204
+ if expected > elapsed:
205
+ time.sleep(expected - elapsed)
206
+ os.replace(tmp, dst); return True
207
+ except Exception:
208
+ if os.path.exists(tmp):
209
+ os.remove(tmp)
210
+ time.sleep(3)
211
+ return False
212
+
213
+
214
+ def cmd_rank(args):
215
+ root = getattr(args, "outdir", None) or BILI_RAW
216
+ meta_path = os.path.join(root, "sources_bili.json")
217
+ os.makedirs(root, exist_ok=True)
218
+ try:
219
+ _op.open("https://www.bilibili.com/", timeout=20).read(2048)
220
+ except Exception:
221
+ pass
222
+ meta = json.load(open(meta_path, "r", encoding="utf-8")) if os.path.exists(meta_path) else []
223
+ done = {m["bvid"] for m in meta}
224
+ seen, cands = set(), []
225
+
226
+ def add(items):
227
+ for it in items:
228
+ b = it.get("bvid"); dur = int(it.get("duration") or 0); title = it.get("title", "")
229
+ if not b or b in seen or b in done or not (args.min_sec <= dur <= args.max_sec):
230
+ continue
231
+ if any(x in title for x in ("直播", "回放", "合集", "教程", "reaction", "解说")):
232
+ continue
233
+ seen.add(b)
234
+ cands.append({"bvid": b, "title": title, "author": (it.get("owner") or {}).get("name", ""),
235
+ "duration": dur})
236
+
237
+ for rid in (3, 129, 1, 181, 71, 160, 4):
238
+ try:
239
+ d = bjson("https://api.bilibili.com/x/web-interface/ranking/v2", {"rid": rid})
240
+ add((d.get("data") or {}).get("list") or [])
241
+ except Exception as e:
242
+ print("RANK_FAIL", rid, e, flush=True)
243
+ time.sleep(1.2)
244
+ for pn in range(1, args.pages + 1):
245
+ try:
246
+ d = bjson("https://api.bilibili.com/x/web-interface/popular", {"ps": 20, "pn": pn})
247
+ add((d.get("data") or {}).get("list") or [])
248
+ except Exception as e:
249
+ print("POPULAR_FAIL", pn, e, flush=True)
250
+ time.sleep(1.2)
251
+ for rid in (3, 129, 1, 181):
252
+ for pn in range(1, 6):
253
+ try:
254
+ d = bjson("https://api.bilibili.com/x/web-interface/dynamic/region",
255
+ {"rid": rid, "ps": 20, "pn": pn})
256
+ add((d.get("data") or {}).get("archives") or [])
257
+ except Exception as e:
258
+ print("REGION_FAIL", rid, pn, e, flush=True); break
259
+ time.sleep(1.0)
260
+ existing = list(done); random.shuffle(existing)
261
+ seeds = ([c["bvid"] for c in cands] + existing[:args.seed_existing])[:args.seeds + args.seed_existing]
262
+ for b in seeds:
263
+ try:
264
+ d = bjson("https://api.bilibili.com/x/web-interface/archive/related", {"bvid": b})
265
+ add(d.get("data") or [])
266
+ except Exception as e:
267
+ print("RELATED_FAIL", b, e, flush=True)
268
+ time.sleep(0.5)
269
+ print("CANDIDATES", len(cands), flush=True)
270
+ saved = 0
271
+ for it in cands:
272
+ if saved >= args.target:
273
+ break
274
+ v = bview(it["bvid"])
275
+ if not v:
276
+ continue
277
+ pu = None
278
+ for qn in (80, 64, 32):
279
+ pu = bplay(it["bvid"], v["cid"], qn)
280
+ if pu:
281
+ break
282
+ if not pu or not pu.get("url"):
283
+ continue
284
+ name = re.sub(r"[^0-9A-Za-z._-]+", "_", it["title"])[:60] or it["bvid"]
285
+ ext = ".flv" if ".flv" in pu["url"] else ".mp4"
286
+ fname = f"{it['bvid']}_{name}{ext}"
287
+ dst = os.path.join(root, fname)
288
+ if os.path.exists(dst):
289
+ continue
290
+ if not bdownload(pu["url"], dst):
291
+ print("DL_FAIL", it["bvid"], flush=True); continue
292
+ meta.append({**it, "file": fname, "cid": v["cid"], "qn": pu["quality"],
293
+ "license": "research use only, not redistributed"})
294
+ saved += 1
295
+ print("OK", fname, flush=True)
296
+ json.dump(meta, open(meta_path, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
297
+ print("TOTAL_FILES", len(meta), flush=True)
298
+
299
+
300
+ # ---------------- tiles ----------------
301
+ PATTERNS = {"1-1": [1], "2-2": [2], "3-3": [3], "mix": [2, 3, 2, 1, 3, 2]}
302
+ VEXT = (".webm", ".ogg", ".ogv", ".mp4", ".mov", ".mkv", ".flv")
303
+
304
+
305
+ def clip_id(stem, seg, pat):
306
+ s = re.sub(r"[^0-9A-Za-z._-]+", "_", stem).strip("_")[:40]
307
+ if any(ord(c) > 127 for c in stem) or len(s) < 4:
308
+ s = (s + "_" if s else "mv_") + hashlib.sha1(stem.encode("utf-8")).hexdigest()[:8]
309
+ return f"{s}_s{seg:02d}_{pat}"
310
+
311
+
312
+ def pulldown(frames, rounds, n):
313
+ out, i = [], 0
314
+ while len(out) < n:
315
+ out.extend([frames[i % len(frames)]] * rounds[i % len(rounds)]); i += 1
316
+ return out[:n]
317
+
318
+
319
+ def cmd_tiles(args):
320
+ rng = random.Random(args.seed); os.makedirs(args.out, exist_ok=True)
321
+ idx_path = os.path.join(args.out, "index.jsonl")
322
+ done = set()
323
+ if os.path.exists(idx_path):
324
+ for line in open(idx_path, "r", encoding="utf-8"):
325
+ try:
326
+ done.add(json.loads(line)["clip"])
327
+ except Exception:
328
+ pass
329
+ videos = [n for n in sorted(os.listdir(args.raw)) if os.path.splitext(n)[1].lower() in VEXT]
330
+ print("VIDEOS", len(videos), "ALREADY", len(done), flush=True)
331
+ total = 0
332
+ with open(idx_path, "a", encoding="utf-8") as idx:
333
+ for vi, name in enumerate(videos):
334
+ cap = cv2.VideoCapture(os.path.join(args.raw, name))
335
+ if not cap.isOpened():
336
+ print("OPEN_FAIL", name, flush=True); continue
337
+ nf = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
338
+ fps = float(cap.get(cv2.CAP_PROP_FPS) or 0)
339
+ w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH) or 0)
340
+ h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT) or 0)
341
+ if nf < args.frames * 3 or w < max(args.tile, args.min_width) or h < args.tile:
342
+ print("SKIP_DIM", name, w, h, nf, flush=True); cap.release(); continue
343
+ max_start = max(0, nf - args.frames * 4)
344
+ for si in range(args.segments):
345
+ start = rng.randint(0, max_start)
346
+ x = rng.randint(0, w - args.tile); y = rng.randint(0, h - args.tile)
347
+ cap.set(cv2.CAP_PROP_POS_FRAMES, start)
348
+ frames = []
349
+ for _ in range(max(4, args.frames // 2)):
350
+ ok, fr = cap.read()
351
+ if not ok:
352
+ break
353
+ frames.append(cv2.cvtColor(fr[y:y + args.tile, x:x + args.tile], cv2.COLOR_BGR2GRAY))
354
+ if len(frames) < 4:
355
+ continue
356
+ for pat, rounds in PATTERNS.items():
357
+ clip = clip_id(os.path.splitext(name)[0], si, pat)
358
+ if clip in done:
359
+ continue
360
+ cdir = os.path.join(args.out, clip); os.makedirs(cdir, exist_ok=True)
361
+ for k, img in enumerate(pulldown(frames, rounds, args.frames)):
362
+ cv2.imwrite(os.path.join(cdir, f"f{k:04d}.png"), img)
363
+ idx.write(json.dumps({"clip": clip, "src": name, "pattern": pat,
364
+ "start_frame": start, "x": x, "y": y,
365
+ "tile": args.tile, "frames": args.frames, "fps": fps},
366
+ ensure_ascii=False) + "\n")
367
+ idx.flush(); total += 1
368
+ cap.release()
369
+ if (vi + 1) % 10 == 0:
370
+ print("PROGRESS", vi + 1, "/", len(videos), "clips", total, flush=True)
371
+ print("TOTAL_CLIPS", total, flush=True)
372
+
373
+
374
+ def cmd_teacher(args):
375
+ os.makedirs(args.labels, exist_ok=True)
376
+ print("SCAN_START", args.tiles, flush=True)
377
+ dirs = [d for d in sorted(os.listdir(args.tiles)) if os.path.isdir(os.path.join(args.tiles, d))]
378
+ print("SCAN_DIRS", len(dirs), flush=True)
379
+ todo = []
380
+ for i, d in enumerate(dirs):
381
+ if not os.path.exists(os.path.join(args.labels, d + ".csv")):
382
+ todo.append(d)
383
+ if (i + 1) % 2000 == 0:
384
+ print("SCAN", i + 1, "/", len(dirs), flush=True)
385
+ print("DIRS", len(dirs), "TODO", len(todo), flush=True)
386
+ for i, d in enumerate(todo):
387
+ subprocess.run([TEACHER, os.path.join(args.tiles, d), os.path.join(args.labels, d + ".csv"),
388
+ "--mode", "uhd", "--crop", "128", "--per-pair", "4", "--limit", "0"],
389
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
390
+ if (i + 1) % 10 == 0:
391
+ print("PROGRESS", i + 1, "/", len(todo), flush=True)
392
+ print("DONE", flush=True)
393
+
394
+
395
+ # ---------------- label / train / eval / pack ----------------
396
+ def soft_from_flow(flow):
397
+ return max(0.0, min(1.0, 1.0 - flow / 2.0))
398
+
399
+
400
+ def cmd_label(args):
401
+ rng = random.Random(args.seed)
402
+ clip_src = {}
403
+ p = os.path.join(TILES, "index.jsonl")
404
+ if os.path.exists(p):
405
+ for line in open(p, "r", encoding="utf-8"):
406
+ try:
407
+ r = json.loads(line); clip_src[r["clip"]] = r["src"]
408
+ except Exception:
409
+ pass
410
+ rows = []
411
+ for name in sorted(os.listdir(LABELS)):
412
+ if not name.endswith(".csv"):
413
+ continue
414
+ clip = name[:-4]; src = clip_src.get(clip, "unknown")
415
+ with open(os.path.join(LABELS, name), "r", encoding="utf-8", newline="") as f:
416
+ for r in csv.DictReader(f):
417
+ try:
418
+ mad, flow, var = float(r["mad"]), float(r["flow"]), float(r["var"])
419
+ stage = r.get("stage", "") or ""
420
+ dup = int(float(r.get("dup", 0) or 0))
421
+ x, y = int(float(r["x"])), int(float(r["y"]))
422
+ except (KeyError, ValueError, TypeError):
423
+ continue # 被中断写坏的 CSV 行(缺列/None)直接跳过
424
+ if dup == 1:
425
+ soft = 1.0 if (flow == 0.0 and var == 0.0) else soft_from_flow(flow)
426
+ elif stage == "Flow":
427
+ if 0.1 <= flow <= 0.4:
428
+ continue
429
+ soft = soft_from_flow(flow)
430
+ else:
431
+ soft = 0.0
432
+ rows.append({"a": os.path.join(clip, r["frame_a"]), "b": os.path.join(clip, r["frame_b"]),
433
+ "x": x, "y": y, "crop": 128,
434
+ "soft": round(soft, 4), "hard": 1 if (stage == "Flow" and mad < 1.0) else 0,
435
+ "stage": stage, "src": src, "clip": clip})
436
+ pos = [r for r in rows if r["soft"] >= 0.9]
437
+ neg = [r for r in rows if r["soft"] <= 0.1]
438
+ mid = [r for r in rows if 0.1 < r["soft"] < 0.9]
439
+ rng.shuffle(pos); rng.shuffle(neg); rng.shuffle(mid)
440
+ neg.sort(key=lambda r: -r["hard"])
441
+ m = int(len(pos) * args.max_neg_ratio)
442
+ neg_keep = neg[:m] if len(neg) > m else neg
443
+ if neg_keep and len(pos) > len(neg_keep):
444
+ pos = pos[:len(neg_keep)]
445
+ mid_keep = mid[:int((len(pos) + len(neg_keep)) * args.mid_ratio)]
446
+ data = pos + neg_keep + mid_keep; rng.shuffle(data)
447
+ val_f = lambda r: sum(ord(c) for c in r["src"]) % 7 == 0
448
+ train = [r for r in data if not val_f(r)]; val = [r for r in data if val_f(r)]
449
+ for path, items in ((os.path.join(DATA, "train.jsonl"), train), (os.path.join(DATA, "val.jsonl"), val)):
450
+ with open(path, "w", encoding="utf-8") as f:
451
+ for it in items:
452
+ f.write(json.dumps(it, ensure_ascii=False) + "\n")
453
+ st = {"raw": len(rows), "kept_pos": len(pos), "kept_neg": len(neg_keep),
454
+ "kept_mid": len(mid_keep), "hard_mined": sum(1 for r in data if r["hard"]),
455
+ "train": len(train), "val": len(val)}
456
+ json.dump(st, open(os.path.join(DATA, "label_stats.json"), "w", encoding="utf-8"),
457
+ ensure_ascii=False, indent=2)
458
+ print(json.dumps(st, ensure_ascii=False), flush=True)
459
+
460
+
461
+ def eval_model(model, loader, device):
462
+ model.eval(); se = n = correct = pe = pn = ne = nn_ = 0
463
+ with torch.no_grad():
464
+ for a, b, y, _ in loader:
465
+ a, b, y = a.to(device), b.to(device), y.to(device)
466
+ p = model(a, b)
467
+ se += torch.abs(p - y).sum().item(); n += y.numel()
468
+ correct += ((p >= 0.5) == (y >= 0.5)).sum().item()
469
+ mm = y >= 0.9
470
+ if mm.any():
471
+ pe += torch.abs(p[mm] - y[mm]).sum().item(); pn += int(mm.sum())
472
+ mm = y <= 0.1
473
+ if mm.any():
474
+ ne += torch.abs(p[mm] - y[mm]).sum().item(); nn_ += int(mm.sum())
475
+ return {"mae": se / max(1, n), "acc": correct / max(1, n),
476
+ "pos_mae": pe / max(1, pn), "neg_mae": ne / max(1, nn_), "n": n}
477
+
478
+
479
+ class ClipGroupSampler(Sampler):
480
+ """按片段分组取样:同一 clip 的���片连续读,机械盘上把随机读变成顺序读。
481
+ 片段之间每个 epoch 重新打乱、片段内部也打乱,样本分布不变。"""
482
+
483
+ def __init__(self, dataset, shuffle=True, seed=1234):
484
+ self.groups = {}
485
+ for i, it in enumerate(dataset.items):
486
+ self.groups.setdefault(it["clip"], []).append(i)
487
+ self.clips = sorted(self.groups.keys())
488
+ self.shuffle = shuffle
489
+ self.seed = seed
490
+ self.epoch = 0
491
+ self.n = len(dataset)
492
+
493
+ def set_epoch(self, e):
494
+ self.epoch = e
495
+
496
+ def __len__(self):
497
+ return self.n
498
+
499
+ def __iter__(self):
500
+ rng = random.Random(self.seed + self.epoch)
501
+ clips = self.clips[:]
502
+ if self.shuffle:
503
+ rng.shuffle(clips)
504
+ for c in clips:
505
+ idx = self.groups[c]
506
+ if self.shuffle:
507
+ idx = idx[:]
508
+ rng.shuffle(idx)
509
+ for i in idx:
510
+ yield i
511
+
512
+
513
+ def _worker_init(_):
514
+ # 每个 DataLoader worker 只用 1 个 torch 线程:默认多线程在 8 个 worker 里会互相抢核,
515
+ # 实测反而拖慢取数(GPU 在等 batch)。解码/切片本身是单线程活。
516
+ try:
517
+ torch.set_num_threads(1)
518
+ except Exception:
519
+ pass
520
+
521
+
522
+ def cmd_train(args):
523
+ random.seed(args.seed); np.random.seed(args.seed)
524
+ torch.manual_seed(args.seed); torch.cuda.manual_seed_all(args.seed)
525
+ torch.backends.cudnn.benchmark = True
526
+ os.makedirs(args.out, exist_ok=True)
527
+ dev = "cuda" if torch.cuda.is_available() else "cpu"
528
+ print("DEVICE", dev, torch.cuda.get_device_name(0) if dev == "cuda" else "", flush=True)
529
+ tr = PairDataset(os.path.join(DATA, "train.jsonl"), TILES, augment=bool(args.augment))
530
+ va = PairDataset(os.path.join(DATA, "val.jsonl"), TILES)
531
+ print("SAMPLES", len(tr), len(va), flush=True)
532
+ tl = DataLoader(tr, batch_size=args.batch, sampler=ClipGroupSampler(tr, shuffle=True),
533
+ num_workers=args.workers, worker_init_fn=_worker_init, prefetch_factor=2,
534
+ pin_memory=(dev == "cuda"), drop_last=True, persistent_workers=args.workers > 0)
535
+ vl = DataLoader(va, batch_size=args.batch, sampler=ClipGroupSampler(va, shuffle=False),
536
+ num_workers=max(2, args.workers // 2), worker_init_fn=_worker_init,
537
+ pin_memory=(dev == "cuda"))
538
+ model = SiameseSFDP().to(dev); print("PARAMS", count_params(model), flush=True)
539
+ opt = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=1e-4)
540
+ steps = max(1, len(tl)); total = args.epochs * steps
541
+
542
+ def lr_at(step):
543
+ warm = args.warmup * steps
544
+ if step < warm:
545
+ return (step + 1) / max(1, warm)
546
+ return 0.5 * (1 + math.cos(math.pi * (step - warm) / max(1, total - warm)))
547
+
548
+ sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_at)
549
+ best = 1e9
550
+ if args.resume and os.path.exists(args.resume):
551
+ ck = torch.load(args.resume, map_location=dev, weights_only=True)
552
+ model.load_state_dict(ck["model"]); opt.load_state_dict(ck["opt"])
553
+ best = ck.get("best", best)
554
+ print("RESUME", args.resume, "prev", ck.get("epoch"), flush=True)
555
+ log = os.path.join(args.out, "train_log.csv")
556
+ if not os.path.exists(log):
557
+ with open(log, "w", newline="", encoding="utf-8") as f:
558
+ csv.writer(f).writerow(["epoch", "lr", "loss", "val_mae", "val_acc", "pos_mae", "neg_mae", "sec"])
559
+ for ep in range(args.epochs):
560
+ if hasattr(tl.sampler, "set_epoch"):
561
+ tl.sampler.set_epoch(ep)
562
+ model.train(); t0 = time.time(); run = cnt = 0
563
+ se = 0
564
+ step_i = 0
565
+ for a, b, y, hard in tl:
566
+ a, b, y, hard = a.to(dev), b.to(dev), y.to(dev), hard.to(dev)
567
+ p = model(a, b); w = 1.0 + (args.hard_weight - 1.0) * hard
568
+ loss = (w * (p - y) ** 2).mean()
569
+ opt.zero_grad(set_to_none=True); loss.backward()
570
+ nn.utils.clip_grad_norm_(model.parameters(), 5.0)
571
+ opt.step(); sched.step()
572
+ run += loss.item() * a.size(0); cnt += a.size(0)
573
+ se += torch.abs(p - y).sum().item()
574
+ step_i += 1
575
+ if step_i % 50 == 0:
576
+ print(json.dumps({"step": step_i, "steps": len(tl), "ep": ep,
577
+ "loss": round(run / max(1, cnt), 5),
578
+ "pmae": round(se / max(1, cnt), 5),
579
+ "elapsed": round(time.time() - t0, 1)}), flush=True)
580
+ m = eval_model(model, vl, dev); sec = time.time() - t0; lr = opt.param_groups[0]["lr"]
581
+ print(json.dumps({"epoch": ep, "loss": round(run / max(1, cnt), 5), "lr": round(lr, 6),
582
+ **{k: round(v, 5) for k, v in m.items()}, "sec": round(sec, 1)}), flush=True)
583
+ with open(log, "a", newline="", encoding="utf-8") as f:
584
+ csv.writer(f).writerow([ep, round(lr, 6), round(run / max(1, cnt), 5), round(m["mae"], 5),
585
+ round(m["acc"], 5), round(m["pos_mae"], 5), round(m["neg_mae"], 5),
586
+ round(sec, 1)])
587
+ ck = {"model": model.state_dict(), "opt": opt.state_dict(), "epoch": ep, "best": best}
588
+ torch.save(ck, os.path.join(args.out, "last.pt"))
589
+ if m["mae"] < best:
590
+ best = m["mae"]; ck["best"] = best
591
+ torch.save(ck, os.path.join(args.out, "best.pt"))
592
+ print("BEST", round(best, 5), "epoch", ep, flush=True)
593
+ print("TRAIN_DONE best_mae", round(best, 5), flush=True)
594
+ ck_best = os.path.join(args.out, "best.pt")
595
+ if not os.path.exists(ck_best):
596
+ ck_best = os.path.join(args.out, "last.pt") # 本轮没刷新最优时用 last.pt 导出,避免导出报 FileNotFoundError
597
+ print("EXPORT_SRC", ck_best, flush=True)
598
+ export_onnx(ck_best, os.path.join(args.out, "easy4k_sfdp_siamese_fp32.onnx"))
599
+
600
+
601
+ def export_onnx(ckpt, out, opset=17):
602
+ import onnx
603
+ ck = torch.load(ckpt, map_location="cpu", weights_only=True)
604
+ m = SiameseSFDP(); m.load_state_dict(ck["model"]); m.eval()
605
+ a = torch.rand(2, 1, 128, 128); b = torch.rand(2, 1, 128, 128)
606
+ torch.onnx.export(m, (a, b), out, input_names=["frame_a", "frame_b"], output_names=["score"],
607
+ dynamic_axes={"frame_a": {0: "batch"}, "frame_b": {0: "batch"}, "score": {0: "batch"}},
608
+ opset_version=opset, do_constant_folding=True)
609
+ o = onnx.load(out); onnx.checker.check_model(o)
610
+ meta = {"model_name": "Easy4K SFDP - Siamese NET AI", "model_version": "0.6.0",
611
+ "task": "frame-duplicate similarity regression",
612
+ "input": "two grayscale 128x128 patches, float32, [0,1]",
613
+ "output": "score in [0,1]; 1 = duplicate/same picture",
614
+ "training": "knowledge distillation from Easy4K SFDP four-stage judge",
615
+ "runtime": "onnxruntime DirectML (NVIDIA / AMD / Intel) or CPU",
616
+ "license": "cc-by-nc-4.0", "params": str(count_params(m))}
617
+ have = {p.key for p in o.metadata_props}
618
+ for k, v in meta.items():
619
+ if k not in have:
620
+ p = o.metadata_props.add(); p.key, p.value = k, v
621
+ onnx.save(o, out)
622
+ print("EXPORTED", out, "size_mb", round(os.path.getsize(out) / 1048576, 2), flush=True)
623
+
624
+ # FP16 版:内部权重/算子转半精度,输入输出仍保持 float32,调用方不用改代码
625
+ try:
626
+ from onnxconverter_common import float16 as _f16
627
+ o16 = _f16.convert_float_to_float16(onnx.load(out), keep_io_types=True)
628
+ p16 = out.replace("_fp32.onnx", "_fp16.onnx") if out.endswith("_fp32.onnx") else (out[:-5] + "_fp16.onnx")
629
+ onnx.save(o16, p16)
630
+ print("EXPORTED_FP16", p16, "size_mb", round(os.path.getsize(p16) / 1048576, 2), flush=True)
631
+ except Exception as e:
632
+ print("EXPORTED_FP16_SKIP", repr(e), flush=True)
633
+
634
+
635
+ def cmd_eval(args):
636
+ ds = PairDataset(os.path.join(DATA, "val.jsonl"), TILES)
637
+ dl = DataLoader(ds, batch_size=512, shuffle=False, num_workers=6)
638
+ ck = torch.load(args.ckpt, map_location="cpu", weights_only=True)
639
+ net = SiameseSFDP(); net.load_state_dict(ck["model"]); net.eval()
640
+ S, Y, ST = [], [], []
641
+ with torch.no_grad():
642
+ off = 0
643
+ for a, b, y, _ in dl:
644
+ p = net(a, b).squeeze(1).numpy(); S.append(p); Y.append(y.squeeze(1).numpy())
645
+ ST.extend([ds.items[off + i]["stage"] for i in range(len(p))]); off += len(p)
646
+ s = np.concatenate(S); y = np.concatenate(Y); t = (y >= 0.5).astype(int); p = (s >= 0.5).astype(int)
647
+ tp = int(((p == 1) & (t == 1)).sum()); fp = int(((p == 1) & (t == 0)).sum())
648
+ fn = int(((p == 0) & (t == 1)).sum()); tn = int(((p == 0) & (t == 0)).sum())
649
+ print(json.dumps({"n": len(s), "acc": round((tp + tn) / len(s), 5),
650
+ "precision": round(tp / max(1, tp + fp), 5),
651
+ "recall": round(tp / max(1, tp + fn), 5)}), flush=True)
652
+ by = {}
653
+ for st in sorted(set(ST)):
654
+ mm = np.array([x == st for x in ST])
655
+ by[st] = {"n": int(mm.sum()), "mean_score": round(float(s[mm].mean()), 4),
656
+ "mean_target": round(float(y[mm].mean()), 4)}
657
+ print("BY_STAGE", json.dumps(by, ensure_ascii=False), flush=True)
658
+ for th in (0.4, 0.5, 0.6, 0.7, 0.8):
659
+ pp = (s >= th).astype(int)
660
+ tp_ = int(((pp == 1) & (t == 1)).sum()); fp_ = int(((pp == 1) & (t == 0)).sum())
661
+ fn_ = int(((pp == 0) & (t == 1)).sum())
662
+ pr = tp_ / max(1, tp_ + fp_); rc = tp_ / max(1, tp_ + fn_)
663
+ print(json.dumps({"t": th, "acc": round(float((pp == t).mean()), 5), "precision": round(pr, 5),
664
+ "recall": round(rc, 5), "f1": round(2 * pr * rc / max(1e-9, pr + rc), 5)}), flush=True)
665
+
666
+
667
+ def cmd_compare(args):
668
+ items = []
669
+ for name in sorted(os.listdir(TEST_LABELS)):
670
+ if not name.endswith(".csv"):
671
+ continue
672
+ clip = name[:-4]
673
+ for r in csv.DictReader(open(os.path.join(TEST_LABELS, name), "r", encoding="utf-8", newline="")):
674
+ flow, var = float(r["flow"]), float(r["var"]); dup = int(float(r.get("dup", 0) or 0))
675
+ stage = r.get("stage", "")
676
+ soft = (1.0 if (flow == 0.0 and var == 0.0) else soft_from_flow(flow)) if dup == 1 else \
677
+ (soft_from_flow(flow) if stage == "Flow" else 0.0)
678
+ items.append((os.path.join(clip, r["frame_a"]), os.path.join(clip, r["frame_b"]),
679
+ int(float(r["x"])), int(float(r["y"])), soft))
680
+ print("TEST_ITEMS", len(items), flush=True)
681
+ A, B = [], []
682
+ for pa, pb, x, y, _ in items:
683
+ a = load_gray(os.path.join(TEST_TILES, pa)); b = load_gray(os.path.join(TEST_TILES, pb))
684
+ h, w = a.shape; x = min(max(0, x), w - 128); y = min(max(0, y), h - 128)
685
+ A.append(torch.from_numpy(np.ascontiguousarray(a[y:y + 128, x:x + 128])).float().div_(255.0).unsqueeze(0))
686
+ B.append(torch.from_numpy(np.ascontiguousarray(b[y:y + 128, x:x + 128])).float().div_(255.0).unsqueeze(0))
687
+ A = torch.stack(A); B = torch.stack(B)
688
+ y = torch.tensor([it[4] for it in items], dtype=torch.float32).unsqueeze(1)
689
+ t = (y >= 0.5).int().numpy()
690
+ for path in args.ckpts.split(","):
691
+ path = path.strip()
692
+ if not os.path.exists(path):
693
+ print("MISSING", path, flush=True); continue
694
+ ck = torch.load(path, map_location="cpu", weights_only=True)
695
+ net = SiameseSFDP(); net.load_state_dict(ck["model"]); net.eval()
696
+ outs = []
697
+ with torch.no_grad():
698
+ for i in range(0, len(A), 512):
699
+ outs.append(net(A[i:i + 512], B[i:i + 512]).numpy())
700
+ s = np.concatenate(outs)
701
+ rec = {"ckpt": os.path.basename(os.path.dirname(path)), "epoch": ck.get("epoch"),
702
+ "mae": round(float(np.abs(s - y.numpy()).mean()), 5)}
703
+ for th in (0.5, 0.6, 0.7):
704
+ pp = (s >= th).astype(int)
705
+ tp = int(((pp == 1) & (t == 1)).sum()); fp = int(((pp == 1) & (t == 0)).sum())
706
+ fn = int(((pp == 0) & (t == 1)).sum()); tn = int(((pp == 0) & (t == 0)).sum())
707
+ rec[f"acc@{th}"] = round((tp + tn) / len(s), 4)
708
+ print(rec, flush=True)
709
+
710
+
711
+ def cmd_pack(args):
712
+ import shutil
713
+ card = open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "card.md"), "r", encoding="utf-8").read()
714
+ os.makedirs(args.out, exist_ok=True)
715
+ shutil.copy2(args.onnx, os.path.join(args.out, "model.onnx"))
716
+ _p16 = args.onnx.replace("_fp32.onnx", "_fp16.onnx")
717
+ if os.path.exists(_p16):
718
+ shutil.copy2(_p16, os.path.join(args.out, "model_fp16.onnx"))
719
+ size = round(os.path.getsize(args.onnx) / 1048576, 2)
720
+ best = {}
721
+ if os.path.exists(args.log):
722
+ for row in csv.DictReader(open(args.log, "r", encoding="utf-8")):
723
+ if not best or float(row["val_mae"]) < float(best["val_mae"]):
724
+ best = row
725
+ ev = {}
726
+ if os.path.exists(args.eval):
727
+ for enc in ("utf-8-sig", "utf-16", "utf-16-le"):
728
+ try:
729
+ for line in open(args.eval, "r", encoding=enc, errors="ignore"):
730
+ line = line.strip()
731
+ if line.startswith("{") and '"precision"' in line:
732
+ ev = json.loads(line); break
733
+ except Exception:
734
+ pass
735
+ if ev:
736
+ break
737
+ st = json.load(open(os.path.join(DATA, "label_stats.json"), "r", encoding="utf-8"))
738
+ clips = len([d for d in os.listdir(TILES) if os.path.isdir(os.path.join(TILES, d))])
739
+ pct = lambda x: ("%.2f%%" % (float(x) * 100)) if str(x) not in ("", "n/a") else "n/a"
740
+ vals = {"params": args.params, "opset": 17, "size_mb": size, "epochs": args.epochs, "clips": clips,
741
+ "samples": st.get("train", 0) + st.get("val", 0), "train": st.get("train"), "val": st.get("val"),
742
+ "ratio": "1 : %.2f" % (st.get("kept_neg", 0) / max(1, st.get("kept_pos", 1))),
743
+ "hard": st.get("hard_mined"), "mae": best.get("val_mae", "n/a"),
744
+ "acc": pct(best.get("val_acc", "n/a")), "precision": pct(ev.get("precision", "n/a")),
745
+ "recall": pct(ev.get("recall", "n/a")), "pos_mae": best.get("pos_mae", "n/a"),
746
+ "neg_mae": best.get("neg_mae", "n/a"), "amd_ms": args.amd_ms, "cpu_ms": args.cpu_ms}
747
+ for k, v in vals.items():
748
+ card = card.replace("{" + k + "}", str(v))
749
+ cfg = {"model_name": "Easy4K SFDP - Siamese NET AI", "model_version": "0.6.0",
750
+ "architecture": "siamese depthwise-separable CNN + difference/product fusion head",
751
+ "input": {"names": ["frame_a", "frame_b"], "shape": ["batch", 1, 128, 128],
752
+ "dtype": "float32", "range": [0, 1], "color": "grayscale"},
753
+ "output": {"name": "score", "shape": ["batch", 1], "meaning": "1 = duplicate"},
754
+ "opset": 17, "providers": ["DmlExecutionProvider", "CPUExecutionProvider"],
755
+ "recommended_threshold": 0.5,
756
+ "metrics": {"val_mae": float(best.get("val_mae", 0)), "val_acc": float(best.get("val_acc", 0)),
757
+ "precision": float(ev.get("precision", 0)), "recall": float(ev.get("recall", 0))},
758
+ "license": "cc-by-nc-4.0"}
759
+ json.dump(cfg, open(os.path.join(args.out, "config.json"), "w", encoding="utf-8"),
760
+ ensure_ascii=False, indent=2)
761
+ open(os.path.join(args.out, "README.md"), "w", encoding="utf-8").write(card)
762
+ open(os.path.join(args.out, "LICENSE"), "w", encoding="utf-8").write(
763
+ "Easy4K SFDP - Siamese NET AI\nCopyright (c) 2026 Easy4K\nCC BY-NC 4.0\n"
764
+ "https://creativecommons.org/licenses/by-nc/4.0/legalcode\n")
765
+ open(os.path.join(args.out, "requirements.txt"), "w", encoding="utf-8").write(
766
+ "onnxruntime-directml>=1.18\nnumpy>=1.26\n")
767
+ src = os.path.join(RAW, "sources.json")
768
+ if os.path.exists(src):
769
+ shutil.copy2(src, os.path.join(args.out, "sources.json"))
770
+ print("PACKED", args.out, flush=True)
771
+
772
+
773
+ def main():
774
+ ap = argparse.ArgumentParser(); sub = ap.add_subparsers(dest="cmd", required=True)
775
+ p = sub.add_parser("rank")
776
+ p.add_argument("--target", type=int, default=40); p.add_argument("--pages", type=int, default=6)
777
+ p.add_argument("--min-sec", type=int, default=60); p.add_argument("--max-sec", type=int, default=420)
778
+ p.add_argument("--seeds", type=int, default=60); p.add_argument("--seed-existing", type=int, default=300)
779
+ p.add_argument("--outdir", default=BILI_RAW); p.set_defaults(func=cmd_rank)
780
+ p = sub.add_parser("tiles")
781
+ p.add_argument("--raw", required=True); p.add_argument("--out", default=TILES)
782
+ p.add_argument("--tile", type=int, default=320); p.add_argument("--frames", type=int, default=32)
783
+ p.add_argument("--segments", type=int, default=4); p.add_argument("--min-width", type=int, default=640)
784
+ p.add_argument("--seed", type=int, default=1234); p.set_defaults(func=cmd_tiles)
785
+ p = sub.add_parser("teacher")
786
+ p.add_argument("--tiles", default=TILES); p.add_argument("--labels", default=LABELS)
787
+ p.set_defaults(func=cmd_teacher)
788
+ p = sub.add_parser("label")
789
+ p.add_argument("--max-neg-ratio", type=float, default=2.0); p.add_argument("--mid-ratio", type=float, default=0.25)
790
+ p.add_argument("--seed", type=int, default=2026); p.set_defaults(func=cmd_label)
791
+ p = sub.add_parser("train")
792
+ p.add_argument("--out", required=True); p.add_argument("--epochs", type=int, default=8)
793
+ p.add_argument("--batch", type=int, default=256); p.add_argument("--lr", type=float, default=2e-3)
794
+ p.add_argument("--workers", type=int, default=8); p.add_argument("--hard-weight", type=float, default=2.0)
795
+ p.add_argument("--warmup", type=int, default=2); p.add_argument("--augment", type=int, default=0)
796
+ p.add_argument("--seed", type=int, default=2026); p.add_argument("--resume", default="")
797
+ p.set_defaults(func=cmd_train)
798
+ p = sub.add_parser("eval"); p.add_argument("--ckpt", required=True); p.set_defaults(func=cmd_eval)
799
+ p = sub.add_parser("compare"); p.add_argument("--ckpts", required=True); p.set_defaults(func=cmd_compare)
800
+ p = sub.add_parser("pack")
801
+ p.add_argument("--onnx", required=True); p.add_argument("--out", required=True)
802
+ p.add_argument("--log", required=True); p.add_argument("--eval", required=True)
803
+ p.add_argument("--params", default="2602017"); p.add_argument("--epochs", default="8")
804
+ p.add_argument("--amd-ms", default="0.56"); p.add_argument("--cpu-ms", default="0.317")
805
+ p.set_defaults(func=cmd_pack)
806
+ args = ap.parse_args(); args.func(args)
807
+
808
+
809
+ if __name__ == "__main__":
810
+ main()
model_fp16.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:666011d5fdee4a9b3e03d58e1c1f04fecc9c8dd25d76a7125c769f9bd1808597
3
+ size 5248967
requirements.txt CHANGED
@@ -1,2 +1,9 @@
1
- onnxruntime-directml>=1.18
2
- numpy>=1.26
 
 
 
 
 
 
 
 
1
+ # 推理
2
+ onnxruntime-directml>=1.18
3
+ numpy>=1.26
4
+
5
+ # 训练 / 导出
6
+ torch>=2.6.0
7
+ opencv-python>=4.9
8
+ onnx>=1.16
9
+ onnxconverter-common>=1.16
train_log.csv ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ epoch,lr,loss,val_mae,val_acc,pos_mae,neg_mae,sec
2
+ 0,0.001001,0.0362,0.03359,0.96952,0.01559,0.04536,529.4
3
+ 1,0.002,0.02679,0.03907,0.96521,0.0104,0.06385,478.4
4
+ 2,0.001866,0.02528,0.03599,0.96902,0.00891,0.05877,477.7
5
+ 3,0.0015,0.02081,0.03631,0.96857,0.00749,0.06233,478.5
6
+ 4,0.001,0.01861,0.03052,0.97374,0.00938,0.04752,477.6
7
+ 5,0.0005,0.01506,0.03056,0.97374,0.00911,0.04821,478.1
8
+ 6,0.000134,0.01389,0.02959,0.97451,0.00888,0.04681,476.9
9
+ 7,0.0,0.01253,0.02788,0.97562,0.01063,0.04085,477.7
训练素材清单.txt ADDED
The diff for this file is too large to render. See raw diff