small-models-for-glam/britannica-illustration-detector-seg-v2

Britannica Illustration Detector

Detects and segments individual illustrations — engravings, figures, maps, diagrams, portraits — on scanned Encyclopaedia Britannica pages (a multi-edition mirror, 1768 onward). RF-DETR Seg small (Roboflow/rf-detr-seg-small, DINOv2-distilled backbone, 29M parameters), fine-tuned for instance segmentation. Default operating threshold: 0.4.

Why this model

Encyclopaedia plates carry 10–20 individual figures on one page; page-level classifiers can say this page has illustrations but not where each one is, which is what extraction, cropping, and image-dataset building need. No labelled data existed for this material, and none was hand-annotated for this model: labels were bootstrapped zero-shot from a large teacher model and corrected in two loops (details under Provenance). Total training cost was roughly $25 of cloud GPU time.

Small is the point: 29M parameters runs at ~1 s/page on a laptop CPU, so a full multi-thousand-page corpus is an overnight local job or a few dollars of batch compute — the operating regime most GLAM institutions actually have. The same recipe (the detection-bootstrap skill, SKILL.md) transfers to other collections with no labels.

Performance

Eval	mAP@[.5:.95]	mAP@50
vs teacher labels, masks (100-page held-out gold slice)	0.618	0.830
vs human gold	not yet measured — annotation pass pending
validation split, boxes (loop-2 final)	0.809	0.898

Validation at loop-2 completion: precision 0.910, recall 0.882, segm-mAP@50 0.895.

Teacher-agreement is flat vs loop-1 by design: loop-2 trained on corrections of the teacher's misses (see Provenance), so it deliberately diverges from the teacher exactly where the teacher was wrong. Recall against human truth is therefore expected to sit above the vs-teacher numbers, but is unmeasured. Human pass/fail gate: 40 fresh never-seen pages reviewed at threshold 0.4 — near-all passed (verbal review; a systematic rate was not recorded).

Predictions at score ≥ 0.3 on pages from the full-corpus store, boxes only — left: a plate of medals, 4th edition 1810 (37 instances, each face boxed separately); centre: a text page, 12th edition 1922, where all three in-text figures are found, including a thin-line circle diagram the 0.4 cut would drop; right: a text page, 8th edition 1858, with two woodcuts. Every box shown was confirmed as an illustration by a VLM judge.

Usage

from rfdetr import RFDETRSegSmall
import numpy as np, torch
from PIL import Image
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("davanstrien/britannica-illustration-detector-seg-v2", "checkpoint_best_ema.pth")
model = RFDETRSegSmall.from_checkpoint(ckpt)
model.model.device = torch.device("cpu")  # or "cuda" / "mps"
dets = model.predict(np.array(Image.open("page.jpg").convert("RGB")), threshold=0.4)
# dets.xyxy, dets.mask, dets.confidence

Runs on CPU (~1 s/page), CUDA, and Apple Silicon (MPS; set the device as above — rfdetr defaults to CUDA internally).

Threshold guidance

• 0.40 — safe default for one-shot use.
• 0.30 — display default in the crop_masks release: on the full-corpus check below, 0.25–0.30 instances were still ~97% illustrations, and this cut recovers the small in-text line diagrams 0.4 drops.
• 0.25–0.30: ~+20% instances on dense plates; more fragments and merges. Use for recall-critical harvesting with human or VLM triage downstream.
• 0.55+: drops faint text-page figures.

Measured on the full corpus (2026-08-26)

After the page store was rebuilt at full resolution (biglam/britannica, source/pages/, ~977k pages), the model was re-checked on a fresh stratified sample of 78 dataset pages (plates and in-text figures × pre-1850 / 1850–1900 / post-1900), run on the 2000 px JPEG copies (source/jpg/). The model resizes internally, so the JPEG and the original give the same predictions; masks are emitted in the original page's frame.

Precision (VLM-judged). Every instance the model proposed at ≥ 0.25 (377 instances) was shown as a margin crop to Qwen3.8-27B with the question "is the central region an illustration?". 374/377 confirmed. By score bin: 0.2–0.3 → 57/59, ≥ 0.4 → 317/318. The three rejects were a library stamp (0.94) and two runs of text lines (0.26, 0.33). Practical consequence: the stored threshold is 0.10 so consumers choose; 0.3 is a sensible display default, and going down to 0.25 costs almost nothing in precision.

Recall (where it misses). 20 of the 78 pages had no detection at any threshold. Asked page-level, the judge said 15 of those have no illustration (the page classifier that built the dataset over-included them — mostly pre-1850 text pages) and 5 are real misses: an atlas map, sponge-spicule diagrams, one anatomical engraving, film-strip traces, and a page of three small line drawings. So whole-page misses ≈ 6 % of illustrated pages, concentrated in small in-text line diagrams; dense plates (pre-1850 average 10.5 instances/page) are the model's strongest case.

Masks are tight. The masks are the raw model output and tend to sit slightly inside the drawn edges (inherited from the teacher's weak labels). For cut-outs, dilate by ~2 % of the box size:

from scipy import ndimage
px = round(0.02 * max(w, h))            # w, h = the instance's box size in pixels
mask = ndimage.binary_dilation(mask, iterations=px)

The crop_masks config on the dataset stores the raw masks and the raw boxes; apply the margin at use time.

Frame. All boxes and RLE masks are in the coordinate frame of the full-resolution original (bucket_url, src_width × src_height), not the JPEG or the dataset thumbnail. To use them on the JPEG, scale by jpg_width / src_width.

Provenance

Labels are zero-shot weak labels plus two correction loops — no human annotations.

1. Teacher pass: falcon-perception-bucket.py (uv-scripts/object-detection) ran tiiuae/Falcon-Perception with query illustration (segmentation task) over 4,285 plate pages → 21,625 mask instances; 1,049 pages labelled empty. Zero-shot; the teacher emits no confidence scores.
2. Filters: box clamping to image bounds; degenerate-sliver removal (extreme aspect ratio + <0.5% area); IoU>0.9 near-duplicate removal within page. ~1% of boxes affected.
3. Loop 1: RF-DETR Seg small trained on the pool above (3,766 train pages; a 100-page stratified gold slice held out and never trained on).
4. Rescue pass: loop-1 run at threshold 0.3 over the teacher's "empty" pages found detections on 328/1,023 (32%) — the teacher's systematic miss. A VLM judge (Qwen3.8-27B-FP8) confirmed 555/624 rescued instances; 34 text-figure pages (a page class the teacher never labelled) added 145 confirmed instances.
5. Loop 2 (this model): retrained on pool + 309 corrected pages (3,833 train pages, 19,568 instances).

Trained by Daniel van Strien using the detection-bootstrap recipe above, run end to end by a coding agent on Hugging Face Jobs.

Limitations

• Dense plates (15+ figures): individual figures are sometimes merged into one mask.
• The teacher's blindness to small text-embedded figures is reduced, not eliminated — loop 2 saw only ~34 such pages.
• About two-thirds of the teacher-"empty" pages produced no student detections and were never re-examined: some true illustrations there remain unlabelled, and those pages were trained as negatives.
• Trained on Britannica scans only; other encyclopaedias and printing styles untested.
• The VLM confirmation layer is itself a model judgment, not human ground truth.

Training configuration

12 epochs, batch 4, rfdetr SegmentationTrainConfig defaults (COCO dataset file, 576 resolution), a10g-large, best-EMA checkpoint at epoch 6 (val segm mAP 0.731). Full config in training_config.json.

Licence & attribution

Model: Apache-2.0 (RF-DETR Seg base). Page scans: public-domain Encyclopaedia Britannica volumes from the Internet Archive, served as a page store in the biglam/britannica bucket; the illustrated subset is the dataset biglam/britannica-illustrated-pages. Teacher: Falcon-Perception (TII). This model is a first-pass distillation product — treat its labels and outputs accordingly.