NagaYu/sumi-ja-pii

STATUS: feature-complete — real-document validation is out of scope by design

All training and evaluation data is synthetic. No real document, no real person's data, nothing derived from either, at any stage. This is not a gap to be closed later: validating a PII detector on real personal data would contradict the premise of the tool. The numbers below are an upper bound measured under synthetic conditions, and they are final.

You must validate on your own documents before deploying. We cannot do that for you, and that is deliberate — not an omission.

墨 Sumi — Japanese PII detection (0.13B, CPU)

A small Japanese PII token classifier that is trained specifically on the confusions English-first tools get wrong: surnames that are also common nouns (森 / 林 / 泉 / 大和 / 青木), place and company homographs, honorific boundaries, and digit strings that merely look like phone numbers.

Pairs with a rule layer for format-determined types and a reversible masking path, so masked text can be sent to an external LLM and the response restored.

Sumi does not guarantee legal or regulatory compliance. It is a tool for reducing the risk of personal data leaving your machine — not a complete detector. Misses will happen. You choose the threshold and the use case. All training and evaluation data is synthetic; no real personal information is used anywhere in this project.

• Code, benchmarks and training pipeline: https://github.com/NagaYu/sumi
• Dataset: https://huggingface.co/datasets/NagaYu/sumi-ja-pii-corpus
• Base model: sbintuitions/modernbert-ja-130m (MIT)
• Parameters: 132M (0.13B)
• Labels: 21 BIO classes over 10 PII types

Detected types

Label	Meaning
NAME	Name (氏名)
ADDRESS	Address (住所)
PHONE	Phone (電話番号)
EMAIL	Email (メールアドレス)
DOB	Birth date (生年月日)
BANK_ACCOUNT	Bank acct (金融口座)
CREDIT_CARD	Card (カード番号)
MYNUMBER	My Number (マイナンバー様式)
MEMBER_ID	Member ID (会員番号)
POSTAL_CODE	Postal (郵便番号)

Usage

The recommended entry point is the sumi package, which combines this model with the rule layer and the calibrator:

pip install "sumi[presidio,onnx] @ git+https://github.com/NagaYu/sumi.git"

from sumi import SumiDetector

det = SumiDetector("NagaYu/sumi-ja-pii")          # or a local directory
for span in det.detect("田中太郎様の連絡先は090-1234-5678です。"):
    print(span.label.value, span.text, round(span.score, 3))
# NAME 田中太郎 0.997
# PHONE 090-1234-5678 0.95

Reversible masking, with the mapping kept locally:

masked, mapping = det.redact(text)   # '<NAME_1>様の連絡先は<PHONE_1>です。'
original = ReversibleMasker().unmask(llm_response, mapping)

Inside Presidio:

from presidio_analyzer import RecognizerRegistry
from sumi.presidio_plugin import register
register(registry)

Raw transformers

The model is a standard ModernBertForTokenClassification. If you use it directly you lose the rule layer, the calibration and the boundary refinement, which is where a good deal of the precision comes from:

from transformers import AutoTokenizer, AutoModelForTokenClassification

tok = AutoTokenizer.from_pretrained("NagaYu/sumi-ja-pii")
model = AutoModelForTokenClassification.from_pretrained("NagaYu/sumi-ja-pii")

What is complete and load-bearing

• EgressGuard. LLMRoundTrip routes every outbound payload through an EgressGuard built from the mapping table. If an original value would be transmitted — raw, NFKC-normalised, or digits-only — the send is blocked, not logged. The guard covers values of three or more characters; one- and two-character values (many Japanese surnames among them) are exempt by design, because they collide with ordinary words — masking still replaces them, but the guard is not a second line of defence for them. The test suite checks this on every synthetic document in its masking fixture, not a sample.
• Three-line Presidio integration — register(registry). Neither spaCy nor GiNZA is required; Sumi analyses Japanese itself.
• ONNX INT8, 133 MB (from 530 MB fp32). Argmax agreement with fp32 is 1.0000 on the export sanity-check sentence, and 0.9964 over 88,106 tokens of 450 evaluation documents (0.9821 on tokens either model labels as PII; 273/450 documents labelled identically). Quantisation is not lossless — the span-level effect is the difference between conditions (D) and (E) in the evaluation below.
• Checksums never gate detection. Card- and number-shaped strings are detected by format; whether the check digit validates is recorded in meta["checksum_valid"] and nothing more.
• The mapping file is written with mode 0600 and never leaves the machine.
• Hard negatives run as a closed loop. After training, false positives on the negative subset are counted per confusion type, and those counts tilt the generation distribution of the next batch (HardNegativeGenerator.reweight_from_errors). The released checkpoint is the first round, trained on uniformly weighted negatives; the weights it produced for the next round (company_as_person 0.146, date_like_nondob 0.146, …) are published as negatives_weights.json.

Evaluation

Measured against four alternatives on identical documents, same CPU, same thread count, each condition in its own process:

• (A) Presidio, default English configuration
• (B) Presidio + GiNZA, carefully mapped (not a straw man)
• (C) Qwen3-4B-Instruct Q4_K_M via llama.cpp, CPU only, prompted
• (D) Sumi fp32, (E) Sumi ONNX INT8

Generalisation set (100 positive / 100 negative documents)

Synthetic PII inserted into Wikipedia / statute / literary prose — no business document templates, so this is the number that reflects real generalisation rather than template memorisation.

Condition	Name	Address	Phone	Birth date	Email	Bank acct	My Number	Member ID	micro F1	FP rate
(A) Presidio, default	0.054	0.000	0.917	0.053	0.192	0.000	0.000	0.000	0.176	0.370
(B) Presidio + GiNZA	0.784	0.077	0.833	0.789	0.231	0.000	0.000	0.024	0.245	0.830
(C) Local LLM 4B (Q4)	0.541	0.897	0.917	0.947	1.000	0.683	0.676	0.659	0.818	0.080
(D) Sumi fp32	0.811	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.978	0.000
(E) Sumi INT8	0.811	0.923	1.000	0.895	1.000	1.000	1.000	1.000	0.953	0.060

The "FP rate" column is the fraction of hard-negative documents (which contain no PII at all) where the system fired at least once.

CPU throughput and memory

Condition	Size	docs/s	ms/doc	Peak RSS (MB)	vs (C)
(A) Presidio, default	0.01B	253.93	4	1231	1825×
(B) Presidio + GiNZA	0.05B	6.05	165	1414	43×
(C) Local LLM 4B (Q4)	4.00B	0.14	7187	5974	baseline
(D) Sumi fp32	0.13B	12.05	83	1192	87×
(E) Sumi INT8	0.13B	22.54	44	1000	162×

Files

File	What it is
model.safetensors	fp32 PyTorch weights
model.onnx	ONNX fp32
model.int8.onnx	ONNX dynamic INT8 — 133 MB, token-level argmax agreement with fp32 0.9964 on evaluation text
onnx/model_quantized.onnx	the same INT8 graph, in the layout transformers.js expects (used by the in-browser demo)
calibrator.json	temperature scaling fitted on held-out spans
sumi_labels.json	label order — keeps ONNX and PyTorch in sync
negatives_weights.json	closed-loop weights: which confusion types the model still gets wrong
mlx/	MLX weights for Apple Silicon
gguf/	GGUF export, weights only, not executable (see limitations)

Training

• 12,000 synthetic documents, 107k gold spans; 2 epochs, batch 32, max length 256
• AdamW, linear warmup and decay, gradient clipping at 1.0
• Checkpoint selected on validation span exact-match F1, not token F1
• Span probability = minimum over constituent token probabilities

Intended use and limitations

Intended for reducing the chance that Japanese personal data reaches an external system — for example masking documents before sending them to a hosted LLM.

Limitations

• On personal names alone, Sumi's lead is small. On the template-independent set Sumi reaches 0.81 against 0.78 for Presidio + GiNZA. Sumi's advantage is not name recall in isolation — it is addresses, the ID/number families, and above all the false-positive rate.
• Trained and evaluated entirely on synthetic data — a fixed upper bound, by design. Using only synthetic data was an ethical decision: a PII detector that had to be trained or proven on real personal data would contradict its own purpose. The same decision sets a ceiling on what can be claimed. Every number in this document was measured under synthetic conditions and is an upper bound; behaviour on real documents cannot be validated here without breaking that premise, and will differ. Validating on your own documents is your step, and it is deliberately left to you.
• Misses are certain. Lowering the threshold raises recall and also raises false positives; the trade-off table above is the honest picture of that.
• Sumi is not a compliance control. It reduces risk; it does not certify anything.
• GGUF is exported but not executable — no mainstream runtime currently runs ModernBERT-style token classification from GGUF. Use ONNX INT8 on CPU.
• Japanese proofreading / typo detection is deliberately out of scope (JWTD and existing public models already cover that).
• The rule layer requires context words for My Number and member IDs. A bare 12-digit string with no surrounding cue is intentionally not flagged, to keep the false-positive rate low.
• Source code docstrings are written in Japanese (the project's working language); all published documentation is in English.

Citation

@software{sumi2026,
  title  = {Sumi: Japanese PII detection that holds up on hard negatives},
  author = {Sumi contributors},
  year   = {2026},
  url    = {https://github.com/NagaYu/sumi}
}