OpenTWBench

An open evaluation suite for Taiwan-domain knowledge in large language models.

📄 arXiv:2610.19848 · 🏆 opentwbench.ai — 108 models, per-subject scores, raw predictions · 📚 Collection

Most Chinese-language benchmarks are built from mainland sources, in Simplified Chinese, about mainland institutions. A model can score well on them while knowing nothing about the Labor Standards Act, the Household Registration Act, or how a Taiwanese pharmacist is licensed. These datasets exist to make that gap measurable.

The two benchmarks

They measure different things on purpose. tw-exam-bench is the professional knowledge Taiwan licenses: 神經疾病物理治療學 and 骨科疾病物理治療學 are separate configs, because a model can know one and not the other. tw-formosa-bench is the Taiwan those examinations never ask about — 蔣渭水's《臨床講義》, who designed 衛武營, the MRT rules, the night markets — curated so that a frozen seven-model pool scored at the passing margin rather than at the ceiling.

from datasets import load_dataset

ds = load_dataset("OpenTWBench/tw-exam-bench", "dentistry", split="test")
ds = load_dataset("OpenTWBench/tw-exam-bench", "field_medicine", split="test")
ds = load_dataset("OpenTWBench/tw-exam-bench", "all", split="test")
ds = load_dataset("OpenTWBench/tw-formosa-bench", "all", split="test")

Every exam item keeps its paper_id and q_no, so any score traces back to the original PDF; every Formosa item keeps the source_title and source_path of the Taiwan.md passage it was written from.

What the leaderboard reports

Every score on opentwbench.ai is a \boxed{} score: the model is told to put the answer letter in \boxed{}, and a response that does not is counted wrong, not pattern-matched into one. The lenient score — what pattern matching would have added — sits beside it in the same row, so what a model knows and what format it writes in are two columns rather than one.

This matters more than it sounds. On identical questions and models, switching extraction from pattern matching to \boxed{} moves accuracy from 20.8% to 93.9%. A board that mixes the two is not one measurement.

Reading the scores honestly

• These are public past papers, very likely in pretraining corpora. Use the suite for relative comparison between models, not to claim a model "passes" an exam.
• Check the \boxed{} rate before believing a low score. A model that knows the answer but will not write it in the requested shape is a formatting failure, and the board shows that rate next to every number.
• Shuffle the options. The source papers carry an answer-position skew, and one placeholder character in a system prompt once moved a 675B model from last place to first on the same questions.
• Answer keys agree across 31,312 duplicated question groups at 99.91%; all 27 disagreements are accounted for item by item. That figure is the noise floor of anything built on this source — differences finer than it are measuring the apparatus, not the model.

Taiwan law is covered separately and in depth by lianghsun/tw-legal-benchmark-v2; questions already published there are excluded from this org, item by item.

Getting your model on the leaderboard

We can run it for you. Publish the model on Hugging Face — gated or private is fine — grant read access to OpenTWBench, and open an issue here with the repo id. We serve it, run the full suite, and publish the result tagged Official alongside the raw prediction files.

You can also run the suite yourself and submit the numbers; those appear tagged Self-reported. Every leaderboard row carries its tag, so a reader can tell who produced the score. See opentwbench.ai for both routes.

The suite is not limited to what we build. Any benchmark that measures Taiwan in Traditional Chinese belongs here, whatever its source. Contributed datasets keep their own home and their own maintainers; we list them, tagged Community, after review. Send us a link from opentwbench.ai.

Contributing

A wrong answer key is worse than a missing question. Open a discussion on the dataset it affects, quoting the paper_id and q_no, and it can be traced to the original PDF.

Citation

@misc{opentwbench2026,
  title        = {OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge
                  Evaluation for Large Language Models, and How the Measurement
                  Apparatus Shapes the Score},
  author       = {Huang, Liang-Hsun and Chen, Min-Yi and Chuang, Chao-Chun and
                  Liu, Yu-Wei and Wu, Che-Yu and Lee, Po-Feng},
  year         = {2026},
  eprint       = {2610.19848},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL}
}

Examination questions and answer keys are official publications of 考選部, released as open data; packaging and metadata are Apache-2.0. tw-formosa-bench is CC BY-SA 4.0, inherited from Taiwan.md.