Blame for README.md

---
license: agpl-3.0
base_model: meta-models/Muse-Glimmer-30B
pipeline_tag: image-text-to-text
language:
- ja
library_name: transformers
tags:
- ocr
- document-transcription
- japanese
- vllm
---

# Wasabi-30B

Wasabi-30B reads a Japanese document page image and writes the page as Markdown:

- full text in reading order;
- tables as HTML `<table>`;
- math as LaTeX, `$...$` inline and `$$...$$` for display;
- ruby (furigana) kept inline as `<ruby>base<rt>reading</rt></ruby>`.

## Model details

- Fine-tune of Muse-Glimmer-30B (https://huggingface.co/meta-models/Muse-Glimmer-30B, revision
  `97c77dff50b2797bcc558fa2d909761dbc575c59`).
- A LoRA adapter trained for Japanese page transcription, merged into BF16 weights.
- Two safetensors shards.

## Files in this repository

| path | what it is |
|---|---|
| `model-00001-of-00002.safetensors`, `model-00002-of-00002.safetensors`, `model.safetensors.index.json` | merged BF16 weights |
| `config.json`, `generation_config.json`, `processor_config.json` | model and processor configuration |
| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` | tokenizer and chat template |
| `inference/` | the page reader package (`wasabi_read`, AGPL-3.0) and its two prompts |
| `LICENSE`, `NOTICE`, `notices/` | licence texts: this model (AGPL-3.0), the base model (Apache-2.0) and its usage policy |
| `SHA256SUMS` | checksums of every file |

## Inference

The package reads a folder of page images (`.jpg`, `.jpeg`, `.png`) and writes one `.md` file per page.

    pip install -e "./inference[dewarp]"

Python 3.10+. The reader flattens every page locally with UVDoc on CPU, so torch and OpenCV are required in every
mode, including endpoint mode.

Local serving needs vLLM with muse_glimmer support, built from commit `6adad08767583f52eb4d2122111af0bf638ed5e6`
(version `0.26.1rc1.dev769`). Released vLLM 0.26.0 has no muse_glimmer model, so it cannot serve this model. A
source build needs a CUDA toolchain:

    pip install "vllm @ git+https://github.com/vllm-project/vllm.git@6adad08767583f52eb4d2122111af0bf638ed5e6" "transformers==5.15.0"

Then, with the Python of that vLLM environment, apply the one-file processor fix. Without it, the muse_glimmer
processor of this vLLM commit rejects the `max_image_tokens` setting that every server below is started with, and the
server does not start. The script edits one file, checks its sha256 before and after, and can undo the edit
(`--check`, `--revert`). A server you start yourself (endpoint mode, direct model use) needs it too.

    python inference/patch_vllm_max_image_tokens.py --apply

An endpoint run needs no local vLLM.

Tested with Python 3.12, vLLM `0.26.1rc1.dev769` (commit 6adad087), transformers 5.15.0, torch 2.13.0,
opencv-python-headless 5.0.0.93, numpy 2.3.5 and Pillow 13.0.0.

Wasabi-30B is the only checkpoint the reader needs: it checks the page orientation and reads the page.

Local vLLM (the package starts and stops the servers itself: first the orientation check, then the reader, both from
the same weights):

    python inference/wasabi_read.py pages/ out/ --engine vllm --model-dir /models/Wasabi-30B --tensor-parallel-size 1 --whole-upscale-tokens 8192

Against OpenAI-compatible servers you already run, both serving Wasabi-30B: the reader started with
`--mm-processor-kwargs '{"max_image_tokens": 8192}'`, the orientation check with
`--mm-processor-kwargs '{"max_image_tokens": 4096}'`, and both with `--reasoning-parser muse_glimmer`. Name what each
server serves with `--endpoint-revision` and `--orient-endpoint-revision`; the addresses and revisions are part of the
output identity, so a rerun against another server is refused instead of mixed into the same output directory.

    python inference/wasabi_read.py pages/ out/ --engine endpoint --endpoint-url http://gpu-host:8000 --endpoint-revision wasabi-30b-v1.0 --orient-endpoint-url http://gpu-host:8001 --orient-endpoint-revision wasabi-30b-v1.0

`--plan` prints the stages and exits without loading a model.

The API key, the UVDoc cache location and offline UVDoc use are described under
[Configuration](inference/README.md#configuration).

### Direct model use

This sends the page image as it is: no orientation check, no flattening, no enlargement and no retries. It is a basic
transcription path, separate from the reader above. It uses the vLLM build and the processor fix from the Inference
section.

    pip install openai
    vllm serve /models/Wasabi-30B --served-model-name wasabi --max-model-len 36864 \
        --mm-processor-kwargs '{"max_image_tokens": 8192}' --reasoning-parser muse_glimmer

Then send one page with the OpenAI-compatible API, using the recommended decoding (greedy, repetition penalty 1.10,
seed 42, up to 24,576 new tokens, low reasoning strength):

    import base64
    from openai import OpenAI

    client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
    png = base64.b64encode(open("page.png", "rb").read()).decode()
    prompt = open("inference/wasabi_read/prompts/draft.txt", encoding="utf-8").read()
    r = client.chat.completions.create(
        model="wasabi",
        messages=[{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": "data:image/png;base64," + png}},
            {"type": "text", "text": prompt}]}],
        temperature=0.0,
        seed=42,
        max_tokens=24576,
        extra_body={"repetition_penalty": 1.10,
                    "repetition_detection": {"max_pattern_size": 512, "min_pattern_size": 4, "min_count": 16},
                    "chat_template_kwargs": {"reasoning_strength": "low", "current_date": "2026-10-16"}},
    )
    print(r.choices[0].message.content)

The transcription prompt is the file `inference/wasabi_read/prompts/draft.txt`. The reader pins `current_date` so that the output does
not depend on the day you run it, and `repetition_detection` stops a looping answer early.

`generation_config.json` keeps the base model's sampling defaults; pass `temperature=0` (as above) for the
recommended greedy transcription.

## What the reader does

Per page:

1. Orientation check: Wasabi-30B answers a four-way upright probe at 4,096 image tokens; a turn is applied only when
   the margin is at least 5.0.
2. Flattening with UVDoc on CPU. The UVDoc code and weights are fetched at first use from a pinned commit and checked
   by sha256 before loading.
3. A whole-page read at 8,192 image tokens, greedy.
4. If the answer is degenerate, a retry ladder at temperatures 0.1, 0.3, 0.5 and 0.8.
5. Output is written per page as `<stem>.md`, atomically. Rerunning the same command resumes: finished pages are
   skipped. Pages with no usable answer are listed in `out/.wasabi-read/run-report.json`.

## Benchmark submission

The MDPBench submission includes additional inference components provided to the organizers. The complete inference
code is maintained in a [private repository](https://github.com/sionic-ai/Wasabi-30B).
The code included here runs independently of that repository.

## Limitations

- Japanese-focused. Other languages are not a target.
- The `transformers` engine has not been measured and is not guaranteed to follow the vLLM path's image size and
  decoding; use the vLLM or endpoint engine for the behaviour documented here.
- Each orientation decision comes from one greedy probe per rotation, so a different serving stack (attention backend,
  CUDA build, batching) can change the turn on a few pages.
- Ruby appears only as the model writes it inline; there is no separate ruby pass.
- Photographed pages and very small print can be misread.
- The reader package runs on Linux and macOS only (it uses POSIX file locks). Its local vLLM engine starts the servers
  with the FlashInfer vision-encoder attention backend; to use another backend, serve the model yourself and use
  endpoint mode.

## License

- This model and the reader package: GNU Affero General Public License v3.0 (`LICENSE`).
- Base model Muse-Glimmer-30B: Apache License 2.0 (`notices/Muse-Glimmer-30B-LICENSE.txt`), together with its
  publisher's usage policy (`notices/Muse-Glimmer-30B-USAGE_POLICY.md`).
- UVDoc code (commit 4c9b82b5): MIT License, Copyright (c) 2023 Tanguy MAGNE. The reader downloads the UVDoc code
  and its pretrained weights at first use and does not redistribute them. The weights carry no separate licence
  statement and remain under their upstream terms.

Copyright 2026 Sionic AI Inc.