saihai-v1-8b-gguf

日本語での紹介記事はこちら: 日本語 Decision モデル "saihai" (采配) と、日本語 Decision モデルベンチマークの公開

A Japanese text decision model that returns yes/no probabilities (Noul), a selection among candidates (Choice), and a rubric score (Score). These GGUF variants of saihai-v1-8b run through llama.cpp's /v1/systemone endpoint. They read candidate logits without generating reasoning or free-form text.

日本語: saihai (采配) は、はい/いいえ・選択・段階評価の判断を確率として返す日本語の Decision モデルです。このリポジトリは llama.cpp 用の GGUF 版で、通常は Q8_0、メモリを抑えたい場合は Q4_K_M がおすすめです。

Other saihai models: saihai-v1-8b (Transformers) and saihai-v1-310m (small encoder).

Recommended variants

Choose Q8_0 for quality or Q4_K_M for lower memory use.

Your priority	Recommended variant	Why choose it?
Preserve BF16 quality	Q8_0	9.13 GB, about 47% smaller than BF16. Japanese Task Avg decreases by only 0.03 points; the highest-probability candidate matches BF16 on 99.03% of judgments.
Reduce memory and download size	Q4_K_M	5.30 GB, about 69% smaller than BF16. Task Avg decreases by 0.66 points; the highest-probability candidate matches BF16 on 92.10% of judgments.

Start with Q8_0 when it fits your memory budget. Choose Q4_K_M when memory is tighter, and check its predictions on your own use case—similar aggregate scores do not imply identical probabilities or decisions. Both delivered about 1.25× BF16 throughput in the separate RTX 5090 measurement. Q4_K_M was only slightly faster than Q8_0, so its main advantage here is lower memory use.

The full comparison is below. Q3_K_M saves more space but loses 2.56 points. Q2_K loses 11.46 points and was slower than Q3_K_M in the measured run.

Variant	Download size	Japanese evaluation score ↑	Difference from BF16 (points)
BF16	17.19 GB	49.53	+0.00
Q8_0	9.13 GB	49.50	-0.03
Q6_K	7.05 GB	49.24	-0.29
Q4_K_M	5.30 GB	48.87	-0.66
Q3_K_M	4.37 GB	46.98	-2.56
Q2_K	3.50 GB	38.08	-11.46

The score is S1MB japanese-v1 Task Avg across 27 benchmarks and 8,067 judgments: the equal-weight average of the Noul, Choice and Score task means. It is not raw accuracy. File sizes exclude KV cache and runtime buffers. See Inference speed for measured throughput and GPU memory.

Quick start

1. Download and start the server

You need the hf CLI and a llama-server build with correct LLM-jp tokenization. See Runtime compatibility for the tested build.

uv pip install 'huggingface-hub>=1.0,<2'
hf download hotchpotch/saihai-v1-8b-gguf saihai-v1-8b-Q8_0.gguf \
  --local-dir ./saihai-v1-8b-gguf

llama-server -m ./saihai-v1-8b-gguf/saihai-v1-8b-Q8_0.gguf \
  --alias saihai-v1-8b-gguf --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 8192 --parallel 1 --flash-attn on --no-context-shift

This example offloads all layers to an NVIDIA GPU and uses an 8K context with one server slot. For the lower-memory recommendation, download saihai-v1-8b-Q4_K_M.gguf instead and replace Q8_0 with Q4_K_M in the server command. The measured peak GPU memory was 12.39 GiB for Q8_0 and 9.21 GiB for Q4_K_M with four 8K slots. Your memory requirement depends on context length and concurrency; these are not minimum requirements.

2. Classify a customer request

This example uses only the Python standard library. It asks whether the customer wants a refund, which department should handle the request, and how urgent it is.

import json
from urllib.request import Request, urlopen

body = {
    "state": "昨日購入した商品を返品したいです。返金をお願いします。",
    "questions": {
        "refund": {
            "type": "noul",
            "instructions": "顧客は返金を求めているか？"
        },
        "department": {
            "type": "choice",
            "instructions": "問い合わせを担当部署に振り分ける。",
            "criteria": {
                "refunds": "返品・返金の対応",
                "delivery": "配送状況・配達の対応",
                "technical": "ログイン・システム障害の対応"
            }
        },
        "urgency": {
            "type": "score",
            "instructions": "問い合わせの緊急度を評価する。",
            "criteria": [
                "通常対応：一般的な問い合わせや依頼",
                "優先対応：利用できず困っている",
                "至急対応：不正利用や被害が進行している"
            ]
        }
    }
}
request = Request(
    "http://127.0.0.1:8080/v1/systemone",
    data=json.dumps(body, ensure_ascii=False).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(request) as response:
    answers = json.load(response)["answers"]

print(f"返金要求あり：{answers['refund']['noul']:.2%}")

department = answers["department"]
names = {"refunds": "返品・返金", "delivery": "配送", "technical": "技術サポート"}
selected = department["choice"]
print(f"担当部署：{names[selected]}（{department['probabilities'][selected]:.2%}）")

urgency = answers["urgency"]
p = urgency["probabilities"]
print(
    f"緊急度：通常対応 {p['0']:.2%}、優先対応 {p['1']:.2%}、"
    f"至急対応 {p['2']:.2%}（スコア：{urgency['score']:.3f}）"
)

The expected interpretation is a refund request, routed to the refunds department, with normal urgency. Actual probabilities vary with quantization and runtime conditions. The urgency score is the probability-weighted average of normal 0, priority 1, and immediate 2.

Inference speed

Measured on an RTX 5090 with four concurrent requests, evaluating 8,067 judgments per variant.

Variant	Judgments/second ↑	Relative to BF16	Observed peak GPU memory
BF16	24.28	1.00×	19.22 GiB
Q8_0	30.44	1.25×	12.39 GiB
Q6_K	28.38	1.17×	10.64 GiB
Q4_K_M	30.58	1.26×	9.21 GiB
Q3_K_M	31.07	1.28×	8.43 GiB
Q2_K	26.56	1.09×	7.70 GiB

Q8_0, Q4_K_M and Q3_K_M delivered approximately 1.25–1.28× BF16 throughput in this run. Lower bit depth does not always mean faster inference.

Throughput uses the end-to-end evaluation time after model startup, including HTTP, scoring and result saving. It is neither generation tokens/second nor single-request latency. The run used an 8K context per slot, prompt-cache reuse and one measurement per variant. Different variants were evaluated concurrently on two GPUs. The memory figures apply to this setup, not to a 64K context.

Inputs, outputs and limits

• Noul: noul is the probability of true.
• Choice: criteria maps candidate names to descriptions. choice is the selected name and probabilities contains every candidate's probability. Supports 1–255 candidates.
• Score: criteria contains 2–10 levels in the intended rubric order. score is the expected zero-based level index.

Questions are evaluated independently. Probabilities are not guaranteed correctness rates and have not been separately calibrated. Input JSON numbers use six significant digits; pass precise quantities as strings. For Score levels with explicit numeric values, compute the expected numeric value from the returned probabilities and your scale.

Longer inputs

The quick-start command uses an 8K context. For the maximum supported context, use -c 65536 --parallel 1. The tested native server reserves one position, allowing up to 65,535 input tokens per slot, including instructions, state, candidates and delimiters. Four slots with 64K capacity each require -c 262144 --parallel 4 and more KV-cache memory. Keep context shifting disabled.

Training used an 8,192-token input limit. Decision quality on longer inputs, including 64K inputs, has not been established by these short-input benchmarks.

Evaluation details

Japanese evaluation — October 13, 2026

All six variants were evaluated with the same S1MB standard llama-cpp input adapter: 27 benchmarks, 8,067 judgments and zero inference failures per variant. Dataset revision: 59b66dd94c430c1ec6db1c948fa2cc0410400a27. Task Avg averages the three task means and differs from the older AVG27 metric, which averages all 27 benchmarks.

Scores by source task and synthetic generalization task

Variant	Source Noul	Source Choice	Source Score	gen-noul	gen-choice	gen-score
BF16	45.17	60.02	55.89	52.67	69.02	21.13
Q8_0	44.98	60.28	55.75	52.22	69.17	21.16
Q6_K	44.79	59.64	55.58	52.67	68.28	21.02
Q4_K_M	44.74	58.86	54.59	53.78	68.43	20.13
Q3_K_M	44.78	57.00	49.95	51.56	66.50	18.97
Q2_K	34.44	48.19	43.34	41.11	60.69	9.46

These columns separate 19 source benchmarks from eight synthetic generalization benchmarks. All values are baseline-adjusted scores ×100.

Training includes train examples from evaluation source families. Evaluation judgments themselves were excluded from training; these results do not establish generalization to unseen source families.

Speed measurement — October 11, 2026

The speed table comes from a separate full run over the same 27 Japanese benchmarks (under its earlier internal name, JDMB-v1), using the same trained tensors before the 64K metadata update. The S1MB quality and speed tables were not measured in the same run. Repeated-run averages and confidence intervals are unavailable. Detailed values, model hashes and evaluation revisions are in benchmark-results.json.

Runtime compatibility

The tested llama.cpp build uses revision 3d65c90d04d337e88f2b1f7f0061f40a5324e662 with the src/llama-vocab.cpp tokenizer fix from PR #28864, commit 7dd044d70a5791ff8205fa26c77074bc77cbe792. This fix is needed for correct LLM-jp Unigram tokenization and byte fallback. No saihai-specific C++ changes or external tokenizer JSON are required.

See the source model card for training details, and gguf-export.json / release-manifest.json for conversion settings and artifact hashes. The training and GGUF conversion code is at hotchpotch/saihai.