rupeshs/laya-typed-decisions-ov-int8

Laya Typed-Decisions, OpenVINO int8

convaiinnovations/laya-typed-decisions converted to OpenVINO IR, with the weights compressed to 8 bit by NNCF. It is 403 MB instead of 1.6 GB and scores the same on the benchmark.

Laya answers typed questions about a state (pick a label, rate on a scale, yes or no) in one forward pass, with no text generation. This checkpoint is a specialist, fine-tuned on the four typed-decisions workflows: agent-trace observability, customer service, invoice processing and security incidents.

Unofficial conversion. Not affiliated with or endorsed by Convai Innovations, the authors of Laya.

Accuracy

Measured on the test split of LocalLLaMA/typed-decisions (400 cases, 2,000 decisions), with all five questions of a case sent in one request, on CPU with OpenVINO 2025.4.

model	weights	size	accuracy
original checkpoint (as published)	fp32	843 MB safetensors	0.766
OpenVINO fp32 export	fp32	1,607 MB	0.7660 (1532 / 2000)
this model	int8	403 MB	0.7655 (1531 / 2000)

The int8 model is not bit-identical to fp32:

• 28 of the 2,000 top answers (1.4%) differ from the fp32 export, in both directions, which nets out to one decision.
• Probabilities move by 0.003 on average and by 0.10 in the worst case, so a case close to a confidence threshold can land on the other side.

Usage

Rust

With laya-ov and its hub feature:

use laya_ov::{Agent, Question, Questions};
use serde_json::json;

fn main() -> laya_ov::Result<()> {
    let agent = Agent::from_pretrained("rupeshs/laya-typed-decisions-ov-int8")?;

    let state = json!({
        "task": "Rotate the expired TLS certificate on the staging load balancer.",
        "constraints": ["Do not exceed a $50 spend on cloud resources"],
        "trace_summary": {"steps": 11, "tool_errors": 0, "irreversible_actions": 0},
    });

    let questions = Questions::new()
        .with("outcome", Question::choice(
            "How did this agent run turn out?",
            [
                ("success", "The agent completed the task correctly."),
                ("partial", "The agent made progress but did not fully complete the task."),
                ("failure", "The agent did not accomplish the task."),
                ("harmful", "The agent took an action that caused damage or violated a constraint."),
            ],
        ))
        .with("risk", Question::score(
            "How risky was the agent's behaviour in this trace?",
            ["Benign", "Low", "Moderate", "High"],
        ))
        .with("needs_review", Question::noul("This trace requires human review."));

    let prediction = agent.predict(&state, &questions)?;
    println!("{}", serde_json::to_string_pretty(&prediction)?);
    Ok(())
}

Python

The directory has the layout that OVAgent in rupeshs/laya-openvino loads.

Files

file
openvino_model.xml, openvino_model.bin	OpenVINO IR, int8 weights
rl_agent_config.json	sequence limits (max_len 1024, head_max_len 256) and calibration temperatures, copied from the original checkpoint
tokenizer/	the original tokenizer

Model inputs are input_ids, attention_mask, marker_pos, marker_mask and qtype (all int64, dynamic shapes); outputs are logits and act_logits.

How it was made

The checkpoint was traced as is with openvino.convert_model, then compressed with nncf.compress_weights(mode=INT8_ASYM). Only the weights are quantized, which needs no calibration data; activations stay in floating point. Nothing was trained.

pip install laya openvino nncf
python scripts/export_openvino.py convaiinnovations/laya-typed-decisions laya-ov-typed-decisions-int8 --int8

The script is in laya-ov-rs.

Limits

These come from the original checkpoint; see its model card for the detail.

• It is a specialist. Outside the four workflows it was fine-tuned on, expect it to do no better than the base laya model.
• Its probabilities are over-confident (the original card reports ECE 0.213). Refit the temperatures on your own data before relying on them.
• action.act_probability was 1.0 for all 2,000 benchmark decisions, in both the fp32 and int8 exports, so it carries no information for this checkpoint.
• English only.
• Keep choice questions under about 20 options: the options of one question share a 256-token budget.

License

Apache 2.0, the same as the original checkpoint.