Qwen3.5-4B-Jev

A locally fine-tuned, prefill-only decision model: state + typed questions → Choice, Score, and Noul.

This release contains the LoRA adapter, FP32 scalar decision head, processor, calibrated temperatures, and standalone inference code. It loads the pinned official Qwen3.5-4B base in NF4 4-bit with double quantization and BF16 computation. The base weights are downloaded separately.

jev-benchmark: 99/100 reference agreement, 10/10 context-contrast pairs passed; 1,521/1,521 valid external decisions across nine tasks and option-reversal controls. The latest three-model comparison compares this release with TypeSafe Jev and Microsoft Decision-1. See the evaluation section below for task-level results, latency, and scope; the 99% figure is specific to the 100-item Chinese implicit-intent diagnostic.

This is an independent Jev-like model, not an official TypeSafe model or a reproduction of its weights.

How it works

state + instructions + complete criteria + candidate
  → Qwen3.5-4B (frozen NF4 base + language LoRA)
  → last valid hidden state → FP32 scalar head
  → per-question softmax / calibrated temperature
  → typed response assembled in code

• Choice: candidate distribution and argmax.
• Score: ordered-level distribution and expected level, starting at 0.
• Noul: probability of the true interpretation from false/true candidate scoring.
• Confidence: 1 − H(p)/log(K), a concentration statistic, not a calibrated probability of correctness.
• No response-token decoding, generated JSON, or self-reported numeric probabilities.

The complete criteria are included in each candidate input. This differs from LLM2Jev's independent yes/no scoring and normalization. The typed API design follows TypeSafe's primitives.

Quick start

Validated on Linux aarch64 / NVIDIA GB10 / CUDA 13 / Python 3.12. Use a CUDA-enabled PyTorch build compatible with your platform, then install requirements.txt. The pinned inference stack is Transformers 5.15.0, PEFT 0.20.0, bitsandbytes 0.50.2 and FLA 0.5.2. Other GPU/platform combinations have not been validated for this release.

hf download xuhaodev/Qwen3.5-4B-Jev --local-dir Qwen3.5-4B-Jev
python -m pip install -r Qwen3.5-4B-Jev/requirements.txt

import sys
sys.path.insert(0, "Qwen3.5-4B-Jev")
from qwen_jev import JevModel

model = JevModel.from_pretrained("Qwen3.5-4B-Jev")
# Optional: base_path="/path/to/the/pinned/Qwen3.5-4B"

result = model.predict(
    state="The order has been paid. The warehouse explicitly records that it has not shipped.",
    questions={
        "status": {
            "type": "choice",
            "instructions": "What is the order's shipping status?",
            "criteria": {"pending": "Not shipped", "shipped": "Already shipped"},
        },
        "paid": {"type": "noul", "instructions": "Has the order been paid?"},
        "progress": {
            "type": "score",
            "instructions": "Rate the order's progress.",
            "criteria": ["Unpaid", "Paid but not shipped", "Shipped"],
        },
    },
)
print(result)

Use this loader rather than a generic AutoModelForCausalLM pipeline. The adapter is attached to the multimodal backbone and needs the scalar head and matching template to reproduce the evaluated model. No trust_remote_code auto-execution is required: the downloaded Python modules are explicitly imported. For reproducibility, pass a fixed Hub revision when downloading.

The API model identifier remains qwen35-4b-jev-v1; the public repository name is Qwen3.5-4B-Jev. String and JSON text states are supported. This release validates text only.

Training

Setting	Value
Base	Qwen/Qwen3.5-4B
Base revision	851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Quantization	NF4 4-bit, double quantization, BF16 compute
Adapter	language LoRA rank 16, alpha 32, dropout 0.05
Trainable parameters	32,467,456
Training questions	6,000: 2,449 semantic references + 3,551 program-verifiable rules
Dev / calibration / internal test	600 / 600 / 600 questions, grouped source separation
Training	2 epochs, 750 optimizer updates, accumulation 16 questions
Learning rate	LoRA 1e-5, scalar head 5e-5
Objective	candidate CE; Score ordinal loss; verified equivalent-view consistency
Selected checkpoint	epoch 2, Dev family × primitive macro NLL

Semantic data were recovered from historical reviewed synthetic training records produced using GPT-6 Luna, Grok 4.7, and GPT-6 Sol. Labels are teacher references, not new human annotations. Program-rule labels are computed from generated facts, with balanced family/label sampling. Source groups are weighted inversely to their question count. Related views stay within the same split. Raw training records are not distributed in this model repository.

The 100-item implicit-intent benchmark was excluded from training, checkpoint selection and calibration. The release was hash-locked before a single 100-request final benchmark run; no weights or calibration were adjusted after seeing that result. A state-only exact-hash exclusion check and cross-split semantic near-duplicate audit were performed. Public benchmark exposure during base pretraining cannot be ruled out.

Evaluation

Source: haxudev/jev-benchmark, latest saved comparison published on 2026-10-12, inspected at commit 1782ce0. These comparisons reuse this model's 2026-10-02 results; TypeSafe Jev (jev-1.13.0) and Microsoft Decision-1 (microsoft-decision-1) were tested on 2026-10-12. The evaluated Qwen release is f5864f83, served as qwen35-4b-jev-v1.

Internal implicit-intent diagnostic: 100 items

Dataset v1.0.0 contains Chinese multi-turn relationship conversations, with 50 marriage-topic items and 50 girlfriend-hint items. Ten contrast pairs share the final utterance but differ in context. Qwen's run completed 2026-10-01 21:40 UTC / 2026-10-02 local time, using benchmark commit d6308af.

Metric	Result
Overall reference agreement	99/100 (99%)
Marriage subset	50/50
Girlfriend-hint subset	49/50
Same-final-utterance context pairs (both correct)	10/10

Only love-073 differs from the reference: predicted B (0.51463), reference D (0.47288).

Model	Reference agreement	Marriage	Girlfriend hints	Context pairs, both correct
TypeSafe Jev	100/100 (100%)	50/50	50/50	10/10
Microsoft Decision-1	99/100 (99%)	49/50	50/50	10/10
Qwen3.5-4B-Jev	99/100 (99%)	50/50	49/50	10/10

Microsoft differs on love-018; Microsoft and Qwen agree on 98/100 choices. Reference agreement and inter-model agreement are distinct metrics, and agreement is not proof of correctness. See the item-level comparison, Qwen report, and packaged raw results.

External evaluation: nine tasks and option reversals

Each model has 1,431 primary items + 90 option-reversal controls = 1,521 valid items. All use the same pinned data, prompt templates, and deterministic sampling manifest: fbb21679cd930b0397a3adcc20d408e422cb816596f0921673ef5602621d4d02. Qwen completed its external run on 2026-10-02 at 03:09 UTC, with 1,521 HTTP requests and no failed or invalid primary responses. Models return native structured Choice, Noul, and Score decisions without receiving reference answers or grading information.

Task	Primary items	TypeSafe Jev	Microsoft Decision-1	Qwen3.5-4B-Jev
JevBench easy	48	100%	100%	48/48 (100%)
JevBench original	72	98.6%	100%	68/72 (94.4%)
JevBench hard	111	72.1%	82.0%	61/111 (55.0%)
AG News classification	200	87.0%	89.5%	165/200 (82.5%)
BoolQ reading comprehension	200	90.0%	86.5%	166/200 (83.0%)
ANLI natural language inference	200	71.5%	63.5%	96/200 (48.0%)
SST-5 five-level sentiment	200	52.0%; MAE 0.598	53.0%; MAE 0.552	85/200 (42.5%); MAE 0.679
STS-B semantic similarity	200	Spearman 0.901; MAE 0.506	Spearman 0.930; MAE 0.447	Spearman 0.878; MAE 0.600
Belebele Chinese reading	200	94.0%	97.0%	169/200 (84.5%)

STS-B has continuous targets and is evaluated with Spearman correlation and MAE, not classification accuracy. The JevBench portion covers 231 pinned public items; the six other datasets use deterministic 200-item slices, with ANLI rounds sampled at 67/67/66 items. This is not the full JevBench leaderboard or a 37-task academic evaluation.

Option-reversal metric	TypeSafe Jev	Microsoft Decision-1	Qwen3.5-4B-Jev
Valid pairs	90/90	90/90	90/90
Same semantic choice after reversing options	89/90 (98.9%)	90/90 (100%)	90/90 (100%)
Both decisions correct	75/90 (83.3%)	81/90 (90.0%)	68/90 (75.6%)

Logical items and physical HTTP attempts are counted separately: Jev made 1,521 attempts, Microsoft made 1,537, and Qwen made 1,521. Microsoft's results include recovery of 16 missing answers; valid wrong predictions were not repeated, so its table is a recovered-coverage result.

Sources: full comparison report, machine-readable comparison, Qwen external report, and Qwen external summary. Detailed reports also include Brier, NLL, ECE, and descriptive risk/coverage curves. Jev probabilities and Score retain the API's two-decimal values without renormalization; rounded zero probabilities affect NLL and prevent a direct full-precision probability-quality ranking.

Successful-request latency

All values are milliseconds, measured at the HTTP client for valid structured responses, including incorrect decisions. HTTP errors, timeouts, invalid responses, inter-request delays, and interruption waits are excluded.

Evaluation	Model	Successful requests	Mean	P50	P90	P95	P99
Internal	TypeSafe Jev	100	844.5	786.5	919.0	1,128.6	1,879.6
Internal	Microsoft Decision-1	100	1,759.9	1,608.1	2,194.2	2,303.3	6,288.5
Internal	Qwen3.5-4B-Jev	100	635.4	633.9	646.2	650.3	666.1
External	TypeSafe Jev	1,521	267.4	253.5	286.5	307.8	575.2
External	Microsoft Decision-1	1,521	1,274.1	1,308.5	1,730.1	1,900.0	2,406.4
External	Qwen3.5-4B-Jev	1,521	539.8	422.2	Not recorded	652.5	Not recorded

Qwen's external historical summary retains mean, P50, and P95 only; P90 and P99 cannot be recomputed without per-request timings. The internal Qwen run was sequential, with a loaded model on GB10 over loopback HTTP, including SDK overhead. Dates, deployments, and connections differ across models: these are not same-hardware pure-inference speed, throughput, or cold-start comparisons. See the latency report and machine-readable latency.

Held-out training-distribution test and calibration

The independently held-out internal test achieved 99.0% agreement over 600 questions: 97.93% on 290 semantic teacher references and 100% on 310 program-rule items. Overall calibrated NLL=0.02248 and Brier=0.01135. Program families share templates across splits; these numbers do not establish unseen-template or broad task generalization.

Temperatures were fitted on the separate design-distribution calibration set: Choice=1.91865, Noul=2.37021, Score=2.02511. Calibration improved internal NLL/Brier but did not improve every metric (ECE increased from 0.00673 to 0.00963).

Scope and limits

• Per-candidate input budget: 4,096 tokens; per-question expanded budget: 32,768 tokens.
• Training main inputs were short (maximum candidate length 671 tokens). The 4K limit is an execution budget, not comprehensive long-context quality validation.
• Nine ~7,800-token controlled padded probes passed; this does not establish general 8K capability.
• Vision structure is retained, but image decision quality is not validated in this text-first release.
• Normalized-entropy confidence is not official Jev confidence equivalence or a correctness guarantee.
• The 100-item benchmark is synthetic, single-author and domain-specific; 99% is reference agreement.
• External evaluation involved no benchmark-directed model tuning. Historical training-data overlap and base-pretraining exposure have not been comprehensively ruled out; deployed weight hashes were not independently verified remotely. See the benchmark's Qwen provenance.
• Original adapter/head/calibration hashes are recorded in provenance.json; public packaging changes only portable configuration paths and inference imports.

License and references

Model adapter, decision head and included inference code: Apache-2.0. Base model: Qwen3.5-4B, Apache-2.0. The benchmark dataset is CC BY 4.0, authored by haxudev; evaluation outputs and links are included with attribution, without redistributing the full dataset.

• Model repository
• Benchmark and report
• TypeSafe introduction
• LLM2Jev
• LocalJev