spybyscript/gliner25-decide-litert

GLiNER2.5-Decide — LiteRT (Android)

A LiteRT conversion of fastino/GLiNER2.5-Decide (revision bbe10ff77ebb238777c17d3a8ac9260e30929057, Apache-2.0, by Fastino), converted by spybyscript. It carries every trained head: classification, plus the span heads that extract entities, structures and relations. gliner2's own code decodes the outputs. For the model itself (training, labels, benchmarks, intended use), see the original model card, copied here as model-card.md. Parts of the graph code are ported from litert-community/GLiNER2.5-Decide-LiteRT.

Files

File	Size	Contents
gliner25_decide_wfp16.tflite	1028 MB	encoder for the CPU and the GPU at FP32: s128 / s160 / s256 / s512 (classifier logits at every position) and e128 … e512 (hidden states), one set of float16-stored weights
gliner25_decide_fp16hw.tflite	1028 MB	the same encoder rewritten for FP16 hardware (GPU at FP16, NPU): LayerNorm power-of-two pre-scale, first feed-forward layer split in two
gliner25_decide_span_wfp16.tflite	76 MB	span heads in a form LiteRT's GPU compiles: w{W} span representations, f{W}_p{P} fused span scores, count, classifier k32 / k128
gliner25_decide_count_wfp16.tflite	38 MB	count embedding c{C}_m{M} (CPU)
gliner25_decide_heads_wfp16.tflite	105 MB	all span heads in the library's layout (CPU or NPU); the span file and the count file replace it
gliner25_decide_embeddings_f16.bin	262 MB	token table, float16 [128011, 1024], little-endian, row-major
runtime/		litert_span_runtime.py (reference runtime), verify_parity.py, parity_corpus.json
npu/		Qualcomm NPU files (below)

config.json lists every graph's signatures, shapes and host steps.

Use

# pip install gliner2==2.0.0 torch numpy ai-edge-litert
import sys; sys.path.insert(0, "runtime")
from litert_span_runtime import LiteRTSpanExtractor

model = LiteRTSpanExtractor(".", span="gliner25_decide_span_wfp16.tflite", count="gliner25_decide_count_wfp16.tflite")
model.classify_text("Please cancel my order.", {"intent": ["cancel", "refund", "track"]})
model.extract_entities("Alice founded Acme in Toronto in 2020.", ["person", "organization", "city"])
model.extract_relations("Alice founded Acme in Toronto.", ["founded", "located_in"])

Every public gliner2 call works: extract, classify_text, extract_entities, extract_json and extract_relations, plus their batch_* and *_long forms. The runtime runs gliner2's span extractor with the five trained modules on LiteRT. Schema handling, scoring and decoding are the library's own code.

On a phone, run the same graphs with LiteRT's CompiledModel:

1. Tokenize with tokenizer.json and pack the prompt as gliner2 2.0.0 does.
2. Look the ids up in the table (row id at byte id * 2048) into inputs_embeds.
3. Call the smallest signature that holds the request.
4. For classification, read each task's [L] marker positions from logits.

GPU settings: set the precision explicitly (FP32 for wfp16, FP16 for fp16hw) and turn on constant tensor sharing. The right priority depends on the GPU; see "GPU priority" below. On the CPU, set XNNPACK's weight cache path. On a Snapdragon NPU, load the chip's precompiled file.

Fixed sizes. The original runs with dynamic shapes; this conversion has fixed limits:

• a request holds at most 512 tokens (schema and text);
• spans up to 8 words, and up to 20 instances per structure (the checkpoint's own limits);
• up to 32 fields or entity types per task.

A larger request is refused with a ValueError, never truncated. For long text, use the *_long calls with a chunk that fits. The weights change only in storage and compute precision.

The span file computes the heads file's span representations with two exact rearrangements (only the order of float sums differs):

• the first layer of the span head is computed once per word instead of once per span;
• f{W}_p{P} folds the last layer into the queries, so a host that needs only the scores skips the [W, 8, 1024] span tensor.

Every tensor is [1, rows, features], the shapes LiteRT's GPU delegate accepts.

Validation

runtime/verify_parity.py runs each request through gliner2 with the original weights and through the reference runtime (LiteRT CPU), then compares the structured results:

• The corpus (runtime/parity_corpus.json): 296 requests over 16 call types (entities, relations, structures, the three record modes, attributes, single- and multi-label classification, mixed schemas). All 296 are identical, with span F1 1.0 and confidences within 0.0013.
• The same check with the span file and the count file: also 296 of 296.
• FluidInference's eleven schema-path fixtures: 11 of 11.
• 2,338 recorded classification decisions: wfp16 gives the library's label on all of them, and fp16hw at FP16 precision on 2,332.

validation-full.json and validation.json hold the numbers.

Phones

LiteRT 2.3.2 with qnn-runtime 2.50.0. Each graph ran on a request recorded from the corpus: 10 calls after a rest, at thermal status 0. The CPU ran 4 threads; the GPU ran at NORMAL priority, timed as whole calls. CPU and NPU times are the model alone. The NPU rows use the chip's precompiled files; at 512 tokens they use the classification file, because the hidden-state file stops at 256.

Galaxy Z Fold 8 (SM8850, Snapdragon 8 Elite Gen 5, Android 17)

Graph	CPU	GPU FP16 (fp16hw)	GPU FP32 (wfp16)	NPU
encoder, 128 / 160 / 256 / 512 tokens	115 / 145 / 262 / 722 ms	60 / 72 / 121 / 404 ms	105 / 137 / 214 / 610 ms	23 / 30 / 95 / 644 ms
span file w64 / w128 / w256 / w512	10 / 20 / 38 / 77 ms	5.3 / 7.1 / 12 / 24 ms	9.5 / 14 / 35 / 76 ms	7.0 / 4.4 / 8.7 / 31 ms
fused scores f64 … f512, 16 queries	6.4 / 14 / 30 / 55 ms	4.4 / 5.4 / 7.9 / 15 ms	7.6 / 10 / 16 / 27 ms	3.1 / 3.9 / 8.2 / 33 ms
count embedding, 1 step × 8 fields / 20 × 32	1.5 / 37 ms	not supported by LiteRT's GPU	not supported	0.9 / 2.6 ms
classifier, 32 / 128 rows	0.2 / 0.8 ms	0.5 / 0.6 ms	1.0 / 1.0 ms	0.5 / 0.8 ms
dirty memory, encoder / span file	2.3 / 1.4 GB	2.8 / 0.8 GB	4.8 / 1.4 GB	0.14 / 0.19 GB

• An extraction request (up to 128 tokens and 64 words) takes about 35 ms on the NPU. On the GPU it takes about 70 ms at FP16 and 125 ms at FP32; on the CPU, about 135 ms.
• Loading: the precompiled NPU files load in 0.3–1.0 s; the encoder's first GPU load takes 11–13 s.
• Answers with the phone's outputs:
  • the encoder: unchanged on the CPU and at GPU FP32 and FP16; on the NPU, 31 of 32 (one family-relation request changed);
  • the span file and the heads file: unchanged on the GPU and the NPU.

Galaxy S23 Ultra (SM8550, Snapdragon 8 Gen 2, Android 16)

Graph	CPU	GPU FP16 (fp16hw)	GPU FP32 (wfp16)	NPU
encoder, 128 / 160 / 256 / 512 tokens	628 / 866 / 1,467 / 3,486 ms	103 / 124 / 227 / 773 ms	202 / 268 / 471 / 1,549 ms	34 / 55 / 168 / 1,149 ms
span file w64 / w128 / w256 / w512	43 / 105 / 221 / 450 ms	10 / 15 / 25 / 47 ms	19 / 26 / 42 / 79 ms	7.3 / 6.0 / 14 / 42 ms
fused scores f64 … f512, 16 queries	26 / 55 / 111 / 220 ms	8 / 10 / 16 / 27 ms	15 / 18 / 28 / 46 ms	3.8 / 5.6 / 14 / 44 ms
count embedding, 1 step × 8 fields / 20 × 32	2.1 / 169 ms	not supported by LiteRT's GPU	not supported	1.4 / 5.4 ms
classifier, 32 / 128 rows	0.9 / 3.8 ms	1.2 / 1.5 ms	1.6 / 2.4 ms	0.8 / 1.3 ms
dirty memory, encoder / span file	1.9 / 1.7 GB	2.7 / 0.8 GB	4.6 / 1.3 GB	0.12 / 0.21 GB

• An extraction request (up to 128 tokens and 64 words) takes about 43 ms on the NPU. On the GPU it takes about 115 ms at FP16 and 220 ms at FP32; on the CPU, about 660 ms.
• Loading: the precompiled NPU files load in 0.5–1.7 s; the encoder's first GPU load takes 6–10 s.
• Answers with the phone's outputs:
  • the encoder: unchanged on the NPU, at GPU FP32 and on the CPU; at GPU FP16, 31 of 32 (one relation request changed);
  • the span file and the heads file: unchanged on the GPU and the NPU.

Each answer check ran 32 recorded requests (two per call type) with the phone's outputs through the rest of the pipeline.

GPU priority beside a rendering app

An app rendered every frame (Godot 4.7, Vulkan) while the model ran three classification calls each round (s160 and two s256, fp16hw at GPU FP16). Each cell gives the app's longest frame per round (median) and the three calls' total time.

The model on	Galaxy Z Fold 8 (Adreno 840, 60 Hz)	Galaxy S23 Ultra (Adreno 740, 120 Hz)
GPU, LOW priority	37 ms, 0.66 s	600 ms, 1.6 s
GPU, NORMAL priority	110 ms, 0.37 s	228 ms, 0.65 s
NPU	39 ms, 0.24 s	81 ms, 0.5 s
CPU	41 ms, 0.81 s	48 ms, 4–5 s

• LOW priority depends on the GPU. On the Fold's Adreno 840, LOW keeps the frames and costs about 0.3 s per round. On the S23's Adreno 740, LOW does not let the app's rendering in, and only makes each call longer.
• Measure on the target GPU before choosing. Use the smallest signature that fits.
• The NPU keeps the frames on both phones and is the fastest option.

Qualcomm NPU files (npu/)

The NPU path runs through LiteRT's Qualcomm accelerator:

• LiteRT 2.3.2 with litert-npu-runtime-qualcomm 2.3.2 and Qualcomm's qnn-runtime 2.50.0 (Qualcomm's AI Stack License);
• files precompiled on x86-64 Linux with QAIRT 2.50.0;
• each signature in one NPU partition;
• each file runs only on its chip (Build.SOC_MODEL) and with those runtime versions.

npu/manifest.json lists every file with its chip, signatures, size and SHA-256.

File	Contents	Size
npu/qualcomm/gliner25_decide_fp16hw_<chip>.tflite	classification, s128 … s512	856–877 MB
npu/qualcomm/gliner25_decide_fp16hw_se128-256_<chip>.tflite	classification and hidden states, 128–256 tokens	777–792 MB
npu/qualcomm/gliner25_decide_heads_<chip>.tflite	the heads file	108–115 MB
npu/qualcomm/gliner25_decide_span_<chip>.tflite	the span file	80–81 MB
npu/gliner25_decide_fp16hw_s128.tflite, _s160, _s256	single-signature cuts that the phone compiles itself, once (on the S23: 21 / 41 / 117 s)	660–710 MB

Chip	Snapdragon	Hexagon	Run on a phone
SM8850	8 Elite Gen 5	V81	all four files (Galaxy Z Fold 8, the results above)
SM8750	8 Elite	V79	not yet
SM8650	8 Gen 3	V75	not yet
SM8550	8 Gen 2	V73	all four files (Galaxy S23 Ultra, the results above)
SM8450	8 Gen 1	V69	not yet

Files

• LICENSE (Apache-2.0) and NOTICE.txt (attribution and changes)
• model-card.md, the original card
• config.json, the contract
• validation.json and validation-full.json
• SHA256SUMS
• the tokenizer files
• source/ (the checkpoint's configs)
• runtime/ and npu/