GLINER quantized INT8

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,141 @@
+---
+library_name: gliner2-onnx
+base_model: fastino/gliner2-multi-v1
+tags:
+  - onnx
+  - gliner
+  - gliner2
+  - ner
+  - named-entity-recognition
+  - zero-shot
+  - classification
+license: mit
+---
+
+> **Experimental ONNX build** - Unofficial ONNX export of [fastino/gliner2-multi-v1](https://huggingface.co/fastino/gliner2-multi-v1).
+
+# gliner2-onnx
+
+GLiNER2 ONNX runtime for Python. Runs GLiNER2 models without PyTorch.
+
+This library is experimental. The API may change between versions.
+
+## Features
+
+- Zero-shot NER and text classification
+- Runs with ONNX Runtime (no PyTorch dependency)
+- FP32 and FP16 precision support
+- GPU acceleration via CUDA
+
+All other GLiNER2 features such as JSON export are not supported. 
+
+## Installation
+
+```bash
+pip install gliner2-onnx
+```
+
+## NER
+
+```python
+from gliner2_onnx import GLiNER2ONNXRuntime
+
+runtime = GLiNER2ONNXRuntime.from_pretrained("lmo3/gliner2-large-v1-onnx")
+
+entities = runtime.extract_entities(
+    "John works at Google in Seattle",
+    ["person", "organization", "location"]
+)
+# [
+#   Entity(text='John', label='person', start=0, end=4, score=0.98),
+#   Entity(text='Google', label='organization', start=14, end=20, score=0.97),
+#   Entity(text='Seattle', label='location', start=24, end=31, score=0.96)
+# ]
+```
+
+## Classification
+
+```python
+from gliner2_onnx import GLiNER2ONNXRuntime
+
+runtime = GLiNER2ONNXRuntime.from_pretrained("lmo3/gliner2-large-v1-onnx")
+
+# Single-label classification
+result = runtime.classify(
+    "Buy milk from the store",
+    ["shopping", "work", "entertainment"]
+)
+# {'shopping': 0.95}
+
+# Multi-label classification
+result = runtime.classify(
+    "Buy milk and finish the report",
+    ["shopping", "work", "entertainment"],
+    threshold=0.3,
+    multi_label=True
+)
+# {'shopping': 0.85, 'work': 0.72}
+```
+
+## CUDA
+
+To use CUDA for GPU acceleration:
+
+```python
+runtime = GLiNER2ONNXRuntime.from_pretrained(
+    "lmo3/gliner2-large-v1-onnx",
+    providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
+)
+```
+
+## Precision
+
+Both FP32 and FP16 models are supported. Only the requested precision is downloaded.
+
+```python
+runtime = GLiNER2ONNXRuntime.from_pretrained(
+    "lmo3/gliner2-large-v1-onnx",
+    precision="fp16"
+)
+```
+
+## Models
+
+Pre-exported ONNX models:
+
+| Model | HuggingFace |
+|-------|-------------|
+| gliner2-large-v1 | [lmo3/gliner2-large-v1-onnx](https://huggingface.co/lmo3/gliner2-large-v1-onnx) |
+| gliner2-multi-v1 | [lmo3/gliner2-multi-v1-onnx](https://huggingface.co/lmo3/gliner2-multi-v1-onnx) |
+
+Note: `gliner2-base-v1` is not supported (uses a different architecture).
+
+## Exporting Models
+
+To export your own models, clone the repository and use make:
+
+```bash
+git clone https://github.com/lmoe/gliner2-onnx
+cd gliner2-onnx
+
+# FP32 only
+make onnx-export MODEL=fastino/gliner2-large-v1
+
+# FP32 + FP16
+make onnx-export MODEL=fastino/gliner2-large-v1 QUANTIZE=fp16
+```
+
+Output is saved to `model_out/<model-name>/`.
+
+## JavaScript/TypeScript
+
+For Node.js, see [@lmoe/gliner-onnx.js](https://github.com/lmoe/gliner-onnx.js).
+
+## Credits
+
+- [fastino-ai/GLiNER2](https://github.com/fastino-ai/GLiNER2) - Original GLiNER2 implementation
+- [fastino/gliner2-large-v1](https://huggingface.co/fastino/gliner2-large-v1) - Pre-trained models
+
+## License
+
+MIT

USAGE.md

@@ -0,0 +1,25 @@
+# gliner2-multi-v1 — INT8 ONNX (CPU)
+
+Dynamically-quantized INT8 ONNX export of `fastino/gliner2-multi-v1`.
+This bundle contains **only** the INT8 graphs — load it with `precision="int8"`.
+
+```python
+from gliner2_onnx import GLiNER2ONNXRuntime
+
+rt = GLiNER2ONNXRuntime(
+    "gliner2-multi-v1-int8",          # this folder
+    precision="int8",                  # required: no fp32 graphs are included
+    providers=["CPUExecutionProvider"],
+)
+
+rt.extract_entities("pagamento polizza tfr A4983AS", ["amount", "reference_number"])
+rt.classify("acquisto gasolio automezzi", ["carburanti", "polizze"], multi_label=True)
+```
+
+For best CPU throughput set intra-op threads to your physical core count via ORT
+`SessionOptions`.
+
+## Contents
+- `gliner2_config.json` — model config, references the INT8 graphs only
+- `config.json`, `tokenizer.json`, `tokenizer_config.json`
+- `onnx/*_int8.onnx` — encoder / classifier / span_rep / count_embed

config.json

@@ -0,0 +1,4 @@
+{
+  "hidden_size": 768,
+  "vocab_size": 250101
+}

gliner2_config.json

@@ -0,0 +1,23 @@
+{
+  "max_width": 8,
+  "special_tokens": {
+    "[SEP_STRUCT]": 250102,
+    "[SEP_TEXT]": 250103,
+    "[P]": 250104,
+    "[C]": 250105,
+    "[E]": 250106,
+    "[R]": 250107,
+    "[L]": 250108,
+    "[EXAMPLE]": 250109,
+    "[OUTPUT]": 250110,
+    "[DESCRIPTION]": 250111
+  },
+  "onnx_files": {
+    "int8": {
+      "encoder": "onnx/encoder_int8.onnx",
+      "classifier": "onnx/classifier_int8.onnx",
+      "span_rep": "onnx/span_rep_int8.onnx",
+      "count_embed": "onnx/count_embed_int8.onnx"
+    }
+  }
+}

onnx/classifier_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5a3b5d015c10301739f8e15f2716f5da3ac7741d4a2b3e9456ef88d77418174
+size 1190416

onnx/count_embed_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b890893dd4e49170ef13dfc01f986c20dfa4a679b422fd9fa0229f8ed38c235
+size 10661306

onnx/encoder_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:162486021e7c65cbd8bba3c264fcfe31921783dc7a3d81c1e90283e44cbca1ce
+size 337059961

onnx/span_rep_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cb04115e8a85de950e81f2551713b34ac8a7b9fd4868e90c2d580fdadc6d6a1
+size 16574642

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1c7ccb287623cccb7c03150953b6d2a09dd95122933393c9151c3a60095c97e
+size 16337353

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": [
+    "[SEP_STRUCT]",
+    "[SEP_TEXT]",
+    "[P]",
+    "[C]",
+    "[E]",
+    "[R]",
+    "[L]",
+    "[EXAMPLE]",
+    "[OUTPUT]",
+    "[DESCRIPTION]"
+  ],
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}