README.md

---
license: other
license_name: audarai-community-license-v1.0
license_link: https://www.audarai.com/license/audarai-community-license-v1.0/
library_name: nemo
language:
- ar
- en
- multilingual
pipeline_tag: audio-classification
inference: false
tags:
- speaker-diarization
- diarization
- streaming
- realtime
- sortformer
- arabic
- multilingual
- nemo
- pytorch
- audar
datasets:
- ami
- alimeeting
- dipco
- icsi
- voxconverse
- chime6
- msdwild
metrics:
- der
model-index:
- name: Audar-Diarization-V1
  results:
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: ami, name: AMI (Headset Mix)}
      metrics: [{type: der, value: 15.24, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: alimeeting, name: AliMeeting (Far)}
      metrics: [{type: der, value: 18.70, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: dipco, name: DiPCo}
      metrics: [{type: der, value: 23.77, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: icsi, name: ICSI}
      metrics: [{type: der, value: 14.46, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: msdwild, name: MSDWild (Few)}
      metrics: [{type: der, value: 21.09, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: msdwild, name: MSDWild (Many)}
      metrics: [{type: der, value: 29.41, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: voxconverse, name: VoxConverse}
      metrics: [{type: der, value: 8.55, name: DER (collar=0.25s)}]
    - task: {type: speaker-diarization, name: Speaker Diarization}
      dataset: {type: chime6, name: CHiME-6}
      metrics: [{type: der, value: 45.00, name: DER (collar=0.25s)}]
---

<div align="center">

# Audar-Diarization-V1

### Real-time streaming speaker diarization — up to 8 speakers, state of the art on 8 corpora.

**From Arabic to the world.**

![License](https://img.shields.io/badge/license-AudarAI%20Community%20v1.0-6f42c1)
![Task](https://img.shields.io/badge/task-speaker%20diarization-blue)
![Params](https://img.shields.io/badge/params-117.7M-f59e0b)
![Speakers](https://img.shields.io/badge/speakers-up%20to%208-f59e0b)
![Streaming](https://img.shields.io/badge/streaming-1.04s%20latency%20%C2%B7%20RTF%200.003-informational)
![DER](https://img.shields.io/badge/8--corpus%20macro%20DER-22.03%25%20%C2%B7%20SOTA-brightgreen)
[![GitHub](https://img.shields.io/badge/GitHub-Audar--diarization--V1-181717?logo=github)](https://github.com/AudarAI/Audar-diarization-V1)

<p><a href="#-what-it-is"><b>🧭 Overview</b></a> · <a href="#-benchmarks"><b>📊 Benchmarks</b></a> · <a href="#-quickstart"><b>⚡ Quickstart</b></a> · <a href="#-real-time-streaming"><b>🎙️ Streaming</b></a> · <a href="#-files"><b>📦 Files</b></a> · <a href="https://github.com/AudarAI/Audar-diarization-V1"><b>🐙 GitHub</b></a> · <a href="https://www.audarai.com"><b>☁️ Audar API</b></a> · <a href="https://www.audarai.com/license/audarai-community-license-v1.0/"><b>📜 License</b></a></p>

</div>

---

## 🧭 What it is

**Audar-Diarization-V1** answers *"who spoke when"* — in real time, for up to **8 speakers**, across
hour-long multi-speaker audio. It is the speaker-attribution engine of the Audar realtime stack: paired
with [**Audar-ASR-V1**](https://huggingface.co/audarai/Audar-ASR-V1-Turbo) it turns a verbatim transcript
into a speaker-labeled one — the difference between an undifferentiated wall of text and a minutes-ready
board record.

It is built on NVIDIA's **Streaming Sortformer v2.1** and advanced in-house through Audar's diarization
program:

- 🔧 **Surgical 4→8 speaker head expansion** — the released Sortformer supports only 4 speaker slots.
  Audar extends the output layer to **8** by modifying exactly two Linear layers (adding just **2,312**
  parameters), cloning the learned 4-speaker weights so the pretrained decision boundary is preserved
  while capacity opens for speakers 5–8.
- 🧊 **Freeze-and-fine-tune** — the **109.55M**-parameter FastConformer acoustic encoder is frozen; only
  the lightweight Transformer encoder + Sortformer assignment modules (**8.15M**) are trained. This buys
  a **4.2-point DER** advantage over full-model fine-tuning and keeps training fast on a single node.
- 🧬 **Correct-by-construction synthetic data** — Audar fixes a systematic bug in the legacy synthetic-
  data generator (97 % of samples had labels running past the audio) and generates **200 h** of clean
  5–8-speaker conversations, on top of **486 h** of real far-field meetings.
- 🎯 **Arrival-Order Speaker Cache (AOSC)** — speakers are assigned to output slots in the order they
  first speak, resolving the permutation problem without global clustering or Hungarian matching. Identity
  is held across sessions **up to 74 minutes**, and the cache auto-sizes to however many speakers are
  actually present.

The result **streams on a single GPU** with **1.04 s** algorithmic latency and a **0.003** real-time
factor (1 s of audio processed in ~3 ms), while posting the **lowest DER of any evaluated system on all
eight benchmark corpora**.

## Model summary

<table>
<tbody>
<tr><td width="220"><b>Model</b></td><td>Audar-Diarization-V1 — streaming speaker diarization (up to 8 speakers)</td></tr>
<tr><td><b>Task</b></td><td>Speaker diarization ("who spoke when") — streaming <i>and</i> offline whole-file</td></tr>
<tr><td><b>Architecture</b></td><td>Sortformer (encoder-label): frozen FastConformer → trainable Transformer encoder → Sortformer modules + AOSC</td></tr>
<tr><td><b>Base</b></td><td>NVIDIA Streaming Sortformer v2.1, surgically extended 4 → 8 speaker slots</td></tr>
<tr><td><b>Total parameters</b></td><td>117,696,272 (117.7M)</td></tr>
<tr><td><b>Trainable / frozen</b></td><td>8.15M trainable · 109.55M frozen (acoustic encoder)</td></tr>
<tr><td><b>Max speakers</b></td><td>8 per session (AOSC auto-sizes to the number present)</td></tr>
<tr><td><b>Prediction frame</b></td><td>80 ms</td></tr>
<tr><td><b>Algorithmic latency</b></td><td>1.04 s (streaming mode)</td></tr>
<tr><td><b>Real-time factor</b></td><td>0.003 (single GPU, batch 1)</td></tr>
<tr><td><b>Sample rate</b></td><td>16 kHz mono</td></tr>
<tr><td><b>Format</b></td><td>safetensors (fp32, lossless) — PyTorch / CUDA via NeMo</td></tr>
<tr><td><b>License</b></td><td>AudarAI Community License v1.0</td></tr>
</tbody>
</table>

## 📊 Benchmarks

Evaluated with **`dscore`** at a **0.25 s collar, ignoring overlap** (DIHARD protocol) on the official
dev/eval splits of **8 corpora** spanning meetings, dinner parties, broadcast, and in-the-wild audio.
Audar-Diarization-V1 posts the **lowest DER on every corpus** and a **macro DER of 22.03 %** — beating
pyannote 3.1 by **7.63 pp** and stock Sortformer v2.1 by **12.16 pp**.

### DER % per corpus (lower is better)

| System | AMI | AliMeeting | DiPCo | ICSI | MSDWild-few | MSDWild-many | VoxConverse | CHiME-6 | **Macro** |
|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|
| **Audar-Diarization-V1** | **15.24** | **18.70** | **23.77** | **14.46** | **21.09** | **29.41** | **8.55** | **45.00** | **22.03** |
| pyannote 3.1 | 28.60 | 27.38 | 30.72 | 22.48 | 27.12 | 34.83 | 12.92 | 53.19 | 29.66 |
| Sortformer v2.1 | 24.84 | 25.94 | 33.80 | 23.22 | 36.92 | 50.77 | 17.06 | 60.97 | 34.19 |

### Where the gain comes from — DER decomposition (macro)

| System | Miss | False alarm | Confusion | **DER** |
|---|--:|--:|--:|--:|
| **Audar-Diarization-V1** | 10.33 | 6.90 | **4.80** | **22.03** |
| pyannote 3.1 | 8.72 | 3.38 | 17.56 | 29.66 |
| Sortformer v2.1 | 11.91 | 5.04 | 17.24 | 34.19 |

The advantage is **confusion: 4.80 % vs 17.2–17.6 %** — a **3.6× reduction**, from the AOSC's stable
identity tracking. The slightly higher false-alarm rate reflects a deliberately assertive streaming VAD
(a missed utterance costs more than a brief false activation in live transcription) and is tunable via
the onset threshold.

### Out-of-domain (CALLHOME, 8 kHz telephony — not in training)

| Audar-Diarization-V1 | Sortformer v2.1 | pyannote 3.1 |
|--:|--:|--:|
| **10.29** | 12.22 | 18.51 |

Identity also holds on the longest sessions in the benchmark — e.g. a 74-minute, 5-speaker ICSI meeting
at 22.2 % DER with ~2 % confusion.

## ⚡ Quickstart

Ships as a single **fp32 safetensors** bundle — `model.safetensors` + `config.yaml` + `load_diarizer.py`.
The loader instantiates the NeMo Sortformer model and loads the weights directly (no `.nemo` tar):

```python
# needs: nemo_toolkit[asr]>=2.6, safetensors, omegaconf
from huggingface_hub import snapshot_download
import sys; sys.path.insert(0, snapshot_download("audarai/Audar-Diarization-V1"))
from load_diarizer import load_diarizer

model = load_diarizer()                                   # fp32, CUDA (device="cpu" also works)
segs  = model.diarize(audio=["meeting.wav"], batch_size=1)
# → RTTM-style [(start_s, end_s, speaker_slot), ...] per file
```

Runnable inference examples (offline diarization + RTTM export), the full 8-corpus benchmark, and
reproduction steps are open at
**[github.com/AudarAI/Audar-diarization-V1](https://github.com/AudarAI/Audar-diarization-V1)**.

## 🎙️ Real-time streaming

The same checkpoint runs **true streaming** via `forward_streaming_step` with persistent `spkcache` / FIFO
state: ~1 s chunks, **80 ms** prediction frames, up to **8 concurrent speakers**, and session-stable slot
labels that never rewrite once committed. Algorithmic latency is **1.04 s** at a **0.003** real-time
factor on a single GPU.

**Speaker-attributed transcription.** The Audar serving gateway runs diarization in parallel with
[**Audar-ASR-V1**](https://huggingface.co/audarai/Audar-ASR-V1-Turbo) and assigns each transcribed word to
the speaker dominant during its time span — so combined latency is the *max* of the two streams, not the
sum. One deployment exposes **ASR-only**, **diarization-only**, and **ASR+diarization** endpoints over HTTP
and an OpenAI-Realtime-compatible WebSocket. For a managed, production-hosted endpoint, see the
[**Audar API**](https://www.audarai.com).

## 📦 Files

| File | What it is |
|---|---|
| `model.safetensors` | **fp32 weights** — bit-exact, lossless, safetensors (safe, zero-copy `mmap`, no pickle) |
| `config.yaml` | model config (the `.nemo`'s `model_config.yaml`) |
| `load_diarizer.py` | self-contained loader (instantiates the NeMo model + loads the weights) |

**Lossless fp32, and faster to load.** The `model.safetensors` carries the full-precision weights
bit-for-bit — verified by a round-trip check (990/990 tensors identical) and by downcasting to the prior
fp16 release with **zero mismatches** across all 971 float tensors, so it reproduces the exact model. The
weight-load step is **~28× faster** than the legacy `.nemo` (≈12 ms `mmap` vs ≈344 ms untar + unpickle),
and `safetensors` is the safe, community-standard format (no arbitrary-code pickle path).

<details>
<summary><b>Re-quantizing to fp16 (optional)</b></summary>

`load_diarizer.py` auto-detects weight dtype, so you can quantize `model.safetensors` to fp16 yourself and
it will load unchanged. When fp16 is detected the loader keeps the preprocessor (STFT/mel) in fp32 and runs
the streaming path under `torch.set_default_dtype(torch.float16)` (NeMo's streaming state is otherwise
created dtype-less). ONNX export is not supported out-of-the-box (NeMo 2.6.2's Sortformer export needs
streaming-state wiring).
</details>

## Intended use & limitations

**Intended use.** Speaker-attributed meeting/broadcast/call-center transcription, board and panel
recordings, and any real-time or offline "who spoke when" task — cloud, on-prem, or edge.

**Limitations.**
- Up to **8 speakers** per session; very large panels beyond 8 are out of scope.
- **Far-field, high-overlap, extreme-noise** audio (e.g. CHiME-6-style dinner parties) remains the hardest
  case for every system.
- Very-low-activity speakers (<2 % of talk time) can have their cache entry decay during long silences.
- Not evaluated for, and must **not** be used for, covert speaker identification.

## 📜 License

Released under the **AudarAI Community License v1.0** — research and limited commercial use for qualifying
Community Entities; enterprise, large-scale, or model-as-a-service use requires an AudarAI Enterprise
License. See
[audarai.com/license/audarai-community-license-v1.0](https://www.audarai.com/license/audarai-community-license-v1.0/),
or contact **contact@audarai.com** for enterprise licensing.

## Citation

```bibtex
@techreport{audar-diarization-v1-2026,
  title       = {Audar-Diarization-V1: Real-Time Streaming Speaker Diarization for Long-Form, Multi-Speaker Audio},
  author      = {Audar AI Team},
  institution = {AudarAI},
  year        = {2026},
  url         = {https://huggingface.co/audarai/Audar-Diarization-V1}
}
```

---

## About AudarAI

<div align="center">

### Leading Arabic-First Multilingual Audio Intelligence

*AudarAI starts with Arabic — and expands to the world.*

</div>

We are building advanced multilingual audio intelligence that helps individuals, enterprises, and
governments communicate across languages, cultures, and borders. By combining Arabic-first speech
technology with global multilingual AI, AudarAI transforms voice into understanding, interaction,
and connection.

Our work spans speech recognition, speech understanding, speaker diarization, voice-enabled digital
assistants, human-computer interaction, and intelligent audio systems designed for real-world impact.
From empowering people to access technology in their native language to helping organizations communicate
globally, AudarAI is shaping a future where every voice can be heard, understood, and connected.

**Arabic-first. Multilingual by design. Human-centered at heart.**

<div align="center">

**[🌐 www.audarai.com](https://www.audarai.com)** · [🤗 Hugging Face](https://huggingface.co/audarai) · [GitHub](https://github.com/AudarAI/Audar-diarization-V1) · contact@audarai.com

© 2026 AUDARAI PTE. LTD. · Licensed under the AudarAI Community License v1.0

</div>