Add full model card: metrics, labels, baseline comparison, data licensing, long-file helper

README.md

@@ -5,50 +5,339 @@ tags:
 - audio
 - music
 - speech
+- singing-voice-detection
+- speech-music-discrimination
 - custom-code
 license: mit
+language:
+- en
+- vi
+metrics:
+- accuracy
+- f1
+model-index:
+- name: smad-crnn
+  results:
+  - task:
+      type: audio-classification
+      name: Speech / music / singing discrimination
+    dataset:
+      type: custom
+      name: SMAD synthetic test split (4,000 clips, source-disjoint)
+    metrics:
+    - type: accuracy
+      value: 0.8795
+      name: Accuracy
+    - type: f1
+      value: 0.8791
+      name: Macro F1
 ---
 
-# SMAD CRNN
+# SMAD CRNN — speech / music / singing discrimination
 
-SMAD is an audio classification model for identifying speech, music, singing,
-and non-vocal content in short audio segments. It is designed for lightweight
-audio analysis workflows where fast, practical content categorization is
-needed.
+A 834k-parameter CRNN that labels a 4-second audio clip as **spoken voice over
+noise**, **spoken voice over music**, **sung voice over music**, or **no voice**.
+Trained from scratch on log-mel spectrograms — no pretrained backbone, no
+fine-tuning. Runs on CPU.
 
-## Load by model id
+**▶ [Try it — upload a file, get a colour-coded timeline](https://huggingface.co/spaces/duclvQ/smad-demo)**
+
+The point of the taxonomy: most speech/music segmenters merge "someone is
+talking over a music bed" with "someone is singing", or call singing *music* and
+stop there. SMAD separates those, which is what you need for lyric/dialogue
+routing, dubbing QC, or MV vs. interview classification.
+
+## Labels
+
+| id | label | meaning |
+|----|-------|---------|
+| 0 | `speech_noise` | spoken voice over non-music background (noise, ambience, silence) |
+| 1 | `speech_music` | spoken voice over a music bed |
+| 2 | `singing_music` | sung voice (lyrics) over music |
+| 3 | `none` | no human voice: instrumental music, noise, or silence |
+
+Input: mono, 16 kHz, 4-second windows. Output: 4 logits; divide by
+`config.temperature` (0.7095) before softmax for calibrated probabilities.
+
+## Quick start
 
 ```python
-import torch
+import librosa, torch
 from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
 
 model_id = "duclvQ/smad"
+feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForAudioClassification.from_pretrained(model_id, trust_remote_code=True).eval()
+
+audio, _ = librosa.load("clip.mp3", sr=16000, mono=True)   # mono float32 @ 16 kHz
 
-feature_extractor = AutoFeatureExtractor.from_pretrained(
-    model_id,
-    trust_remote_code=True,
-)
-model = AutoModelForAudioClassification.from_pretrained(
-    model_id,
-    trust_remote_code=True,
-).eval()
-
-# `audio` must be mono float32 audio sampled at 16 kHz. For longer files, run
-# this over 4-second windows.
 inputs = feature_extractor(audio, sampling_rate=16000, return_tensors="pt")
 with torch.no_grad():
     logits = model(**inputs).logits
-    probs = torch.softmax(logits / model.config.temperature, dim=-1)
+probs = torch.softmax(logits / model.config.temperature, dim=-1)[0]
 
-label_id = int(probs.argmax(-1)[0])
-label = model.config.id2label[label_id]
-confidence = float(probs[0, label_id])
+label_id = int(probs.argmax())
+print(model.config.id2label[label_id], float(probs[label_id]))
 ```
 
-For arbitrary file paths, load/resample with `librosa`:
+The feature extractor pads or truncates to exactly 4 seconds. For anything
+longer, use the helper below.
+
+## Audio of any length: sliding window + smoothing
+
+Copy-paste. No dependencies beyond `librosa`, `numpy`, `torch`, `transformers`.
 
 ```python
-import librosa
+import librosa, numpy as np, torch
+
+
+def analyze(path, model, feature_extractor,
+            hop_seconds=1.0, smooth_windows=5, batch_size=64):
+    """Label a file of any length. Returns merged segments:
+    [{"start": 0.0, "end": 12.0, "label": "singing_music", "confidence": 0.93}, ...]
+
+    hop_seconds    step between windows (window itself is fixed at 4 s by training)
+    smooth_windows median filter width over time; 1 disables smoothing
+    """
+    sr = feature_extractor.sampling_rate
+    win = int(model.config.segment_seconds * sr)
+    hop = max(int(hop_seconds * sr), 1)
+
+    audio, _ = librosa.load(path, sr=sr, mono=True)
+    if len(audio) < win:
+        audio = np.pad(audio, (0, win - len(audio)))
+    starts = list(range(0, len(audio) - win + 1, hop))
+
+    probs = []
+    for i in range(0, len(starts), batch_size):
+        batch = [audio[s:s + win] for s in starts[i:i + batch_size]]
+        inputs = feature_extractor(batch, sampling_rate=sr, return_tensors="pt")
+        with torch.no_grad():
+            logits = model(**inputs).logits
+        probs.append(torch.softmax(logits / model.config.temperature, dim=-1))
+    probs = torch.cat(probs).numpy()
+
+    # Median filter across time: kills isolated one-window flips, which are the
+    # dominant error mode on real audio (a single drum fill read as `none`).
+    if smooth_windows > 1:
+        pad = smooth_windows // 2
+        padded = np.pad(probs, ((pad, pad), (0, 0)), mode="edge")
+        probs = np.stack([np.median(padded[i:i + smooth_windows], axis=0)
+                          for i in range(len(probs))])
+
+    # Merge runs of equal labels. A run ends where the next one starts, so the
+    # returned segments tile the audio without overlapping -- consecutive
+    # windows overlap by design whenever hop_seconds < 4.
+    segments = []
+    for start, p in zip(starts, probs):
+        cid = int(p.argmax())
+        if segments and segments[-1]["_id"] == cid:
+            segments[-1]["_conf"].append(float(p[cid]))
+        else:
+            segments.append({"start": start / sr, "_id": cid, "_conf": [float(p[cid])]})
+
+    duration = len(audio) / sr
+    for cur, nxt in zip(segments, segments[1:]):
+        cur["end"] = nxt["start"]
+    segments[-1]["end"] = min(starts[-1] / sr + win / sr, duration)
+
+    return [{"start": round(s["start"], 2), "end": round(s["end"], 2),
+             "label": model.config.id2label[s["_id"]],
+             "confidence": round(float(np.mean(s["_conf"])), 4)}
+            for s in segments]
+
+
+for seg in analyze("song.mp3", model, feature_extractor):
+    print(f"{seg['start']:7.1f} - {seg['end']:7.1f}s  {seg['label']:<14s} {seg['confidence']:.2f}")
+```
+
+Segment boundaries are quantised to `hop_seconds`, and a boundary is only ever
+accurate to within one 4-second window — that window is the model's unit of
+decision, not a frame-accurate onset. Lower `hop_seconds` for finer boundaries
+at linear cost; raise `smooth_windows` if the timeline flickers.
+
+## Results
+
+4,000 held-out clips, 1,000 per class, source-disjoint from training (no
+speaker, song, or noise file appears in both). Chance is 25%.
+
+**Overall accuracy: 87.95%** · macro F1 0.879 · 834,728 parameters
+
+### Per class
+
+| label | precision | recall | F1 | support |
+|---|---|---|---|---|
+| `speech_noise` | 0.950 | 0.957 | **0.954** | 1000 |
+| `speech_music` | 0.951 | 0.942 | **0.946** | 1000 |
+| `none` | 0.782 | 0.867 | **0.823** | 1000 |
+| `singing_music` | 0.841 | 0.752 | **0.794** | 1000 |
+
+### Confusion matrix
+
+Rows = true, columns = predicted.
+
+| | `speech_noise` | `speech_music` | `singing_music` | `none` |
+|---|---|---|---|---|
+| **`speech_noise`** | **957** | 43 | 0 | 0 |
+| **`speech_music`** | 47 | **942** | 11 | 0 |
+| **`singing_music`** | 1 | 6 | **752** | 241 |
+| **`none`** | 2 | 0 | 131 | **867** |
+
+The two speech classes are close to solved. `singing_music` ↔ `none` holds 372
+of the 482 total errors — 77%. Telling a sung voice apart from the instrumental
+track under it is the open problem, not an implementation gap. If your use case
+only needs "is anyone talking", the relevant number is the top-left 2×2 block.
+
+### Accuracy by mixing difficulty
+
+The test mixer records the SNR (or gain, for single-source clips) of each clip:
+
+| bucket | accuracy | n |
+|---|---|---|
+| SNR [15,20] dB | 0.984 | 244 |
+| SNR [5,10) dB | 0.962 | 547 |
+| SNR [0,5) dB | 0.939 | 512 |
+| SNR [10,15) dB | 0.938 | 448 |
+| SNR [-5,0) dB | 0.932 | 249 |
+| gain [0,5] dB (single-source) | 0.768 | 354 |
+| gain [-5,0) dB (single-source) | 0.755 | 318 |
+| gain [-10,-5) dB (single-source) | 0.732 | 328 |
+
+Voice buried at negative SNR is *not* the hard case. Single-source clips are —
+they are the `none` class, where there is no second source to key on.
+
+### Calibration
+
+Trained with label smoothing 0.1 + SpecAugment, then temperature-scaled on the
+validation split (T = 0.7095, fitted — not tuned by hand).
+
+| metric | raw | after temperature scaling |
+|---|---|---|
+| expected calibration error | 0.0404 | **0.0335** |
+| errors made at >0.99 confidence | 0.0% | 1.0% |
+| mean confidence when correct | — | 0.934 |
+| mean confidence when wrong | — | 0.753 |
+
+Why this matters: an earlier transformer trained *without* label smoothing put
+>0.99 confidence on **24.5%** of its own mistakes — its confidence score was
+useless as a filter, which is the first thing anyone actually wants from it.
+This model's score is usable as a threshold. Temperature scaling is monotonic,
+so it never changes which class wins.
+
+## Compared with inaSpeechSegmenter
+
+Same 1,000 clips, drawn from the test split above. The two systems don't share a
+taxonomy — inaSpeechSegmenter tags singing as `music` **by design**, and merges
+speech-over-music with speech-over-noise — so a naive head-to-head would be
+meaningless. Two separate measurements instead:
+
+**Task A — is a spoken voice present?** The question both systems are built to
+answer. Neither is handicapped.
+
+| system | accuracy | precision | recall | F1 |
+|---|---|---|---|---|
+| inaSpeechSegmenter | 0.896 | 0.997 | 0.794 | 0.884 |
+| **SMAD** | **0.996** | 0.998 | **0.994** | **0.996** |
+
+**Task B — is *any* voice present, spoken or sung?** SMAD is built for this;
+inaSpeechSegmenter is built *not* to distinguish it. Reporting B is not a
+criticism of their tool — it quantifies what the extra class buys, which is the
+only reason to carry it.
+
+| system | accuracy | precision | recall | F1 |
+|---|---|---|---|---|
+| inaSpeechSegmenter | 0.648 | 1.000 | 0.531 | 0.693 |
+| **SMAD** | **0.908** | 0.957 | **0.919** | **0.937** |
+
+On the 250 clips that genuinely contain singing, inaSpeechSegmenter labelled 228
+`music`, 14 `noise`, 7 `noEnergy`, and **1** `speech`. SMAD got 188 right.
+
+Size is comparable, not smaller: SMAD is 834,728 parameters (3.3 MB) against
+inaSpeechSegmenter's speech/music/noise CNN at ~789k stored values (3.2 MB). The
+gain here is the taxonomy and the recall, not the footprint.
+
+*Caveat, stated plainly:* the clips come from SMAD's own synthetic mixer, so this
+is home turf. inaSpeechSegmenter was trained on different data for a different
+label set and never saw this distribution. Read this as "what the extra class
+buys on this task", not as a general ranking.
+
+## Training data
+
+40,000 synthetic 4-second mixtures. The classes are defined by *what is mixed
+together*, so segments are synthesised from clean stems rather than scraped from
+labelled corpora — that gives frame-exact labels and direct control over mixing
+SNR. Splits are disjoint at the **source** level (speaker id, song, noise file),
+not the clip level.
+
+| role | corpus | license |
+|---|---|---|
+| speech (EN) | [`openslr/librispeech_asr`](https://huggingface.co/datasets/openslr/librispeech_asr) `train.clean.100` | CC BY 4.0 |
+| speech (VI) | Common Voice VI | CC0 1.0 |
+| speech (VI) | FLEURS VI | CC BY 4.0 |
+| singing + accompaniment | [`danjacobellis/musdb18HQ`](https://huggingface.co/datasets/danjacobellis/musdb18HQ) | **CC BY-NC-SA 4.0 — non-commercial** |
+| instrumental music | [`benjamin-paine/free-music-archive-small`](https://huggingface.co/datasets/benjamin-paine/free-music-archive-small), `instrumental == Yes` | per-track Creative Commons; metadata MIT |
+| noise | [`FluidInference/musan`](https://huggingface.co/datasets/FluidInference/musan) (noise subset only) | CC BY 4.0 |
+| music (VI) | Wikimedia Commons Vietnamese traditional music | CC BY-SA / CC0, per file |
+| real-world | crawled MV / news audio, pseudo-labelled with Silero VAD, **training split only** | no license grant — see below |
+
+**Before you deploy this commercially, read this.** Two ingredients constrain it:
+
+1. **MUSDB18-HQ is CC BY-NC-SA 4.0 — non-commercial research only.** It is the
+   only separated-stem source in the pipeline and supplies both halves of
+   `singing_music`. Whether a model trained on NC-licensed audio is itself
+   encumbered is unsettled and jurisdiction-dependent. Get your own advice.
+2. **The crawled real-world subset carries no license grant.** It went into
+   *training only* — validation and test stay purely synthetic, so every number
+   above remains comparable with every number measured before and after it was
+   added. Each crawled file has a JSON sidecar recording source URL, video id,
+   uploader, license field, the search query that surfaced it, and a sha256, so
+   any source can be audited and removed on request.
+
+The `license: mit` tag on this repo covers the **code and weights as published
+by the author**; it does not and cannot re-license the upstream training corpora.
+
+## Limitations
+
+- **87.95% is an upper bound, not an estimate.** Train and test come from the
+  same synthetic mixer and share its biases: no room impulse responses, one SNR
+  distribution, one pool of speakers and songs. Expect real-world numbers below
+  this. Evaluation on genuine recordings (e.g. MIR-1K) is the honest next step
+  and has not been done.
+- `singing_music` is capped by data, not architecture: only 281 distinct sources
+  exist for that class project-wide, because MUSDB18-HQ has 150 tracks.
+- Speech coverage is English + Vietnamese. Other languages are untested.
+- The 4-second window is fixed by training. Shorter clips are zero-padded;
+  a 4-second window straddling a speech→music cut gets one label for both.
+- Music with heavy vocal-like synths, and speech over strongly rhythmic
+  ambience, are the known confusions beyond the `singing_music`/`none` pair.
+
+## Architecture
+
+`TinyAudioCRNN`: 4 conv blocks (32→64→128→128, BN + ReLU + max-pool) over an
+80-bin log-mel spectrogram, then a 1-layer BiGRU (hidden 128), then mean+max
+pooling over time into a linear classifier. Per-mel-bin input standardisation
+(`feat_mean` / `feat_std`) ships **inside** the weights rather than in a config
+file, so inference physically cannot feed the model a different input scale than
+it trained on.
+
+Features must match training exactly: 80 mels, `n_fft=400` (25 ms),
+`hop_length=160` (10 ms), `power=2.0`, `librosa.power_to_db`. The bundled feature
+extractor does this for you.
+
+Ablations at the same training recipe: transformer (683k params) 82.53%;
+transformer without label smoothing + SpecAugment 80.13%. So the recipe is worth
++2.40 points and the CRNN architecture +5.42.
+
+Selected at epoch 15 of 24 by validation accuracy (90.48%); early-stopped at 23.
+~10 minutes on an RTX A4500.
+
+## Citation
 
-audio, _ = librosa.load("clip.mp3", sr=16000, mono=True)
+```bibtex
+@software{smad_crnn,
+  title  = {SMAD: a small CRNN for speech / music / singing discrimination},
+  author = {duclvQ},
+  year   = {2026},
+  url    = {https://huggingface.co/duclvQ/smad}
+}
 ```