README.md

metadata

title: Contextual VAD Trainer
colorFrom: blue
colorTo: green
sdk: gradio
app_file: app.py
pinned: false
license: mit

Contextual VAD

Small CPU-friendly classifier for improving VAD events on top of an STT + LLM + TTS voice-agent pipeline.

Live Playground

The Space includes a browser playground for quick event testing:

• record from the browser microphone
• upload an audio file
• optionally type the STT text you want the model to see
• set assistant/TTS state to test interruption behavior
• inspect a 100ms frame-level event timeline

The playground does not run a full STT model on Free CPU. Audio drives VAD/timing features, while the optional transcript box simulates the STT partial/stable text features used by the classifier.

How It Works

Raw acoustic VAD is fast, but it only answers "is there speech-like audio right now?" A voice agent needs richer decisions:

• did the user start speaking?
• did the user finish their turn?
• is a silence just a pause?
• is the user interrupting the assistant?
• is the user only backchanneling with "yeah", "mhm", or "right"?
• is this a false VAD trigger from echo or noise?

contextual-vad wraps a streaming voice pipeline:

audio VAD + streaming STT partials + assistant/TTS state + dialogue context
  -> small tabular ML classifier
  -> event probabilities
  -> deterministic turn-taking policy

The model consumes derived frame-level features such as:

• VAD probability, speech duration, silence duration, and energy
• STT confidence, stable transcript length, partial transcript length, and word counts
• semantic hints such as continuation endings and whether required slots are filled
• assistant/TTS state, including whether the assistant is speaking and estimated echo risk
• dialogue context such as expected answer type

It predicts probabilities for:

• listening
• speech_started
• endpoint_candidate
• turn_committed
• user_resumed
• interruption_started
• interruption_confirmed
• backchannel_detected
• false_alarm

The Space trains a bootstrap model from synthetic event rows by default. Upload a CSV with the same feature schema and an event_label column to train on real call logs. Each training run pushes both the model artifact and the training_data.csv file used for that artifact to the model repo.

Research Reference

This implementation is inspired by Voice Activity Projection (VAP):

Erik Ekstedt and Gabriel Skantze, "Voice Activity Projection: Self-supervised Learning of Turn-taking Events", Interspeech 2022.

• Paper: https://arxiv.org/abs/2205.09812
• PDF: https://arxiv.org/pdf/2205.09812
• Reference implementation: https://github.com/ErikEkstedt/VoiceActivityProjection

VAP's key idea is to predict future voice activity and turn-taking events instead of relying only on current speech/non-speech detection. This Space applies that design direction to a practical STT + LLM + TTS wrapper: the current model uses lightweight tabular features rather than raw-audio transformers, but the event taxonomy follows the same spirit of distinguishing holds, shifts, backchannels, and true interruptions.