---
title: Contextual VAD Trainer
colorFrom: blue
colorTo: green
sdk: gradio
app_file: app.py
pinned: false
license: mit
---

# Contextual VAD

Small CPU-friendly classifier for improving VAD events on top of an STT + LLM + TTS voice-agent pipeline.

## Live Playground

The Space includes a browser playground for quick event testing:

- record from the browser microphone
- upload an audio file
- optionally type the STT text you want the model to see
- set assistant/TTS state to test interruption behavior
- inspect a 100ms frame-level event timeline

The playground does not run a full STT model on Free CPU. Audio drives VAD/timing features, while the optional transcript box simulates the STT partial/stable text features used by the classifier.

## How It Works

Raw acoustic VAD is fast, but it only answers "is there speech-like audio right now?" A voice agent needs richer decisions:

- did the user start speaking?
- did the user finish their turn?
- is a silence just a pause?
- is the user interrupting the assistant?
- is the user only backchanneling with "yeah", "mhm", or "right"?
- is this a false VAD trigger from echo or noise?

`contextual-vad` wraps a streaming voice pipeline:

```text
audio VAD + streaming STT partials + assistant/TTS state + dialogue context
  -> small tabular ML classifier
  -> event probabilities
  -> deterministic turn-taking policy
```

The model consumes derived frame-level features such as:

- VAD probability, speech duration, silence duration, and energy
- STT confidence, stable transcript length, partial transcript length, and word counts
- semantic hints such as continuation endings and whether required slots are filled
- assistant/TTS state, including whether the assistant is speaking and estimated echo risk
- dialogue context such as expected answer type

It predicts probabilities for:

- `listening`
- `speech_started`
- `endpoint_candidate`
- `turn_committed`
- `user_resumed`
- `interruption_started`
- `interruption_confirmed`
- `backchannel_detected`
- `false_alarm`

The Space trains a bootstrap model from synthetic event rows by default. Upload a CSV with the same feature schema and an `event_label` column to train on real call logs. Each training run pushes both the model artifact and the `training_data.csv` file used for that artifact to the model repo.

## Research Reference

This implementation is inspired by Voice Activity Projection (VAP):

Erik Ekstedt and Gabriel Skantze, "Voice Activity Projection: Self-supervised Learning of Turn-taking Events", Interspeech 2022.

- Paper: https://arxiv.org/abs/2205.09812
- PDF: https://arxiv.org/pdf/2205.09812
- Reference implementation: https://github.com/ErikEkstedt/VoiceActivityProjection

VAP's key idea is to predict future voice activity and turn-taking events instead of relying only on current speech/non-speech detection. This Space applies that design direction to a practical STT + LLM + TTS wrapper: the current model uses lightweight tabular features rather than raw-audio transformers, but the event taxonomy follows the same spirit of distinguishing holds, shifts, backchannels, and true interruptions.