Coordinating a real-time voice model with a separate task-execution agent: a practical architecture

Sharing a technical writeup on a problem that comes up whenever you try to combine full-duplex speech with agentic device control: the models good at real-time conversation (parallel-stream architectures like Moshi, dual-tower dialogue modeling from Generative Spoken Dialogue Language Modeling) aren't necessarily the models you want doing complex visual reasoning over long-running tasks.

Our approach: don't force one model to do both. A real-time voice model handles the live exchange. A separate, stronger model handles device-control tasks in the background, visually grounded (screen inspection + keyboard/mouse/touch action). The two coordinate asynchronously through a task queue rather than a shared agent loop.

A few implementation details that might be useful if you're working on something similar:

• Completion vs. success as separate signals. The task queue distinguishes Completed (execution reached its end state) from Failed (an exception interrupted it) — neither implies the intended result was achieved. This lets the foreground conversational layer report outcomes honestly instead of treating a technically-finished-but-unsuccessful task as a success.
• 500ms notification aggregation. When backend results land close together, the window extends and results get combined before delivery to the foreground agent, avoiding a burst of competing responses.
• Runtime state as appended messages, not prompt rewrites. Two runtime-specific message types (State, Notice) get converted to ordinary UserMessage objects at the model boundary — State carries environment changes (device type, app state) without rewriting the system prompt and losing cached prompt material; Notice carries runtime events like loop-detection corrections or backend task results.
• Serial execution by design. Only one device task runs at a time — most tasks need exclusive control of one device, one screen, one input path, and concurrent execution would make that ownership ambiguous.

Full technical writeup with the message-type diagrams and tool design: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/

Anyone here who's tackled the foreground/backend split differently, or hit different failure modes with full-duplex + agentic control combined. What's your interruption-handling approach when the backend is mid-task and a new voice input comes in?