Natural speech is only the visible edge. A production voice agent must also handle latency, interruption, telephony state, business tools and a graceful human transfer.
Optimise the complete conversation loop, including the moment automation should stop.
Design the conversation as a state machine
A call is not a stream of independent prompts. It has a beginning, identity and consent checks, a goal, tool calls, confirmation points, possible interruptions and a definite end state.
Make those states explicit. The model can decide how to phrase a response, but deterministic call state should decide whether audio is playing, a tool is pending, the caller interrupted or a transfer has started.
Treat latency as part of the experience
Users notice the whole turn: voice activity detection, transcription, reasoning, tool execution and speech generation. Improving only model response time rarely fixes an awkward call.
- Stream audio and partial results where the technology allows it.
- Keep common lookups fast and make longer work audible and honest.
- Support barge-in so callers can correct or redirect the agent naturally.
- Never let filler language imply that an action succeeded before confirmation exists.
Put business tools behind narrow contracts
A voice agent should not receive broad database or browser access. Give it small tools such as find an order, read an appointment, propose a reschedule or request a transfer. Each tool should validate identity, permission and required fields independently of the model.
For payments, account changes, regulated advice and other sensitive actions, use explicit confirmation and human approval. A confident voice must never be mistaken for authority.
Make transfer a first-class success path
Human handover is part of the product, not an error fallback. Transfer the caller with a concise summary, the verified identity context, the attempted actions and the reason for escalation.
Test with noise, accents, interruptions, silence, tool timeouts, repeated questions and callers who change their mind. A production-ready agent is one that fails safely and preserves the conversation context.
