The most common voice agent failure modes, what causes them, and how to fix them fast. Part of the Voice agents section of the TrainYourAgent documentation, written to be executed against rather than skimmed.
If the agent feels sluggish, look at the latency budget layer by layer. Start with speech-to-text first-byte latency, then look at the language model time to first token, then the time to first audio out of the TTS engine. In our experience, the single most common cause of slow agents is a TTS service that is not streaming. Switch to streaming TTS first, then look at the LLM call. If the language model is slow…
This is almost always an orchestrator-level barge-in detection bug. The voice activity detection threshold is too high, or the agent's playback is not being interrupted when speech is detected. Lower the threshold, confirm the orchestrator immediately pauses TTS playback when speech-onset is fired, and add a short cool-down so a quick laugh or a "uh-huh" does not constantly interrupt the agent. Test by deliberately interrupting the agent on every prompt and verifying it yields cleanly.
If the agent consistently mishears a particular product name, model number, or proper noun, that is a transcription accuracy problem at the speech-to-text layer. Most major STT providers support a "hints" or "vocabulary boost" feature where you can pre-load the model with the terms it should expect. Add your top fifty domain-specific terms there. For phone numbers, addresses, and dates, use the STT provider's structured output mode rather than free-form transcription.
If the agent invents information — wrong prices, wrong hours, wrong policy — the cause is almost always that the prompt asks the model to be helpful without giving it a way to answer the question correctly. The fix is a retrieval tool. Move the source of truth out of the prompt and into a tool the model can call. Then add a guardrail: if the tool returns nothing, the model should say it does…
This article is part of the Voice agents section. The most common voice agent failure modes, what causes them, and how to fix them fast.