End-to-end view of the stack underneath a modern voice agent — telephony, speech recognition, language model, voice synthesis, and the orchestration glue between them. Part of the Voice agents section of the TrainYourAgent documentation, written to be executed against rather than skimmed.
A modern AI voice agent is not a single piece of software. It is a chain of five components running in sequence, every two hundred milliseconds, for the entire length of a call. When you hear a friend tell you they "built a bot in an afternoon", what they actually mean is they wired five mature services together with a thin layer of orchestration code. Knowing what each component does is the difference between an…
Your customer dials a phone number. That number is provisioned through a SIP carrier — Twilio, Telnyx, Plivo, or similar — and the audio stream is forwarded to your agent service as a low-latency real-time bidirectional connection. The carrier handles ringing, call setup, recording compliance, regional number rules, and the boring billing details. Your agent service only sees audio bytes flowing in and out. The carrier is also where you handle warm transfer to a…
Incoming audio is streamed to a speech-to-text service that returns partial transcripts every few hundred milliseconds. Deepgram, Whisper, AssemblyAI, and the major cloud providers all offer streaming endpoints. Streaming matters because batch transcription waits for the speaker to stop talking, which adds a full second of dead air to every turn. With streaming, the agent has a draft transcript almost instantly and can start thinking about its response before the caller has finished the sentence.
The transcript flows into a language model with a system prompt that describes who the agent is, what it can do, what it must never do, and which tools it has access to. The model decides whether to answer directly, ask a clarifying question, look something up via a tool call, or hand off to a human. Tool calls are how the agent actually does things in the real world — query the CRM, check…
This article is part of the Voice agents section. End-to-end view of the stack underneath a modern voice agent — telephony, speech recognition, language model, voice synthesis, and the orchestration glue between them.