The Voice AI Stack: Cascaded Pipelines vs Speech-Native Models

Voice is the most natural human interface, and in 2026 it has finally become a first-class way to build AI applications. But "voice AI" is not one thing. Under the hood there are two fundamentally different architectures for making a machine listen, think, and speak. Understanding the trade-off between them is the single most important decision you will make on any voice project, because it shapes your latency, your cost, your control, and the very feel of the conversation.

The cascaded (pipeline) architecture

The classic approach chains three specialized components in sequence. First, speech-to-text (STT), also called automatic speech recognition (ASR), converts the user's incoming audio into text. That text goes to a large language model (LLM) — such as Claude Sonnet 5, GPT-5.5, or Gemini 3.1 Pro — which reasons over the transcript and produces a text reply. Finally, text-to-speech (TTS) synthesizes that reply back into audio the user hears. Audio in, text in the middle, audio out.

This is called a cascade because each stage feeds the next. Its great virtue is modularity and control. You can swap Deepgram for Whisper at the STT stage, change LLMs without touching anything else, and pick a specific TTS voice from ElevenLabs or Cartesia. Every intermediate step is inspectable: you can log the exact transcript, run guardrails on the text, apply retrieval-augmented generation, and call tools deterministically. For regulated domains where every word must be auditable, this transparency is decisive.

The cost of the cascade is latency and information loss. Each stage adds delay, and the stages are somewhat serial. Worse, when speech becomes text, you throw away everything that is not words: tone, emotion, hesitation, sarcasm, emphasis, and the speaker's pace. The LLM never hears that the user sounded frustrated or that they trailed off uncertainly. The reply is generated from a flattened transcript, then re-voiced with prosody the TTS model guesses at.

The speech-native (speech-to-speech) architecture

The newer approach uses a single multimodal model that ingests audio directly and emits audio directly, without a text bottleneck in the middle. The OpenAI Realtime API with GPT Realtime is the best-known example in 2026; Google's Gemini Live API is another. These models were trained on audio, so they perceive and produce the paralinguistic signal — emotion, emphasis, laughter, pacing — that the cascade discards. The result feels dramatically more natural and responds faster, because there is no serial STT to LLM to TTS relay.

Speech-native models excel at expressiveness and low latency. They can react to how something was said, handle interruptions gracefully, and produce speech with genuinely conversational prosody. The trade-off is less control and less transparency. The reasoning happens inside audio-space, so you have weaker guarantees about exactly what text was "understood," tool-calling can be less deterministic, and swapping components is not possible — you take the model as a whole. Costs and available voices are set by the provider.

A practical comparison

The hybrid reality

In practice, many 2026 production systems are hybrids. A common pattern uses a speech-native model for the fast conversational surface while still transcribing audio in parallel for logging, analytics, and compliance. Another pattern uses the cascade but with modern streaming components so tight that the latency gap nearly closes. Frameworks such as Pipecat and LiveKit Agents are designed to let you assemble either architecture, and even switch, without rewriting your application. You should treat the choice as reversible engineering, not a religion.

Why the choice matters

Consider two products. A medical intake assistant must produce an exact, auditable transcript, apply strict guardrails, refuse out-of-scope questions, and call verified tools to look up records. The cascade's transparency and control make it the safer default. Now consider a language-learning tutor whose entire value is natural, expressive back-and-forth conversation that reacts to the learner's tone and hesitation. Here the speech-native model's expressiveness and low latency are the product. Same technology family, opposite correct answers.

What "good" feels like

Regardless of architecture, users judge a voice agent on a few visceral qualities: does it respond fast enough to feel alive (roughly under 800 milliseconds to first sound), can I interrupt it and be heard, does it sound human rather than robotic, and does it actually understand me? Every architectural and engineering decision in this course ultimately serves those four felt qualities. Keep them in mind as your north star.

A note before we build

Throughout this course we will treat voice as a responsible technology. A voice agent records people's speech, which is personal — and sometimes biometric — data. It must disclose that the user is talking to an AI. It must never impersonate a real person without consent. We introduce these obligations now, not as an afterthought, because in voice AI they are load-bearing engineering requirements, not legal footnotes. With the landscape mapped, the next lessons dive into each layer of the stack.

Choosing in practice: a quick decision aid

When a new project lands on your desk, resist the urge to pick an architecture from fashion. Instead, write down four things: the latency your users will tolerate, whether every word must be auditable, how deterministic your tool calls must be, and how much the emotional texture of the conversation matters to the product. If auditability and deterministic tools dominate, lean cascade; if natural, expressive, low-latency conversation is the whole point, lean speech-native; if you are unsure, prototype both behind the same orchestration layer and let measured latency and real user reactions decide. Remember that the decision is rarely permanent — frameworks let you swap or blend architectures — so bias toward the option that lets you ship, learn, and adjust. Above all, keep the four felt qualities in view: fast enough to feel alive, interruptible, human-sounding, and genuinely understanding. Every architectural choice is a means to those ends, and a technically elegant pipeline that fails them is the wrong pipeline. Start from the experience you want the person on the other end to have, and work backward to the stack that delivers it, not the other way around.

Decomposing where the time and money go

To choose an architecture with your eyes open, decompose a single conversational turn into its cost and latency contributors rather than treating "the stack" as a black box. In a cascade, a turn spends time in five places: capturing and buffering audio, running streaming STT to a stable transcript, the LLM producing its first token and then the full reply, TTS synthesizing the first audio chunk, and network transport in both directions. Each of these is independently measurable and independently optimizable — which is precisely the cascade's advantage. In a speech-native model, most of that collapses into a single provider call whose internal breakdown you cannot see; you optimize it by sending less audio, keeping the session warm, and shortening replies, not by tuning individual stages.

Cost follows the same split. A cascade bills you separately for STT (per minute of audio), the LLM (per input and output token), and TTS (per character or per second of audio), plus your own infrastructure. A speech-native model typically bills audio input and audio output tokens at rates well above text tokens, because audio is information-dense. The practical consequence is that speech-native models can be simpler to operate but more expensive per minute of conversation, while a well-tuned cascade gives you more levers to cut cost — for example, routing simple turns to a cheaper LLM.

A capability matrix by use case

Use this matrix to sanity-check an instinct before you commit. It is not a law, but it captures how the trade-offs usually resolve.

A worked decision: outbound appointment reminders

Imagine you must build an agent that calls patients to confirm appointments. Walk the four questions. Latency tolerance: phone conversation, so under roughly 800 ms to first sound matters but is not extreme. Auditability: yes — you need a verbatim record of what was asked and answered, because a missed cancellation has real consequences. Tool determinism: high — the agent must reliably call a scheduling API and record a structured outcome. Emotional texture: low — warmth is nice, but this is a transactional call. Three of four questions point to the cascade, and the fourth is weak. Decision: streaming cascade, with a parallel transcript retained for compliance. Notice that you reached a defensible answer in minutes, and you can write down why — which is exactly what a reviewer or auditor will ask for later.

Common mistakes when choosing

• Choosing by hype. Speech-native models feel magical in a demo, but a demo does not reveal whether your tool calls will fire reliably or whether your compliance team can audit the conversation. Decide from your four requirements, not from the newest release.
• Assuming the choice is permanent. Because Pipecat and LiveKit Agents abstract the transport and the components, you can often prototype both and swap later. Treat the decision as reversible engineering.
• Ignoring the hidden transcript need. Even with a speech-native model, most regulated products still need a transcript for logging and evaluation, which means you are running ASR anyway. Budget for it.
• Forgetting the felt qualities. A technically elegant pipeline that responds in 1.5 seconds, cannot be interrupted, and sounds robotic is the wrong pipeline no matter how clean the diagram looks.

Checklist before you commit to an architecture

• I have written down the latency budget my users will tolerate, in milliseconds.
• I know whether every word must be auditable, and why.
• I know how deterministic my tool calls must be and what happens if one misfires.
• I have decided how much the emotional texture of the conversation matters to the product.
• I have confirmed I can meet my legal obligations (AI disclosure, recording consent, biometric handling) under whichever architecture I pick.
• I have a plan to measure real latency and real user reactions, not just to reason about them.

In practice

Most teams that ship successful voice agents in 2026 start with a streaming cascade because it is transparent, debuggable, and cheap to reason about, then selectively adopt speech-native models on the surfaces where expressiveness is the product. The architecture is a means to the four felt qualities — fast, interruptible, human-sounding, understanding — never an end in itself. Start from the experience you want the person on the other end to have, and let that pull the stack into shape.