The Voice AI Stack: Cascaded Pipelines vs Speech-Native Models
From the course Voice AI and Realtime Multimodal Agents
Built-in AI Professor Exclusive
Ask anything about the lesson and get an instant answer. The AI Professor knows the course content and helps you learn more effectively.
Voice is the most natural human interface, and in 2026 it has finally become a first-class way to build AI applications. But "voice AI" is not one thing. Under the hood there are two fundamentally different architectures for making a machine listen, think, and speak. Understanding the trade-off between them is the single most important decision you will make on any voice project, because it shapes your latency, your cost, your control, and the very feel of the conversation.
The cascaded (pipeline) architecture
The classic approach chains three specialized components in sequence. First, speech-to-text (STT), also called automatic speech recognition (ASR), converts the user's incoming audio into text. That text goes to a large language model (LLM) — such as Claude Sonnet 5, GPT-5.6 Sol, or Gemini 3.1 Pro — which reasons over the transcript and produces a text reply. Finally, text-to-speech (TTS) synthesizes that reply back into audio the user hears. Audio in, text in the middle, audio out.
This is called a cascade because each stage feeds the next. Its great virtue is modularity and control. You can swap Deepgram for Whisper at the STT stage, change LLMs without touching anything else, and pick a specific TTS voice from ElevenLabs or Cartesia. Every intermediate step is inspectable: you can log the exact transcript, run guardrails on the text, apply retrieval-augmented generation, and call tools deterministically. For regulated domains where every word must be auditable, this transparency is decisive.
The cost of the cascade is latency and information loss. Each stage adds delay, and the stages are somewhat serial. Worse, when speech becomes text, you throw away everything that is not words: tone, emotion, hesitation, sarcasm, emphasis, and the speaker's pace. The LLM never hears that the user sounded frustrated or that they trailed off uncertainly. The reply is generated from a flattened transcript, then re-voiced with prosody the TTS model guesses at.
The speech-native (speech-to-speech) architecture
The newer approach uses a single multimodal model that ingests audio directly and emits audio directly, without a text bottleneck in the middle. The OpenAI Realtime API with GPT Realtime is the best-known example in 2026; Google's Gemini Live API is another. These models were trained on audio, so they perceive and produce the paralinguistic signal — emotion, emphasis, laughter, pacing — that the cascade discards. The result feels dramatically more natural and responds faster, because there is no serial STT to LLM to TTS relay.
Speech-native models excel at expressiveness and low latency. They can react to how something was said, handle interruptions gracefully, and produce speech with genuinely conversational prosody. The trade-off is less control and less transparency. The reasoning happens inside audio-space, so you have weaker guarantees about exactly what text was "understood," tool-calling can be less deterministic, and swapping components is not possible — you take the model as a whole. Costs and available voices are set by the provider.
Pick up exactly where you left off
Create your free account in under a minute, then pick the option that fits you best:
What's next in this lesson
- A practical comparison
- The hybrid reality
- Why the choice matters
- What "good" feels like
- A note before we build
- Choosing in practice: a quick decision aid
- Decomposing where the time and money go
- A capability matrix by use case
Everything you'll learn in this course
1 Foundations of Voice AI in 2026 3 lessons
- The Voice AI Stack: Cascaded Pipelines vs Speech-Native Models Reading now 52 min
- Conversation Anatomy: Latency, Turn-Taking, and Endpointing 51 min
- Consent, Privacy, and AI Transparency: The Legal Foundation 50 min
2 Speech-to-Text: Modern ASR 3 lessons
- How Modern Speech Recognition Works: Whisper, Deepgram, and the ASR Landscape 51 min
- Streaming Transcription: Partial Results, Interim Hypotheses, and Endpointing 50 min
- ASR Accuracy: WER, Diarization, and Handling Real-World Audio 50 min
3 Text-to-Speech: Neural Voices 3 lessons
- Neural TTS in 2026: ElevenLabs, Cartesia, and Hosted Voices 51 min
- Streaming TTS: Latency, Chunking, and Voice Consistency 50 min
- Voice Cloning: Ethics, Consent, and the Law 49 min
4 The Realtime API and Speech-Native Models 3 lessons
- Speech-to-Speech Models: The OpenAI Realtime API and GPT Realtime 51 min
- Bidirectional Streaming: WebSockets, WebRTC, and Audio Framing 51 min
- Sessions, Events, and Managing Realtime State 50 min
5 Building a Voice Agent: Orchestration 4 lessons
- Orchestration Frameworks: Pipecat and LiveKit Agents 51 min
- Turn Detection, VAD, and Barge-In in Practice 51 min
- The Latency Budget: Engineering Natural Conversation 50 min
- Conversation Design and System Prompts for Voice Agents 50 min
6 Function Calling and Multimodal Voice 3 lessons
- Tool and Function Calling in Voice Agents 51 min
- Multimodal Agents: Combining Audio, Text, and Vision 50 min
- Realtime Video and Vision Agents 49 min
7 Telephony and SIP for Voice Agents 3 lessons
- Connecting Voice Agents to the Phone Network: Twilio and SIP 51 min
- Media Streams, DTMF, and Call Control 50 min
- Building a Production Phone Agent 50 min
8 Evaluation, Testing, and Safety 3 lessons
- Evaluating Voice Agents: Metrics and Methods 50 min
- Testing, Simulation, and Regression 50 min
- Safety, Guardrails, and Responsible Voice AI 51 min
9 Deployment, Cost, and Case Studies 4 lessons
- Deployment Architecture and Scaling 50 min
- Cost Modeling and Optimization 50 min
- Case Studies: Real-World Voice Agents 49 min
- Observability, Monitoring, and Incident Response for Voice Agents 50 min
10 Final Quiz — Voice AI and Realtime Multimodal Agents 1 lessons
- Final Assessment — Voice AI and Realtime Multimodal Agents 52 min
Everything you need to learn effectively
Interactive quizzes
Check your knowledge at the end of every lesson with scored quizzes and feedback.
Personal notes
Save notes on every lesson, accessible anytime from your dashboard.
Scheduled reviews
Revisit lessons exactly when it matters, at the right intervals — so you remember for the long term.
Progress & Achievements
Track your progress, unlock achievements, and visualize what you've learned.
Bookmarks
Save the lessons that matter and find them instantly when you need them.
Questions & Answers
Ask questions right on the lesson and get answers from our team.
Good to know before you start
How do I get access to the course?
You can read the beginning of the first lesson for free, right on this page. For the course you create an account, pick the subscription that fits — a single course or a bundle — and get access immediately after your payment is confirmed. Everything happens 100% online.
Can I cancel my subscription anytime?
Yes. Cancel anytime, straight from your account, in just a few clicks. Your access stays active until the end of the period you have already paid for.
What does the subscription for this course include?
All 30 lessons in the course, interactive quizzes, the AI professor built into every lesson (select any passage and it explains it on the spot), personal notes, automatically saved progress, and content updates included.
Is there a fixed learning schedule?
No. You learn at your own pace, on any device. Lessons are structured step by step, and the platform saves your progress automatically, so you can pick up right where you left off — anytime.
Ready to unlock all the content?
Just this course — €99 / month, VAT included — or every IT Pro course, with smart quizzes and the full AI Professor, in the bundle at €399 / month, VAT included.
