Real-Time AI Agent Architecture, End to End

Direct answer: A real-time AI agent architecture has six stages between the moment a person stops speaking and the moment they see and hear a response: voice activity detection, automatic speech recognition, LLM reasoning, text-to-speech synthesis, lipsync generation, and face rendering. Each stage adds latency. The design target for a conversational agent that feels natural is a total round-trip under 200-300ms, which means every stage has to be individually optimised and, where possible, run in parallel or streamed rather than sequentially.

Building a real-time AI agent is fundamentally a latency budgeting problem before it is a model quality problem. A pipeline that produces perfect answers at 3 seconds of latency will feel worse to a user than a pipeline that produces good answers at 300 milliseconds. Human conversational turn-taking has a natural rhythm, the gap between one person finishing a sentence and the other beginning a response is typically 200-500ms in natural speech. Any AI agent pipeline that exceeds this window starts to feel mechanical, regardless of how good the underlying model is.

This article walks through the six stages of the pipeline, where latency accumulates at each one, and the architectural decisions that determine whether a system hits the sub-200ms target or drifts into the multi-second range that breaks conversational flow.

Stage 1: Voice activity detection and turn-taking

Before any processing of what was said, the system has to determine when the person has finished speaking. This sounds trivial and is not. A voice activity detection (VAD) model that waits too long to determine end-of-turn adds unnecessary latency to every single response. A VAD model that cuts off too early interrupts the user mid-sentence.

The engineering challenge is distinguishing a genuine pause (someone finished their thought) from a mid-sentence pause (someone is thinking, or breathing, or pausing for emphasis). Modern VAD systems use a combination of acoustic silence detection and semantic completion signals, is the sentence structurally complete, does the intonation pattern suggest a question or a continuation, to make this call in real time. This stage typically adds 100-300ms of unavoidable latency, because the system needs some silence window to be confident the turn has ended. Reducing this window too aggressively increases interruption rate; the tradeoff is tuned per use case.

Barge-in handling is the related capability: allowing a user to interrupt the AI agent mid-response, the way humans interrupt each other in natural conversation. A pipeline without barge-in support forces the user to wait for the agent to finish speaking even when they want to redirect the conversation immediately, which feels distinctly unnatural and is one of the fastest ways to make an AI agent feel robotic.

Stage 2: Automatic speech recognition (ASR)

The audio stream is converted to text. Modern streaming ASR systems process audio incrementally as it arrives, rather than waiting for the full utterance to complete before starting transcription. This is a critical architectural choice: a non-streaming ASR system adds the full utterance duration as latency before any downstream processing can begin. A streaming system starts producing partial transcripts within 100-200ms of speech starting, and the downstream LLM stage can begin reasoning on partial transcripts before the utterance is even complete, in systems designed for this.

ASR accuracy matters as much as speed for the overall system quality, a fast but inaccurate transcription produces an LLM response to the wrong input, which is a worse failure mode than a slightly slower but accurate transcription. Domain-specific vocabulary (product names, technical terms, proper nouns specific to the deployment) requires either fine-tuning or a custom vocabulary injection at this stage to avoid transcription errors that cascade through the rest of the pipeline.

Stage 3: LLM reasoning

This is the stage most people think about when they think about "the AI," and it is often the largest and most variable latency contributor. Time-to-first-token (TTFT), how long before the model starts generating its response, is the critical metric here, not total generation time, because the downstream TTS stage can begin as soon as the first few words of the response are available.

Architectural choices that reduce latency at this stage: using a model sized appropriately for the task (a smaller, faster model for simple qualification logic, a larger model reserved for complex reasoning), streaming token generation rather than waiting for the complete response, and caching or pre-computing responses to extremely common queries where appropriate. Context window management also matters, a bloated context with unnecessary conversation history or knowledge base content slows down every single turn of the conversation, not just the first one.

Independent benchmarking backs up why this stage rewards careful model selection rather than defaulting to whatever is largest. Artificial Analysis's ongoing performance leaderboard measures time-to-first-token and output speed across major model providers on a rolling basis, and as of mid-2026 the fastest mainstream APIs (Gemini 2.5 Flash, Claude Haiku 4.5) sit consistently under 600ms TTFT on medium-length prompts, with specialised inference providers pushing meaningfully lower still on selected open models. The gap between the fastest and slowest providers on this single metric is routinely a matter of several hundred milliseconds, which is the entire latency budget for some use cases.

Stage 4: Text-to-speech (TTS) synthesis

The LLM's text response needs to become audio. Streaming TTS architectures generate audio incrementally as text arrives from the LLM, rather than waiting for the complete text response before starting synthesis. This is the single highest-leverage optimisation in the entire pipeline: a non-streaming TTS system adds the LLM's full generation time plus the TTS synthesis time as sequential latency. A properly streamed pipeline overlaps these stages almost entirely, TTS begins synthesising the first sentence while the LLM is still generating the third.

Voice quality and latency are historically in tension: higher-fidelity voice models with more natural prosody have traditionally been slower. Modern real-time TTS systems (including the voice layer inside Ojin's Human Agents stack) are engineered specifically to break this tradeoff, achieving natural-sounding speech at streaming speeds that keep pace with LLM token generation.

Stage 5: Lipsync generation

For agents with a visual face component, the audio needs to be mapped to facial movements, specifically, the mouth and jaw movements (visemes) that correspond to the phonemes being spoken. This stage runs in parallel with, or immediately following, TTS synthesis, and needs to stay tightly synchronised with the audio output. A lipsync system that drifts even 100-150ms out of sync with the audio produces a visibly uncanny effect that undermines the naturalness of the entire interaction, even when the underlying voice quality is excellent.

Stage 6: Face rendering

The final stage renders the visual output, the agent's face, expression, and lipsync-driven mouth movement, as a video stream delivered to the user's device. This stage's latency depends heavily on rendering approach: some architectures render frames on demand per response, others maintain a continuously rendered stream that the lipsync data drives in real time. The rendering approach affects not just latency but also visual consistency across a long conversation.

Why sequential architecture fails and streaming architecture wins

The naive architecture for this pipeline processes each stage sequentially and waits for the previous stage to fully complete: wait for ASR to finish, then send the complete transcript to the LLM, wait for the LLM to finish generating, then send the complete text to TTS, wait for audio to finish, then generate lipsync, then render. Every stage's full latency stacks on top of the previous one. This architecture can easily produce 2-4 second round-trip times, which feels unmistakably slow and mechanical to a user.

The production-grade architecture streams data between stages continuously: ASR produces partial transcripts as speech arrives, the LLM can begin reasoning on stable partial transcripts, TTS begins synthesising as soon as the first sentence of the LLM's response is available, lipsync and rendering process the audio stream as it is produced. This overlapping, streaming design is what compresses the same six-stage pipeline into a 200-300ms experience rather than a multi-second one. The stages are the same; the orchestration is fundamentally different.

Frequently asked questions

What is considered "real-time" for a [conversational AI agent](/insights/conversational-ai-agent)?

There is no universally agreed threshold, but the widely used target in the industry is a round-trip response latency under 200-300ms, which approximates the natural turn-taking gap in human conversation. Systems that exceed 500ms-1 second start to feel noticeably delayed; systems above 2 seconds feel broken for conversational use cases, even if acceptable for asynchronous ones.

Does adding a visual face component to an AI agent significantly increase latency compared to voice-only?

It adds some latency (lipsync generation and rendering), but a well-architected pipeline runs these stages in parallel with audio synthesis rather than after it, so the marginal latency cost of the visual layer is typically in the tens of milliseconds, not hundreds, when properly engineered.

Can this architecture run entirely in the browser, or does it require server-side processing?

Most production real-time AI agent architectures use a hybrid approach: audio capture and playback happen client-side (browser or app), while ASR, LLM reasoning, TTS, and rendering typically run server-side or at the edge, streamed to the client over WebRTC or a similar low-latency transport protocol. Fully client-side inference is emerging for smaller models but is not yet standard for production-grade multimodal agents.

See also: [Real-Time Agent API, the full guide](https://ojin.ai/insights/real-time-agent-api) · Sub-second AI agent conversations · Low-latency AI inference · Pipecat avatar integration · Demo: [docs.ojin.ai](https://docs.ojin.ai)