What a real time agent API is, and why latency is the hard part

Direct answer: A real time agent API runs a live session: you stream audio and events in, and a synchronised face and voice stream back turn after turn. Ojin holds the face model to a sub-200ms target and pipelines every other stage around it. Above that rhythm, users feel the lag and stop treating the agent as present.

A real time agent API has one job: answer before the person notices the gap.

Send it audio and events, get back a synchronised face and voice that responds in the same beat as a human conversation. It is not a rendered clip, and not a templated talking head reading a script. What runs instead is a real-time Human AI Agent that listens, reasons and replies while the user is still leaning in, as a live, embodied session rather than a scripted flow. The face model at the centre of it targets sub-200ms latency, which is that model's own figure and not the full round trip. Speech-to-speech and real-time voice APIs cover the audio half of this stack; the difference here is a synchronised face in the same session.

What a real-time agent API actually does

A real-time agent API runs an open session: audio streams in from the user's mic, gets transcribed and routed to your reasoning layer, usually an LLM you already own, and the reply streams back as synchronised audio and an animated face, frame by frame, while the words are still being generated, with the session staying open for the next turn.

By contrast, most "AI avatar" APIs are batch tools in real-time clothing. You submit a script and a voice id, wait, and download a video. That is asynchronous generation: fine for marketing reels, useless for a conversation. Round-trip latency is the metric that matters, because every turn pays it again. Build-vs-buy is therefore sharper here than in most AI tooling, because the pieces look easy to assemble until you measure the millisecond budget.

The millisecond budget comes from human behaviour rather than engineering convenience. Stivers and colleagues published their cross-linguistic study in PNAS in 2009, under the title "Universals and cultural variation in turn-taking in conversation". It measured turn-taking gaps across ten languages and found an average response latency of roughly 200 milliseconds, consistent enough across unrelated cultures that the authors treat it as a universal. Telecoms standards were built around the same instinct: the ITU's Recommendation G.114 on one-way transmission time is the long-standing reference for how much delay an interactive voice connection can carry before the exchange stops feeling conversational. An API that cannot hold that rhythm is speaking a different, slower language than the person on the other end.

How does a conversation move through the pipeline?

A live conversation moves through four real-time subsystems, transport, speech-to-text, reasoning, and voice-and-face rendering, in that order, and each one is a place where latency accrues before the reply reaches the user.

Transport

The session rides a low-latency transport, in practice WebRTC, built for real-time media with jitter buffering and the IETF's congestion control requirements for interactive real-time media. The IETF RTCWeb / WebRTC specifications define the guarantees you lean on, and the browser-side API is standardised in the W3C's WebRTC recommendation, which reached full Recommendation status in March 2025. Bolt this onto plain HTTP streaming and you inherit head-of-line blocking between streams on every turn.

Speech in

Incoming audio is transcribed continuously, with endpointing to decide when a turn is done. Too patient and you add dead air; too eager and you cut the user off. A June 2025 arXiv preprint on streaming endpointing, evaluated on the SpokenWoZ dialogue corpus, treats balancing this latency-accuracy trade-off as the central design problem.

Reasoning

Transcribed text goes to your LLM. Ojin deliberately does not own this layer; your prompts, tools, and retrieval are your moat. The one constraint is that reasoning must stream tokens, because the voice has to start before the full reply exists. This is also where Ojin's architecture diverges hardest from the voice-AI startup field: some voice-only platforms are explicitly aggregators, orchestrating third-party LLMs and TTS providers and billing accordingly, while others bundle STT, LLM, and TTS into one opinionated stack you cannot swap piece by piece. Ojin stays modular by design: bring your own LLM and swap any component without rebuilding the integration.

Voice and face

This is the multimodal core. The voice layer turns streaming reply text into streaming, multilingual audio with voice cloning. Inside this stack, text to speech runs as a streaming component rather than a product you call and wait on, emitting audio chunks as tokens arrive. Oris Portrait renders the face fast and at scale, and Oris Presence is the higher-fidelity tier of the same face-model class, built for maximum expressiveness. Both are face models and nothing more; the bundled product around them is Human Agents, which pairs speech recognition, your reasoning layer, the face model and speech synthesis in one session.

The research line underneath the face layer is public. Prajwal and colleagues presented Wav2Lip at ACM Multimedia 2020 under the title "A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild". Their result was that a dedicated lip-sync discriminator is what pushes generated mouth movement towards professionally synced quality on unconstrained video of arbitrary people. A voice-only real-time API cannot do this, because its pipeline has no face slot to fill.

Where the milliseconds go

A real-time agent API has to fit transport, speech recognition, reasoning time-to-first-token, voice synthesis and lipsync into one budget that the user perceives as a single response. The face model itself is held under 200ms, and every other stage is kept as tight as the pipeline allows. "Real-time" is a marketing word until you account for where those milliseconds go, because it is easy to claim a sub-200ms model and hard to keep every stage in the loop tight.

Ojin's sub-200ms figure is the face model's own latency. The full multimodal round trip, audio in to audio and video out, is a separate and larger number. For scale, the independent benchmarking outfit Artificial Analysis publishes a time-to-first-audio leaderboard for speech-to-speech models. It measures the seconds needed to produce the first token of audio output across the Big Bench Audio question set. On its figures the fastest models sit in the region of 0.44 to 0.70 seconds, and that is audio only, with no face anywhere in the loop. Per-stage p50/p95 figures belong with a published measurement methodology rather than being asserted here.

In production, the budget rarely goes where you expected. It goes on the endpointer waiting an extra beat to be sure the user finished, or on a transport renegotiation over a flaky network.

Why first-chunk time dominates

The budget is shaped less by any model's peak throughput than by time-to-first-token and time-to-first-frame at every stage. A model with great average speed but a slow first chunk blows the budget anyway, because the user is timing the gap before the agent starts, not the length of the answer. Two consequences fall out. First, the stages overlap; you pipeline them, so the voice layer is already speaking the start of the reply while the LLM is still generating the end. Second, the slowest first chunk dominates, so optimising average throughput on an already-fast stage buys nothing.

Serving research points the same way. Kwon and colleagues' PagedAttention paper at SOSP 2023, the work behind vLLM, reports two to four times the throughput of earlier systems at the same latency. Those gains came from memory management in the serving layer, not from a faster model.

Why orchestration is the moat, not any one model

You can find a good face model, a good voice, and a serviceable lipsync model. Keeping three streaming models phase-locked, turn after turn, on real networks, with the face model holding its sub-200ms budget throughout, is a different problem, and it is the one Ojin is built around.

The voice layer and the face model, Oris Portrait or Oris Presence, which render the face and nothing else, cannot drift; if audio leads the lips by a few frames, the uncanny-valley alarm fires and the conversation breaks. Yasu and colleagues, writing in PLOS ONE in 2016 on a sample of eleven listeners, found that the perceptual window is wider for audio lags than for audio leads, so the two error directions are not equally forgiving. Broadcast engineering has had a tolerance standard for decades on the same grounds: the ITU-R's Recommendation BT.1359 on the relative timing of sound and vision exists because audiences detect the mismatch long before they can name it.

The reasoning stream feeds both, so backpressure in one model has to propagate without stalling the others. The multimodal infrastructure problem sits underneath the friendly face, and it also explains the competitive map. Two adjacent fields fall short of it for different reasons. Recorded avatar tools ship pre-rendered or template video rather than live conversation. Voice-only agent platforms ship excellent live voice, and no face slot in the architecture at all. Neither camp ships the full real-time multimodal stack, face plus voice plus presence, as one orchestrated agent. That gap is the position Ojin holds, and it is structural rather than a feature-roadmap item, because adding a face to a voice-only pipeline means rebuilding the orchestration from the ground up.

How should you design against a real-time agent API?

Treat the agent as a stateful session, not a request. Stream everything: your reasoning streams tokens and the API streams audio and frames back. Any stage that waits for a complete payload breaks the budget. Lean on events rather than polling, since turn start, turn end, interruption and barge-in all arrive as webhooks and events. Instrument the critical path from day one, recording per-stage time-to-first-chunk. Model cost per minute of live conversation rather than per request, against the current rates on Ojin's pricing page. The point is that the reasoning layer stays yours. You bring the LLM; Ojin supplies the real-time face, voice and presence around it, plus the orchestration that holds the budget.

The self-serve entry point, sign up and get a live agent embedded the same day, is covered in our self-serve AI platform pillar. If you are still fixing terms, what a Human AI agent is sets out the definitions this page assumes. For the broader pattern of building a conversational AI agent around a reasoning layer you already own, the same session model applies whether or not a face is in the pipeline.

Frequently asked questions

What is a real-time agent API?

An interface for running a live AI agent as an open session. You stream user audio and events in; you get a synchronised speaking face and voice back in real time, with the face model held to a sub-200ms target. It differs from batch video-generation APIs, which render a clip from a script and return it asynchronously.

How is it different from a text to speech API?

A standalone text to speech API converts text to audio and stops. In a real-time agent platform, the TTS step is one streaming component that emits audio as reasoning tokens arrive, synchronised with a face. The voice never runs as an isolated product here.

Why does sub-200ms matter?

Roughly 200ms is the gap human speakers leave between turns. Ojin's sub-200ms figure is a separate measurement: the face model's own latency, not the full round trip, which is a larger number. Holding the face model to that budget forces a pipelined, streaming architecture rather than a sequential one, and the rest of the loop is then tuned around it.

What does Ojin orchestrate?

Two real-time face model tiers plus a bundled voice layer, kept phase-locked: Oris Portrait for speed and scale, Oris Presence for maximum expressiveness, and a voice layer for streaming multilingual speech. Human Agents is the bundle that holds them together with speech recognition and speech synthesis. You supply the reasoning layer.

Can I bring my own LLM?

Yes. The reasoning layer stays yours, along with your prompts, tools and retrieval. Ojin provides the real-time face, voice and presence, plus the orchestration around them.

How is Ojin different from recorded avatar tools or voice-only providers?

Those are pre-rendered, template-based, or voice-only. Ojin ships the full real-time multimodal stack, face plus voice plus presence, as one live agent, and is EU-based in Berlin.

How is Ojin's API different from voice-only agent platforms?

Some voice-only platforms aggregate third-party LLM and TTS providers, which means the rate you end up paying depends on which providers you plug in rather than on the platform's own line item alone. Others bundle STT, LLM and TTS into one opinionated stack you cannot swap piece by piece. Ojin stays modular, so any component can be swapped, and adds a real-time face that neither architecture has a slot for.

How to start building against the API

Wire it up and watch the latency yourself. Build your first agent at ojin.ai/signin and use docs.ojin.ai to guide you. New accounts get $10 in free signup credit, enough to run a live session end to end.

Already running a voice pipeline and only need the face? Oris drops into an existing stack without touching your speech recognition, LLM or speech synthesis, so the orchestration work is the part you are buying.