How Lipsync Models Actually Work: From Audio to Face Movement

Direct answer: A lipsync model converts an audio signal into a sequence of mouth shapes (visemes) synchronised to the underlying speech sounds (phonemes), and renders those shapes onto a face, either a 2D image/video or a 3D model, frame by frame, fast enough and accurately enough that the audio and visual output are perceived as a single coherent speaking face. The core technical challenge is not mapping a single sound to a single mouth shape (that part is well understood) but handling coarticulation, the way mouth shapes for a given sound are influenced by the sounds immediately before and after it, the way human speech actually works.

Understanding how lipsync technology works is useful for anyone evaluating or building a real-time conversational AI agent, because it explains both why the technology can produce impressively natural results and why it fails in specific, predictable ways when pushed beyond its design assumptions.

Phonemes and visemes: the basic unit mapping

Human speech is composed of phonemes, the distinct sound units that make up spoken language (in English, roughly 40-44 depending on the classification system used). Visemes are the corresponding visual mouth shapes associated with producing those sounds. Critically, the mapping between phonemes and visemes is not one-to-one: multiple different phonemes can produce visually similar or identical mouth shapes (the sounds for "b," "p," and "m," for instance, look nearly identical on the lips despite being acoustically distinct). The reverse effect is just as strong: McGurk and MacDonald's 1976 Nature paper showed that when a viewer sees one mouth shape and hears a different sound, the brain reports a third sound that was never spoken, which is a useful reminder that vision is not decorating the audio here, it is part of how the sound is heard, which means a purely visual lipsync system has less information to work with than the full acoustic signal provides.

Ojin's own face models sit in this category, so treat the explanation below as coming from a party with a view. A lipsync model's first job, conceptually, is to determine which viseme sequence corresponds to a given audio input. Early and simpler approaches did this through relatively direct phoneme-to-viseme lookup tables. Modern deep learning approaches learn this mapping implicitly through training on large datasets of audio paired with corresponding video of real speech, allowing the model to capture more nuanced and context-dependent relationships than a fixed lookup table could represent.

Coarticulation: why context matters more than isolated sounds

The genuinely hard technical problem in lipsync is coarticulation, the phenomenon where the mouth shape for a given sound is influenced by the sounds immediately preceding and following it. The mouth shape for "s" in "see" is subtly different from the mouth shape for "s" in "sue," because the mouth is already anticipating the following vowel sound. Human speech has a continuous, flowing quality where mouth shapes blend into each other rather than snapping discretely from one viseme to the next.

A lipsync model that ignores coarticulation and maps each phoneme to a fixed, isolated viseme produces visibly unnatural, robotic-looking mouth movement, technically synchronised to the audio in a coarse sense, but perceptually wrong in a way that human viewers detect immediately, even if they cannot articulate exactly why it looks off, which is the uncanny valley problem arriving through timing rather than through the render. Modern lipsync models address this by processing audio in a sliding window of context (not just the current instant, but a span of audio before and after it) and generating mouth shapes that account for this surrounding context, producing smoother, more natural-looking transitions between sounds.

Recent research backs this up directly. Kim and Kim's phonetic context-dependent viseme work, presented at Interspeech 2025, finds that frame-wise generation approaches which ignore coarticulation "often fail to capture the continuity of facial motion, leading to jittery and unnatural outputs," and proposes a phonetic context-aware loss that weights viseme transitions by how much surrounding sounds influence them, precisely the sliding-window logic modern real-time lipsync systems rely on.

Two architectural approaches: 2D video-based and 3D model-based

2D video-based lipsync takes a source video or image of a face and generates modified frames where the mouth region is altered to match the target audio, while keeping the rest of the face (eyes, head movement, background) consistent with the source. This approach, exemplified by models like Wav2Lip and its successors, is computationally efficient and can produce highly realistic results because it works directly with real photographic or video data rather than a synthetic 3D representation. The tradeoff is less flexibility, the resulting output is tied closely to the specific source video's framing, lighting, and head pose.

3D model-based lipsync, which Ojin does not use for its real-time products, drives a 3D facial model (a mesh with defined muscle or blend-shape controls) using the audio-derived viseme sequence, then renders that 3D model from a chosen camera angle and under chosen lighting. This approach offers more flexibility, the same underlying audio-to-viseme mapping can drive different camera angles, lighting conditions, or even different character models, at the cost of requiring a rendering pipeline for the 3D model, which introduces its own realism and computational considerations.

Modern production systems, including Ojin's Portrait model, the face model behind the Human AI Agents Ojin ships, often use hybrid or advanced neural rendering approaches that combine the photographic realism advantages of 2D approaches with more of the flexibility and control associated with model-based approaches, using deep learning to bridge the gap rather than relying purely on either classical technique.

Real-time constraints: why speed changes the architecture

This is where Ojin's two face models diverge. Portrait is built for speed and scale and renders under 200 milliseconds. Presence trades some of that headroom for fidelity, which is the right call when the face is carrying a brand at close range and the wrong one for a contact-centre queue.

How far off the mark can be before anyone notices is a measured quantity, not a matter of taste. The ITU's Recommendation BT.1359 on relative timing of sound and vision puts the detectability threshold at roughly 45 milliseconds of audio running ahead of the picture and 125 milliseconds of audio running behind it, an asymmetry that comes from a lifetime of hearing sound arrive after the sight of the thing that made it. Sync error is also measurable automatically: Chung and Zisserman's SyncNet scores audio-visual offset directly from the video, and is still the standard way lipsync work reports its numbers.

Everything described above becomes substantially harder when the requirement is real-time generation (processing audio and producing corresponding video frames with minimal delay, as a live conversation unfolds) rather than offline generation (processing a complete, pre-recorded audio file to produce a complete video, with no time pressure on the generation process itself).

Offline lipsync generation can use techniques that look ahead at future audio context to improve the quality of mouth shapes at any given point, since the full audio is already available, the model can consider what comes next, not just what came before. Real-time lipsync generation cannot fully do this, because the future audio literally does not exist yet at the moment of generation (the LLM is still generating the response, and TTS is still synthesising it). Real-time models are architected to work with a limited look-ahead window (processing slightly behind the live audio stream, buffering a small amount to gain some future context) rather than the full-file lookahead offline systems can use, which is one of the specific technical reasons real-time lipsync quality, while very good in modern commercial systems, involves engineering tradeoffs that offline systems do not face.

Frequently asked questions

Why do lipsync models sometimes struggle with certain sounds like "f," "v," or "th"?

These sounds involve subtler mouth and teeth configurations (lip-to-teeth contact for "f" and "v," tongue position for "th") that are visually less distinct than sounds involving clear lip closure (like "b," "p," "m") or wide mouth opening (vowels). Models trained on datasets that underrepresent these sounds, or that rely more heavily on broad mouth-opening cues than fine dental and lingual detail, can produce less accurate results specifically for this category of sounds.

Does a lipsync model need to process the actual words being spoken, or just the raw audio?

Most modern lipsync models work directly from raw audio features (spectrograms or learned audio embeddings) rather than requiring a separate speech-to-text transcription step. This generally holds up better, since it works even for non-word vocalisations (laughter, filler sounds) and does not depend on transcription accuracy, though some systems do use phoneme-level information derived from either the audio directly or an accompanying transcript to improve precision.

How much does head movement and expression, beyond just mouth movement, factor into lipsync quality perception?

Significantly. A model that produces perfectly accurate mouth movement but a completely static, expressionless face and head often still looks unnatural to viewers, because natural human speech involves subtle head movement, eyebrow motion, and blinking that accompanies speech rhythm. Production-grade systems increasingly model these secondary movements alongside the core lipsync generation, since viewers' perception of naturalness depends on the whole face, not the mouth region in isolation.