Why do lipsync models sometimes struggle with sounds like f, v, or th?

These sounds involve subtler mouth and teeth configurations that are visually less distinct than sounds with clear lip closure or wide mouth opening. Models underrepresenting these sounds in training data produce less accurate results for them.

Does a lipsync model need transcribed text, or does it work from raw audio?

Most modern lipsync models work directly from raw audio features rather than requiring transcription, which is more robust since it handles non-word vocalisations and doesn't depend on transcription accuracy.

How Lipsync Models Actually Work: From Audio to Face Movement

Direct answer: A lipsync model turns audio into mouth shapes, then draws those shapes onto a face frame by frame. The mouth shapes are called visemes and the speech sounds they correspond to are called phonemes. Done fast enough and accurately enough, the sound and the picture read as one speaking face rather than two things happening at once. The hard part is not matching one sound to one shape. That is well understood. The hard part is coarticulation: the way a mouth shape changes depending on the sounds either side of it, which is how speech actually works.

Knowing how this works helps if you are buying or building a real-time conversational AI agent. It explains why the output can look so natural, and why it fails in the specific ways it does.

Phonemes and visemes: the basic unit mapping

Human speech is composed of phonemes, the distinct sound units that make up spoken language (in English, roughly 40-44 depending on the classification system used). Visemes are the corresponding visual mouth shapes associated with producing those sounds. Critically, the mapping between phonemes and visemes is not one-to-one: multiple different phonemes can produce visually similar or identical mouth shapes (the sounds for "b," "p," and "m," for instance, look nearly identical on the lips despite being acoustically distinct). It runs the other way too. McGurk and MacDonald showed in 1976 that if you see one mouth shape and hear a different sound, your brain reports a third sound that nobody made.

So the face is not decorating the audio. It is part of how the sound is heard.

Ojin's own face models sit in this category, so treat the explanation below as coming from a party with a view. A lipsync model's first job, conceptually, is to determine which viseme sequence corresponds to a given audio input. Early and simpler approaches did this through relatively direct phoneme-to-viseme lookup tables. Modern deep learning approaches learn this mapping implicitly through training on large datasets of audio paired with corresponding video of real speech, allowing the model to capture more nuanced and context-dependent relationships than a fixed lookup table could represent.

Coarticulation: why context matters more than isolated sounds

The genuinely hard technical problem in lipsync is coarticulation, the phenomenon where the mouth shape for a given sound is influenced by the sounds immediately preceding and following it. The mouth shape for "s" in "see" is subtly different from the mouth shape for "s" in "sue," because the mouth is already anticipating the following vowel sound. Human speech has a continuous, flowing quality where mouth shapes blend into each other rather than snapping discretely from one viseme to the next.

Map each sound to one fixed mouth shape and you get robotic movement. It is synchronised in a crude sense and it looks wrong, and viewers spot it instantly without being able to say why. That is the uncanny valley problem arriving through timing rather than through the render. Modern models handle it with a sliding window. Instead of looking only at the current instant, they take a span of audio either side and shape the mouth accordingly. The transitions come out smoother.

Recent research backs this up. Kim and Kim, at Interspeech 2025, found that models generating each frame in isolation "often fail to capture the continuity of facial motion, leading to jittery and unnatural outputs".

Their fix weights the transitions between mouth shapes by how much the surrounding sounds affect them. That is the same sliding-window logic real-time systems use.

Two architectural approaches: 2D video-based and 3D model-based

2D video-based lipsync starts from a real photo or video and redraws the mouth to match the audio. Eyes, head movement and background stay as they were in the source. This approach, exemplified by models like Wav2Lip and its successors, is computationally efficient and can produce highly realistic results because it works directly with real photographic or video data rather than a synthetic 3D representation. The tradeoff is less flexibility, the resulting output is tied closely to the specific source video's framing, lighting, and head pose.

3D model-based lipsync, which Ojin does not use for real-time products, drives a 3D face mesh from the same viseme sequence, then renders it from whatever camera angle and lighting you choose. It is more flexible, since one audio-to-viseme mapping can drive several camera angles or characters. The cost is a rendering pipeline, with its own realism and compute problems.

Most production systems now sit between the two. Ojin's Human Portrait model, the face behind the Human AI Agents it ships, uses neural rendering to get the photographic realism of the 2D approach with more of the control of the model-based one.

Real-time constraints: why speed changes the architecture

This is where Ojin's two face models diverge. Human Portrait is built for speed and scale and renders under 200 milliseconds. Human Presence trades some of that headroom for fidelity, which is the right call when the face is carrying a brand at close range and the wrong one for a contact-centre queue.

How far off the mark can be before anyone notices is a measured quantity, not a matter of taste. The ITU's Recommendation BT.1359 puts the threshold at roughly 45 milliseconds if the audio runs ahead of the picture, and 125 milliseconds if it runs behind. The asymmetry comes from a lifetime of hearing sound arrive after the sight of whatever made it. Sync error is also measurable automatically: Chung and Zisserman's SyncNet scores audio-visual offset directly from the video, and is still the standard way lipsync work reports its numbers.

All of this gets harder in real time. Offline generation takes a finished audio file and produces a finished video with no clock running. Real-time generation has to produce frames as the conversation happens.

An offline model can look ahead, because the whole audio file already exists. It knows what comes next, not just what came before. Real-time lipsync generation cannot fully do this, because the future audio literally does not exist yet at the moment of generation (the LLM is still generating the response, and TTS is still synthesising it). Real-time models buffer a fraction of a second and run slightly behind the live audio, which buys a little future context. Offline systems can look at the whole file. That gap is why real-time lipsync involves tradeoffs an offline renderer never has to make.

What this means when you are comparing vendors

Four things separate a lipsync system that holds up in conversation from one that only looks good

in a reel.

Ask what the look-ahead window is. Every real-time system buffers something. A vendor who

cannot tell you roughly how much is not measuring it.

Watch the transitions, not the frames. A still frame from almost any modern system looks

fine. The failures live in the movement between sounds, which is where coarticulation either was

or was not handled.

Test a language the demo did not use. Viseme behaviour differs across languages, and a model

tuned on English can produce visibly worse mouth movement in German or Arabic.

Push it with fast speech. Rapid delivery compresses the time available per viseme, and

systems that look fine at a measured pace fall apart when someone talks quickly.

Ojin's Human Portrait model is built for the volume end of this and Human Presence for the close-scrutiny

end, and both should be judged on the four tests above rather than on a showreel.

Why this is harder than text to speech

Speech synthesis has a single output stream and one quality bar: does it sound right. Lipsync has

two streams that must agree with each other, and a second bar on top: does the picture match the

sound at every instant.

That difference is why a face pipeline cannot simply be bolted onto a working voice pipeline. The

timing contract runs between the two, so adding a face means rebuilding the orchestration rather

than adding a component. It is also why voice-only platforms have no face option: the gap is

architectural, not a missing feature.

Most of the above is invisible in a demo, which is the practical problem.

A showreel is rendered offline, from a finished audio file, with unlimited time per frame. Every

constraint described here is absent. A demo that looks perfect tells you almost nothing about how

the same system behaves mid-conversation on a home broadband connection.

So ask for a live session rather than a video, on a normal connection, in a language other than

English, with someone speaking quickly. Those four conditions surface almost every weakness this

article describes, and they cost nothing to arrange.

A note on the word "realistic"

Vendors use realistic to mean two different things, and buyers should separate them.

One is fidelity: how convincing a single frame looks, judged as a photograph. The other is

plausibility of movement: whether the face behaves the way a face behaves while speaking. A

system can be excellent at the first and poor at the second, and that combination is the most

unsettling of all, because the closer the render gets the more the timing errors stand out.

Judge the movement, not the frame.

Frequently asked questions

Why do lipsync models sometimes struggle with certain sounds like "f," "v," or "th"?

These sounds involve subtler mouth and teeth configurations (lip-to-teeth contact for "f" and "v," tongue position for "th") that are visually less distinct than sounds involving clear lip closure (like "b," "p," "m") or wide mouth opening (vowels). Models trained on datasets that underrepresent these sounds, or that rely more heavily on broad mouth-opening cues than fine dental and lingual detail, can produce less accurate results specifically for this category of sounds.

Does a lipsync model need to process the actual words being spoken, or just the raw audio?

Most modern lipsync models work directly from raw audio features (spectrograms or learned audio embeddings) rather than requiring a separate speech-to-text transcription step. That holds up better. It handles laughter and filler sounds, and it does not inherit transcription errors. Some systems still use phoneme-level information, taken from the audio or a transcript, to sharpen the result.

How much does head movement and expression, beyond just mouth movement, factor into lipsync quality perception?

Significantly. A model that produces perfectly accurate mouth movement but a completely static, expressionless face and head often still looks unnatural to viewers, because natural human speech involves subtle head movement, eyebrow motion, and blinking that accompanies speech rhythm. Production-grade systems increasingly model these secondary movements alongside the core lipsync generation, since viewers' perception of naturalness depends on the whole face, not the mouth region in isolation.