Lip Sync

Lip sync is a technology that makes a person's mouth movements in a video match their voice perfectly. When you watch a dubbed foreign film, it can feel awkward if the actor's lips are moving in English while the voice you hear is in Korean. Lip sync removes that mismatch, making the mouth shapes look like they're actually producing whatever language the voice has been changed to.

In the past, voice actors would adjust their scripts to fit the original mouth movements while recording. Now, AI analyzes the audio waveform and automatically generates matching mouth-movement video. This technology is used when creating corporate training videos in multiple languages, or when producing multilingual versions of AI avatar videos featuring people. However, lip sync is different from technology that swaps out an entire face. Since it only matches the voice and mouth shape, the face and expressions themselves remain the same as the original.

It's worth noting, though, that as lip sync becomes more sophisticated, the risk of it being used to manipulate footage to make someone appear to say things they never actually said also grows.

As Synthesia expanded its dubbing feature to multiple languages, it described the result as "lip sync matched to the original speaker's mouth movements." In other words, it doesn't just translate and overlay a new voice—it naturally regenerates the mouth shapes for speaking in the new language as well. Since lip sync is a technology that matches mouth shape to sound rather than altering the face itself, it should be understood as distinct from deepfakes, which synthesize an entire face.