Cross-lingual voice transfer

Think of a dubbing studio. If an actor performed a scene in Korean and you want that same voice to speak an English script, you'd normally have to cast a different voice actor. Cross-lingual voice transfer skips that step: instead of hiring a new voice actor, it extracts the original speaker's tone and vocal color and applies them to sentences in another language. It separates the content of speech (the language) from the identity of the voice (the speaker), keeping the speaker fixed while swapping out only the language.

This matters because in multilingual content, the same person needs to sound the same across every language they speak. If you're making a video where one presenter speaks Korean, English, and Japanese, and a different person dubs each language, the character's consistency breaks down. Cross-lingual voice transfer solves this by anchoring on the original speaker's voice and swapping only the language layer.

Because this technology directly manipulates the vocal characteristics of a real person, using someone's voice as a source without their consent immediately raises legal and ethical problems. The fact that the technology is now possible doesn't mean it's permissible to use.

audio.cpp 0.5 was introduced as adding cross-lingual voice transfer through a model called Confucius4. A common misunderstanding here is mistaking this for a translation feature. In reality, the language of the sentence is already fixed (translation is a separate step); the technology is closer to reading that sentence aloud in the original speaker's vocal characteristics. It's used to maintain speaker consistency in multilingual dubbing.