Tencent Releases an Open-Source Model That Generates and Edits Speech

What is worth noticing here is not a leaderboard position but the fact that the entry point has been reduced to one. Until now, audio work meant swapping tools. Reading a sentence aloud called for a speech synthesizer, stripping out noise called for a restoration model, and pulling a voice away from its backing track called for a separation model. AuK handles all of it in the same place, given only a written instruction and a piece of reference audio. On its official account, the company introduced the model as "nano banana for audio."

There are 16 capabilities attached to it, grouped into five families. The first is speech generation: zero-shot synthesis that hears a new voice briefly and imitates it, and synthesis where an instruction sets the delivery. The second is content editing, swapping out what a recording says or rewriting lyrics. The third is acoustic editing of pitch, speed and volume. The fourth is paralinguistic editing, which changes emotion and timbre, removes an accent, strips out sounds that are not words, and turns ordinary speech into a whisper. The fifth covers noise removal, speaker separation, music separation and target speaker extraction.

The scale of the training is written out in the technical report. There are 3.03 billion instances pairing an instruction with audio, and 1.95 million hours of supervision actually used in training. The researchers say in the technical report that they built this material themselves across the five task families described above. Thirty-three authors are named on the report.

The architecture stitches together three pieces. A multimodal language model sits where the meaning of speech is handled, and a VAE trained jointly on human speech, general audio and music sits where the texture of sound is handled. Above them, a hybrid rectified-flow Transformer runs through blocks that split into two streams and then merge into one to produce sound. The training order was also split: generation was learned first on its own, then generation and editing were learned together, and the final stage attached preference optimization based on human feedback to the editing side and reward-based reinforcement learning to the generation side.

AuK-Flash, the lightweight version released alongside it, may matter more in practical work. It is a distilled model that answers in four steps, and it drops the classifier-free guidance step normally used to clean up results. Measured under matched conditions, the researchers say wall-clock time came out about 4.5 times faster than the full model. When the wait on a single edit shrinks, the working style of rerunning something until it feels right becomes possible.

The report also records where the company did not come out ahead. The researchers claim leading performance on zero-shot synthesis, instruction-based generation and instruction-guided editing, while saying only that the model "remains competitive" on signal-level restoration tasks. In the traditional territory of removing noise and reviving broken sound, it has not yet passed the dedicated models built to do only that.

For people building things, the condition that changes is the license. MIT allows commercial use and permits modified redistribution, so the model can be dropped straight into a video editing tool or a podcast service. The researchers wrote in the report that they "release both the source code and model weights to support reproducibility and further research," and the weights are posted on both Hugging Face and ModelScope. For teams weighing what to put in the slot where they used to book a voice actor or rent a studio, there is now one more option worth testing.

Another company in the same market took the opposite road. METAL reported earlier that Suno retrained v6 on licensed catalogs from Warner and BMG, and if that route led to a closed service with the rights cleared, this release sits at the other end, with the weights left open. In technology that handles the human voice, the line between rights and openness keeps getting sharper.

The release sequence was short. The technical report went up on arXiv on September 8, and two days later the company's official account announced that the code, the weights and a demo were all live and asked people to try it and send feedback. It is common to publish a paper first and release weights much later; here all three arrived in the same week. For anyone using it, that means the performance claims can be checked by running the model on their own data rather than only reading about them. The worth of a model released as open source is decided less by the numbers in the announcement than by how quickly that check can happen.

The official introduction video METAL reviewed runs one minute and 53 seconds and carries no speech, only images in sequence. It shows, one after another, how the sound changes when an instruction is entered. The repository had drawn 218 stars shortly after release.

Tencent Hunyuan has put out a model that merges making and fixing a voice behind one interface, with the weights opened as well. The weight of the announcement lies less in what becomes newly possible than in the fact that work previously done by chaining several tools together can now be handled in one place with a written instruction. Two things are worth watching next: whether the gap with dedicated restoration models narrows in the next version, and when a service actually ships with these MIT-licensed weights inside it.