AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

arXiv:2607.261782026-07-27

AI voice assistants should take turns differently depending on the situation, and a small amount of human feedback can teach them how

Full-duplex AI voice assistants that can listen and speak at once currently use one fixed style of turn-taking no matter the situation, which doesn't match how humans actually behave. DuplexGen fixes this by having a language model generate candidate turn-taking moments, then calibrating its predictions against a small set of human annotations collected per scenario. Across six scenarios like tutoring, negotiation, and social chat, the calibrated model matched human turn-taking preferences far better than prompting alone or training on generic conversation data, and a full-duplex model trained on its output showed scenario-specific behavior that human raters preferred.

METAL LAB explanatory visual

How DuplexGen turns plain text dialogue into scenario-aware turn-taking data

Evidence statusMeasured results reported

  1. 1. Spoken-style conversionAn LLM rewrites text dialogue into speech-like transcripts with fillers and disfluencies (e.g., 'you know', 'I, I use it').
  2. 2. Slot identificationCandidate turn-taking points are marked roughly every 8 tokens at clause or sentence boundaries.
  3. 3. Human + LLM annotation5 human annotators (from 248 Prolific participants) and an LLM each label LISTEN / BACKCHANNEL / TAKE_FLOOR preferences at each slot per scenario.
  4. 4. Calibration via fine-tuningAn LLM (Qwen3) is fine-tuned to match its predicted action distribution to the human soft-label distribution, minimizing KL divergence.
  5. 5. Scenario-adapted synthesis & trainingThe calibrated model generates turn-taking dialogues per scenario, which are used to fine-tune a full-duplex model (PersonaPlex) that then shows scenario-specific behavior.
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. DuplexGen converts written dialogue into a spoken-style transcript with fillers and disfluencies, then flags candidate 'slots' in each utterance where a listener could act, roughly every 8 tokens.
  2. At each slot, an LLM predicts a probability distribution over three actions - stay silent (LISTEN), give a short acknowledgment (BACKCHANNEL), or interrupt to speak (TAKE_FLOOR) - and this prediction is fine-tuned to match soft-label distributions gathered from 5 human annotators per slot across 248 Prolific participants.
  3. The framework was applied to six tasks: three cooperative (tutoring, planning, interview) and three competitive (negotiation, persuasion, social chat), built from existing text datasets like MultiWOZ and CraigslistBargain.
  4. Combining fine-tuning on Switchboard human-human speech data with DuplexGen's small calibration set (SWBD+DUPLEXGEN) gave the closest match to human judgment (lowest KL divergence) across most model sizes and scenarios, beating prompting-only and Switchboard-only baselines.
  5. A full-duplex speech model (PersonaPlex) fine-tuned on 1,623 hours of DuplexGen-generated dialogue showed more scenario-varied turn-taking, taking the floor more in competitive scenarios, and was rated higher by human listeners on turn-taking naturalness and by GPT-4.1 on instruction following than the untuned baselines.
Figure 1: Scenario-specific turn-taking behaviors.
Figure 1: Scenario-specific turn-taking behaviors.

Findings

  • Human turn-taking preferences differ significantly across scenarios (chi-square tests, p<0.05) except among competitive scenarios, which share a similar norm; backchanneling is more common in cooperative tasks and floor-taking more common in competitive ones.
  • SWBD+DUPLEXGEN achieved the lowest average KL divergence against human judgments across model sizes (e.g., 0.335 average at 4B, 0.360 at 14B), outperforming prompt-only and Switchboard-only training in most settings.
  • Despite being trained on a much larger corpus (2.4K dialogues vs. DuplexGen's small calibration set), SWBD-ONLY was consistently less aligned with human judgments than DUPLEXGEN-ONLY, tending to overestimate silence and underestimate backchanneling.
  • PP-DG (PersonaPlex fine-tuned on DuplexGen data) showed more scenario-dependent floor-taking and backchannel frequency than Moshi or untuned PersonaPlex, with stronger floor-taking in competitive scenarios (negotiation, persuasion, social chat).
  • In human listening tests, PP-DG scored higher on average Turn-Taking Naturalness (3.69) and Instruction Following (3.55) than PersonaPlex (3.56, 3.41) and Moshi (3.48, 2.61), with pairwise Welch's t-tests showing PP-DG significantly outperforms PP (p=0.034) and Moshi (p=0.0026).
Figure 1: Scenario-specific turn-taking behaviors.
Figure 1: Scenario-specific turn-taking behaviors.

Where it can be used

  • Generating training data for full-duplex voice assistants that need different turn-taking styles for tutoring versus negotiation versus casual chat.
  • Building calibration pipelines that use a small number of human annotations per scenario instead of large-scale generic speech corpora to adapt AI conversational timing.
  • Evaluating existing voice AI systems for whether their interruption and backchannel frequency matches human expectations in a given task context.
Figure 6: Example of human turn-taking annotation. Human participants are asked to annotate the appropriate turn-taking action for each slot in streaming utterance (backchanneling is selected in the previous slot in this example).
Figure 6: Example of human turn-taking annotation. Human participants are asked to annotate the appropriate turn-taking action for each slot in streaming utterance (backchanneling is selected in the previous slot in this example).

Limits and open work

  • The synthesized datasets cover only six scenarios; broader scenario and interaction-type coverage remains untested.
  • Model behavior was found to be highly sensitive to training data and hyperparameters like turn-switching latency and pause duration, so a robust training recipe is not yet established.
  • Calibration is based on text transcripts revealed incrementally and does not directly use prosody, pause duration, intonation, speech rate, overlap, or visual cues, so it's unclear whether text-based preferences match audio-based ones.
  • The downstream listening study evaluates realized audio but does not establish agreement between text-based and audio-based human preference judgments; paired text/audio annotation is left for future work.
  • Two of three full-duplex model comparisons (PP-DG vs. PP, and vs. Moshi) were tested with pairwise significance tests, but broader model coverage and larger-scale multimodal calibration remain future directions.
Figure 7: Example of human turn-taking annotation. Human participants are asked to annotate the appropriate turn-taking action for each slot in streaming utterance (floor-taking is selected in the current slot, truncating the remaining user turn in this example).
Figure 7: Example of human turn-taking annotation. Human participants are asked to annotate the appropriate turn-taking action for each slot in streaming utterance (floor-taking is selected in the current slot, truncating the remaining user turn in this example).

Why it matters

Voice AI systems that can listen and talk simultaneously (full-duplex) currently apply a single, one-size-fits-all timing style regardless of whether they're tutoring a student or negotiating a price, which can feel socially off. This work shows that a modest amount of targeted human feedback per scenario, rather than more raw speech data or cleverer prompting, is what actually teaches models when to jump in versus stay quiet.

Terms in this paper

  • Full-duplex model · An AI voice system that can listen and speak at the same time, rather than waiting for strict turns.
  • Turn-taking · The moment-to-moment decision of who holds the conversational floor - staying silent, giving a brief acknowledgment, or taking over to speak.
  • Backchannel · A short verbal signal like 'uh-huh' that shows attention without claiming the speaking turn.
  • KL divergence · A statistical measure of how different two probability distributions are; lower means the model's predictions match human judgments more closely.
  • Calibration · Adjusting a model's raw predictions using a small set of human-labeled examples so they better reflect real human preferences.

Figures we cannot republish

  • Figure 0
  • Figure 1: Scenario-specific turn-taking behaviors.
  • Figure 4
  • Figure 5
  • Figure 2: Overview of the DUPLEXGEN framework. (1) Spoken-style dialogue conversion: converting text-based dialogue to spoken-style transcript format. (2) Turn-taking slot identification: identifying potential turn-taking slots within utterances. (3) Slot annotation and calibration: using a small set of human and LLM annotations to calibrate turn-taking predictions. (4) Turn-taking dialogue synthe
  • Figure 7
See the figures in the original paper →

Original abstract (English)

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human-human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce Du

Authors · Takyoung Kim

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Takyoung Kim et al., arXiv:2607.26178, CC BY-SA 4.0