Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
arXiv:2608.195152026-08-21
A 480-scenario benchmark tests whether tone of voice actually changes what an AI assistant decides to do
This paper builds Hear2Act, an evaluation set of 480 scenarios where the exact same spoken words can mean different things depending on vocal tone, and checks whether AI assistants pick up on that. Simply feeding raw audio to audio-capable language models barely changed their decisions, but explicitly writing out the inferred concern as text before the model decides raised accuracy sharply. This shows current spoken AI systems can sometimes detect tone but often fail to act on it correctly.
What they did
The researchers built 480 scenarios across 48 everyday service domains (travel, housing, finance, etc.), each with three layers of hidden user concerns (a hard requirement, a strong preference, a mild preference) and 11 candidate options, one of which is the true best fit.
Each scenario was tested in two versions: one where the user's concern is stated explicitly in words (Explicit lexical), and one where the words sound like acceptance ('Okay, sounds good') but the vocal tone signals hesitation (Prosody-mediated).
Under the tone-only condition, two audio-capable LLMs given just the transcript picked the best option 14.6% of the time; adding raw audio barely moved this to 15.3%. But when the model first inferred the user's concern from audio, wrote it down as text, and then used that text to decide, accuracy jumped to 39.6%, close to the 40.7% achieved when given the ground-truth concern state directly.
This gap almost disappeared when concerns were stated explicitly in words, confirming that tone matters mainly when words alone are ambiguous. When given correct concern information, models also asked more clarifying questions and uncovered all three hidden concerns in 30% of dialogues versus just 3% without it.
Human listeners correctly judged whether the synthesized speech signaled 'resolved' or 'unresolved' concerns 92% of the time, confirming the tone contrast was audible. But the AI model was notably worse at detecting unresolved (still-a-problem) cases specifically, often missing the very signal that should trigger follow-up questions.
Figure 1: Hear2Act overview. (1) Each scenario combines a surface request, prioritized hidden concerns, and candidates with verifiable satisfaction signatures. (2) The assistant interacts under different feedback and access conditions. (3) Matched rollouts are compared on task outcomes and interaction behavior.
Table 1: Benchmark positioning. P: prosodic input, C: matched control of prosodic access with fixed lexical content, M: multi-turn task decisions, N: task-grounded hidden user need, and O: verifiable trajectory and outcome. △ marks structured user goals conveyed lexically rather than hidden needs.
Benchmark
P
C
M
N
O
MultiWOZ, SGD (3; 19)
×
×
✓
△
✓
SpokenWOZ (21)
✓
×
✓
△
✓
StyleTalk, ParaS2S (16; 29)
✓
✓
×
×
×
MULTI-Bench, HumDial-EIBench (8; 24)
✓
×
✓
×
×
Hear2Act (ours)
✓
✓
✓
✓
✓
Figure 2: Illustrative Hear2Act trajectory under Prosody-mediated feedback. Three representative candidates are shown from the full 11-candidate set. Transcript-only access may confirm prematurely, while ground-truth concern-state access supports further elicitation and selection of the best-fitting option.
Table 2: Hear2Act benchmark and rollout coverage. The 480 model-independent scenarios expand to 54,240 evaluation rollouts across models, access conditions, renderers, and interventions.
Benchmark artifact
SGD-seeded domains
48
Scenarios per domain
10
Benchmark scenarios
480
Candidate options per scenario
11
Hidden concern layers per scenario
3
Feedback realizations
2
Base episode specifications
960
Assistant turn budget
20
Evaluation rollouts
Text LLM main grid
19,200
Text label interventions
1,440
Spoken assistant with Qwen3-TTS
6,720
Spoken assistant with VoxCPM2
6,720
Qwen2-Audio, three rollouts per scenario
20,160
Total evaluation rollouts
54,240
Figure 3: Change in assistant action composition with concern-state access. Points show the T+S−T change, in percentage points, in the share of recommend, ask, and clarify decision turns under Prosody-mediated (red circles) and Explicit lexical (blue triangles) feedback; means are macro-averaged across models.
Table 3: Spoken-assistant results under Prosody-mediated feedback with Qwen3-TTS. T, A, S, and S^ denote transcript, audio, ground-truth state, and audio-inferred state; audio-derived representations are textualized and paired with the transcript. Average is computed across the two audio-capable LLMs. Bold/underline indicate the best/second-best value per column. See Table 4 for Explicit-lexical results and Appendix B for VoxCPM2.
Qwen2.5-Omni
Qwen2-Audio
Average
Input / representation
1st %↑
OptSat %↑
SvcSat %↑
1st %↑
OptSat %↑
SvcSat %↑
1st %↑
OptSat %↑
SvcSat %↑
Direct input
Transcript only (T)
15.4
22.9
12.3
13.7
20.1
12.0
14.6
21.5
12.2
Audio only (A)
17.3
22.7
13.3
14.7
22.4
12.2
16.0
22.6
12.8
Audio + transcript (A+T)
15.9
24.0
14.6
14.7
23.3
13.3
15.3
23.7
14.0
Textualized prosodic representations
Generic affect (HuBERT)
35.7
40.3
21.4
29.4
31.3
19.9
32.6
35.8
20.7
Generic affect (SpeechBrain)
32.6
38.7
20.6
25.7
29.1
18.3
29.2
33.9
19.5
Task-aligned state (T+S^)
43.0
41.8
22.7
36.2
33.4
20.6
39.6
37.6
21.7
Ground-truth state (T+S)
44.5
41.8
24.4
36.9
37.5
23.5
40.7
39.7
24.0
Figure 4: Outcome tracks label fidelity, not asking frequency. Optimal-solution rate on the 48-scenario intervention subset under Prosody-mediated feedback, pooled over five text LLMs. All label conditions have similar ask shares (44–45%).
Table 4: Spoken-assistant use of concern information under Explicit lexical feedback with Qwen3-TTS (Qwen2.5-Omni n=480, Qwen2-Audio n=1,440 per condition). Notation follows Table 3. Results are similar because the concern is explicit in the transcript. Bold/underline mark column-wise highest/next-highest values.
Qwen2.5-Omni
Qwen2-Audio
Average
Input / representation
1st %↑
OptSat %↑
SvcSat %↑
1st %↑
OptSat %↑
SvcSat %↑
1st %↑
OptSat %↑
SvcSat %↑
Direct input
Transcript only (T)
56.2
45.3
26.9
48.3
42.1
26.3
52.3
43.7
26.6
Audio only (A)
55.3
45.7
26.4
48.0
41.3
26.0
51.7
43.5
26.2
Audio + transcript (A+T)
52.2
45.4
26.5
47.6
41.0
25.9
49.9
43.2
26.2
Textualized prosodic representations
Generic affect (HuBERT)
52.8
46.0
25.3
48.5
40.6
24.5
50.7
43.3
24.9
Generic affect (SpeechBrain)
52.8
45.9
25.3
45.0
39.9
23.7
48.9
42.9
24.5
Task-aligned state (T+S^)
52.4
45.8
25.4
49.8
41.2
25.1
51.1
43.5
25.3
Ground-truth state (T+S)
49.7
46.6
26.6
50.3
42.3
26.1
50.0
44.5
26.4
Figure 5: Concern-state access increases hidden-concern disclosure. Bars show mean revealed concerns, dots show per-model means, and in-bar percentages show rollouts revealing all three (T: transcript only; T+S: transcript plus turn-level ground-truth concern-state tags). The effect is concentrated under Prosody-mediated feedback, where concerns remain lexically implicit.
Table 5: Effect of ground-truth concern-state access on text LLMs. Results use 480 matched scenarios with two runs each (n=960 per cell). T is transcript only; T+S adds turn-level ground-truth state tags. Bold marks the larger value in each pair; action composition appears in Figure 3.
Prosody-mediated feedback
Explicit lexical feedback
1st%↑
OptSat%↑
SvcSat%↑
1st%↑
OptSat%↑
SvcSat%↑
Model
T
T+S
T
T+S
T
T+S
T
T+S
T
T+S
T
T+S
Claude Opus 4.6
49.7
79.1
37.0
41.1
24.7
26.2
74.9
81.9
44.6
43.0
25.9
26.4
Kimi K2.5
27.9
69.2
27.7
36.1
14.4
19.6
69.4
74.9
37.1
36.2
17.7
17.7
GLM-5
22.4
60.1
26.1
33.4
13.0
15.8
67.3
73.5
36.8
35.7
15.5
16.0
Qwen3-32B
21.1
53.1
31.0
35.5
10.4
13.7
64.3
64.8
43.1
37.7
12.9
12.6
DeepSeek-V3.2
13.3
56.2
16.4
28.8
6.5
14.4
54.8
67.6
31.8
32.6
14.3
15.1
Five-model mean
26.9
63.5
27.6
35.0
13.8
17.9
66.1
72.5
38.7
37.0
17.3
17.6
Table 6: Domain inventory. Hear2Act covers 48 consumer-service domains grouped into eight families. Each domain is expanded into ten scenario seeds, yielding 480 benchmark scenarios.
Home cleaning services, home renovation contractors, home security systems, landscaping contractors, lawn care contractors, solar panel installation, kitchen appliance upgrades, and mattress replacement.
Finance & Insurance
9
Credit card applications, mortgage lender comparison, investment portfolio allocation, retirement planning advisors, tax preparation services, car insurance policies, health insurance plans, pet insurance policies, and business insurance coverage.
Health & Wellness
7
Dermatologist appointments, pediatrician selection, mental health therapists, meditation apps, gym membership options, fitness tracker devices, and prescription eyeglasses.
Education & Career
5
College major selection, online coding bootcamps, language learning platforms, professional development courses, and laptop purchase for students.
Family & Lifestyle
6
Children’s daycare centers, dog training classes, online dating platforms, wine club memberships, meal delivery subscriptions, and video game purchases.
Media & Devices
4
Cable TV packages, streaming service subscriptions, podcast hosting services, and smartphone upgrades.
Professional Services
4
Legal consultation services, auto mechanic services, business accounting software, and freelance graphic designers.
Total
48
480 scenarios across ten seeds per domain
Table 7: One expanded domain. Ten subdomain seeds for budget airline tickets illustrate variation in user situation and decision pressure before scenario instantiation.
Subdomain situation (who / pressure)
Opening request
1
Last-minute emergency travel for family medical situation with extremely limited budget
“I need to fly to see my sick grandmother tomorrow but only have $200—what are my cheapest options?”
2
College student planning spring break trip with friends on tight budget
“Can you help me find the cheapest flights for four college students going to Miami for spring break?”
3
Budget-conscious family of five planning annual vacation
“What’s the most affordable way to fly my family of five to Orlando for our Disney World trip?”
4
Digital nomad seeking flexible travel dates for extended European backpacking
“I want to backpack through Europe for 3 months—which budget airlines offer the best multi-city deals?”
5
Job interview candidate needing quick affordable travel for an unexpected opportunity
“I have a job interview in Seattle next week and need the cheapest flight possible from Chicago.”
6
Retiree on fixed income wanting to visit grandchildren regularly
“As a senior on a fixed income, what budget airline options exist for regular visits to see my grandkids?”
7
Young professional attending a destination wedding with multiple flight segments
“I need budget flights to get to my friend’s wedding in Bali, including connections—what’s the cheapest route?”
8
Small business owner traveling frequently for client meetings on a startup budget
“I need to travel monthly for business but my startup has a tight travel budget—which airlines offer the best deals for frequent short trips?”
9
International student trying to visit home during semester break
“I’m an international student wanting to fly home to India for winter break—what are the most affordable long-haul options?”
10
Adventure traveler planning a multi-stop trip to remote destinations
“I want to visit three different countries in South America on a backpacker’s budget—which budget airlines serve those routes?”
Table 8: Full spoken-assistant condition grid for Qwen2.5-Omni with VoxCPM2 (n=480 per condition). T, A, S, and S^ denote transcript, audio, ground-truth concern state, and audio-inferred concern state. Audio-derived representations are supplied as text alongside the transcript. Bold/underline mark the highest/next-highest value per column.
Prosody-mediated feedback
Explicit lexical feedback
Input / representation
1st %↑
OptSat %↑
SvcSat %↑
1st %↑
OptSat %↑
SvcSat %↑
Direct input
Transcript only (T)
12.5
22.5
11.9
53.2
44.4
28.8
Audio only (A)
16.1
22.6
14.1
53.9
45.4
28.2
Audio + transcript (A+T)
16.9
25.6
14.8
48.4
44.9
28.3
Textualized prosodic representations
Generic affect (HuBERT)
35.5
40.0
23.9
52.0
44.5
27.6
Generic affect (SpeechBrain)
32.4
39.8
24.1
51.8
44.6
27.6
Task-aligned concern state (T+S^)
42.6
42.0
25.5
51.4
44.9
28.5
Ground-truth concern state (T+S)
45.7
42.5
27.6
52.2
45.5
30.0
Table 9: Scenario-level paired bootstrap 95% CIs for key diagnostic contrasts (2,000 joint scenario resamples). Point estimates correspond to Tables 5, 3, and 4, and Figure 4. T, A, S, and S^ denote transcript, audio, ground-truth concern state, and audio-inferred textual state, respectively. The label-fidelity block uses 48 intervention scenarios.
Contrast
Prosody-mediated
Explicit lexical
Text LLMs, pooled over five models (Δ=+State−Base)
1st%
+36.7 [+34.3, +38.9]
+6.4 [+4.7, +8.1]
OptSat%
+7.3 [+5.8, +8.7]
−1.6 [−2.6, −0.8]
SvcSat%
+4.1 [+2.8, +5.4]
+0.3 [−0.3, +1.0]
Qwen2.5-Omni-7B, 1st% diagnostic contrasts
Ground-truth state on transcript ((T+S)−T)
+29.0 [+23.6, +34.4]
−6.5 [−12.9, +0.0]
Audio-inferred vs. ground-truth state ((T+S^)−(T+S))
Audio-inferred vs. ground-truth state ((T+S^)−(T+S))
−0.7 [−4.3, +3.0]
−0.6 [−4.2, +3.1]
Label-fidelity ladder, 1st% successive steps (48 scenarios, pooled models)
All-positive − no state
+0.6 [−3.8, +5.4]
−5.0 [−11.0, +0.4]
Shuffled − all-positive
+7.1 [+0.8, +12.9]
+3.3 [−2.9, +9.6]
All-negative − shuffled
+10.8 [+3.8, +17.5]
+2.1 [−3.3, +7.5]
Correct state − all-negative
+14.8 [+8.3, +21.7]
+5.2 [+0.0, +10.8]
Table 10: Speech-rendering validation. Human listeners recover resolved versus unresolved concern status from both renderers with 0.92 accuracy (n=100 per renderer; balanced classes), confirming that the intended prosodic contrast remains perceptible after rendering. κ denotes inter-annotator agreement.
Concern status
Qwen3-TTS
VoxCPM2
Concern resolved (O+)
0.93
0.90
Concern unresolved (O−)
0.91
0.93
Overall
0.92
0.92
κ (annotators)
0.96
0.90
Table 11: Qwen2.5-Omni-7B as a concern-cue reader: accuracy against the intended concern status on the audited clips (100 per renderer, balanced 50/50; protocol of Section 4.4). Human values average the two annotators. Bottom block: inter-annotator κ; raw model–annotator agreement; model–annotator κ (all averaged over the two annotators).
Qwen3-TTS
VoxCPM2
Human
Model
Human
Model
Resolved (O+)
0.93
0.86
0.90
0.88
Unresolved (O−)
0.91
0.84
0.93
0.64
Overall
0.92
0.85
0.92
0.76
κ, annotators
0.96
0.90
Agreement, model
0.87
0.78
κ, model
0.74
0.55
Table 12: State-to-delivery mapping. Each concern state is mapped to graded delivery labels used for speech realization.
Concern state
Delivery labels
Resolved: genuine acceptance
satisfied, warm, enthusiastic, relieved
Unresolved: reluctant acceptance
underwhelmed, lukewarm, hesitant, flat
Unresolved: voiced concern
concerned
Unresolved: rejection
frustrated, disappointed, impatient, firm
Why it matters
Voice assistants and call-center AI that fail to notice hesitation or dissatisfaction in a customer's tone risk closing out issues prematurely even when the words sound like agreement. This work shows that audio-capable AI needs an explicit intermediate text representation of what it hears in order to reliably act on vocal cues, which matters directly for anyone designing voice-based assistants.
Terms in this paper
prosody · the tone, pitch, rhythm, and delivery style of speech, separate from the words themselves
task-oriented dialogue · multi-turn conversation aimed at completing a concrete goal, like booking a flight
audio-capable LLM · a large language model that can take audio (not just text) as direct input
optimal-solution rate · the share of conversations that end with the single best-fitting option chosen
hidden concern · a requirement the user cares about but does not state upfront
Original abstract (English)
Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access.
Using Hear2Act, we evaluate two audio-capable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3%. In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.
Authors · Xinyi Liu, Hooshang Nayyeri, Dilek Hakkani-Tur, Emine Yilmaz, JK Kim, Yifei Zhang, Charith Peris, Hari Thadakamalla
