Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems
To test interview-style chatbots you need many different fake users, so this work has an LLM automatically generate those fake user personalities
Interview dialogue systems that ask users about things like travel plans or dessert preferences are costly to test with real humans. This paper has an LLM generate a large, diverse set of fake user personas from just a handful of hand-written examples, then uses those personas to drive simulated users. Experiments show the resulting simulated conversations are more varied than with a fixed set of manually written personas.
What they did
- The method was tested on two Japanese interview dialogue systems: one interviewing users about travel, one about sweets preferences.
- Starting from 10 hand-written 'seed' personas, GPT-4o generated 100 new personas per condition using in-context few-shot prompting.
- Persona generation also specified two personality traits tied to communication style: degree of anthropomorphism (treating the system as an object vs. as a human) and degree of elaborateness (redundant vs. direct speech).
- Generated personas drove a GPT-4o-based user simulator talking to a GPT-4o-mini-based interview system, and diversity was measured with metrics like utterance length variation and lexical type-token ratio.
- Simply generating new personas with an LLM already increased content diversity (content-word type-token ratio rose from .106 to .122 for travel and .109 to .133 for sweets versus the baseline), and adding the elaborateness trait increased stylistic diversity, with utterance length standard deviation rising from 7.0 to 18.2 for travel and 8.0 to 17.7 for sweets.
| Condition | Personality | #Dialogues | Ave. utterance length (S.D.) | Ave. utterance | length (S.D.) | Total words | Total | words | Unique words | Unique | words | Unique bigrams | Unique | bigrams | TTR |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ave. utterance | |||||||||||||||
| length (S.D.) | |||||||||||||||
| Total | |||||||||||||||
| words | |||||||||||||||
| Unique | |||||||||||||||
| words | |||||||||||||||
| Unique | |||||||||||||||
| bigrams | |||||||||||||||
| BL | 100 | 28.2 | (7.7) | 42,260 | 15,747 | 31,620 | .373 | ||||||||
| noPT | 100 | 28.5 | (7.0) | 42,784 | 16,362 | 32,730 | .382 | ||||||||
| APM | All | 100 | 28.4 | (7.2) | 42,563 | 16,089 | 32,334 | .378 | |||||||
| High | 50 | 30.1 | (7.6) | 22,547 | 8,406 | 17,098 | .373 | ||||||||
| Low | 50 | 26.7 | (6.3) | 20,016 | 7,683 | 15,236 | .384 | ||||||||
| EL | All | 100 | 36.0 | (18.2) | 53,951 | 18,556 | 39,010 | .344 | |||||||
| High | 50 | 50.7 | (13.9) | 38,001 | 12,175 | 26,883 | .320 | ||||||||
| Low | 50 | 21.3 | (5.7) | 15,950 | 6,381 | 12,127 | .400 | ||||||||
| APM+EL | All | 100 | 31.4 | (13.4) | 47,111 | 16,856 | 34,839 | .358 | |||||||
| High+High | 25 | 46.2 | (12.3) | 17,308 | 5,624 | 12,309 | .325 | ||||||||
| High+Low | 25 | 23.6 | (4.8) | 8,836 | 3,451 | 6,787 | .391 | ||||||||
| Low+High | 25 | 35.2 | (9.9) | 13,184 | 4,651 | 9,796 | .353 | ||||||||
| Low+Low | 25 | 20.8 | (5.9) | 7,783 | 3,130 | 5,947 | .402 |
| Condition | Personality | #Dialogues | Ave. utterance length (S.D.) | Ave. utterance | length (S.D.) | Total words | Total | words | Unique words | Unique | words | Unique bigrams | Unique | bigrams | TTR |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ave. utterance | |||||||||||||||
| length (S.D.) | |||||||||||||||
| Total | |||||||||||||||
| words | |||||||||||||||
| Unique | |||||||||||||||
| words | |||||||||||||||
| Unique | |||||||||||||||
| bigrams | |||||||||||||||
| BL | 100 | 25.9 | (7.8) | 25,152 | 11,138 | 20,534 | .443 | ||||||||
| noPT | 100 | 25.6 | (8.0) | 25,509 | 11,105 | 20,634 | .435 | ||||||||
| APM | All | 100 | 25.6 | (8.2) | 25,680 | 11,124 | 20,862 | .433 | |||||||
| High | 50 | 26.5 | (8.2) | 13,269 | 5,723 | 10,824 | .431 | ||||||||
| Low | 50 | 24.6 | (8.0) | 12,411 | 5,401 | 10,038 | .435 | ||||||||
| EL | All | 100 | 33.9 | (17.7) | 34,020 | 13,214 | 26,117 | .388 | |||||||
| High | 50 | 47.5 | (14.8) | 23,755 | 8,578 | 17,808 | .361 | ||||||||
| Low | 50 | 20.3 | (6.1) | 10,265 | 4,636 | 8,309 | .452 | ||||||||
| APM+EL | All | 100 | 28.1 | (12.2) | 28,067 | 11,549 | 22,182 | .411 | |||||||
| High+High | 25 | 39.4 | (13.1) | 9,857 | 3,743 | 7,643 | .380 | ||||||||
| High+Low | 25 | 22.2 | (6.7) | 5,560 | 2,458 | 4,533 | .442 | ||||||||
| Low+High | 25 | 30.6 | (10.5) | 7,642 | 3,087 | 6,015 | .404 | ||||||||
| Low+Low | 25 | 20.0 | (5.9) | 5,008 | 2,261 | 3,991 | .451 |
Why it matters
This lets developers stress-test interview dialogue systems against a wide range of user behaviors without recruiting real human testers, cutting labor and cost. Broader persona diversity increases the chance of catching unexpected system failures before deployment.
Terms in this paper
- persona · a profile of personality, preferences, and speaking style assigned to a simulated user
- interview dialogue system · a conversational AI that asks users questions to collect information from them
- user simulator · a simulated conversational partner that stands in for a real human when testing a dialogue system
- type-token ratio (TTR) · the ratio of unique words to total words used, a measure of vocabulary diversity
- few-shot in-context learning · giving a language model a few examples in the prompt so it produces new outputs in a similar style
Original abstract (English)
This paper addresses the issue of the significant labor required to test interview dialogue systems. While interview dialogue systems are expected to be useful in various scenarios, like other dialogue systems, testing them with human users requires significant effort and cost. Therefore, testing with user simulators can be beneficial. Since most conventional user simulators have been primarily designed for training task-oriented dialogue systems, little attention has been paid to the personas of the simulated users. During development, testing interview dialogue systems requires simulating a wide range of user behaviors, but manually creating a large number of personas is labor-intensive. We propose a method that automatically generates personas for user simulators using a large language model. Furthermore, by assigning personality traits related to communication styles when generating personas, we aim to increase the diversity of communication styles in the user simulator. Experimental results show that the proposed method enables the user simulator to generate utterances with greater variation.
Read on arXivLatest papers
- LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive AlignmentPeeking at a few early training gradients before fine-tuning starts to set up LoRA smarter
- Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right call
- Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy CorrectionWhen text input is missing or broken, this AI doesn't guess once and move on—it revises its guess step by step to read emotions more reliably
- Rethinking Patch Based Multivariate Time Series Forecasting with Semantic Structured PartitioningA new way to slice time series into meaningful chunks instead of arbitrary equal-length pieces
- Reliable Financial Named Entity Recognition under Domain ShiftAn AI's confidence trained on formal filings turns unreliable once it reads tweets
- Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysisA system that makes AI show its work when analyzing brain-imaging data, not just deliver an answer
- GenMatch: An End-to-End Generative Matching Framework for Micro-View Order-Dispatching in Ride-HailingDiDi replaced its multi-step ride-hailing dispatch pipeline with one generative model and saw real-world gains
- Auditing Cross-Lingual Fairness in Language Model WatermarkingAI text watermarks that are supposed to catch machine-written content work far less reliably in many non-English languages, and the gap tracks language families, not individual languages
Latest from METAL LAB
- Google Discover adds chatbot that adjusts your feed based on spoken preferences
- OpenAI Closes In on Anthropic Again in Enterprise Spending Share
- Meta Unveils First 10 Tasks in WildArtifactBench, a Benchmark for AI Agents
- Musk: "Optimus + Grok will one day handle healthcare for all humanity"
- 35% of Web Pages Published Since ChatGPT Show Signs of AI Authorship
Figures: Mikio Nakano et al., arXiv:2608.19549, arxiv-nonexclusive