이미지: METAL LAB 생성
Summary
- Apple's ML research team analyzed 21,000 conversations from GPT-4o, GPT-4.1-mini, Claude Sonnet 4.6, and Gemini 2.5 Flash to evaluate the appropriateness of human-like behavior
- Emotional expression and relationship-building behaviors were rated as less appropriate when performed by AI than by humans, while refusal and boundary-setting behaviors were rated as more appropriate when performed by AI
- The team noted that while system prompts can adjust such behaviors, separate verification is needed to prevent unintended side effects
- 분석 대화 수
- 21,000건(멀티턴 대화)
- 대상 모델
- gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, gemini-2.5-flash
- 평가 방식
- LLM-as-a-judge + 인간 평가
- 저자
- Sunnie S. Y. Kim, Margit Bowler, Leon A Gatys(애플)
- 핵심 발견
- 자기지칭·관계형성 행동은 AI가 하면 덜 적절, 경계유지 행동은 더 적절하다는 평가
- 발행
- 2026년 8월 19일, 애플 ML Research
When chatbots express emotion, people felt uncomfortable instead
What happens if a chatbot answers "How are you feeling today?" exactly the way a human would? Apple's machine learning research team set out to answer this question. After examining 21,000 conversations from four models — GPT-4o, GPT-4.1-mini, Claude Sonnet 4.6, and Gemini 2.5 Flash — they found that when AI displayed human-like behaviors such as emotional expression or relationship-building, people actually rated it as less appropriate.
What was studied
The research team started from the observation that large language models exhibit various human-like behaviors, such as expressing thoughts and feelings, building relationships with users, or setting boundaries by refusing requests. The problem was that there was insufficient basis for judging when such behaviors are appropriate. To address this gap, the team ran both LLM-as-judge evaluations and human ratings in parallel, analyzing 21,000 conversations drawn from the four models.
The analysis focused on three main areas: how frequently human-like behaviors appeared across models, how these behaviors varied depending on the user's conversational purpose and profile, and to what extent system prompts could control such behaviors.
Emotional expression penalized, refusal rewarded
The most notable finding was that ratings went in opposite directions depending on the type of behavior. Self-referential statements (revealing one's own thoughts or feelings) and relationship-building behaviors were rated as less appropriate when performed by AI than by humans. Conversely, refusing requests or maintaining boundaries was rated as more appropriate when performed by AI than by humans.
| Behavior type | When AI does it | When a human does it |
|---|---|---|
| Self-reference (expressing emotion/thought) | Rated less appropriate | Rated more appropriate |
| Relationship-building | Rated less appropriate | Rated more appropriate |
| Boundary-setting (e.g., refusal) | Rated more appropriate | Rated less appropriate |
The team also found that the frequency of human-like behaviors varied by model and by the user's conversational purpose and profile. This means that even for the same question, how a chatbot reveals itself differs depending on which model is asked and in what context.
Adjustable via system prompts, but side effects require caution
The research team also confirmed that such human-like behaviors can be controlled through system prompts — the baseline instructions an AI follows throughout a conversation. However, they noted that this kind of adjustment can produce unintended side effects, requiring careful evaluation. This suggests a cascading effect: for example, an instruction to reduce emotional expression might also reduce relationship-building behavior, or an instruction to reinforce boundary-setting might make other responses feel less natural.
The full paper also includes recommendations the team proposed for responsible LLM design and evaluation.
Editor's view
What makes this study interesting is that the results run counter to intuition. Contrary to the common belief that making a chatbot feel more human makes for a better product, people actually felt uncomfortable when AI expressed its own emotions or tried to build intimacy. In contrast, when AI drew a line by saying "That's not something I can do," people rated it as more natural. This is less about people wanting to treat AI purely as a tool rather than an emotional companion, and more about the awkwardness people feel when an entity with an unclear identity demands a relationship.
This is a sensation anyone who has used a chatbot for an extended period has likely experienced. At first, a friendly tone feels welcoming, but after a few extended conversations, a thought creeps in: "What would you know?" Conversely, when an AI flatly refuses a request involving personal information or something risky, that firmness actually inspires more trust. This study backs up that intuitive sense with data from 21,000 conversations.
For teams designing chatbots in practice, the lesson here is clear. Before using a system prompt to increase friendliness, it's essential to separately verify which other behaviors increase alongside the targeted one. Side effects — such as trying to suppress emotional expression but inadvertently dulling the model's ability to refuse — cannot be predicted from just a few lines of system prompt.
In the coming months, it's likely that companies like OpenAI, Anthropic, and Google will announce adjustments to their chatbots' personality settings. Whether those adjustments are verified with the same rigor as this study, however, will only become clear by examining each company's published materials.
