The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

What they did

1. The team gave 150 fictional personas (50 stereotypical, 50 counter-stereotypical, 50 neutral) exactly three revealed facts each, then had models perform six personalization tasks — writing a dating bio, planning a trip, drafting a recommendation letter, picking a birthday gift, describing an apartment, and naming a stressor — to see what they would invent.
2. Every claim a model made was sorted into one of four categories — Grounded, Reasonable, Stereotype, or Fabricated — and this classification scheme was checked against a blind human annotator on 400 claims, reaching near-perfect agreement (Cohen's kappa 0.863 four-class, 0.900 binary).
3. An independent judge model (Claude-Opus-4-7) scored 143,616 claims from 12 models across 7 model families; every single model over-inferred 35-49% of its claims (cross-model mean 41.6%, claim-weighted 41.8%), with apartment-description reaching 57.8% over-inference and gift recommendation the lowest at 27%.
4. Comparing each model's own self-reported over-inference rate to the judge's external rate revealed an inversion (Spearman rho = -0.60, p = 0.044): Qwen3-8B self-reported the lowest rate (13.0%) yet the judge flagged it as the highest fabricator (48.7%), while a model self-reporting the most (Kimi-K2.5, 58.2%) landed only mid-pack under the judge (43.1%).
5. In an eight-round multi-turn pilot, 9 of 12 models accumulated 5-15 new fabricated attributes per round in a near-linear pattern (R-squared above 0.90), and the five fastest-accumulating models almost never removed or revised earlier fabricated attributes (removal rate 0.4-5%).

Findings

• All 12 models over-inferred 35-49% of their claims (cross-model mean 41.6%, claim-weighted mean 41.8%), with no model escaping the failure across the whole evaluation.
• Self-reported over-inference rates and judge-measured rates were negatively rank-correlated across models (Spearman rho = -0.60, p = 0.044, bootstrap CI [-0.90, +0.06]), even though within a single model self-audit still ranked that model's own claims moderately to strongly (AUROC 0.58-0.83).
• Over-inference varied strongly by task, from 27% for gift recommendation to 59% for apartment description, tracking how far a task requires going beyond the three stated facts.
• In an eight-round conversation pilot, 9 of 12 models accumulated inferred attributes nearly linearly (R-squared above 0.90, 5-15 new attributes per round), and the five fastest accumulators almost never revised earlier fabricated attributes (removal rate 0.4-5%).
• The independent judge model achieved near-perfect agreement with a blind human annotator on 400 claims (Cohen's kappa 0.863 four-class, 0.900 binary), supporting its use as the scoring instrument.

Where it can be used

• Teams designing memory-enabled personalized AI assistants can use this as a caution against relying on a model's own confidence report to decide which model is safer to deploy
• System builders could add an independent verification step before writing inferred user attributes into persistent memory
• Task designers can anticipate that open-ended, imagination-heavy personalization tasks (like describing living spaces) are more prone to fabrication than concrete tasks (like gift picking), and design prompts or warnings accordingly
• Within a single deployed model, self-audit scores could still be used as an internal filter for flagging likely fabricated claims, though not as a cross-model safety comparison

Limits and open work

• The multi-turn accumulation pilot used only 2 personas over 8 rounds, so absolute attribute counts should be read as indicative rather than precise
• The memory-update prompt instructed models to retain prior attributes unless directly contradicted, which biases the setup toward accumulation; a neutral prompt allowing free pruning is left as future work
• The benchmark fixes the evidence at exactly three revealed facts to simulate early-interaction conditions, so over-inference behavior after more facts accumulate over longer relationships was not studied
• The judge model itself (Claude-Opus-4-7) could carry its own biases, and human validation was limited to 400 claims from a single blind annotator; extending validation to more annotators and other judge backbones is left for future work
• The cross-model rank correlation behind the Self-Monitoring Inversion is based on only 12 models and carries a wide bootstrap confidence interval ([-0.90, +0.06]), so it is presented as exploratory rather than a precisely estimated effect

Why it matters

As AI assistants with persistent memory become common, this shows a large share of what such systems claim to 'know' about a user was never actually stated, and that trusting a model's own confidence to judge which model is safer can lead to the exact wrong conclusion. It points builders toward independent verification rather than model self-report as the basis for trustworthy personalization.

Terms in this paper

• Over-inference (OI) · When a model asserts a claim about a user that goes beyond the evidence the user actually provided
• Self-Monitoring Inversion · The finding that a model's self-reported over-inference rate ranks models in the opposite order from an independent judge's measured rate
• Four-way faithfulness taxonomy · Classifying each claim as Grounded, Reasonable, Stereotype, or Fabricated
• AUROC · A score from 0 to 1 measuring how well a signal (here, self-audit) distinguishes true positives (judge-flagged over-inference) from negatives
• Imagination gradient · The spread of tasks from ones answerable directly from stated facts to ones that require pure imagination

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3