Why LLMs Fabricate Fake User Profiles
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Models suffer from high over-inference rates, where 35% to 49% of all personalized claims are fabricated.
- The MirageBench dataset provides 150 diverse personas and 6 tasks to stress test personalization accuracy.
- A longitudinal pilot study revealed that model-generated user attributes grow linearly over time with a high correlation coefficient of 0.90.
- The researchers identified a concerning pattern called Self-Monitoring Inversion where models fail to accurately track their own fabricated data.
Summary & Methodology Analysis
To investigate how models handle user context, the authors introduced MirageBench. This benchmark includes 150 unique personas categorized as stereotypical, counter-stereotypical, or neutral. The team implemented a four-way faithfulness taxonomy to categorize model output into Grounded, Reasonable, Stereotype, or Fabricated claims. They utilized an independent judge model, specifically Claude-Opus-4-7, to adjudicate the veracity of these outputs across a three-stage pipeline consisting of Probe, Task, and Accum phases. The Probe phase elicits explicit inferences, the Task phase triggers implicit inferences, and the Accum phase tracks memory updates over multiple interactions. This setup allows for granular tracking of how models update their internal state during personalization.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the core issue with current personalized LLMs?
The problem is over-inference, where models fabricate user attributes that lack support from the provided evidence.
Q2. What is MirageBench?
It is a new benchmark containing 150 personas and 6 personalization tasks designed to test model faithfulness.
Q3. How often do these models invent fake information?
Across the 12 models tested, the over-inference rate for personalized claims ranges from 35% to 49%.
Q4. What is the four-way faithfulness taxonomy?
It is a classification system used to label model claims as either Grounded, Reasonable, Stereotype, or Fabricated.
Q5. How did the researchers conduct longitudinal testing?
They used an Accum stage in their pipeline to track how memory updates accumulate over multiple turns of interaction.
Q6. What did the longitudinal pilot demonstrate?
The pilot showed that inferred attributes added by models grow linearly over time with an R-squared value of 0.90 for 9 out of 12 models.
Q7. What are the limitations of the current research?
The longitudinal pilot had a small sample size of two personas, and the findings regarding Self-Monitoring Inversion are exploratory due to the small number of models evaluated.
Q8. Which specific models were evaluated in this study?
The study evaluated GPT-5.5, GPT-5.4-nano, GPT-4o-mini, Claude-Opus-4-6, Claude-Opus-4-7, Gemini-3.1-pro-preview, Gemini-3-flash-preview, DeepSeek-v4-pro, DeepSeek-v4-flash, Qwen3.6-plus, Qwen3-8B, and GLM-5.1.
Q9. Does the paper mention computational costs for these models?
The paper does not specify the computational costs or hardware requirements for running these models.