Evaluating LLMs as Mobile Personal Assistants
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- SPIEval tests LLMs on five cognitive capabilities including reasoning, disambiguation, integration, preference inference, and multi-intent decomposition.
- The benchmark covers 250 tasks grounded in 4,335 personal records across 10 simulated mobile applications.
- Performance gaps are significant, with the best model achieving 57.3% accuracy compared to 16.4% for the weakest.
- Data retrieval is a major bottleneck, as 79% of task failures stem from inaccurate information localization.
Summary & Methodology Analysis
The SPIEval benchmark addresses the challenge of deploying LLMs as mobile assistants that must aggregate scattered personal information. The methodology constructs a unified user profile and simulates 10 mobile applications (Accommodation, Alarm, Contacts, Meeting, Notes, Schedule, SMS, Transactions, Transport, and Voicemail) with structured schemas. To interact with these apps, the researchers designed 21 tools: 11 retrieval tools that support substring, regex, and fuzzy search, alongside 10 execution tools for task completion. Each of the 250 tasks was manually created and verified by at least two researchers to ensure the ground-truth solutions are unique and human-verifiable.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary purpose of SPIEval?
It serves as a benchmark to evaluate LLM performance when executing instructions that require leveraging personal information scattered across multiple mobile applications.
Q2. What kind of tasks does the benchmark cover?
It covers 250 tasks requiring five cognitive capabilities: reasoning, disambiguation, integration, preference inference, and multi-intent decomposition.
Q3. How do researchers determine if a model succeeds?
They use a binary accuracy metric based on the model's final execution tool calls compared to human-annotated gold answers.
Q4. What role does information retrieval play in task failure?
Inaccurate information localization is the primary cause of failure, accounting for 79% of all failed tasks.
Q5. Do models effectively utilize advanced search features during retrieval?
No, models rely heavily on simple methods with 98.5% of retrievals using plain substring queries, while less than 2% utilize regular expressions or fuzzy matching.
Q6. How does model performance vary across the tested models?
There is a wide performance gap, with GPT-5.5 (xhigh) achieving 57.3% accuracy while the weakest model reaches only 16.4%.
Q7. What specific cognitive tasks do models find most difficult?
Models struggle specifically with preference inference and multi-intent decomposition compared to other cognitive capabilities.
Q8. Why do models tend to stop searching prematurely?
The paper indicates that models often perform fewer retrievals on failed tasks because they stop searching once they encounter information that appears plausible but is actually incorrect.
Q9. How many personal records are included in the dataset?
The benchmark includes 4,335 personal records used to ground the 250 tasks.