Testing AI Navigation in Virtual Cities

Summary & Methodology Analysis

UrbanGround provides a sandbox environment built from 3D Visualisation Map and 3D Pedestrian Network data, which are sourced from the Hong Kong Lands Department. The system integrates a Microsoft Rocketbox avatar library to simulate a busy urban environment. It specifically evaluates Multimodal Large Language Model (MLLM) agents, which are systems that process both text and visual inputs, across a series of tasks ranging from local spatial reasoning to full-scale city navigation. The research measures performance through a first-person perspective interaction layer, allowing agents to respond to complex, real-scale city features.

Evaluation results indicate a significant performance gap between local perception and global planning. For example, Claude-Opus-5 demonstrates 91.3% accuracy in visual recognition but struggles significantly with long-range navigation, achieving only a 2.5% success rate in LongNav. Similarly, models like GPT-5.5 and Kimi-K3 yielded overall navigation success rates of 20.8% and 22.5% respectively across eight tasks. Despite these low success rates in completing full tasks, over 50% of episodes result in the agent reducing the distance to the goal, suggesting that the agents possess basic spatial progression capabilities even when they fail to reach the specific target.

A primary bottleneck for these agents is their inability to maintain spatial state over extended periods. As the agents explore, they accumulate errors without effective correction, leading to a failure to compose local actions into successful goal-directed behavior. Furthermore, the agents exhibit poor adaptation to environmental dynamics. When a route becomes invalidated, such as during a road closure, the agents frequently continue to execute locally compliant movements despite their original navigation plan becoming obsolete. The paper highlights that current MLLM architecture lacks the necessary mechanisms to reconcile these persistent spatial planning errors during long-horizon navigation.