Testing LLM Spatial Reasoning and Mapping

Summary & Methodology Analysis

The research team constructed a novel benchmark by leveraging UK metropolitan district wards to generate atomic triplets represented as (x, r, y). These instances are derived from geometric measurements and GeoSPARQL predicates to capture fundamental spatial concepts. To test higher-order reasoning, the authors combined these atomic units into composite triplets using logical AND operations. This allows for rigorous Binary-QA and MCQ evaluations, including negated statements to check for logical consistency. The study incorporates Llama, Mistral, and Qwen variants ranging from 0.6B to 8B parameters to compare performance across different architectures. Analysis of these models involved linear probes, which are lightweight linear models trained to predict concepts from latent vector representations, to inspect whether the internal layers effectively encode abstraction and compositional logic. The researchers also examined contextual grounding by injecting explicit numerical coordinate and geometric data into the prompts to see if providing raw spatial context improves performance. The analysis highlights that while most models show consistent latent structures, the Mistral family consistently failed to generalize across the benchmark instances. The current limitations of this study include a constrained geographic scope restricted to two regions, the possibility that chosen spatial concepts do not capture full real-world complexity, and the use of linear classifiers which may not reveal non-linear dependencies in the model representations.