Unifying Spatial Perception and Reasoning

Summary & Methodology Analysis

SPARGen addresses the challenge of unifying 3D reconstruction, dense correspondence estimation, and spatial reasoning into a single instruction-conditioned multimodal generative framework. It achieves this without relying on task-specific prediction heads or external geometric modules. Instead, the framework builds upon the Bagel model and employs a Mixture-of-Transformer-Experts architecture, which is a modular neural network design that routes inputs to specialised sub-networks, containing an understanding expert and a generation expert. Input image sequences are encoded into visual-understanding tokens using a Vision Transformer encoder, a neural network architecture based on self-attention mechanisms for processing visual data, while natural-language instructions are represented as text tokens. Sequence-based tasks, such as textual answers, sparse geometric states, and structured camera poses, are formulated to be generated autoregressively, predicting tokens one by one.

For dense-field tasks like depth maps, point maps, and optical flow, SPARGen formulates them as image-aligned fields and maps their target fields to clean target latents using a frozen variational autoencoder, a neural network that compresses data into a lower-dimensional latent space without updating its weights during training. It then constructs a linear rectified-flow path between the clean VAE latent and Gaussian noise for dense-field training, where the generation pathway predicts the velocity field. At inference time, the model performs optical flow refinement through a predict-warp-predict procedure by aligning the second frame to the first and predicting a residual flow. This design allows the model to handle diverse spatial tasks within a single pipeline.

Despite its strong performance, the architecture has notable limitations. Spatial compression using the frozen VAE acts as a bottleneck for geometric edges and high-precision physical quantities. In terms of quantitative results, SPARGen-7B achieves an average score across four spatial-reasoning benchmarks outperforming compared non-proprietary models, improving the average score by 9.85 points on MindCube, 1.97 points on OmniSpatial, 4.99 points on OST, and 24.71 points on SPAR. Additionally, it achieves zero-shot optical flow results on KITTI with an Endpoint Error of 4.09 and an F1-all score of 13.34.