Improving 3D Vision Models at Runtime

Summary & Methodology Analysis

Self-Geometry introduces a test-time adaptation (TTA) pipeline designed to enforce multi-view geometric consistency on pretrained Vision Foundation Models (VFMs) like Depth Anything 3 (DA3). Since pretrained models often lack explicit geometric coherence, the pipeline utilizes external feature matchers, such as LightGlue, to extract 2D pixel correspondences. These correspondences serve as a supervisory signal to align depth and pose predictions, allowing the system to refine the model weights in a specific scene without prior ground truth. By inserting LoRA (a parameter-efficient fine-tuning technique that updates a small subset of weights) into the model architecture, the system optimizes these adapters while the core model remains frozen. This approach specifically targets the under-fitted regime, yielding substantial performance boosts in geometry estimation for smaller model variants.

The architecture incorporates specific mechanisms to handle geometric constraints. It leverages depth-independent point-to-line constraints and multi-view consistency to refine the model's output, ensuring that predictions across multiple viewpoints are coherent. To manage this optimization, the pipeline carefully handles gradient flow to prevent conflicts between different geometric requirements. The authors validate this across a variety of visual geometry benchmarks, including 7Scenes, ScanNet++, ETH3D, and HiRoom. The inclusion of these modules effectively bridges the gap between raw pretrained outputs and high-fidelity 3D reconstruction requirements for downstream tasks.

Despite these performance gains, the system faces practical constraints regarding its deployment profile. The adaptation process, while completed within two minutes for up to 40 input views on an NVIDIA RTX PRO 6000, remains far from meeting real-time requirements. Furthermore, the quality of the adaptation is strictly coupled with the reliability of the feature matcher. In scenarios characterized by repetitive textures, textureless surfaces, or wide-baseline viewpoint changes with limited overlap, the matcher often fails. This failure weakens the supervision signal, leading to a measurable degradation in adaptation quality, which developers should consider when evaluating the pipeline for production environments.