Dynamic Modality Selection for 3D Scenes

Summary & Methodology Analysis

SmartMage improves 3D scene understanding by replacing fixed modality pipelines with a dynamic orchestration system. It first encodes diverse inputs like text, RGB images, depth maps, and point clouds into a shared embedding space. The core innovation is the SMART module, which employs a semantic prior estimator and similarity scorer to evaluate the necessity of each modality relative to the user query. This data informs the MAGE module, which uses a sparse Mixture of Experts (MoE) architecture, a design that routes data through specialized sub-networks to reduce computation, to activate specific model components based on modality priors. This approach avoids the semantic noise and wasted computation inherent in processing all input types for every query. The implementation relies on a selective fine-tuning strategy that optimizes only the router and expert branches end-to-end, keeping the visual encoders and primary LLM weights frozen to preserve base capabilities. The system shows substantial performance gains over baselines like Ross3D, reaching a +5.1 increase in Acc@0.5 on ScanRefer and a +6.4 F1@0.5 increase on Multi3DRefer. Despite these gains, the architecture faces challenges with input viewpoint selection. The FoVSR algorithm may skip critical angles, leading to potential gaps in spatial context. Furthermore, the use of Farthest Point Sampling to downsample point clouds reduces density, which may constrain the model when representing fine-grained object features.