Efficient Token Pruning for 3D Vision Models
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- Reduces token counts to 10% on general 3D tasks with near-zero performance loss.
- Retains over 90% performance on VSI-Bench when using only 15% of total visual tokens.
- Introduces CoverPrune-Lite for high-performance inference using Morton code ordering and grouping.
- Optimizes token selection by prioritizing visual evidence coverage rather than simple diversity.
Summary & Methodology Analysis
3D Vision-Language Models (3D VLMs) often suffer from significant computational bottlenecks due to the massive number of visual tokens (discrete units of image or spatial data used as inputs for transformers). Existing approaches typically focus on token diversity, but this paper shifts the objective to maintaining visual evidence coverage through an optimal transport formulation. This approach treats the selection process as a semi-relaxed Optimal Transport (OT) problem, which computes the cost of moving probability mass between distributions. The researchers designed a Feature-Spatial-Temporal (FST) transport cost to model feature similarity, 3D spatial proximity, and temporal coherence, ensuring that the model retains the most informative parts of the scene.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem addressed by this paper?
The paper addresses the high computational cost of processing massive visual token counts in 3D VLMs during inference.
Q2. Does this method impact model performance?
The paper reports that CoverPrune and CoverPrune-Lite maintain near-zero performance loss with 10% of visual tokens on general 3D tasks.
Q3. What is the primary contribution of CoverPrune?
It formulates token pruning as an optimal transport problem to better preserve visual evidence coverage compared to diversity-based selection.
Q4. How does CoverPrune-Lite achieve efficient pruning?
It uses 3D-aware Morton code ordering and capacity-guided grouping for local block-structured matching.
Q5. What specific metrics support the performance claims on VSI-Bench?
At a 20% retention ratio using GS-Reasoner, the method preserves 92.4% of full-token performance, and it retains over 90% performance at a 15% retention ratio.
Q6. What components make up the FST transport cost?
The FST cost models feature similarity, 3D spatial proximity, and temporal coherence.
Q7. What algorithmic approach is used to approximate the OT objective?
The paper implements a Spatial-Guided Greedy Selection (SGS) algorithm to approximate the optimal transport objective.
Q8. Are there any known limitations to the current approach?
The paper notes that SOTA performance on general 3D QA tasks has largely saturated.
Q9. Which models and datasets are evaluated?
Evaluations involve models like GS-Reasoner, Honeybee, and LLaVA-UHD, using datasets including ScanQA, SQA3D, Scan2Cap, and VSI-Bench.