Modular Experts for Multi-Task Video Models
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- The VideoLLM-MoTE-1B+5E model achieves a 62.9 percent average top-1 accuracy across five benchmarks on the COIN dataset.
- Converting to the MoTE decoder architecture improves KIE task performance on CORD from 20.07 percent to 95.79 percent Micro-F1.
- The modular design allows the five-expert model to activate approximately 2 billion LLM parameters per sample.
- Incremental addition of experts preserves performance on existing routes by freezing old modules.
Summary & Methodology Analysis
The architecture addresses task interference in multi-task video models by replacing standard transformer decoder feed-forward networks with a Mixture of Task Experts (MoTE). This design retains a shared backbone, including the visual encoder and language token embedding, while routing inputs through specific task-level computation paths. By initializing experts from pretrained parameters and incorporating a shared layer active for every input, the system enables the model to manage heterogeneous procedural tasks without overwriting previously learned representations.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the core contribution of this work?
The authors propose a modular decoder design called MoTE that exposes distinct computation paths for different tasks to reduce interference in video-language models.
Q2. What kind of tasks does this model perform?
The model is designed for multi-task video understanding, including action recognition, structured field extraction, and procedural intent analysis.
Q3. Does this model improve over previous approaches?
Yes, it achieves higher average top-1 accuracy on COIN benchmarks compared to recent VideoLLM baselines.
Q4. How does the model handle task extension?
It adds new task experts while freezing old modules, which protects existing routes from performance degradation.
Q5. What is the computational cost of the five-expert model?
The model activates approximately 2 billion LLM parameters per sample.
Q6. What datasets were used for validation?
The authors evaluated the model on COIN, Ego4D, SROIE, and CORD datasets.
Q7. Are there limitations to the routing mechanism?
Yes, prompt-conditioned routing is currently limited to predefined intent categories and may struggle with ambiguous, overlapping, or compositional requests.
Q8. What is the trade-off of freezing old modules?
While freezing old modules protects existing routes, it reduces the overall plasticity of the model for learning new tasks.
Q9. How is the task routing implemented?
The design uses a prompt-conditioned selector to map inputs to specific task experts, supplemented by a shared expert layer.