Audio-Visual Flamingo for Complex Videos

Summary & Methodology Analysis

The paper tackles the scarcity of suitable large-scale audio-visual datasets and the limitations of existing audio-visual large language models in joint perception, temporal alignment, and multi-event reasoning over long and complex videos. To solve this, the authors introduce a system architecture that relies on several standard and novel components. For vision, it uses a SigLip vision encoder, which is a vision transformer (an architecture using attention to process image patches) that extracts spatial features from images or video frames. For audio, it uses the AF-Whisper audio encoder with a sliding-window feature extraction mechanism to process long-form audio. Two-layer MLPs project these encoder outputs into the embedding space of Qwen2.5-7B, which serves as the central text-only reasoning backbone LLM (large language model). It also incorporates a streaming TTS (text-to-speech) module implemented as a decoder-only transformer for voice-to-voice interaction.

To power cross-modal learning, the paper introduces a novel collection called Audio-Visual-Skills, featuring paired captions and QA training instances designed for short and long videos. It also proposes Temporal Audio-Visual Interleaved Chain-of-Thought, a reasoning framework that explicitly grounds intermediate reasoning steps to timestamps in long audio-visual streams. The model undergoes a novel three-stage training curriculum. This curriculum consists of pre-training for initialization and short-context training, mid-training for long-context training with AV-Skills-Long, and post-training involving supervised fine-tuning and GRPO-based RL (reinforcement learning, a method of training models via reward signals) on the AV-Think dataset.

Evaluating performance across multiple benchmarks, the models show strong results. On WorldSense, AVF-Instruct achieves an accuracy of 50.3 and AVF-Think achieves 51.6, outperforming Qwen2.5-O at 45.4 and OmniVinci at 48.2. On DailyOmni, AVF-Instruct reaches 72.4 and AVF-Think reaches 73.9, outperforming OmniVinci at 66.5. On MMOU, AVF-Instruct achieves 56.9 and AVF-Think achieves 60.2, outperforming Minicpm-o 4.5 at 46.8 and Gemini-2.5 Pro at 64.2. On the MMAU-v05.15.25 test average audio understanding, AVF-Instruct achieves 73.49, outperforming Audio Flamingo 3 at 72.42 and OmniVinci at 71.60. On Video-MME with subtitles, AVF-Instruct hits an accuracy of 71.2, outperforming OmniVinci at 68.6. Despite these successes, the paper notes several limitations. AV-Skills is built from public datasets and open-internet videos, which may introduce source bias and potential overlap with prior training data. Reasoning over very long and highly dense videos remains challenging, especially when evidence is sparse or temporally dispersed. Furthermore, current benchmarks do not fully capture open-ended real-world deployment.