Benchmarking Real Time Omni Modal Assistants

Summary & Methodology Analysis

The researchers developed OmniAssistBench to address the difficulty of evaluating large language models that process video and audio in real time. Standard static datasets often fail to capture how a model response influences future user behavior. To solve this, the authors created a pipeline that filters video content to ensure suitability for assistant interaction, logically deduces user goals, and segments these into multi-turn clips. These clips are then processed by the model, which must handle inputs provided via text-to-speech and subtitles to mimic a live assistant environment. The evaluation uses an automated judge to score responses against a 5-point rubric that penalizes hallucinations and redundant information across two tiers: perception and goal-oriented tasks. Performance is summarized by a total score out of 100 points, where the proprietary Gemini-3-Pro achieves 66.4 and the open-source Qwen3-Omni-Instruct achieves 51.2. The evaluation reveals significant bottlenecks in how these models process transient information. Specifically, models often fail to delay their output until a target event occurs, which is a critical requirement for synchronizing actions with video timestamps. Furthermore, they struggle to interpret visual prompts such as hand gestures, which suggests a gap in fine-grained perception for interactive agents. A major limitation is the loss of cross-turn context, where models forget the user goal when new visual stimuli appear in subsequent turns. This tendency to drift suggests that while these models can process multimodal data, they lack the robust state management required for long-running, goal-directed assistance.