Back to Feed
Multimodal

Relative Scene Graph Memory for Video QA

Original: R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video
By Ke Ma, Yamin Mao, Weiming Li, Shuai Tan, Yijie Zhong, Hao Chen, Haofen Wang, Meng Wang Published 2026-08-11

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 2 concepts

Key Takeaways

  • Long egocentric video lacks persistent structured object-centric memory, making object history and state change tracking difficult for AI assistants.
  • The method constructs frame-wise scene graphs with relative spatial relationships and establishes persistent object tracks using semantic, geometric, and anchor-neighborhood consistency.
  • On a 255-question object-related subset from EgoLifeQA, the proposed approach achieved a 6.7-point overall accuracy gain over EgoRAG-Text and a 12.5-point gain on when questions.
  • The current empirical findings are restricted to a single-subject dataset and do not demonstrate multi-subject generalization.

Summary & Methodology Analysis

AI assistants processing long egocentric video recorded from wearable cameras struggle with object history and state changes because traditional methods lack persistent, structured object-centric memory. Existing approaches often rely on captioning, temporal grounding, or strong 3D geometric assumptions that fail in monocular, free-motion settings. To solve this, the authors introduce a relative 4D scene graph memory system that reorganizes raw video into semantic episodes based on activity and location labels. It performs 2D object segmentation using promptable video segmentation guided by episode semantics, and applies RGB-only lifting to estimate relative 3D metadata including object location and extent.

The system constructs frame-wise scene graphs incorporating space, activity, and object nodes with relative spatial relationships. It establishes persistent object tracks across frames using semantic, geometric, and anchor-neighborhood consistency. Instead of relying on global coordinates, it identifies static anchors and dynamic objects to represent object states through anchor-relative transitions. Finally, it writes retrieval-ready memory entries at the segment window level containing temporal, place, activity, object, state, and interaction summaries.

Evaluation on a 255-question object-related subset from EgoLifeQA shows a 6.7-point overall accuracy gain over EgoRAG-Text and a 12.5-point gain on when questions. However, the approach has notable limitations. The empirical findings are restricted to a single-subject dataset and do not demonstrate multi-subject generalization, and the memory system is restricted to offline, retrospective use with limited persistence across days.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What problem does the paper address?

Long egocentric video lacks persistent, structured object-centric memory, making it difficult for AI assistants to answer questions about object history and state changes.

Q2. What is the core proposed method?

A relative 4D scene graph memory system that reorganizes video into semantic episodes, performs object segmentation, estimates relative 3D metadata, and builds frame-wise scene graphs with persistent object tracks.

Q3. What performance gains were achieved?

On a 255-question object-related subset from EgoLifeQA, the approach achieved a 6.7-point overall accuracy gain over EgoRAG-Text and a 12.5-point gain on when questions.

Q4. How are raw videos initially processed?

Raw video is reorganized into semantic episodes based on activity and location labels.

Q5. How does the system estimate 3D metadata from video?

It applies SAM 3D-style RGB-only lifting to estimate relative 3D metadata including object location and extent.

Q6. How does the memory system handle object states without global coordinates?

It identifies static anchors and dynamic objects to represent object states through anchor-relative transitions rather than global coordinates.

Q7. What data is included in the retrieval-ready memory entries?

They contain temporal, place, activity, object, state, and interaction summaries at the segment window level.

Q8. What datasets and benchmarks are mentioned in the paper?

The paper references EgoLifeQA, EgoRAG-Text, SAM 3, SAM 3D, DUSt3R, MASt3R, Open3DSG, VL-SAT, OpenFunGraph, 3DGraphQA, EMQA, EgoSchema, GroundVQA, AMEGO, G-Retriever, KG 2 RAG, GNN-RAG, 3DSSG, SceneGraphFusion, EgoSG, HostSG, Action Scene Graphs, GraphVideoAgent, SAMJAM, and Qwen3.5-27B.

Q9. What are the limitations of the current approach?

The empirical findings are restricted to a single-subject dataset and do not demonstrate multi-subject generalization, and the memory system is restricted to offline, retrospective use with limited persistence across days.

Browse the archive →
Multimodal / Safety & Alignment Programmable Backdoor Attacks on Vision Models The paper introduces a flexible backdoor paradigm that enables dynamic, post-training control over a Vision Language Model output by injecting trigger patterns into training data. Benchmarks & Evals / Multimodal Evaluating Image Captions with Dense Question Answering CapProbe is a new benchmark for evaluating the factual accuracy of image captions by decomposing images into specific regions and using dense, region-anchored question answering to detect hallucinations and omissions. Multimodal / Benchmarks & Evals Removing Person Knowledge from Multimodal Models The paper introduces a corpus-free framework and benchmark for deleting specific person-related knowledge from multimodal large language models without needing the original training data.