Predicting Robot Interactions from 360 Degree Video
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 1 concepts
Key Takeaways
- The HUI360 dataset provides 4,310 tracks across 71.3 hours of egocentric video data.
- Interaction prediction is formulated as a binary classification task to determine if a human will engage with a robot platform.
- The approach uses YOLOv11x for detection and SAM2.1-Video for human segmentation.
- LSTM models achieved an AUC of 0.84 during cross-dataset transfer tests from HUI360 to SSUP-A.
Summary & Methodology Analysis
The HUI360 methodology centers on creating a structured pipeline for interaction anticipation using 360 degree egocentric video captured by a mobile robot. The system utilizes YOLOv11x for object detection and SAM2.1-Video, which is a segment anything model for video that produces temporal masks, to track humans. To ensure data quality, the team filters tracks based on pose keypoint validity and mask quality before labeling them based on physical intersection with interaction zones like robot plates or trashcans. The final dataset includes over 1M detections, providing a large-scale resource for training predictive models.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of the HUI360 project?
The project aims to improve human-robot interaction in real-world environments by training models to anticipate future physical contact with a robot.
Q2. What kind of data does the dataset contain?
It contains 71.3 hours of 360 degree egocentric video captured from a mobile robot across diverse environments.
Q3. How large is the HUI360 dataset?
The dataset includes 4,310 tracks and over 1M detections.
Q4. Which specific machine learning models were tested for anticipation?
The authors evaluated Random Forest, Multi-Layer Perceptron, and Long Short-Term Memory recurrent neural networks.
Q5. How is the interaction zone defined in the training data?
Interactions are defined as the intersection of a person's segmentation mask with predefined zones, such as a trashcan or a robot plate.
Q6. What inputs are used for the binary classification model?
Input features include mask size representing distance, bounding box coordinates, and pose keypoints extracted via ViTPose and Sapiens-0.6B.
Q7. What were the results of the cross-dataset transfer tests?
The LSTM model achieved an AUC of 0.84 when trained on HUI360 and tested on SSUP-A.
Q8. What are the limitations regarding social signal modeling?
The dataset currently only models physical contact and does not explicitly account for non-physical social signals like gaze or verbal engagement.
Q9. Does the paper provide specific latency or inference speed metrics?
The paper does not specify these metrics.