Improving Robot Action Models Using Semantic Foresight

Summary & Methodology Analysis

World-Action Models often fail when visual inputs change because their latent spaces focus too heavily on pixel-level appearance rather than actionable dynamics. Robust-WAM modifies the standard architecture of these models by inserting learnable query tokens into the Action DiT (a transformer model that generates action sequences). These tokens are assigned temporal positional encodings to represent specific future steps, allowing the model to focus on the semantic content of the scene rather than just pixel reconstruction. The training process uses a semantic foresight objective, which forces the query tokens to align their output hidden states with frozen DINOv3 CLS embeddings (feature vectors representing class-level information from images). This alignment process effectively imports strong semantic priors into the action prediction pipeline. During inference, the DINOv3 teacher and the alignment logic are discarded, leaving only the query-enhanced model for execution. This avoids adding runtime overhead associated with the teacher model while retaining the performance benefits. Experimental results on the LIBERO-Plus benchmark showed that the method improved success rates for the GE-Act baseline from 78.0% to 80.9%, and for the FastWAM baseline from 49.7% to 58.9%. Real-world evaluation on a Franka Research 3 platform showed a significant boost in out-of-distribution performance, raising the GE-Act success rate from 57.3% to 80.0%. One limitation identified in the paper is that combining DINOv3 CLS embeddings with depth-based targets (DepthAnything3) resulted in performance degradation. This suggests that over-constraining the model with multiple simultaneous alignment targets can interfere with the learning process.