Language-Controlled Quadrotor Flight via Visual Models

Summary & Methodology Analysis

Language-conditioned quadrotor flight requires policies to ground semantic goals, anticipate visual consequences, and output smooth, dynamically executable control references from rapidly changing first-person views. Existing methods using discrete actions or high-level waypoints provide limited supervision for these tasks. Furthermore, applying World-Action Models to quadrotor navigation is challenging due to the difficulty of collecting large-scale, synchronized multimodal flight data and the high inference-time cost and error accumulation from autoregressive future-video generation. To resolve these challenges, AeroAct adapts a pretrained video diffusion transformer (a neural network architecture based on self-attention mechanisms that processes data in parallel by weighting the significance of different sequence parts) to predict local trajectory action chunks from egocentric visual history, proprioception, and language. It uses future first person frames as dense consequence supervision only during training. The system is designed so that deployment directly decodes actions without generating future video, employing a blockwise causal mask to prevent action tokens from accessing future frame tokens. It constructs a scalable hybrid data pipeline combining DiffAero dynamics, Isaac Lab, and 3D Gaussian splatting renderers to generate aligned multimodal data including egocentric video, language, proprioception, and dynamically feasible action chunks. It also introduces a low-cost handheld collection device that couples camera observations with motion estimates to recreate flight-like egocentric trajectories for real-world data collection, formulates a smooth chunked action space based on local fifth-order trajectory segments, and introduces an inference-time self-guidance procedure to improve temporal consistency across overlapping trajectory chunks by replacing the prefix velocity of a new sample with the corresponding suffix from the previous action chunk.

Regarding the architecture and components, the methodology integrates several models and hardware tools. The models or datasets involved include a pretrained video diffusion transformer, DiffAero, Isaac Lab, 3D Gaussian splatting, Wan video diffusion transformer, Wan2.1-1.3B, Intel RealSense T265 stereo camera, Intel RealSense D435i camera, Radxa ROCK 5C, OddityRC 35Pro frame, and the AdamW optimizer. The system relies on a smooth chunked action space based on local fifth-order trajectory segments and a blockwise causal mask to manage token dependencies.

Despite its effectiveness, the approach has clear limitations. Real-world experiments are currently confined to short indoor trajectories. The current temporal context is sufficient for single-stage target reaching, but inadequate for complex instructions requiring multiple semantic sub-goals, recovery behaviors, or long-horizon memory. Additionally, real-world deployment relies on offboard inference due to the computational cost of the video diffusion backbone. The paper does not specify precise latency figures, parameter counts, or dollar costs beyond noting the high inference-time cost of the video diffusion backbone.