Running Bimanual Robotics on Jetson
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- A zero-copy GStreamer capture pipeline utilizing NVIDIA NVMM buffers reduced peak single-core CPU utilization from 98.0% to 77.0% and maximum pipeline latency from 117.31 ms to 101.52 ms.
- Action Chunking with Transformers achieved a 95% success rate on a bimanual pick-and-place task, whereas Diffusion Policy failed to produce a usable policy.
- TensorRT successfully calibrated and quantized the ResNet18 vision backbone to FP16 and INT8, though it failed to quantize ACT transformer layers.
- The system was successfully deployed and evaluated on an NVIDIA Jetson Orin Nano Super within an 8 GB budget.
Summary & Methodology Analysis
The paper investigates the challenge of deploying bimanual robot control policies onto resource-constrained embedded hardware, specifically targeting an entry-level Jetson platform with limited memory and CPU. To address memory and compute bottlenecks during three-camera sensing, the authors implemented a zero-copy GStreamer capture pipeline utilizing NVIDIA NVMM buffers. This architectural change bypasses unnecessary memory copies, successfully lowering peak single-core CPU utilization from 98.0% down to 77.0% and reducing maximum pipeline latency from 117.31 ms to 101.52 ms.
For policy execution, the authors evaluated two models trained on identical demonstration datasets: Action Chunking with Transformers, a policy model that predicts sequences of actions using attention mechanisms, and Diffusion Policy, a generative model framing action generation as a denoising process. While Diffusion Policy failed to produce a usable policy (0 out of 10 trials) even at twice the gradient steps, Action Chunking with Transformers converged on the bimanual pick-and-place task and achieved a 95% success rate across 20 trials. To optimize inference on the edge, the Action Chunking with Transformers model was converted to ONNX and built into TensorRT engines targeting the NVIDIA Jetson Orin Nano Super. TensorRT successfully applied INT8 quantization to the ResNet18 vision backbone, though it failed to quantize any of the model's 145 transformer layers.
The study faces a couple of notable limitations. First, the 200k-step reference training budget for Diffusion Policy was originally tuned on the simpler PushT task, which likely explains why it failed to converge on the more complex bimanual task. Second, the evaluation relied on a relatively small sample size of 20 trials per condition, limiting the statistical significance of small differences in success rates. The paper does not specify other limitations or broader hardware deployment constraints beyond these factors.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What problem does this paper address?
The paper addresses the challenge of deploying bimanual manipulation policies on resource-constrained embedded hardware, specifically investigating whether bimanual sensing and inference can operate effectively on an entry-level Jetson platform with limited memory and CPU.
Q2. What hardware platform was used for the edge deployment?
The paper used the NVIDIA Jetson Orin Nano Super.
Q3. What was the main performance result of the robot task?
Action Chunking with Transformers achieved a 95% success rate, completing 19 out of 20 trials on a bimanual pick-and-place task involving a deformable beanbag.
Q4. How did the authors reduce CPU utilization for camera capture?
They implemented a zero-copy GStreamer capture pipeline using NVIDIA NVMM buffers to reduce CPU utilization and latency for three-camera sensing.
Q5. What models were compared in the study?
The study compared Action Chunking with Transformers and Diffusion Policy, both trained on offline workstation setups with identical demonstration datasets.
Q6. How did the two models compare in task success?
Action Chunking with Transformers achieved a 95% success rate (19 out of 20 trials), while Diffusion Policy failed to produce a usable policy with a 0% success rate (0 out of 10 trials) despite using twice the gradient steps.
Q7. What tools were used to optimize the model for the edge?
The authors converted the model to ONNX and built TensorRT engines for FP16 and INT8 inference.
Q8. Did TensorRT successfully quantize all parts of the Action Chunking with Transformers model?
No. TensorRT's INT8 calibration successfully quantized the ResNet18 vision backbone, but failed to quantize any of the 145 transformer layers.
Q9. What limitations are noted regarding the training budgets and evaluation?
The paper notes that Diffusion Policy's 200k-step training budget was tuned on the simpler PushT task, which may explain its failure to converge. Additionally, the study used a small sample size of 20 trials per condition, limiting statistical significance.