Hand Visibility Detector for Keypoints

Summary & Methodology Analysis

Existing hand pose estimation methods lack explicit, accurate, and standalone visibility estimation, treating visibility only as an auxiliary signal for pose estimation rather than as a primary task. To solve this, the paper implements a pipeline starting with a frozen Vision Transformer (ViT), a neural network architecture using attention mechanisms to process data, backbone loaded from a large scale pretrained Hand Pose Estimation model to extract image features. These feature maps are compressed using a 1x1 convolution layer, and the resulting flattened tokens are processed through a Gated Attention Unit (GAU) to model global spatial dependencies. The features are then projected into J channels using a 1x1 convolution, followed by spatial average pooling and a sigmoid activation function to output per-joint visibility probabilities. Only the lightweight visibility head is trained using binary cross-entropy loss, while the backbone remains frozen.

The proposed method delivers strong empirical results, outperforming baselines with an mAP of 0.931 and an F1 score of 0.896 on the HInt dataset. Additionally, visibility-weighted triangulation reduced the mean reprojection error by up to 10.1% on DexYCB, HO3D, and H2O datasets. The paper evaluates various models and datasets, including HaMeR, WiLoR, RTMPose, Contact4D, COCO-WholeBody, CSPNeXt, ResNet-50, DINOv3, and ImageNet-1k, but the extracted facts do not provide further architectural breakdowns for these comparisons.

A key limitation observed by the authors is that the method relies entirely on frozen backbones. Specifically, fine-tuning the backbone for the visibility task actually degrades performance, meaning the feature extractor must remain static during training of the visibility head. The paper does not specify training time, hardware requirements, or inference latency figures beyond the reported accuracy metrics and error reductions.