Learning from Metadata for Better Bioacoustics

Summary & Methodology Analysis

Bioacoustic foundation models frequently encounter domain shifts when moving from training data sourced on citizen science platforms to target applications like continuous passive acoustic monitoring. To tackle this, the paper presents MetaPerch, a system designed for robust feature representations that generalize more effectively across species distribution and acoustic domain shifts. The base architecture uses random window sampling with 5s windows, mixup data augmentation, and an EfficientNet-B3 lightweight vision model to embed 2D spectrograms of audio windows. A prototype learning classifier and a linear classifier predict species using cross-entropy loss, where gradients from the linear head back-propagate to the embedding model. The novel core contribution is multi-task learning where the model jointly trains with auxiliary metadata prediction tasks alongside primary species identification. These auxiliary metadata prediction heads are constructed as small multilayer perceptrons, which are networks of interconnected artificial neurons, that project spatially aggregated embeddings into dimensions for metadata sources including location, season, and background species. Missing metadata are addressed by assigning placeholder values and zeroing out their loss contributions. Mixup augmentation with metadata is handled by formulating metadata prediction tasks as classification tasks and constructing multihot targets, mixing audio from all recordings while only mixing metadata that is present. Additionally, adversarial training via a gradient reversal layer, which reverses gradients during backpropagation to remove specific information from representations, is optionally used between the backbone and metadata prediction heads to encourage embeddings to be agnostic to spurious metadata.

MetaPerch demonstrates consistent performance gains across multiple benchmarks and task types. It achieves an average increase of 0.015 in ROC-AUC over BioBaseline across the BirdSet benchmark datasets. On underrepresented regions in BirdSet, including Peru, Hawai'i, Colombia, and Costa Rica, it yields an average improvement of 0.025 ROC-AUC points. Furthermore, MetaPerch improves accuracy on BEANS classification tasks by 0.016 on average, including specific gains of 0.021 on Watkins and 0.012 on Bats. It also outperforms BioBaseline on BEANS detection tasks with an improvement of 0.006 on Mean cmAP, including improvements of 0.009 and 0.019 ROC-AUC on ENABirds and DCASE, respectively. On the globally distributed WABAD dataset, MetaPerch improves over the baseline by 0.070 and 0.018 in ROC-AUC for linear and prototype learning probes, respectively.

Despite these improvements, the paper documents specific limitations regarding the utility of metadata. The benefits of metadata diminish when both species and acoustic domain shifts are present in conjunction. Furthermore, metadata benefits are not uniform across biomes, showing a marked drop for deserts. In the BIRB few-shot linear probe experiment on the Xeno-Canto held-out set, MetaPerch underperformed BioBaseline at higher shots. The models and datasets utilized across the research span Xeno-Canto, iNaturalist, Tierstimmenarchiv, FSD50K, Perch 2.0, BioBaseline, MetaPerch, EfficientNet-B3, BirdSet, BEANS, Powdermill, Caples, Weldy calltype, DCLDE, NOAA, ReefSet, ENABirds, HumbugDB, Hainan Gibbons, PER, UHH, NES, HSN, NBP, SSW, SNE, Watkins, Bats, Dogs, DCASE, Hiceas, RFCX, CBI, WABAD, ESC-50, Speech Cmds, GeoCLIP, SatCLIP, and AlphaEarth.