Learning Hierarchical Representations for Symbolic Music

Summary & Methodology Analysis

The MIDI-RAE-JEPA architecture addresses the inability of traditional models to handle the hierarchical, multiscale nature of music. The system encodes binary piano rolls as 128x128 images, which are then processed by a Swin Transformer V2, a vision model that partitions inputs into hierarchical stages to capture features at multiple scales. To avoid common distortions caused by standard image-based transformations, the model uses an equivariance loss to internalize pitch and time-shift relationships, alongside a masked embedding predictor that learns context across the hierarchical layers. Representation collapse is mitigated by the use of SIGReg to enforce an isotropic Gaussian prior on the embeddings, and a soft factorization loss is applied to keep pitch and time dimensions geometrically orthogonal in latent space. The pipeline concludes with a separate decoder trained on frozen embeddings for reconstruction and a flow matching model for generation. The performance is validated through an F1 score of 0.995 for reconstruction and superior 4-class accuracy of 0.488 on the EMOPIA emotion classification dataset compared to Haar scattering and DINOv2. A primary limitation is that most representational capacity is concentrated at the fine hierarchical levels, leaving coarser levels unable to capture high-level musical abstractions. Additionally, increasing the maximum time shift during training negatively impacts reconstruction accuracy if the factorization loss is omitted.