Making Music Transformers Shift-Invariant

Summary & Methodology Analysis

Standard music transformers, as they grow larger or train longer, suffer from a lack of equivariance to time-shifted or pitch-transposed inputs. Instead of recognizing relative patterns, they allocate model capacity to memorizing absolute positions. For example, the Anticipatory Music Transformer exhibits progressively less equivariance over time and suffers a significant drop in output quality under shifted prompts. To address this, the paper proposes the Equivariant Music Transformer, which maintains consistent quality across shifts.

The methodology relies on a base transformer architecture derived from Moonbeam. It implements Fundamental Music Embedding for feature-level equivariance and employs multi-dimensional relative attention, which is a mechanism to weigh input tokens, to encode relative musical positions. The model adds an auxiliary equivariance regularization loss and is jointly optimized using a standard next-token prediction loss and an auxiliary Kullback-Leibler divergence loss. Furthermore, it applies random pitch and temporal shifts to inputs and uses stop-gradient operations on anchor predictions to enforce equivariance.

Evaluation shows that the Equivariant Music Transformer outperforms state-of-the-art baselines and traditional data augmentation methods in both generative capability and generalization to musical shifts. However, the approach has a notable limitation: models trained on the LakhMIDI dataset may reproduce fragments of their training examples. The paper does not specify further limitations, hardware requirements, or exact computational costs.