Automated Audio to Sheet Music Transcription
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Introduces the SheetSage A2S dataset to provide high quality audio and symbolic score pairs for popular music.
- Utilizes a frozen pre trained MuQ encoder to extract audio features, feeding into an autoregressive Transformer decoder.
- Achieves a 4.98 percent symbol error rate on classical music and 20.92 percent on the popular music dataset.
- Employs offline data augmentation via pitch shifting and time stretching to improve model robustness.
Summary & Methodology Analysis
The system utilizes an autoregressive Transformer decoder, a sequence modeling architecture that generates outputs token by token based on previous predictions. To increase capacity, the feedforward dimension was expanded from 256 to 1024, incorporating a Pre Norm architecture for training stability. Instead of relying on a standard CNN, the model employs a frozen MuQ model to extract high level features from input audio, which are then processed via a linear projection and LayerNorm before entering the decoder stack. The workflow involves processing audio through Mel-band Roformer for vocal extraction and RMVPE for absolute pitch inference, ultimately converting data into the kern format for transcription.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of this research?
The project aims to improve audio to score transcription, specifically moving beyond classical music to support popular music genres.
Q2. What is the SheetSage A2S dataset?
It is a new collection of data created using existing JSON annotations to provide audio and corresponding symbolic scores for popular music.
Q3. How well does the model perform?
It achieves a 4.98 percent symbol error rate on the Quartets dataset and 20.92 percent on the SheetSage A2S dataset.
Q4. What happens to the model architecture to handle more complexity?
The researchers expanded the feedforward dimension from 256 to 1024 and adopted a Pre Norm architecture.
Q5. How is the raw audio preprocessed before reaching the Transformer?
Vocal melodies are extracted using Mel-band Roformer and absolute pitch is inferred using RMVPE.
Q6. Does the model use standard CNN encoders?
No, the model replaces the vanilla CNN encoder with a frozen pre trained MuQ model.
Q7. What are the limitations regarding chord transcription?
The model has difficulty with chords beyond simple triads and struggles with chord inversion and quality labeling.
Q8. Are there specific timing issues identified in the transcription output?
Yes, the model struggles with misidentifying note offsets, consolidating consecutive notes, and incorrectly splitting single notes.
Q9. Does the model use any external data for training?
The paper mentions several datasets including Quartets, Chorales, MuseSyn, RISM, MARBLE, and the Million Song Dataset, though it focuses on the new SheetSage A2S collection.