Generating Mixed Audio Scenes with Language Models

Summary & Methodology Analysis

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains challenging because current approaches typically rely on a disjointed pipeline with a frozen text encoder feeding a separate audio decoder. To solve this, the authors propose MiDashengLM-Gen, an end-to-end framework coupling a pre-trained sequential language model backbone with per-token conditional flow matching for autoregressive, variable-length mixed audio scene generation. The framework adopts the encoder of a custom MiDashengLM-0.6B model as a unified semantic-acoustic audio tokenizer, uses structured multi-view captions for supervision across six textual views, uses a pre-trained causal language model as the sequential LLM backbone, and applies conditional flow matching with a Diffusion Transformer-based decoder. To bridge the modalities, the system performs an audio-text alignment pre-training stage via an audio adapter that maps the tokenizer latents into the language model token space prior to generation training.

Several specific architectural rules and mechanisms govern the generation process. The system enforces a specific structural convergence prerequisite where the Diffusion Transformer decoder width must strictly exceed the audio latent dimensionality. Variable-length generation is determined via a learned stop head that predicts a binary continue or stop distribution over language model hidden states. Audio is finally decoded using a 5x convolutional upsampler followed by a Vocos vocoder. For supervision and evaluation, the pipeline leverages various datasets and models, including the Seed-TTS benchmark, MECAT, AudioCaps, MusicCaps, MiniMax-Speech, CV3-Eval, ACAVCaps, Emilia, LibriTTS, LJSpeech, AISHELL-3, WenetSpeech4TTS, TangoFlux, MusicGen, UniFlow-Audio, Dasheng AudioGen, and Qwen3-TTS.

Despite its strong performance on the Seed-TTS benchmark, where the English Word Error Rate drops from 12.15% to 2.79% to approach dedicated text-to-speech system performance at 1.24%, the approach has clear boundaries. Variable-length generation is bounded by the training data distribution ranging from 1 to 20 seconds. Furthermore, the model supports only coarse speaker-style control without voice cloning or explicit speaker identity conditioning.