Real Time Speech Enhancement With Mamba

Summary & Methodology Analysis

This paper addresses real-time, streaming speech enhancement under strict algorithmic latency and real-time factor constraints. The authors propose a fully causal Mamba-based architecture combined with progressive knowledge distillation to achieve efficient long-form sequence modeling on edge devices. Standard or existing mechanics involve computing complex short-time Fourier transform domain representations, deploying causal time-frequency Mamba blocks, propagating a fixed-size recurrent hidden state across time steps, updating convolution state buffers for depthwise one-dimensional convolutions, and maintaining a temporal frame buffer for past feature frames required by temporal convolutions.

The paper's own specific novel contribution is reformulating the non-causal SEMamba architecture into a fully causal RT-SEMamba operating online with a 1-frame-in and 1-frame-out mode and a 25 ms algorithmic latency constraint. It applies channel-wise LayerNorm with causal padding along the time axis, adds a multilayer perceptron after each causal time-frequency Mamba block, makes Time Mamba unidirectional along time without lookahead while keeping frequency-axis modeling bidirectional, and applies a progressive knowledge distillation scheme incorporating output-level loss for magnitude, phase, and complex outputs and intermediate feature distillation using sample-normalized aggregated blocks to compress an 8-layer teacher into a 1-layer or 2-layer student.

The key results show that the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint. The paper does not specify limitations, hardware requirements, or further performance metrics.