Efficient Memory-Saving Transformers Through Recurrent Depth

Summary & Methodology Analysis

The Gated Recurrent Transformer improves architectural efficiency by partitioning the transformer into three distinct stages: a fixed prelude block, a shared core block iterated R times, and a coda block. This structure allows the model to gain expressive depth through repeated modulation of the internal state. During execution, the input to the shared core is concatenated with the hidden state from the previous iteration plus stochastic noise. A gated residual update, which uses an elementwise gate to control how much new information is absorbed into the hidden state, manages the flow of data across these recurrent steps. By reusing the weights of the core block, the model achieves significant parameter savings compared to standard designs that require unique weights for every layer.

Performance benchmarks demonstrate clear trade-offs between memory footprint and execution speed. At large scale, the architecture achieves a 62% reduction in parameters and 59% less peak decoding memory, though this comes with a 10% increase in compiled generation latency. When constrained to an iso FLOPs budget, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline. Furthermore, under an iso parameter constraint, the recurrent design achieves a 2.76 validation loss, outperforming a non-recurrent counterpart which reaches 2.84, suggesting that deeper recurrence effectively leverages fixed parameters for better predictive accuracy.

Despite these efficiency gains, the approach has notable limitations that impact production deployment. The recurrence depth R is fixed at inference time and does not support per-token halting, which limits the ability to dynamically adjust compute based on input complexity. Additionally, the specific gate bias and noise magnitudes used to train the model may require re-tuning for architectures outside of the GPT-2 family. Finally, while the paper validates the approach against competitors like Mixture-of-Recursions, the optimal fraction of layers that should be shared versus fixed remains an open question that requires further systematic study.