Long Short Term Memory Networks

Summary & Methodology Analysis

Traditional recurrent neural network architectures suffer from a core problem where they struggle to store information over extended time intervals. This failure happens because of insufficient and decaying error backflow during training, making it difficult for models to learn dependencies that span long sequences. The paper reviews Hochreiter's analysis of this decaying error backflow and builds a solution designed to maintain a stable gradient signal over thousands of discrete time steps.

To solve the decay issue, the authors implement constant error carousels inside special units that enforce a constant error flow. They combine these carousels with multiplicative gate units, which act as dynamic controllers that learn when to open and close access to the constant error flow. Furthermore, the gradient is truncated where it does not cause harm. This combination enables the network to successfully bridge time lags in excess of 1000 discrete time steps, vastly outperforming alternative architectures like real time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking in both success rate and learning speed. Additionally, the computational complexity of the long short term memory model sits at O(1) per time step and weight.

The primary limitation of this approach is tied to its gradient handling. Specifically, the method involves the truncation of the gradient, which must be carefully applied only where it does not cause harm to the training process.