Retrieval Augmented Generation for Time Series

Summary & Methodology Analysis

The TS-RAG architecture addresses the failure of traditional time series models to leverage external data by introducing a retrieval-based pipeline. The process begins by identifying relevant historical segments from a database using a similarity-based function. Input sequences and these retrieved segments are then converted into non-overlapping patches. A key contribution is the use of learnable reference tokens, which are prepended to the input sequence to represent broader historical patterns before computing embeddings and positional encodings. This structure allows the model to treat external historical context as a first-class feature rather than just appending raw data to the input stream. For the model layers, TS-RAG employs self-attention to capture internal temporal dependencies, while cross-attention is specifically utilized to integrate the retrieved sequences into the current input. The final hidden representations are processed through a standard feed-forward network to generate the forecast. The CM variant further improves results by explicitly modeling inter-channel dependencies, leading to an average MSE of 0.310 and MAE of 0.348. The performance impact is most pronounced at longer horizons, where it achieves up to an 18.2% reduction in MSE for 720-step predictions compared to simple concatenation. However, the model has limitations. Performance can degrade if excessive sequences are retrieved due to noise and redundancy, suggesting an optimal limit on reference data. Additionally, the standard TS-RAG variant lacks the explicit inter-channel modeling found in the CM version.