Saving In-Context Learning State Between Sessions

Summary & Methodology Analysis

The paper approaches the problem of state persistence by framing it as a pre-query state-coding challenge. Instead of attempting to save an entire context window, the method uses Bayesian decision theory to identify a compact handover record. This record balances the need to reproduce earlier inputs against the requirement to preserve the target distribution for future queries. The handover record acts as a serialized snapshot of the model's in-context learning state, which is defined by three components: exact constraints, evidence-based statistics, and any residual observations that do not fit into simplified statistical buckets. For linear Gaussian regression, the framework proves that storing the Gram matrix and the vector of input-output products provides predictive sufficiency, meaning this record captures all necessary information from the prior session to inform future predictions under Gaussian noise. For more complex scenarios, the authors utilize a nonparametric approach based on quantized, cell-based counts and sums, which allows for finite storage management. The process relies on deterministic checks to ensure the validity of the serialized data, while the associated overhead is attributed entirely to the chosen writing method. A significant limitation is the assumption of query exogeneity, where future queries are treated as independent of previous context, which simplifies the model into a quotient space. Furthermore, the reliance on Gaussian sufficient statistics requires prior knowledge of the noise variance. Finally, the authors note that for interactive agent tasks, the definition of predictive sufficiency is more complex because agent actions can alter future observations, making the one-step regression model insufficient for long-term state maintenance.