Checking LLM Generated Trading Strategy Reliability

Summary & Methodology Analysis

The FIDES protocol aims to bridge the gap between an LLM's natural language rationale, its executable code, and actual market outcomes. The methodology employs dual delivery where a single prompt triggers the generation of both a strategy logic description and a self-contained strategy function. This code is then executed within a sandbox environment, and researchers perform a lag-one out-of-sample backtest to prevent look-ahead bias, ensuring that model positions are calculated using only prior market data rather than future information. Finally, the system measures concordance by scoring the alignment between the model's intent, the actual code execution, and the final empirical result.

The study evaluated four distinct models, including gpt-4o-mini, gpt-4o, gpt-4.1-mini, and qwen-2.5-7b-instruct, using a single seed at a temperature of 0.2. The team faced significant infrastructure constraints during the evaluation process, as many targeted models, such as Claude, Gemini, DeepSeek, Llama-70B, and Qwen-72B, were inaccessible through their inference catalog. Due to these limitations, the authors emphasize that they report observed effects rather than statistical confidence intervals based on their sample size of 40 strategies.

Key technical limitations were identified throughout the study, particularly regarding the validation tools. The researchers attempted to implement a runtime future-information probe intended to detect if code was gaming the backtest, but the probe failed to trigger on both clean and injected code samples. This failure is categorized by the authors as a limitation of the probe itself rather than evidence that the generated code was clean. Additionally, the small sample size and reliance on a single specific inference setup restrict the generalizability of these findings, though they provide a concrete look at the current reliability of LLMs in financial logic generation.