Real Time LLM Forecasting Evaluation

Summary & Methodology Analysis

The researchers addressed the prevalent issue of data leakage in LLM benchmarks by designing WorldCup Arena. Traditional retrospective evaluations often include test data in the model training set, leading to inflated performance scores. This framework ensures validity by using daily retrieval of news, injury reports, and betting odds to build a rolling dossier for 48 teams. The system enforces a snapshot lock, which restricts model access to only the information available before a specific match, and a kickoff lock, which mandates that all predictions be submitted before the match starts. Predictions were generated using a fixed, pre-defined prompt template covering seven betting markets per match. Evaluations were scored against objective, official 90-minute scoreline results. The evaluation pipeline utilized 46 timestamped snapshots over a 39-day period to capture real-time forecasting capability without the risk of future information bleeding into the inference process. Models were tested on their ability to reason through dynamic event variables, but the study remains constrained to a single, non-repeatable event. Furthermore, the evaluation was limited to one prediction choice per market per match, potentially restricting the granular assessment of probability distribution handling. While the paper provides a robust mechanism to mitigate contamination in forecasting tasks, the scope is localized to this specific tournament format.