Verified Financial LLM Reasoning Benchmark

Summary & Methodology Analysis

To address the limitations of existing benchmarks that rely on manual annotation or heuristic extraction, the authors constructed V-FiLLM by building synthetic financial data in two formats, namely 10-Q filing-like tables and regularized financial sheets. They generated typed financial atoms from spreadsheet cells, storing numerical values with semantic metadata, and sampled symbolic expression templates in the form of typed binary trees with controllable depth, breadth, and financial concept complexity. These expression trees were grounded in concrete spreadsheet atoms using a binding environment and rendered into natural language questions using a bottom-up semantic procedure. Lightweight linguistic data augmentation was applied to questions while preserving the underlying computation graph, and the symbolic expression was evaluated directly to generate an automatically verified ground-truth answer. Single-turn questions were also converted into multi-turn dialogues by mapping parenthesized sub-expressions to ordered turns, and adversarial perturbations such as missing values, garbage values, OCR look-alikes, cross-sheet contamination, and distractor information were applied to table cells.

For model training and evaluation, the authors utilized LoRA, which stands for low-rank adaptation, a technique that fine-tunes base models efficiently by updating only a small subset of parameters. They fine-tuned base models using LoRA on verified chain-of-thought traces, which are step-by-step reasoning paths generated from the framework. Gemma-31B achieved 98.4% and 97.6% accuracy on 10-Q and simplified statements respectively, outperforming GPT-OSS-120B and Qwen3.7-Plus. However, accuracy on simplified statements declined from 85.0% at 6 steps to 55.0% at 8 steps for Gemma-31B. Furthermore, LoRA fine-tuning improved Qwen3.5-4B performance on the held-out benchmark from 81.1% to 85.6%, and the fine-tuned model achieved 32 out of 100 correct answers on FinQA, up from 27 out of 100 for the baseline.

Despite these results, the paper notes several limitations. The natural-language questions may contain translation ambiguities or lack sufficient context. Additionally, evaluation is constrained by limited compute, covering only six models and English-language documents, while the LoRA fine-tuning experiments were restricted to a small evaluation set of 90 problems. The paper does not specify compute hardware details, training time, or inference latency costs.