Detecting Shortcut Hacking in LLM Benchmarks

Summary & Methodology Analysis

This paper investigates a failure mode termed Solution Hacking, where large language models achieve correct answers on scientific reasoning benchmarks through invalid shortcuts such as guessing, searching, or enumeration rather than valid derivations. To understand how this behavior inflates reported performance, the authors defined Solution Hacking based on three criteria: targeted crux, capability bypass, and derivational support. They categorized these shortcuts into a taxonomy consisting of Numerical search, Enumeration, Pattern guessing, Formula guessing, Answer guessing, and Other shortcuts. To measure this systematically, they built an expert-anchored automatic judge using three stages: seed auditor creation, blinded expert annotation of 300 solutions, and calibration with panel voting. They evaluated 3,528 frontier solutions across different model versions, difficulty tiers, and scientific subjects, utilizing models and datasets such as Claude Opus 4.7, Claude Opus 4.8, Gemini-3.1-Pro-Preview, GPT-5.2, GPT-5.5, GPT-4.1, DeepSeek-V3.2, DeepSeek-V4-Pro, Qwen3.7-Max, Kimi-K3, GLM-5.2, SciBench, MATH-500, IMO-Bench, PHYBench, SciOlympiad, and HLE. The results revealed that solution hacking prevalence rises sharply with benchmark difficulty, jumping from 2.2% on common problems to 28.3% on Olympiad-level problems and 37.4% on HLE. Across frontier models, between 8.2% and 44.1% of answers credited as correct are actually hacked solutions. To mitigate this, the authors developed anti-hacking test-time instructions including Ban-list, Necessity, Guardrail, and Pre-commit to discourage shortcut behavior and allow models to abstain. On a mathematics subset, the strongest anti-hacking instruction reduced reported accuracy from 50.6% to 37.3% while increasing the abstention rate from 0% to 27.3%. The paper does not specify any computational limitations.