Reusable Execution Experience for AI Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page 5 sections
Related concepts 6 concepts
Key Takeaways
- EvoMap converts successful task execution trajectories into reusable Genes that boost task completion rates.
- On 252 verifier-confirmed tasks, evolved Genes improved model performance by 8.7 to 15.5 percentage points compared to baseline methods.
- Using these Genes allowed Claude Opus to complete 39 more tasks while reducing solve-time token usage by 9.9 percent.
- The quality of the source model matters, as Opus-authored Genes outperformed Gemini-authored Genes by 4.4 to 11.7 percentage points.
Summary & Methodology Analysis
The researchers developed LongWoF-Bench, a benchmark containing 778 machine-verifiable tasks spanning code generation, agent-environment synthesis, mathematical reasoning, and rule following. The methodology involves a producer model, specifically Claude Opus, interacting with a verifier until a successful execution trajectory is achieved. This trajectory is distilled into a structured object termed a Gene, which acts as a form of external procedural experience. By offloading these successful strategies into an EvoMap, consumer models like Gemini 3.1 Pro Preview and MiniMax M3 can access this knowledge during one-shot inference without requiring additional verifier feedback.
The system shows measurable efficiency and performance gains. When deployed for Claude Opus, Gene reuse increased task completion by 39 tasks and achieved a 9.9 percent reduction in solve-time token consumption. Across seven evaluated models, Genes consistently outperformed standard methods by 8.7 to 15.5 percentage points. Furthermore, the provenance of the Gene is a significant factor in performance, as evidenced by the 4.4 to 11.7 percentage point advantage of Opus-authored Genes over those produced by Gemini. The authors note that reference-distilled Genes do not provide similar advantages, indicating that the value lies in the verified experience rather than just the compact representation of the solution.
Despite these improvements, the approach faces distinct constraints. The authors observe that EvoMap does not eliminate the total cost of discovery, as that cost must still be incurred initially and is then simply amortized over subsequent runs. Additionally, Genes are not a universal fix for poor underlying reasoning. The paper reports that when the primary bottleneck is a model's intrinsic mathematical reasoning capability, Genes are unable to fully compensate for these limitations. Finally, the utility of the approach is highly dependent on having verifier-confirmed trajectories available for the distillation process.
Interactive System Flowchart
Illustrative Implementation
A short sketch of the paper's core idea, not the authors' own code.
# Illustrative sketch (not from the paper)\nimport torch\n\n# Mock benchmark (LongWoF-Bench)\ntasks = [\"t1\"]\n\n# Producer model placeholder\ndef produce(state):\n return {\"act\":\"step\",\"state\":state+1}\n# Verifier placeholder\ndef verify(traj): return len(traj)>=3\n# Distill gene from successful trajectory\ndef distill(traj): return {\"gene\":traj[-2:]}\nEvoMap={}\nfor t in tasks:\n s=0; traj=[]\n for _ in range(5):\n step=produce(s); traj.append(step); s=step[\"state\"]\n if verify(traj):\n EvoMap[t]=distill(traj); break\n# Consumer reuses gene\ndef consume(t):\n g=EvoMap.get(t)\n return f\"Done {t}\" if g else \"Fallback\"\nprint(consume(\"t1\"))// Illustrative sketch (not from the paper)\nconst tasks = ['t1'];\nfunction produce(s){return{act:'step',state:s+1};}\nfunction verify(traj){return traj.length>=3;}\nfunction distill(traj){return{gene:traj.slice(-2)};}\nconst EvoMap={};\nfor(const t of tasks){\n let s=0,traj=[];\n for(let i=0;i<5;i++){\n const step=produce(s);traj.push(step);s=step.state;\n if(verify(traj)){EvoMap[t]=distill(traj);break;}\n }\n}\nfunction consume(t){return EvoMap[t]?`Done ${t}`:'Fallback';}\nconsole.log(consume('t1'));Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of this research?
The research aims to improve task completion in complex workflows by treating verified execution experience as a reusable asset to avoid repeating past failure modes.
Q2. What is a Gene in the context of EvoMap?
A Gene is a structured distillation of a verifier-confirmed execution trajectory that captures critical knowledge such as strategies, corrections, and failure guards.
Q3. Can any model use these Genes?
Yes, Genes are designed as external procedural experience and can be transferred between different model families, including Claude Opus, Gemini 3.1 Pro Preview, and MiniMax M3.
Q4. How many tasks are included in the LongWoF-Bench?
The benchmark includes 778 machine-verifiable tasks across various categories such as code generation and mathematical reasoning.
Q5. Do Genes work if they are just distilled from a reference solution?
No, the paper notes that reference-distilled Genes do not exhibit the same performance advantage as those derived from verifier-confirmed trajectories.
Q6. Can Genes fix a model that is bad at math?
No, Genes cannot fully compensate for underlying model reasoning limitations, especially in mathematical reasoning tasks.
Q7. Does this method eliminate the cost of finding a solution?
No, it does not eliminate the cost of discovery; it allows that cost to be amortized across subsequent executions.
Q8. What was the performance difference between Opus-authored and Gemini-authored Genes?
Opus-authored Genes outperformed Gemini-authored Genes by 4.4 to 11.7 percentage points across a common set of 180 tasks.
Q9. Are there any specific hardware requirements mentioned for running this?
The paper does not specify any hardware requirements.