Benchmarking LLMs on Scientific Diagram Tasks
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- Diagram-MMU evaluates models on foundational abilities and four agentic dimensions, including tool use and planning.
- Models demonstrate strong performance in reasoning tasks with up to 86 percent accuracy, but struggle significantly with parsing and editing.
- Planning is currently the weakest agentic skill, with models showing accuracy drops as high as 8.8 percent on question answering tasks.
- Claude-4.6 Opus is identified as the only model that consistently improves across all three evaluated task categories in agentic workflows.
Summary & Methodology Analysis
Diagram-MMU is a comprehensive benchmark focused on scientific diagram analysis, incorporating 6,849 TikZ source code samples sourced from official documentation and community resources. The construction process involved filtering and compilation verification via standard LaTeX engines (pdflatex, lualatex, or xelatex) to ensure visual fidelity. The dataset is structured into 3,744 unique diagrams categorized across six domains: charts, planar geometry, 3D shapes, graphs, chemistry, and circuits. To support agentic evaluation, the researchers deployed a TikZ search tool as an MCP server, which provides models with selective, documentation-based syntax references during execution.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is Diagram-MMU?
It is a benchmark designed to assess Multimodal Large Language Models on scientific diagram parsing, editing, and question answering.
Q2. What specific skills does the benchmark test?
It tests both foundational diagram understanding and agentic capabilities like context utilization, tool use, state management, and planning.
Q3. How were the diagrams in the benchmark verified?
The diagrams underwent compilation checks using pdflatex, lualatex, or xelatex to ensure the rendered images matched the source code.
Q4. How does model performance vary by task type?
Models achieve high reasoning accuracy up to 86 percent, but demonstrate lower performance in parsing, with F1 scores ranging from 31 to 57 percent.
Q5. What role does the TikZ search tool play in the evaluation?
The tool functions as an MCP server, allowing models to reference specific syntax documentation during the evaluation process.
Q6. How many models were evaluated in this study?
The study evaluated 12 Multimodal Large Language Models.
Q7. What are the limitations regarding the scope of the diagram languages tested?
The benchmark is limited to TikZ code and does not cover other diagram representation languages like Python or SVG.
Q8. Which model showed the most consistent performance in agentic settings?
Claude-4.6 Opus was the only model to show consistent improvement across parsing, editing, and reasoning tasks in agentic settings.
Q9. Does the benchmark cover all aspects of agentic diagram creation?
No, it does not evaluate full agentic abilities such as text drafting or citation management.