ContractScrub: A Benchmark for Legal Contracts
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- The benchmark introduces 3,014 annotated tasks spanning nine categories of drafting errors.
- It uses 44 contracts derived from the CUAD dataset, enriched with both existing and injected errors.
- Performance is measured using a custom metric based on normalized tuple matching compared against human-annotated gold standards.
- The top performing model, GPT-5.5, achieved a macro-average recall score of 0.750, though all F1 scores remained under 0.650.
Summary & Methodology Analysis
ContractScrub focuses on the final review process of transactional agreements. The methodology begins with 44 contracts sourced from the CUAD dataset. Researchers performed manual annotations to identify existing drafting errors and purposefully injected additional errors to ensure coverage across nine defined categories. This process resulted in a total of 3,014 annotated tasks, providing a standardized set of challenges for models designed to handle legal text review. The evaluation protocol requires models to return structured JSON, testing their ability to extract specific error data in a programmatic format. Performance is then determined by comparing these outputs against human-generated gold standards using normalized tuple matching, a custom evaluation metric designed for task-specific accuracy.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary purpose of ContractScrub?
It serves as a benchmark for evaluating how effectively AI models can identify errors like inconsistent language or incorrect cross-references in legal contracts.
Q2. What kind of documents does the benchmark use?
The dataset contains 44 English-language contracts, each approximately 10 to 15 pages in length.
Q3. How do researchers determine the model's accuracy?
They compare model-generated detections against human-annotated gold standards using a specific metric called normalized tuple matching.
Q4. How many error categories are included in the evaluation?
The benchmark covers nine distinct categories of drafting errors.
Q5. Which model performed best on this benchmark?
The GPT-5.5 model achieved the highest performance with a macro-average recall score of 0.750.
Q6. Did any models achieve high F1 scores?
No, all tested models recorded F1 scores below 0.650.
Q7. Does the requirement for JSON output affect performance scores?
Yes, the requirement for structured JSON output may depress performance scores due to the strict instruction-following constraints imposed on the models.
Q8. Are there limitations regarding the language of the contracts?
Yes, the dataset is limited to English-language contracts, which restricts its generalizability to other languages or legal traditions.
Q9. Is the size of the dataset large?
The paper describes the corpus of 44 contracts as modest in scale.