Why AutoML Performance Results Look Inflated
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Key Takeaways
- Protocol defects involving how models are selected and how time is measured lead to inflated performance metrics in AutoML comparisons.
- Correcting the testing process reduced the win rate of the Orcetra engine from 57.1 percent to 34.3 percent on a subset of datasets.
- Peeking at the test set during model selection, rather than using a validation split, contributed 4.8 percentage points to the inflated win rate.
- No significant performance difference remained between the analyzed frameworks once the protocol was corrected.
Summary & Methodology Analysis
The paper investigates the validity of AutoML benchmarks running on short time budgets, specifically focusing on the Orcetra engine compared against FLAML and AutoGluon. The researchers identified three critical flaws in standard evaluation protocols: testing models on the test set, failing to enforce strict time budgets, and inconsistent merging of separate result sweeps. To address these, the team developed a corrected protocol that enforces an external deadline for the search process and pins frameworks to an equal share of machine compute resources. Crucially, they modified the search loop to select models based on a validation split carved from the training data, rather than the test split, to ensure the test set remains untouched during the search process.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem identified by the paper?
AutoML comparisons running at short time budgets are often miscalculated due to protocol errors, resulting in inflated performance figures for some systems.
Q2. Which frameworks were compared in this research?
The paper compared Orcetra, FLAML, and AutoGluon, while referencing other frameworks like Auto-WEKA, auto-sklearn, TPOT, and H2O AutoML.
Q3. Did the corrections change the performance rankings?
Yes, once the protocol was corrected, no significant performance differences remained between the frameworks, and Orcetra's win rate dropped from 57.1 percent to 34.3 percent.
Q4. What role did test set peeking play in the results?
Selecting models by peeking at the test set rather than using a validation split accounted for 4.8 percentage points of the inflated win rate.
Q5. How did the researchers ensure fair compute time?
They enforced an external deadline for the search process to prevent budget overshoot and pinned all framework executions to an equal share of machine compute resources.
Q6. What datasets were used for the evaluation?
The initial analysis used 513 OpenML datasets, while the corrected re-run was performed on a random subsample of these datasets.
Q7. Were there limitations in the evaluation methodology?
Yes, the study used a single 80/20 holdout split per dataset instead of standard 10-fold cross-validation, and AutoGluon was run without its optional fastai dependency.
Q8. How was the testing protocol corrected?
The researchers implemented a search loop that selects models using a validation split carved from the training data instead of the test split.
Q9. Does the paper mention the impact of specific hardware on these results?
The paper does not specify the exact hardware used, only that they pinned framework executions to an equal share of machine compute resources.