Automated Terminal Task Synthesis Framework
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 3 concepts
Key Takeaways
- FACET generates 6,078 validated tasks, each averaging 22.77 executable tests to ensure high verification coverage.
- The FACET-Terminal-Qwen3.5-27B model improved its Terminal-Bench 2.1 score by 6.75 points, reaching 47.57.
- The framework employs a manifest-based planning approach to materialize container environments, maintaining strict alignment between source intent and execution.
- Validation and repair mechanisms use execution traces to fix specific artifacts without requiring full task regeneration.
Summary & Methodology Analysis
FACET addresses the drift between generated task components (instruction, environment, solution, verifier) by adopting a state-grounded construction process. It begins by collecting and filtering agent skills, followed by a model-based judge that selects relevant combinations to build a five-dimensional representation: goal, context, capability, state, and I/O-tool. The environment materialization phase uses a manifest-based planning approach to populate the necessary dependencies and services within a container, ensuring the runtime environment matches the intended agentic scenario.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of the FACET framework?
FACET aims to preserve source intent and cross-artifact consistency while synthesizing terminal tasks that are both information-rich and executable.
Q2. Does FACET improve agent performance on benchmarks?
Yes, the FACET-Terminal-Qwen3.5-27B model achieved a score of 47.57 on Terminal-Bench 2.1, which is a 6.75 point increase over its base model.
Q3. How many tasks does the FACET dataset provide?
The dataset includes 6,078 validated tasks.
Q4. What is the role of the manifest-based planning approach?
It is used during environment materialization to populate files, dependencies, and services, ensuring the container state is properly configured for the task.
Q5. How does FACET handle errors in task artifacts?
It uses execution traces to perform targeted repairs, fixing specific artifacts without the need to regenerate the entire task bundle.
Q6. How do FACET task success rates compare to other benchmarks?
FACET tasks exhibit lower pass rates, specifically 27.00 P@1 and 35.00 P@3, compared to other datasets because agents must satisfy every required condition across multiple checkpoints.
Q7. Does the dataset contain a high volume of trajectories?
FACET contains 1.2K training trajectories, which is fewer than benchmarks like Nemotron-Terminal or Terminal-Lego, though its trajectories are comparatively longer.
Q8. What model powered the generation of rollouts for the 6K tasks?
The Terminus-2 agent, powered by DeepSeek-V4-Pro, was used to generate rollouts.
Q9. What is the main limitation regarding task completion in FACET?
The lower success rates are linked to the high number of checkpoints required for task completion, as completing the main workflow is insufficient if any individual requirement is unmet.