Back to Feed
Agents / Benchmarks & Evals

Improving Agent Skills Through Contrastive Distillation

Original: SKILL-KD: Contrastive Skill Distillation for LLM Agents
By Qiming Shi, Yibo Dou, Jiawen Zhu, Yulong Tao, Linbo Jin, Zhaolu Kang, Yunfan Zhou, Di Weng Published 2026-08-04

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • Increases student agent performance, specifically raising Qwen3.5-4B from 43.5 percent to 66.8 percent across five benchmarks.
  • Boosts performance for larger student models like Qwen3.6-35B-A3B from 57.9 percent to 74.6 percent.
  • Utilizes a trace-linked skill library that records rationale and edit history for every procedural adjustment.
  • Implements a Drift-Aware Skill Consolidation process to manage the lifecycle of skill patches by reviewing historical evidence.

Summary & Methodology Analysis

SKILL-KD addresses the difficulty of enabling frozen LLM agents to acquire reusable procedural skills. Instead of conventional fine-tuning, which modifies internal model weights, the method treats skills as textual rules. Each skill is represented as a structured object containing a title, content, rationale, and trace. When a student agent fails a task, the system compares the student trajectory to a successful teacher trajectory. This contrastive analysis identifies the specific behavioral gap, allowing the system to propose a targeted skill patch.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary goal of SKILL-KD?

The goal is to enable frozen LLM agents to acquire reusable procedural skills from more capable teacher models.

Q2. Does this method require retraining the model?

No. The method focuses on providing actionable guidance to frozen agents, meaning the underlying model parameters are not updated through gradient-based training.

Q3. What kind of performance gains were observed?

The Qwen3.5-4B model improved from 43.5 percent to 66.8 percent, and the Qwen3.6-35B-A3B model improved from 57.9 percent to 74.6 percent.

Q4. How are skills stored and managed in the library?

Skills are stored as trace-linked textual rules that include a title, content, rationale, and trace, with a persistent edit history for all modifications.

Q5. What happens during the Drift-Aware Skill Consolidation phase?

The system reviews historical context and trace-linked evidence to determine whether to add, modify, delete, or skip specific skill patches.

Q6. Does SKILL-KD work with any teacher model?

The paper uses models like Qwen3.7-plus and ChatGPT-5.5 as teachers, though it emphasizes the need to contrast student failures with teacher demonstrations.

Q7. Are there any limitations to this approach?

The primary limitation is that the framework relies on the availability of a successful or partially successful teacher trajectory to contrast against the student failure.

Q8. What benchmarks were used to test the system?

The system was evaluated using SearchQA, SpreadsheetBench, DocVQA, LiveMath, and ALFWorld.

Q9. How does this compare to existing agent distillation methods?

The paper compares its approach against methods like Trace2Skill, SkillOpt, Reflexion, ExpeL, and AgentDistill.