A Large Dataset of Agent Skills

Summary & Methodology Analysis

To address the lack of systematic data regarding agent skills, the researchers developed a methodology to crawl GitHub for SKILL.md files. Given the constraints of the GitHub code-search API, they partitioned the search space by file size to ensure complete retrieval. They then performed deduplication by grouping files by content hash to select unique representatives. This process resulted in a dataset containing 3,797,117 files from 282,200 repositories, capturing 1,877,981 distinct contents. Enrichment was applied to each representative file by capturing full text, parsed YAML front matter, folder contents, repository metadata, and sampled commit history. To ensure data privacy, the team performed anonymization by replacing commit author accounts with keyed one-way codes and redacting personal names and email addresses. The resulting dataset provides a foundation for analyzing how developers write and maintain instructions for language-model agents, which are systems that use a transformer (a deep learning model architecture that processes data sequences using self-attention mechanisms) to perform tasks. The collection is limited to files found on default branches in public repositories, excludes files larger than 384 KB, and only tracks certain active repositories. These factors mean the current dataset size serves as a lower bound for the total number of agent skills available on the platform.