Pass@k
pass@k is an evaluation metric measuring the probability that at least one of k generated code samples passes a suite of unit tests.
What it is
To calculate pass@k, you generate k code snippets for a given prompt using an LLM with non-zero temperature to ensure diversity. These candidates are executed against a set of functional unit tests in an isolated sandbox. If at least one snippet passes all assertions, the problem is considered solved. For complex tasks, k is often set to values like 10, 100, or even 1000 to measure the model's upper potential rather than its single-shot accuracy.
Why it matters
Understanding pass@k helps you calibrate the compute budget for your AI-powered features. If you rely on a single-shot generation that fails frequently, you can shift to a generate-and-verify workflow where you run multiple samples and discard the failures. Ignoring this metric leads to overestimating model reliability when the reality is that the model needs multiple attempts to produce valid, executable code.
In practice
In production, this manifests as your system's retry logic or reranking pipeline. You might set a high 'n' parameter in your inference API to generate multiple candidates and then use a script or a secondary 'judge' model to select the one that passes tests or satisfies constraints before returning it to the user.
The tradeoff
Increasing k improves the probability of success but linearly increases your latency and token costs per request.