Evaluating Large Language Models Trained on Code

Summary & Methodology Analysis

This paper addresses the challenge of program synthesis and the evaluation of code-writing capabilities in large language models trained on code. Specifically, it focuses on generating standalone Python functions from docstrings and measuring functional correctness. The methodology involves assembling each evaluation problem into a prompt consisting of a header, a signature, and a docstring. Tokens are sampled from the language model until encountering specific stop sequences using nucleus sampling, which is a method for generating text by sampling from the subset of tokens whose cumulative probability exceeds a threshold. Generated code samples are then evaluated automatically through unit tests in a secure sandbox environment. Pass@k metrics are computed using an unbiased estimator based on the number of correct samples and total samples generated.

Key results demonstrate the performance differences across models and strategies. Codex solves 28.8% of the problems on HumanEval with a single sample, while GPT-3 solves 0% and GPT-J solves 11.4%. Repeated sampling allows Codex-S to solve 70.2% of problems with 100 samples per problem. Furthermore, Codex-S solves 37.7% of problems with a single sample after fine-tuning on standalone, correctly implemented functions, where fine-tuning is the process of adjusting pre-trained model weights on a specific downstream dataset.

Despite these results, the paper identifies several limitations. Codex has difficulty with docstrings describing long chains of operations. It also has difficulty with binding operations to variables. Additionally, Codex is not sample efficient to train, meaning it requires a large amount of data or compute relative to the performance gained.