Evaluating Large Language Models Trained on Code
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Fine-tuning a GPT language model on publicly available code from GitHub creates the Codex model.
- Codex achieves a 28.8% success rate on the HumanEval benchmark, compared to 0% for GPT-3 and 11.4% for GPT-J.
- Repeated sampling allows the model to solve 70.2% of problems using 100 samples per prompt.
Summary & Methodology Analysis
The paper addresses the challenge of evaluating the functional correctness of large language models in synthesizing computer programs from natural language docstrings. The methodology involves fine-tuning a GPT language model on publicly available code from GitHub to create the Codex model. To evaluate performance, the authors use a new benchmark called HumanEval, which is designed to measure program synthesis performance. Additionally, they apply repeated sampling as a strategy to generate working solutions for difficult prompts.
Codex achieves a 28.8% success rate on HumanEval, compared to 0% for GPT-3 and 11.4% for GPT-J. Furthermore, the use of repeated sampling allows the model to solve 70.2% of problems using 100 samples per prompt. The models and datasets involved in this research include Codex, GPT, GitHub, GitHub Copilot, HumanEval, GPT-3, and GPT-J.
Despite these results, the model faces certain limitations. Specifically, the paper notes that the model struggles with docstrings that involve long operation chains and binding operations to variables. The paper does not specify any other architectural limitations or computational costs beyond this finding.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the core challenge addressed in the paper?
The paper addresses the challenge of evaluating the functional correctness of large language models in synthesizing computer programs from natural language docstrings.
Q2. What is Codex?
Codex is a model created by fine-tuning a GPT language model on publicly available code from GitHub.
Q3. What is HumanEval?
HumanEval is a new benchmark designed to measure program synthesis performance.
Q4. What success rate did Codex achieve on HumanEval?
Codex achieves a 28.8% success rate on HumanEval.
Q5. How did GPT-3 and GPT-J perform on HumanEval in comparison?
GPT-3 achieved a 0% success rate and GPT-J achieved an 11.4% success rate on HumanEval.
Q6. What strategy was used to help the model solve difficult prompts?
The authors applied repeated sampling as a strategy to generate working solutions for difficult prompts.
Q7. How effective is repeated sampling for solving problems?
Repeated sampling allows the model to solve 70.2% of problems using 100 samples per prompt.
Q8. What specific limitations does the Codex model have?
The model struggles with docstrings that involve long operation chains and binding operations to variables.
Q9. What models or datasets are mentioned in the paper?
The mentioned models or datasets are Codex, GPT, GitHub, GitHub Copilot, HumanEval, GPT-3, and GPT-J.