Measuring Multitask Language Understanding Across Subjects

Summary & Methodology Analysis

To address the disconnect between existing benchmarks and actual model capabilities, the authors designed the Massive Multitask Language Understanding benchmark, known as MMLU. This benchmark measures knowledge acquired during pretraining, which is the initial unsupervised training phase where a transformer model learns general language patterns from massive text corpora, by evaluating models exclusively in zero-shot settings, where a model performs a task without prior task-specific training examples, and few-shot settings, where a model is given a small number of demonstration examples before generating an answer. The dataset covers 57 subjects across STEM, the humanities, the social sciences, and other areas, ranging from elementary to advanced professional difficulty. The authors manually collected 15,908 multiple-choice questions from freely available online sources, including practice questions for the Graduate Record Examination, United States Medical Licensing Examination, undergraduate courses, and Oxford University Press books. The collection was split into a few-shot development set containing 5 questions per subject, a validation set with 1,540 questions, and a test set with 14,079 questions, ensuring each subject contains a minimum of 100 test examples.

For evaluation, prompts were structured by beginning each prompt with the text indicating the subject, adding up to 5 demonstration examples with answers for few-shot evaluation, appending the question, and ending all prompts with the answer prefix. Model performance was measured by treating the highest probability option among the tokens A, B, C, and D produced by the model as the prediction and computing the classification accuracy across all examples and tasks in the test. The authors also evaluated the calibration of GPT-3 by testing how well its average confidence estimates its actual accuracy for each subject. Results showed that few-shot models up to 13 billion parameters achieve a random chance performance of 25 percent accuracy, whereas the 175 billion parameter GPT-3 model reaches 43.9 percent accuracy. Meanwhile, unspecialized humans from Amazon Mechanical Turk achieve 34.5 percent accuracy, and expert-level accuracy is estimated to be approximately 89.8 percent. The largest UnifiedQA model, utilizing 11 billion parameters, attained 48.9 percent accuracy, and even the smallest UnifiedQA variant with 60 million parameters achieved approximately 29.3 percent accuracy.

Despite these results, the evaluation revealed major limitations across current large-scale models. On every one of the 57 tasks, the best models still require substantial improvements to reach expert-level accuracy, exhibiting lopsided performance and frequently lacking awareness of their incorrect predictions. For GPT-3, 9 out of the 10 lowest accuracy tasks were STEM subjects emphasizing mathematics or calculations, and models showed near random accuracy on certain socially important subjects like morality and law. Furthermore, models are poorly calibrated, with GPT-3's average confidence differing from its actual accuracy by up to 24 percent, and elementary mathematics exhibiting an RMS calibration error of 19.4 percent in the zero shot setting. Even in the few shot setting, models remain miscalibrated, with the gap between accuracy and confidence reaching up to 14 percent. The paper does not specify compute training costs or hardware runtimes.