MMLU

Explainer

MMLU (Massive Multitask Language Understanding) is a knowledge-and-reasoning benchmark that measures a model's accuracy on multiple-choice questions drawn from 57 subjects spanning the humanities, STEM, the social sciences, and professional and legal domains. Each item presents a question with answer options, and a model's score is reported as percent accuracy on a 0–100 scale. In Policy Window's catalog, MMLU sits in the general-reasoning domain, reflecting its design intent: rather than probing a single skill, it samples broadly across fields a knowledgeable generalist might be expected to handle, from elementary mathematics and US history to law and clinical medicine.

The benchmark originated with Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt in "Measuring Massive Multitask Language Understanding," released in 2020 (arXiv:2009.03300) and presented at ICLR 2021 (https://arxiv.org/abs/2009.03300). The authors framed it as a test of what they describe as requiring extensive "world knowledge and problem solving ability" acquired largely during pretraining, evaluated in zero- and few-shot settings rather than after task-specific fine-tuning. Over the following years MMLU became the de facto default for reporting general capability: it appeared in nearly every frontier model release and system card, which made it a convenient common yardstick precisely because so many labs reported it.

That ubiquity is now also its central limitation. MMLU is, in the catalog's terms, saturated: leading frontier models cluster in roughly the high-80s to low-90s percent accuracy, and small gaps near the ceiling no longer reliably separate frontier systems from competent mid-tier ones. Two distinct problems compound the saturation. First, the benchmark has a high contamination risk: because MMLU questions are widely published on the open web, they can leak into the very pretraining corpora used to build the models that are later evaluated on them, which inflates measured scores in ways that are difficult to disentangle from genuine capability. This concern is well documented in the evaluation literature — Sainz et al., "NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark" (Findings of EMNLP 2023, https://aclanthology.org/2023.findings-emnlp.722/), argues that test-set contamination can undermine the validity of benchmark results and calls for routine contamination auditing. Second, even setting contamination aside, a four-option multiple-choice format with a saturated ceiling has limited headroom to discriminate among the strongest models; the four-option format also permits a 25 percent chance baseline and measures selection from given options rather than open-ended generation, so a model can identify the right answer without being able to produce it.

These pressures motivated a harder successor. MMLU-Pro — Wang, Ma, Zhang, and colleagues, "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" (2024, arXiv:2406.01574, https://arxiv.org/abs/2406.01574), published at the NeurIPS 2024 Datasets and Benchmarks Track — expands the answer set from four to ten options, adds more reasoning-intensive items, and removes trivial and erroneous questions. The authors report that these changes drop accuracy by roughly 16 to 33 percent relative to MMLU while improving stability across prompt variations, restoring discriminative headroom. Policy Window records MMLU-PRO (slug mmlu-pro) as MMLU's catalog successor.

What an MMLU score does and does not establish is worth stating plainly. A high score evidences broad recall of factual and academic knowledge under a constrained multiple-choice format; it is not, on its own, evidence of open-ended reasoning, reliable tool use, or agentic competence in realistic tasks, and contamination can inflate the headline number relative to true held-out ability. For governance and procurement readers, MMLU is therefore best read as one coarse, increasingly saturated signal among many rather than a summary verdict on capability. Policy Window holds this editorial read provisionally: as contamination-auditing methods and successor benchmarks mature, the appropriate weight to place on MMLU may shift.