MMLU is at least 29% contaminated, according to analysis published in 2024 — meaning roughly a third of its questions were findable in models’ pretraining corpora before evaluation began. EvoEval, which rewrites HumanEval problems using seven types of semantic and syntactic transformation, found top models’ pass@1 rates dropping by 19 to 47 percentage points compared to the originals — a contamination fingerprint: the models had memorized solutions, not learned to solve problems. These are not edge cases or methodological quibbles. They are evidence that two of the most widely cited benchmarks in LLM evaluation have been producing inflated, unreliable scores for the frontier models most in need of rigorous assessment.

The problem is structural. Pretraining corpora for large models span hundreds of billions of tokens scraped from the internet. Benchmark datasets, by virtue of being published and discussed, also end up on the internet. Decontamination — removing benchmark-adjacent text before training — requires knowing which benchmarks will matter before training concludes. In practice, models are trained, then evaluated against benchmarks that were online before the training run finished. The gap cannot be closed by good intentions alone. Earlier coverage on this site walked through what MMLU, HELM, GSM8K, and HumanEval actually measure at a general level; this piece goes deeper on why contamination specifically undermines those measurements and what a credible pipeline does about it.

What Contamination Actually Does to Benchmark Scores

Contamination inflates scores by substituting memorization for generalization. When a model’s pretraining data includes a benchmark question — or discussions of it, answer keys, academic papers describing it, forum threads where users posted their answers — the model is not reasoning to an answer at inference time. It is pattern-matching against statistical residue from training.

The practical consequence is that a 2–3 point gap between two models on MMLU may reflect a genuine capability difference, a data preprocessing difference, or a contamination artifact. From the outside, these are almost always indistinguishable.

The contamination trajectory follows a predictable arc. A new benchmark launches with genuinely clean data. Researchers cite it; model cards reference it; practitioners discuss it on forums. Within one to two training cycles — roughly two to three years for high-visibility benchmarks — enough adjacent material has accumulated online that training corpus decontamination becomes unreliable. The benchmark has, in effect, joined the training data. This is not a solvable problem with current static benchmark designs; it is inherent to the evaluation methodology.

MMLU-CF, a contamination-free reformulation published on arXiv (arXiv:2412.15194), attempts to mitigate this through systematic paraphrasing, back-translation, and surface-level permutation. The approach reduces direct memorization artifacts but cannot fully eliminate the underlying issue: if the benchmark’s domain knowledge has permeated the pretraining corpus, surface changes still leave models advantaged.

Why Frontier Models Are Most Affected

Contamination is not uniformly distributed across the capability range. Frontier models — the ones most often compared on public leaderboards — are precisely the models that have ingested the largest pretraining corpora, are most likely to have encountered benchmark-adjacent text, and whose performance differences are smallest and most contested.

A model with a 30% gap below the leaderboard leader is unlikely to have its ranking changed by contamination. A model that differs from the leader by 1–2 points is an entirely different situation. The comparisons that matter most for research and procurement decisions are the ones most sensitive to contamination artifacts.

The evaluator-as-judge approach compounds this problem. MT-Bench and similar frameworks use GPT-4 to score model outputs on conversational tasks. The correlation with human preferences is reasonable, but the approach introduces circularity: a model trained on data that reflects GPT-4’s output style will score well on GPT-4-judged evaluations independently of actual quality. When the judge and the candidate share significant training signal, the evaluation is partially circular.

What LiveBench and Dynamic Benchmarks Get Right

LiveBench, which appeared as a Spotlight Paper at ICLR 2025, takes a different architectural approach: questions are derived from recently released sources — arXiv papers, news articles, IMDb synopses — with new questions released monthly. Because questions are based on post-training material, they cannot have appeared in models’ pretraining data. Answers are objectively verifiable without a model judge, which removes the circularity problem.

The benchmark currently covers 23 tasks across 7 categories including math, coding, reasoning, language, data analysis, and instruction following. Automatic scoring against ground truth avoids the biases introduced by LLM judging or crowdsourced human evaluation.

LiveBench’s limitations are worth stating clearly. Monthly question refresh requires ongoing maintenance infrastructure that static benchmarks do not. Coverage is limited to tasks with objectively scorable answers — open-ended writing quality, nuanced reasoning, and many real-world deployment scenarios resist objective ground-truth scoring. And dynamic benchmarks face their own validity questions: are tasks sampled consistently across refreshes? Are question difficulty distributions stable? These are methodological problems, not fatal flaws, but they require ongoing calibration work.

AdEval (arXiv:2501.13983) proposes alignment-based dynamic evaluation, using Wikidata revision logs to construct questions about provably post-training facts. LiveCodeBench applies the same principle to code tasks, building problems from programming contest problems released after known training cutoffs.

These approaches converge on the same design principle: evaluation validity requires temporal isolation between training data and test data, with that isolation enforced by benchmark design rather than hoped for through decontamination.

The Task-Specific Evaluation Case

Standard benchmarks are general-purpose. Deployment tasks are not. A model evaluated on MMLU mathematics questions may perform very differently on the specific mathematical reasoning required for your application. A model with high HumanEval pass@1 may fail systematically on the code generation patterns your codebase actually uses.

Task-specific evaluation — building held-out test sets from your actual use case — is the most contamination-resistant approach available because your private evaluation data cannot appear in public training corpora. It is also the most practically valid: it measures what you actually care about. The objection is cost. Building a representative task-specific benchmark requires curating questions, establishing answer quality criteria, and maintaining the benchmark as the task evolves.

The effort is justified for high-stakes deployments. The evaluation literature increasingly acknowledges that organizations operating AI systems for consequential decisions — hiring, credit, medical triage — need task-specific evidence that a model’s performance generalizes to their distribution. General benchmark scores are not a defensible substitute.

GPQA Diamond and similar expert-level benchmarks occupy a middle ground: highly difficult questions in specific domains (biology, chemistry, physics) that resist memorization by requiring multi-step expert reasoning. They are closer to task-specific than MMLU while remaining public. The tradeoff is that difficulty concentrates coverage in narrow domains.

What a Credible Evaluation Pipeline Looks Like

No single benchmark solves the contamination problem. A credible pipeline combines:

Temporal isolation via dynamic benchmarks. Use LiveBench or domain equivalents as a baseline layer. Questions derived from post-training material provide a floor on contamination-resistance. Accept that coverage is incomplete.

Static benchmarks with contamination adjustment. Where MMLU or similar benchmarks remain relevant for comparison, prefer contamination-aware variants (MMLU-CF, MMLU-Redux, MMLU-Pro). Report confidence intervals rather than point estimates, and treat small score differences as statistically indistinguishable.

Private task-specific evaluation. For any deployment with real consequences, build a held-out test set from your own task distribution. Keep it private. Rotate test examples periodically as deployment volumes accumulate. This is the only layer that directly measures what you are deploying.

Behavioral probing over aggregate scores. Supplement score-based evaluation with structured adversarial testing: does the model degrade gracefully on distribution-shifted inputs? Does it fail systematically on a specific input class that MMLU doesn’t cover? Behavioral probing catches capability gaps that aggregate scores miss.

Documented evaluation protocols. Record which benchmarks were used, which model version was evaluated, what decontamination steps were applied, and what score variance was observed across runs. This documentation is a prerequisite for meaningful model-to-model comparison and increasingly relevant to AI governance requirements — including the technical documentation and evidence obligations engineering teams will need to satisfy under the EU AI Act for high-risk systems, where the same instrumentation discipline (recorded protocols, versioned evaluation runs, retained score history) is what a conformity assessment actually expects to see.

The goal is not a perfect score. It is a valid one — evidence that actually tracks generalization rather than memorization, on tasks that actually matter for the deployment in question.

Frequently Asked Questions

What is benchmark contamination in LLM evaluation?

Benchmark contamination occurs when questions from an evaluation benchmark, or texts closely related to them, appear in a model’s pretraining corpus. When this happens, a model answering correctly is pattern-matching against memorized material rather than demonstrating the reasoning or knowledge the benchmark intends to measure. The result is inflated scores that overstate actual capability.

Is MMLU still useful for comparing language models?

MMLU retains some utility as a broad knowledge benchmark, but documented contamination rates — estimated at 29% or higher for some models — mean that score differences below 3–5 points between frontier models are unreliable indicators of genuine capability gaps. More carefully controlled variants (MMLU-CF, MMLU-Pro) partially address this but do not eliminate the underlying issue. MMLU is most useful when combined with other evaluation approaches, not when used alone.

What makes LiveBench more contamination-resistant?

LiveBench derives questions from recently published sources — arXiv papers, news articles, film synopses — that were released after models’ training cutoffs. Questions are refreshed monthly. Answers are objectively verifiable without using a model as judge. These three design choices make it structurally difficult for benchmark questions to appear in training data.

Should teams build their own evaluation benchmarks?

For high-volume or high-stakes deployments, yes. Private task-specific evaluation is the most contamination-resistant approach because the data never enters public training corpora. The cost is real — curation, quality criteria, rotation — but the alternative is relying on general benchmarks that measure correlates of your task, not the task itself. Organizations subject to AI governance requirements increasingly need documented evidence of task-specific evaluation.

How does the LLM-as-judge approach introduce evaluation bias?

When a model like GPT-4 evaluates other models’ outputs, it tends to favor outputs that resemble its own training distribution — syntactically, stylistically, and structurally. A model trained on similar data to the judge will score higher on judge-based evaluations independently of actual output quality. This creates circularity: labs that use the same data sources as the judge tend to produce models that score better. Objective ground-truth scoring, as used in LiveBench, avoids this problem for tasks where ground truth is available.