A model that answers factual questions correctly and writes clean code can still produce content that causes real harm — instructions for building weapons, convincing disinformation, or outputs that expose private data. Capability benchmarks were never built to catch this. That gap is what AI safety benchmarks exist to close, and the discipline around them has matured quickly since 2022, moving from ad hoc internal checklists to a recognizable — if still fragmented — evaluation practice.
This article covers what distinguishes a safety benchmark from a capability benchmark, the major categories of safety evaluation in use today, how red-teaming methodology has evolved from manual to model-assisted approaches, and the structural limitations that keep even well-designed safety benchmarks from fully predicting real-world risk.
What Makes a Benchmark a “Safety” Benchmark?
Capability benchmarks measure whether a model can do something: solve a math problem, summarize a document, write functioning code, answer a trivia question correctly. Safety benchmarks measure whether a model should do something, or whether it does it in a way that avoids harm. The two are not the same axis, and a model can score well on one while scoring poorly on the other.
A safety benchmark typically evaluates one or more of the following properties: whether the model refuses genuinely harmful requests, whether it resists being manipulated into bypassing its own guidelines, whether its outputs are biased against protected groups, and whether it possesses capabilities that would be dangerous if misused (chemical synthesis guidance, cyberweapon development, or autonomous replication, for instance). None of these are captured by a benchmark built to test reasoning or code generation.
The distinction matters for procurement and deployment decisions. A technical decision-maker evaluating a model for a customer-facing application needs both scores — a highly capable model with poor refusal behavior is arguably a worse deployment choice than a moderately capable model with strong safety properties, depending on the use case. Vendor marketing materials do not always make this distinction clearly, which is itself a reason to treat published capability scores and safety claims as separate lines of evidence requiring separate scrutiny.
Categories of AI Safety Benchmarks
Safety evaluation is not a single test; it is a cluster of related but distinct measurement approaches, each targeting a different failure mode.
Jailbreak and adversarial-prompt resistance. These benchmarks test whether a model can be manipulated — through role-play framing, encoded instructions, multi-turn context manipulation, or other prompt-engineering techniques — into producing outputs it would otherwise refuse. Resistance is typically measured as an attack success rate: the percentage of adversarial prompts from a curated or dynamically generated set that successfully elicit a policy-violating response. Because new jailbreak techniques are discovered continuously, these benchmarks age quickly; a resistance score measured against a fixed prompt set from a year prior tells you little about resistance to current techniques.
Harmful-content refusal rates. This category measures how reliably a model declines requests for content in well-defined harm categories — weapons instructions, child safety violations, self-harm encouragement, targeted harassment — without also over-refusing benign adjacent requests. Over-refusal is a real and measured failure mode in its own right: a model that declines to discuss chemistry homework because the question mentions a controlled substance is failing a different, less-discussed axis of the same evaluation.
Bias and fairness probes. These test whether model outputs vary systematically and inappropriately across demographic groups — in sentiment, association, recommendation quality, or refusal rate. Fairness probes typically rely on template-based prompt sets that swap demographic markers while holding the rest of the prompt constant, then measure statistical divergence in outputs. Results are highly sensitive to which demographic categories and which templates are chosen, which is one reason fairness benchmark results are difficult to compare across research groups using different test sets.
Dangerous-capability evaluations. This is the newest and most specialized category, focused on frontier risks: could the model meaningfully assist someone with insufficient existing expertise in synthesizing a biological or chemical weapon, conducting a sophisticated cyberattack, or acquiring capabilities that would let an AI system act with reduced human oversight? These evaluations are typically conducted by specialized teams with domain expertise (biosecurity researchers, offensive security professionals) rather than general-purpose annotators, and results are often not published in full detail for the same reason the underlying capability is concerning — publication itself could be a partial uplift.
Red-Teaming Methodology: Manual and Model-Assisted Approaches
Red-teaming is the active, adversarial complement to static benchmark scoring. Where a benchmark applies a fixed or semi-fixed set of test cases, red-teaming is an open-ended search for failure modes that existing tests have not anticipated.
Manual red-teaming uses human testers — internal staff, contracted specialists, or crowdsourced participants — who attempt to elicit harmful, biased, or policy-violating outputs through creative and adversarial prompting. Manual red-teaming benefits from human judgment about what a subtly harmful output looks like and can surface failure modes that no automated system has been designed to look for. Its costs are real: it is slow, expensive to scale, and its coverage depends heavily on the creativity and domain knowledge of the specific testers involved. Two red-teaming exercises against the same model, run by different teams, routinely surface substantially different sets of vulnerabilities.
Automated and model-assisted red-teaming uses one language model to generate adversarial test cases against a target model, then classifies or scores the resulting outputs — sometimes with a human in the loop for final judgment, sometimes fully automated. The foundational method here comes from Anthropic researchers’ 2022 paper Red Teaming Language Models with Language Models, which demonstrated that a red-team language model could generate test cases via zero-shot prompting, few-shot prompting, and reinforcement learning, surfacing tens of thousands of offensive outputs from a target model that manual review alone would have been unlikely to find at that scale. Model-assisted red-teaming trades some of the judgment quality of human testers for dramatically greater scale and repeatability, which makes it well suited to regression testing across model versions — checking whether a known vulnerability class has actually been fixed, rather than searching for genuinely novel ones.
In practice, mature safety evaluation programs use both: automated red-teaming for broad, repeatable coverage and regression tracking, and manual red-teaming — often with domain specialists — for the harder, more open-ended search for genuinely novel failure modes, particularly in the dangerous-capability category described above.
Known Limitations of AI Safety Benchmarks
Safety benchmarks are a necessary tool, not a sufficient one, and the field is reasonably candid about several structural limitations.
Benchmark gaming. When a specific numeric score becomes the target of optimization — whether through explicit training against benchmark-like data or through narrower fine-tuning aimed at the test distribution — the score can improve without a corresponding improvement in the underlying property the benchmark was meant to measure. A model fine-tuned to recognize and refuse prompts resembling a known jailbreak benchmark’s format may still be vulnerable to structurally different attacks that achieve the same underlying goal.
Evaluation-set contamination. Safety benchmark prompts and answer patterns can leak into training data, whether through direct inclusion, discussion of the benchmark in scraped web text, or derivative datasets built from benchmark outputs. Contamination inflates measured performance without improving real-world safety behavior, and it is difficult to detect after the fact because labs rarely have full visibility into everything in a large pretraining corpus.
The gap between static benchmarks and deployment risk. A safety benchmark evaluates a model against a fixed set of inputs at a fixed point in time. Real deployment involves an open-ended space of user behavior, product integrations, and adversaries actively searching for exploitable behavior — conditions no static test set can fully anticipate. A model can score well on every available safety benchmark and still produce a serious harmful output in production once exposed to the full diversity of real usage, including multi-turn conversations, tool use, and integration with other systems that static single-turn benchmarks do not model.
Lack of standardization across labs. There is no single agreed-upon safety benchmark suite that all developers report against, and organizations differ in which categories they test, how they define harm thresholds, and how they disclose results. This makes it difficult for a technical decision-maker or policy analyst to compare safety claims across models on equal footing — a “low jailbreak success rate” claim from one lab may reflect a materially easier test set than the same claim from another. This is a distinct problem from the broader benchmark contamination and reliability issues that affect capability evaluation as well, though the two compound each other: a field without standardized capability evaluation is unlikely to converge quickly on standardized safety evaluation either.
Where This Evaluation Work Fits in AI Governance
Safety benchmarking and red-teaming do not exist in a governance vacuum — they are the technical measurement layer that frameworks like the NIST AI Risk Management Framework are built to sit on top of. The Framework’s Measure function explicitly calls for adversarial and stress testing as evaluation approaches, without mandating a specific test suite or numeric threshold — which is precisely why the benchmark landscape described above remains fragmented rather than standardized. NIST has continued building out more specific guidance since the Framework’s initial release, including a generative AI profile, but implementers are still largely left to define their own measurement methodology within that structure.
This has direct operational consequences. Because safety evaluation results feed into governance and compliance processes, they need the same evaluation discipline that any other MLOps output requires: versioned test sets, reproducible scoring, and a documented evaluation pipeline sitting alongside the deployment pipeline rather than bolted on afterward. Teams that already treat evaluation as an integrated step in moving models from training to production are better positioned to make safety evaluation continuous — re-running red-team suites against each model version — rather than a one-time pre-launch checkbox that goes stale the moment the model or its usage patterns change.
Standards bodies beyond NIST are also active in this space, including ISO/IEC work on AI management systems and sector-specific efforts from groups like the Partnership on AI and MITRE’s ATLAS framework for adversarial threat tracking in machine learning systems. None of these bodies currently mandate a specific safety benchmark score as a condition of deployment; the practical effect of the governance layer, at least as of this writing, is to require that safety evaluation be planned, documented, and traceable — not to specify exactly what a passing result looks like.
Frequently Asked Questions
What is the difference between a capability benchmark and a safety benchmark?
A capability benchmark measures whether a model can perform a task correctly — solving math problems, writing code, answering questions. A safety benchmark measures whether the model’s outputs avoid harm: resisting jailbreak attempts, refusing genuinely dangerous requests, avoiding biased outputs, and lacking capabilities that would be dangerous if misused. The two axes are measured separately and do not predict each other.
What is model-assisted red-teaming?
Model-assisted (or automated) red-teaming uses one language model to generate adversarial test cases against a target model, then scores or classifies the resulting outputs, sometimes with human review. It trades some of the nuanced judgment of human testers for far greater scale, making it useful for repeatable regression testing across model versions rather than open-ended discovery of entirely novel failure modes.
Can a model pass safety benchmarks and still be unsafe in deployment?
Yes. Static benchmarks test a fixed set of inputs at a point in time, while real deployment exposes a model to an open-ended range of user behavior, multi-turn conversations, tool integrations, and adversaries actively searching for exploitable weaknesses. A model can score well across available safety benchmarks and still produce harmful outputs once exposed to the full diversity of production usage.
Does any regulation require passing a specific AI safety benchmark?
No single benchmark score is currently mandated by law or by the NIST AI Risk Management Framework. The Framework’s Measure function calls for adversarial and stress testing as evaluation approaches but leaves specific metrics and thresholds to implementers. Some jurisdictions impose conformity assessment obligations on higher-risk systems that reference adversarial testing generally, without naming a specific benchmark suite.
Why don’t AI labs use the same safety benchmarks?
There is no single industry-agreed safety benchmark suite, and labs differ in which harm categories they test, how they define thresholds, and how much detail they disclose publicly. This makes safety claims difficult to compare across organizations on equal footing, since a strong result on one lab’s test set may reflect an easier evaluation than a similar-sounding claim from another lab.