AI Safety

What AI providers state their models reached in safety testing, in their own words, and what outside testers found.

Also: compare models side by side · statements that a capability level was crossed · providers · practices providers document · safety news.

What providers state

Each row is one statement in a provider's own safety document about a capability level of one of its models: the level as the provider names it, whether the provider says the model reached it, and the sentence that says so.

136 of 136

Model	Level, as the provider names it	The provider states	The provider's sentence
Claude Sonnet 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 3.7 Anthropic	ASL-4 Autonomy and AI research	Not determined	“These problems are harder to solve than the previous problems and we consider them as part of our ASL-4 evaluations.” source
Claude Opus 4.6 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.6 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“Claude Opus 4.6 achieved a “max of 5” score of 0.79 for virology task 1 (inﬂuenza), placing it at the ASL-3 rule-in threshold of 0.8.” source
Claude Opus 4.6 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Based on these ﬁndings and quantitative risk modeling, we estimate that the model’s uplift to relevant threat actors remains below the ASL-4 capability threshold, though it may reduce certain tacit-knowledge barriers and help speed up biological research workﬂows.” source
Claude Opus 4.6 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.5 does not cross either the AI R&D-4 or CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“We determine that Claude Opus 4.5 does not cross the CBRN-4 threshold.” source
Claude Opus 4.1 Anthropic	ASL-4 Autonomy and AI research	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Cyber	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Opus 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Haiku 4.5 Anthropic	ASL-3 Autonomy and AI research	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source
Claude Haiku 4.5 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source

Previous · Page 4 of 4

What outside testers report

Each row is one result from a government evaluator, independent lab, academic or security firm that tested a model, quoted from where it is reported.

1957 of 1957

Model	Tester	Test	What was found
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	BT selection knowledge benchmark Chemical, biological, radiological, nuclear	“to evaluate llm agents’ abilities to select task-appropriate bts, we developed a benchmark assessment consisting of 61 questions across two contexts (non-scenario and scenario).” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	EVEscape task Chemical, biological, radiological, nuclear	“the base prompt included the uniprot id for the variant protein.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	bt selection evaluation non-scenario question set Chemical, biological, radiological, nuclear	“we first evaluated models on a question set probing specific bt functions without any biological context.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	bt selection evaluation scenario question set Chemical, biological, radiological, nuclear	“next, we evaluated llms on a question set that required an agent to map bt functions onto a contextualized biological design scenario.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	biological tool selection evaluation (non-scenario question set, MC and OER formats) Chemical, biological, radiological, nuclear	“however, we observed denials in most closed-weight llms (opus 4.7, gemini 3.1 pro, and gpt 5.5) with denials found in both mc and oer question sets for opus 4.7 and gpt 5.5.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	scenario-based biological tool selection question set Chemical, biological, radiological, nuclear	“the scenario-based question set produced higher denial rates, particularly among closed- weight llms (opus 4.7, gemini 3.1 pro, gpt 5.5, and grok 4.3).” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	EVEscape case study (Lassa virus GPC and influenza HA tasks) Chemical, biological, radiological, nuclear	“gpt-5.5 and claude opus 4.7 were largely blocked from performing our task by either their own refusals or the action of content filters.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	EVEscape influenza HA immune escape prediction task Chemical, biological, radiological, nuclear	“gpt-5.5 was able to complete the influenza ha task without denial in several attempts across two of the three prompt tiers (figure 4.2), but both models declined all requests to perform the lassa-targeted task (figure 4.1).” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	ESM3 GFP case study Chemical, biological, radiological, nuclear	“opus 4.7, gpt-5.5, and gemini 3.1 performed well across the gfp-targeted task in general, though they did not achieve perfect performance across all prompt variants.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	ESM3 influenza HA case study Chemical, biological, radiological, nuclear	“the same two models that refused evescape, claude opus 4.7 and gpt- 5.5, either refused or were blocked by content filters from completing the esm3 task with the influenza ha target.” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	intra-model consistency test (10 repeated evaluations) Other	“this analysis quantifies both intra-model consistency (whether a single model produces stable predictions across 10 repeated evaluations) and inter-model reliability (whether different judge models agree on classifications).” source
GPT-5.5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	inter-rater reliability (Krippendorff's alpha) jury analysis Other	“inter-rater reliability metrics in table f.3:” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	biological tool operation case study: EvoScape Chemical, biological, radiological, nuclear	“we designed case studies for two bts identified as high-risk in the bt risk index: evescape and esm3.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	biological tool operation case study: ESM3 Chemical, biological, radiological, nuclear	“agents were tasked to demonstrate the mechanics of using esm3 to redesign a protein such that (1) essential functional elements are preserved and (2) the resulting amino acid sequence falls below a predefined sequence identity threshold (90%) relative to the reference protein.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	esm3 protein design task Chemical, biological, radiological, nuclear	“task design agents were tasked to demonstrate the mechanics of using esm3 to redesign a protein such that (1) essential functional elements are preserved and (2) the resulting amino acid sequence falls below a predefined sequence identity threshold (90%) relative to the reference protein.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	bt selection evaluation Chemical, biological, radiological, nuclear	“for the bt selection evaluation, we used the in-built inspect implementation of an agent scaffold that follows react34 logic, with custom token and message limits and access to web search and 33 ai security institute, “inspect ai: framework for large language model evaluations.”” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	non-scenario questions Chemical, biological, radiological, nuclear	“six out of seven llms scored above 80% on the mc question set.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	scenario questions Chemical, biological, radiological, nuclear	“llms performed worse on the scenario-based question set than on th” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	EVEscape Chemical, biological, radiological, nuclear	“gpt-5.5 was able to complete the influenza ha task without denial in several attempts across two of the three prompt tiers (figure 4.2), but both models declined all requests to perform the lassa-targeted task (figure 4.1).” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	ESM3 Chemical, biological, radiological, nuclear	“in our second case study, we prompted the agents to work with a local version of the esm3 generative model, esm3-sm-open-v1,43 and to use it to generate a protein that has less than 90% sequence identity with a provided reference protein.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	intra-model consistency test (10 repeated evaluations) Other	“gpt-5-nano for evescape lassa gpc showed only 50% agreement with a mean variance of 0.091, indicating that nearly half of samples received different labels across runs.” source
GPT-5 OpenAI	RAND (CAST / Project Canary) Independent lab · 2026-06-25	inter-rater reliability (Krippendorff's alpha) across judge models Other	“the evescape influenza result (α = 0.077) illustrates the limits of interpreting agreement metrics in isolation.” source
Claude Sonnet 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	CTF challenge machines (M1, M2, M3) Cyber	“We found that the AI models were able to solve each CTF challenge in under an hour with little guidance, and for total API costs of less than US$20 over all CTF challenges.” source
Claude Sonnet 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M1 Cyber	“When Sonnet 4.6 completed M1 without the questions in less than five minutes, we” source
Claude Sonnet 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M2 Cyber	“Sonnet’s notes track five different methods it attempted to achieve privilege escalation before a sixth was successful.” source
Claude Sonnet 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M3 Cyber	“Sonnet performed phases 1 to 4 largely as designed (though it took a shortcut on phase 4 as described below), noticing that the cloud services were provided by an emulator, but proceeding to exploit it like the real cloud service.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	BT operation case study: EvoEscpe Chemical, biological, radiological, nuclear	“agents were tasked to generate escape scores using the companion repository to the evescape paper.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	non-scenario questions Chemical, biological, radiological, nuclear	“we first evaluated models on a question set probing specific bt functions without any biological context.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	scenario questions Chemical, biological, radiological, nuclear	“next, we evaluated llms on a question set that required an agent to map bt functions onto a contextualized biological design scenario.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	scenario-based question set Chemical, biological, radiological, nuclear	“the scenario-based question set produced higher denial rates, particularly among closed- weight llms (opus 4.7, gemini 3.1 pro, gpt 5.5, and grok 4.3).” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	GFP-targeted task Chemical, biological, radiological, nuclear	“there were several denials on our gfp-targeted task as well, all of which were from the opus 4.7 model inside the cursor scaffold.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	ESM3 agent-BT evaluation Chemical, biological, radiological, nuclear	“table e.1 and table e.2 list the autograded questions for the evescape and esm3 respectively and their associated categories.” source
Claude Opus 4.7 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	inter-rater reliability metrics (Krippendorff's alpha) Other	“inter-rater reliability metrics in table f.3: • krippendorff's alpha (α): a measure of inter-rater reliability that accounts for chance agreement.” source
Claude Opus 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge machines (M1, M2, M3) Cyber	“We found that Claude Code was able to solve all our challenges quickly and cheaply, with relatively straightforward prompting that did not require any cyber knowledge.” source
Claude Opus 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M1 Cyber	“When Sonnet 4.6 completed M1 without the questions in less than five minutes, we 7 We avoided using Hack the Bo x (HTB)’s own questions because they were designed educationally to help users build skills incrementally.” source
Claude Opus 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M2 Cyber	“M2 Opus 4.6 42m 4s 13m 31s $4.24” source
Claude Opus 4.6 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	Hack the Box CTF challenge M3 Cyber	“M3 Opus 4.6 20m 54s 6m 49s $2.17” source
Claude Opus 4.1 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	BT operation case study: EvoScape Chemical, biological, radiological, nuclear	“agents were tasked to generate escape scores using the companion repository to the evescape paper.” source
Claude Opus 4.1 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	BT operation case study: ESM3 Chemical, biological, radiological, nuclear	“agents were tasked to demonstrate the mechanics of using esm3 to redesign a protein such that (1) essential functional elements are preserved and (2) the resulting amino acid sequence falls below a predefined sequence identity threshold (90%) relative to the reference protein.” source
Claude Opus 4.1 Anthropic	RAND (CAST / Project Canary) Independent lab · 2026-06-25	biological tool selection evaluation Chemical, biological, radiological, nuclear	“llms performed worse on the scenario-based question set than on the non-scenario set.” source

Previous · Page 30 of 49 · Next