AI Safety

What AI providers state their models reached in safety testing, in their own words, and what outside testers found.

Also: compare models side by side · statements that a capability level was crossed · providers · practices providers document · safety news.

What providers state

Each row is one statement in a provider's own safety document about a capability level of one of its models: the level as the provider names it, whether the provider says the model reached it, and the sentence that says so.

136 of 136

Model	Level, as the provider names it	The provider states	The provider's sentence
Claude Sonnet 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 3.7 Anthropic	ASL-4 Autonomy and AI research	Not determined	“These problems are harder to solve than the previous problems and we consider them as part of our ASL-4 evaluations.” source
Claude Opus 4.6 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.6 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“Claude Opus 4.6 achieved a “max of 5” score of 0.79 for virology task 1 (inﬂuenza), placing it at the ASL-3 rule-in threshold of 0.8.” source
Claude Opus 4.6 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Based on these ﬁndings and quantitative risk modeling, we estimate that the model’s uplift to relevant threat actors remains below the ASL-4 capability threshold, though it may reduce certain tacit-knowledge barriers and help speed up biological research workﬂows.” source
Claude Opus 4.6 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.5 does not cross either the AI R&D-4 or CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“We determine that Claude Opus 4.5 does not cross the CBRN-4 threshold.” source
Claude Opus 4.1 Anthropic	ASL-4 Autonomy and AI research	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Cyber	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Opus 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Haiku 4.5 Anthropic	ASL-3 Autonomy and AI research	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source
Claude Haiku 4.5 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source

Previous · Page 4 of 4

What outside testers report

Each row is one result from a government evaluator, independent lab, academic or security firm that tested a model, quoted from where it is reported.

1957 of 1957

Model	Tester	Test	What was found
Claude Opus 4.7 Anthropic	UK AI Security Institute Government evaluator · 2026-07-21	cybersecurity capability evaluations Cyber	“In AISI’s cybersecurity capability evaluations, models are tasked with finding a ‘flag’ (a sequence of letters and numbers) hidden within a simulated environment, by carrying out a sequence of offensive cyber actions, such as reverse-engineering compiled code or exploiting security vulnerabilities.” source
Claude Opus 4.7 Anthropic	UK AI Security Institute Government evaluator · 2026-07-21	automated LLM monitor Deception and sabotage	“To analyse cheating behaviour at scale, we used an automated LLM monitor.” source
Claude Opus 4.7 Anthropic	UK AI Security Institute Government evaluator · 2026-07-21	self-report Deception and sabotage	“One proposed way to detect cheating is to ask the model whether it took any prohibited or suspicious actions.” source
Claude Opus 4.7 Anthropic	UK AI Security Institute Government evaluator · 2026-07-21	chain-of-thought monitoring Deception and sabotage	“Another possible method for identifying cheating is monitoring the model’s chain-of-thought - the written reasoning it produces whilst working through a task, which is usually, but not always, human-legible (in today’s models).” source
Claude Opus 4.1 Anthropic	METR Independent lab · 2026-07-21	agentic optimization runs against NanoGPT Autonomy and AI research	“We also report the status of preliminary agentic optimization runs against NanoGPT: after more than \$10K of expenditure we estimate expenditure horizons of \$0-\$3K.” source
Claude Opus 4.1 Anthropic	METR Independent lab · 2026-07-21	six high-expenditure agentic optimization runs Autonomy and AI research	“We ran six high-expenditure agentic optimization runs starting from record #78 of the speedrun (March ’26, 85.56 seconds of training time).” source
Claude Opus 4.1 Anthropic	METR Independent lab · 2026-07-21	NanoGPT speedrun agentic optimization run Autonomy and AI research	“GPT-5 and Opus-4.1 did not make any meaningful progress at all.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check Chemical, biological, radiological, nuclear	“We tested a range of tool-level safeguards (text or code embedded within biological software tools to restrict agent usage) and evaluated their efficacy in preventing AI agents from assisting with the tool’s use.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check Deception and sabotage	“On the ESM3 task, both Opus 4.5 and Gemini 3.1 Pro make these false claims in 50% of trials and 63% of trials, respectively.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	environmental identifier safeguard Deception and sabotage	“However, we found that Opus 4.5 and Gemini 3.1 Pro frequently delete these variables to evade detection.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	agents.md safeguard Deception and sabotage	“Despite the fact that AGENTS.md files are intended to contain instructions written by humans for AI agents (Agents.md project, 2026), we 16 found that agents would create and modify AGENTS.md files in order to bypass this safeguard.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check displaying a message stating an intent to synthesize variola major Harmful content	“While Opus 4.5 and Gemini 3.1 Pro were either blocked by content filters or refused to proceed in every trial, GPT 5.2 proceeded without refusal in every trial.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	false identity claim experiment Deception and sabotage	“All frontier LLMs we tested falsely claimed to be human when doing so enabled them to accomplish the task.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	agents.md experiments Prompt injection and agents	“We designed safeguards that leverage harness’s treatment of AGENTS.md files and that detect harness’s environment identifiers.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check Misuse and jailbreaks	“The strongest safeguard, the pre-run check, was only highly effective on one of the three tasks, and even on this task the help denial rate was as low as 90% (for Opus 4.5).” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	jailbreak of the strongest safeguard Misuse and jailbreaks	“Jailbreaks reduce the efficacy of our strongest safeguard to near-zero, illustrating that LLM guardrails can fail and highlighting the need for a defense in depth strategy.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	documentation changes Misuse and jailbreaks	“We ran experiments where statements prohibiting AI agent usage were placed into the readme.md documentation files of RFdiffusion.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	greedy coordinate gradient (GCG) optimization of safeguard text Misuse and jailbreaks	“We ran this algorithm with an abliterated gpt-oss-20b model on the RFdiffusion tasks.” source
Gemini 3.1 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	GCG-optimized prompt injection misdirection Misuse and jailbreaks	“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	tool-level safeguards Chemical, biological, radiological, nuclear	“We tested a range of tool-level safeguards (text or code embedded within biological software tools to restrict agent usage) and evaluated their efficacy in preventing AI agents from assisting with the tool’s use.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	jailbreaks Misuse and jailbreaks	“The tool-level safeguards we developed are compromised by subversion of LLM guardrails1: adversarial subversion techniques such as jailbreaks and refusal-vector ablation reduce the efficacy of safeguards, indicating that reliance on LLM guardrails does not currently provide robust risk mitigation.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	help denial rate Chemical, biological, radiological, nuclear	“Our evaluation metric, the help denial rate, measures the frequency with which the safeguard stops the agent from both (i) running the biological tool and (ii) providing assistance to the user on the use of the tool.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check Chemical, biological, radiological, nuclear	“Across all tasks and all LLMs, the pre-run check increased the help denial rate, except in the one case in which the baseline help denial rate was already 100% (Opus 4.5 on the EVEscape task).” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	jailbreak robustness test Misuse and jailbreaks	“We test our safeguards’ robustness to jailbreaks by appending the jailbreak text to each initial task prompt, repeating the methodology described above, and observing how the jailbreak affects refusal-to-help rate.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	safety abliteration Misuse and jailbreaks	“We also explored methods to optimize a subset of our safeguards against small open-weight LLMs, gpt-oss-20b (Agarwal, 2025), and Qwen 3 4B Instruct (Yang, et al., 2025), whose safeguards have been subverted using a technique called safety abliteration (Arditi et al., 2024).” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	jailbreak text added to task prompts Misuse and jailbreaks	“Including these jailbreaks in the task prompt reduced the help denial rate to near zero for Opus 4.5 and Gemini 3 Pro (Figure 2A).” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	refusal abliteration Misuse and jailbreaks	“We performed refusal abliteration on two small open-weight LLMs, GPT OSS 20B (Agarwal et al., 2025) and Qwen 3 4B Instruct (Yang et al., 14 2025), and found that this reduced help denial to near zero for both LLMs (Figure 2B).” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check on the esm3 task Deception and sabotage	“On the ESM3 task, both Opus 4.5 and Gemini 3.1 Pro make these false claims in 50% of trials and 63% of trials, respectively.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	safeguard that detects environmental identifiers of the cursor cli agent Deception and sabotage	“However, we found that Opus 4.5 and Gemini 3.1 Pro frequently delete these variables to evade detection.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	safeguard that utilizes the agents.md protocol Deception and sabotage	“Despite the fact that AGENTS.md files are intended to contain instructions written by humans for AI agents (Agents.md project, 2026), we 16 found that agents would create and modify AGENTS.md files in order to bypass this safeguard.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	pre-run check in which the biological tool displays a message stating an intent to synthesize variola major Harmful content	“While Opus 4.5 and Gemini 3.1 Pro were either blocked by content filters or refused to proceed in every trial, GPT 5.2 proceeded without refusal in every trial.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	tool-level safeguard evaluation (pre-run check, agents.md, environment identifiers) in biological software tasks Other	“Our results suggest that safeguards implemented into biological software cannot currently induce help denial reliably in LLM agents.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	documentation changes (statements prohibiting AI agent usage placed into README.md of RFdiffusion) Misuse and jailbreaks	“We ran experiments where statements prohibiting ai agent usage were placed into the readme.md documentation files of rfdiffusion.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	GCG optimization of safeguard text on RFdiffusion tasks with an abliterated gpt-oss-20b model Misuse and jailbreaks	“We ran this algorithm with an abliterated gpt-oss-20b model on the rfdiffusion tasks.” source
Gemini 3 Pro Google DeepMind	RAND (CAST / Project Canary) Independent lab · 2026-07-20	GCG optimization of misdirection prompt injection text on the abliterated gpt-oss-20b model Misuse and jailbreaks	“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
GLM-5.2 Z.ai (Zhipu AI)	US CAISI (NIST) Government evaluator · 2026-07-17	ExploitBench Cyber	“Cyber ExploitBench 21.4 ± 0.9 57.2 ± 1.1 38.1 ± 1.1 40.7 ± 1.1 N/A” source
GLM-5.2 Z.ai (Zhipu AI)	US CAISI (NIST) Government evaluator · 2026-07-17	CTF-Archive-Diamond Cyber	“Cyber CTF Archive Diamond 39.3 ± 2.9 65.6 ± 2.8 48.4 ± 3.0 70.5 ± 2.7 32**” source
GLM-5.2 Z.ai (Zhipu AI)	US CAISI (NIST) Government evaluator · 2026-07-17	agent hijacking attacks Cyber	“GLM-5.2 was never successfully hijacked by publicly available agent hijacking attacks (Figure 5.1), and appeared to withstand more iterations of adversarial red-teaming more than other evaluated open-weight models, although less than evaluated U.S. closed-weight models (Figure 5.2).” source
GLM-5.2 Z.ai (Zhipu AI)	US CAISI (NIST) Government evaluator · 2026-07-17	automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries Misuse and jailbreaks	“Figure 6.1: On automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries, GLM-5.2 complied with a low percentage of requests , both with and without the use of a public jailbreak.” source
GLM-5.2 Z.ai (Zhipu AI)	US CAISI (NIST) Government evaluator · 2026-07-17	automated benchmark evaluations against malicious cyber queries Misuse and jailbreaks	“Figure 6.2: On automated benchmark evaluations against malicious cyber queries, the best jailbreak tended to reduce – rather than increase – the level of request-relevant detail in GLM-5.2’s responses.” source

Previous · Page 25 of 49 · Next