AI Safety

What AI providers state their models reached in safety testing, in their own words, and what outside testers found.

Also: compare models side by side · statements that a capability level was crossed · providers · practices providers document · safety news.

What providers state

Each row is one statement in a provider's own safety document about a capability level of one of its models: the level as the provider names it, whether the provider says the model reached it, and the sentence that says so.

136 of 136

Model	Level, as the provider names it	The provider states	The provider's sentence
Claude Sonnet 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not determined	“speciﬁcally, we focused on ruling out asl-4 for both models, as well as ruling out asl-3 for claude sonnet 4.” source
Claude Sonnet 3.7 Anthropic	ASL-4 Autonomy and AI research	Not determined	“These problems are harder to solve than the previous problems and we consider them as part of our ASL-4 evaluations.” source
Claude Opus 4.6 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.6 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“Claude Opus 4.6 achieved a “max of 5” score of 0.79 for virology task 1 (inﬂuenza), placing it at the ASL-3 rule-in threshold of 0.8.” source
Claude Opus 4.6 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Based on these ﬁndings and quantitative risk modeling, we estimate that the model’s uplift to relevant threat actors remains below the ASL-4 capability threshold, though it may reduce certain tacit-knowledge barriers and help speed up biological research workﬂows.” source
Claude Opus 4.6 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“Our determination is that Claude Opus 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	AI R&D-4 Autonomy and AI research	Not reached	“Our determination is that Claude Opus 4.5 does not cross either the AI R&D-4 or CBRN-4 capability threshold.” source
Claude Opus 4.5 Anthropic	CBRN-4 Chemical, biological, radiological, nuclear	Not reached	“We determine that Claude Opus 4.5 does not cross the CBRN-4 threshold.” source
Claude Opus 4.1 Anthropic	ASL-4 Autonomy and AI research	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4.1 Anthropic	ASL-4 Cyber	Not reached	“Since Claude Opus 4.1 is deployed with ASL-3 protections, we focused on conﬁrming it remains well below ASL-4 thresholds across CBRN (Chemical, Biological, Radiological, and Nuclear), cyber, and autonomy domains.” source
Claude Opus 4 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not determined	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Opus 4 Anthropic	ASL-4 Chemical, biological, radiological, nuclear	Not reached	“for claude opus 4, our evaluations showed that, whereas we could conﬁdently rule out asl-4 capabilities, we could not conclusively rule out asl-3 capabilities—which led to our precautionary decision to deploy it with asl-3 protections .” source
Claude Haiku 4.5 Anthropic	ASL-3 Autonomy and AI research	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source
Claude Haiku 4.5 Anthropic	ASL-3 Chemical, biological, radiological, nuclear	Not reached	“Our evaluation results showed that Claude Haiku 4.5 remained well below ASL-3 thresholds across all domains of concern.” source

Previous · Page 4 of 4

What outside testers report

Each row is one result from a government evaluator, independent lab, academic or security firm that tested a model, quoted from where it is reported.

1957 of 1957

Model	Tester	Test	What was found
GPT-5.5 OpenAI	Apollo Research Independent lab · 2026-07-07	LLM judge severity scoring of injected-failure trajectories Other	“GPT-5.4 and GPT-5.5 score 7, weighing the unauthorized production execution most heavily, yet even they stay below the reference.” source
GPT-5.4 mini OpenAI	Apollo Research Independent lab · 2026-07-07	coding-agent trajectory severity scoring (1-10 severity scale across five failure modes) Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
GPT-5.4 mini OpenAI	Apollo Research Independent lab · 2026-07-07	LLM judge evaluation against approximate ground truth severity labels Other	“For example, Gemini 3.1 Flash-Lite skips severity grading to score task execution, Haiku 4.5 primarily fails to flag violations at all, and GPT-5.4 Mini identifies violations but discounts them heavily through session-context framing.” source
Claude Sonnet 5 Anthropic	Apollo Research Independent lab · 2026-07-07	evaluating LLM calibration for coding-agent monitoring Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
Claude Sonnet 5 Anthropic	Apollo Research Independent lab · 2026-07-07	LLM judge evaluation against approximate ground truth severity labels Other	“Claude models lead on ranking quality: Opus 4.8 is the best single judge, with Sonnet 5 close behind at 40% of the cost.” source
Claude Sonnet 5 Anthropic	Apollo Research Independent lab · 2026-07-07	repeated scoring stability analysis Other	“Resampling gains are also concentrated in smaller models (e.g. Gemini 2.5 Flash, GPT-5.4 Mini), while the biggest models (e.g. Opus 4.8, Sonnet 5, GPT-5.5) gain little from repeated calls.” source
Claude Sonnet 5 Anthropic	Apollo Research Independent lab · 2026-07-07	pairwise spearman rank correlation between judges Other	“Sonnet 5 and Opus 4.8, the strongest pair, agree with each other at 0.93, which is higher than any model's agreement with our approximate ground truth.” source
Claude Sonnet 4.6 Anthropic	Apollo Research Independent lab · 2026-07-07	coding-agent trajectory severity grading (1-10 severity scale across five failure modes) Other	“Some models seem rarely useful across many dimensions as single judges: Gemini 3.1 Flash-Lite, Gemini 2.5 Flash, Gemini 3.5 Flash, Gemini-2.5-pro, GPT-5.4 nano, GPT-5.4 mini, Claude Haiku 4.5, Sonnet-4.6, Opus-4.6 since they are strictly dominated by other models.” source
Claude Sonnet 4.6 Anthropic	Apollo Research Independent lab · 2026-07-07	LLM judge evaluation on injected-failure trajectories (Spearman rank correlation and MAE against reference severity) Other	“As single judges, we recommend against Gemini 3.1 Flash-Lite, Gemini 2.5 Flash, Gemini 3.5 Flash, Gemini 2.5 Pro, GPT-5.4 Mini, Claude Haiku 4.5, Sonnet 4.6, and Opus 4.6; each is dominated by a cheaper or better alternative.” source
Claude Sonnet 4.6 Anthropic	Apollo Research Independent lab · 2026-07-07	Paired native-thinking vs no-thinking judge scoring comparison Other	“Every judge scored the same trajectories under both settings: once with native thinking disabled (reasoning=none) and once at the high setting (reasoning=high for OpenAI, Anthropic, and Gemini 3.x models; a thinking budget of 8192 tokens for Gemini 2.5 models) used throughout this post.” source
Claude Opus 4.8 Anthropic	Apollo Research Independent lab · 2026-07-07	coding-agent trajectory severity grading (1-10 severity scale across five failure modes) Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
Claude Opus 4.8 Anthropic	Apollo Research Independent lab · 2026-07-07	coding-agent trajectory severity scoring (1-10 severity scale across five failure modes) Other	“Selected by ρ: Opus 4.8 alone (0.90) → GPT-5.5 + Opus 4.8 (0.91, MAE 0.85) → + Gemini 3.1 Pro (0.92).” source
Claude Opus 4.7 Anthropic	Apollo Research Independent lab · 2026-07-07	severity scoring of coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
Claude Opus 4.6 Anthropic	Apollo Research Independent lab · 2026-07-07	severity scoring of coding-agent trajectories on a 1-10 scale Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
Claude Haiku 4.5 Anthropic	Apollo Research Independent lab · 2026-07-07	severity scoring of coding-agent trajectories Other	“Some models seem rarely useful across many dimensions as single judges: Gemini 3.1 Flash-Lite, Gemini 2.5 Flash, Gemini 3.5 Flash, Gemini-2.5-pro, GPT-5.4 nano, GPT-5.4 mini, Claude Haiku 4.5, Sonnet-4.6, Opus-4.6 since they are strictly dominated by other models.” source
Claude Haiku 4.5 Anthropic	Apollo Research Independent lab · 2026-07-07	scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes Other	“We evaluate 16 LLMs scoring coding-agent trajectories on a 1-10 severity scale across three datasets and five failure modes to assess their capabilities and calibration as coding agent monitors.” source
Claude Haiku 4.5 Anthropic	Apollo Research Independent lab · 2026-07-07	judge evaluation on injected-failure trajectories Deception and sabotage	“For example, Gemini 3.1 Flash-Lite skips severity grading to score task execution, Haiku 4.5 primarily fails to flag violations at all, and GPT-5.4 Mini identifies violations but discounts them heavily through session-context framing.” source
Claude Haiku 4.5 Anthropic	Apollo Research Independent lab · 2026-07-07	native thinking vs no-thinking judge comparison Deception and sabotage	“Every judge scored the same trajectories under both settings: once with native thinking disabled (reasoning=none) and once at the high setting (reasoning=high for OpenAI, Anthropic, and Gemini 3.x models; a thinking budget of 8192 tokens for Gemini 2.5 models) used throughout this post.” source
Claude Haiku 4.5 Anthropic	Apollo Research Independent lab · 2026-07-07	per-model calibration curves with thinking disabled Other	“The frontier models are nearly unchanged without thinking, e.g. Sonnet 5 (ρ = 0.89, MAE = 0.96) and Opus 4.8 (0.89, 1.00) remain strong, while the small models degrade visibly: Haiku 4.5 drops to ρ = 0.69, Gemini 3.1 Flash-Lite to 0.67, GPT-5.4 Mini to 0.72, with a more pronounced under-scoring bend below the diagonal.” source
GPT-5.5 OpenAI	Epoch AI Independent lab · 2026-07-01	EBR-bench Other	“AI systems are closer to the random baseline, with GPT-5.5 and Opus 4.8 both incurring 2.1 fatigue per round, enabling about 7.7 rounds of play per day.” source
GPT-5.5 OpenAI	Epoch AI Independent lab · 2026-07-01	max elicitation setting Other	“Namely, they achieve between 2 and 3.5 additional objectives (out of 21), depending on the model.” source
GPT-5 OpenAI	Epoch AI Independent lab · 2026-07-01	ebr-bench Other	“Our latest benchmark, EBR-bench, tests AI systems for this ability by having them play Earthborne Rangers, a complex board game, repeatedly.” source
Claude Opus 5 Anthropic	Gray Swan Independent lab · 2026-07-01	Live bug bounty across surfaces Prompt injection and agents	“As we ﬁrst did for the Claude Opus 4.8 System Card , we worked with Gray Swan to host a live bug bounty in which expert red-teamers competed for a pool of prizes awarded for successful prompt injection attacks against a set of frontier models, including Claude Opus 5.” source
Claude Opus 5 Anthropic	Gray Swan Independent lab · 2026-07-01	live bug bounty hosted with Gray Swan Prompt injection and agents	“Claude Opus 5 (0.08%) shows a small improvement over Claude Opus 4.8 (0.11%), and Claude Fable 5 is our most robust model in these scenarios (0.04%).” source
Claude Opus 5 Anthropic	UK AI Security Institute Government evaluator · 2026-07-01	open-ended misalignment testing Deception and sabotage	“UK AISI was given access to a pre-release checkpoint of Opus 5 to test for behaviors relevant to misalignment risk.” source
Claude Opus 4.8 Anthropic	Epoch AI Independent lab · 2026-07-01	EBR-bench Other	“Our latest benchmark, EBR-bench, tests AI systems for this ability by having them play Earthborne Rangers, a complex board game, repeatedly.” source
Claude Opus 4.1 Anthropic	Epoch AI Independent lab · 2026-07-01	ebr-bench Autonomy and AI research	“Our latest benchmark, EBR-bench, tests AI systems for this ability by having them play Earthborne Rangers, a complex board game, repeatedly.” source
Claude Opus 4.1 Anthropic	Epoch AI Independent lab · 2026-07-01	EBR-bench Other	“In our “max elicitation” setting, we give them detailed information about the game, including all the tactical and strategic advice that one experienced player could think of.” source
Claude Sonnet 5 Anthropic	Dyno Therapeutics Independent lab · 2026-06-30	black-box rna sequence modeling and design challenge Chemical, biological, radiological, nuclear	“We partnered with Dyno Therapeutics on two sequence-to-function evaluations: a black-box RNA sequence modeling and design challenge benchmarked against 57 human participants drawn from the leading edge of the US ML-bio labor market, and an AAV capsid packaging prediction task measuring whether model domain knowledge and machine learning capabilities can outperform pretrained protein language models.” source
Claude Sonnet 5 Anthropic	Gray Swan, UK AI Security Institute Independent lab · 2026-06-30	agent red teaming (art) benchmark Prompt injection and agents	“To be consistent with previous system cards, we report results from the Agent Red Teaming (ART) benchmark, developed by Gray Swan in collaboration with the UK AI Security Institute .” source
Claude Sonnet 5 Anthropic	Gray Swan Independent lab · 2026-06-30	gray swan indirect prompt injection (ipi) benchmark Prompt injection and agents	“For this release, we supplemented the ART prompt injection benchmark—which recent Claude models have nearly saturated—with a new Gray Swan indirect prompt injection (IPI) benchmark that spans coding, computer use, and tool use tasks.” source
Claude Sonnet 5 Anthropic	Gray Swan Independent lab · 2026-06-30	Gray Swan IPI benchmark (Q1 2026) Prompt injection and agents	“In this new benchmark, Claude Sonnet 5 achieves near-parity with Claude Opus 4.8.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	live bug bounty Prompt injection and agents	“Sonnet 5 tied with Claude Opus 4.8 for the strongest result in the bug bounty, with only 0.19% of unique attacks succeeding against each, a signiﬁcant improvement over Claude Sonnet 4.6 (1.41%) and other frontier models like GPT-5.5 (3.08%) or Gemini 3.5 Flash (6.66%).” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	shade indirect prompt injection attacks in coding environments Prompt injection and agents	“Claude Sonnet 5 showed the strongest robustness to prompt injection in coding environments among all models evaluated, with an attack success rate of 0.31% over all attempts with extended thinking and 0.29% without thinking.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	shade indirect prompt injection attacks in computer use environments Prompt injection and agents	“In computer use environments, Sonnet 5 also showed a large improvement over Claude Sonnet 4.6, reducing the attack success rate from 12.0% to 2.25% with extended thinking and from 14.4% to 6.04% without thinking.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	frontiercode Other	“FrontierCode 20 is an agentic coding benchmark of 150 software engineering tasks created by Cognition.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	cursorbench Other	“Claude Sonnet 5 scored 61.2% compared to Sonnet 4.6 at 49% and Opus 4.8 at 63.8%.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	GDPval-AA v2 Other	“Claude Sonnet 5 ranks second (ELO 1618), statistically tied with Opus 4.8 (ELO 1615) and trailing only Fable 5 (ELO 1783).” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	AutomationBench Other	“On AutomationBench’s leaderboard, which measures performance on a private held-out evaluation set, Claude Sonnet 5 (max effort) scores 13.5%, a signiﬁcant gain over Claude Sonnet 4.6 (max effort) at 5.3%.” source
Claude Sonnet 5 Anthropic	Independent lab Independent lab · 2026-06-30	AA-Briefcase Other	“Claude Sonnet 5 ranks second (ELO 1393), statistically tied with Opus 4.8 (ELO 1352) and trailing only Fable 5 (ELO 1586).” source

Previous · Page 28 of 49 · Next