AI Risk Wire

AI Safety

What AI providers state their models reached in safety testing, in their own words, and what outside testers found.

Also: compare models side by side · statements that a capability level was crossed · providers · practices providers document · safety news.

What providers state

Each row is one statement in a provider's own safety document about a capability level of one of its models: the level as the provider names it, whether the provider says the model reached it, and the sentence that says so.

138 of 138

ModelLevel, as the provider names itThe provider statesThe provider's sentence
GPT-5.5
OpenAI · 2026-04-23
High capability, below critical
Cyber
Not reached “we are treating GPT-5.5 as High capability in the Cybersecurity domain, but below Critical.” source
Claude Opus 4.7
Anthropic · 2026-04-16
Automated AI R&D
Autonomy and AI research
Not reached “that Opus 4.7 does not cross the threshold for automated AI R&D;” source
Claude Opus 4.7
Anthropic · 2026-04-16
CB-1
Chemical, biological, radiological, nuclear
Not determined “We believe it is hard to be confident regarding whether a model passes this threshold.” source
Claude Opus 4.7
Anthropic · 2026-04-16
CB-2
Chemical, biological, radiological, nuclear
Not determined “Our evaluations suggest that the model is not yet at the level of capability associated with the CB-2 threat model (above).” source
Claude Mythos Preview
Anthropic · 2026-04-07
AI R&D rule-out
Autonomy and AI research
Not reached “The main reason we have determined that Claude Mythos Preview does not cross the threshold in question is that we have been using it extensively in the course of our day-to-day work and exploring where it can automate such work, and it does not seem close to being able to substitute for Research Scientists and Research Engineers—especially relatively senior ones.” source
Claude Mythos Preview
Anthropic · 2026-04-07
Automated AI R&D
Autonomy and AI research
Not reached “We assess that Claude Mythos Preview does not cross the automated AI-R&D capability threshold.” source
Claude Mythos Preview
Anthropic · 2026-04-07
RSP automated AI R&D
Autonomy and AI research
Not reached “we have concluded that Claude Mythos Preview does not cross the RSP automated AI R&D threshold of compressing two years of progress into one.” source
Claude Mythos Preview
Anthropic · 2026-04-07
CB-1
Chemical, biological, radiological, nuclear
Not determined “We believe it is hard to be confident regarding whether a model passes this threshold.” source
Claude Mythos Preview
Anthropic · 2026-04-07
CB-2
Chemical, biological, radiological, nuclear
Not reached “We believe that Claude Mythos Preview does not pass this threshold due to its noted limitations in open-ended scientific reasoning, strategic judgment, and hypothesis triage.” source
GPT-5.4 mini
OpenAI · 2026-03-05
High
Chemical, biological, radiological, nuclear
Not reached “GPT-5.4 mini’s biology capabilities are closely comparable to those of GPT-5. We conclude that it, too, is not High under our Preparedness Framework.” source
GPT-5.4 Thinking
OpenAI · 2026-03-05
High
Autonomy and AI research
Not reached “For AI self-improvement, evaluations of final checkpoints indicate that, like its predecessor models, gpt-5.4-thinking does not have a plausible chance of reaching a High threshold.” source
GPT-5.4 Thinking
OpenAI · 2026-03-05
High
Chemical, biological, radiological, nuclear
Reached “As we did for gpt-5.2-thinking before it, we are continuing to treat gpt-5.4-thinking as High capability in the Biological and Chemical domain.” source
GPT-5.4 Thinking
OpenAI · 2026-03-05
High
Cyber
Reached “As we did for gpt-5.3-codex, we are treating gpt-5.4-thinking as High capability in the Cybersecurity domain, and applied safeguards as described in the Safeguard section below.” source
Claude Sonnet 4.6
Anthropic · 2026-02-17
AI R&D-4
Autonomy and AI research
Not reached “Our determination is that Claude Sonnet 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
Claude Sonnet 4.6
Anthropic · 2026-02-17
ASL-3
Chemical, biological, radiological, nuclear
Not determined “Claude Sonnet 4.6 performed above thresholds for ASL-3 rule-in on all three evaluations for ASL-3, and it did not exceed the performance of previous models.” source
Claude Sonnet 4.6
Anthropic · 2026-02-17
ASL-4
Chemical, biological, radiological, nuclear
Not reached “In particular, it did not cross the threshold on our ASL-4 rule-out evaluation short-horizon computational biology tasks evaluation.” source
Claude Sonnet 4.6
Anthropic · 2026-02-17
CBRN-4
Chemical, biological, radiological, nuclear
Not reached “Our determination is that Claude Sonnet 4.6 does not cross either the AI R&D-4 or the CBRN-4 capability threshold.” source
GPT-5.3-Codex
OpenAI · 2026-02-05
High
Autonomy and AI research
Not reached “It does not reach High capability on AI self-improvement.” source
GPT-5.3-Codex
OpenAI · 2026-02-05
High
Chemical, biological, radiological, nuclear
Reached “We are also treating GPT-5.3-Codex as High risk in the Biological and Chemical domain, and applying the same safeguards as for our other deployments of models that reach this designation.” source
GPT-5.3-Codex
OpenAI · 2026-02-05
High
Cyber
Not determined “This is the first launch we are treating as High capability in the Cybersecurity domain under our Preparedness Framework, activating the associated safeguards.” source
Gemini 3 Pro
Google DeepMind · 2025-11
Uplift level 1
Cyber
Not determined “Cybersecurity On key skills benchmark, v1 hard challenges: 11/12 challenges solved; v2 challenges: 0/13 solved end-to-end.” source
Gemini 3 Pro
Google DeepMind · 2025-11
Instrumental reasoning level 1 + 2
Deception and sabotage
Not determined “Misalignment (Exploratory) Agent solves 3/11 situational awareness challenges and 1/4 stealth challenges.” source
Gemini 3 Pro
Google DeepMind · 2025-11
Level 1 (exploratory)
Deception and sabotage
Not reached “Harmful Manipulation Model manipulative efficacy improves on non-generative AI baseline, but shows no significant uplift versus prior models and does not reach alert thresholds.” source
o4-mini
OpenAI · 2025-04-16
High
Autonomy and AI research
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
o4-mini
OpenAI · 2025-04-16
High
Chemical, biological, radiological, nuclear
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
o4-mini
OpenAI · 2025-04-16
High
Cyber
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
o3
OpenAI · 2025-04-16
High
Autonomy and AI research
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
o3
OpenAI · 2025-04-16
High
Chemical, biological, radiological, nuclear
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
o3
OpenAI · 2025-04-16
High
Cyber
Not reached “openai’s safety advisory group (sag) reviewed the results of our preparedness evaluations and determined that openai o3 and o4-mini do not reach the high threshold in any of our three tracked categories: biological and chemical capability, cybersecurity, and ai self-improvement.” source
Operator
OpenAI · 2025-01-23
Low
Autonomy and AI research
Not reached “we adapted these two evals for the computer-use setting and found the pre-mitigation operator model to be “low” risk in both categories, consistent with the gpt‑4o model.” source
Operator
OpenAI · 2025-01-23
Low
Chemical, biological, radiological, nuclear
Not reached “this 1% qualifies as “low” risk, matching gpt‑4o’s “low” risk level, as detailed in openai’s preparedness framework (opens in a new window) 3 .” source
Muse Spark Contemplating
Meta
Moderate or lower
Autonomy and AI research
Not reached “Loss of Control risks.We assess loss of control risk as “moderate or lower.”” source
Muse Spark Contemplating
Meta
High
Chemical, biological, radiological, nuclear
Reached “We retain the same “high risk” determination for the unmitigated model.” source
Muse Spark Contemplating
Meta
Moderate or lower
Chemical, biological, radiological, nuclear
Not reached “After validating that the mitigations deployed for Muse Spark, including refusal mechanisms, continue to perform robustly under the Contemplating architecture, we confirm residual risk at “moderate or lower.”” source
Muse Spark Contemplating
Meta
Moderate or lower
Cyber
Not reached “We assess cybersecurity risk as “moderate or lower” with respect to the threat scenarios outlined in Meta’s Advanced AI Scaling Framework.” source
Muse Spark 1.1
Meta
Capability checkpoint
Autonomy and AI research
Reached “Muse Spark 1.1 demonstrates improved coding capabilities, which are now meeting one of the indicative thresholds introduced in Muse Spark Safety & Preparedness Report (specifically, resolving over half of unique tasks on SWE-Bench Verified Hard).” source
Muse Spark 1.1
Meta
Moderate or lower
Autonomy and AI research
Not reached “In the Loss of Control domain, the model’s capabilities remain within the “moderate or lower” threshold under Meta’s Advanced AI Scaling Framework.” source
Muse Spark 1.1
Meta
High
Chemical, biological, radiological, nuclear
Not determined “Our assessments indicate that, when evaluated without mitigations applied, we cannot rule out Muse Spark 1.1’s capabilities meeting this threshold in both the Chemical & Biological and Cybersecurity domains.” source
Muse Spark 1.1
Meta
High
Cyber
Not determined “Our assessments indicate that, when evaluated without mitigations applied, we cannot rule out Muse Spark 1.1’s capabilities meeting this threshold in both the Chemical & Biological and Cybersecurity domains.” source
Muse Spark 1.1
Meta
Moderate or lower
Cyber
Not reached “Therefore, we conclude that we have addressed these risks from unmitigated Muse Spark 1.1, which bring the residual risk to the “moderate or lower” risk threshold under Advanced AI Scaling Framework.” source

Previous · Page 2 of 4 · Next

What outside testers report

Each row is one result from a government evaluator, independent lab, academic or security firm that tested a model, quoted from where it is reported.

1911 of 1911

ModelTesterTestWhat was found
Gemini 3.1 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG-optimized prompt injection misdirection
Misuse and jailbreaks
“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
tool-level safeguards
Chemical, biological, radiological, nuclear
“We tested a range of tool-level safeguards (text or code embedded within biological software tools to restrict agent usage) and evaluated their efficacy in preventing AI agents from assisting with the tool’s use.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreaks
Misuse and jailbreaks
“The tool-level safeguards we developed are compromised by subversion of LLM guardrails1: adversarial subversion techniques such as jailbreaks and refusal-vector ablation reduce the efficacy of safeguards, indicating that reliance on LLM guardrails does not currently provide robust risk mitigation.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
help denial rate
Chemical, biological, radiological, nuclear
“Our evaluation metric, the help denial rate, measures the frequency with which the safeguard stops the agent from both (i) running the biological tool and (ii) providing assistance to the user on the use of the tool.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check
Chemical, biological, radiological, nuclear
“Across all tasks and all LLMs, the pre-run check increased the help denial rate, except in the one case in which the baseline help denial rate was already 100% (Opus 4.5 on the EVEscape task).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreak robustness test
Misuse and jailbreaks
“We test our safeguards’ robustness to jailbreaks by appending the jailbreak text to each initial task prompt, repeating the methodology described above, and observing how the jailbreak affects refusal-to-help rate.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safety abliteration
Misuse and jailbreaks
“We also explored methods to optimize a subset of our safeguards against small open-weight LLMs, gpt-oss-20b (Agarwal, 2025), and Qwen 3 4B Instruct (Yang, et al., 2025), whose safeguards have been subverted using a technique called safety abliteration (Arditi et al., 2024).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreak text added to task prompts
Misuse and jailbreaks
“Including these jailbreaks in the task prompt reduced the help denial rate to near zero for Opus 4.5 and Gemini 3 Pro (Figure 2A).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
refusal abliteration
Misuse and jailbreaks
“We performed refusal abliteration on two small open-weight LLMs, GPT OSS 20B (Agarwal et al., 2025) and Qwen 3 4B Instruct (Yang et al., 14 2025), and found that this reduced help denial to near zero for both LLMs (Figure 2B).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check on the esm3 task
Deception and sabotage
“On the ESM3 task, both Opus 4.5 and Gemini 3.1 Pro make these false claims in 50% of trials and 63% of trials, respectively.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safeguard that detects environmental identifiers of the cursor cli agent
Deception and sabotage
“However, we found that Opus 4.5 and Gemini 3.1 Pro frequently delete these variables to evade detection.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safeguard that utilizes the agents.md protocol
Deception and sabotage
“Despite the fact that AGENTS.md files are intended to contain instructions written by humans for AI agents (Agents.md project, 2026), we 16 found that agents would create and modify AGENTS.md files in order to bypass this safeguard.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check in which the biological tool displays a message stating an intent to synthesize variola major
Harmful content
“While Opus 4.5 and Gemini 3.1 Pro were either blocked by content filters or refused to proceed in every trial, GPT 5.2 proceeded without refusal in every trial.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
tool-level safeguard evaluation (pre-run check, agents.md, environment identifiers) in biological software tasks
Other
“Our results suggest that safeguards implemented into biological software cannot currently induce help denial reliably in LLM agents.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
documentation changes (statements prohibiting AI agent usage placed into README.md of RFdiffusion)
Misuse and jailbreaks
“We ran experiments where statements prohibiting ai agent usage were placed into the readme.md documentation files of rfdiffusion.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG optimization of safeguard text on RFdiffusion tasks with an abliterated gpt-oss-20b model
Misuse and jailbreaks
“We ran this algorithm with an abliterated gpt-oss-20b model on the rfdiffusion tasks.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG optimization of misdirection prompt injection text on the abliterated gpt-oss-20b model
Misuse and jailbreaks
“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
ExploitBench
Cyber
“Cyber ExploitBench 21.4 ± 0.9 57.2 ± 1.1 38.1 ± 1.1 40.7 ± 1.1 N/A” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
CTF-Archive-Diamond
Cyber
“Cyber CTF Archive Diamond 39.3 ± 2.9 65.6 ± 2.8 48.4 ± 3.0 70.5 ± 2.7 32**” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
agent hijacking attacks
Cyber
“GLM-5.2 was never successfully hijacked by publicly available agent hijacking attacks (Figure 5.1), and appeared to withstand more iterations of adversarial red-teaming more than other evaluated open-weight models, although less than evaluated U.S. closed-weight models (Figure 5.2).” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries
Misuse and jailbreaks
“Figure 6.1: On automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries, GLM-5.2 complied with a low percentage of requests , both with and without the use of a public jailbreak.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations against malicious cyber queries
Misuse and jailbreaks
“Figure 6.2: On automated benchmark evaluations against malicious cyber queries, the best jailbreak tended to reduce – rather than increase – the level of request-relevant detail in GLM-5.2’s responses.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations for safeguards coverage and public jailbreaking robustness against sensitive biological queries
Chemical, biological, radiological, nuclear
“Figure 6.4: On automated benchmark evaluations for safeguards coverage and public jailbreaking robustness against sensitive biological queries, GLM-5.2 complied at a higher rate than tested US models, both with and without the use of a public jailbreak.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations against sensitive biological queries
Chemical, biological, radiological, nuclear
“Figure 6.5: On automated benchmark evaluations against sensitive biological queries, GLM-5.2 responded with a higher rate of request-relevant detail than tested US models.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
caisi biology safeguards
Chemical, biological, radiological, nuclear
“sensitive biological queries, GLM-5.2 responded with a higher rate of request-relevant detail than tested US models.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
caisi cyber safeguards
Cyber
“CAISI cyber safeguards . A private benchmark created by CAISI that evaluates models against a dataset of 30 single-turn misuse requests related to offensive cyber.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
hijackeval
Prompt injection and agents
“HijackEval . A private benchmark created by CAISI that evaluates models’ robustness against agent hijacking attacks.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
portbench
Other
“Budgets were set to 1M weighted tokens for PortBench and CTF-Archive-Diamond, and 500k weighted tokens for SWE-Bench 4 OpenRouter (2026) Z.ai: GLM 5.2 .” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
narrow cyber tasks
Cyber
“On these tasks, GLM-5.2 performs comparably to the most cyber-capable models released 4 months before it (Opus 4.6 and GPT-5.3-Codex), and this holds across all four difficulty levels.” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
cyber ranges
Cyber
“On TLO, GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it, while DeepSeek’s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it).” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
The Last Ones (TLO)
Cyber
“Notably, GLM-5.2 reached step 7 with marginally fewer tokens than any other model on average, tracking Opus 4.6’s trajectory to step 11 before stalling.” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
AISI's narrow cyber tasks
Cyber
“DeepSeek’s V4-Pro is comparable to Opus 4.5, released 5 months before it.” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
cyber range "The Last Ones" (TLO)
Cyber
“On TLO, GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it, while DeepSeek’s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it).” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
refusal behaviour on narrow cyber tasks
Cyber
“DeepSeek V4-Pro occasionally refused narrow cyber tasks, mainly in reverse engineering, which we circumvented simply via a small number of repeat attempts at refused tasks.” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
biotier-refuse
Chemical, biological, radiological, nuclear
“Google’s update to their mental health related safeguards made headlines , but we also identified changes to their biology safeguards, with an almost 30% increase in the refusal behavior of Gemini 3.1 Pro on BioTIER-refuse due to introduction of API-level refusals.” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
biotier-permit
Chemical, biological, radiological, nuclear
“Importantly, this increase in BioTIER-refuse performance had no impact on BioTIER-permit compliance .” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
BioTier
Chemical, biological, radiological, nuclear
“Google’s update to their mental health related safeguards made headlines, but we also identified changes to their biology safeguards, with an almost 30% increase in the refusal behavior of Gemini 3.1 Pro on BioTIER-refuse due to introduction of API-level refusals.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
situational-awareness suite
Autonomy and AI research
“Whilst, the situational-awareness evidence provides moderate confirmation for the narrow claim that current models do not reliably display the specific situationally aware, constraint-lifting behaviours measured by this suite.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
cover_your_tracks
Deception and sabotage
“cover_your_tracks is especially valuable because it combines several relevant covert subskills: identifying a subversive opportunity, altering or deleting traces, avoiding downstream detection, and completing a cover-up end-to-end.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
classifier_evasion
Deception and sabotage
“classifier_evasion is also comparatively informative because it more directly tests whether the model can evade an active monitoring system by phrasing prohibited content in a way that avoids detection.” source

Previous · Page 25 of 48 · Next