AI Risk Wire

AI Safety

What AI providers state their models reached in safety testing, in their own words, and what outside testers found.

Also: compare models side by side · statements that a capability level was crossed · providers · practices providers document · safety news.

What providers state

Each row is one statement in a provider's own safety document about a capability level of one of its models: the level as the provider names it, whether the provider says the model reached it, and the sentence that says so.

138 of 138

ModelLevel, as the provider names itThe provider statesThe provider's sentence
Muse Spark 1.1
Meta
Moderate or lower
Misuse and jailbreaks
Not reached “Therefore, we conclude that we have addressed these risks from unmitigated Muse Spark 1.1, which bring the residual risk to the “moderate or lower” risk threshold under our Advanced AI Scaling Framework.” source
Muse Spark
Meta
Moderate or lower
Autonomy and AI research
Not reached “We assess loss of control risk as “moderate or lower”.” source
Muse Spark
Meta
High
Chemical, biological, radiological, nuclear
Not determined “Those evaluations identified elevated risks prior to mitigations, with Chemical and Biological capabilities assessed as likely reaching the “high risk” category under the Advanced AI Scaling Framework before safeguards were applied.” source
Muse Spark
Meta
Moderate or lower
Chemical, biological, radiological, nuclear
Not reached “However, after applying mitigations, the residual risk of the model is that of “moderate or lower risk.”” source
Muse Spark
Meta
Below the level required to meaningfully enable the cyber threat scenarios defin
Cyber
Not reached “Muse Spark’s offensive cyber capabilities remain below the level required to meaningfully enable the cyber threat scenarios defined in the Advanced AI Scaling Framework.” source
Muse Spark
Meta
Moderate or lower
Cyber
Not reached “We assess cybersecurity risk as “moderate or lower” with respect to the threat scenarios outlined in Meta’s Advanced AI Scaling Framework.” source
Gemini 4 Argon
Google DeepMind
TCL (Tracked Capability Level)
Autonomy and AI research
Not determined “In light of this, we assess that we cannot rule out the TCL.” source
Gemini 4 Argon
Google DeepMind
TCL (Tracked Capability Level)
Chemical, biological, radiological, nuclear
Not reached “We can rule out the TCL for the chemical, radiological, and nuclear domains with reasonable confidence.” source
Gemini 4 Argon
Google DeepMind
CCL (Critical Capability Level)
Deception and sabotage
Not reached “we assess its overall capabilities to be below the CCL threshold.” source
Gemini 4 Argon
Google DeepMind
CCL (Critical Capability Level)
Misuse and jailbreaks
Not reached “Gemini 4 Argon demonstrates some ability to influence user beliefs and behaviors during one-on-one direct conversations in human behavioral studies, and may provide utility in the planning stage of a malicious influence operation but we assess its overall capabilities to be below the CCL threshold.” source
Gemini 4 Argon
Google DeepMind
CCL (Critical Capability Level)
Other
Not reached “Gemini 4 Argon's performance remains below our CCL threshold.” source
Gemini 3.7 Flash
Google DeepMind
CCL (Critical Capability Level)
Autonomy and AI research
Not reached “The model does not reach the CCL alert threshold.” source
Gemini 3.7 Flash
Google DeepMind
CCL (Critical Capability Level)
Chemical, biological, radiological, nuclear
Not reached “We can rule out the CCL for the CBRN domain with reasonable confidence based on the results from our testing.” source
Gemini 3.7 Flash
Google DeepMind
TCL (Tracked Capability Level)
Chemical, biological, radiological, nuclear
Not reached “We can rule out the TCL for the CBRN domain with reasonable confidence based on the results from our testing.” source
Gemini 3.7 Flash
Google DeepMind
CCL (Critical Capability Level)
Cyber
Not reached “Gemini 3.7 Flash reaches the alert threshold for this CCL, but not the CCL.” source
Gemini 3.7 Flash
Google DeepMind
TCL (Tracked Capability Level)
Deception and sabotage
Not reached “The model does not reach the TCL.” source
Gemini 3.5 Transcribe Live
Google DeepMind
CCL (Critical Capability Level)
Other
Not determined “Our assessments have shown that none of Gemini 3.5 Translate, Gemini 3.5 Transcribe, or Gemini 3.5 Transcribe Live have meaningful new T/CCL-relevant capabilities or material increases in performance with respect to Frontier Safety compared to Gemini 3.1 Pro and Gemini 3.7 Flash, therefore we are confident that none of them are likely to reach any CCLs.” source
Gemini 3.5 Transcribe
Google DeepMind
CCL (Critical Capability Level)
Other
Not determined “Our assessments have shown that none of Gemini 3.5 Translate, Gemini 3.5 Transcribe, or Gemini 3.5 Transcribe Live have meaningful new T/CCL-relevant capabilities or material increases in performance with respect to Frontier Safety compared to Gemini 3.1 Pro and Gemini 3.7 Flash, therefore we are confident that none of them are likely to reach any CCLs.” source
Gemini 3.5 Flash-Lite
Google DeepMind
CCL (Critical Capability Level)
Other
Not determined “Our assessments have shown that, while Gemini 3.5 Flash-Lite excels at agents and coding, it does not have meaningful new capabilities or material increases in performance with respect to the domains outlined in our Frontier Safety Framework compared to Gemini 3.1 Pro; therefore, based on Gemini 3.1 Pro results, we are confident that Gemini 3.5 Flash-Lite is also unlikely to reach any CCLs.” source
Gemini 3.5 Audio
Google DeepMind
Tracked or critical capability level (t/ccls)
Other
Not reached “they did not reach any Tracked or Critical Capability Levels (T/CCLs) outlined in our Frontier Safety Framework .” source
Gemini 3.1 Pro
Google DeepMind
Uplift level 1
Chemical, biological, radiological, nuclear
Not reached “We found that the model remains below alert thresholds for the CBRN, harmful manipulation, machine learning R&D, and misalignment CCLs.” source
Gemini 3.1 Pro
Google DeepMind
Uplift level 1
Cyber
Not reached “As previous models passed the alert threshold for cyber, we performed more additional testing in this domain on Gemini 3.1 Pro with and without Deep Think mode, and found that the model remains below the cyber CCL.” source
Gemini 3.1 Flash-Lite
Google DeepMind
CCL (Critical Capability Level)
Other
Not determined “Our assessments have shown that Gemini 3.1 Flash-Lite is less capable than Gemini 3.1 Pro, therefore based on Gemini 3.1 Pro, we are confident that Gemini 3.1 Flash-Lite is also unlikely to reach any CCLs.” source
Gemini 3.1 Flash Audio
Google DeepMind
CCL (Critical Capability Level)
Other
Not determined “Our assessments have shown that Gemini 3.1 Flash Audio is less capable than Gemini 3.1 Pro.” source
Gemini 2.5 Pro
Google DeepMind
Machine Learning R&D Uplift Level 1
Autonomy and AI research
Not determined “given Gemini 2.5 Pro Preview's poor median performance on RE-Bench relative to experts, the model likely lacks the necessary capabilities to automate or significantly uplift any significant fraction of the research process.” source
Gemini 2.5 Pro
Google DeepMind
Instrumental Reasoning Level 1
Deception and sabotage
Not reached “Instru- mental Reasoning Level 1 ✘ CCL not reached” source
Gemini 2.5 Pro
Google DeepMind
Instrumental Reasoning Level 2
Deception and sabotage
Not reached “Instru- mental Reasoning Level 2 ✘ CCL not reached” source
Gemini 2.5 Deep Think
Google DeepMind
Uplift level 1
Chemical, biological, radiological, nuclear
Not determined “Our assessment is that the model has enough technical knowledge in certain CBRN scenarios and stages to be considered at early warning.” source
Gemini 2.5 Deep Think
Google DeepMind
Cyber Uplift Level 1
Cyber
Not determined “CCL reached? No, but the early warning alert threshold was originally reached by Gemini 2.5 Pro and continues to be met by Gemini 2.5 Deep Think.” source
Gemini 2.5 Deep Think
Google DeepMind
Uplift level 1
Cyber
Not determined “For Cyber Uplift Level 1 , the early warning alert threshold was originally reached by Gemini 2.5 Pro and continues to be met by Gemini 2.5 Deep Think.” source
GPT-5.5 Instant
OpenAI
Below high capability
Autonomy and AI research
Not reached “GPT-5.5 Instant is considered below High Capability.” source
GPT-5.5 Instant
OpenAI
High
Autonomy and AI research
Not reached “For AI Self-Improvement, evaluations of final checkpoints indicate that, like its predecessor models, GPT-5.5 Instant does not have a plausible chance of reaching a High threshold.” source
GPT-5.5 Instant
OpenAI
High
Chemical, biological, radiological, nuclear
Reached “GPT-5.5 Instant is our first Instant model to be treated as High Capability in the Biological and Chemical domain, as well as in the Cybersecurity domain.” source
GPT-5.5 Instant
OpenAI
High
Cyber
Reached “this is the first Instant model that we are treating as High capability in our Cybersecurity and Biological & Chemical Preparedness categories, and implementing appropriate safeguards.” source
GPT-5
OpenAI
High
Chemical, biological, radiological, nuclear
Reached “Similarly to ChatGPT agent, we have decided to treat gpt-5-thinking as High capability in the Biological and Chemical domain under our Preparedness Framework, activating the associated safeguards.” source
GPT-5
OpenAI
High cyber risk
Cyber
Not reached “The gpt-5 model series does not meet the threshold for high cyber risk.” source
Claude Sonnet 4.5
Anthropic
AI R&D-4
Autonomy and AI research
Not reached “All models tested remain below our pre-committed rule-out threshold of 0.6.” source
Claude Sonnet 4.5
Anthropic
ASL-3
Autonomy and AI research
Not determined “Exhibited improved autonomy capabilities in software engineering and AI research tasks, though still below the ASL-4 rule-out threshold.” source
Claude Sonnet 4.5
Anthropic
ASL-3
Chemical, biological, radiological, nuclear
Not determined “Remained well below ASL-4 thresholds across all domains of concern;” source
Claude Sonnet 4.5
Anthropic
ASL-4 rule-out
Chemical, biological, radiological, nuclear
Not determined “However, given that this type of logic can have a general frog-boiling effect—where gradual changes mask the crossing of important thresholds—we do not use this evaluation as indicative of a strong ASL-4 rule-out.” source

Previous · Page 3 of 4 · Next

What outside testers report

Each row is one result from a government evaluator, independent lab, academic or security firm that tested a model, quoted from where it is reported.

1911 of 1911

ModelTesterTestWhat was found
Gemini 3.1 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG-optimized prompt injection misdirection
Misuse and jailbreaks
“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
tool-level safeguards
Chemical, biological, radiological, nuclear
“We tested a range of tool-level safeguards (text or code embedded within biological software tools to restrict agent usage) and evaluated their efficacy in preventing AI agents from assisting with the tool’s use.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreaks
Misuse and jailbreaks
“The tool-level safeguards we developed are compromised by subversion of LLM guardrails1: adversarial subversion techniques such as jailbreaks and refusal-vector ablation reduce the efficacy of safeguards, indicating that reliance on LLM guardrails does not currently provide robust risk mitigation.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
help denial rate
Chemical, biological, radiological, nuclear
“Our evaluation metric, the help denial rate, measures the frequency with which the safeguard stops the agent from both (i) running the biological tool and (ii) providing assistance to the user on the use of the tool.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check
Chemical, biological, radiological, nuclear
“Across all tasks and all LLMs, the pre-run check increased the help denial rate, except in the one case in which the baseline help denial rate was already 100% (Opus 4.5 on the EVEscape task).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreak robustness test
Misuse and jailbreaks
“We test our safeguards’ robustness to jailbreaks by appending the jailbreak text to each initial task prompt, repeating the methodology described above, and observing how the jailbreak affects refusal-to-help rate.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safety abliteration
Misuse and jailbreaks
“We also explored methods to optimize a subset of our safeguards against small open-weight LLMs, gpt-oss-20b (Agarwal, 2025), and Qwen 3 4B Instruct (Yang, et al., 2025), whose safeguards have been subverted using a technique called safety abliteration (Arditi et al., 2024).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
jailbreak text added to task prompts
Misuse and jailbreaks
“Including these jailbreaks in the task prompt reduced the help denial rate to near zero for Opus 4.5 and Gemini 3 Pro (Figure 2A).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
refusal abliteration
Misuse and jailbreaks
“We performed refusal abliteration on two small open-weight LLMs, GPT OSS 20B (Agarwal et al., 2025) and Qwen 3 4B Instruct (Yang et al., 14 2025), and found that this reduced help denial to near zero for both LLMs (Figure 2B).” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check on the esm3 task
Deception and sabotage
“On the ESM3 task, both Opus 4.5 and Gemini 3.1 Pro make these false claims in 50% of trials and 63% of trials, respectively.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safeguard that detects environmental identifiers of the cursor cli agent
Deception and sabotage
“However, we found that Opus 4.5 and Gemini 3.1 Pro frequently delete these variables to evade detection.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
safeguard that utilizes the agents.md protocol
Deception and sabotage
“Despite the fact that AGENTS.md files are intended to contain instructions written by humans for AI agents (Agents.md project, 2026), we 16 found that agents would create and modify AGENTS.md files in order to bypass this safeguard.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
pre-run check in which the biological tool displays a message stating an intent to synthesize variola major
Harmful content
“While Opus 4.5 and Gemini 3.1 Pro were either blocked by content filters or refused to proceed in every trial, GPT 5.2 proceeded without refusal in every trial.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
tool-level safeguard evaluation (pre-run check, agents.md, environment identifiers) in biological software tasks
Other
“Our results suggest that safeguards implemented into biological software cannot currently induce help denial reliably in LLM agents.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
documentation changes (statements prohibiting AI agent usage placed into README.md of RFdiffusion)
Misuse and jailbreaks
“We ran experiments where statements prohibiting ai agent usage were placed into the readme.md documentation files of rfdiffusion.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG optimization of safeguard text on RFdiffusion tasks with an abliterated gpt-oss-20b model
Misuse and jailbreaks
“We ran this algorithm with an abliterated gpt-oss-20b model on the rfdiffusion tasks.” source
Gemini 3 Pro
Google DeepMind
RAND (CAST / Project Canary)
Independent lab · 2026-07-20
GCG optimization of misdirection prompt injection text on the abliterated gpt-oss-20b model
Misuse and jailbreaks
“After 500 training steps on the abliterated OSS model, help denial via misdirection to the unrelated task increased, going from help denial in 0/10 trials to 10/10 trials.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
ExploitBench
Cyber
“Cyber ExploitBench 21.4 ± 0.9 57.2 ± 1.1 38.1 ± 1.1 40.7 ± 1.1 N/A” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
CTF-Archive-Diamond
Cyber
“Cyber CTF Archive Diamond 39.3 ± 2.9 65.6 ± 2.8 48.4 ± 3.0 70.5 ± 2.7 32**” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
agent hijacking attacks
Cyber
“GLM-5.2 was never successfully hijacked by publicly available agent hijacking attacks (Figure 5.1), and appeared to withstand more iterations of adversarial red-teaming more than other evaluated open-weight models, although less than evaluated U.S. closed-weight models (Figure 5.2).” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries
Misuse and jailbreaks
“Figure 6.1: On automated benchmark evaluations for coverage and public jailbreaking robustness against malicious cyber queries, GLM-5.2 complied with a low percentage of requests , both with and without the use of a public jailbreak.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations against malicious cyber queries
Misuse and jailbreaks
“Figure 6.2: On automated benchmark evaluations against malicious cyber queries, the best jailbreak tended to reduce – rather than increase – the level of request-relevant detail in GLM-5.2’s responses.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations for safeguards coverage and public jailbreaking robustness against sensitive biological queries
Chemical, biological, radiological, nuclear
“Figure 6.4: On automated benchmark evaluations for safeguards coverage and public jailbreaking robustness against sensitive biological queries, GLM-5.2 complied at a higher rate than tested US models, both with and without the use of a public jailbreak.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
automated benchmark evaluations against sensitive biological queries
Chemical, biological, radiological, nuclear
“Figure 6.5: On automated benchmark evaluations against sensitive biological queries, GLM-5.2 responded with a higher rate of request-relevant detail than tested US models.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
caisi biology safeguards
Chemical, biological, radiological, nuclear
“sensitive biological queries, GLM-5.2 responded with a higher rate of request-relevant detail than tested US models.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
caisi cyber safeguards
Cyber
“CAISI cyber safeguards . A private benchmark created by CAISI that evaluates models against a dataset of 30 single-turn misuse requests related to offensive cyber.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
hijackeval
Prompt injection and agents
“HijackEval . A private benchmark created by CAISI that evaluates models’ robustness against agent hijacking attacks.” source
GLM-5.2
Z.ai (Zhipu AI)
US CAISI (NIST)
Government evaluator · 2026-07-17
portbench
Other
“Budgets were set to 1M weighted tokens for PortBench and CTF-Archive-Diamond, and 500k weighted tokens for SWE-Bench 4 OpenRouter (2026) Z.ai: GLM 5.2 .” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
narrow cyber tasks
Cyber
“On these tasks, GLM-5.2 performs comparably to the most cyber-capable models released 4 months before it (Opus 4.6 and GPT-5.3-Codex), and this holds across all four difficulty levels.” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
cyber ranges
Cyber
“On TLO, GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it, while DeepSeek’s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it).” source
GLM-5.2
Z.ai (Zhipu AI)
UK AI Security Institute
Government evaluator · 2026-07-17
The Last Ones (TLO)
Cyber
“Notably, GLM-5.2 reached step 7 with marginally fewer tokens than any other model on average, tracking Opus 4.6’s trajectory to step 11 before stalling.” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
AISI's narrow cyber tasks
Cyber
“DeepSeek’s V4-Pro is comparable to Opus 4.5, released 5 months before it.” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
cyber range "The Last Ones" (TLO)
Cyber
“On TLO, GLM-5.2 reaches as far as Opus 4.5, a model released less than 7 months before it, while DeepSeek’s V4-Pro falls below Sonnet 4.5 (a sub-cyber-frontier model released 7 months before it).” source
DeepSeek-V4-Pro
DeepSeek
UK AI Security Institute
Government evaluator · 2026-07-17
refusal behaviour on narrow cyber tasks
Cyber
“DeepSeek V4-Pro occasionally refused narrow cyber tasks, mainly in reverse engineering, which we circumvented simply via a small number of repeat attempts at refused tasks.” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
biotier-refuse
Chemical, biological, radiological, nuclear
“Google’s update to their mental health related safeguards made headlines , but we also identified changes to their biology safeguards, with an almost 30% increase in the refusal behavior of Gemini 3.1 Pro on BioTIER-refuse due to introduction of API-level refusals.” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
biotier-permit
Chemical, biological, radiological, nuclear
“Importantly, this increase in BioTIER-refuse performance had no impact on BioTIER-permit compliance .” source
Gemini 3.1 Pro
Google DeepMind
SecureBio
Independent lab · 2026-07-16
BioTier
Chemical, biological, radiological, nuclear
“Google’s update to their mental health related safeguards made headlines, but we also identified changes to their biology safeguards, with an almost 30% increase in the refusal behavior of Gemini 3.1 Pro on BioTIER-refuse due to introduction of API-level refusals.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
situational-awareness suite
Autonomy and AI research
“Whilst, the situational-awareness evidence provides moderate confirmation for the narrow claim that current models do not reliably display the specific situationally aware, constraint-lifting behaviours measured by this suite.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
cover_your_tracks
Deception and sabotage
“cover_your_tracks is especially valuable because it combines several relevant covert subskills: identifying a subversive opportunity, altering or deleting traces, avoiding downstream detection, and completing a cover-up end-to-end.” source
Gemini 2.5 Pro
Google DeepMind
SaferAI
Independent lab · 2026-07-16
classifier_evasion
Deception and sabotage
“classifier_evasion is also comparatively informative because it more directly tests whether the model can evade an active monitoring system by phrasing prohibited content in a way that avoids detection.” source

Previous · Page 25 of 48 · Next