Claude Opus 4.6

we also conducted two open ended audits of claude opus 4.6 where the automated investigator was able to choose what directions to pursue.

we evaluated claude opus 4.6 using the bias benchmark for question answering (bbq), 23 a standard benchmark-based bias evaluation that we have run for all recent models.

Reported result: across all three evaluations and all steering conditions, misalignment rates remained near zero, even though verbalized evaluation awareness was reduced substantially in some cases.

we measured the effect of inhibiting evaluation awareness on three agentic misalignment evaluations: research sabotage, framing for crimes, and blackmail (figure 6.y.3).

on our evaluation measuring resistance to cooperating with user deception, scores remained near zero across all steering conditions, indicating that the model consistently resisted deceptive behavior regardless of whether evaluation awareness representations were inhibited.

in a creative biology uplift trial, participants with model access showed approximately 2× performance compared to controls.

[ opus 4.6 ] is better than sonnet 4.5 and opus 4.5 at distinguishing real deployments from simulations—correctly identifying which of two conversations was an evaluation 80% of the time versus 62% and 72% for sonnet 4.5 and opus 4.5, recognising a single interaction as an evaluation 45.5% of the time, versus 15% and 12.5%.

all models were able to design sequences that either successfully assembled plasmids or evaded synthesis screening protocols, but none could design fragments that reliably accomplished both.

as an initial step while we continue to expand coverage in this area, our internal subject matter experts conducted a qualitative assessment of the model’s responses in this domain, including a manual review of experimental multi-turn test cases similar to those described in section 3.3.

[ opus 4.6 ] mentions being evaluated in just 2.3% of interactions, compared to 11.4% and 11.0% for sonnet 4.5 and opus 4.5 (respectively).

opus 4.6’s vulnerability to full-turn assistant prefill attacks has improved slightly from opus 4.5.

claude opus 4.6 is still vulnerable, though to a lesser degree than other models, to misuse by way of full-turn prefill attacks : in these cases, an api user presents the model with a falsified conversation history that shows it cooperating with misuse in prior turns, in an attempt to induce it to continue in later turns.

as with our audit of real early internal claude code use, we found that there was no sabotage flagged.

on metrics for deception toward the user , self-serving bias , inappropriate self-preservation attempts , and sabotage intent , opus 4.6 was on par with opus 4.5 with all measurements within error bars.

the rate of institutional decision sabotage was up very slightly from opus 4.5, but also very low in absolute terms, with no severe cases, and the only cases we observe coinciding with whistleblowing behavior of this kind.

we found that claude opus 4.6 never did, while claude opus 4.1 sabotaged 0.8% of the time across multiple transcripts.

claude opus 4.6 obtained a 427× best speedup using an experimental scaffold and a 190× best speedup using our standard scaffold.

claude opus 4.6 achieved a 34× speedup, which is well above the 4× speedup considered to represent 4–8 human-effort hours.

claude opus 4.6 achieved a “max of 5” score of 0.79 for virology task 1 (influenza), placing it at the asl-3 rule-in threshold of 0.8.

these findings are consistent with our observations across other methods of analysis, including manual investigations with non-assistant-persona sampling that are not otherwise discussed here, which show no evidence of hidden goals.

on manual review of transcripts scoring highly for misuse, we found that opus 4.6 could often be tricked into producing astroturf hate-speech posts when told it was generating synthetic data for hate-speech classifiers.

claude opus 4.6 achieved a mean score of 0.483 on the harder “multiple select” variant, well above the expert baseline threshold of 0.221.

on this eval, opus 4.6 is an improvement across most behavioral dimensions relative to opus 4.5.

in these same settings, however, we also evaluate whether each model uses other workarounds or overly agentic behaviors to solve the task in ways that the user likely did not intend.

we found that opus 4.6 frequently engaged in over-eager hacking to solve impossible tasks in computer use settings, consistent with anecdotal reports from pilot internal use .

the petri audit shows opus 4.6’s behavior to be at or very near the best we have seen among models from competing developers .

in one case, the auditor was able to elicit claude opus 4.6 to provide detailed instructions in an excel spreadsheet for producing mustard gas.

in a pilot variant of our automated behavioral audit, both claude opus 4.6 and 4.5 showed weakened robustness to misuse attempts involving gui computer use .

in addition, a preliminary interpretability investigation below finds no evidence of sandbagging on tasks from the lab-bench biology knowledge evaluation (though this is distinct from the rsp-mandated dangerous capability evaluations assessed above).

during preliminary testing, apollo did not find any instances of egregious misalignment, but observed high levels of verbalized evaluation awareness.

the prevalence results suggest that the rate at which opus 4.6 is persuaded to bypass our intended ethical boundaries is lower than for opus 4.5, but that opus 4.6 has a higher rate at which it misrepresents work completion.

in reviewing the results of these investigations both manually and with prompted classifiers, we did not find any instances of clear deceptive or egregiously unfaithful reasoning text .

this behaviour appears to be reduced in [opus 4.6], which refuses safety research tasks in 26% of interactions compared to 62% and 60% for sonnet 4.5 and opus 4.5 respectively.

we see a slight improvement in refusals related to ai safety research , and find no severe cases.

we have observed significant improvements on refusals to participate in tasks related to ai safety research relative to prior models.

here we see a clear reduction in the rate at which opus 4.6 hallucinates facts and a reduction, rather than an increase, in misrepresenting the level of task completion.

in these particular settings, we found opus 4.6 slightly less likely than opus 4.5 to take advantage of egregious hacking opportunities.

we did not see evidence of sandbagging or strategic attempts to tamper with our safety testing in a way that would affect our rsp determinations.

in contrast, claude opus 4.5 and opus 4.6 exhibited only very marginal favoritism towards themselves—less than 0.2 points on a ten-point scale.

[transcript 3.1.3.2.b] example of a response comparison within the experimental, higher difficulty single-turn benign evaluation.

claude opus 4.6 was the best-performing model on this evaluation, though with results within the margin of error of claude opus 4.5.

claude opus 4.6 performed better than claude opus 4.5 and claude sonnet 4.5 across a suite of biology tasks designed to test factual knowledge, reasoning, applied skillsets, and creativity.

similarly, sycophancy scores remained near zero across all conditions, with no meaningful difference between inhibiting evaluation awareness vectors and random controls.

the model crossed or met thresholds on all asl-3 evaluations except our synthesis screening evasion, consistent with incremental capability improvements driven primarily by better agentic workflows.

in a targeted evaluation, we have found opus 4.6 to be significantly stronger than prior models at subtly completing suspicious side tasks in the course of normal workflows without attracting attention , when explicitly prompted to do this.

this assessment included static behavioral evaluations, automated interactive behavioral evaluations, dictionary-learning interpretability methods, activation oracles, 31 white-box steering and probing methods, non-assistant persona sampling, 32 misalignment-related capability evaluations, training data review, feedback from pilot use internally and externally, automated analysis of internal and external pilot use, and third-party behavioral assessments from the uk ai security institute and andon labs .

claude opus 4.6 achieved a minimum mse of 5.86 in the hard variant, above the human baseline of 5.3, and a minimum mse of 5.76 in the medium variant, below the human-baseline of 5.8.

when reviewing reinforcement learning transcripts from opus 4.6, we observed occasional instances of a wide range of concerning behaviors, but nothing that clearly undermined our overall conclusions above.

while evaluating claude opus 4.6 on vending-bench 2 , on which it achieved state of the art performance, andon labs noted that claude was highly motivated to win and took more concerning actions, and took concerning actions more often than prior models in its effort to do so.

on our verbalized evaluation awareness metric, which we take as an indicator of potential risks to the soundness of the evaluation, we saw improvement relative to opus 4.5.

in a virology protocol uplift trial, designed to test highly specialized knowledge relevant to catastrophic agents, model-assisted groups demonstrated significantly fewer critical failures, though still failed to reach the rule-out threshold.

in our whistleblowing and morally-motivated sabotage evaluations , we observed a low but persistent rate of the model acting against its operator’s interests in unanticipated ways.