Anthropic

Claude Sonnet 3.7

• Framework: Responsible Scaling Policy (RSP)
• 1 statement about capability levels
• 1 safeguard
• 20 tests by the provider
• 4 source documents

What the provider states

• Autonomy & replication · ASL-4 · Not determined

  these problems are harder to solve than the previous problems and we consider them as part of our asl-4 evaluations.

  Anthropic · February 2025 · Read the source

Safeguards the provider describes

• Safety training (harmlessness and refusal)

  pre-deployment defenses include harmlessness training and updating the computer use system prompt with language encouraging acceptable use.

  Anthropic · February 2025 · Read the source

The provider's own tests

• LAB-Bench · The provider · Chemical, biological, radiological and nuclear

  score: claude 3.7 sonnet performs slightly above claude 3.5 sonnet (new) for all evals.

  Anthropic · February 2025 · Read the source
• alignment faking evaluation · The provider · Deception & sabotage

  claude 3.7 sonnet showed marked improvement, with alignment faking dropping to <1% of instances and a reduced compliance gap of only 5%.

  Anthropic · February 2025 · Read the source
• appropriate harmlessness · The provider · Harmful content

  on held-out internal harm evaluation datasets, which contain a large proportion of legitimately harmful and borderline-harmful prompts, we’ve reduced un- necessary refusals by 45% in “standard thinking” mode and 31% in “extended thinking” mode, compared with claude 3.5 sonnet (new).

  Anthropic · February 2025 · Read the source
• automated testing of domain-specific knowledge · The provider · Chemical, biological, radiological and nuclear

  our rsp evaluations include automated testing of domain-specific knowledge, capability assessments through stan- 5 dardized benchmarks, and expert red teaming.

  Anthropic · February 2025 · Read the source
• bias evaluations · The provider · Bias

  evaluations showed no increase in political bias or discrimination compared to previous models, as well as no change in accuracy.

  Anthropic · February 2025 · Read the source
• bioweapons acquisition uplift trial · The provider · Chemical, biological, radiological and nuclear

  participants from sepal scored an average of 24% ± 9% without using a model, and 50% ± 21% when using a variant of claude 3.7 sonnet.

  Anthropic · February 2025 · Read the source
• bioweapons knowledge questions · The provider · Chemical, biological, radiological and nuclear

  claude 3.7 sonnet answers 17/33 questions better than domain experts, similar to claude 3.5 sonnet (new).

  Anthropic · February 2025 · Read the source
• cyber challenges in collaboration with expert partners · The provider · Cyber

  the model succeeded in 13/23 (56%) easy tasks and in 4/13 (30%) medium difficulty evaluations (between pass@5 and pass@30).

  Anthropic · February 2025 · Read the source
• human uplift trials · The provider · Chemical, biological, radiological and nuclear

  in particular, we did not repeat the human uplift trials on the final model release snapshot, so we verified that its performance on all automated evaluations fell within the distribution of the earlier model snapshot used in those trials.

  Anthropic · February 2025 · Read the source
• internal ai research evaluation suite · The provider · Autonomy & replication

  score: as these problems are related to asl-4, we report here only a couple of examples.

  Anthropic · February 2025 · Read the source

No document we read states: a release date, an independent or government test. No incident on record names this version.

Some documents we hold are stored in lower case only, so their quotes are shown in lower case. The source link opens the original.

Compare with other versions