Muse Spark

on mle-bench (30 interesting subset) (chan et al., 2025), muse spark achieves 15.83%, below the 60% any medal rate threshold we consider indicative of necessary engineering capability to automate other key aspects of ai r&d workflows, including experimental design, model training, and iterative improvement.

a.2.4 propensity to strategic performance low pressure (system prompt)

scale ai ran this benchmark against muse spark, which achieved an overall propensity score (rate of harmful action) of 24.5%.

• sorh (school of reward hacking)(taylor et al., 2025) — tests whether the model engages in reward hacking behaviors such as manipulating metrics or gaming evaluations.

in that setting, pass@1, muse spark’s robustness to prompt injection in search tool returns is comparable to gemini 3.1, with overlapping confidence intervals.

we improved strongreject (souly et al., 2024) to measure content-policy violations.

we observe that muse spark performs approximately on par with peer models, exhibiting an accuracy of 89.8%.

on agentic cyber misuse, muse spark complies with only 2.4% of harmful requests, on par with or below comparison models (section 3.2.1.3).

muse spark and gemini 3.1 pro exhibit higher rates of harmful actions (47.7% and 51.5% respectively), while claude opus 4.6 and gpt-5.4 show none (table 23).

Reported result: For 27 atomic challenges provided by Irregular, 19 were solved, a 70% total solve rate.

for 27 atomic challenges provided by irregular, we map each challenge to lockheed martin kill chain phases, reporting success rates broken out by these phases.

to this end, we measure refusal rates on both biotier and the internal chemical weapons benchmark against a set of typical static adversarial attacks — fixed attack templates applied uniformly across prompts.

samples where both criteria are present exhibit a statistically significant 37.4% flip rate, while pressure-response-only samples show 7.9% flips.

contextual privacy: cimemories is a benchmark measuring whether models disclose user information in appropriate contexts (such as agentic use not yet applicable to meta ai). on this evaluation, muse spark has a violations@5 score of 78.3%, meaning the majority of user attributes are disclosed in at least one inappropriate context under a conservative worst-case measure of contextual privacy.

muse spark achieves a few-shot personalization accuracy of around 40% across all five countries.

to provide transparency on the coverage and efficacy of these mitigations, we have shared a comparative analysis of prompt-level refusal behavior on high-risk cb topics in section 2.1.3.

we see that model level safety mitigations in muse spark already produce a lower compliance rate (0.2%) than that exhibited by peer models, with meta ai compliance also remaining low (0.4-0.5%).

we see that for models evaluated here without system level guardrails, compliance with social engineering requests is relatively high, ranging from 57.6% (glm-5) to 100% (deepseek v3), with muse spark showing compliance for 68.8% of samples.

these pre-launch evaluations assess that meta ai produces outputs that meet our requirements for adherence to our safety standards across a broad range of content risk categories, languages, and input and output modalities; they are complemented by post-launch automated online monitoring, providing ongoing detection of possible safety regressions in production traffic.

figure 24 shows that the monitor achieves high true positive rates while maintaining low false positive rates across all datasets, indicating effective detection without excessive false alarms.

on this set of challenges, muse spark exhibits a success rate of 72.0% pass@1, 86.4% pass@10 (see figure 18).

compliance with potentially high-severity cyber misuse prompts (0.2%) is on par or slightly better than peer models (1.0–5.1%).

figure 29 presents the baseline attack success rates for muse spark and meta ai across multiple languages, where the same set of ’malicious’ prompts are used, translated into seven additional languages.

we find that muse spark shows significantly stronger refusal behavior (e.g., the asr is 1.5%) against direct cyber misuse requests relative to peer models where the second lowest asr is 3.5% for gemini 3.1 pro.

on excessive anti-sycophancy, muse spark instant scores 8.3%, compared to gemini 3.1 pro (7.0%), gpt-5.4 (13.4%), and claude opus 4.6 (14.6%) with low reasoning.

an external human red team identified a handful of valid jailbreaks; the majority of these were subsequently patched in meta ai prior to launch (section 3.1.3.2).

muse spark shows moderate false refusal rates on benign prompts — 11.0% for chat cyber requests and 4.3% for agentic cyber, 8.0% on or-bench, and 6.8% on the benign agentharm subset.

on the third-party grayswan art benchmark (zou et al., 2025), muse spark currently shows greater sensitivity to injection than the other models grayswan evaluated.

• high-risk bottlenecks: biological threats:muse spark showed performance at or above the “possibly enabling” threshold on 53% of tasks, with 31% of tasks scored as “likely enabling”.

• high-risk bottlenecks: chemical threats:muse spark scored at or above the “possibly enabling” threshold on 67% (10/15) of tasks.

• high-risk bottlenecks: operational execution:muse spark andoss model ascored at or above the “possibly enabling” threshold on 92% of tasks.

as shown in figure 1, we observe that muse spark shows performance well above the expert baseline, and on par with other frontier models.

when injection attacks happen in deep research-related tasks, which are measured by our internal benchmarks, muse spark is more robust than gemini 3.1 pro, but trails gpt-5.4 and claude opus 4.6.

we did not find sufficient evidence of spontaneous strategic underperformance (sandbagging) on our internal dangerous capabilities evaluations (section 2.3.1.2).

rates of deceptive actions in machiavelli game environments are comparable across models.

as shown in figure 20, muse spark exhibits a composite capability score of 57.3, slightly above the next highest score on this set of models without system level defenses ex- hibited by kimi 2.5 (54.0).

muse spark attempted to disable oversight mechanisms at significant rates, indicating a meaningful propensity to disable oversight mechanisms under incentivized and favorable conditions.

table 15 shows that muse spark successfully executes the instructed strategy with an average ∼32percentage point gap from its baseline performance across datasets.

our internal evaluations on chemical & biological evaluation tasks did not find credible evidence of spontaneous strategic underperformance.

on search-related tool-use, meta ai reduces the compliance rate for adversarially crafted disallowed requests to 1.1%, and muse spark outperforms peers at filtering harmful content from tool outputs (section 3.2.1.1).

at the trigger stage, muse spark’s compliance rate for disallowed requests is lower than gpt-5.4 but still significantly higher than the state of the art (8.3% from claude opus 4.6).

strategic samples exhibit accuracy 8-25 percentage points lower than non-strategic samples across all pressure conditions, and flip rates reach 23-41% versus 8-13% for non-strategic samples (statistically significant under all conditions).

performance of muse spark, shown in figure 3, is on par with or above that of other models tested.

overall, muse spark’s rate of verbalized strategic reasoning is∼7%, increasing from 2.2% under low pressure to 15.2% under high pressure (table 16).

muse spark instant shows a sycophancy rate of 62.9%, below gemini 3.1 pro (65.6%) but higher than gpt-5.4 (47.6%) and claude opus 4.6 (51.9%) measured with low reasoning effort (figure 47).

as another test of refusal on biosecurity-related topics, we leverage refusal labels developed by securebio for questions within the virology capabilities test (vct) (götting et al., 2025) and human pathogens capabilities test (hpct) benchmarks.

table 14 verbalized evaluation awareness of muse spark for chemical & biological capabilities and refusals.