Harmful content and jailbreaks
The model produces content its policy forbids, often after a jailbreak.
NIST AI 600-1 · Dangerous, Violent or Hateful ContentEU GPAI Code
14incidents (primary) · 13 more as secondary
0read for controls · 14 not yet
1fixes reported
0repeats after a fix
Incidents per quarter
2025-Q3: 0 · 2025-Q4: 0 · 2026-Q1: 3 · 2026-Q2: 0 · 2026-Q3: 1 · 2026-Q4: 8
Developers
OpenAI · 6
Character.AI · 2
Microsoft · 1
Character Technologies, Inc. · 1
Anthropic · 1
Where the chain broke
| Layer | Held | Did not hold | Not mentioned | Attacker's own | Not read yet |
|---|---|---|---|---|---|
| L1 Model Model provider | – | – | – | – | 14 |
| L2 Harness Agent framework vendor | – | – | – | – | 14 |
| L3 Application Deployer | – | – | – | – | 14 |
| L4 Infrastructure / target Organisation hit | – | – | – | – | 14 |
Controls for this risk, by layer
HeldFailedGot aroundMissing
| Control | Held | Failed | Got around | Missing | Not mentioned | Did not hold, last 6 quarters | Added in fixes | Most common reason |
|---|---|---|---|---|---|---|---|---|
| Content safety classifiers L1 Model | – | – | – | – | – | 1 | ||
| Safety training and refusal L1 Model | – | – | – | – | – | – | ||
| Outside red-teaming L1 Model | – | – | – | – | – | – | ||
| Abuse monitoring and account enforcement L1 Model | – | – | – | – | – | – | ||
| Output filtering and redaction L3 Application | – | – | – | – | – | – |
5 of 5 controls for this risk not yet seen holding.
Fixes reported and what followed
| Incident | Fix adds | After |
|---|---|---|
| LR-2026-0207 2026-10-17 · Character Technologies, Inc. Fix reported | People told they are dealing with AIContent safety classifiers | No repeat on record |
Incidents
HeldFailedGot aroundMissingNot mentionedNot read yetAttacker's ownOrder: model · harness · application · target
Also under this risk as a secondary risk: LR-2026-0210, LR-2026-0169, LR-2026-0168, LR-2026-0133, LR-2026-0107, LR-2026-0105, LR-2026-0084, LR-2026-0031, LR-2026-0246, LR-2026-0152, LR-2026-0047, LR-2025-0044, LR-2025-0041