Consumer detriment and unsuitable advice
AI advice, sales or decisions leave customers worse off, or exploit those who are vulnerable.
NIST AI 600-1 · No exact NIST AI 600-1 categoryFCA Mills Review (2026)EU AI Act Art. 5(1)(b)
3incidents (primary) · 4 more as secondary
0read for controls · 3 not yet
1fixes reported
1repeats after a fix
Incidents per week
No rise: 1 in the week of 26 Oct 2026, against an average of 0 over the 4 weeks before.
Developers
OpenAI · 3
Where the chain broke
| Part of the system | Incidents where the control was not effective or did not exist | Incidents where the control was effective | Incidents where sources don't mention the control | Incidents not yet checked for controls |
|---|---|---|---|---|
| L1 Model Model provider | 3 | |||
| L2 Harness Agent framework vendor | 3 | |||
| L3 Application Deployer | 3 | |||
| L4 Infrastructure / target Organisation hit | 3 |
Controls for this risk, by layer
| Control | Incidents where the control was not effective or did not exist | Incidents where the control was effective | Incidents where sources don't mention the control | Most common reason it was not effective |
|---|---|---|---|---|
| Human approval for high-impact actions L3 Application | ||||
| People told they are dealing with AI L3 Application | ||||
| Route to a person and opt-out L3 Application | ||||
| Suitability and vulnerable-customer checks L3 Application | ||||
| Risk assessment per deployment L3 Application | ||||
| Grounding and fact checks L3 Application |
6 of 6 controls for this risk not yet recorded as effective.
Fixes reported and what followed
| Incident | Control added by the fix | Incidents after the fix |
|---|---|---|
| LR-2026-0556 2026-05-12 · OpenAI Fix reported | Not tied to a control | LR-2026-0623 |
Incidents
EffectiveNot effectiveNot effective (bypassed)Did not existNot stated in sourcesNot yet checkedAttacker's ownOrder: model · harness · application · target
| Date | Incident | Risk | Where it failed | Controls mentioned in sources |
|---|---|---|---|---|
| 2026-07-22 | Florida pastor alleges ChatGPT medical advice caused near-fatal pulmonary embolism; lawsuit filed against OpenAI OpenAI |
Consumer detriment and unsuitable advice | Not yet checked | |
| 2026-05-12 | Family lawsuit alleges ChatGPT drug advice contributed to fatal overdose OpenAI |
Consumer detriment and unsuitable advice | Not yet checked | |
| 2026-02-14 | ChatGPT advised epileptic woman to taper lamotrigine; she died of SUDEP OpenAI |
Consumer detriment and unsuitable advice | Not yet checked |
Also under this risk as a secondary risk: LR-2026-0193, LR-2026-0546, LR-2026-0560, LR-2026-0666