How moderation works

Holding a hard line on the content in Prohibited content without reading your private conversations is a design problem. Here is how we solve it: in layers, most of them automated, with people involved only where they have to be.

Safety in layers

No single filter catches everything, so we stack several independent checks. Something has to get past all of them to reach anyone.

The layers

• Before generation. Your prompt or request is screened first. The strictest matches — anything reading as a minor above all — are blocked outright, before a single image or line is produced.
• After generation. Whatever is produced is checked again, on its own, against the hard lines before it is shown to you. Every generated photo goes past an image classifier that looks for anything reading as a minor, and when it is unsure, the photo is blocked.
• Strikes. A request or image that crosses the hard lines puts a strike on the account. The first takes away image features; the second suspends the account.
• Human review. Reports and appeals are read by a person, who removes the content with a written reason, or rejects the report and says why.
• Your reports. Anyone can flag content. Reports in the most serious categories — apparent minors first — jump the queue.

What this means for your privacy

The automated classifiers run on content, including private content, because that is the only reliable way to catch the hard lines. People are different: our reviewers do not browse your private chats or generations. A human sees your content only when it has been flagged by a classifier, reported by someone, or is part of a specific investigation. We do not sell your conversations, and we do not put them on display.

What gets stopped most

In rough order: content sexualizing apparent minors, real-person likenesses, scenes built on removing consent, and other illegal material.

If we got it wrong

Automated systems over-block sometimes. If something of yours was blocked or removed and you think it was a mistake, appeal it — a different reviewer takes a fresh look, and clear misfires help us tune the filters.

Illegal content

Where content is criminal, moderation stops being about the platform and starts being about the law: we remove it, close the account, and report it to the relevant authorities.