Design a Content Moderation System at Scale

TL;DR. At 500M posts/day and a hard 200 ms pre-publish budget, you cannot run a 12B-parameter multimodal model on every post. The answer is a classifier cascade: cheap regex and hash filters exit 60% of traffic in 0.1 ms, small transformers handle 35%, and only 0.5% reaches human reviewers ^[1][2]. The pivotal trade-off is precision versus recall at speed: 95% precision on "remove" still produces 50,000 false positives per million actions, each becoming an appeal. TikTok spent over $2B on trust and safety in 2024 ^[2:1]; Meta actioned 16M pieces of hate speech in Q1 2024 alone ^[1:1]. The cascade is the only architecture that reconciles cost, latency, and accuracy at this scale.

Learning Objectives#

• Design a classifier cascade that holds per-post cost under a cent while hitting a 200 ms p99
• Model a policy taxonomy with severity tiers mapped to enforcement actions (allow, downrank, remove, escalate)
• Build a reviewer workflow with regional, language, and specialty routing plus anti-burnout rotation
• Specify an appeals state machine that is auditable, bounded-latency, and resistant to brigading
• Reason about PR-curve-driven threshold tuning and calibration drift under adversarial load
• Extend the text pipeline to images (perceptual hashing, CLIP) and video (frame sampling + audio transcription)

Intuition#

A single post takes 50 ms to classify with a multimodal model. Multiply by 500 million posts per day and you need 290,000 GPU-seconds per second, which is physically impossible on any cluster that exists. Meanwhile, YouTube ingests roughly 500 hours of video per minute ^[3], and NCMEC received 36.2 million reports of suspected child exploitation in 2023 containing over 100 million files ^[4]. No human workforce can review that volume either.

The naive approach (one big model on every post) works at 10,000 posts/day. At 500 million it collapses on three fronts: latency (50 ms per item blocks the publish path), cost (LLM inference at $150 per million items means $75,000/day just for the judge), and recall (a single model cannot cover text, images, video, and 200+ languages equally well).

The insight that unlocks the design: most content is benign. If you can exit 95% of traffic in the first two cheap tiers (regex, hash matching, small transformers), the expensive models and humans only see the genuinely ambiguous 5%. The cascade is not an optimization. It is the only architecture that fits the physics of the problem.

Requirements#

Clarifying Questions#

• Q: What content types are in scope? Assume: Text, images, video, audio, and livestream. Each has its own preprocessing pipeline feeding the shared cascade.
• Q: Pre-publish blocking or post-publish async? Assume: Both. Pre-publish blocking for high-confidence severe violations (CSAM, terrorism). Post-publish async re-check for ambiguous content.
• Q: One universal policy or per-market? Assume: Per-market overlays. EU DSA Article 17 requires statements of reasons ^[5]; German NetzDG has 24-hour removal windows; UK Online Safety Act 2023 requires risk assessments and safety duties enforced by Ofcom (Codes of Practice in force from 2025) ^[6].
• Q: What is the precision target? Assume: >95% precision on "remove" for critical tiers (CSAM, terrorism, self-harm). Recall negotiated per category.
• Q: Is appeals required from day one? Assume: Yes. DSA Article 20 makes internal complaint handling non-optional in the EU ^[5:1].
• Q: Livestream in scope? Assume: Yes. Rewrites the latency budget from 200 ms to "seconds per chunk" with frame sampling.

Functional Requirements#

• Classify every post into a policy taxonomy (hate, harassment, self-harm, sexual, violence, spam, CSAM, terrorism, misinformation) with severity tiers
• Enforce actions: allow, label, downrank, age-gate, remove, escalate to human, escalate to law enforcement
• Route ambiguous 0.5% to human reviewers with per-tier SLA (critical <30 min, low <48 h)
• Accept appeals, re-review with double-blind second reviewer, restore wrongly removed content
• Track repeat offenders with escalating consequences across linked accounts
• Expose transparency API for DSA-mandated quarterly reporting

Non-Functional Requirements#

• Load: 500M posts/day; 15K/sec peak during elections or viral events
• Latency: Pre-publish p99 < 200 ms; 95% of posts exit by Tier 2
• Human review: 0.5% of traffic (~2.5M/day); p50 decision < 30 min for critical
• Precision: >95% on "remove" for critical categories
• Cost: < $0.001 per post all-in (classifier + storage + amortized reviewer labor)
• Availability: 99.99% on sync path; graceful degrade to "allow + async re-check" on outage

Capacity Estimation#

Cost per million posts by tier: Tier 1 ~$0.01, Tier 2 ~$2, Tier 3 ~$20, Tier 4 ~$150, Tier 5 ~$8,000. The cascade keeps the blended cost under $1 per million by ensuring 95% of traffic never passes Tier 2. At 500M posts/day, that is $500/day for Tiers 1-2, plus $4,500 for Tier 3, $375 for Tier 4, and $20,000 for Tier 5 (human labor dominates).

API and Data Model#

API Design#

POST /moderate/v1/evaluate
  Body: { "content_id": "uuid", "author_id": "uuid", "modalities": ["text","image"],
          "text": "...", "media_refs": ["s3://..."], "locale": "en", "surface": "feed" }
  Returns: 200 { "decision": "allow|block|review", "policy_tags": ["hate"],
                  "confidence": 0.97, "tier_reached": 2, "latency_ms": 12 }
  SLA: 200 ms p99

POST /moderate/v1/appeals
  Body: { "content_id": "uuid", "author_id": "uuid", "reason": "..." }
  Returns: 201 { "appeal_id": "uuid", "state": "submitted" }

POST /internal/reviewers/decisions
  Body: { "reviewer_id": "uuid", "content_id": "uuid",
          "outcome": "remove|restore", "rationale_tags": ["hate_speech"] }
  Returns: 200 { "state": "decided" }

GET /moderate/v1/decision/{content_id}
  Returns: 200 { "final_decision": "removed", "policy_tags": [...],
                  "appeal_available": true }

Data Model#

-- Decisions store (ClickHouse for analytics, append-only)
table content_moderation_decisions (
  content_id       UUID,
  author_id        UUID,
  created_at       DateTime64,
  final_decision   Enum('allow','remove','label','downrank'),
  policy_tags      Array(String),
  tier_reached     UInt8,
  classifier_scores JSON,
  reviewer_id      Nullable(UUID),
  appeal_id        Nullable(UUID)
) ENGINE = MergeTree() ORDER BY (created_at, content_id)

-- Reviewer queues (Redis Streams, partitioned by queue_id)
-- queue_id = hash(severity, language, specialty)
-- Fields: content_id, enqueued_at, priority_score, locked_by, locked_until

-- Appeals (PostgreSQL, state machine)
table appeals (
  appeal_id        UUID PRIMARY KEY,
  content_id       UUID,
  author_id        UUID,
  state            Enum('submitted','assigned','in_review','decided','closed'),
  original_decision VARCHAR,
  appeal_decision  Nullable(VARCHAR),
  reviewer_id      Nullable(UUID),
  created_at       TIMESTAMP,
  decided_at       Nullable(TIMESTAMP)
)

High-Level Architecture#

Each cascade tier exits most traffic cheaply; only the genuinely ambiguous 0.5% reaches human reviewers, keeping cost under $0.001/post.

The write path is synchronous through Tiers 1-2 (combined p99 < 15 ms). If Tier 2 escalates, the item enters an async queue for Tiers 3-4 while the post is held in a "pending" state (not yet visible). For surfaces where brief exposure is tolerable (long-form articles), the post publishes immediately and Tiers 3-4 run post-publish with async takedown on violation.

The CSAM path is isolated: PhotoDNA hash matching runs in Tier 1 against the NCMEC database. On match, the item is blocked immediately, the account is frozen, and a CyberTipline report is filed automatically ^[7][8]. No LLM judge is in this path (prompt injection would be catastrophic).

Deep Dives#

Cascade architecture and threshold economics#

The cascade exists because of a simple cost inequality. Running Llama Guard 4 (12B parameters ^[9]) on every post at 500M/day would cost ~$10M/day in dedicated GPU fleet compute (vs. ~$75K/day at API pricing for a text-only judge). Running a distilled 50M-parameter BERT costs 1/100th as much per item. The cascade exploits the fact that most content is unambiguous.

Each tier emits a confidence score. If confidence exceeds a high threshold (clean or clearly violating), the item exits. If it falls in the ambiguous band, it escalates. The thresholds are the product of PR-curve analysis per category per surface. A newsfeed surface tolerates more false positives (aggressive removal) than a DM surface (privacy-sensitive).

OpenAI reported that using GPT-4 to interpret policy documents cut the policy-update cycle from months to hours ^[10]. This is the Tier 4 value proposition: zero-shot on novel policies with legible rationale. But LLM judges cost 100-1000x more per item than a small transformer, and they introduce a prompt-injection surface ^[11]. The architecture never lets the LLM judge be the sole gate.

Confidence-based escalation ensures 95% of traffic never touches a GPU-heavy model; each tier's threshold is tuned via PR-curve analysis per category.

Drift detection is critical. A classifier at 95% precision on launch day can silently degrade to 80% within weeks as adversaries probe its seams ^[12]. Mitigations: weekly retraining on reviewer-labeled data, a versioned adversarial eval set refreshed monthly, shadow deploys, and automatic rollback on PR-curve regression.

CSAM pipeline and mandatory reporting#

CSAM moderation is architecturally separate from the general cascade. It is legally mandated (18 USC 2258A ^[7:1]), deterministic by design, and carries the highest stakes for both false negatives (child exploitation continues) and false positives (innocent accounts destroyed).

The pipeline: (1) Every uploaded image is hashed with PhotoDNA before storage ^[13]. (2) The hash is compared against the NCMEC database and the GIFCT Hash-Sharing Database (408,000 distinct terrorist/CSAM items at end of 2024 ^[14]). (3) On match, the upload is blocked, the account is frozen, and a CyberTipline report is filed via NCMEC's REST API containing the legally required fields: IP address, URL, email, and identifying information ^[7:2][8:1]. (4) A specialist reviewer (not a general moderator) confirms the match within the isolated CSAM queue.

Meta open-sourced PDQ in 2019: a 256-bit DCT-based perceptual hash with a recommended matching threshold of Hamming distance 31 or less and a quality-score cutoff of 50 or above (the PDQ README recommends discarding hashes with quality <= 49) ^[15]. The reference FAISS-based PDQ matcher handles 4,000 images/sec on a single process ^[15:1]. For video, TMK+PDQF produces a 256 KB signature; vPDQ determines matches based on the share of matching frames ^[15:2].

Reviewer welfare is an architecture concern. More than 140 former Facebook moderators in Kenya were diagnosed with severe PTSD after reviewing CSAM and gore content ^[16]. Mitigations: hard rotation caps (4 hours/day maximum on CSAM), blur-by-default tooling with deliberate "unblur" action, embedded counselors, and compensation floors in vendor contracts ^[17].

Adversarial evasion and the multilingual gap#

Adversaries do not stand still. TokenBreak attacks inject zero-width spaces and homoglyphs to shift tokenization so the classifier sees different tokens from what a human reads ^[12:1]. Algospeak creates community dictionaries of euphemisms ("unalive" for suicide, "le$bian" to evade sexuality filters) that classifiers must learn to map back to canonical forms ^[18]. A 2024 paper demonstrated that homoglyph attacks effectively evade AI-generated-content detectors by shifting token log-likelihoods ^[19].

Mitigations form a defense-in-depth stack: Unicode normalization (NFKC) before tokenization, homoglyph confusables mapping from Unicode TR39, OCR of image-embedded text through the same text pipeline, adversarial augmentation during training, and a versioned red-team eval set refreshed monthly that blocks deploys on regression.

The multilingual gap compounds the problem. Amnesty International documented that Meta's algorithms "proactively amplified and promoted" anti-Rohingya content in Myanmar beginning as early as 2012, contributing to the 2017 ethnic cleansing that displaced more than 700,000 Rohingya ^[20][21]. The root cause: Burmese-language moderation was an order of magnitude below English-language moderation. The Oversight Board's 2025 assessment noted that "users outside the United States experienced limited content moderation tailored to their own regions" ^[22].

The architectural fix: a multilingual backbone (XLM-R class) with per-language fine-tunes for the top 20 languages, regional specialist reviewer pools for the tail, and per-language precision/recall tracking with alerts when any language drops below threshold. Low-resource languages (Amharic, Burmese, Tigrinya) require partnerships with local NGOs for red-team data ^[23].

Appeals state machine and anti-brigading#

Appeals are not a bolt-on. DSA Article 20 makes internal complaint handling mandatory ^[5:2], and appeals are the primary labeled-data source for measuring false-positive rates. The state machine: submitted to assigned (router picks queue by category and language), assigned to in_review (reviewer locks the item), in_review to decided (outcome submitted), decided to closed (author notified). If the outcome is "restore," a double-blind second review is triggered before closure.

Bounded-latency appeal lifecycle with double-blind second review for restore decisions; rate limiting rejects brigading attempts at the gate.

Anti-brigading: when a visible account is removed, tens of thousands of same-second appeals arrive (author plus brigaders plus bots). Mitigations: per-author rate limit (one appeal per content_id), deduplication on (author_id, content_id), anti-brigading filter that discounts nth-order reports from related accounts, and priority deferral of repeat submissions. The Oversight Board cases repeatedly process high-visibility removals that trigger downstream brigading ^[24].

Real-World Example#

Meta Community Standards Enforcement (2019-2025)

Meta operates the most publicly documented moderation system at scale. In Q1 2024, Meta actioned approximately 16 million pieces of hate speech on Facebook and Instagram ^[1:2]. The system uses a hybrid pre-publish block (for high-confidence severe violations) plus async post-publish re-check.

In January 2025, Meta announced the end of its third-party fact-checking program in the United States (operational since December 2016), replacing it with a crowdsourced Community Notes model ^[25]. The change removed professional fact-checkers from the US moderation pipeline while retaining automated enforcement for policy violations (hate speech, violence, CSAM). International fact-checking partnerships remain in place. Meta reported a roughly 50% reduction in US enforcement mistakes from Q4 2024 to Q1 2025, partly attributed to appeal and measurement improvements ^[26].

The stack is partially open-sourced. Meta released PDQ and TMK+PDQF in the ThreatExchange repository, along with the Hasher-Matcher-Actioner (HMA) platform for AWS deployment ^[15:3]. The architecture: ingest extracts modalities, pre-publish classifiers include text transformers (multilingual), image perceptual hashing, and CLIP-style image understanding. Post-publish, heavier models re-score and human reviewers handle the residual.

The Oversight Board (20 members as of 2026, operational since 2020) reviews contested decisions. Between January 2021 and December 2025, Meta responded to 326 Board recommendations; approximately 65% were fully or partially implemented ^[27][28]. In 2024, Meta adopted the Board's recommendation to label AI-created or altered content ^[24:1].

The canonical failure: the 2019 Christchurch livestream was viewed by fewer than 200 people during its 17-minute broadcast but re-uploaded 1.5 million times in 24 hours ^[29][30]. Automated detection failed because of the first-person body-cam framing. Post-Christchurch, Meta retrained systems on police/military body-cam footage to detect the pattern ^[31].

Meta's hybrid: hash matching blocks known-bad pre-upload; classifiers catch novel violations; the Oversight Board handles the contested residual.

Trade-offs#

The single biggest trade-off: precision versus recall at speed. High precision (few false positives) means more harmful content stays up. High recall (few false negatives) means more innocent content is removed, generating appeals and eroding trust. Real platforms resolve this differently per category: CSAM gets maximum recall (block everything suspicious, sort later); political speech gets maximum precision (only remove with high confidence, accept some harmful content staying up temporarily).

Scaling and Failure Modes#

At 10x load (5B posts/day): Tier 2 GPU fleet saturates. Mitigation: raise Tier 1 exit thresholds (more aggressive regex/hash matching), add GPU autoscaling with queue-depth triggers, and graceful degradation that raises Tier 2 confidence thresholds (more items exit as "clean" with async re-check).

At 100x load (50B posts/day): Human review becomes the bottleneck. Even at 0.5%, that is 250M items/day for humans. Mitigation: shift to LLM-assisted review where the model pre-labels and the human confirms/rejects (10x throughput per reviewer), accept higher automation rates for low-severity categories. TikTok began this transition in 2025, laying off hundreds of human moderators in the UK and Germany while increasing AI-driven moderation, despite new UK Online Safety Act obligations ^[32].

At 1000x load: The architecture shifts to edge-first classification. Lightweight models run on-device before upload; only content that passes device-side screening reaches the server cascade. Apple's on-device CSAM scanning (proposed 2021, shelved) was an early attempt at this pattern.

Failure mode: Classifier service outage. Response: graceful degrade to "allow + async re-check." All posts publish immediately; a backlog queue accumulates for re-scoring when classifiers recover. Detection: health check failures on Tier 2-3 endpoints. Recovery: drain backlog within SLA (critical content re-scored within 30 minutes).

Failure mode: Christchurch-style viral re-upload. Response: GIFCT Incident Response Framework activation ^[14:1]. Emergency hash injection into Tier 1 within minutes. Temporarily lower matching thresholds (accept more false positives). Activate cross-platform hash sharing. The GIFCT activated this protocol seven times in 2024 ^[14:2].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Cascade threshold tuning#

Your Tier 2 classifier has precision 96% and recall 88% at the current threshold for hate speech. Product wants recall raised to 93%. Using the PR curve, you find that achieving 93% recall drops precision to 89%. Calculate: (a) the additional false positives per day at 500M posts (assume 0.1% base rate of hate speech), and (b) the additional human review load this creates.

Key Takeaways#

• Cascades are physics, not optimization: LLMs are too expensive and too slow for 500M posts/day at 200 ms. The art is picking escalation thresholds that keep 95%+ of traffic in cheap tiers.
• CSAM is a separate pipeline: Legally mandated, deterministic, no LLM in the loop. Treat it as infrastructure, not a feature.
• Adversarial drift is the silent killer: Launch-day accuracy is not a long-term guarantee. Continuous red-teaming and PR-curve monitoring are the only durable defenses.
• Human reviewers are the labeled-data generator: Design the reviewer tool as a data-collection system, not just a queue. Their decisions keep classifiers calibrated.
• Appeals are first-class architecture: DSA compliance, user trust, and false-positive measurement all depend on a well-designed appeal state machine.
• Multilingual coverage inverts risk: The worst failures concentrate in the least-served languages. Per-language precision/recall tracking is mandatory.

Further Reading#

• Meta Community Standards Enforcement Report. Quarterly structured transparency data; the most detailed public view of large-platform enforcement numbers.
• facebook/ThreatExchange on GitHub. PDQ, TMK+PDQF, vPDQ, and the Hasher-Matcher-Actioner (HMA) reference platform, open-source BSD-licensed.
• OpenAI, "Using GPT-4 for content moderation". How LLM judges can interpret policy documents and cut the policy-update cycle from months to hours.
• Inan et al., "Llama Guard: LLM-based Input-Output Safeguard". Open-weight LLM-based moderation; representative Tier 3-4 architecture with configurable taxonomy.
• EU Digital Services Act (Regulation 2022/2065). Legal baseline for appeals, transparency, and notice-and-action in Europe; Articles 17, 20, and 24.
• Amnesty International, "The Social Atrocity". Canonical post-mortem on multilingual moderation failure at scale and Meta's role in the Rohingya crisis.
• GIFCT Annual Report 2024. Terrorism hash-sharing database stats, Incident Response Framework activations, and cross-platform coordination.
• OWASP LLM Top 10 (2025). Prompt injection and adversarial attack surface for LLM-in-the-loop moderation systems.

Flashcards#

References#