Design a Notification System (Push, SMS, Email at Scale)

TL;DR. A notification system is three problems in a trenchcoat: fan-out (one event produces N deliveries across M channels for K devices), provider integration (APNs, FCM, Twilio, and SES each have different rate limits, payload sizes, and error semantics), and delivery semantics (at-least-once with deduplication, because exactly-once across external providers is impossible). LinkedIn's Air Traffic Controller processes over 1 billion requests per day^[1] and cut member complaints in half by centralizing orchestration. The pivotal trade-off: throughput versus correctness under heterogeneous provider constraints.

Learning Objectives#

After this module, you will be able to:

• Design a fan-out architecture that isolates channel failures and scales each provider fleet independently
• Implement a device-token lifecycle driven by provider feedback (APNs 410, FCM UNREGISTERED)
• Apply caller-side and provider-side idempotency to achieve effectively-once delivery
• Justify retry strategies using delay topics with exponential backoff and jitter
• Estimate capacity for a platform delivering 10B notifications per day across 4 channels
• Trade off delivery latency against cost in a push-to-SMS fallback waterfall

Intuition#

A notification platform looks like a trivial CRUD app at 10 users. Accept an event, call APNs, done. At 10 billion notifications per day it collapses, and the reason is channel heterogeneity.

APNs can absorb effectively unlimited pushes over multiplexed HTTP/2 connections. Twilio SMS short codes cap at 100 messages per second^[2]. FCM's default quota is 600,000 messages per minute per project^[3]. If you put all channels on one queue, SMS backpressure stalls push delivery. If a provider goes down, the entire pipeline backs up.

The insight that unlocks the design: treat each channel as an independent delivery system with its own worker fleet, its own queue, its own rate limits, and its own failure domain. A thin orchestration layer sits in front, handling dedup, preferences, and fan-out. This is exactly what LinkedIn built with Air Traffic Controller^[1:1], what Uber built for marketing push^[4], and what Slack rebuilt in 2026 for notification preferences^[5].

The second insight: sending more is not better. Every low-quality notification erodes trust. LinkedIn's pre-ATC world let every team decide its own notification policy, producing "non-regulated, excessive, and low-quality member notification experience"^[1:2]. Centralization cut complaints in half with double-digit engagement lift. The platform is a gatekeeper, not just a delivery pipe.

Requirements#

Clarifying Questions#

• Q: Which channels must we support day one? Assume: Push (iOS + Android), SMS, email, and in-app inbox. Web push is a stretch goal.

• Q: What is the latency SLA for transactional vs promotional? Assume: Transactional (password reset, security alert) p99 < 5 seconds end-to-end. Promotional can be scheduled within a 1-hour window.

• Q: Do we need delivery confirmation? Assume: Yes for transactional. APNs provides no positive receipt^[6], so we rely on client-side read receipts for push confirmation.

• Q: Multi-region? Assume: Active-active in 2 regions. Kafka MirrorMaker for cross-region replication of notification state.

• Q: Who are the callers? Assume: Internal product services (ride-arrived, new-message, marketing-campaign). External webhook triggers are out of scope.

• Q: What compliance frameworks apply? Assume: TCPA for US SMS, GDPR for EEA, CAN-SPAM for email, A2P 10DLC for US 10-digit long codes^[7], RFC 8058 one-click unsubscribe for bulk email^[8].

Functional Requirements#

• Accept notification requests from product services with idempotency keys
• Fan out to all of a user's registered devices across all enabled channels
• Enforce per-user, per-category, per-channel preferences and frequency caps
• Retry transient failures with exponential backoff; dead-letter permanent failures
• Track delivery state (accepted, sent, delivered, read) per notification per device
• Support scheduled sends and digest batching for promotional categories

Non-Functional Requirements#

• Load: 10B notifications/day, 100K/sec peak ingest, 300K/sec peak provider calls (after fan-out)
• Latency: p99 < 5s for transactional push; p99 < 60s for email/SMS
• Availability: 99.95% on the ingest path; 99.9% on delivery (provider-dependent)
• Consistency: at-least-once delivery with effectively-once semantics via dedup
• Durability: no notification silently dropped; permanent failures land in DLQ for human review

Capacity Estimation#

• Read:write ratio: 1:10 (writes dominate; reads are inbox queries and admin dashboards)
• Hot partition risk: celebrity users or viral campaigns can produce 100K+ fan-out from a single trigger
• Bandwidth: ~50 Gbps egress at peak (540K msgs/sec * 12 KB avg rendered payload)
• SMS throughput ceiling: A2P 10DLC MMS was capped at 1 MPS account-level before March 2026^[9]; short codes provide 100 MPS guaranteed^[2:2]

API and Data Model#

API Design#

POST /v1/notifications
  Idempotency-Key: <uuid>
  Body: {
    "user_id": "u_abc",
    "category": "security",
    "priority": "high",
    "channels": ["push", "sms"],
    "template_id": "password_reset_v2",
    "params": { "code": "482910", "expires_min": 10 },
    "ttl_seconds": 300
  }
  Returns: 202 { "notification_id": "n_xyz", "status": "accepted" }
  Errors: 409 duplicate (returns original notification_id), 429 rate limited

GET /v1/notifications/{id}/status
  Returns: 200 {
    "notification_id": "n_xyz",
    "deliveries": [
      { "channel": "push", "device": "d_1", "state": "delivered", "at": "..." },
      { "channel": "sms", "state": "sent", "at": "..." }
    ]
  }

GET /v1/users/{id}/inbox?cursor=...&limit=50
  Returns: 200 { "items": [...], "next_cursor": "..." }

PUT /v1/users/{id}/preferences
  Body: { "category": "marketing", "channel": "push", "enabled": false }
  Returns: 200

Pagination uses opaque cursor tokens (base64-encoded notification_id). Rate limiting: 1,000 req/sec per caller service, enforced at the API gateway. Note: AWS SES sandbox limits to 200 messages/day and 1 msg/sec; production accounts scale to millions/day^[10].

Data Model#

-- Notification state (Cassandra, partitioned by user_id)
CREATE TABLE notifications (
  user_id       text,
  notification_id timeuuid,
  category      text,
  priority      text,
  template_id   text,
  params        map<text, text>,
  created_at    timestamp,
  ttl_seconds   int,
  PRIMARY KEY (user_id, notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC);

-- Delivery tracking (Cassandra, partitioned by notification_id)
CREATE TABLE deliveries (
  notification_id timeuuid,
  device_id     text,
  channel       text,
  state         text,   -- accepted | sent | delivered | read | failed
  updated_at    timestamp,
  provider_id   text,   -- apns-id, twilio-sid, ses-message-id
  PRIMARY KEY (notification_id, device_id)
);

-- Device registry (Cassandra, partitioned by user_id)
CREATE TABLE devices (
  user_id       text,
  device_id     text,
  platform      text,   -- ios | android | web
  token         text,
  state         text,   -- registered | active | stale | invalid
  last_seen     timestamp,
  PRIMARY KEY (user_id, device_id)
);

Core entities: a user has preferences and devices; each notification spawns one delivery record per device-channel pair.

High-Level Architecture#

Each channel's worker fleet scales independently; a Twilio outage cannot stall APNs delivery. Delay topics handle retries without blocking worker threads.

Write path. A product service POSTs a notification. The Ingest API checks the idempotency key in Redis (SET NX, 24h TTL). On first-seen, it produces to notifications.ingest partitioned by user_id and returns 202. The fan-out processor consumes, loads preferences and device tokens from a Redis-backed cache, renders the template, and emits one message per (device, channel) pair to the appropriate channel topic.

Delivery path. Channel workers consume from their topic, call the provider API, and classify the response. Success updates Cassandra state. Transient failures (429, 5xx) go to a delay topic with exponential backoff. Permanent failures (410, UNREGISTERED) invalidate the device token and drop the message.

Receipt path. Provider webhooks (FCM delivery receipts, Twilio status callbacks, SES events) feed a webhook receiver that updates delivery state in Cassandra. APNs provides no positive delivery receipt^[6:1]; true push confirmation requires client-side analytics.

Priority isolation. Separate Kafka topics per priority tier (push.ios.transactional, push.ios.promotional). Workers consume transactional first. A password reset never sits behind 10 million marketing pushes.

Deep Dives#

Deep dive 1: Fan-out, ordering, and backpressure across heterogeneous channels#

The fan-out processor is the brain of the system. It takes one ingest message and produces N delivery messages. The challenge: channels have wildly different latencies and throughput ceilings.

Throughput asymmetry. APNs accepts effectively unlimited pushes over multiplexed HTTP/2 connections (one connection, many concurrent streams). FCM allows 600,000 messages per minute per project^[3:1]. Twilio short codes cap at 100 MPS; toll-free numbers at 3 MPS baseline^[2:3]. A single marketing campaign targeting 10M users produces 10M push messages (absorbed in seconds) but 10M SMS messages (which would take 28 hours at 100 MPS on a single short code).

Per-user ordering. Kafka partitioning by user_id guarantees that all notifications for one user are processed in order by the fan-out processor^[1:3]. Cross-channel ordering is weaker: a user can receive the SMS before the push if the SMS worker is faster. This is acceptable because channels serve different purposes.

Backpressure. When a provider returns 429, the worker must not block. It publishes to a delay topic with a backoff timestamp. A scheduler consumer re-injects messages when the backoff elapses. At 20K QPS with a 30-second in-process retry, you would need 600,000 blocked threads^[11]. Delay topics cost nothing.

Circuit breaking. If a provider returns 429 for >30 seconds continuously, a circuit breaker pauses consumption from that channel's topic entirely. On recovery, traffic ramps gradually (1% to 5% to 25% to 100%) to avoid thundering-herd amplification^[11:1].

Hot-path flow from ingest through fan-out to APNs, showing idempotency check, per-attempt delivery, and retry via delay topic.

The 410 response in the sequence above triggers a device token state transition. Every token follows this lifecycle:

Token lifecycle driven by client re-registration and provider feedback; Invalid is a terminal sink that stops wasted sends.

Deep dive 2: Dedup, preferences, and quiet-hours engine on the hot path#

Every notification passes through three gates before dispatch: deduplication, preference check, and timing rules. All three must execute in the hot path without adding significant latency.

Layered deduplication. (1) Caller-supplied Idempotency-Key checked in Redis at ingest (SET NX, 24h TTL, following Stripe's pattern^[12]). (2) Provider-side idempotency: APNs accepts apns-id for dedup within its window^[6:2]; Twilio supports idempotency headers. (3) Content-hash dedup: hash of (template_id + params + user_id) catches semantic duplicates from different callers. (4) Time-windowed dedup: drop identical content to the same user within 5 minutes as a last line.

At 10B notifications/day with 24h TTL, the idempotency cache holds ~10B entries. A Bloom filter in front cuts Redis memory from ~170 GB to ~17 GB: 99% of checks return "definitely not a duplicate" from the filter, and only suspected duplicates hit Redis.

Preference enforcement. The fan-out processor loads preferences from a Redis-backed cache keyed by (user_id, category, channel). The preference model encodes: enabled/disabled, daily frequency cap, quiet hours (start/end in user's timezone), digest window, and consent flags (TCPA opt-in for SMS, GDPR consent for EEA)^[7:1]. Slack's 2026 rebuild separated "what to notify about" from "how to receive it" into independent fields, and saw settings engagement increase 5x post-launch^[5:1].

Quiet hours. The processor checks the user's stored timezone against current UTC. If the notification falls within quiet hours and is not time-sensitive priority, it is deferred to a scheduler topic that re-injects at quiet-hours-end. Stale timezone data (user traveled) is the main failure mode; a 1-hour grace window mitigates.

iOS interruption levels. Since iOS 15, server-side priority maps to client-side gating: passive notifications are silently delivered, active respect Focus modes, time-sensitive break through Focus, and critical (requires Apple entitlement) override Do Not Disturb^[13]. Apple Mail Privacy Protection also pre-fetches tracking pixels, making email open-rate metrics unreliable for MPP users^[14]. The fan-out processor sets apns-priority and interruption-level based on notification category.

RFC 8058 compliance. Every promotional email includes List-Unsubscribe and List-Unsubscribe-Post: List-Unsubscribe=One-Click headers, DKIM-signed, POST-able without cookies or auth^[8:1]. Gmail and Yahoo require this for bulk senders (>5,000 emails/day) since 2024^[8:2].

Deep dive 3: Delivery tracking, retries, and fallback waterfall#

The delivery subsystem answers: "Did the user actually see this notification?" The answer varies dramatically by channel.

Provider response classification. Workers classify responses into three buckets: transient (retry), permanent (drop/DLQ), and policy (TTL expired). FCM recommends at least 10 seconds before first retry, exponential backoff with jitter, and drops after ~60 minutes^[11:2]. FCM error codes like UNREGISTERED and INVALID_ARGUMENT require different handling paths^[15]. APNs returns 200 on acceptance but provides no positive delivery receipt^[6:3]. Twilio provides status callbacks (queued, sent, delivered, undelivered, failed). SES provides bounce, complaint, and delivery events via SNS^[10:1].

Retry mechanics. Delay topics implement retries without blocking workers. The worker publishes to a delay topic with a deliver_after timestamp. A scheduler consumer (or Kafka's built-in delayed delivery if available) re-injects when the backoff elapses. Backoff schedule: 10s, 30s, 60s, 120s, 300s, then drop to DLQ. Jitter is mandatory to prevent thundering herd on provider recovery^[11:3].

Fallback waterfall. For security-critical categories (password reset, 2FA, fraud alert), if push is not confirmed via client-side read receipt within 120 seconds, the system escalates to SMS. If SMS fails, it escalates to email. Promotional notifications never fall back to paid channels.

Only security-critical categories waterfall to paid channels; fallback is driven by client-side read receipts, not provider acceptance.

Cost control. Blind SMS fallback is dangerous. Since APNs gives no positive receipt, naive "if no APNs confirmation in 5 minutes, send SMS" fires on every push. At ~$0.0075/segment (Twilio US)^[2:4], a million daily users equals ~$7,500/day in wasted SMS. The fix: drive fallback off actual client-side read receipts (in-app analytics SDK reports "notification displayed"), not provider response codes.

Observability. Slack models each notification as its own OpenTelemetry trace (trace_id = notification_id) with spans at trigger, notify, sent, received, opened, and read-in-app^[16]. Span links connect to the sender's message trace without creating billion-span fan-out traces. Sampling is 100% for notification traces (customer-experience engineers need fidelity) versus 1% for message sends. This cut triage time by 30%^[16:1].

Real-World Example#

LinkedIn Air Traffic Controller#

LinkedIn's Air Traffic Controller (ATC) is the canonical production notification orchestrator. It processed over 1 billion requests per day for 546+ million members as of 2018^[1:4].

Architecture. ATC runs on Apache Samza with state in RocksDB and transport on Kafka^[17]. Services write notification requests to a Kafka topic. Partitioners distribute by hash of recipient member_id so all state for one member lives on one Samza task. Relevance processors score each request using ML models pushed from offline jobs and stored in RocksDB. Pipeline processors aggregate scores and make final decisions: drop, in-app only, in-app + push, digest, or Delivery Time Optimization^[1:5]. Facebook Messenger's Iris system uses a similar per-consumer pointer model for mobile sync, where each device maintains its own read cursor into a totally-ordered queue^[18].

Key decisions:

• Partition by recipient. All of a member's state (history, devices, settings, preferences) is co-located. Lookups are local RocksDB reads at "a couple of milliseconds" versus remote calls at 10-100ms^[1:6].
• ML models in RocksDB. Offline model outputs ingested via Kafka streams into local RocksDB avoid remote calls on the hot path.
• Samza Async API. Thread parallelism hides remote-call latency for unavoidable sender-side lookups. This reduced P90 end-to-end push latency from ~12 seconds to ~1.5 seconds^[1:7].
• Send-time optimization. Delivery Time Optimization picks each member's highest-engagement hour from historical data, smoothing spikes.

Impact. ATC "cut member complaints in half and created double-digit increases in member engagement site-wide"^[1:8]. The pre-ATC world was fragmented: every team decided its own notification policy. Centralization was the fix.

Uber's parallel evolution. Uber's RAMEN platform handles 1.5 million concurrent persistent connections and 250,000 messages per second for in-app state pushes^[19]. The 2022 reboot moved to gRPC/QUIC for bi-directional streaming, solving HTTP/1.1 SSE limitations on mobile^[20]. For marketing push, Uber's Consumer Communication Gateway (CCG) was introduced to address notifications "being sent within minutes and hours of each other" with conflicting messaging, after push volume grew to billions per month by late 2020^[4:1]. Their scheduler uses an XGBoost model combined with an integer linear program solver to jointly pick send time and notification selection per user, subject to per-user frequency caps and send-window constraints^[4:2].

Trade-offs#

Scaling and Failure Modes#

At 10x load (100B/day, 1M/sec peak): The fan-out processor becomes the bottleneck. Mitigation: horizontally scale Samza/Flink tasks; increase Kafka partitions for channel topics. The idempotency Bloom filter grows to ~170 GB (still fits in a Redis cluster). Provider quotas become the hard ceiling: request FCM quota increase, add more Twilio short codes.

At 100x load (1T/day, 10M/sec peak): Single-region Kafka cannot absorb 10M/sec writes. Mitigation: multi-region Kafka clusters with geo-routing (notifications processed in the user's home region). Template rendering becomes CPU-bound; pre-render templates at campaign creation time. SMS cost at this scale ($750K/day) forces aggressive push-first strategy with SMS reserved for security-only.

At 1000x load: The architecture shifts to edge-first: notification decisions made at edge PoPs with local preference caches, only state updates replicated to core. Provider APIs become the fundamental bottleneck; negotiate dedicated capacity with Apple and Google.

Failure mode: Provider outage (FCM down for 30 minutes). Circuit breaker pauses Android push topic consumption. Messages accumulate in Kafka (hours of retention). On recovery, gradual ramp prevents thundering herd. Messages past TTL are dropped. Users receive a single collapsed notification via collapse_key instead of 40 stale pushes.

Failure mode: Preference cache poisoned. A bad deployment writes incorrect preferences. Users receive notifications they opted out of. Detection: anomaly detection on opt-out rate spikes. Recovery: rollback cache, replay from source-of-truth DB, issue apology notification. Blast radius: bounded by cache TTL (5 minutes).

Failure mode: Kafka consumer lag spike. Fan-out processors fall behind during a viral campaign. Transactional notifications (security alerts) are delayed. Detection: consumer lag metric > 10K. Mitigation: priority topics ensure transactional traffic has dedicated consumer groups that are never starved by promotional volume.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Fan-out capacity for a ride-sharing app#

A ride-sharing app sends driver-match notifications. At peak, 200,000 ride requests per hour. Each request notifies the nearest 50 drivers. Each driver has 1.5 devices on average. Design the fan-out path and estimate peak provider QPS. What happens if FCM returns 429 for 30 seconds?

Key Takeaways#

• Three problems, one platform: fan-out, provider integration, and delivery semantics. Solve them independently with per-channel worker fleets connected by Kafka topics.
• Idempotency is non-negotiable: caller-supplied keys in Redis at ingest, provider-side dedup (APNs apns-id), and content-hash as last line.
• Device tokens are a dataset you maintain: synchronous invalidation on 410/UNREGISTERED prevents significant wasted sends.
• Retries belong in delay topics: backoff with jitter, respect TTLs, permanent failures to DLQ for human review.
• Centralize preference enforcement: LinkedIn's fragmented approach led to excessive notifications; centralization cut complaints in half^[1:11].
• APNs is write-only: true delivery confirmation requires client-side analytics, not provider response codes.

Further Reading#

• Air Traffic Controller: Member-First Notifications at LinkedIn. The canonical write-up explaining partition-by-recipient, RocksDB-first architecture, and the engagement impact of centralized notification orchestration.
• Tracing Notifications at Slack. Modeling notifications as OpenTelemetry traces with span links; explains why 100% sampling for notifications is worth the cost and how it cut triage time by 30%.
• How Uber Optimizes the Timing of Push Notifications using ML. ML + linear programming for send-time optimization; documents how push grew to billions per month and how conflicting, back-to-back notifications drove the Consumer Communication Gateway redesign.
• How Slack Rebuilt Notifications. Preference-model refactor separating "what to notify about" from "how to receive it"; read-time migration for rollback safety.
• FCM: Best Practices at Scale. Traffic spike taxonomy, exponential backoff requirements, and ramp-up schedules from Google's own documentation.
• APNs Provider API Reference. HTTP/2 headers, status codes, 4 KB payload limit, and the critical absence of positive delivery receipts.
• RFC 8058: One-Click Unsubscribe. The POST-based unsubscribe mechanism now required by Gmail and Yahoo for bulk email senders since 2024.
• Stripe: Designing Robust APIs with Idempotency. The foundational pattern for idempotent ingestion APIs that notification systems adopt wholesale.

Flashcards#

References#