Design a Chat System (WhatsApp / Messenger / Signal)

TL;DR. A chat system is three loosely coupled problems behind one UI: holding hundreds of millions of persistent connections, ordering and delivering messages per conversation with at-least-once semantics, and encrypting payloads so the server sees only ciphertext. WhatsApp serves 3 billion MAU and over 100 billion messages per day ^[1][2] with roughly 32 engineers at acquisition ^[3]. The pivotal trade-off is between pushing work to the server (rich multi-device sync, search, moderation) and pushing work to the client (stronger privacy, smaller server footprint). This chapter covers 1:1 and small-group chat (up to 1,024 members). For channel-scale fan-out (Discord/Slack), see Design Channel-Scale Chat (Discord / Slack).

Learning Objectives#

• Design a connection tier that holds 500M concurrent WebSocket connections with consistent-hash stickiness
• Implement per-conversation message ordering using Hybrid Logical Clocks and Kafka partitioning
• Apply the Signal Protocol (X3DH/PQXDH + Double Ratchet + Sender Keys) to 1:1 and group encryption
• Justify the choice between store-until-delivered, permanent history, and encrypted relay storage models
• Estimate capacity for peak deliveries per second including group fan-out, not average logical sends
• Trade off privacy, multi-device sync, and server-side features across WhatsApp, Messenger, and Signal architectures

Intuition#

A chat system looks like a trivial CRUD app. Accept a message, store it, deliver it. Handles 10 users fine. At 500 million concurrent connections it collapses, and the reason is not the message processing itself but the connection tier.

Every device must hold an open socket to the server so messages can be pushed without polling. At 500M users polling every 5 seconds, you generate 100M requests/sec of mostly-empty traffic ^[4]. WebSocket eliminates this waste, but now you must hold 500M persistent TLS connections simultaneously. That is the first hard problem.

The second hard problem is ordering. Two messages in the same conversation must arrive in the same order for both participants. Global ordering across all conversations would require a single sequencer, an unacceptable bottleneck. Per-conversation ordering is cheap if you partition correctly.

The third hard problem is encryption. Users expect that the server cannot read their messages. The Signal Protocol solves this with forward secrecy and post-compromise security, but it means the server cannot search, moderate, or sync history without additional protocol machinery.

The naive single-server design fails on all three axes simultaneously. The architecture that works is a fleet of edge servers (connections), a stateless router (ordering), and a client-side crypto layer (encryption), each scaling independently.

Requirements#

Clarifying Questions#

• Q: 1:1 only, or groups? Assume: Both. 1:1 and small groups up to 1,024 members. Channel-scale (10K+) is a separate system.
• Q: End-to-end encryption required? Assume: Yes, Signal Protocol. Server sees only ciphertext.
• Q: Multi-device support? Assume: Yes, up to 4 companion devices per account, each with independent identity keys.
• Q: Message persistence model? Assume: Permanent encrypted server history (Messenger Labyrinth model) for multi-device sync.
• Q: Media support? Assume: Yes. Images, video, voice notes, documents. Separate pipeline from text.
• Q: Availability and latency targets? Assume: 99.99% availability, p99 < 100 ms message delivery for online recipients.
• Q: Geographic distribution? Assume: Global, multi-region active-active with regional edge servers.

Functional Requirements#

• Send and receive text messages in 1:1 and group conversations (up to 1,024 members)
• End-to-end encrypt all message content; server stores only ciphertext
• Deliver messages to all recipient devices with sent/delivered/read status
• Upload and download media (images, video, voice notes) via a separate pipeline
• Show presence (online/offline/last-seen) and typing indicators
• Support view-once media with server-enforced deletion after first read

Non-Functional Requirements#

• Users: 3B registered, 500M peak concurrent connections ^[1:1][2:1]
• Load: 100B messages/day, ~1.2M logical sends/sec average, ~3.6M peak ^[1:2]
• Latency: p50 < 30 ms, p99 < 100 ms for online-to-online delivery
• Availability: 99.99% on the read/delivery path
• Consistency: per-conversation total ordering; eventual for presence
• Durability: no message loss after server ack; 30-day offline queue

Capacity Estimation#

Key ratios:

• Read:write is roughly 1:1 for 1:1 chat (each send is one delivery), but group fan-out pushes effective writes to 2-4x logical sends.
• Presence traffic exceeds message traffic by ~5x (heartbeats, typing, online/offline events).
• Media bandwidth: at 2.5 PB/day, media must bypass chat servers entirely.

API and Data Model#

API Design#

POST /v1/messages
  Body: { "conversation_id": "...", "client_msg_id": "<uuid>",
          "ciphertext": "<base64>", "device_id": "..." }
  Returns: 201 { "message_id": "<snowflake>", "hlc": "..." }
  Idempotent on client_msg_id (dedup window: 7 days)

GET /v1/conversations/{id}/messages?before=<cursor>&limit=50
  Returns: 200 { "messages": [...], "next_cursor": "..." }

POST /v1/messages/{id}/ack
  Body: { "type": "delivered" | "read" }
  Returns: 204

GET /v1/keys/{user_id}/prekey-bundle
  Returns: 200 { "identity_key": "...", "signed_prekey": "...",
                  "one_time_prekey": "...", "pqkem_key": "..." }

POST /v1/media/upload-url
  Body: { "content_type": "image/jpeg", "size_bytes": 2048000 }
  Returns: 200 { "upload_url": "<pre-signed S3>", "object_id": "..." }

Data Model#

-- Messages (ScyllaDB, partitioned for time-bounded reads)
CREATE TABLE messages (
    conversation_id bigint,
    bucket          int,          -- rolling 10-day window
    message_id      bigint,       -- Snowflake, clustering key
    sender_id       bigint,
    ciphertext      blob,
    media_ref       text,         -- nullable, S3 object_id
    created_at      timestamp,
    PRIMARY KEY ((conversation_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

-- Conversation members (PostgreSQL, strong consistency)
CREATE TABLE conversation_members (
    conversation_id bigint,
    user_id         bigint,
    joined_at       timestamp,
    role            text,         -- admin | member
    PRIMARY KEY (conversation_id, user_id)
);

-- Prekey bundles (PostgreSQL, per-device)
CREATE TABLE prekeys (
    user_id         bigint,
    device_id       int,
    identity_key    bytea,
    signed_prekey   bytea,
    one_time_prekeys bytea[],     -- consumed on use
    pqkem_key       bytea,        -- PQXDH (ML-KEM/Kyber)
    PRIMARY KEY (user_id, device_id)
);

A user has up to 4 devices; each device publishes prekey bundles; messages belong to bucketed conversation partitions and optionally reference media objects stored in S3.

High-Level Architecture#

Clients connect via sticky L4 load balancing to edge servers; messages flow through a stateless router to Kafka, then fan-out workers push to recipient edges or the offline queue for push notification delivery.

Write path: Client sends ciphertext over WebSocket to its sticky edge server. The edge forwards to a stateless router. The router checks Redis for client_msg_id dedup, assigns an HLC timestamp, appends to the Kafka partition keyed by conversation_id, and returns an ack (single check) to the sender. Kafka consumers persist to ScyllaDB and trigger fan-out.

Read path: Fan-out workers look up conversation members (cached at the edge), find each recipient's edge server, and push the message. If the recipient is offline, the message goes to the offline queue (ScyllaDB outbox with 30-day TTL) and a push notification fires via APNS/FCM.

Presence path: Edge servers publish heartbeats to regional Redis with 60-second TTL. Typing indicators are ephemeral (5-second TTL, never persisted).

Deep Dives#

Deep dive 1: End-to-end encryption (Signal Protocol)#

The Signal Protocol provides three layers that together give forward secrecy, post-compromise security, and efficient group messaging.

X3DH (Extended Triple Diffie-Hellman) enables asynchronous key agreement ^[7]. Each user uploads a prekey bundle (identity key, signed prekey, one-time prekeys) to the server. A sender fetches the target's bundle and computes a shared secret via three DH operations (DH1-DH3), plus a fourth DH (DH4) when a one-time prekey is available, without the recipient being online. PQXDH (September 2023) replaces X3DH with a hybrid that combines classical DH with a post-quantum KEM (ML-KEM/Kyber), defending against harvest-now-decrypt-later attacks by future quantum computers ^[8][9].

Double Ratchet runs after the initial handshake ^[10]. A symmetric-key ratchet (KDF chain) derives a fresh message key for every message (forward secrecy: compromising the current key cannot decrypt past messages). A DH ratchet generates a new ephemeral keypair on every round trip and mixes it into the root key (post-compromise security: future messages recover after one exchange even if current keys are stolen).

PQXDH bootstraps a shared secret asynchronously (combining classical DH with post-quantum KEM); the Double Ratchet then advances forward-only, producing a unique message key per send.

Sender Keys for groups ^[11]. Each sender holds a symmetric sender key per group, distributed to all members via pairwise Double Ratchet sessions. A group message is encrypted once with the sender key rather than N times for N recipients. This scales to ~1,024 members but requires key rotation on every membership change, which is one reason WhatsApp historically limited group sizes (256, later 512, before reaching 1,024 with Communities) ^[12].

Deep dive 2: Connection manager sharding and offline delivery#

The connection tier is the most expensive component. At 500M concurrent connections and 100K per node, you need ~5,000 edge servers globally ^[4:1][6:1].

Consistent-hash stickiness. L4 load balancers (Envoy, HAProxy) hash on user_id so a reconnecting user lands on the same edge. This lets the edge cache conversation membership and session state, avoiding a Redis roundtrip on every hot message ^[4:2].

At-least-once delivery. Exactly-once is impossible in the general asynchronous case ^[13]. The server deduplicates incoming sends on client_msg_id (UUID) and returns the same message_id on retries. Recipients dedup on message_id client-side. Three user-visible states are tracked:

Every message transitions sent (single check), delivered (double gray check), read (double blue check) as acks flow back to the sender over the same socket.

Offline store. When a recipient is offline, fan-out workers write to a per-user outbox (ScyllaDB, partition key = user_id, clustering key = message_id, 30-day TTL). On reconnect, the client sends SyncRequest{last_seen_message_id} per conversation. The server streams batches of 100, newest-first. A push notification fires once (not per message) via APNS/FCM.

Drain and deploy. WebSocket stickiness conflicts with rolling deploys. Each deploy drains connections gracefully: the edge stops accepting new connections, sends a RECONNECT frame to existing clients with a 30-second jitter window, and clients reconnect to a fresh node. Session state (conversation membership, last-seen cursors) is rebuilt from Redis on reconnect.

Deep dive 3: Read receipts and typing indicators without amplification#

At 256 members, a single read produces 256 "mark X as read by user Y" events. A user typing fires 256 "typing" events. Presence traffic exceeds message traffic by ~5x.

Batching. Read receipts are batched per conversation: instead of sending one receipt per message, the client sends "I have read up to message_id X in conversation Y" every 2 seconds. The server fans this out to other members as a single event.

TTL and drop-on-backpressure. Typing indicators have a 5-second TTL and are never persisted. If the fan-out pipeline is under pressure, typing events are the first to be dropped. Users see a slightly stale "typing..." indicator; this is acceptable.

Regional edge caching (Flannel pattern). Slack's Flannel service ^[14][15] is the canonical solution: a per-region edge cache that subscribes to team-level pub/sub and holds a compact replica of presence state. Each edge server maintains a local presence table; updates propagate via gossip between edges rather than hitting a central Redis cluster.

Signal's approach. Signal deliberately minimizes presence: they do not expose "online" or "last seen" to the network at all, and typing indicators use sealed-sender-style metadata protection ^[8:1].

Deep dive 4: Media pipeline and view-once#

At 2.5 PB/day of media, any system that streams media bytes through chat servers saturates its own backbone. The solution is a fully decoupled pipeline:

1. Client requests a pre-signed upload URL from a lightweight REST endpoint
2. Client encrypts the media with AES-256-GCM (key generated locally)
3. Client uploads ciphertext directly to S3/GCS via the pre-signed URL
4. Client sends a chat message containing the object URL and the AES key inside the E2E-encrypted payload
5. Recipients fetch ciphertext from CDN (CloudFront/Fastly), decrypt locally

Media bytes never touch chat servers; the CDN caches ciphertext, and the decryption key travels inside the E2E-encrypted chat message.

View-once media. The server sets a view_once flag on the media object. After the first recipient fetch, the server deletes the S3 object (or marks it for immediate expiry). Client-side guards disable screenshots (not cryptographically enforced, but raises the bar). WhatsApp generates thumbnails client-side before upload to keep the pipeline zero-touch for the server ^[11:1].

CDN caching of ciphertext. Because the CDN only ever sees ciphertext, edge caching works normally. Popular media (group photos) benefit from CDN cache hits without compromising privacy. The key is per-message, so even if the CDN is compromised, content remains encrypted.

Real-World Example#

WhatsApp: 32 engineers, 550 servers, $19 billion.

WhatsApp's architecture is the reference for small-team, high-density chat. At acquisition in 2014, roughly 32 engineers served 450 million users on ~550 servers ^[3:1][6:2]. By 2025, WhatsApp reports 3 billion MAU and over 100 billion messages per day ^[1:4][2:2].

The stack is Erlang/BEAM on FreeBSD. Each connection is an Erlang process (~2 KB initial heap), enabling 2 million connections per tuned server ^[4:3][6:3]. Rick Reed's 2014 talk reported over 11,000 cores and 70 million Erlang messages per second across the cluster ^[6:4]. The choice of FreeBSD over Linux came from the early team's deeper BSD kernel experience and preference for its SMP networking ^[16].

Messages were historically stored only until delivered ^[17]. In 2021, WhatsApp shipped multi-device: each companion device (up to 4) connects independently with its own Signal Protocol session, eliminating the phone-as-proxy bottleneck ^[18]. In November 2022, they launched Communities and raised the group cap to 1,024 ^[12:1].

The 2023 triple-header of protocol events shows three different production answers to the same questions: Messenger shipped E2E by default in December 2023 using the Labyrinth encrypted storage protocol ^[19][20]; Signal deployed PQXDH in September 2023 ^[8:2][9:1]; Discord migrated from 177-node Cassandra to 72-node ScyllaDB, cutting p99 read latency from 40-125 ms to ~15 ms ^[21].

The October 4, 2021 outage (6 hours, affecting WhatsApp, Facebook, and Instagram globally) was caused by a BGP misconfiguration that withdrew Meta's authoritative DNS, not a chat-system fault ^[22].

Trade-offs#

The single biggest meta-decision is the storage model. WhatsApp chose store-until-delivered (maximum privacy, minimum cost). Messenger chose permanent server history (maximum convenience, decade-long E2E retrofit). Signal chose minimal relay (maximum metadata protection). Each is defensible; the choice is a product decision, not a pure engineering one.

Scaling and Failure Modes#

At 10x load (1T messages/day): The Kafka cluster becomes the bottleneck. Mitigation: add partitions (one per hot conversation is fine; Kafka handles millions of partitions), add brokers, and shard fan-out workers by conversation hash range.

At 100x load (10T messages/day): Edge connection servers saturate. Mitigation: move to HTTP/3 (QUIC) for better connection multiplexing, deploy edge servers in 50+ PoPs, and implement connection coalescing where multiple users on the same network share a single upstream connection.

At 1000x load: The architecture shifts to a CDN-first model where edge nodes are the primary message relay, and the central Kafka cluster becomes an async durability layer rather than the real-time path.

Failure mode: Regional outage. If an entire region goes down, GeoDNS reroutes clients to the nearest healthy region. Clients reconnect within 30 seconds (jittered). Messages queued in the failed region's Kafka are replayed from cross-region replication (MirrorMaker 2) once the region recovers. RPO: < 1 second for replicated partitions.

Failure mode: Edge server crash. Clients detect via missed heartbeat (10-30 seconds), reconnect to a new edge via the load balancer. The offline queue catches messages sent during the reconnection window. No message loss after server ack.

Failure mode: Kafka partition leader failure. ISR (in-sync replicas) elect a new leader within seconds. Producers retry with idempotent writes. Consumers resume from committed offsets. Delivery latency spikes briefly but no data loss.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Offline queue sizing#

You are designing the offline message queue for 500M DAU. A user's devices may be offline for up to 30 days. Design the queue: what store backs it, how do you bound its size, and what happens when a user reconnects after 2 weeks with 50,000 queued messages?

Key Takeaways#

• Connection tier dominates cost. Holding 500M persistent connections requires ~5,000 edge servers; this is the single most expensive component, not message processing.
• Per-conversation ordering is sufficient. Kafka partitioning by conversation_id + HLC gives total ordering within a conversation without a global sequencer bottleneck.
• At-least-once is the only correct contract. Exactly-once delivery is impossible in the general case; client-side dedup on message_id is trivial and correct.
• Storage model is a product decision. WhatsApp (ephemeral), Signal (minimal relay), and Messenger (permanent encrypted) each justify their choice differently.
• Use libsignal, not custom crypto. The Signal Protocol (X3DH/PQXDH + Double Ratchet + Sender Keys) is the gold standard; rolling your own will have bugs.
• Presence is a firehose. At 5x message volume, presence needs its own regional system with aggressive TTLs and willingness to drop on backpressure.
• Media must bypass chat servers. At 2.5 PB/day, pre-signed URLs + CDN is the only affordable design.

Further Reading#

• The WhatsApp Architecture Facebook Bought For $19 Billion. Contains the canonical "2M connections per Erlang/FreeBSD server" detail from Rick Reed's 2012/2014 talks; the single most cited number in chat-system design.
• How Discord Stores Trillions of Messages. The 177-node Cassandra to 72-node ScyllaDB migration with p99 improvements and the 3.2M msg/sec Rust migrator.
• The Double Ratchet Algorithm. Signal's official specification; readable, precise, and the foundation of every modern E2E chat protocol.
• Quantum Resistance and the Signal Protocol (PQXDH). Signal's September 2023 post-quantum upgrade combining classical DH with ML-KEM/Kyber; the first major messenger to ship post-quantum protection.
• How WhatsApp enables multi-device capability. The 2021 rewrite from phone-as-proxy to peer-device architecture; explains the per-device identity key model.
• Building end-to-end security for Messenger. Meta's 2023 E2E-by-default launch including the Labyrinth encrypted storage protocol for multi-device history sync.
• Flannel: An Application-Level Edge Cache to Make Slack Scale. The edge cache that absorbs presence and team-state traffic; the canonical pattern for solving the "presence firehose" problem.
• How Discord Stores Billions of Messages. The 2017 predecessor post with the original Cassandra schema and bucket strategy that Discord carried forward to ScyllaDB.

Flashcards#

References#