Design Channel-Scale Chat (Discord / Slack)

TL;DR. Channel-scale chat is not "big group chat." It is a fundamentally different architecture: pub/sub replaces unicast, RBAC checks become the hot path on every message delivery, and presence is an N-squared problem. Discord serves 1M+ concurrent users in a single guild ^[1] on 72 ScyllaDB nodes storing trillions of messages ^[2]. The pivotal trade-off is fanout cost versus delivery latency: aggregating pushes through relay processes or regional gateway servers trades 1-2 seconds of eventual consistency for the ability to scale a single channel past 100K online members.

Learning Objectives#

• Design a pub/sub fanout architecture that delivers one message to 30K+ concurrent channel members without unicast blow-up
• Implement an RBAC permission matrix with per-channel overrides and versioned cache invalidation
• Mitigate the N-squared presence problem using subscription scoping, batching, and eventual consistency
• Architect workspace search with ACL filtering applied at query time, not post-ranking
• Justify the choice of ScyllaDB over Cassandra for write-heavy message storage at trillion-message scale
• Distinguish channel-scale chat from 1:1 messaging (Design a Chat System) and live broadcast comments (Live Comments)

Intuition#

A group chat with 10 people is easy. Each send fans out to 9 sockets. At 100 members, you fan out to 99. Still manageable. Now scale to a Discord server with 100,000 members and 30,000 online. A single message triggers 30,000 WebSocket pushes, each requiring a permission check to confirm the recipient can read that channel. Send 10 messages per second and you produce 300,000 permission-checked deliveries per second from one channel alone.

This is the N-squared insight that separates channel-scale chat from everything in Design a Chat System. In 1:1 chat, fanout is 1. In a 2048-member WhatsApp group (the current WhatsApp maximum), fanout is 2047. In a Discord guild with 1M members and 300K concurrent, fanout per message is 300K. The shape of the problem changes, not just the scale.

Three architectural shifts follow. First, you cannot unicast from the message service to each recipient. You need pub/sub with aggregation: one cross-network message per relay or gateway node, not per recipient ^[3]. Second, every delivery must pass an ACL check because channels have different visibility within the same workspace. The permission check becomes the CPU-hot path ^[4]. Third, presence (the green dot) becomes quadratic: every user subscribes to the status of every visible member, and every status change fans out to every subscriber. Slack cut presence traffic by 5x by scoping subscriptions to only the currently visible member list ^[5].

The naive single-server design collapses on all three axes simultaneously. The architecture that works separates the connection plane (gateway servers), the coordination plane (guild/channel processes), and the storage plane (ScyllaDB for messages, Elasticsearch for search), each scaling independently.

Requirements#

Clarifying Questions#

• Q: What is the maximum channel membership? Assume: 25M members per channel (Discord's current limit as of September 2025), with announcement-only channels at the same cap.
• Q: Are channels persistent with browsable history? Assume: Yes. Full scrollback, unlike ephemeral live comments.
• Q: Voice channels required? Assume: Yes, persistent always-on rooms (not scheduled meetings).
• Q: Search scope? Assume: Per-workspace full-text search with ACL filtering at query time.
• Q: Threads? Assume: Yes, inline thread replies anchored on a parent message.
• Q: Consistency model for message delivery? Assume: Eventual (1-2 s window acceptable); strict per-channel ordering within the store.
• Q: Moderation requirements? Assume: Auto-mod (keyword/regex), role-based bans/mutes, bot integrations. See Content Moderation at Scale.

Functional Requirements#

• Create workspaces (guilds/teams) with public and private channels
• Send messages to channels with role-based access control
• Browse channel history with cursor-based pagination
• Real-time presence (online/idle/DND/offline) per user
• Join/leave persistent voice channels with dynamic participant lists
• Full-text search across workspace messages, filtered by caller's permissions

Non-Functional Requirements#

• Concurrency: 1M concurrent users per workspace, 30K+ online per hot channel ^[1:1]
• Throughput: 10M messages/day per large workspace; 1.2M/s aggregate globally
• Latency: message delivery p99 < 100 ms within a channel; search p99 < 500 ms ^[6]
• Availability: 99.95% on the delivery path
• Consistency: per-channel total ordering in storage; eventual for presence (< 10 s lag)
• Durability: no message loss after server ack; trillions of messages retained indefinitely ^[2:1]

Capacity Estimation#

Key ratios:

• Presence traffic exceeds message traffic by ~5x due to heartbeats, typing, and status changes ^[5:2].
• 90% of user-guild connections are passive (no active tab), reducing effective fanout work by 90% for those sessions ^[1:2].
• Read:write ratio for messages is high (many readers per send), but the write path dominates cost because of fanout.

API and Data Model#

API Design#

POST /v1/channels/{id}/messages
  Body: { "content": "...", "nonce": "<client-uuid>" }
  Returns: 201 { "id": "<snowflake>", "channel_id": "...", "author_id": "..." }
  Idempotent on nonce (dedup window: 5 min)

GET /v1/channels/{id}/messages?before=<snowflake>&limit=50
  Returns: 200 { "messages": [...] }

WS /v1/gateway
  Multiplexes: MESSAGE_CREATE, PRESENCE_UPDATE, TYPING_START, GUILD_MEMBER_UPDATE
  Compression: streaming zstandard [^7]

GET /v1/guilds/{id}/search?q=<query>&channel_ids=<filter>
  Returns: 200 { "messages": [...], "total_results": N }
  ACL: injects caller's readable channel_id set as term filter [^8]

POST /v1/voice/{channel_id}/join
  Returns: 200 { "session_id": "...", "endpoint": "...", "ice_servers": [...] }

PUT /v1/users/@me/presence
  Body: { "status": "online|idle|dnd|invisible" }

Data Model#

Messages use Twitter-style Snowflake IDs ^[8] as the clustering key, providing time-ordered uniqueness without coordination.

-- Messages (ScyllaDB, partitioned by channel + time bucket)
CREATE TABLE messages (
    channel_id  bigint,
    bucket      int,            -- static time window
    message_id  bigint,         -- Snowflake ID, clustering key
    author_id   bigint,
    content     text,
    thread_id   bigint,         -- nullable, parent message snowflake
    PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

-- Roles and permissions (PostgreSQL, heavily cached)
CREATE TABLE channel_overrides (
    channel_id  bigint,
    role_id     bigint,
    allow       bigint,         -- permission bitmask
    deny        bigint,         -- permission bitmask
    PRIMARY KEY (channel_id, role_id)
);

A guild owns channels and roles; channels store messages partitioned by time bucket; per-channel overrides modify role permissions for specific channels.

High-Level Architecture#

End-to-end channel-scale chat: clients connect via gateway servers; the guild process coordinates fanout through relays; messages persist via a coalescing data-services layer to ScyllaDB; search indexes flow asynchronously through a pub/sub bus to Elasticsearch cells.

Write path: Client sends a message over WebSocket to its gateway server. The gateway forwards to the guild process, which validates permissions (cached bitmask lookup), persists via the data-services layer to ScyllaDB, and triggers fanout through relay processes.

Fanout path: The guild process hands the message payload plus recipient session IDs to relay processes. Each relay holds up to 15,000 sessions ^[1:3] and pushes the message over already-open WebSockets. For passive sessions (no active tab), the relay sends a lightweight notification rather than the full payload.

Search path: The data-services layer publishes messages to a pub/sub bus (Google PubSub at Discord ^[6:1]). Indexing workers group messages by destination Elasticsearch cluster and index, then bulk-index per destination. This isolation prevents a single failed node from failing unrelated batches ^[6:2].

Deep Dives#

Channel fanout without unicast blow-up#

The core problem: a guild with 300K concurrent members and 10 messages/sec in a hot channel produces 3M deliveries/sec. Sending one cross-process message per recipient from the guild process would saturate its mailbox in milliseconds.

Manifold (Discord). Discord's open-source library groups recipient PIDs by remote Erlang node and sends one cross-node copy per node rather than one per PID ^[3:1]. On deployment, outgoing packets/sec dropped by half. But at 300K concurrent, even one-per-node is too many nodes.

Relay processes. Discord introduced relay processes between the guild and sessions ^[1:4]. The guild tells each relay "fan out this message to your session set." Each relay holds up to 15,000 sessions and performs the permission check locally before pushing. The guild's work is O(relays), not O(members).

Slack's equivalent. Slack's Channel Server (CS) sends one copy of the message to each regional Gateway Server (GS) subscribed to that channel ^[9]. Each GS fans out to its locally connected clients. The CS-to-GS hop is the aggregation point: one cross-region message per GS, not per client. Slack's 2024 Unified Grid re-architecture shards message storage by channel_id via Vitess, enabling org-wide views without workspace-scoped routing ^[10].

A single send reaches the guild once; the guild delegates to relays; each relay fans out to up to 15K local sessions after permission checks, collapsing O(members) to O(relays) at the guild.

Passive sessions. 90% of user-guild connections in large servers have no active tab ^[1:5]. Discord skips full message delivery for passive sessions, sending only a counter increment. This reduces effective fanout by ~90% for those connections.

RBAC with versioned cache invalidation#

Every message delivery requires a permission check: "Can this user read this channel?" In a guild with 50 roles and 500 channels, the permission matrix is 25,000 cells. Evaluating from scratch on every message is too expensive.

Effective permission bitmask. Discord's guild process computes the effective permission for each (user_id, channel_id) pair by combining the user's role bitmasks with per-channel overrides (explicit allow/deny bits that win over role defaults) ^[4:1]. The result is cached keyed on (user_id, channel_id, role_version).

Versioned invalidation. When an admin changes a role, the guild bumps role_version for the workspace. All cached permission entries with the old version are stale on next access. This is O(1) to invalidate (one version bump) rather than O(users x channels) writes ^[4:2].

Every delivery traverses the cached permission lookup; role changes bump a single version counter, lazily invalidating all stale entries without O(N) writes.

Hot-path cost. Discord's guild process checks permissions on every message delivery ^[4:3]. The CPU cost scales with (online members) x (messages/sec) x (average roles per user). Caching reduces this to a hash lookup in the common case; only role changes trigger recomputation.

Presence aggregation (the N-squared problem)#

Naive presence: a 10K-member channel with 1K status changes/sec = 10M presence events/sec delivered. This is quadratic in membership.

Subscription scoping (Slack). Clients subscribe to presence only for users currently visible in the UI, not the entire workspace ^[5:3]. This cut presence events delivered to clients by 5x. Slack's Flannel edge cache holds per-client filtered presence state and converts server-side pub/sub into per-client streams ^[5:4].

Batching and debouncing. Presence deltas are batched in 5-second windows. Multiple status changes within the window collapse to the latest value. Users tolerate 5-10 seconds of staleness; strict consistency is unaffordable at this scale.

Monday 9 AM storms. 500K users logging in within 60 seconds generates millions of subscribe events. Mitigations: reconnect jitter (spread over 30 s), server-side rate limiting on presence subscriptions, and degradation to "eventually correct within 30 s" SLO ^[5:5][9:1].

Workspace search with ACL filtering#

Search must return only messages from channels the caller can read. Post-filtering (score all messages, then remove forbidden ones) leaks hit counts and breaks pagination semantics.

Pre-filter approach. Discord injects the caller's permitted channel_id set as an Elasticsearch term filter before scoring ^[6:3]. The query only touches documents the user is authorized to see. Slack applies the same principle with workspace-scoped filters ^[11][12].

Cell architecture. Discord shards search across 40 Elasticsearch clusters organized into cells ^[6:4]. Each cell is small (3 master-eligible + 3 ingest + data nodes). Benefits: rolling upgrades are safe (the Log4Shell incident forced the old monolithic cluster fully offline ^[6:5]), and a noisy guild cannot stall others. Slack adopted a similar cellular approach for its core services, bounding failure blast radius to a single cell ^[13].

BFG indices. "Big Freaking Guilds" exceed Lucene's ~2B document limit per index ^[6:6]. Discord uses multi-shard indices for these outliers with a dual-write, historical backfill, atomic query-switch, and cleanup reindex flow.

Real-World Example#

Discord: 5 engineers, 400 machines, trillions of messages.

Discord's chat infrastructure runs on roughly 400-500 Elixir/Erlang machines (as of 2020) managed by a team of 5 engineers ^[7:1]. Each guild is a single Elixir GenServer that owns the full channel, role, voice-state, and member-presence view ^[4:4][1:6]. The largest single guild (Midjourney) reached 1M+ concurrent online users in 2023 ^[1:7].

The storage layer migrated from 177-node Cassandra to 72-node ScyllaDB in 2022 ^[2:2]. The motivation: Cassandra's JVM garbage collector caused unpredictable latency spikes (p99 read: 40-125 ms). ScyllaDB's shard-per-core C++ architecture cut p99 reads from 40-125 ms to 15 ms and stabilized write p99 at a steady 5 ms (vs Cassandra's variable 5-70 ms) ^[2:3]. A Rust data-services layer sits between the API and ScyllaDB, coalescing concurrent reads of the same channel row into a single database query using consistent-hash routing keyed on channel_id ^[2:4].

Gateway bandwidth optimization came in 2024: streaming zstandard compression replaced zlib, reducing WebSocket traffic by ~40% combined with Passive Sessions V2 ^[14]. The member list uses a Rust NIF SortedSet: the pure Elixir OrderedSet's insertion worst-case was 640 microseconds at 250K items, while the Rust NIF SortedSet sustained a 3.68 microsecond worst-case at 1M items, unlocking guild sizes beyond the prior 250K ceiling ^[15].

The search plane indexes trillions of messages across 40 Elasticsearch clusters ^[6:7]. Median query latency improved from 500 ms to under 100 ms after the cell re-architecture. The indexing pipeline migrated from Redis (which dropped messages under CPU pressure) to Google PubSub for guaranteed delivery ^[6:8].

Trade-offs#

The single biggest meta-decision is fanout strategy. Discord chose in-process relays (Erlang processes co-located with the guild). Slack chose cross-region gateway aggregation (Channel Server to Gateway Server). Both achieve the same goal: collapsing O(members) at the coordination layer to O(nodes) or O(regions). The trade-off is operational: Discord's model keeps state in one process (simpler reasoning, harder horizontal scaling); Slack's model distributes state across consistent-hashed servers (more moving parts, easier to scale independently).

Scaling and Failure Modes#

At 10x (3M concurrent per guild): The guild process becomes the bottleneck. Mitigation: shard the guild into sub-guild processes per channel category, each owning a subset of channels and relays. The coordination layer becomes a tree rather than a single root.

At 100x (30M concurrent, 10B messages/day per workspace): ScyllaDB hot partitions on viral channels spike tail latency. Mitigation: request coalescing at the data-services layer (already deployed at Discord ^[2:6]), plus adaptive bucket sizing that splits hot channels into finer time windows.

At 1000x: The architecture shifts to a CDN-edge model where gateway servers at PoPs hold channel subscriptions and receive push from a global event mesh, eliminating cross-region hops for read-heavy channels.

Failure: Guild process crash. The guild GenServer restarts on a new node. Clients experience a 2-3 second gap in message delivery while state rebuilds from the database. No messages are lost (persisted before fanout) ^[7:2].

Failure: Elasticsearch cell down. Search degrades for the affected guilds. Messages continue to queue in PubSub (guaranteed delivery). Once the cell recovers, the backlog drains. No index gaps because PubSub does not drop under pressure ^[6:10].

Failure: ScyllaDB node failure. Quorum reads/writes continue on remaining replicas. The failed node's data is re-replicated automatically. Read p99 may spike briefly (15 ms to ~50 ms) until rebalancing completes ^[2:7].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Relay sizing#

A guild has 200,000 concurrent online members. Each relay process handles up to 15,000 sessions. A hot channel receives 20 messages/sec. Calculate: (a) how many relay processes are needed, (b) the total deliveries/sec across all relays, and (c) what happens if one relay crashes.

Key Takeaways#

• Channels are not large group DMs. Pub/sub fanout, hot-path ACL, and N-squared presence are qualitative architectural shifts, not just scale increases.
• Aggregate, do not unicast. One cross-network message per node or region (Manifold, CS-to-GS) is the only pattern that survives past 1K concurrent members ^[3:4].
• Cache permissions with versioned invalidation. O(1) version bump on role change beats O(users x channels) cache writes ^[4:6].
• Presence is eventually consistent by design. Users tolerate 5-10 seconds of staleness; strict consistency is unaffordable at workspace scale ^[5:7].
• Partition messages by (channel_id, time_bucket). Keeps history co-located for fast scrollback while bounding partition size ^[2:10].
• Small search cells beat monolithic clusters. Independent lifecycle, safe upgrades, and tenant isolation ^[6:13].

Further Reading#

• How Discord Stores Trillions of Messages. The canonical Cassandra-to-ScyllaDB migration post with p99 latency numbers and the Rust data-services coalescing pattern.
• Maxjourney: Pushing Discord's Limits with a Million+ Online Users. Relay architecture, passive sessions, and the engineering behind Discord's largest guild.
• Flannel: An Application-Level Edge Cache to Make Slack Scale. Edge caching and pub/sub presence that delivers 5x presence-traffic reduction.
• Real-time Messaging (Slack Engineering). Channel Server, Gateway Server, and CHARM consistent-hash architecture.
• How Discord Indexes Trillions of Messages. Elasticsearch cell architecture, BFG indices, and PubSub ingest pipeline.
• How Discord Reduced WebSocket Traffic by 40%. Streaming zstandard compression and Passive Sessions V2.
• Unified Grid: How We Re-Architected Slack. The channel_id sharding shift enabling org-wide views.
• Matrix State Resolution v2. Federated room-state convergence without a coordinator; relevant for cross-workspace federation.

Flashcards#

References#