Design Live Comments at Scale (FB Live / YouTube Live / Twitch Chat)

TL;DR. Live comments invert the chat fan-out problem: one room with 10M subscribers instead of many rooms with a few. A viral stream draws 50K comments/sec that must reach millions of viewers within 2 seconds. The pivotal insight is delta-batched aggregation: collapse a 500 ms window of messages into one compressed WebSocket frame per subscriber shard, reducing 500 billion naive pushes/sec to a tractable 200 frames/sec per shard. Moderation runs as a three-stage cascade (Bloom filter, regex, ML classifier) that clears 98% of messages in under 5 ms. The celebrity-stream hotspot is contained by per-stream shard affinity and automatic slow-mode activation.

Learning Objectives#

• Design a one-to-many broadcast fan-out that delivers 50K comments/sec to 10M viewers within 2 seconds
• Implement delta-batched WebSocket push that reduces naive per-recipient fan-out by 100x
• Build a three-stage moderation cascade that stays under 500 ms p99 while filtering 50K msg/sec
• Justify ephemeral (Redis buffer) versus durable (Kafka archive) storage per product requirements
• Contain celebrity-stream hotspots with per-stream shard affinity and dynamic worker scaling
• Contrast this broadcast shape with the many-to-many fan-out in Design a Chat System

Intuition#

A live-comment system looks like a chat room. Accept messages, show them to everyone. Handles 100 viewers fine. At 10 million concurrent viewers on a single stream, it collapses, and the reason is fan-out arithmetic.

If 50,000 commenters each post once per second, that is 50K messages/sec arriving at one room. Naive per-recipient push means 50K messages times 10M viewers equals 500 billion pushes per second. No infrastructure survives that. The naive approach is not "a bit slow." It is physically impossible. Facebook Live's watermelon-exploding video peaked at 800,000 concurrent viewers and 300,000 comments on a single 45-minute clip ^[1]. TheGrefg's Fortnite skin reveal hit 2,468,668 concurrent viewers on one Twitch channel ^[2][3]. These are not hypothetical numbers.

The insight that unlocks the design: humans cannot read faster than about 60 messages per minute ^[4][5]. A viewer watching a fast-moving chat stream is already sub-sampling. The system can exploit this by batching: accumulate all messages in a 500 ms window, compress them into one frame, and push that single frame to each subscriber shard. Now 50K messages/sec becomes 2 frames/sec per viewer, and the fan-out fleet pushes 2 frames times 100 subscriber shards (each holding 100K connections) equals 200 pushes/sec per shard. Tractable.

This is fundamentally different from 1:1 chat. Design a Chat System covers many rooms with few recipients per room. Live comments is one room with millions of recipients. Chat demands per-message ordering and exactly-once semantics. Live comments accepts at-most-once delivery, loose ordering, and graceful degradation under overload. The techniques are inverted: chat optimizes for correctness, live comments optimizes for throughput.

Requirements#

Clarifying Questions#

• Q: Ephemeral or durable comments? Assume: Both modes. Twitch-style ephemeral (30-min TTL) and FB Live-style durable (permanent archive for replay and legal discovery).
• Q: Pre-publish or post-publish moderation? Assume: Pre-publish for brand-safety platforms (YouTube, FB Live); post-publish with hide for community platforms (Twitch).
• Q: Strict global ordering across all commenters? Assume: No. Per-commenter FIFO is sufficient. Loose global ordering within a batch window is acceptable.
• Q: Super chats / paid highlighted messages in scope? Assume: Yes. Paid messages bypass standard rate limits and get visual prominence.
• Q: Replay for mid-stream joiners? Assume: Yes. Last-N (configurable, default 200) served from cache on join.
• Q: Rate-limit modes (slow mode, subscriber-only, emote-only)? Assume: Yes. These are load-shedding levers the streamer controls.

Functional Requirements#

• Post a comment to a live stream; receive acknowledgment with moderation status
• Subscribe to a stream's comment feed via WebSocket; receive delta-batched frames
• Moderate: auto-filter (banned words, ML toxicity), manual mod actions (ban, timeout, delete)
• Rate-limit per user with configurable slow mode, subscriber-only, and emote-only modes ^[6]
• Support paid/highlighted messages (Super Chat, Bits) with tier-based pin duration ^[7]
• Replay last-N comments for mid-stream joiners

Non-Functional Requirements#

• Viewers: 10M concurrent on a single hot stream; 100M globally across all streams
• Commenters: 100K concurrent writers on a hot stream
• Write rate: 50K comments/sec peak on one stream; 1B events/day globally
• Visibility latency: p99 < 2 seconds from post to viewer screen
• Moderation latency: p99 < 500 ms for automated pipeline
• Availability: 99.9% during live events
• Delivery: at-most-once acceptable; drop rate < 5% under overload

Capacity Estimation#

Key ratios:

• Read:write amplification: 200:1 (each message reaches 100 shards as part of a batch frame)
• Moderation pass rate: ~98% clear automatically (stages 1-3 combined); only ~2% reach human review ^[8]
• Human review volume: ~1% of total (500 msg/sec on a hot stream)

API and Data Model#

API Design#

POST /v1/streams/{stream_id}/comments
  Body: { "content": "...", "emojis": [...], "reply_to": null }
  Returns: 201 { "comment_id": "<snowflake>", "status": "published" | "pending" }
  Rate-limited: per-user token bucket (default 20/30s) [^9]

WS /v1/streams/{stream_id}/subscribe
  Server push: delta-batched frames every 500ms
  Resume: { "last_frame_id": "<cursor>" }
  Backpressure: slow clients dropped after 3 missed frames

DELETE /v1/streams/{stream_id}/comments/{comment_id}
  Returns: 204 (mod action or self-delete)

POST /v1/streams/{stream_id}/mode
  Body: { "slow_seconds": 3, "subs_only": true, "emote_only": false }
  Returns: 200 (broadcasts ROOMSTATE to all subscribers) [^6]

GET /v1/streams/{stream_id}/replay?limit=200
  Returns: 200 { "comments": [...], "next_cursor": "..." }

Data Model#

-- Redis: live comment ring buffer (ephemeral mode)
-- Key: stream:{stream_id}:comments (sorted set, score = timestamp)
-- TTL: 30 min after stream ends
-- Max entries: 10,000 (ZREMRANGEBYRANK on overflow)

-- Kafka: comment.log (durable mode)
-- Topic: comments, partitioned by stream_id
-- Retention: 7 days hot, archive to S3 for permanent

-- Redis: rate limits
-- Key: rate:{stream_id}:{user_id} (token bucket counter)
-- Key: mode:{stream_id} (hash: slow_seconds, subs_only, emote_only)

-- Redis: banned words
-- Key: banned:{stream_id} (set of strings, synced to mod workers)

Comments belong to a stream and a user; the stream holds mode flags (slow, subs-only) and the current viewer count for shard-scaling decisions.

High-Level Architecture#

Commenters write to an ingress that rate-limits and pre-moderates; accepted messages flow through Kafka to a sharded fan-out fleet that delta-batches frames to 10M viewers over WebSocket.

Write path: A commenter posts via HTTP. The ingress service checks the per-user token bucket and stream mode flags (slow mode, subs-only). If admitted, the message enters the moderation cascade. Accepted messages are appended to the Kafka partition for that stream_id and simultaneously written to the Redis ring buffer for replay.

Read path: Each fan-out shard consumes its stream's Kafka partition, accumulates messages in a 500 ms window, compresses the batch into a single WebSocket frame, and pushes to all connections it owns. Slow clients that miss 3 consecutive frames are disconnected with a RECONNECT hint.

Replay path: A mid-stream joiner hits the replay endpoint, which serves the last 200 messages from the Redis sorted set. For hot streams, this response is edge-cached with a 2-second TTL to absorb thundering-herd joins.

Deep Dives#

Deep dive 1: Delta-batched fan-out#

The single most important lever in this system. Without batching, 50K messages/sec times 10M viewers equals 500 billion pushes/sec. With 500 ms batching, the math changes entirely.

Mechanism: Each fan-out shard maintains a per-stream accumulator. Every 500 ms, the shard:

1. Drains the accumulator (up to 25K messages in a hot window)
2. Serializes the batch as a compressed Protocol Buffer frame
3. Writes the frame once to a kernel send buffer
4. The kernel copies the frame to each of the shard's 100K TCP connections

The per-shard egress is one frame times 100K connections. At 500 B per compressed message and 25K messages per batch, the frame is ~12.5 MB uncompressed, ~2 MB with gzip. Per-shard egress: 2 MB times 2 frames/sec times 100K connections... still too high. The solution: client-side sub-sampling.

Sub-sampling: The frame contains all 25K messages, but the client renders only the most recent 30 (matching the ~60/min human reading ceiling ^[4:1]). The client picks which 30 to show based on recency, paid-message priority, and followed-user boost. This means the frame can be further trimmed server-side: include only the top 200 messages per batch (sorted by priority), reducing frame size to ~100 KB compressed.

Twitch's 2016 engineering blog describes the client-side version of this: buffering incoming TMI messages for 100 ms and rendering once per window, which halved render time and eliminated dropped video frames ^[9]. YouTube's pollingIntervalMillis achieves the same effect server-side: the server dictates how often clients may poll, returning batches of 200 to 2,000 messages ^[10][11].

A 500 ms accumulation window collapses 25K messages into one compressed frame; clients sub-sample to the human-readable ceiling of ~60 messages/min.

Deep dive 2: Three-stage moderation cascade#

At 50K messages/sec, you cannot run every message through a GPU-based toxicity classifier. The cascade exploits the fact that most messages are either benign or hit well-known banned-word patterns.

Stage 1: Banned-word hash lookup (~1 ms). A hash set of channel-specific banned words plus a global profanity list. In our design, catches ~5% of messages immediately. Twitch's msg_rejected_mandatory notice fires at this stage ^[6:1]. False-positive rate is near zero because the list is curated.

Stage 2: Regex and pattern matching (~5 ms). Catches evasion patterns (l33tspeak, Unicode homoglyphs, zero-width characters). Flags another ~3% of messages. Combined with stage 1, ~8% are blocked before any ML inference.

Stage 3: ML toxicity classifier (~50 ms). A lightweight transformer scores the remaining 92% of messages. Messages scoring above 0.7 are auto-hidden. Messages scoring 0.3 to 0.7 (roughly 2% of total) are routed to human moderators ^[8:1]. Messages below 0.3 pass immediately. These stage percentages are illustrative design targets; no platform publishes exact production figures.

Cheap filters reject 8% of messages in under 5 ms; only 2% of ambiguous content reaches human review, keeping p99 moderation latency under 500 ms.

GPU sizing: At 50K msg/sec with 92% reaching stage 3, the classifier processes 46K msg/sec. At 50 ms per inference and 30 RPS per GPU instance, you need ~1,500 classifier slots. Batching inference (8 messages per forward pass) brings this to ~190 GPU instances. The cascade's value: without stages 1-2, you would need 1,667 GPU instances instead of 190.

Deep dive 3: Celebrity hot-shard problem#

Stream popularity follows a power law. TheGrefg's Fortnite skin reveal drew 2,468,668 concurrent viewers on a single Twitch channel ^[12][13]. Facebook Live's watermelon video peaked at 800,000 viewers with 300,000 comments ^[14]. A single celebrity stream can concentrate 50-95% of global traffic on one Kafka partition.

Detection: A heat-detection service monitors per-partition lag and per-stream viewer count. When a stream crosses a threshold (e.g., 500K viewers or 10K writes/sec), it is flagged as "hot."

Isolation: Hot streams get dedicated fan-out worker pools. Cold streams share worker pools. This prevents a celebrity's traffic from starving neighboring streams on the same broker.

Automatic slow mode: When write rate exceeds a threshold (e.g., 20K msg/sec), the system auto-enables slow mode (3-second cooldown per user), reducing effective write rate by 60-80% ^[6:2]. Twitch's ROOMSTATE broadcasts the mode change to all clients instantly.

Passive-connection optimization: Discord discovered that ~90% of user-guild connections in large servers are "passive" (member but not actively viewing) ^[15]. Skipping fan-out to passive connections gave a 10x reduction in fan-out work. The same principle applies here: viewers who have the chat panel minimized or are in a background tab do not need real-time frames. A heartbeat-based activity signal lets the fan-out shard skip inactive connections.

Hot-stream detection routes celebrity partitions to dedicated workers and auto-enables slow mode; passive-connection skipping reduces fan-out by 90%.

Facebook Live's Cartographer system addresses the geographic dimension: it maps subnets to PoPs and uses cubic-spline load prediction (not linear, because load curves are nonlinear) to route viewers to the nearest PoP with spare capacity before saturation ^[14:1]. The measurement-to-decision window was tightened from 90 seconds to 3 seconds.

Real-World Example#

Twitch TMI: IRC at 2.4 million concurrent viewers.

Twitch's chat backend is TMI (Twitch Messaging Interface), exposed as a modified RFC 1459 IRC interface with IRCv3 message tags ^[6:3][16]. Clients connect over wss://irc-ws.chat.twitch.tv:443, authenticate with OAuth, and JOIN #channel. Each PRIVMSG carries IRCv3 tags: badges, emote positions, bit counts, and a tmi-sent-ts server-side UNIX timestamp used as the per-stream ordering key ^[6:4].

The channel is the shard unit. room-id in each tag is the partition key. When a room exceeds 1,000 users, Twitch suppresses JOIN/PART presence events entirely because the bookkeeping is unworkable at scale ^[6:5]. Rate limits are tiered: regular users get 20 messages per 30 seconds, the broadcaster/moderators/VIPs get 100, and verified bots get 7,500 ^[17].

During TheGrefg's January 2021 stream (2,468,668 concurrent viewers ^[12:1], a record later surpassed repeatedly by Ibai Llanos: 3.3M in June 2022, 3.4M in 2023, 3.8M in 2024, and 9.3M at La Velada del Ano 5 in July 2025), TMI stayed operational but chat rate limits and client-side rendering were stressed ^[18]. The previous individual-channel record had been Ninja and Drake's Fortnite stream at 628,000 concurrent viewers in March 2018, which TheGrefg first surpassed in December 2020 ^[19]. Twitch's 2016 engineering blog describes the rendering fix: buffering TMI messages for 100 ms and flushing once per window halved Ember render time from ~200 ms to ~100 ms per frame, eliminating dropped video frames caused by chat rendering ^[9:1].

The architectural insight: Twitch chose an existing well-understood text protocol (IRC) rather than inventing a binary one. Every language has an IRC library. The trade-off is verbosity (text headers per message), but delta batching at the rendering layer compensates. Twitch has since overlaid EventSub (a modern webhook/WebSocket subscription API) for new integrations and now recommends EventSub over IRC for all new chatbots; IRC remains functional but is considered legacy ^[16:1].

Bilibili's goim (open-source, same lineage as their danmaku pipeline) benchmarks at 35.9M message deliveries/sec (fan-out to 1M online connections at 40 broadcasts/sec) on a single Xeon server, with 4.39 Gbps egress ^[20]. The IPL cricket stream in 2023 reportedly peaked at 32M concurrent viewers ^[21], pushing the same broadcast shape to an even larger scale. These numbers establish the per-server ceiling for the broadcast problem.

Trade-offs#

The single biggest meta-decision is ephemeral versus durable. Twitch optimizes for immediacy and cost: comments are transient, like speech in a room. Facebook and YouTube optimize for permanence: comments are content, searchable, replayable, and legally discoverable. The 10x storage cost difference (500 GB/day ephemeral vs 5 TB/day with full metadata) is the price of durability.

Scaling and Failure Modes#

At 10x (100M viewers, single stream): Fan-out shards scale linearly (1,000 shards at 100K conns each). The Kafka partition for the hot stream becomes the bottleneck. Mitigation: split the single partition into N sub-partitions with a secondary routing layer; each fan-out shard consumes one sub-partition.

At 100x (1B concurrent globally): The WebSocket connection fleet needs 10,000+ servers. Mitigation: move to a CDN-edge model where edge PoPs hold connections and consume a replicated message stream, eliminating cross-region hops for fan-out.

At 1000x: The architecture shifts to multicast at the network layer (IP multicast within PoPs) with application-layer dedup at the edge.

Failure: Fan-out shard crash. Viewers on that shard miss frames until reconnect (2-5 seconds with jitter). At-most-once semantics mean no replay of missed frames. Acceptable: users see a brief gap in chat, then resume.

Failure: Kafka partition leader failure. ISR elects a new leader in seconds. Fan-out shards resume from committed offset. Comments posted during the leader election are buffered at the ingress (backpressure to commenters via HTTP 503 with Retry-After).

Failure: WebSocket reconnect storm. A backend blip disconnects 10M clients simultaneously. Without jitter, all reconnect in a 1-second window, saturating the auth service. Mitigation: exponential backoff with random jitter (Twitch's RECONNECT command assumes clients implement this ^[6:8]); per-region admission control caps new-connection rate at 100K/sec.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Batch window sizing#

A stream has 2M concurrent viewers and 10K comments/sec. Your fan-out shards each hold 50K WebSocket connections. The network team says each shard can sustain 500 MB/s egress. What is the maximum batch window you can use before egress becomes the bottleneck? Assume each compressed batch frame is 50 KB.

Key Takeaways#

• Live comments are broadcast, not chat. One room times 10M subscribers demands topic-level fan-out, not per-recipient delivery.
• Batch or die. Delta-batched aggregation every 500 ms is the difference between 500G impossible pushes/sec and 200 tractable frames/sec per shard.
• Moderation is a cascade. Cheap filters first (hash, regex) clear 92% in under 5 ms; GPUs handle only the ambiguous remainder.
• Celebrity streams are the interesting case. Per-stream shard affinity, auto slow-mode, and passive-connection skipping are the load-shedding levers.
• Ephemeral versus durable is a product decision. Twitch and FB Live land on opposite sides for defensible reasons; the 10x cost delta is real.
• At-most-once is acceptable. Humans sub-sample fast chat anyway; dropped messages on a 50K msg/sec stream are invisible to viewers.

Further Reading#

• How Facebook Live Streams to 800,000 Simultaneous Viewers. The canonical write-up on request coalescing, cubic-spline load prediction, and PoP capacity estimation for broadcast fan-out.
• Improving Chat Rendering Performance (Twitch Engineering, 2016). The batching rationale that inspired delta-batched fan-out at the render layer; concrete before/after frame-time measurements.
• Maxjourney: Pushing Discord's Limits with a Million+ Online Users in a Single Server. The deepest public look at the celebrity-shard problem; the 90% passive-connection finding changes how you think about fan-out sizing.
• YouTube liveChatMessages API Reference. Every paid message type, tombstones, moderation events, and the pollingIntervalMillis server-dictated cadence.
• YouTube liveChatMessages.streamList (gRPC). The push alternative to polling; nextPageToken resume semantics for reconnection without replay.
• Terry-Mao/goim on GitHub. Open-source IM with the same shape as Bilibili's danmaku pipeline; the benchmark README establishes per-server connection and throughput ceilings.
• Twitch IRC Concepts (TMI Reference). The canonical reference for the wire format: tags, modes, notices, rate limits, and the full ROOMSTATE surface.

Flashcards#

References#