Message Queues and Streaming: Decoupling at Scale

TL;DR: Message systems split into two models, not two products. A queue delivers each message to one consumer and deletes it after acknowledgement. A log (or stream) is an append-only, ordered, durable record that many consumer groups read independently at their own pace. Kafka, Kinesis and Pulsar are log-first; SQS is queue-only; RabbitMQ ships both, since streams give it "non-destructive consumer semantics" alongside its classic and quorum queues^[1]. Jay Kreps's 2013 essay reframed the log as the source of truth from which every downstream system derives its state^[2]. The practical delivery guarantee is at-least-once with idempotent consumers; exactly-once holds "when reading, processing and writing data on Kafka topics," and reaching other systems "generally requires cooperation with such systems"^[3]. As of October 2019, LinkedIn processed over 7 trillion messages per day across 4,000+ brokers^[4], and Netflix processes hundreds of billions of events daily through its Keystone pipeline. The partition is the unit of parallelism, the boundary of ordering, and the cap on consumer-group scaling (though not on share groups).

Learning Objectives#

After this module, you will be able to:

• Distinguish a queue from a log and pick the right one for a workload
• Design for at-least-once delivery with idempotent consumers
• Reason about Kafka partitions, consumer groups, and ordering guarantees
• Handle backpressure, dead-letter queues, and poison messages
• Compare Kafka, RabbitMQ, SQS, Kinesis, and Pulsar on real criteria
• Explain why partition count is a permanent architectural decision

Intuition#

Think of two services at the post office.

The first is the package counter. You hand a parcel to a clerk. The clerk gives it to exactly one delivery driver. Once delivered, the receipt is shredded. If you want to send the same parcel to two people, you need two parcels. This is a queue: one message, one consumer, gone after processing.

The second is the newspaper archive. Every edition is printed, numbered, and shelved in order. Any subscriber can walk in, find edition #4,217, and read forward from there. A new subscriber does not need to wait for tomorrow's paper; they can start from any past edition. Adding a subscriber does not slow down existing readers. This is a log: append-only, ordered, replayable, multi-subscriber.

The queue is simpler. The log is more powerful. Most confusion in system design comes from reaching for a queue when you need a log, or paying the operational cost of a log when a queue would suffice.

The rest of this chapter teaches you to tell the difference and pick correctly.

Theory#

Queue vs stream: the fundamental distinction#

A queue (SQS, a RabbitMQ classic or quorum queue) tracks per-message state: visible, in-flight, acknowledged. Competing consumers drain work faster than one could alone. Once acknowledged, the message is gone. If you add a new consumer service next month, it starts from "now" with no history^[5].

A log (Kafka, Kinesis, Pulsar, RabbitMQ streams) stores durable, offset-addressable segments. Each consumer group tracks its own offset. Adding a new consumer group replays history from offset 0 without disturbing existing consumers. The log is not just a transport mechanism; it is the source of truth that downstream systems (search indexes, caches, warehouses, microservices) derive their state from^[2:1].

Pick the model first, then the product. Several brokers implement both, so "does this product support replay?" is the wrong question; "which queue type am I declaring?" is the right one.

A pragmatic decision tree: start with replay and fan-out needs, then narrow by routing complexity and operational appetite.

Use a log when you need replay, multi-subscriber fan-out, or CDC. Use a queue when you need simple work distribution with per-message acks and no history.

Delivery guarantees: at-most, at-least, exactly-once#

Three levels exist, and only one is honest at scale:

• At-most-once (Kafka acks=0, fire-and-forget): the producer sends and moves on. Messages can be lost on broker crashes.
• At-least-once (the practical default): the producer retries until it gets an ack, and the consumer may see the same record twice. Be precise about where the duplicate comes from. enable.idempotence defaults to true, so the broker dedupes producer retries by PID and sequence number^[6]; producer retries are not your duplicate source on a modern client. The duplicate comes from the consumer side: a consumer that processes records and then crashes before committing its offset will reprocess them after rebalance, which Kafka's own docs describe as the at-least-once case^[3:1]. Consumers must therefore be idempotent. Netflix's Keystone pipeline uses acks=1 (leader-only acknowledgement), a pragmatic middle ground that accepts a small data loss risk on leader failure in exchange for lower latency and higher availability^[7].
• Exactly-once (Kafka EOS): requires enable.idempotence=true (producer sequence numbers deduped broker-side), a transactional.id (atomic writes across partitions plus consumer offset commit in the same transaction), and consumers using isolation.level=read_committed^[8]. The often-quoted "3% overhead" needs its baseline: Confluent measured 1 KB messages and 100 ms transactions declining "only by 3%, compared to the throughput of a producer configured for at least once, in-order delivery (acks=all, max.in.flight.requests.per.connection=1)", and "by 20% compared to ... at most once delivery with no ordering guarantees (acks=1, max.in.flight.requests.per.connection=5)", which was the default at the time^[8:1]. Cite the pairing, not the 3% alone.

Kafka architecture: topics, partitions, ISR, KRaft#

A Kafka topic is split into N partitions. Each partition is a replicated log with one leader and several followers. The ISR (in-sync replicas) is the subset of replicas that have caught up to the leader.

Key invariants:

• acks=all + min.insync.replicas=2 on replication factor 3 guarantees no data loss under single-broker failure^[6:1]. It also costs availability, which the durability framing hides: if acks=all and the ISR falls below min.insync.replicas, "the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend)"^[9]. On RF=3 with min.insync.replicas=3, losing one broker stops writes to that partition entirely. Raising the setting buys durability by spending availability; 2-of-3 is the usual compromise for that reason.
• Producer idempotence (default since Kafka 3.0) assigns each producer a PID and each batch a sequence number; brokers dedupe retries with max.in.flight.requests.per.connection <= 5^[6:2].
• Partition throughput: 10-50 MB/s per partition in production^[10].
• Adding partitions is not a free scale-out. Kafka's operations docs are explicit: "adding partitions doesn't change the partitioning of existing data ... if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way"^[9:1]. A key's future records land on a new partition while its history stays put, so the per-key ordering guarantee is broken across the change and two consumers can process one key's stream concurrently. Over-provision at creation, or plan a topic migration rather than an --alter.

Producers write keyed events into partitioned topics; two consumer groups read independently at their own offsets, and poison messages route to a DLQ after N retries.

KRaft (KIP-500) replaced ZooKeeper with an internal Raft-based metadata quorum. The rollout took years: KIP-833 marked KRaft "production ready for new clusters only" in Kafka 3.3 (announced 3 October 2022), 3.5 was the bridge release that allowed ZooKeeper-to-KRaft migration and deprecated ZooKeeper, and Kafka 4.0 (18 March 2025) is "the first major release to operate entirely without Apache ZooKeeper"^[11][12][13]. This eliminates an entire system from the operational footprint and enables faster controller failover.

Tiered storage (KIP-405, GA in Kafka 3.9) offloads older log segments to object storage (S3, GCS), significantly reducing storage cost for long-retention topics while keeping recent data on local disks for hot reads^[14][15].

RabbitMQ and AMQP: exchanges, flexibility, routing#

RabbitMQ implements AMQP 0-9-1. Producers publish to an exchange; bindings attach queues to exchanges with a routing key pattern^[16]:

• Direct exchange: routes to queues whose binding key exactly matches the routing key.
• Fanout exchange: ignores routing keys, delivers to every bound queue.
• Topic exchange: uses dotted routing keys (orders.eu.paid) with * (one word) and # (zero or more words) wildcards.

A topic exchange routes by wildcard pattern; one message fans out to multiple queues without duplication logic in the producer.

Modern RabbitMQ (3.8+) provides quorum queues built on Raft, replacing the deprecated mirrored-queue feature with clear failure semantics^[17]. Throughput per queue (tens of thousands msgs/sec) is an order of magnitude below Kafka's per-partition figures^[18].

What RabbitMQ does not lack is replay. Since 3.9 it also ships streams, which the docs describe as "an append-only log of messages that can be repeatedly read until they expire," with explicitly "non-destructive consumer semantics": consumers attach at any point in the log, read the same messages as many times as they want, and track their own offset^[1:1]. Streams were built for large fan-outs, time-travel replay, high throughput, and large backlogs, the four things classic queues handle badly^[1:2].

The honest framing is therefore queue versus stream as models, not RabbitMQ versus Kafka as products. A classic or quorum queue has destructive consume: once acknowledged, the message is gone and a consumer added next month starts from now. A stream does not. Both models are available in RabbitMQ, and streams are declared per queue with x-queue-type: stream^[1:3].

Use RabbitMQ when you need rich routing (topic/fanout/headers), low-latency task distribution, or RPC-style request/reply patterns. Use Kafka when you need a durable, replayable event log.

SQS, SNS, Kinesis, Pulsar, and Redpanda#

SQS uses a visibility timeout (default 30s): after ReceiveMessage, the message is invisible to other consumers. If the consumer crashes before DeleteMessage, the message reappears. A redrive policy moves messages to a DLQ after maxReceiveCount failed receives^[21].

For fan-out in AWS, combine SNS (pub/sub) with SQS (per-subscriber queue): SNS delivers to N SQS queues, each consumed independently. This gives you log-like multi-subscriber semantics without running Kafka.

Consumer patterns: groups, competing consumers, DLQ, backpressure#

Consumer groups distribute partitions across members, and one partition goes to exactly one member. A consumer group therefore cannot have more active consumers than partitions, which is the classic ceiling on parallelism^[10:1]. That ceiling is now conditional on the group type. Share groups (KIP-932, "Queues for Kafka") went production-ready in Kafka 4.2, and there "partitions may be assigned to multiple consumers" and "the number of consumers in a share group can exceed the number of partitions in a topic"^[3:3][22]. Records are acquired under a time-limited lock (share.record.lock.duration.ms, 30 s by default) and acknowledged individually, so you trade per-partition ordering for queue-like parallelism. If your work is genuinely order-independent, share groups remove the partition-count ceiling; if you need per-key ordering, the ceiling still applies.

Rebalancing has moved on twice. KIP-429 cooperative rebalancing (Kafka 2.4+) narrowed the older "stop-the-world" protocol so only partitions that need to move are revoked^[23][24]. The current answer is KIP-848, the next-generation consumer rebalance protocol, Generally Available since Kafka 4.0. It is "fully incremental" and "no longer relies on a global synchronization barrier," heartbeat interval and session timeout are now server configs, and the assignor itself moved server-side (group.consumer.assignors, defaulting to uniform and range). Groups on the new protocol are called Consumer groups; the old ones are Classic. On the client you must opt in with group.protocol=consumer^[13:1][25].

Backpressure is managed via bounded in-flight windows:

• Kafka: max.poll.records and fetch.max.bytes
• SQS: approximately 120,000 in-flight messages per standard queue, "depending on queue traffic and message backlog"; past it, short polling returns OverLimit^[26]
• RabbitMQ: prefetch (QoS) count

Dead Letter Queue (DLQ): after N retries with exponential backoff, produce the failing record to a <topic>.DLQ, commit the offset, and move on. This is the Dead Letter Channel pattern from Hohpe and Woolf's Enterprise Integration Patterns^[27]. Alert on DLQ depth. Build a redrive tool to re-inject fixed messages.

Lag monitoring: the difference between log-end offset and committed offset. LinkedIn's Burrow is the canonical consumer-lag monitor^[28].

The producer retry creates a duplicate at the broker; the idempotent producer dedupes by PID+sequence, and the consumer deduplicates using a persisted idempotency key before committing the offset.

Real-World Example#

LinkedIn Kafka: 7 trillion messages per day (as of October 2019).

LinkedIn is where Kafka was born, and it remains the largest deployment with published numbers. Every figure below is from a single post dated 8 October 2019, and LinkedIn has not refreshed them since: over 100 Kafka clusters with 4,000+ brokers, 100,000+ topics, 7 million partitions, and more than 7 trillion messages per day^[4:1]. Treat them as a 2019 snapshot, not a current reading. The post does not name a largest cluster; it says "some larger clusters have more than 140 brokers and host one million replicas in a single cluster"^[4:2].

Every LinkedIn service runs a Kafka client (or a REST proxy for non-JVM languages). Events flow to regional Kafka clusters and are mirrored cross-region by Brooklin, which replaced Kafka MirrorMaker in 2018; as of 11 April 2022, LinkedIn reported mirroring "more than seven trillion messages per day between the clusters using Brooklin"^[29]. Downstream they are consumed by Samza stream processors, Hadoop, Pinot, and hundreds of microservices.

Key engineering decisions that kept this running:

• Maintenance mode brokers: brokers flagged for decommission stop receiving new partition assignments, letting SREs safely drain hardware without triggering rebalance storms^[4:3].
• Controller memory optimization: reusing UpdateMetadataRequest objects to prevent cascading controller failures in clusters with millions of replicas^[4:4].
• Cruise Control: automated partition rebalancing and self-healing across clusters.

The failures are instructive. LinkedIn's KIPs document production issues: KIP-291 (separating controller from data-plane connections to avoid head-of-line blocking), KIP-354 (maximum log compaction lag, after a compaction stall filled disks), and KIP-380 (detecting outdated control requests after broker bounces)^[4:5].

The lesson: at that volume, every configuration choice (partition count, acks setting, retention policy, rebalance strategy) is the difference between a healthy pipeline and a 3am page.

Trade-offs#

Decision rule: Use Kafka for partitioned ordered streams with replay. Use SQS when you do not need ordering or replay and want zero ops. Use RabbitMQ when you need flexible routing patterns. Use Kinesis or Pulsar when your constraints (AWS-native or multi-tenant) demand them.

Common Pitfalls#

Exercise#

Design Challenge: You are building the event backbone for a fintech platform. Requirements: 100 million events per day, 3 consumer types (fraud detection with sub-second latency, billing with exactly-once semantics, analytics warehouse with hourly batch loads), and strict per-user ordering of financial events.

Key Takeaways#

• A queue delivers each message to one consumer and deletes it. A log is an append-only, replayable record that many consumer groups read independently. These are models, not products: RabbitMQ ships both.
• At-least-once with idempotent consumers is the honest, practical delivery guarantee. Kafka's exactly-once holds for reading, processing and writing on Kafka topics; other destinations need cooperation (offset stored with the output) rather than being impossible.
• The partition is the unit of parallelism and the boundary of ordering. A consumer group cannot have more active consumers than partitions; a share group (KIP-932, production-ready in Kafka 4.2) can, at the cost of per-partition ordering.
• Ordering is only meaningful per partition. Design your partition key for the ordering you need (typically entity ID).
• Partition count is nearly permanent. Increasing it later reshuffles hash(key) % N, and Kafka "will not attempt to automatically redistribute data in any way," so a key's history and its future live on different partitions and per-key ordering breaks across the change. Over-provision at creation.
• DLQs are not optional. Poison messages will happen, and without a DLQ they block the entire partition indefinitely.
• Use Kafka for ordered event streams with replay. Use SQS for zero-ops async jobs. Use RabbitMQ for rich routing and task distribution.

Further Reading#

• The Log: What every software engineer should know (Jay Kreps, 2013) - the foundational essay that reframes messaging as derived views over a durable log; read this before anything else in the space. LinkedIn has taken the original page down, so this is the archived copy.
• How LinkedIn customizes Apache Kafka for 7 trillion messages/day - the canonical large-scale operations story; covers controller tuning, maintenance mode, and Cruise Control.
• Exactly-Once Semantics Are Possible (Confluent, 2017) - authoritative walkthrough of idempotent producer + transactions; read the performance section carefully, since the 3% figure is measured against acks=all, max.in.flight=1 and the same sentence reports 20% against the then-default at-most-once config.
• Scaling Slack's Job Queue - the best "add Kafka incrementally without rewriting everything" story; shows Kafka as a durable buffer in front of Redis workers.
• Kafka Inside Keystone Pipeline (Netflix, 2016) - running Kafka in a cloud that treats brokers as cattle; covers the acks=1 trade-off and cold-standby failover.
• Designing Data-Intensive Applications, Ch. 11 (Kleppmann) - conceptual framing of messaging and stream processing; the textbook treatment of exactly-once semantics.
• Enterprise Integration Patterns (Hohpe and Woolf) - the DLQ, routing, and channel patterns everyone reinvents badly; read the Dead Letter Channel and Competing Consumers patterns.
• KIP-500: Replace ZooKeeper with Self-Managed Metadata Quorum - the design document for KRaft; explains why ZooKeeper was the operational bottleneck.

Flashcards#

References#