Load Balancers: Spreading Traffic, Absorbing Failure

TL;DR: A load balancer accepts a virtual IP, picks a backend, and quietly removes the ones that stop responding. Layer 4 forwards packets by 5-tuple at line rate: Google Maglev saturates a 10 Gbps link on a single machine^[1]. Layer 7 terminates TCP, parses HTTP, and routes on headers, at the cost of CPU per request. Most production stacks run both: a stateless L4 tier absorbs raw packet volume, and an L7 tier adds routing, retries, and observability^[2].

Learning Objectives#

After this module, you will be able to:

• Choose between L4 and L7 load balancing for a given workload
• Explain the trade-offs of round robin, least-connections, consistent-hash, and power-of-two-choices algorithms
• Design health checks that catch real failures without flapping
• Reason about sticky sessions, connection draining, and graceful deploys
• Describe how anycast and DNS-based GSLB route traffic globally
• Compare HAProxy, NGINX, Envoy, and cloud load balancers on features, throughput, and cost

Intuition#

Imagine you arrive at an airport terminal with 20 boarding gates. A single gate agent checks every passenger's ticket, assigns them a gate, and redirects anyone whose gate is closed. Without her, passengers would crowd gate 1 while gates 2 through 20 sit empty.

She does three jobs. First, she distributes passengers across gates so no single gate is overwhelmed. Second, she monitors gate status: if a gate's jetbridge breaks, she stops sending people there. Third, she handles the paperwork: scanning boarding passes, verifying IDs, and handing back a simplified stub so the gate agent downstream does less work.

A load balancer does the same three jobs for network traffic: distribution, health tracking, and termination (TLS, protocol parsing). The simplest version just reads the destination on your ticket (the 5-tuple) and points you to a gate. A smarter version reads your frequent-flyer status, your connection time, and your luggage weight before deciding. The simple version is faster. The smart version makes better decisions. Production airports use both: a fast first pass at the terminal entrance, then a smarter second pass at the gate cluster.

That is the L4/L7 split in one analogy. The rest of this chapter makes it concrete.

Theory#

Layer 4 vs layer 7#

An L4 load balancer operates on TCP/UDP packets. It hashes the 5-tuple (source IP, source port, destination IP, destination port, protocol) into a backend slot and forwards the packet without ever reading the payload. Responses typically bypass the LB entirely via Direct Server Return (DSR), so the LB only touches ingress traffic^[1:1][3].

An L7 load balancer terminates the TCP connection, decrypts TLS, parses HTTP (or gRPC, WebSocket, HTTP/2), and opens a new connection to the chosen backend. This lets it route by path, header, cookie, or method, retry failed requests, inject observability headers, and enforce per-tenant rate limits^[4][5].

The cost is CPU. Every request parsed is a request the LB must buffer, decrypt, and re-serialize. L7 proxies top out an order of magnitude lower on requests per second than L4 forwarders on the same hardware.

L4 forwards packets without seeing HTTP and uses DSR for replies; L7 terminates the connection and proxies both directions.

Use L4 when you need raw throughput, protocol-agnostic forwarding, or a DDoS absorption tier. Use L7 when you need path-based routing, retries, auth, or per-request metrics. Most production stacks use both: Cloudflare runs Unimog (L4, XDP) in front of Pingora (L7)^[2:1][6]; AWS customers put NLB in front of ALB or Envoy^[7].

Algorithms#

Round robin walks an ordered list. It is dead simple and distributes evenly across homogeneous hosts. It ignores actual load: a stalled host keeps receiving traffic until a health check removes it^[8].

Weighted round robin repeats higher-weight hosts proportionally. Use it when backends have different capacities (mixed-generation hardware).

Least-connections / least-outstanding-requests picks the host with the fewest in-flight requests. The naive version herds: when many LB instances simultaneously pick the same "least loaded" backend, that backend spikes.

Power of two choices (P2C) fixes the herding problem. Pick two backends at random; send the request to whichever has fewer outstanding requests. Mitzenmacher's 2001 result proves that d=2 choices gives an exponential improvement over d=1 (random), while d=3 gives only a constant factor further improvement^[9]. This is why Envoy, NGINX (via its random two directive), Netflix Zuul, and Finagle all offer P2C as a load balancing option^[4:1][8:1].

Consistent hashing routes requests by hashing a stable key (user ID, cache key, 5-tuple) into a server slot. When one of N backends is removed, roughly 1/N of keys move. Google's Maglev hash fills a 65,537-slot lookup table via a permutation algorithm; packet-path selection is a single modulo and array lookup^[1:2][10].

A Maglev-style lookup table maps 65,537 slots to backends by permutation; the 5-tuple hashes into a slot in O(1).

Use round robin for homogeneous stateless services. Use P2C least-request as the default for heterogeneous or latency-sensitive workloads. Use consistent hash only when you need cache affinity or flow pinning, and accept that hot keys can overload one backend^[11].

Health checks and outlier detection#

Active health checks probe backends on a schedule: a TCP connect, an HTTP GET to /healthz, or a gRPC health RPC. They find dead backends fast even when traffic is low^[12].

Passive (outlier) health checks observe real request traffic. Envoy's default outlier detector ejects a host after 5 consecutive 5xx responses for a base ejection time of 30 seconds, with linear backoff on re-ejection^[12:1][13]. Passive checks catch gray failures (slow responses, partial errors) that active checks miss.

Connection draining stops sending new connections to a backend while existing requests finish. In Kubernetes, the preStop hook sleeps for the LB's deregistration delay before the container receives SIGTERM^[14][15].

Slow start ramps traffic to a newly launched backend over a configurable window. Netflix uses 90 seconds to avoid overloading a cold JVM^[8:2].

A backend transitions through probation, healthy, unhealthy, and draining states; the LB only routes new traffic to healthy backends.

Session persistence#

Sticky sessions bind a client to a specific backend via a cookie, header, or source-IP hash. AWS ALB uses an AWSALB cookie^[16]; NLB pins each TCP connection via a 6-tuple flow hash that adds the TCP sequence number, so separate connections from one client can land on different targets^[17].

Global load balancing#

Above a single datacenter, traffic is steered globally by two mechanisms:

BGP anycast advertises the same VIP from every POP. Internet routers converge on the nearest location for each client. Cloudflare operates in 330+ cities with 500 Tbps of external capacity^[18]. When a POP goes dark, it withdraws its BGP route and traffic reconverges in seconds, with no DNS TTL delay^[2:2].

DNS-based GSLB (Route 53, NS1) returns different A records based on client subnet, RTT, or weighted policy. It is coarser: stale caches at ISPs can pin users to dead regions for minutes. Only new DNS lookups are affected; existing TCP connections stay put^[19].

A single anycast VIP reaches the nearest of many POPs; inside each POP, a stateless L4 LB fans out to backends.

Use anycast when you need sub-second failover and can operate BGP. Use DNS GSLB when you need geographic steering without BGP infrastructure, and accept minutes-long failover on stale caches.

TLS termination#

Three patterns dominate:

1. Termination at the LB. The LB decrypts, inspects HTTP, and re-originates to the backend (cleartext or re-encrypted). This is the default for AWS ALB, Cloudflare's L7 tier, and Envoy ingress. It enables header routing, WAF, and retries, but puts all private keys on the LB^[20].
2. Passthrough. The LB forwards encrypted bytes untouched (L4 or SNI-based routing). AWS NLB TLS listener and Istio PASSTHROUGH mode fit here. No HTTP routing, but end-to-end encryption is preserved^[21].
3. mTLS in a service mesh. Istio and Linkerd terminate outer TLS at the sidecar and re-originate mTLS to the upstream sidecar. Automatic cert rotation, zero-trust between services^[21:1].

Use termination when you need L7 features. Use passthrough when compliance requires end-to-end encryption. Use mTLS when you need authenticated service-to-service communication.

Real-World Example#

Cloudflare Unimog: every server is a load balancer#

Cloudflare's Unimog is an L4 load balancer deployed across their entire edge network: 330+ cities, 500 Tbps of capacity^[18:1]. Its defining design choice is that every server in a datacenter is simultaneously a load balancer and an application server. There is no separate LB tier.

How it works. Routers ECMP packets to any server. An XDP program chain runs per packet: l4drop (DDoS mitigation) first, then Unimog. Unimog hashes the 5-tuple, looks up a forwarding table with tens of thousands of buckets (roughly 100x the server count), and GUE-encapsulates the packet to the target server^[2:3].

Connection persistence. When the forwarding table changes (server added or removed), Unimog uses "daisy chaining" borrowed from the Beamer paper: each bucket has two slots (current and previous). If the current server has no matching TCP socket, it forwards to the previous server. Less than 1% of packets take this "second hop"^[2:4].

CPU overhead. Less than 1% of each server's CPU is spent on load-balancing work. XDP runs before the Linux kernel network stack touches the packet, so the cost is near zero^[2:5].

The oscillation incident. Early on, Cloudflare hit a feedback-loop problem: when a datacenter became overloaded and the control plane (conductor) diverted new connections to less-loaded servers, those servers then degraded while the original recovered, causing oscillation. The fix: teach the conductor to distinguish individual-server degradation from datacenter-wide saturation^[2:6].

DDoS absorption. In 2025, Cloudflare mitigated a 31.4 Tbps DDoS attack through the l4drop/Unimog chain with no human intervention^[18:2].

The lesson: a stateless L4 LB that runs on every server eliminates the "LB tier" as a separate capacity-planning problem. The LB scales with the fleet.

Trade-offs#

Where does the load balancer sit?#

Which algorithm picks the backend?#

Common Pitfalls#

Exercise#

Design the load balancing tier for a public API that serves 200k RPS across 80 stateless pods, needs TLS termination, path-based routing to four services, per-tenant rate limiting, and zero-downtime deploys. Decide between ALB, NLB + Envoy, or NGINX, and justify the choice against cost, feature fit, and operational burden.

Key Takeaways#

• L4 is fast and dumb; L7 is smart and expensive. Most production stacks use both: L4 in front for packet volume, L7 behind for routing intelligence.
• P2C least-request is the best default algorithm. It self-balances, resists herding, and handles heterogeneous backends without configuration.
• The load balancing algorithm matters less than good health checks and connection draining. A perfect algorithm with broken health checks still sends traffic to dead servers.
• Sticky sessions are a smell. Externalize state to Redis or a JWT; use consistent hash only for cache affinity where losing affinity degrades latency, not correctness.
• Anycast with stateless L4 gives sub-second global failover. DNS-based GSLB is simpler but fails over in minutes due to TTL caching.
• Every server can be a load balancer. Cloudflare's Unimog proves that eliminating a separate LB tier removes a capacity-planning problem at less than 1% CPU overhead^[2:8].
• Enable SO_REUSEPORT on any multi-core proxy. It is one config line that eliminates accept-queue contention and can double throughput^[24:1].

Further Reading#

• Maglev: A Fast and Reliable Software Network Load Balancer (Eisenbud et al., NSDI 2016) - The foundational L4-at-scale paper; introduces the Maglev hash and explains how Google serves all production traffic without shared LB state.
• Unimog: Cloudflare's Edge Load Balancer (2020) - The most detailed public write-up of a global L4 LB, including daisy-chaining for connection persistence and the oscillation incident.
• Open-sourcing Katran (Meta Engineering, 2018) - How XDP+eBPF lets an L4 LB run colocated with backend applications at 10M+ pps per core.
• Rethinking Netflix's Edge Load Balancing (2018) - P2C plus server-reported utilization with real production incident data; the best public case for why round robin fails.
• The Power of Two Random Choices: A Survey of Techniques and Results (Mitzenmacher, 2001) - The theoretical basis cited by every major modern LB; proves d=2 is exponentially better than d=1.
• Envoy Load Balancer Documentation - The reference L7 LB's menu of algorithms (ring hash, Maglev, P2C, weighted round robin) and why each exists.
• Socket Sharding in NGINX 1.9.1 (F5/NGINX, 2015) - The SO_REUSEPORT benchmark that unlocks multi-core scaling; one of the highest-ROI tuning changes for any proxy.
• HAProxy Forwards Over 2M RPS on a Single Arm Instance (2021) - A public software LB benchmark near the top of what one box can do; useful for capacity planning.
• Consistent Hashing and Random Trees (Karger et al., STOC 1997) - The original consistent hashing paper; prerequisite reading for understanding Maglev, ring hash, and jump hash.
• GLB: GitHub's Open Source Load Balancer (2018) - Rendezvous hashing and the "second chance" technique for stateless connection persistence; a good complement to Unimog.

Flashcards#

References#