Design a DNS Service (Cloudflare 1.1.1.1 / Google 8.8.8.8)

TL;DR. A public recursive resolver answers the lookup that precedes every TCP connection on the internet. Cloudflare 1.1.1.1 handles 1.9 trillion queries per day from 300+ anycast POPs^[1], while Google 8.8.8.8 reportedly serves over 1 trillion^[2]. The hard part is not resolving a name; it is serving the entire internet from one IPv4 address at p99 under 20 ms, surviving multi-terabit amplification DDoS, validating DNSSEC within the latency budget, and keeping a 50 GB per-POP cache coherent without global coordination. The pivotal trade-off: latency (you cannot wait 300 ms for a root walk on every miss) vs. survivability (your open resolver is a weaponizable amplifier).

Learning Objectives#

• Design a global anycast recursive resolver for 22M+ qps and reason about per-POP capacity
• Walk a recursive resolution from root to TLD to authoritative with DNSSEC validation at each step
• Compare UDP/53, TCP/53, DoT/853, DoH/443, and DoQ/853 and justify transport selection
• Design a per-POP cache with TTL-bounded staleness, negative caching, and scan-resistant eviction
• Mitigate amplification DDoS with source-address validation, Response Rate Limiting, and anycast absorption
• Estimate storage, bandwidth, and query fan-out for a Cloudflare-class resolver

Intuition#

A single-server DNS resolver works fine for a home network. Install BIND, point your router at it, and every name resolves in a few milliseconds from cache. At 10 users, this is trivial.

At 10 million concurrent users, three forces break the naive approach. First, latency. A cache miss walks root to TLD to authoritative, each hop potentially 50+ ms away. With millions of unique domains, cold misses dominate and p99 climbs past 500 ms. Second, DDoS. An open resolver on a single IP is a weapon: attackers send 60-byte queries with a spoofed source, and your server reflects 4 KB DNSSEC-signed responses at the victim. Amplification factors exceed 50x. Third, availability. A single server in one location means one BGP path, one power feed, one failure domain. When Dyn's authoritative DNS went down in 2016, Twitter, Netflix, and Reddit became unreachable^[3].

The insight that unlocks the design: announce one IP address from 300 locations simultaneously via BGP anycast. Each client packet routes to the topologically nearest POP with zero client configuration. DDoS traffic spreads across all POPs instead of concentrating on one. Each POP runs its own cache, its own rate limiter, and its own recursive resolver. There is no global coordination, no distributed lock, no cross-POP invalidation. TTL-bounded staleness is the only coherence guarantee, and that is fine because DNS was designed as an eventually consistent system from day one^[4].

Requirements#

Clarifying Questions#

• Q: Recursive resolver (1.1.1.1) or authoritative (Route 53)? Assume: Recursive. We resolve any name on the internet, not host zone files.

• Q: DNSSEC validation required from day one? Assume: Yes. Validate the chain of trust when zones are signed; serve without validation for unsigned zones.

• Q: Which encrypted transports? Assume: UDP/53 default, plus DoT (RFC 7858), DoH (RFC 8484), and DoQ (RFC 9250)^[5][6][7].

• Q: Privacy posture? Assume: No client IP logging beyond 24 hours. QNAME minimization (RFC 9156) to limit data leaked to root/TLD servers^[8].

• Q: Performance SLA? Assume: p50 < 10 ms cache hit, p99 < 20 ms cache hit, p99 < 500 ms cache miss. Measured at the POP.

• Q: How many POPs day one? Assume: 300+ locations, single anycast /24 prefix per address family.

Functional Requirements#

• Accept queries on UDP/53, TCP/53, DoT/853, DoH/443, DoQ/853
• Resolve A, AAAA, CNAME, MX, TXT, NS, SRV, CAA, and all standard record types
• Follow NS delegations iteratively from root to authoritative
• Cache positive responses under their TTL; cache NXDOMAIN/NODATA per RFC 2308^[9]
• Validate DNSSEC when the zone publishes DS/DNSKEY records
• Return SERVFAIL on broken DNSSEC chains rather than silently serving unvalidated data

Non-Functional Requirements#

• Load: 22M qps steady state (1.9T queries/day), peak 2-3x during events^[1:1]
• Latency: p50 < 10 ms, p99 < 20 ms for cache hits; p99 < 500 ms for cold misses
• Availability: 99.99% per POP; effectively 100% globally via anycast failover
• Cache hit ratio: > 80% at the POP level^[1:2]
• DDoS survivability: absorb multi-Tbps amplification attacks without service degradation

Capacity Estimation#

Key derivations:

• Transport mix: 86.6% UDP, 9.6% DoT, 2.0% TCP, 1.7% DoH (Feb 2025)^[1:6]. DoQ is growing but still < 1%.
• Negative cache: RFC 2308 default 300 s TTL for NXDOMAIN^[9:1]. Without it, typo floods hammer TLD auth servers.
• DNSSEC overhead: ~10 ms extra per cache miss for signature verification; under 20% of queries hit signed zones^[1:7].

API and Data Model#

API Design#

DNS uses a binary wire protocol (RFC 1035), not REST. The "API" is the query/response format:

QUERY (UDP/53, DoT/853, DoH/443, DoQ/853):
  Header: ID(16b), QR=0, RD=1, QDCOUNT=1
  Question: QNAME=example.com, QTYPE=A, QCLASS=IN

RESPONSE:
  Header: ID(match), QR=1, AA=0, RA=1, RCODE=NOERROR
  Answer: example.com A 93.184.216.34 TTL=3600
  Authority: example.com NS ns1.example.com
  Additional: ns1.example.com A 198.51.100.1

DoH variant (RFC 8484):
  POST https://1.1.1.1/dns-query
  Content-Type: application/dns-message
  Body: <wire-format query>
  Response: 200 OK, application/dns-message

  GET https://1.1.1.1/dns-query?dns=<base64url-encoded-query>

Design notes:

• EDNS0 (RFC 6891): extends the 512 B UDP limit to 4096 B; required for DNSSEC responses carrying RRSIGs.
• TC flag: if the response exceeds the negotiated buffer, set TC=1 and the client retries over TCP.
• 0x20 encoding: mix case in QNAME (eXaMpLe.CoM) as additional entropy against cache poisoning^[10].

Data Model#

-- Per-POP in-memory cache (concurrent hash map)
key:   (qname, qtype, qclass, DO_bit, CD_bit)
value: {
    rrset:       []ResourceRecord,
    rrsig:       []RRSIG,          -- if DNSSEC-signed
    ttl_expiry:  unix_timestamp,
    insert_time: unix_timestamp,
    cache_class: POSITIVE | NXDOMAIN | NODATA | SERVFAIL
}
-- Partition key: fnv64(qname, qtype, DO, CD)[^16]
-- Eviction: ARC (scan-resistant, no single scan evicts working set)

-- Negative cache (separate store, shorter TTLs)
key:   (qname, qtype)
value: {
    soa_minimum: uint32,           -- from SOA MINIMUM field
    ttl_expiry:  unix_timestamp,
    nxdomain:    bool
}

-- Rate-limit state (per-POP shared memory)
key:   source_ip_prefix (/32 for IPv4, /48 for IPv6)
value: {
    tokens:      float64,
    last_refill: unix_timestamp
}

High-Level Architecture#

Each POP is an independent resolver with its own cache. Anycast routes clients to the nearest POP; no GSLB or client configuration required. DDoS spreads across all 300+ POPs automatically.

Write path (cache population): On a cache miss, the resolver iteratively queries root, TLD, and authoritative servers. Each response is validated (DNSSEC if signed), then inserted into the local cache with the authoritative TTL. No cross-POP propagation occurs.

Read path (cache hit): The query arrives, passes the rate limiter, hits the cache in < 1 ms, passes the response rate limiter, and returns to the client. Over 80% of queries take this path^[1:8].

DDoS path: Ingress rate limiting drops floods before they reach the cache. Egress Response Rate Limiting (RRL) throttles identical responses to the same /24 prefix, forcing legitimate clients to retry over TCP while dropping amplification traffic^[11].

Deep Dives#

Anycast routing and BGP survivability#

Every POP announces the same /24 prefix (1.1.1.0/24) via BGP peering with local transits and IXPs^[12]. A client's ISP picks the topologically closest origin via shortest-AS-path. No GeoDNS, no GSLB, no client-side failover logic.

Why anycast works for DNS: DNS queries are stateless UDP datagrams. Unlike TCP (where mid-connection rerouting breaks the session), a DNS query-response pair completes in one round trip. If a POP fails, BGP reconverges in seconds to minutes, and subsequent queries route to the next-nearest POP. The client retries automatically (stub resolvers retry after 1-5 seconds).

BGP risks: Anycast's Achilles heel is BGP itself. Route leaks and hijacks can blackhole the prefix globally. In May 2018, approximately eight weeks after launch, AS58879 (Shanghai Anchang Network) briefly announced 1.1.1.0/24; Hurricane Electric (AS6939) propagated the leak before other peers filtered it, and the hijack lasted under two minutes^[13]. Cloudflare's 2024-06-27 incident combined a hijack and a route leak, degrading 1.1.1.1 for a small percentage of users^[12:1]. In 2025, an internal config change accidentally withdrew the prefix for 62 minutes^[14].

Mitigations: RPKI Route Origin Authorization (ROA) for the prefix; automated BGP monitoring that detects anomalous withdrawals within seconds; peer-filter policies requiring RPKI-valid origins; and operational runbooks for rapid re-announcement.

Anycast routes each client to the topologically nearest POP. On POP failure, BGP reconverges and traffic shifts to the next-nearest origin within seconds.

DNSSEC chain of trust and validation#

DNSSEC provides cryptographic proof that a DNS response has not been tampered with between the authoritative server and the resolver^[15]. Without it, cache poisoning attacks (Kaminsky 2008, CVE-2008-1447) can redirect bank.com to an attacker-controlled IP by flooding forged responses^[10:1].

Chain of trust: The IANA root KSK is the trust anchor. Each zone has a Key Signing Key (KSK) that signs the DNSKEY RRset, and a Zone Signing Key (ZSK) that signs all other records. The parent zone publishes a DS record containing a hash of the child's KSK. Validation walks: root KSK signs root DNSKEY, root ZSK signs .com DS, .com KSK signs .com DNSKEY, .com ZSK signs example.com DS, and so on down to the target RRSIG^[15:1].

Performance cost: ~10 ms extra per cache miss for cryptographic verification. Once validated, the result is cached with its RRSIG, so subsequent hits pay no validation cost. Under 20% of queries hit signed zones in practice^[1:9]. About 93% of gTLDs are signed, but only ~65% of ccTLDs^[1:10].

Failure mode: A single misconfigured signature (expired RRSIG, wrong DS hash) returns SERVFAIL for all validating resolvers. DNSSEC is "very unforgiving"^[16]. Cloudflare implements Negative Trust Anchors to temporarily bypass validation for known-misconfigured zones, preventing user-visible breakage from operator errors.

Root KSK anchors trust. Each parent publishes a DS hash of the child's KSK, forming an unbroken cryptographic chain to the target record's RRSIG.

DDoS mitigation and Response Rate Limiting#

An open recursive resolver is inherently weaponizable. An attacker sends a 60-byte query with the victim's IP as the spoofed source; the resolver sends a 4 KB+ DNSSEC-signed response to the victim. Amplification factors exceed 50x^[11:1]. The resolver becomes a booter without knowing it.

Defense layers:

1. BCP 38 source-address validation: upstream ISPs should drop packets with spoofed sources. Not universally deployed, but reduces attack surface.
2. Per-source-IP rate limiting at ingress: token bucket keyed on source /32 (IPv4) or /48 (IPv6). Drops floods before they consume resolver CPU.
3. Response Rate Limiting (RRL) at egress: the canonical mechanism from ISC Technical Note 2012-1^[11:2]. A token bucket keyed on (source-prefix, response-class). Each identical response debits one token; when the bucket is empty, responses are dropped or truncated (TC=1, forcing TCP retry for legitimate clients)^[17].
4. Anycast dilution: attack traffic spreads across 300+ POPs. No single POP absorbs the full volume.
5. Drop ANY queries: the ANY query type returns all records for a name, producing the largest possible response. Modern resolvers return minimal answers or refuse ANY entirely^[17:1].

The Dyn lesson: On 2016-10-21, the Mirai botnet (~100,000 IoT devices) hit Dyn's authoritative DNS across two major attack windows followed by smaller aftershocks; Dyn measured packet-flow bursts 40-50x normal and noted that external estimates of ~1.2 Tbps could not be verified from their own data^[3:1][18]. Twitter, Netflix, GitHub, and Reddit went dark. The blast radius was amplified because many sites used only Dyn nameservers. Design lesson: multi-provider redundancy and shuffle-sharding (Route 53 uses 2,048 virtual nameservers yielding ~730 billion unique shard combinations)^[19] are essential for authoritative DNS.

Inbound queries pass per-source-IP rate limiting, then hit the cache fast path. Responses pass through RRL before egress. Throttled responses set TC=1, forcing legitimate clients to retry over TCP while amplification traffic is dropped.

Caching architecture and negative caching#

The cache is the performance engine. Over 80% of queries answer from cache in under 1 ms^[1:11]. The design challenge is maximizing hit rate while resisting scan attacks and handling TTL expiry gracefully.

Cache structure: Per-POP in-memory hash map, ~100M entries at ~500 B each (~50 GB). The cache key includes (qname, qtype, DO bit, CD bit) because DNSSEC-aware and non-aware clients can legitimately receive different answers for the same name^[20].

Eviction: Cloudflare's BigPineapple uses Adaptive Replacement Cache (ARC), which resists scan attacks by maintaining ghost lists of recently evicted entries^[20:1]. A single zone-enumeration scan cannot flush the working set.

Negative caching (RFC 2308): NXDOMAIN and NODATA responses are cached using the SOA MINIMUM field as TTL (default 300 s)^[9:2]. Without negative caching, a typoed domain or malware beacon asking for random123.example.com millions of times would flood the .com TLD authoritative servers.

Thundering herd on TTL expiry: When a popular domain's TTL expires, 300+ POPs simultaneously issue upstream queries. Mitigations: stale-while-revalidate (serve the expired record for a few seconds while refreshing asynchronously); prefetch popular entries before TTL expires; de-duplicate in-flight queries within the POP so only one upstream call is made per (qname, qtype) even if thousands of clients ask simultaneously^[20:2].

Consistent hashing within a POP: Queries for the same registered domain are steered to the same subset of nodes within a POP, improving cache hit ratio by reducing redundant upstream queries^[20:3].

Real-World Example#

Cloudflare 1.1.1.1 (BigPineapple) launched on April 1, 2018, built on CZ NIC's Knot Resolver for proven DNSSEC validation^[21]. The team wrapped it with Cloudflare's edge stack: Unimog (L4 load balancer) distributes queries within a POP, TLS terminators handle DoT/DoH, and Quicksilver propagates configuration globally in seconds.

Over time, the team replaced the C-based core with a Rust-based resolver on the tokio async runtime, rewrote the plugin system to use WebAssembly sandboxes (via Wasmer), and replaced the LMDB cache with an ARC-based cache that resists zone-enumeration scans^[20:4]. Wasm plugins run in isolated memory spaces, can be hot-swapped worldwide via Quicksilver, and pay a memory-copy cost amortized by shared-memory mapping at instantiation.

Scale (Feb 2025): 1.9 trillion queries/day (~22M qps steady state) from 300+ POPs across ~250 countries^[1:12]. Cache hit ratio exceeds 80%. Transport mix: 86.6% UDP, 9.6% DoT, 2.0% TCP, 1.7% DoH. QNAME minimization ships by default since launch^[21:1]. Privacy commitment: no client IP stored, logs deleted within 24 hours, independently audited.

Failures that shaped the architecture: A 2024 BGP hijack+leak degraded service for a small percentage of users^[12:2]. A 2025 internal config change accidentally withdrew the anycast prefix for 62 minutes^[14:1]. A 2026 code change reordered CNAME records, breaking resolution for some clients^[22]. Each incident drove improvements: automated BGP anomaly detection, prefix-withdrawal circuit breakers, and stricter response-ordering tests.

The one insight non-experts miss: DDoS is not an edge case for DNS. It is the design driver. Every architectural decision (anycast, RRL, per-POP independence, no cross-POP state) exists because the service must survive being attacked continuously.

Trade-offs#

The single biggest meta-decision: per-POP independence vs. cross-POP coordination. Per-POP independence means no global lock, no distributed cache invalidation, and no single point of failure. The cost is TTL-bounded staleness and redundant upstream queries across POPs. Cross-POP gossip would improve cache efficiency but would 300x the write path and introduce a coordination failure mode. Every production public resolver chooses per-POP independence.

Scaling and Failure Modes#

At 10x load (220M qps):

• Per-POP QPS rises to ~730K average. Top POPs hit 3-7M qps. Horizontal scaling within the POP (more resolver nodes behind Unimog) handles this.
• Cache memory pressure increases as more long-tail domains are queried. ARC eviction keeps the working set hot.
• Upstream authoritative servers see 10x more misses. Prefetch and stale-while-revalidate reduce the spike.

At 100x load (2.2B qps):

• Anycast prefix must be announced from 1,000+ POPs to maintain per-POP headroom.
• DoH/DoT TLS termination becomes CPU-bound. Hardware acceleration (AES-NI, kernel TLS offload) is required.
• BGP convergence time matters more: a 60-second reconvergence at 100x load means 60 seconds of dropped queries at the failed POP.

At 1000x load (22B qps):

• Architectural shift: edge-compute model where the resolver runs on every CDN edge server (Cloudflare already does this). No dedicated "DNS POP"; every server is a resolver.

Failure modes:

• BGP hijack/leak: upstream AS announces the resolver prefix, blackholing traffic. Response: RPKI ROA, automated monitoring, rapid re-announcement. Blast radius limited by anycast (only affected ASes lose service)^[12:3][14:2].
• DNSSEC key rollover failure: a misconfigured KSK/ZSK breaks an entire zone for validating resolvers. Response: Negative Trust Anchors temporarily bypass validation for the affected zone^[16:1].
• Amplification DDoS: attacker weaponizes the resolver against a victim. Response: RRL at egress, BCP 38 upstream, anycast dilution^[11:3].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Capacity estimation for a new POP#

You are deploying a new POP in Mumbai that will serve 500K qps at peak. The cache hit ratio is expected to be 75% initially (lower than the global 80% because the cache is cold). How much memory do you need for the cache, and how many upstream queries per second will hit authoritative servers during the warm-up period?

Key Takeaways#

• Anycast serves the internet from one IP. GeoDNS serves it from many. Public resolvers use anycast because it requires zero client configuration and spreads DDoS automatically.
• DDoS is the design driver, not an afterthought. Rate limiting, RRL, and anycast absorption are load-bearing architecture for any open resolver.
• Each POP is independent. No cross-POP coordination, no global lock, no distributed cache. TTL-bounded staleness is the only coherence guarantee.
• Negative caching protects the hierarchy. RFC 2308 NXDOMAIN caching prevents typo floods from DDoSing TLD authoritative servers.
• DNSSEC is the only complete defense against cache poisoning. Source-port randomization and 0x20 encoding raise the bar; DNSSEC makes it cryptographic.
• Five transports matter. UDP (default), TCP (fallback), DoT (privacy), DoH (censorship resistance), DoQ (next-gen encrypted with 0-RTT).

Further Reading#

• Introducing DNS Resolver 1.1.1.1. The launch post covering Knot Resolver choice, QNAME minimization, DoT/DoH, and aggressive negative caching.
• How Rust and Wasm power Cloudflare's 1.1.1.1. Detailed BigPineapple internals: ARC cache, tokio runtime, Wasm plugin sandbox, consistent hashing within POPs.
• Some TXT about new DNS insights on Cloudflare Radar. Authoritative 2025 numbers for query volume, transport mix, cache hit ratio, and DNSSEC adoption.
• Workload isolation using shuffle-sharding. Colm MacCarthaigh on how Route 53 uses 2,048 virtual nameservers for DDoS isolation.
• ISC Technical Note 2012-1: Response Rate Limiting. The original RRL specification that protects authoritative servers from amplification abuse.
• RFC 4033: DNS Security Introduction and Requirements. The DNSSEC architecture specification.
• DDoS attacks on Dyn (Wikipedia). Canonical public record of the 2016-10-21 Mirai incident that took down half the internet's east coast.

Flashcards#

References#