Design a Feature Flag Service (LaunchDarkly / Harness FME / Unleash)

TL;DR. A feature flag service replaces deploy-time conditionals with runtime decisions, enabling trunk-based development, canary rollouts, kill switches, and A/B testing on a single platform. LaunchDarkly processes roughly 20 trillion flag evaluations per day^[1] with a 99.999% availability target. The pivotal architectural decision is SDK-side evaluation with streaming config distribution: flag checks become sub-millisecond local function calls, the SDK survives total control-plane outage via last-known-good cache, and kill switches propagate in under 60 seconds globally via SSE streams.

Learning Objectives#

• Design an SDK-side evaluation architecture that handles 50M flag checks/sec with sub-millisecond latency
• Implement deterministic user bucketing via SHA-256(salt + user_id) mod 10000 that prevents correlated rollouts across flags
• Architect a config distribution layer (streaming SSE + CDN polling) that propagates kill switches to 10K servers in under 60 seconds
• Size an evaluation-event pipeline for 250 TB/day ingestion without drowning the analytics store^[1:1]
• Reason about experimentation statistics: sequential testing (SPRT), CUPED variance reduction, and false-discovery correction at scale
• Handle SDK version skew gracefully so unknown rule operators fall back safely rather than crashing production

Intuition#

A feature flag looks trivial. if (flag.isOn("new-checkout", user)) { ... }. A junior engineer could build this with a database table and an API call. It handles 10 users fine.

At 10,000 servers checking 500 flags each on every request, the architecture collapses in three ways simultaneously. First, latency: a 10 ms RPC per flag check times 500 checks per page load adds 5 seconds of pure flag-evaluation overhead^[2]. Second, availability: that central flag service is now a single point of failure for your entire fleet. Third, propagation: when a payment bug is melting down production and you flip the kill switch, "eventually consistent in 60 seconds" is the difference between a blip and a front-page outage.

The insight that unlocks the design: push the rules to the SDK, not the answers to the server. The SDK downloads the full targeting ruleset, evaluates locally in sub-millisecond time, and updates via a streaming connection. The flag service becomes a control plane that publishes config, not a data plane that answers queries. This is the architecture LaunchDarkly, Statsig, and every mature platform converged on^[3][4][5].

Requirements#

Clarifying Questions#

• Q: SDK-side evaluation or client-server call per check? Assume: SDK-side for server SDKs (full ruleset downloaded). Client SDKs get pre-evaluated values (no rule leakage to browsers).

• Q: What flag types do we support? Assume: Boolean (release/kill-switch), string-variant (A/B/n experiments), JSON-blob (dynamic config). All share one evaluation engine.

• Q: Kill-switch propagation SLA? Assume: Under 60 seconds p99 globally. Streaming is mandatory; polling-only has a 30-60 second floor^[5:1].

• Q: Multi-tenant SaaS or single-company internal? Assume: Multi-tenant SaaS for thousands of customers on a shared control plane, per-tenant isolation for config and events^[1:2].

• Q: Experimentation scope? Assume: Classic A/B with sequential testing, mutual-exclusion layers for 10K concurrent experiments, CUPED variance reduction.

• Q: Regulatory requirements? Assume: SOC 2 audit log (who changed what when), RBAC per environment, data residency for EU/federal tenants^[6].

Functional Requirements#

• CRUD flags with targeting rules (attribute predicates, segment membership, percentage rollouts)
• Deterministic user bucketing: same user always gets same variant for same flag
• Kill-switch with sub-60s global propagation
• A/B experiment assignment with exposure logging and statistical analysis
• Immutable audit log of every flag change
• SDKs in 10+ languages with offline-safe last-known-good fallback

Non-Functional Requirements#

• Evaluations: 20T/day (~230M/sec peak)^[1:3]
• SDK eval latency: sub-millisecond (local function call, no network)
• Kill-switch propagation: p99 < 60 seconds globally
• Control-plane availability: 99.999%^[1:4]
• Event ingestion: 250 TB/day^[1:5]
• Tenants: thousands of customers, 100K flags per enterprise tenant

Capacity Estimation#

Key ratios:

• Read:write ratio: ~1,000,000:1 (evaluations vs. flag changes)
• CDN hit rate: >99% for config bundles (changes are rare)
• Event sampling: 10% default, 100% for flags in active experiments
• Propagation budget: 60s total = DB write (5ms) + bundle build (200ms) + CDN invalidation (2s) + SSE push (500ms) + SDK swap (instant) + buffer

API and Data Model#

API Design#

POST /v1/flags
  Body: { "key": "new-checkout", "variants": ["control","treatment"], "rules": [...] }
  Returns: 201 { "key": "new-checkout", "version": 1 }

PATCH /v1/flags/{key}
  Body: { "rules": [...], "comment": "ramp to 50%" }
  Returns: 200 { "version": 42 }
  Side-effect: triggers config publish + SSE push

POST /v1/flags/{key}/kill
  Returns: 200 { "killed": true, "propagation_started": "2026-05-04T..." }
  Bypasses approval workflow; <60s SLA

GET /sdk/v1/config?env_key={key}&etag={version}
  Returns: 200 (full config bundle, gzipped) | 304 Not Modified
  Served from CDN edge

SSE /sdk/v1/stream?env_key={key}
  Pushes: { "type": "patch", "path": "/flags/new-checkout", "data": {...} }

POST /sdk/v1/events
  Body: [{ "flag_key": "new-checkout", "variant": "treatment", "user_hash": "a3f2...", "ts": 1714800000 }]
  Fire-and-forget; batched by SDK every 30s

Data Model#

-- Source of truth (PostgreSQL, per-tenant schema)
CREATE TABLE flags (
  tenant_id   UUID NOT NULL,
  key         TEXT NOT NULL,
  version     BIGINT NOT NULL,
  variants    JSONB NOT NULL,
  rules       JSONB NOT NULL,  -- ordered list of targeting predicates
  default_variant TEXT NOT NULL,
  killed      BOOLEAN DEFAULT FALSE,
  PRIMARY KEY (tenant_id, key)
);

CREATE TABLE segments (
  tenant_id   UUID NOT NULL,
  key         TEXT NOT NULL,
  user_ids    TEXT[],           -- explicit membership
  rules       JSONB,           -- attribute-based membership
  PRIMARY KEY (tenant_id, key)
);

CREATE TABLE audit_log (
  id          BIGSERIAL PRIMARY KEY,
  tenant_id   UUID NOT NULL,
  actor       TEXT NOT NULL,
  flag_key    TEXT NOT NULL,
  old_version BIGINT,
  new_version BIGINT,
  diff        JSONB NOT NULL,
  created_at  TIMESTAMPTZ DEFAULT NOW()
);

Config bundles are built per (tenant_id, environment_id, version), signed, gzipped, and stored in S3 behind a CDN. Evaluation events flow through Kafka partitioned by tenant_id into ClickHouse for experiment analysis.

High-Level Architecture#

Control-plane writes propagate to SDKs via CDN (polling) and SSE (streaming); evaluation events flow back through Kafka to ClickHouse and the experimentation engine.

Write path: An operator changes a flag rule via the Admin UI. The API writes to PostgreSQL (incrementing the flag version and appending to the audit log), then enqueues a config-publish job. The Config Publisher builds a new per-environment bundle, uploads to S3, invalidates the CDN, and publishes a change event to the SSE fanout service.

Read path (server SDK): At startup, the SDK opens an SSE connection and receives the current config bundle. Flag checks are local function calls against the in-memory ruleset. When the SSE stream pushes a patch, the SDK atomically swaps the in-memory spec. If the stream disconnects, the SDK falls back to polling the CDN every 30 seconds, then to the on-disk last-known-good cache^[5:2].

Read path (client SDK): Browser and mobile SDKs cannot receive the full ruleset (it would leak targeting logic and user allowlists). Instead, the server evaluates for the specific user and pushes pre-computed values over a personalised SSE stream^[3:1].

Deep Dives#

Deterministic bucketing: the hash that prevents flicker#

The core evaluation primitive: given a (flag_key, user_id, rollout_percentage) tuple, deterministically assign the user to a variant without any server-side state.

The algorithm:

function bucket(unitID: string, salt: string): number {
  const hash = SHA256(salt + unitID)
  const head = hash.readBigUInt64BE(0)
  return Number(head % 10000n)  // 0-9999
}

The bucket (0-9999) is compared against rollout thresholds. If bucket < rollout_pct * 100, the user gets the treatment variant^[4:1].

Why the salt is load-bearing: If you hash on user_id alone, the same user lands in the same bucket for every flag. The first 10% of users by hash see every new feature simultaneously, creating correlated rollouts. Including a per-flag salt decorrelates assignments across flags. Statsig uses modulus 10,000 for experiments and 1,000 for mutual-exclusion layers^[4:2].

Why this prevents flicker: Rolling a flag from 0% to 50% back to 0% back to 50% re-exposes the same 50% of users each time. No server-side user -> bucket cache is needed. Statsig notes that customers who tried memoising assignments in Redis ended up paying more for Redis than for the flag service itself^[4:3].

Including the per-rule salt in the hash decorrelates rollouts across flags and guarantees the same user lands in the same bucket across re-fetches.

Rule evaluation order: Rules are an ordered list of predicates evaluated top-down with short-circuit on first match^[7]. A typical flag has: (1) internal employees get treatment at 100%, (2) beta segment gets treatment, (3) public gets treatment at 10%, (4) default is control.

Targeting rules short-circuit top-down; the first matching rule determines the variant.

Kill-switch propagation: the 60-second SLA#

When production is on fire, the kill switch must propagate globally in under 60 seconds. Every edge in the propagation chain contributes to the SLA; the SLA is the sum of the worst-case edges, not the mean.

The propagation sequence:

Every edge contributes to the 60-second kill-switch SLA; streaming SDKs receive the change in under 3 seconds, while polling-only SDKs wait up to 30 seconds.

The thundering-herd problem: An early LaunchDarkly design pushed a "something changed" ping on the stream, causing every connected SDK to re-fetch the full config simultaneously. In their own words: "we had created the ability to potentially DDoS ourselves"^[3:2]. The fix was twofold: (1) random jitter on reconnect so SDKs do not stampede, and (2) personalised streams that push already-evaluated values so the SDK never needs to re-fetch^[3:3].

Multi-region streaming: LaunchDarkly runs SSE fanout in three regions (North America, Europe, Asia Pacific) with Route 53 latency-based routing. Moving from single-region to three-region cut APAC stream-initialisation latency by more than 75%^[8].

Relay proxy: For customers with large SDK fleets (tens of thousands of instances), a relay proxy maintains one upstream SSE connection and fans out to local SDKs. An idle relay consumes approximately 11 MiB of memory and handles tens of thousands of concurrent SDK connections^[9][10].

Experimentation statistics: honest peeking at scale#

Once flags gate A/B tests, the platform inherits statistics problems that pure infrastructure engineers rarely encounter.

Sequential testing (SPRT): Product managers will peek at results regardless of what statisticians tell them. Sequential testing via mixture SPRT produces always-valid p-values so peeking does not inflate the false-positive rate^[11][12]. The cost: wider confidence intervals than fixed-horizon tests. The benefit: experiments can stop early when the effect is large, saving weeks of runtime.

CUPED variance reduction: Introduced by Microsoft researchers at WSDM 2013, CUPED uses a pre-experiment covariate (typically the pre-period outcome) to shrink variance by a factor of 1 - rho^2, where rho is the correlation between the covariate and the outcome^[13][14]. In practice that is a 30-70% reduction for well-correlated metrics, cutting required sample sizes proportionally. Statsig's cloud product uses a 7-day window before each user's exposure as the default covariate^[15]. CUPED is now standard at Netflix, Booking, Meta, Airbnb, and Statsig^[14:1].

False discovery rate (FDR): With 10K concurrent experiments at alpha=0.05, you expect ~500 false discoveries. Benjamini-Hochberg (1995) controls FDR by ranking p-values and is less conservative than Bonferroni^[16]. This is the right balance when running hundreds to thousands of simultaneous tests.

Sample ratio mismatch (SRM): A chi-squared test on realised vs. planned allocation. If a 50/50 experiment arrives as 51/49 with statistical significance, the assignment pipeline is biased. Statsig treats SRM as a hard stop because CUPED cannot fix biased assignment^[4:4][13:1].

Mutual-exclusion layers: To prevent experiment collisions on the same UI surface, the bucketing space is partitioned into non-overlapping layers. Statsig uses modulus 1,000 for layers: experiments within a layer are mutually exclusive, experiments across layers are independent^[4:5].

Real-World Example#

LaunchDarkly: from 4 billion to trillions of evaluations per day.

LaunchDarkly's architecture evolved through three distinct phases. In its early years, it served approximately 4 billion flag evaluations per day using a polling-only model: SDKs fetched a JSON config bundle on a timer^[17]. By July 2020, LaunchDarkly publicly reported it was evaluating "over 3 trillion flags a day"^[3:4]. More recent AWS case study material cites roughly 20 trillion evaluations per day and 250 TB of event data ingested daily via Amazon Kinesis^[1:7]. Published figures vary across marketing sources and point-in-time snapshots; the direction of travel (single-digit billions to low-tens of trillions over roughly a decade) is what matters architecturally.

The polling-to-streaming migration was forced by kill-switch SLA requirements. Polling has a floor of 30-60 seconds (the poll interval). For payment-path kill switches, that is unacceptable. LaunchDarkly moved to SSE streaming, initially buying the infrastructure from a third party, then building in-house after outgrowing it^[3:5].

The critical architectural decision was personalised streams for client-side SDKs. Server-side SDKs receive the full ruleset and evaluate locally. But shipping targeting rules to browsers would leak user-ID allowlists and internal logic. Instead, the server evaluates for each connected user and pushes only the applicable values over a per-connection SSE stream^[3:6].

LaunchDarkly chose SSE over WebSockets deliberately. SSE is a one-way server-to-client protocol that works through more proxies and CDNs than WebSocket upgrades, which require HTTP Upgrade negotiation that many corporate proxies block^[18].

The relay proxy solved the last-mile problem: enterprise customers with 50K+ SDK instances cannot maintain 50K upstream SSE connections. A single relay proxy maintains one connection to LaunchDarkly and fans out locally, consuming approximately 11 MiB idle^[9:1][10:1].

Trade-offs#

The biggest meta-decision: SDK-side vs. client-server evaluation. SDK-side wins overwhelmingly for general use because it eliminates the flag service as a runtime dependency. Client-server evaluation is reserved for the rare case where you need a central audit log of every single evaluation (compliance) or cannot tolerate any config lag (payment kill switches during active incidents).

Scaling and Failure Modes#

At 10x (200T evals/day):

• SSE fanout servers saturate connection limits. Mitigation: add regional fanout clusters; Route 53 latency-based routing already supports this^[8:1].
• Event pipeline Kafka partitions become hot. Mitigation: increase partition count; re-key by (tenant_id, flag_key) for better distribution.

At 100x (2 quadrillion evals/day):

• Config bundle size for mega-tenants exceeds practical SDK memory. Mitigation: sharded config subscriptions (SDK subscribes only to flags it references); server-side eval for the long tail.
• CDN invalidation storms on high-churn tenants. Mitigation: per-tenant rate limits on flag changes; stale-while-revalidate at the edge.

At 1000x:

• Architectural rewrite: move to a tiered evaluation model where hot flags (top 1%) are edge-evaluated at CDN PoPs, warm flags use SDK-side eval, and cold flags (rarely checked) use on-demand fetch.

Failure modes:

• Control-plane outage: SDKs continue evaluating from last-known-good in-memory spec. No user-visible impact until a flag change is needed. Recovery: restore control plane; SDKs reconnect automatically with exponential backoff.
• SSE fanout failure: SDKs fall back to CDN polling (30s interval). Kill-switch SLA degrades from 3s to 30s. Detection: SDK emits stream_disconnected metric.
• Thundering herd on config change: Mitigated by jitter on reconnect, personalised streams, and per-tenant CDN rate limits^[3:9].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Design the kill-switch propagation path#

Your payment service is processing $10M/hour. A bug in the new checkout flow (gated by flag new-checkout-v2) is causing 5% of transactions to fail. Design the propagation path from "operator clicks kill switch" to "first SDK evaluates the flag as OFF." Identify every edge, its worst-case latency, and the total SLA.

Key Takeaways#

• The hard part is not the if. It is propagation latency, SDK safety, deterministic bucketing, and experimentation statistics at scale.
• SDK-side eval with streaming config is the default. Sub-millisecond checks, no central SPOF, survives control-plane outage via last-known-good^[5:5].
• Include the flag key in the bucketing hash. That one detail prevents correlated rollouts and guarantees session consistency without server-side state^[4:8].
• Kill-switch SLA is the sum of worst-case edges, not the mean. Design for the streaming-reconnect case, not the happy path.
• Flag debt is the silent killer. Knight Capital lost $460M from a reused flag key^[21:1]. Enforce expiration dates, unique keys, and code-reference scans.
• 10K experiments means confronting statistics. Sequential testing, CUPED, FDR correction, and mutual-exclusion layers are not optional at scale^{[11:1][13:2][16:1]}.

Further Reading#

• Pete Hodgson, "Feature Toggles (aka Feature Flags)". The definitive taxonomy (release, experiment, ops, permission) and the implementation-pattern reference that every flag platform builds on.
• LaunchDarkly's Evolution from Polling to Streaming. The canonical public retrospective on scaling a commercial flag platform, including the self-DDoS story and the move to personalised streams.
• AWS Case Study: LaunchDarkly and Kinesis. Concrete numbers: 20T evals/day, 250 TB ingested/day, 99.999% availability target.
• Sujeet Jaiswal on Statsig SDK Architecture. Dense third-party analysis of deterministic bucketing, DataAdapter pattern, and bootstrap initialisation.
• OpenFeature Specification. CNCF incubating vendor-neutral SDK spec; the emerging standard for portable flag evaluation.
• Netflix ABlaze: It's All A/Bout Testing. Netflix's experimentation platform at 150K-450K RPS with stratified sampling and Cassandra-backed allocation.
• Deng, Xu, Kohavi, Walker, "Improving Sensitivity of Online Controlled Experiments" (WSDM 2013). The CUPED paper that cut experiment duration in half at Microsoft, now standard across the industry.
• Benjamini & Hochberg, 1995. The original FDR-control paper; necessary reading once you run 100+ concurrent experiments.

Flashcards#

References#