Design a Metrics Pipeline (Prometheus / InfluxDB / Thanos)

TL;DR. A metrics pipeline ingests, stores, queries, and alerts on time-series data from every service in your fleet. At 10M active series per cluster, Prometheus compresses samples to ~1.37 bytes each using Gorilla-style XOR encoding^[1], but local disk tops out at 2-4 weeks of retention. The pivotal trade-off is cardinality versus dimensionality: every unique label combination is a new series that costs memory, and a single unbounded label like user_id can OOM your monitoring stack during the exact incident you need it most^[2]. The architecture converges on per-cluster Prometheus for scrape, object-storage-backed long-term retention (Thanos or Mimir), tiered downsampling, and independent failure domains so the metrics pipeline survives what it observes.

Learning Objectives#

After this module, you will be able to:

• Design a pull-based metrics ingestion pipeline that handles 10M+ active series per cluster
• Identify cardinality explosion risks and implement relabel-based mitigations before they OOM Prometheus
• Justify the choice between Thanos (sidecar federation) and Cortex/Mimir (horizontal write path) for long-term storage
• Estimate capacity for a metrics pipeline using back-of-envelope math (samples/sec, bytes/sample, retention cost)
• Implement tiered downsampling (raw to 5m to 1h) and explain why it reduces storage 10-20x
• Trade off push versus pull ingestion and articulate when each model wins

Intuition#

A single Prometheus server monitoring a Kubernetes cluster looks trivial. Install the Helm chart, point Grafana at it, done. It handles 10 services fine. At 10,000 services with 50 metrics each and 20 label dimensions, you have millions of active time series, and the architecture collapses in three ways simultaneously.

First, memory. Every active series costs an in-memory head chunk plus an inverted-index entry. A careless developer adds request_id as a label, and your 5,000-series metric becomes a 50-million-series metric in one deploy cycle. Prometheus is OOM-killed. Your dashboards go dark during the incident you need them for^[3].

Second, retention. Raw 15-second samples for two years at 10M series is ~84 GB per day before compaction. That is 30 TB per year on local NVMe. Economically impossible, operationally fragile.

Third, survivability. If Prometheus runs in the same cluster it monitors, a cluster-wide failure takes both down. Facebook's Gorilla paper^[2:1] and Google SRE's monitoring chapter^[4] both identify this as a fundamental design constraint: the monitoring system must be independent infrastructure.

The insight that unlocks the design: treat the metrics pipeline as a tiered storage system with independent failure domains. Hot data (last 2 hours) lives in RAM for fast queries. Warm data (2 hours to 2 weeks) lives on local SSD. Cold data (weeks to years) lives in object storage at downsampled resolution. Each tier has different cost, latency, and durability characteristics. The scrape layer, the storage layer, and the query layer scale independently.

Requirements#

Clarifying Questions#

• Q: What is the fleet size and series cardinality? Assume: 10,000 scrape targets across 5 Kubernetes clusters, 10M active time series per cluster (50M total).

• Q: What retention do we need? Assume: 30 days raw (15-second resolution), 2 years downsampled (5-minute and 1-hour resolution).

• Q: Pull or push ingestion? Assume: Pull (Prometheus scrape) for long-lived services. Push (OTLP via OpenTelemetry Collector) for Lambda and batch jobs.

• Q: Multi-tenant or single-tenant? Assume: Multi-tenant. Engineering teams share infrastructure but have per-tenant cardinality caps.

• Q: What is the alerting SLA? Assume: Alert evaluation every 30 seconds on 5,000 rules. Page delivery within 60 seconds of threshold breach.

• Q: Must the metrics pipeline survive a full cluster outage? Assume: Yes. The monitoring stack runs in a dedicated failure domain, not colocated with the workloads it observes.

Functional Requirements#

• Scrape /metrics endpoints at configurable intervals (default 15 seconds in this design; Prometheus OSS default is 1 minute)
• Store counters, gauges, histograms, and summaries with dimensional labels^[5]
• Evaluate alerting rules and route notifications via Alertmanager
• Serve PromQL queries for dashboards with sub-second p99 for 1-hour ranges
• Downsample raw data to 5-minute and 1-hour resolution for long-range queries
• Support multi-cluster federation: query across all clusters from a single Grafana

Non-functional Requirements#

• Ingestion: 1M samples/sec per cluster (50M total across 5 clusters)
• Latency: dashboard query p99 < 1 second for 1-hour range; < 5 seconds for 30-day range
• Availability: 99.9% for the query path; alerting path must survive single-node failures
• Retention: 30 days raw, 2 years downsampled
• Cardinality cap: 100K series per metric per tenant; hard reject above threshold

Capacity Estimation#

Key ratios:

• Compression ratio: 11.7x (16 B raw to 1.37 B compressed)^[1:2]
• Downsample savings: 20x at 5-min resolution, 240x at 1-hour resolution
• Read:write ratio: ~10:1 (dashboards refresh every 5-30 seconds across many users)
• Alert evaluation cost: 5,000 rules x 30-second interval = 167 rule evaluations/sec

API and Data Model#

API Design#

GET  /api/v1/query?query=rate(http_requests_total[5m])&time=<ts>
GET  /api/v1/query_range?query=...&start=<ts>&end=<ts>&step=15s
POST /api/v1/write                    # Remote Write (Snappy-compressed Protobuf)
GET  /api/v1/labels                   # List all label names
GET  /api/v1/label/{name}/values      # List values for a label
GET  /api/v1/status/tsdb              # Cardinality stats (head series, label pairs)
POST /api/v1/admin/tsdb/delete_series # Tombstone series matching selector

Remote Write is the critical integration point. Prometheus scrapes locally, then pushes Snappy-compressed Protobuf batches to the long-term backend^[8]. The write payload is a list of TimeSeries messages, each containing labels and samples.

Data Model#

A time series is uniquely identified by its metric name plus label set:

http_requests_total{method="GET", endpoint="/api/users", status="200", cluster="us-east-1"}

The underlying storage model (Prometheus TSDB)^[6:1]:

./data
  01BKGV7JBM69T2G1BGBGM6KB12/     # 2-hour block
    chunks/000001                   # up to 512 MB per chunk segment
    index                           # inverted index: label -> posting list -> chunk offsets
    tombstones
    meta.json
  chunks_head/000001                # mmapped head chunks (currently writing)
  wal/                              # 128 MB WAL segments for crash recovery
    000000002
    checkpoint.00000001/

Each unique label combination creates a distinct series; the inverted index maps label matchers to posting lists for fast PromQL resolution.

High-Level Architecture#

A Thanos-based metrics pipeline: per-cluster Prometheus pairs scrape targets, sidecars upload 2-hour blocks to object storage, the compactor produces downsampled tiers, and the querier fans out across sidecars (recent) and store gateways (historical) to serve Grafana.

Write path: Prometheus scrapes targets every 15 seconds. Samples land in the WAL (crash safety), then the in-memory head block. Every 2 hours, the head flushes to an immutable on-disk block. The Thanos sidecar detects the new block and uploads it to S3/GCS^[9].

Read path: Grafana queries hit the Query Frontend, which splits long ranges into per-block sub-queries and caches results. The Querier fans out to sidecars (for the last 2 hours still in Prometheus) and store gateways (for historical blocks in object storage). Results are merged and deduplicated (HA pairs produce duplicate samples).

Alert path: Prometheus evaluates recording and alerting rules locally every 30 seconds. Firing alerts go to Alertmanager, which deduplicates across HA pairs, groups related alerts, and routes to PagerDuty or Slack.

Deep Dives#

Cardinality control: the silent killer#

Every unique combination of label values creates a new time series. A metric http_requests_total{endpoint, status, method} with 100 endpoints, 10 statuses, and 5 methods produces 5,000 series. Add user_id with 10M users and you have 50 million series from a single metric^[10][11].

Why it is catastrophic: Prometheus pays the cost at write time. Each series needs a head chunk (~120 bytes), a memSeries struct, and postings-list entries in the inverted index. At 50M series, that is ~6 GB of RAM just for the index, plus the head chunks. Prometheus OOMs. Scrapes time out. Alerting stops. The on-call engineer loses visibility during the incident^[3:1].

Detection: Alert on prometheus_tsdb_head_series exceeding a per-metric threshold. Track rate(prometheus_tsdb_head_series_created_total[5m]) for sudden jumps. Datadog surfaces this as ingested-vs-indexed divergence in their Metrics without Limits feature^[12].

Mitigation stack:

1. Scrape-time relabeling: metric_relabel_configs with action: labeldrop removes the offending label before it enters the TSDB. No restart needed after POST /-/reload^[13].
2. Per-tenant cardinality caps: Cortex/Mimir reject samples above a configurable series-per-tenant limit^[14].
3. Automatic suppression: FreshTracks' "Bomb Squad" pattern monitors series-creation rate and auto-drops labels that exceed a threshold^[3:2].

A single unbounded label multiplies series count by its cardinality; relabel configs intervene at scrape time before the label enters the TSDB.

Long-term storage: Thanos vs. Cortex/Mimir#

Prometheus local TSDB has no replication, no clustering, and retention limited by local disk^[6:2]. For 2-year retention across 5 clusters, you need a long-term storage layer.

Thanos (sidecar federation model): Keep vanilla Prometheus unchanged. A sidecar process watches the TSDB directory, uploads closed 2-hour blocks to S3/GCS, and exposes a gRPC StoreAPI. The Querier fans out across sidecars and store gateways. The Compactor runs as a singleton, merging blocks and producing 5-minute and 1-hour downsampled blocks^[15][9:1].

Cortex/Mimir (horizontal write path): Prometheus remote-writes to stateless distributors. Distributors hash (tenant_id, series_labels) via a consistent hash ring to place samples on N ingesters (replication factor 3). Ingesters hold 2 hours in memory, flush to object storage. Store gateways serve historical queries. Grafana Mimir scaled this to 1 billion active series in a 2022 load test^[16][17].

Cortex/Mimir's horizontal write path: stateless distributors hash series to semi-stateful ingesters with replication factor 3, then flush 2-hour blocks to object storage for long-term queries.

Gorilla compression: why 1.37 bytes per sample#

The Gorilla paper (Facebook, VLDB 2015)^[2:2] introduced paired compression that makes in-memory time-series storage practical:

Delta-of-delta for timestamps: Scrape intervals are nearly constant (15 seconds). The first derivative is ~15; the second derivative is ~0. Encoding the second derivative costs 1-2 bits when it is zero, which it almost always is^[19].

XOR for float64 values: Consecutive samples of the same metric are usually close. current XOR previous produces a value with many leading and trailing zeros. Gorilla stores: 1 control bit + 5 bits leading-zero count + 6 bits significant-bit width + the significant bits themselves^[20]. If the XOR is exactly 0 (value unchanged), only a single 0 bit is written.

Worked example: Two consecutive samples of a gauge reading 1.0 (IEEE 754: 0x3FF0000000000000). XOR = 0x0000000000000000. Encoding: single bit 0. Cost: 1 bit per sample instead of 64 bits.

Result: Facebook's Gorilla achieved ~12x compression (from 16 bytes to ~1.37 bytes per sample on average)^[2:3][19:1]. Prometheus achieves an average of 1.37 bytes per sample in production^[1:3], down from 16 bytes uncompressed. This is why a single Prometheus node can hold 10M active series in ~6-8 GB of RAM for the head block.

Real-World Example#

Uber M3: 2,500 QPS serving 8.5 billion data points per second.

Uber's metrics infrastructure evolved from Graphite + WhisperDB (2014) through a painful period of "firefighting" to M3, a purpose-built metrics platform^[21]. By November 2018, M3Query served approximately 2,500 queries per second, each scanning a 7-day rolling window and streaming ~8.5 billion data points per second from M3DB^[22][23].

M3 has three components:

1. M3Coordinator + M3Aggregator: Streaming aggregation with dynamic resolution per namespace. Infrastructure teams get 10-second resolution for 2 days; business teams get 1-minute resolution for 13 months. Same ingestion API, different cost profiles^[21:1].

2. M3DB: A distributed time-series database with an embedded inverted index. The index is the crucial piece: it resolves PromQL label-matcher expressions to posting lists before touching any samples, avoiding full scans.

3. M3Query: A PromQL-compatible engine that streams compressed blocks from M3DB and decompresses them lazily, one datapoint at a time. Functions are applied inline during decompression, minimizing network transfer and memory allocation^[22:1].

The key insight Uber's team had: multiple retention/resolution namespaces in one cluster eliminate the need for separate short-term and long-term systems. A single write path fans out to multiple namespaces with different aggregation windows. This is architecturally simpler than Thanos's two-tier model but requires a custom TSDB (M3DB) rather than vanilla Prometheus.

Netflix Atlas takes the opposite approach: an in-memory dimensional TSDB optimized for operational queries with a 6-hour hot window^[24][25]. Atlas prioritizes operational queries (the last few hours) over long retention, echoing Gorilla's philosophy that recent data is far more valuable for incident response^[2:4].

Trade-offs#

The biggest meta-decision: Thanos versus Mimir. If you already run Prometheus and want to add long-term storage incrementally, Thanos wins on operational simplicity. If you are building a multi-tenant metrics platform from scratch and expect to exceed 10M series per tenant, Mimir's horizontal write path is the better foundation. Both use the same block format and object storage backend; the difference is where the write path lives.

Scaling and Failure Modes#

At 10x (100M active series, 5 clusters):

• Single Prometheus per cluster hits memory ceiling (~32 GB for 20M series). Mitigation: shard Prometheus by namespace or service using hashmod relabeling. Two Prometheus instances per cluster, each scraping half the targets.
• Object storage query latency grows. Mitigation: Query Frontend with result caching (Memcached) and query splitting by time range.

At 100x (1B active series):

• Thanos architecture breaks down (too many sidecars, compactor cannot keep up). Mitigation: migrate to Mimir's horizontal write path with split-and-merge compaction^[17:1].
• Ingester memory becomes the bottleneck. Mitigation: increase ingester count, reduce replication factor from 3 to 2 for non-critical tenants.

At 1000x (10B active series):

• Architectural rewrite: streaming ingestion (Kafka-backed write path), per-tenant isolation at the storage layer, and aggressive pre-aggregation at the edge (aggregate before shipping).

Failure modes:

• Prometheus OOM from cardinality explosion: Detection: alert on prometheus_tsdb_head_series > threshold. Response: relabel-drop the offending label, delete tombstoned series, clean tombstones to reclaim memory. Recovery: minutes if automated, hours if manual^[3:3].
• Object storage unavailable: Recent data (last 2 hours) still served from sidecars/ingesters. Historical queries fail gracefully with partial results. Alerting continues from local Prometheus.
• Ingester failure (Mimir): Replication factor 3 means losing one ingester loses no data. The hash ring redistributes its token range to surviving ingesters. In-flight samples are retried by distributors.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Cardinality incident response#

It is 2 AM. A deploy pushed a bug that tags every http_request_total metric with request_id. Your Prometheus cardinality goes from 1M to 50M series in 15 minutes. Prometheus starts OOMing. Design the response: how you detect the explosion automatically, how you relabel-drop the bad label without restarting Prometheus, how you recover the series count, and how you prevent recurrence.

# Add to scrape config's metric_relabel_configs:
- source_labels: [__name__]
  regex: http_request_total
  action: labeldrop
  regex: request_id

# Tombstone the offending series
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/delete_series' \
  -d 'match[]=http_request_total{request_id=~".+"}'

# Reclaim disk and memory
curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

Key Takeaways#

• Cardinality is the primary scaling constraint. Not disk, not CPU, not network. Every unbounded label is a time bomb that detonates during incidents^[10:2].
• Gorilla compression makes local TSDB practical. 1.37 bytes per sample (11.7x compression) means 10M series fits in single-digit GB of RAM^[1:4].
• Object storage is the universal long-term backend. Thanos, Mimir, VictoriaMetrics, and M3 all converge on S3/GCS as the durable layer; they differ only in the write and query paths above it.
• Downsampling is not optional at 2-year retention. Raw 15-second samples for 2 years costs 20x more than tiered 5-minute and 1-hour rollups^[15:2].
• The metrics pipeline must survive what it observes. Independent failure domains, not colocated with monitored workloads^[4:3].
• Pull for long-lived services, push for ephemeral. This is not a religious debate; it is a failure-mode analysis. Pull gives you up{} for free; push works where there is no long-lived HTTP server^[13:3].

Further Reading#

• Pelkonen et al., "Gorilla: A Fast, Scalable, In-Memory Time Series Database" (VLDB 2015). The paper every metrics engineer should read; XOR/delta-of-delta compression and the in-memory-first philosophy that Prometheus, M3DB, and VictoriaMetrics all reimplement.
• Prometheus TSDB storage documentation. Canonical reference for block layout, WAL segments, retention flags, and remote read/write configuration.
• Ganesh Vernekar, "Prometheus TSDB: Compaction and Retention". The best deep-dive on head block lifecycle, compaction ranges, and retention mechanics, by a core Prometheus maintainer.
• Grafana Labs, "How we scaled Mimir to 1 billion active series". Concrete methodology for the billion-series load test; covers split-and-merge compaction and ingester scaling.
• Thanos quick tutorial. Sidecar, querier, compactor, and store gateway in one readable walkthrough; the fastest path from single Prometheus to multi-cluster long-term storage.
• Uber Engineering, "The Billion Data Point Challenge". M3Query serving 8.5B data points/sec with lazy decompression and streaming evaluation.
• Google SRE Book: Monitoring Distributed Systems. Four Golden Signals, symptoms vs. causes, and why alerting philosophy matters more than alerting tooling.
• FreshTracks, "Bomb Squad: Automatic Cardinality Explosion Detection". The definitive post on automatic detection and suppression of cardinality bombs in production Prometheus.

Flashcards#

References#