Design a Logging Platform (ELK / Loki / Splunk)

TL;DR. A logging platform ingests every textual event from an application fleet, buffers it through Kafka for durability, stores it in either an inverted-index system (Elasticsearch, Splunk) or a label-indexed object-store system (Loki), and exposes sub-5-second queries over days of data. The pivotal trade-off is what you index at ingest versus what you scan at query time: Elasticsearch indexes every field at 3-10x storage cost^[1], while Loki indexes only labels and grep-scans compressed chunks on demand^[2]. At hyperscale, Meta Scribe ingests over 2.5 TB/sec^[3], Character.AI processes 450 PB of raw log data per month^[4], and Uber compresses Spark logs 169:1 with CLP^[1:1]. The architecture converges on a Kafka buffer as the durability boundary, tiered retention (hot/warm/cold), and structured logging as the cost-control mechanism.

Learning Objectives#

After this module, you will be able to:

• Design a log ingestion pipeline with backpressure that survives downstream outages without data loss
• Compare index-based (ELK) and label-based (Loki) storage architectures and justify the choice for a given workload
• Apply BM25 scoring to understand how full-text log search ranks results
• Estimate capacity for a 10 PB/year logging platform using back-of-envelope math
• Implement tiered retention (hot, warm, cold, frozen) to satisfy compliance at minimal cost
• Justify structured logging over unstructured free-text at scale

Intuition#

Logging looks trivial at small scale. kubectl logs works. tail -f works. A single Elasticsearch node handles your startup's output. Then you grow to 500 microservices across 3,000 pods, and a single misbehaving retry loop emits 1,000 errors per second per pod. Across a 1,000-pod deployment, that is 1M log events per second from one bug. Your Elasticsearch cluster OOMs. Your Kafka partitions fill. Your on-call engineer cannot search the logs from the incident because the incident killed the logging platform.

The naive approach (write everything to one big index) fails for three reasons. First, cost: indexing every field of every log line costs 3-10x the raw data in storage^[1:2]. At 10 PB/year raw, that is 30-100 PB of index storage. Second, durability: without a buffer between producers and the index, a downstream outage means log loss. Third, query speed: searching 30 days of unindexed data is a full scan measured in minutes, not seconds.

The insight that unlocks the design: separate the durability boundary from the query boundary. Kafka absorbs ingest at any rate and holds data for days regardless of downstream health. The index (or chunk store) serves queries. These two systems scale independently, fail independently, and cost differently. Everything else, including agents, parsers, tiering, and retention policies, is plumbing between these two boundaries.

Requirements#

Clarifying Questions#

• Q: What is the peak ingest rate? Assume: 10M events/sec peak (1M/sec average), approximately 3 MB/sec per service at steady state.

• Q: What retention is required? Assume: 30 days hot (fast queries), 90 days warm, 13 months cold (compliance). PCI-DSS requires 12 months with 3 months immediately available^[5].

• Q: Full-text search or structured filters only? Assume: Both. Engineers need free-text keyword search during incidents; dashboards use structured label filters.

• Q: Multi-tenant? Assume: Yes. Each service team has per-tenant ingest quotas and query concurrency limits.

• Q: What is the query latency target? Assume: p99 under 5 seconds for queries spanning 7 days of hot data.

• Q: Must we handle PII in logs? Assume: Yes. PCI-DSS requires scrubbing cardholder data before it reaches persistent storage^[6].

Functional Requirements#

• Ingest structured and unstructured logs from any container, VM, or serverless function
• Parse, enrich, and optionally sample logs at the pipeline level
• Full-text search with relevance ranking (BM25) over hot data
• Label-based filtering with sub-second response for recent logs
• Tiered retention with automatic lifecycle transitions
• Per-tenant rate limiting and quota enforcement

Non-Functional Requirements#

• Ingestion: 10M events/sec peak, lossless up to 2x burst
• Latency: p99 < 5 seconds for 7-day queries; p99 < 30 seconds for 30-day queries
• Availability: 99.9% write path, 99.95% read path
• Durability: zero log loss during downstream outages up to 24 hours^[7]
• Compliance: SOX 7 years, HIPAA 6 years, PCI-DSS 12 months^[5:1]

Capacity Estimation#

Key ratios:

• Write:read ratio: ~100:1 (most logs are never searched)
• Compression: 15-20x typical with ZSTD on columnar stores^[4:2]; 169x with domain-specific CLP^[1:3]
• Index overhead (Elasticsearch): 3-10x raw size for inverted index + _source^[1:4]
• Cost at scale: Datadog charges ~$0.10/GB ingested (ingestion only; indexing charged separately)^[8][9]; Splunk (now Cisco, acquired March 2024) lists at $150-225/GB/day^[10]

API and Data Model#

API Design#

POST /v1/ingest
  Content-Type: application/x-protobuf (OTLP)
  Body: ExportLogsServiceRequest { resource_logs: [...] }
  Returns: 200 { "accepted": 50000 }
  Errors: 429 rate limited, 503 backpressure

GET /v1/search?query=error+timeout&start=2026-05-01T00:00:00Z&end=2026-05-04T00:00:00Z&limit=100
  Returns: 200 { "hits": [...], "total": 4521, "next_cursor": "..." }

GET /v1/streams?labels={app="checkout",env="prod"}&start=...&end=...&filter=|="timeout"
  Returns: 200 { "streams": [...], "stats": { "bytes_processed": "2.1GB" } }

POST /v1/tail
  WebSocket upgrade for live tail with label selector
  Returns: streaming log lines matching selector

GET /v1/retention/policies
  Returns: 200 { "policies": [{ "name": "hot", "days": 30 }, ...] }

Data Model#

The OpenTelemetry log record defines the canonical schema^[11]:

LogRecord {
  timestamp:          uint64 (nanoseconds since epoch)
  observed_timestamp: uint64 (collection time)
  trace_id:           bytes (W3C correlation)
  span_id:            bytes
  severity_number:    int (1-24: TRACE/DEBUG/INFO/WARN/ERROR/FATAL)
  severity_text:      string
  body:               AnyValue (structured or free-text)
  resource:           { service.name, k8s.pod.name, ... }
  attributes:         map<string, AnyValue>
}

Storage schemas differ by backend:

High-Level Architecture#

The canonical logging pipeline: agents collect and scrub on-host, Kafka provides the durability boundary, ingesters write to the hot index, and ILM transitions data through warm and cold tiers.

Write path: Agents tail container stdout, parse into structured records, apply PII scrubbing, sample INFO-level traffic, and batch-produce to Kafka. Ingester workers consume partitions, enforce per-tenant rate limits, and write to the hot index. Kafka's 7-day retention means a 24-hour downstream outage causes backlog, not loss^[7:1]. Meta's Scribe architecture uses over 40 distinct components to achieve this at 2.5 TB/sec input and 7 TB/sec output^[3:1].

Read path: The query frontend receives a search request, splits it by time range into sub-queries, fans out to the hot index (recent) and warm tier (older), merges results, and returns ranked hits. For Loki-style backends, the frontend parallelizes chunk fetches from S3.

Alert path: Streaming queries over the hot index trigger alerts on error-rate thresholds. Alert rules evaluate every 30 seconds; notifications route through Alertmanager to PagerDuty or Slack.

Deep Dives#

Index-based vs. label-based storage#

The fundamental architectural decision is how much you index at write time.

Ecosystem context (as of 2026-05). Elasticsearch relicensed from Apache 2.0 to ELv2+SSPL in January 2021; Elastic added AGPL as a third option in September 2024, making it OSI-approved open source again. AWS forked Elasticsearch as OpenSearch in April 2021; OpenSearch moved to the Linux Foundation in September 2024. Splunk was acquired by Cisco in March 2024 for $28 billion. Emerging alternatives include Datadog Logs (SaaS, per-GB pricing), Grafana Loki (label-indexed, object-store-native), VictoriaLogs (single-binary, 30x less RAM than Elasticsearch), and ClickHouse-based stacks.

Index-everything (Elasticsearch, Splunk): Every field of every log record is tokenized and added to an inverted index. Each Elasticsearch index is divided into shards, each a self-contained Lucene inverted index with posting lists mapping terms to document IDs^[12:1][14]. Queries against any field are fast because the posting list already points at matching documents. The cost: storage is 3-10x the raw log size (inverted index + stored _source field), and ingest CPU is dominated by tokenization^[1:5].

Label-based (Grafana Loki): Only a small set of stream labels (typically under 15 per stream) are indexed^[15]. The log body is compressed into chunks and flushed to object storage. The write path uses a consistent hash ring: the distributor hashes each stream's labels to an ingester, which builds compressed chunks per (tenant, label_set) and flushes them to S3/GCS along with a TSDB index mapping labels to chunk offsets^[13:1]. Full-text search is a parallel grep: the querier fetches chunks matching the label selector, decompresses them, and scans linearly for the filter string.

Columnar (ClickHouse): No traditional index. Data is stored column-by-column with LZ4/ZSTD compression, achieving 15-20x compression ratios^[4:4]. MergeTree primary keys provide sparse ordering; bloom-filter skip indexes accelerate high-cardinality field lookups. Character.AI processes 50 billion sampled log entries per month on ClickHouse Cloud with queries completing in seconds over 15-day ranges^[4:5].

The same 1 GB of raw logs consumes vastly different storage depending on the indexing strategy; ClickHouse and CLP achieve 7-169x compression by avoiding per-field inverted indexes.

The trade-off is stark: Elasticsearch gives you sub-second arbitrary queries but costs 10-50x more storage than Loki. Loki gives you cheap ingest but slow full-text search. Character.AI explicitly chose ClickHouse over Loki because "you must know the structure of your logs ahead of time" with label-based systems, and during incidents, engineers need free-text keyword search^[4:6].

Ingestion pipeline with backpressure#

The path from a running process to durable storage must survive three failure modes: the downstream store being slow, a single service flooding logs, and a network partition between agents and aggregators.

Agent layer: Fluent Bit runs as a DaemonSet with ~1 MB baseline memory (versus Fluentd's ~40 MB)^[16]. It tails container log files, applies parsing (JSON, logfmt, regex), executes PII scrubbing transforms, and produces batches to Kafka. On-disk buffering at the agent handles brief Kafka unavailability.

Kafka buffer: The critical durability boundary. With 7-day retention, a multi-hour downstream outage becomes bounded backlog rather than log loss. Datadog's 2023-03-08 outage illustrates why this matters: a systemd-networkd restart caused network connectivity loss starting at 06:00 UTC; on AWS, approximately 60% of instances were terminated and replaced by 08:00 UTC, and recovery drove traffic to roughly twice the usual throughput as the platform processed the backlog^[7:2]. Partitioning by tenant isolates noisy neighbors.

Backpressure and quotas: Meta's Scribe enforces per-category write quotas: "Quota excesses can then result in Scribe block listing a category, accepting a fraction of its logs, or applying backpressure to the Producers"^[3:2]. The same pattern applies here: per-tenant rate limits at the ingester reject excess traffic with HTTP 429, and the agent backs off exponentially.

Kafka absorbs ingest during an index outage, converting downstream failure into bounded backlog; the ingester drains at elevated rate after recovery.

Log-flood mitigation: A misbehaving service logging errors in a tight loop (1,000 errors/sec x 1,000 pods = 1M events/sec from one bug) can saturate the entire pipeline. Defense: per-tenant ingest quotas at the Kafka producer level, aggressive sampling of INFO/DEBUG at the agent (Character.AI samples backbone services at 1/10,000 for INFO^[4:7]), and circuit-breaker alerts on partition lag.

Search ranking with BM25#

When an engineer types error timeout payment into the search bar, the system must rank millions of matching log lines by relevance. BM25 (Best Matching 25) is the algorithm Elasticsearch and most full-text search engines use^[17].

The formula: For each query term q_i and document D, BM25 scores:

score(D, Q) = sum over q_i of:
  IDF(q_i) * (f(q_i,D) * (k1 + 1)) / (f(q_i,D) + k1 * (1 - b + b * |D| / avgDL))

Where IDF(q_i) penalizes common terms (a log line containing "error" in a sea of errors gets less boost), k1=1.2 controls term-frequency saturation (seeing "timeout" 50 times in one line does not score 50x higher), and b=0.75 normalizes for document length (longer log lines do not get unfair advantage)^[17:1].

Why it matters for logs: Most log queries are structured filters (service=checkout AND level=ERROR), where boolean match suffices. But during incidents, engineers search free-text patterns they have never seen before. BM25 surfaces the most relevant lines first, saving minutes of scrolling. Label-based systems like Loki cannot do this; they return lines in timestamp order via LogQL stream selectors^[18].

Limitation: BM25 requires an inverted index. Systems without one (Loki, ClickHouse) rely on timestamp ordering or simple grep-match. This is the core reason index-based systems persist despite their cost: relevance-ranked search is qualitatively different from filtered scan. Cloudflare processes 6M HTTP requests per second of analytics through ClickHouse^[19], but uses timestamp-ordered results rather than relevance ranking.

Full-text BM25 query returns relevance-ranked results via posting lists; Loki's label-match plus grep fetches and scans chunks linearly, returning results in timestamp order only.

Real-World Example#

Uber: from Elasticsearch to CLP, 169x compression, $1.8M to $10K/year.

Uber's Spark cluster generates up to 200 TB of logs per day at INFO verbosity^[1:6]. The original architecture used Elasticsearch, but at this scale "Elasticsearch incurs prohibitive operational and hardware costs"^[1:7]. SSDs burned out prematurely because Log4j writes many small records spread across job durations, causing write amplification. The temporary fix was dropping verbosity from INFO to WARN, losing critical debugging visibility.

The solution was CLP (Compressed Log Processor), deployed in two phases. Phase 1 runs a CLP-aware Log4j appender inside each JVM: it parses each log line into a timestamp, a log type (the static template), and variables (dictionary and non-dictionary), then writes the parsed intermediate representation to a ZSTD stream with a 4 MB buffer aligned to SSD blocks^[1:8]. This eliminated write amplification immediately.

Phase 2 runs offline, aggregating IRs across JVMs into CLP's final columnar format. The result: 5.38 PB of uncompressed Spark logs over 30 days compresses to 31.4 TB, a 169:1 ratio^[1:9]. CLP's compression is 2.16x better than ZSTD alone and 2.28x better than Gzip. The OSDI 2021 paper shows CLP outperforms Elasticsearch and Splunk's ingestion by over 13x^[20].

The cost impact: 30-day retention went from a projected $1.8M/year to $10K/year after Phase 1 compression. Retention improved from 3 days to 30 days at 17x lower cost^[1:10]. For non-Spark JSON logs, Uber migrated to ClickHouse, which handles structured data with columnar compression and sparse indexes more efficiently than Elasticsearch.

ILM transitions indices through tiers as they age; each tier trades query speed for cost, with object storage 10-50x cheaper than SSD-backed hot nodes.

Trade-offs#

The biggest meta-decision: index-based versus label-based. If your engineers search logs daily during incidents and need free-text keyword search, pay for the index (Elasticsearch or ClickHouse with bloom filters). If logs are primarily for compliance and occasional debugging, Loki's 10-50x cost advantage wins. Most large organizations run both: ELK for security/compliance teams, Loki for application debugging.

Scaling and Failure Modes#

At 10x (100M events/sec):

• Kafka partitions saturate. Mitigation: increase partition count per topic, shard by service namespace.
• Elasticsearch hot tier OOMs from burst ingestion. Mitigation: separate hot/warm/cold tiers with ILM, index-per-hour rollover instead of per-day^[12:2].

At 100x (1B events/sec):

• Single Kafka cluster cannot handle the throughput. Mitigation: regional Kafka clusters with cross-region replication for compliance queries.
• Query frontend becomes the bottleneck. Mitigation: result caching (Memcached), query splitting by time range, per-tenant concurrency limits.

At 1000x (Meta/Uber scale):

• Architectural rewrite: custom transport (Scribe/LogDevice), domain-specific compression (CLP), aggressive pre-sampling at the agent, and dedicated query engines per use case^[3:3][1:12].

Failure modes:

• Index cluster OOM: Kafka absorbs ingest; alert on partition lag. Recovery: scale hot nodes, force-merge segments, or temporarily route to warm tier. Cascading shard reallocation can stress adjacent nodes^[14:1].
• Kafka broker failure: Replication factor 3 means no data loss on single-broker failure. Producers retry to ISR replicas. Recovery: automatic via controller re-election.
• Agent log-flood loop: Per-tenant quotas at the ingester reject excess. Scribe's model: block-list the category and accept only a fraction^[3:4]. Alert on fluentbit_output_errors_total for silent drops.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Burst ingestion design#

Your fleet emits 3 MB/sec of logs at steady state. During a specific outage, a library starts logging errors in a loop, spiking to 50 MB/sec. Design the ingestion so that the spike does not drop logs, the downstream index does not OOM, and ops teams can still query logs at the moment of the outage. Specify: Kafka retention needed, autoscaling behavior of the ingester, and the alert that tells you ingestion is throttling.

Key Takeaways#

• What you index is what you pay for at ingest; what you do not index is what you pay for at query time. This is the single trade-off that determines your architecture.
• Kafka as a durability buffer is not optional at scale. It converts downstream failures into bounded backlog rather than log loss^[3:6].
• Structured logging (OTLP) is a 10x quality-of-life improvement. Push it upstream; fighting unstructured logs is lost time^[11:3].
• Tiered retention (hot/warm/cold) reduces storage cost 10-50x. Auditors rarely need sub-second query speed over year-old data^[5:3].
• Per-tenant rate limits prevent a single misbehaving service from killing the platform. Scribe's quota model is the gold standard^[3:7].
• BM25 relevance ranking is qualitatively different from timestamp-ordered grep. It justifies the cost of an inverted index for incident response^[17:2].

Further Reading#

• Scribe: Transporting petabytes per hour via a distributed, buffered queueing system. Meta's 12-year retrospective on the daemon+service+durable-log pattern at 2.5 TB/sec ingest.
• Reducing Logging Cost by Two Orders of Magnitude using CLP. Uber's 169x compression case study with the two-phase Log4j appender design.
• CLP: Efficient and Scalable Search on Compressed Text Logs (OSDI 2021). The original paper showing 13x ingestion advantage over Elasticsearch and Splunk.
• Grafana Loki architecture. Canonical docs for distributor, ingester, querier, chunk format, and the write/read paths.
• Practical BM25, Part 2: The BM25 Algorithm and its Variables. Readable walkthrough of k1, b, IDF, and the full scoring formula with worked examples.
• OpenTelemetry Logs Data Model. The standard schema for structured logs; read the SeverityNumber table for cross-language normalization.
• Scaling observability at Character.AI with ClickStack. 450 PB/month raw, 15x compression, 50% cost reduction with ClickHouse Cloud and OpenTelemetry Collectors.
• Loki label cardinality best practices. The canonical explanation of why high-cardinality labels break Loki and the structured metadata escape hatch.

Flashcards#

References#