Design a Semantic Cache for LLM Applications

TL;DR. A semantic cache keys on embedding vectors rather than prompt hashes, enabling 60-70% hit rates on natural-language LLM traffic where exact-match caching yields near-zero hits^[1]. The architecture embeds each prompt, runs an ANN lookup against an HNSW index, and returns the stored answer when cosine similarity exceeds a calibrated threshold. The pivotal trade-off is the similarity threshold: lowering it from 0.99 to 0.75 raises cost savings from 15.8% to 86.3% while accuracy drops less than one percentage point on general chatbot traffic^[2]. The design must handle false positives (wrong answers that look correct), per-tenant isolation, and invalidation when the underlying knowledge base changes.

Learning Objectives#

• Design an embedding-indexed cache lookup that returns in <10 ms at 10k QPS over 100M entries
• Calibrate a cosine-similarity threshold using a held-out evaluation set and defend the precision/recall trade-off
• Invalidate entries on source-document changes, embedding-model drift, and TTL expiry without re-embedding the entire corpus
• Isolate tenants via namespace, per-tenant index, or separate clusters based on compliance posture
• Measure false-positive cost in tokens, dollars, and user trust, and distinguish it from traditional cache staleness
• Distinguish cacheable prompts (deterministic RAG Q&A) from non-cacheable ones (tool-use, agentic loops) and enforce the boundary at the gateway

Intuition#

A traditional cache is a dictionary: hash the key, look it up, done. If two users type the exact same prompt, the second one gets a cached answer. But natural language is not a dictionary key. "What is Python?" and "Tell me about the Python programming language" mean the same thing, yet their SHA-256 hashes share zero bits.

The naive approach (exact-match cache) hits only when prompts are byte-identical. On real chatbot traffic, that happens less than 5% of the time. You pay for an LLM call on every paraphrase, every rephrasing, every slight variation. At $0.06 per 1K output tokens and 10K requests per second, the bill grows fast.

The insight: embed each prompt into a vector, then look up the nearest stored vector. If the distance is small enough, the answers are interchangeable. This converts a dictionary lookup into a nearest-neighbour search, and the hit rate jumps from near-zero to 60-70%^[1:1]. But it introduces a new failure mode that traditional caches never had: the cache can return the wrong answer with a 200 OK and sub-100ms latency. The user has no signal that anything went wrong. Every architectural decision in this chapter flows from managing that false-positive risk.

Requirements#

Clarifying Questions#

• Q: Which prompts are cacheable? Assume: Deterministic RAG Q&A, classification, extraction, summarization. Tool-use, function-calling, agentic loops, and time-sensitive queries bypass the cache entirely.

• Q: What is the blast radius of a false positive? Assume: For general support bots, a mildly-off answer triggers a thumbs-down. For medical/legal, it is a compliance violation. Design supports per-tenant threshold overrides.

• Q: How strict is tenant isolation? Assume: Shared index with payload filter for self-serve tenants; dedicated index for mid-tier; dedicated cluster for HIPAA/financial.

• Q: When the underlying corpus changes, how are we notified? Assume: CDC from the knowledge base via Kafka. Polling is not acceptable for sub-minute freshness.

• Q: What staleness is acceptable? Assume: Seconds for trading data (bypass cache), minutes for news, days for product documentation.

• Q: Is the LLM deterministic? Assume: Temperature 0 with fixed seed for cacheable paths. Non-deterministic calls bypass the cache.

Functional Requirements#

• Accept a prompt + tenant_id + optional context_hash; embed the prompt; ANN lookup; return cached answer if similarity >= threshold AND context_hash matches AND entry is not invalidated
• On miss, forward to the LLM and write-back on success
• Expose invalidation channels: by tenant, source_doc_id, or embedding_model_version
• Emit per-tenant hit/miss telemetry with similarity-score histograms

Non-Functional Requirements#

• Load: 10K QPS steady, 20K peak
• Latency: p99 lookup < 10 ms (embed + ANN + threshold check, excluding LLM)
• Capacity: 100M entries, growing 10M/month
• Hit rate: 70%+ on repeat-heavy traffic
• Availability: 99.9% cache-plane; LLM fallback on outage (fail-open)
• Cost: cache infra < 10% of the LLM bill it displaces

Capacity Estimation#

Key derivations:

• Embed latency budget: 2 ms per prompt on batched GPU inference (batch size 32, 2 GPUs). ANN lookup adds 2-5 ms at 100M vectors with ef=64^[4]. Total budget: 7 ms ANN + 2 ms embed + 1 ms network = 10 ms p99.
• Break-even hit rate: With a remote vector DB (30 ms miss cost), break-even is 15.4%. With in-memory HNSW (2 ms miss cost), break-even drops to 1%^[5]. This justifies the RAM investment.
• Cost savings at 70% hit rate: 7K hits/sec x avg 500 output tokens x $0.06/1K = $0.21/sec saved = ~$18K/day.

API and Data Model#

API Design#

POST /v1/cache/lookup
  Body: { "tenant_id": "acme", "prompt": "...", "context_hash": "sha256:...",
          "model": "gpt-4o", "threshold_override": 0.92 }
  Returns: 200 { "hit": true, "answer": "...", "similarity": 0.94,
                  "entry_id": "uuid", "latency_ms": 7 }
           200 { "hit": false, "answer": "...", "latency_ms": 1200 }
  Headers: X-Cache-Policy: no-store | store-only | read-through

POST /v1/cache/write
  Body: { "tenant_id": "acme", "prompt": "...", "answer": "...",
          "context_hash": "sha256:...", "source_doc_ids": ["doc-1", "doc-2"],
          "model": "gpt-4o", "embedding_model_version": "v3-small-2024" }
  Idempotent on (tenant_id, prompt_hash)
  Returns: 201 { "entry_id": "uuid" }

POST /v1/cache/invalidate
  Body: { "tenant_id": "acme", "mode": "source_doc", "target": "doc-1" }
  Returns: 202 { "job_id": "uuid", "estimated_entries": 20 }

GET /v1/cache/metrics?tenant_id=acme
  Returns: { "hit_rate": 0.72, "p50_ms": 4, "p99_ms": 9,
             "tokens_saved_24h": 45000000, "invalidation_backlog": 0 }

Data Model#

-- Vector index (Redis Stack or Qdrant collection per tenant-group)
cache_entries (
  id                     UUID PRIMARY KEY,
  vector                 FLOAT32[1536],   -- or INT8 quantized
  tenant_id              TEXT,            -- payload filter key
  prompt_hash            CHAR(64),        -- SHA-256 for exact-match L1
  answer                 TEXT,            -- <= 8 KB inline; overflow to S3
  source_doc_ids         TEXT[],          -- for CDC invalidation fan-out
  context_hash           CHAR(64),        -- RAG context fingerprint
  model                  TEXT,
  embedding_model_version TEXT,
  created_at             TIMESTAMP,
  last_hit_at            TIMESTAMP,
  hit_count              INT DEFAULT 0,
  ttl_expires_at         TIMESTAMP
)
-- Partition key: tenant_id (payload-based in Qdrant, hash tag in Redis)

-- Secondary index for invalidation (Redis hash)
source_doc_index: source_doc_id -> [entry_id, ...]

-- Tenant config (PostgreSQL)
tenant_config (
  tenant_id              TEXT PRIMARY KEY,
  threshold_override     FLOAT,           -- NULL = use global default
  ttl_policy             INTERVAL,
  isolation_mode         ENUM('namespace', 'dedicated_index', 'dedicated_cluster'),
  allowed_models         TEXT[],
  pii_redaction_enabled  BOOLEAN DEFAULT true
)

High-Level Architecture#

Two-tier cache with exact-match L1 and semantic L2. Misses fall through to the LLM; write-back populates both tiers. CDC-driven invalidation tombstones entries when the source corpus changes.

The gateway is stateless. On each request it first checks L1 (prompt hash lookup, ~1 ms, zero false-positive risk). On L1 miss, it embeds the prompt and queries the HNSW index. If the top-1 neighbour's cosine similarity exceeds the tenant's threshold AND the context_hash matches AND the entry is not tombstoned, it returns the cached answer. Otherwise it forwards to the LLM, writes back on success, and emits telemetry. Redis 8 Community Edition reaches 90% recall@100 at 200 ms median latency on one billion 768-dim FLOAT16 vectors with 50 concurrent queries^[4:1] (the benchmark measures the top-100 nearest-neighbour case; a cache's top-1 lookup at smaller scale is typically sub-10 ms), making it viable for large-scale deployments.

The cache fails open: on cache-plane outage, traffic flows directly to the LLM. Availability of the cache is a cost optimization, not a correctness requirement. The concept was popularized by GPTCache in 2023, which described itself as "an open-source semantic cache for LLM applications enabling faster answers and cost savings"^[6].

Deep Dives#

Similarity threshold calibration#

The threshold is the single highest-leverage knob in the system. AWS tests on chatbot traffic using Claude 3 Haiku with Titan Embeddings showed that moving from 0.99 to 0.75 raised hit rate from 23.5% to 90.3% and cost savings from 15.8% to 86.3%, while accuracy dropped less than one percentage point (92.1% to 91.2%)^[2:1].

Calibration process: Label a few-thousand-pair evaluation set where each row is (prompt_a, prompt_b, answer_is_interchangeable). Sweep thresholds from 0.75 to 0.99 in 0.01 steps. Plot precision against recall. Pick the threshold where precision meets the SLO (e.g., 95% for general chatbots, 99% for medical). OpenAI's automatic prompt caching uses a routing hash of ~256 tokens to pin requests to the same cache shard^[7], a complementary approach that avoids the threshold problem entirely for prefix-stable traffic.

GPT Semantic Cache swept 0.6 to 0.9 in 0.05 steps on all-MiniLM-L6-v2 (384 dim) and settled on 0.8: below 0.8, hit rate rose but positive-hit rate collapsed; above 0.8, hit rate fell sharply with little accuracy gain^[1:2].

The static-threshold trap: A single threshold under-fits heterogeneous workloads. Sparse embedding spaces (conversational queries, 10th-NN distance ~0.38) tolerate thresholds around 0.75-0.78. Dense spaces (code, 10th-NN ~0.12) need 0.88-0.90^[5:1]. Production systems need per-category or per-tenant threshold overrides.

Drift detection: Traffic mix shifts weekly. Re-calibrate on a rolling evaluation set. Alarm when production precision drops below the per-tenant SLO. Portkey reports ~99% user-rated accuracy across 250M+ cache requests at high-confidence thresholds (~0.95)^[8].

Every check that must pass before a cache hit is returned. Any failure falls through to the LLM. The threshold check is the critical gate where false positives enter.

Invalidation pipeline#

Three independent signals can mark a cached entry as invalid:

Channel 1: Embedding-model version. Swapping from text-embedding-3-small to text-embedding-3-large changes the vector space entirely. Prior entries' vectors are incomparable with new query embeddings. The cache key must include embedding_model_version. On upgrade, dual-write both old and new embeddings, shadow-query the new index (score but do not serve), then cut over when the new index exceeds the target hit rate^[9].

Channel 2: Source-corpus CDC. When a RAG knowledge-base document changes, a CDC stream (Kafka topic keyed on tenant_id:source_doc_id) fans out to invalidation workers. Workers look up affected entries via the secondary index (source_doc_id -> [entry_id]) and tombstone them. Tombstones (lazy delete on next read) absorb invalidation storms better than synchronous deletes^[5:2].

Channel 3: TTL. Every entry has an expiry. Category-aware TTLs vary from 5 minutes (financial data, 80% change per hour) to 3-9 days (code patterns, 0.01% change per day)^[5:3]. TTL alone is never sufficient for a corpus with edits.

Source-corpus CDC fans out via Kafka to invalidation workers that tombstone affected cache entries. The secondary index enables O(1) fan-out per document change.

TTL jitter is mandatory. Synchronous TTL expiry across many entries creates a thundering herd. Add +/- 15% random jitter to all expiry times^[10].

Per-tenant isolation and multi-tenancy#

Three levels trade cost against blast radius^[11]:

Level 1: Shared index with payload filter. A single HNSW collection stores all tenants. Every query includes a tenant_id filter. Qdrant's payload-indexed filter short-circuits graph traversal, making filtered search competitive with unfiltered^[11:1]. Cost: cheapest. Risk: a filter bug is a cross-tenant data leak.

Level 2: Dedicated index per tenant. Each paying tenant gets their own collection. Enables per-tenant threshold tuning, independent TTL policies, and eliminates cross-tenant filter-bug risk. Cost: ~3x infrastructure^[11:2].

Level 3: Dedicated cluster per tenant. Required for HIPAA, financial, or government workloads. The tenant owns encryption keys. Cost: 10x+, plus operational overhead of version skew across clusters^[11:3].

Production guidance: Start with Level 1 for self-serve. Promote to Level 2 when a tenant's volume justifies it or when they request per-tenant threshold control. Reserve Level 3 for compliance-mandated isolation.

Cache stampede and singleflight#

A viral prompt that misses the cache can trigger hundreds of concurrent LLM calls in the write-back window. Singleflight (canonical Go implementation in golang/groupcache^[10:1]) coalesces N concurrent requests for the same key into one backend call. The remaining N-1 wait and share the result.

For semantic caches, the "key" for singleflight is the prompt hash (exact match) or, more aggressively, the hash of the nearest existing neighbour embedding. Under burst load, this collapses hundreds of redundant LLM calls to one.

TTL-expiry stampede: When a popular entry expires, all concurrent users see the cache as missing simultaneously. Mitigation: stale-while-revalidate (serve the expired entry while one caller refreshes), TTL jitter, and singleflight^[10:2].

Real-World Example#

GPTCache (Zilliz, open source) is the reference implementation for semantic caching. Evaluated on 2,000 test queries across four categories, it achieved 61.6% to 68.8% cache hit rates with 92.5% to 97.3% positive-hit rates at a 0.8 cosine threshold^[1:3].

The architecture is a modular pipeline: pre_embedding_func normalizes the prompt, embedding_func produces the vector (OpenAI API, ONNX, or Cohere), a data_manager handles search and save across a scalar store (SQLite/PostgreSQL) and a vector store (Milvus/FAISS/Qdrant/Redis), a similarity_evaluation component scores candidates, and a post_process_messages_func handles tie-breaking^[12][9:1].

A critical engineering decision: the cache_health_check step compares the embedding stored in the scalar cache against the embedding in the vector store. On mismatch (dual-store drift), it logs critical, forces the similarity score to zero (rejecting the candidate), and self-heals by overwriting the vector store with the cache-store embedding^[9:2]. This pattern acknowledges that any system with two stores will eventually drift.

GPTCache exposes a cache_skip flag that lets callers force-miss based on request temperature: it skips cache entirely on temperature >= 2 and softmax-samples in (0, 2)^[9:3]. This is the gateway-level enforcement of "which calls are cacheable."

Portkey operates semantic caching as production middleware, reporting ~99% user-rated accuracy across 250M+ cache requests^[8:1]. One large food-delivery platform handling tens of millions of AI requests cut LLM spend by over $500K using Portkey's caching, routing, and fallbacks^[13].

Provider-side prefix caching (Anthropic, OpenAI, Gemini) is a complementary but distinct technique. Anthropic's prompt caching charges cache reads (hits and refreshes) at 0.1x the base input price (a 90% discount on reads), while the initial cache write costs 1.25x base input for a 5-minute TTL or 2x for a 1-hour TTL; the cache stores the KV-cache for identical prefixes, with a 5-minute default TTL^[14]. The minimum cacheable prefix is model-dependent: 4,096 tokens for Claude Opus 4.5 and newer Opus models (including Opus 4.7, 4.6, and Haiku 4.5), 2,048 tokens for Sonnet 4.6, and 1,024 tokens for Sonnet 4.5/4/3.7, Sonnet 4, and Opus 4.1/4^[14:1]. One Rails team reported dropping their monthly Claude bill from $42K to under $5K after enabling prefix caching on stable system prompts^[15]. OpenAI's automatic caching activates at 1,024+ token prefixes with up to 90% input cost reduction^[16]. Gemini offers explicit context caching via a dedicated API with configurable TTL^[17]. These are deterministic (zero false-positive risk) but only help when long prefixes are byte-identical across requests. A March 2026 regression silently reduced Anthropic's cache TTL from 1 hour to 5 minutes, causing cost inflation for users paying the 2x 1-hour write premium^[18].

Three-tier caching stack: provider-side prefix cache (L0, zero FP), exact-match hash (L1, zero FP), and semantic HNSW (L2, threshold-gated). Each layer catches traffic the layer above missed.

Trade-offs#

Three axes run through the semantic-cache design: the cache mechanism (exact vs semantic vs provider-native prefix cache), the storage backend, and the similarity threshold tuned to domain risk.

The single biggest meta-decision: threshold vs domain risk. For general chatbots, the accuracy curve is remarkably flat between 0.75 and 0.99 (91.2% to 92.6% on AWS test data)^[2:3], so you should push the threshold as low as your SLO allows. For domain-specific or high-stakes applications, the curve is steeper and you must calibrate per-category.

Scaling and Failure Modes#

At 10x load (100K QPS): Embedding latency dominates. Batch aggressively, co-locate the embed model with the gateway, and prefer smaller models (768 dim) unless evaluation proves you need 1536+. HNSW graph updates serialize; keep write rate under 20% of read rate by sharding by tenant-group^[21].

At 100x load (1M QPS): Single-region HNSW saturates. Deploy region-local caches with independent indexes. Cross-region invalidation replicates via Kafka MirrorMaker. Accept that a new entry written in region A takes seconds to appear in region B. Azure OpenAI's prompt caching documentation recommends keeping static content at the beginning of prompts for maximum cache reuse across regions^[22].

At 1000x load: The index exceeds single-cluster capacity. Tier the architecture: hot entries (high hit_count) stay in-memory HNSW; cold entries move to disk-tiered quantized indexes. The hybrid in-memory approach drops break-even hit rate from 15% to 1%^[5:4].

Failure modes:

• Cache-plane outage: Fail open. All traffic goes to the LLM. Cost spikes but correctness is preserved. Alert on hit-rate drop to zero.
• Embedding service down: L1 exact-match still works. L2 semantic lookups fail open to LLM. Partial degradation, not total failure.
• Invalidation lag: Stale answers served until tombstones propagate. Monitor invalidation backlog; alert if lag exceeds TTL policy.
• False-positive storm: A threshold miscalibration or embedding-model drift causes a spike in wrong answers. Detection: user thumbs-down rate spikes + LLM-judge sampling flags divergence. Mitigation: circuit-break the cache (force all traffic to LLM) and re-calibrate^[13:1].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Threshold selection#

Your semantic cache serves a customer-support chatbot. You have a 5,000-pair evaluation set. At threshold 0.85, hit rate is 72% and precision is 94%. At threshold 0.92, hit rate is 48% and precision is 99%. Your SLO requires >= 97% precision. The LLM costs $100K/month. Calculate the monthly savings at each threshold and recommend which to deploy.

Key Takeaways#

• Semantic caching is not a drop-in for exact-match caching. The similarity threshold is a dial between savings and wrong-answer risk. Measure both sides of the equation.
• The threshold is the highest-leverage knob. Moving from 0.99 to 0.75 can 5x your savings with negligible accuracy loss on general traffic^[2:4], but domain-specific workloads need per-category calibration.
• False positives are invisible without instrumentation. A bad cache hit returns 200 OK with fast latency. Without LLM-judge sampling and user-feedback loops, degradation is silent^[13:5].
• Invalidation needs three channels. Embedding-model version, source-corpus CDC, and TTL. TTL alone is never sufficient for a corpus that gets edited^[5:5].
• Tenant isolation is a compliance decision. Pick namespace / dedicated-index / dedicated-cluster based on blast-radius tolerance, then engineer backwards^[11:5].
• Cache the right calls. Deterministic read-only prompts pay back. Tool-use, agentic loops, and time-sensitive queries must bypass the cache at the gateway level.

Further Reading#

• GPTCache: Semantic Cache for LLM Applications (Zilliz). The reference open-source implementation; read the adapter.py hot path and pluggable similarity_evaluation for the canonical architecture.
• Reducing LLM Costs and Latency via Semantic Embedding Caching (Regmi and Pun, 2024). Reproducible threshold-sweep methodology on an 8,000-pair eval set; the source for hit-rate and positive-hit-rate numbers.
• Category-Aware Semantic Caching for Heterogeneous LLM Workloads (IBM, 2025). The hybrid in-memory HNSW + external storage economics; essential for understanding why break-even hit rate drops from 15% to 1%.
• Anthropic Prompt Caching Documentation. Authoritative on cache_control, 5m vs 1h TTL, pre-warming, and the distinction between deterministic prefix caching and semantic caching.
• Redis 8 Billion-Vector Benchmark. End-to-end HNSW numbers at 1B scale with precision vs latency trade-off curves.
• Qdrant Multitenancy Guide. Payload-based vs collection-based isolation with performance notes for the per-tenant architecture decision.
• Portkey Semantic Caching Thresholds. The AWS 0.99-to-0.75 sweep data and production monitoring patterns for threshold calibration.
• HNSW Paper (Malkov and Yashunin, IEEE TPAMI 2020). The original algorithm; read once before tuning M and ef_construction.

Flashcards#

References#