Design a Fraud Detection System (Stripe Radar / PayPal / Feedzai)

TL;DR. Fraud detection sits in the synchronous critical path of every payment: a verdict must ship in under 100 ms or the authorization times out^[1]. The architecture converges on a three-stage cascade (deterministic rules, gradient-boosted trees, deep model on the ambiguous tail) backed by a unified online/offline feature store that eliminates training-serving skew, an asynchronous graph pipeline for ring detection, and a closed feedback loop where analyst labels and chargebacks retrain the model weekly. Stripe Radar scores over 1,000 signals per transaction and prevents more than $500M in fraud per month on this pattern^[2][3]. The pivotal trade-off is false positives versus false negatives: 33% of consumers never return after a single false decline^[4].

Learning Objectives#

• Size a synchronous scoring service to a less-than-100 ms p99 budget across feature fetch, rules, and ML inference
• Justify a classifier cascade (cheap rules, GBDT, deep model) as a latency and cost control mechanism
• Design an online/offline feature store that guarantees training-serving parity
• Reason about graph-based ring detection as an asynchronous feature producer, not a hot-path component
• Close the feedback loop from analyst labels and chargebacks to shadow-deployed retrained models
• Navigate the tension between GDPR explainability and model-stealing risk

Intuition#

A naive fraud detector is a single if-statement: block the transaction if the amount exceeds $10,000. This works for your first 100 merchants. At 1 million events per second, three things break simultaneously.

First, the base rate is hostile. Only about 1 in 1,000 online payments is fraudulent^[2:1]. Your classifier must find a needle in a haystack without accidentally blocking the haystack. A false negative costs the merchandise value plus a $15 to $40 chargeback fee; a false positive loses the sale and, per Stripe, one-third of those customers never come back^[4:1]. The cost function is asymmetric and merchant-specific.

Second, latency is non-negotiable. The card network expects an authorization response in under 100 ms^[1:1]. You cannot run a 500 ms deep-learning model on every transaction. But you also cannot skip it entirely, because the cheap model misses novel patterns. The insight: a cascade where each stage resolves the majority of traffic and passes only the ambiguous tail to the next, more expensive stage.

Third, fraudsters are an adaptive adversary. They probe your system with small transactions, learn your thresholds, and evolve within days. Your training data is structurally stale because chargebacks arrive 30 to 180 days after the transaction^[5]. The system is not a one-shot classifier but a closed loop of decisioning, labeling, retraining, and safe deployment.

Requirements#

Clarifying Questions#

• Q: Payment fraud only, or also account takeover and promo abuse? Assume: All three. CNP fraud is primary; ATO and promo abuse share the same feature infrastructure.
• Q: Synchronous (block pre-auth) or asynchronous (flag post-hoc)? Assume: Synchronous for payments; asynchronous for lower-stakes flows like ad clicks.
• Q: What false-positive tolerance? Assume: Merchant-configurable. Default block rate around 1%; aggressive merchants accept 0.5%.
• Q: Is graph analysis required? Assume: Yes, but asynchronous. Ring detection writes features back to the online store.
• Q: Regulatory scope? Assume: PCI-DSS for payment data, GDPR for decision explainability, PSD2/SCA for European step-up auth.
• Q: Multi-region? Assume: Yes. Scoring runs in-region; training is centralized.

Functional Requirements#

• Ingest a payment, login, or signup event and return a verdict (approve, challenge, decline) with reasons.
• Enrich events with 100+ features from transaction history, device fingerprint, and user profile.
• Support merchant-editable rules (blocklists, velocity thresholds) with shadow and A/B modes.
• Accept analyst labels and chargebacks; retrain and shadow-deploy new models.
• Provide an audit trail with SHAP-based explanations for every decision.

Non-Functional Requirements#

• Load: 1M events/sec sustained, 3M peak.
• Latency: p99 under 100 ms for the synchronous verdict path.
• Availability: 99.9% (fraud system down means payments down).
• Feature freshness: Online store updated within 1 to 5 seconds of event.
• Recall: 99.9% on known fraud patterns at less than 0.1% false-positive rate on flagged population.

Capacity Estimation#

• Read:write ratio: 1:1 on the scoring path (every event is both a read from the feature store and a write to the event log).
• Cache hit rate: 95%+ on the online feature store; most features are pre-computed.
• Bandwidth: 2 GB/sec ingest to Kafka; 200 MB/sec to the graph pipeline (1% flagged events).

API and Data Model#

API Design#

POST /v1/score
  Body: { "event_type": "payment", "subject_id": "usr_abc", "merchant_id": "m_xyz",
          "amount": 4999, "currency": "usd", "device": {...}, "ip": "..." }
  Returns: 200 { "verdict": "approve", "risk_score": 0.12, "reasons": [],
                  "model_version": "v42", "decision_id": "dec_123" }
  Errors: 429 rate_limited, 503 service_unavailable

POST /v1/feedback
  Body: { "decision_id": "dec_123", "label": "fraud", "source": "chargeback" }
  Returns: 202 accepted

GET /v1/decisions/{decision_id}
  Returns: 200 { "features": {...}, "rules_fired": [...], "shap_values": {...} }

PUT /v1/rules/{rule_id}
  Body: { "condition": "velocity_1h > 10", "action": "decline", "mode": "shadow" }
  Returns: 200 { "rule_id": "r_456", "version": 3 }

Data Model#

-- Event log (Kafka -> Parquet on S3, partitioned by day + merchant)
-- Online feature store (Redis / ScyllaDB, keyed by entity_id)
-- Graph index (Neo4j: nodes = user, device, ip, card, email; edges = shared_usage)

CREATE TABLE decisions (
  decision_id   UUID PRIMARY KEY,
  event_type    TEXT,
  subject_id    TEXT,
  merchant_id   TEXT,
  risk_score    FLOAT,
  verdict       TEXT,         -- approve | challenge | decline
  model_version TEXT,
  features      JSONB,
  rules_fired   TEXT[],
  created_at    TIMESTAMPTZ
);

CREATE TABLE labels (
  decision_id   UUID REFERENCES decisions(decision_id),
  label         TEXT,         -- fraud | legit
  source        TEXT,         -- chargeback | analyst | merchant
  labeled_at    TIMESTAMPTZ
);

Decisions store the full feature snapshot at scoring time; labels arrive asynchronously and join back for retraining.

High-Level Architecture#

The synchronous scoring path returns under 100 ms while asynchronous pipelines update features, detect rings, and retrain models on fresh labels.

Write path. A payment event hits the API gateway, which fetches the pre-computed feature vector from the online store (5 to 10 ms), runs the scoring cascade, and returns the verdict. The event is published to Kafka for downstream processing.

Read path. The analyst console queries the decisions table for flagged events, joining SHAP explanations and graph neighborhood context for triage.

Async path. Flink consumes from Kafka and updates streaming features (velocity counters, device reputation) in the online store within seconds. A batch graph job runs every 10 minutes, detecting rings and writing cluster features back. The training pipeline consumes labeled data and produces new model candidates for shadow deployment.

Deep Dives#

Scoring cascade for sub-100 ms latency#

The core insight: not every transaction needs a deep model. A cascade routes traffic through progressively expensive stages, and each stage resolves the majority of what remains^[2:2][6].

Stage 1: Rules engine (~1 ms). Deterministic checks: blocklists, velocity thresholds (more than 10 transactions in 1 hour from the same card), geo-mismatch (billing country differs from IP country), device fingerprint on a known-bad list, and sanctions screening. Rules resolve 60 to 70% of events with zero ambiguity. They are auditable, editable by ops in minutes, and satisfy regulatory requirements for explainability^[7].

Stage 2: Gradient-boosted trees (~1 to 10 ms). XGBoost or LightGBM operating on hundreds of features. Handles the next 25 to 30% of traffic. Stripe's previous architecture used a Wide (XGBoost) + Deep (DNN) ensemble; dropping XGBoost naively cost 1.5 percentage points of recall, so they replaced it with Shield NeXt, a ResNeXt-inspired multi-branch DNN that preserves the memorization advantage in a single model and cut training time by 85% to under 2 hours^[2:3].

Stage 3: Deep model (~50 ms, on the ambiguous 5%). A Transformer or DNN over event sequences that captures temporal patterns invisible to tabular models. Only the ambiguous tail reaches this stage, keeping p99 under 100 ms while the mean stays below 20 ms^[2:4][6:1].

The fast path resolves the majority of events in under 10 ms; the deep model runs only on the ambiguous tail, keeping p99 under the 100 ms budget.

Why not skip the cascade and run deep on everything? At 1M events/sec, a 50 ms deep model requires 50,000 concurrent inference slots. At the cascade's 5% routing rate, you need only 2,500 slots, a 20x cost reduction with negligible recall loss.

Feature store and training-serving parity#

Training-serving skew is the top silent failure mode in production ML. One widely cited pattern: for example, a model catches 94% of fraud offline but only 71% in production because features are computed differently between the Spark training job and the Flink online pipeline^[8]. One industry estimate attributes approximately 37% of production ML bugs to this class of error, though no primary study has been traced^[9].

The fix: declare each feature once. Stripe built Shepherd on top of Airbnb's open-source Chronon to unify batch and streaming feature definitions^[10]. A single declaration like count_transactions_7d for user_id compiles to both a BigQuery table for training and a Redis read path for online scoring^[11]. Feast's reference architecture ships the same pattern: offline store (BigQuery/Snowflake) and online store (Redis/Datastore) share identical feature view definitions^[11:1].

Point-in-time correctness. The training pipeline must join transactions against feature values as they existed at the time of the transaction, not at query time. Without this, the model sees "future" features during training (label leakage) and collapses in production. Feast's get_historical_features uses the event_timestamp column to enforce this^[11:2].

Streaming aggregations. Features like transaction_count_7d are maintained by Flink using event-time windows with watermarks. Late-arriving events can corrupt counters unless the system uses a configurable late-arrival policy (e.g., update the counter and log the correction)^[12][13].

Online fetch pattern. The scoring service calls the feature store by entity ID and receives a pre-computed vector in sub-10 ms. No feature computation happens on the hot path; all aggregation is pre-materialized^[11:3].

Graph-based ring detection#

Tabular models see one transaction at a time. They cannot detect that four accounts sharing two devices and three overlapping IPs are a coordinated fraud ring. Graph analytics surfaces these structural patterns^[14][15].

Architecture. A graph database (Neo4j, TigerGraph) stores the entity graph: users, cards, devices, IPs, and emails as nodes; shared usage as edges. Batch jobs run every 5 to 10 minutes computing connected components, community detection (Louvain), and degree centrality. Output features (ring_size, ring_prior_chargeback_rate, device_shared_by_N_accounts) are written back to the online feature store for the next synchronous scoring call^[14:1][15:1].

Why not synchronous graph queries? A 3-hop neighborhood lookup in Neo4j can run in hundreds of milliseconds under load. On top of the 50 ms deep-model slow path, the latency budget is gone^[16]. The BRIGHT paper (Lu et al., CIKM 2022) formalizes the solution: a Two-Stage Directed Graph with Lambda Neural Network that decouples offline entity-embedding batch inference from online transaction prediction, cutting end-to-end p99 by over 75% and speeding inference 7.8x versus a naive GNN^[16:1].

Four accounts sharing two devices and overlapping IPs cluster into one anomaly component; cluster features feed the next synchronous score.

PayPal operates a real-time graph database for ATO detection, surfacing patterns like "ABABA" repeated send-backs between two accounts and asset-sharing anomalies that relational schemas would need 4-way self-joins to detect^[17].

Real-World Example#

Stripe Radar: $500M+ fraud prevented per month#

Stripe Radar scores every transaction on the Stripe network in under 100 ms p99, evaluating over 1,000 signals per transaction with a 99.9% correct-verdict rate across billions of legitimate payments^[2:5]. Radar prevents more than $500 million in payment fraud per month^[3:1].

Network advantage. 90% of cards used on the Stripe network have been seen more than once^[18]. This gives Radar cross-merchant aggregate features no single merchant could replicate. A fraud pattern discovered on one Brazilian merchant generalizes to US merchants automatically through learned embeddings^[2:6][18:1].

Shield NeXt architecture. Stripe replaced a Wide (XGBoost) + Deep (DNN) ensemble with Shield NeXt, a ResNeXt-inspired multi-branch DNN. The split-branch structure preserves XGBoost's memorization advantage inside a single model, enabling parallel training, transfer learning, and embedding adoption. Training time dropped from overnight to under 2 hours (85%+ reduction)^[2:7].

Continuous improvement. Retraining the same architecture on fresher data alone adds up to 0.5 percentage points of recall per month^[18:2]. Stripe tripled their model release cadence by investing in tooling for shadow deployment and per-merchant guardrails. Every release is measured not just on aggregate F1 but on per-merchant false-positive, block, and authorization rates^[18:3].

Explainability. Risk insights surface the top contributing features, a location map of billing/shipping/IP distances, and linked-email authorization rates. Elasticsearch powers related-transaction lookups for the analyst console^[2:8].

Trade-offs#

The single biggest meta-decision: where to set the operating point on the precision-recall curve. This is not a technical choice but a business one. A high-margin merchant (digital goods, 80% margin) tolerates more false positives because the cost of a false negative (chargeback + merchandise) is high relative to a lost sale. A low-margin merchant (electronics, 5% margin) cannot afford to block 2% of legitimate customers. The system must expose a per-merchant threshold, not a global one^[18:4].

Scaling and Failure Modes#

• At 10x (10M events/sec): The online feature store (Redis) saturates. Shard by entity_id across a Redis Cluster with 100+ nodes. The scoring cascade scales horizontally; add inference pods behind a load balancer.
• At 100x (100M events/sec): Single-region architecture saturates. Deploy scoring in-region (US, EU, APAC) with a centralized training pipeline. Feature store replicates per-region. Kafka partitions scale to 1,000+.
• At 1000x: The graph database becomes the bottleneck for batch jobs. Partition the graph by geographic region. Move to a streaming GNN architecture (BRIGHT-style Lambda NN) for sub-minute feature freshness^[16:2].

Failure modes:

• Feature store unavailable: The scoring service falls back to a degraded model using only request-level features (amount, currency, IP). Recall drops but latency stays within budget. Alert fires for immediate remediation.
• Model inference timeout: If the deep model exceeds its 50 ms budget, the cascade returns the GBDT score. The 5% of ambiguous transactions get a slightly less accurate verdict rather than a timeout.
• Chargeback pipeline delay: Labels arrive later than 180 days for some dispute types^[5:1]. The model trains on stale data; unsupervised anomaly detection acts as a drift sentinel for novel patterns until labels catch up.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Cascade routing threshold#

Your cascade routes transactions to the deep model when the GBDT score falls between 0.3 and 0.7 (the "ambiguous zone"). A new attack pattern produces scores clustered at 0.28, just below the threshold, and the deep model never sees them. How do you detect and fix this?

Key Takeaways#

• The latency budget is the product budget. A cascade that routes 95% of events to cheap paths keeps p99 under 100 ms without sacrificing model quality.
• Training-serving parity beats model sophistication. A GBDT with consistent features outperforms a Transformer with skew. Define features once; serve from one store^[8:2][10:2].
• Graph analytics belong off the hot path. Compute asynchronously, write results back as features, let the synchronous model benefit without paying the traversal cost^[16:4].
• Feedback loops are a system, not a job. Analyst labels, shadow deploys, canary promotions, and drift alarms are all part of the fraud service.
• Explainability is a regulatory feature with a privacy cost. Bucket reasons externally; reserve raw SHAP for internal analysts.
• The operating point is a business decision. Expose per-merchant thresholds; high-margin merchants tolerate more false positives than low-margin ones.

Further Reading#

• How Stripe Detects Fraudulent Transactions Within 100 ms. End-to-end walkthrough of Radar, Shield NeXt, and the training-serving infrastructure; the most practically useful single read on a production fraud stack.
• A primer on machine learning for fraud detection (Stripe). Stripe's canonical explanation of precision-recall trade-offs, cost of FP versus FN, and score-distribution monitoring.
• BRIGHT: Graph Neural Networks in Real-Time Fraud Detection (CIKM 2022). Two-Stage Directed Graph and Lambda NN architecture for GNN inference inside a sub-100 ms budget, evaluated on e-commerce marketplace data.
• How PayPal Uses Real-time Graph Database to Fight Fraud. Asset-sharing anomaly detection and transaction-pattern mining against account takeover.
• Uber Project RADAR: Intelligent Early Fraud Detection. The explainability-first hybrid-AI argument and why expert rules stay in the stack alongside ML.
• Feast fraud detection tutorial. Runnable reference architecture for online/offline feature parity with code excerpts.
• Shepherd: How Stripe adapted Chronon. First-party source on how Stripe unified batch and streaming feature definitions at scale.
• PSD2 SCA exemptions (Stripe docs). The regulatory thresholds that shape any European fraud engine's challenge logic.

Flashcards#

References#