Design a Price Tracking Service (CamelCamelCamel / Honey / Keepa)

TL;DR. A price tracker is a specialized web crawler that repeatedly visits 100M product URLs on a priority schedule, diffs extracted prices against stored history, and fans alerts to 10M subscribers on threshold crossings. The hard problems are not fetching pages but scheduling them politely (Amazon reportedly reprices millions of items per day^[1]), evading anti-bot systems (Cloudflare Bot Management, DataDome), separating real price moves from DOM noise, and absorbing the alert storm when a flash sale crosses 50M thresholds in minutes. The pivotal trade-off: scrape frequency versus politeness compliance versus proxy cost.

Learning Objectives#

• Design a priority-bucketed scrape schedule for 100M URLs under per-retailer politeness budgets
• Justify the choice of residential versus data-center proxies against active anti-bot defenses
• Separate real price changes from extraction-rule drift using canary sets and dual-extractor confirmation
• Plan alert fanout for bursty flash-sale events with per-user rate limiting and idempotency
• Estimate storage for 2-year price history using time-series compression
• Identify legal boundaries (hiQ v. LinkedIn, EU Database Directive) that constrain scraping architecture

Intuition#

A price tracker looks like a trivial CRUD app. Store a URL, fetch it once a day, compare the price, send an email. At 10 users watching 10 products, a cron job and a Postgres table handle it.

At 100 million products across 10 retailers, it collapses. Amazon alone reportedly reprices millions of items per day^[1:1], so a once-daily scrape is already stale for the popular tail. The Amazon Product Advertising API starts every account at an initial limit of 1 request per second^[2] (scaling up to 10 req/sec only as attributed shipped revenue grows), which covers 86,400 URLs per day, three orders of magnitude short of a 100M catalog. The gap between "what the API allows" and "what the tracker needs" is closed by HTML scraping through rotating proxies, which brings anti-bot detection, CAPTCHAs, legal risk, and extraction fragility.

The insight that unlocks the design: the scrape is easy; the schedule is the architecture. Not every product deserves the same frequency. The top 1% (1M URLs with many subscribers) need hourly checks. The middle 80% need daily. The cold tail needs weekly at most. This three-bucket priority model cuts total scrape volume by roughly 10x compared to uniform daily polling, while keeping hot products fresh against Amazon's continuous repricing.

Once you internalize that the scheduler is the core component, the rest follows: per-domain token buckets for politeness, proxy rotation for anti-bot evasion, dual-extractor confirmation for change detection, and partitioned fanout for alert delivery.

Requirements#

Clarifying Questions#

• Q: Single retailer (Amazon-only like CamelCamelCamel) or cross-retailer? Assume: Cross-retailer (Amazon, Walmart, Best Buy, Target, eBay). This forces product matching across catalogs.

• Q: Alert channels? Assume: Email, push, and browser extension overlay. SMS for paid tier only. Downstream delivery via a Notification System.

• Q: Free tier vs. premium? Assume: Free gets daily checks and email alerts. Premium gets hourly checks, push, and SMS.

• Q: How do we handle MAP and fake "was" prices? Assume: Track against our own observed history, not the retailer's claimed reference price.

• Q: Must we render JavaScript? Assume: No. Schema.org JSON-LD covers the common path^[3]. Headless rendering reserved for edge cases.

• Q: Legal posture? Assume: Scrape only public, unauthenticated pages. Honor robots.txt per RFC 9309^[4]. No account creation on target retailers.

Functional Requirements#

• Add a product URL and set an alert threshold (notify when price drops below X)
• Scrape products on a priority schedule (hourly, daily, weekly buckets)
• Detect real price changes and store them in a time-series history
• Trigger alerts on threshold crossings and deliver via email, push, or SMS
• Render a 2-year price-history chart per product
• Match the same product across retailers via GTIN/UPC identifiers^[5]

Non-Functional Requirements#

• Scrape throughput: 100M URLs total; ~1,500 scrapes/sec sustained, 5,000/sec peak
• Alert latency: p99 < 60 seconds for paid tier, < 5 minutes for free tier
• Alert delivery: 99% within SLO
• Freshness: < 1 hour for hot products, < 24 hours for warm, < 7 days for cold
• Storage: 2-year retention at 90-98% compression^[6]
• Availability: 99.9% for API and chart reads; scraper fleet tolerates individual node failure
• Politeness: honor robots.txt; per-domain rate limit of 1 req/sec per IP

Capacity Estimation#

Key ratios: read:write on the API is ~20:1 (chart views dominate). Alert fanout amplification is 10-30x (one price_alert becomes N user notifications across M channels).

API and Data Model#

API Design#

POST /v1/watches
  Body: { "product_url": "https://...", "alert_below": 299.99, "channels": ["email", "push"] }
  Returns: 201 { "watch_id": "w_abc123", "product_id": "p_xyz" }
  Idempotency-Key: <uuid>

GET /v1/products/{id}/history?range=2y&resolution=1d
  Returns: 200 { "product_id": "p_xyz", "points": [{"ts": "...", "price": 329.00}, ...] }

DELETE /v1/watches/{watch_id}
  Returns: 204

GET /v1/products/search?q=Sony+WH-1000XM5
  Returns: 200 { "results": [{"product_id": "p_xyz", "title": "...", "current_price": 299.99}] }

Pagination uses cursor-based tokens. Rate limiting follows the Rate Limiter pattern: 100 req/min free, 1,000 req/min paid.

Data Model#

-- Product catalog (PostgreSQL, sharded by product_id)
table products (
  product_id      uuid PRIMARY KEY,
  canonical_title text,
  gtin            varchar(14),       -- GTIN-13/14 for cross-retailer match
  category        text,
  created_at      timestamp
);

-- Per-retailer listings
table listings (
  listing_id      uuid PRIMARY KEY,
  product_id      uuid REFERENCES products,
  retailer        varchar(32),
  url             text,
  priority_bucket enum('hot','warm','cold'),
  last_scraped_at timestamp,
  rule_version    int
);

-- Price history (TimescaleDB hypertable, partitioned by month)
table price_history (
  listing_id      uuid,
  observed_at     timestamptz,
  price           numeric(10,2),
  currency        char(3),
  availability    varchar(16)
) PARTITION BY RANGE (observed_at);

-- User watches (PostgreSQL)
table watches (
  watch_id        uuid PRIMARY KEY,
  user_id         uuid,
  product_id      uuid,
  alert_below     numeric(10,2),
  channels        text[],
  last_alerted_at timestamp
);
CREATE INDEX idx_watches_product ON watches(product_id);

High-Level Architecture#

End-to-end price tracking pipeline: the scheduler enqueues due listings into a priority queue, scraper workers fetch through rotating proxies, the extractor and diff detector separate real price moves from noise, and Kafka-backed alert fanout delivers notifications to subscribers.

The write path: the scheduler scans listings due for refresh (based on priority_bucket and last_scraped_at), scores them by staleness and subscriber count, and pushes into a Redis sorted set. Workers pop the highest-priority listing, check out a proxy from the pool, fetch the page, extract the price via Schema.org JSON-LD (primary) or per-retailer CSS rules (fallback), and pass the result to the diff detector. If the price changed, it writes to TimescaleDB and checks subscriber thresholds.

The alert path: on a threshold crossing, the diff detector emits a price_alert event to Kafka (partitioned by product_id). The alert service consumes, looks up subscribers from PostgreSQL, applies per-user rate limiting (one alert per watch per hour), and enqueues per-user notifications into the Notification System.

The read path: chart requests hit TimescaleDB with downsampling (raw for 30 days, hourly buckets for 30-365 days, daily buckets for 1-2 years). Product search hits Elasticsearch.

Deep Dives#

Scraping politeness and priority scheduling#

The scheduler is the core component. It must balance three forces: freshness (hot products need hourly checks), politeness (1 req/sec per domain per IP), and cost (residential proxies at ~$5-8/GB versus data-center at ~$1/GB).

Three-bucket priority model:

Priority bucket promotion and demotion: products move between tiers based on subscriber count and engagement, ensuring popular items stay fresh without wasting budget on the long tail.

The hot tier (1M URLs at 24 scrapes/day) consumes 24M scrapes/day. The warm tier (80M at 1/day) consumes 80M. The cold tier (19M at 1/7 days) consumes ~2.7M. Total: ~107M scrapes/day, or ~1,240/sec average.

Per-domain token bucket: each (domain, proxy-IP) pair gets a token bucket at 1 token/sec with a 5-token burst. With a pool of 1,000 IPs, Amazon sees at most 1,000 req/sec from the fleet, spread across distinct source IPs. RFC 9309 compliance means fetching /robots.txt every 24 hours and honoring Disallow rules^[4:1].

Dynamic promotion: a newly viral product (e.g., a TikTok-trending gadget) starts in the warm tier. When subscriber count crosses 100 within an hour, the scheduler promotes it to hot immediately rather than waiting for the nightly recomputation. This prevents a popular product from sitting stale for 23 hours.

Anti-bot evasion and proxy management#

Retailers deploy Cloudflare Bot Management^[7], DataDome^[8], Akamai Bot Manager, and PerimeterX (HUMAN) to classify requests. Cloudflare assigns a bot score from 1 to 99 using ML models trained on the global request stream^[7:1] combined with JA3/JA4 TLS fingerprints^[9]. A low score means automation.

Evasion strategy (layered):

1. Schema.org JSON-LD first. Most retailers embed <script type="application/ld+json"> with canonical price data for Google Shopping^[3:1]. A simple HTTP GET with a realistic User-Agent often suffices because the JSON-LD is in the initial HTML response, no JS rendering needed.

2. Stealth plugins on escalation. When JSON-LD is absent or stale, fall back to headless Chromium with playwright-stealth, which patches navigator.webdriver, the HeadlessChrome UA marker, and WebGL mismatches^[10]. Per ScrapingAnt benchmarks, Playwright stealth achieves ~92% evasion success against basic anti-bot systems^[10:1].

3. Residential proxy fallback. Data-center IPs first (cheap, ~$1/GB). On burn detection (3 consecutive 403/429 from one IP), cool the IP for 1 hour and escalate to residential proxies. Bright Data offers 400M+ monthly residential IPs across 195 countries^[11]; Oxylabs provides 175M+^[12]. Residential is ~5-8x the cost but significantly less likely to be blocked.

Proxy health state machine: each IP cycles between healthy, cooling, and evicted states; subnet-level burn triggers escalation to the residential pool.

Cost model: at 107M scrapes/day averaging 50 KB per page, total bandwidth is ~5 TB/day. At $1/GB data-center, that is $5,000/day. If 10% escalates to residential at $5-8/GB, add $2,500-4,000/day. Total proxy cost: ~$7,500-9,000/day, or ~$225-270K/month. This is why priority scheduling matters: uniform hourly polling of 100M URLs would cost 24x more.

Change detection and extraction-rule drift#

The non-obvious hard problem. A retailer ships a DOM change, the CSS selector returns null, the pipeline converts null to $0, and every subscriber gets a "price dropped to $0!" alert. This is the most common production failure mode for price trackers.

Dual-extractor confirmation: every scrape runs two independent extractors in parallel:

1. Schema.org JSON-LD parser (structured, maintained by the retailer for SEO)^[3:2]
2. Per-retailer CSS/XPath rule (fragile, breaks 1-4 times per year per retailer)

A price change is confirmed only if both extractors agree, or if one returns null and the other shows a change within the historical range (50-200% of last observed price). If both return null or disagree wildly, the scrape is quarantined for manual review rather than alerting users.

Canary set: a representative sample of 100 URLs per retailer is scraped on every extraction-rule deploy and hourly in production. Any rule that returns null, empty, or outside historical bounds flags the rule version for rollback. This catches drift within one scrape cycle rather than after user complaints.

Downsampling for charts: raw price ticks for the last 30 days (query by listing_id + time range). 1-hour buckets for 30-365 days. 1-day buckets for 1-2 years. TimescaleDB's continuous aggregates handle this automatically with 90-98% compression on the columnar chunks^[6:2][13].

Alert fanout under flash-sale load#

A Black Friday or Prime Day event can drop 5M prices inside 5 minutes. At an average of 10 subscribers per watch, that is 50M per-user alert events, each expanding into 1-3 channel deliveries (email + push + SMS). Without backpressure, the notification system saturates and users unsubscribe.

Alert fanout sequence: the diff detector emits an idempotent price_alert per crossing; the alert service applies per-user rate limiting before handing off to the notification system.

Idempotency: the key is (watch_id, crossing_ts) at second-bucket resolution. If the same crossing is detected twice (retry, duplicate scrape), the notification system deduplicates.

Digest mode: during known high-volume windows (Black Friday, Prime Day), the alert service switches to batch digest mode: "5 of your watches just dropped" as a single notification rather than 5 separate ones. This reduces downstream volume by 5-10x.

Backpressure: Kafka partitions by product_id for ordering. The alert service consumer group scales horizontally. If consumer lag exceeds 5 minutes, the system sheds load by dropping free-tier alerts and prioritizing paid-tier delivery.

Real-World Example#

Keepa: 6B+ Amazon products across 11 marketplaces.

Keepa (Keepa GmbH, Kemnath, Germany, founded 2011) is the largest Amazon-focused price tracker, monitoring over 6 billion product listings across 11 Amazon marketplaces (US, UK, Germany, Japan, France, Canada, Italy, Spain, India, Mexico, and Brazil)^[14][15]. The Chrome extension has 4M+ users with a 4.7-star rating^[14:1]. Paid tier costs approximately 19 EUR/month for API access and unlimited tracking^[16].

Keepa's architecture exploits a key simplification: Amazon-only scope means the ASIN is the canonical product identifier, eliminating cross-retailer matching entirely^[15:1]. The system tracks sub-price breakouts (Amazon vs. 3rd-party New vs. Used vs. Warehouse) because arbitrage sellers, its primary paying audience, need that granularity^[14:2].

The browser extension uses contentScripts to inject price-history charts directly into the Amazon product page DOM^[14:3]. This is the same pattern Honey used for coupon injection^[17]. The back end cycles through ASINs at tiered frequencies: hot SKUs (bestsellers, deal-of-the-day candidates) approach continuous scraping; the long tail refreshes daily or slower.

Because Amazon reportedly reprices millions of items per day^[1:2], Keepa's hot-tier cycle must keep pace. The PA-API's default 1 req/sec initial limit^[2:1] covers only 86,400 URLs/day, so the bulk of data acquisition happens through HTML scraping. Amazon's anti-bot escalations periodically reduce Keepa coverage for hours, per seller-forum reports, though no official post-mortem exists.

The cautionary counterexample is Honey (PayPal, $4B acquisition in January 2020^[18][19]). Honey's 17M-user browser extension observed prices client-side from real shoppers, elegantly sidestepping anti-bot^[17:1]. But its monetization via last-click affiliate cookie replacement led to the 2024 class action (In re PayPal Honey Browser Extension Litigation, N.D. Cal. Case 5:24-cv-09470-BLF)^[20][21], Chrome user decline from 20M to 15M in six months^[22], and Google's March 2025 Chrome Web Store policy update restricting affiliate-link manipulation^[23].

Trade-offs#

The biggest meta-decision: scrape frequency versus proxy cost versus detection latency. Keepa resolves it by going Amazon-only (one retailer, one ID scheme, one anti-bot system to master). A cross-retailer tracker like ours accepts higher proxy cost in exchange for broader coverage.

Scaling and Failure Modes#

At 10x (1B products): the priority queue becomes the bottleneck. Redis sorted sets handle ~100K ops/sec per node; at 1B products with frequent score updates, shard across 10+ Redis instances. TimescaleDB partitions grow to ~1.8 TB compressed; add read replicas for chart queries.

At 100x (10B products, Keepa-scale): proxy cost dominates. At 10x current bandwidth, proxy spend exceeds $3M/month. Mitigation: client-side price capture (the Honey model) where browser-extension users contribute prices passively, reducing server-side scrape volume by 50-80%.

At 1000x (continuous global price intelligence): architectural rewrite to streaming. Flink processes price events in real-time from a mix of API feeds, scraper output, and client-contributed data. The scheduler becomes a streaming join between "what we know" and "what is stale."

Failure modes:

• Proxy pool burn (subnet ban). Cloudflare blocks an entire /24 subnet. Detection: rolling 5-minute 403 rate per subnet. Response: evict the subnet, escalate to residential, alert on-call if residential success rate drops below 70%.
• Extraction-rule drift. Retailer ships a DOM change; CSS selector returns null. Detection: canary set flags within one scrape cycle. Response: quarantine affected listings, roll back rule version, suppress alerts until confirmed.
• Kafka consumer lag during flash sale. Alert delivery exceeds SLO. Detection: consumer lag > 5 minutes. Response: shed free-tier alerts, scale consumer group horizontally, switch to digest mode.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Priority bucket sizing#

Your tracker has 50M products. Analytics show: 500K products have > 50 subscribers each, 40M have 1-10 subscribers, and 9.5M have zero subscribers. Design the priority buckets and estimate total daily scrape volume. What is the proxy bandwidth cost at $1/GB data-center assuming 50 KB average page size?

Key Takeaways#

• The scrape is easy; the schedule is the design problem. Priority buckets (hourly/daily/weekly) beat uniform polling by ~10x in cost efficiency while keeping hot products fresh.
• Dual-extractor confirmation prevents $0 alert disasters. Schema.org JSON-LD plus CSS rules, with canary-set drift detection, catches extraction failures before users see them.
• Alert fanout needs per-user rate limiting. Without it, a flash sale becomes a spam incident. Idempotency key (watch_id, crossing_ts) deduplicates retries.
• Proxy cost is the dominant expense. Data-center first with residential fallback on burn optimizes the 5-8x cost differential.
• Legal boundaries shape architecture. Scrape only public unauthenticated pages; honor RFC 9309; avoid account creation that creates ToS privity^[25:1].

Further Reading#

• RFC 9309: Robots Exclusion Protocol. The normative spec for robots.txt; every polite scraper must implement it, and courts increasingly treat violations as evidence of bad faith.
• hiQ Labs v. LinkedIn, 31 F.4th 1180 (9th Cir. 2022). The post-Van Buren reaffirmation that public-data scraping sits outside the CFAA; essential reading for any scraping architecture's legal posture.
• Schema.org Product vocabulary. The structured-data standard retailers maintain for Google Shopping; the lowest-maintenance extraction signal for price trackers.
• Cloudflare Bot Management documentation. Defender-side explanation of ML scoring, JA3/JA4 fingerprinting, and verified-bot allowlists; know your adversary.
• TimescaleDB compression documentation. The 90-98% columnar compression that makes 2-year price history viable on PostgreSQL-compatible infrastructure.
• The Verge: Honey scandal timeline. Curated timeline of the 2024 affiliate-theft scandal with links to court filings; a cautionary tale for monetization design.
• OAG Q&A: Hopper Disrupts the OTA Market. Primary-source numbers on Hopper's 70T data points and prediction architecture; the "predict, don't just track" alternative.
• Amazon Product Advertising API rate limits. The canonical reference for PA-API's 1 req/sec cap and scaling rules; motivates why scraping is necessary.

Flashcards#

References#