Design a Photo Sharing Service (Instagram)

TL;DR. A photo-sharing service at Instagram scale is three decoupled subsystems sharing one photo ID: an upload pipeline (presigned direct-to-S3, resumable multipart), a transcoding pipeline (libvips producing a fixed derivative ladder in 3 formats), and a read pipeline (CDN with origin shield collapsing viral stampedes into a single origin fetch). Instagram serves 3B+ MAU uploading 100M photos/day against a read-write ratio between 100:1 and 1,000:1 ^[1][2][3]. The pivotal trade-off: pay the upload cost once (eagerly transcode all derivatives) so every subsequent view is a cache hit.

Learning Objectives#

After this module, you will be able to:

• Design a resumable upload pipeline that bypasses the API tier for photo bytes using presigned URLs
• Justify pre-generating a fixed derivative ladder over resize-on-demand at scale
• Apply CDN origin shielding to protect object storage from viral-content stampedes
• Estimate storage, compute, and bandwidth for a 100M-uploads/day photo service
• Separate photo storage from feed delivery and articulate the coupling contract
• Choose between Haystack-style custom blob stores and managed object storage based on scale

Intuition#

A photo-sharing service looks like a trivial CRUD app. Accept a JPEG, store it, serve it back. Handles 10 users fine. At 100 million uploads per day and 10 billion reads, it collapses, and the reason is the read-write asymmetry.

Every photo uploaded triggers one expensive write (megabytes over a flaky mobile radio, CPU for transcoding, a mandatory moderation scan) but then gets read hundreds to thousands of times in its first week ^[4]. The naive approach, storing one original and resizing on each request, means every read is a CPU-bound cold computation. At 10 billion reads/day, that is not a photo service; it is a compute cluster pretending to be one.

The insight that unlocks the design: pay the upload cost once, eagerly, to convert an unbounded-scale read problem into a cache lookup. Generate all derivative sizes at upload time. Serve them from a CDN. Make the origin invisible behind a shield that coalesces duplicate requests. Now your read path is a static file fetch, and your write path is a background job that can tolerate seconds of latency without the user noticing.

The second pressure is storage temperature. A photo uploaded today is hot for a week, warm for a month, and cold forever after ^[4:1]. Treating all bytes the same way is the fastest way to bankrupt the service. Facebook's journey from NFS to Haystack to f4 exists because of this single economic fact ^[5][6].

Requirements#

Clarifying Questions#

• Q: Authenticated uploads only, or anonymous? Assume: Authenticated. Anonymous viewing with rate limits; registered users get upload quotas.
• Q: What is the SLA target? Assume: 99.9% upload availability, p99 < 200 ms image load globally, 99.99% read availability.
• Q: Multi-region required? Assume: Yes. Active-passive writes (primary region), active-active reads via CDN.
• Q: Do we store video or only photos? Assume: Photos only in this design. Video is a separate pipeline (see Follow-up Questions).
• Q: What is the maximum photo size? Assume: 20 MB per upload. Larger files rejected at presign time.
• Q: Do we need real-time feed integration? Assume: Yes. After transcoding and moderation, the photo appears in followers' feeds within 5 seconds.

Functional Requirements#

• Upload a photo (JPEG, PNG, HEIC, WebP) up to 20 MB with caption and tags
• View a photo at the appropriate resolution for the requesting device
• Delete a photo (soft-delete with 30-day recovery window)
• Like, comment on, and share photos
• Browse a user's photo grid and a personalized home feed

Non-Functional Requirements#

• Load: 100M uploads/day (~1,200/sec average, 3,600/sec peak); 5B+ reads/day
• Latency: p50 < 50 ms, p99 < 200 ms on image load (CDN edge); p99 < 2s on upload complete
• Availability: 99.99% read path, 99.9% write path
• Consistency: eventual for feed appearance (< 5s); strong for ownership metadata
• Durability: 11 nines on originals; derivatives are regeneratable

Capacity Estimation#

Key ratios:

• Read:write ratio: 50:1 to 1,000:1 depending on photo popularity ^[1:1][8]
• CDN edge hit rate target: 95%+ (derivatives are immutable, cache-friendly)
• Storage growth after tiering: up to 68% savings on originals not accessed for 90+ days via Intelligent-Tiering's Archive Instant Access tier (actual savings vary by access pattern) ^[9]

API and Data Model#

API Design#

POST /v1/uploads/init
  Body: { "content_type": "image/jpeg", "size_bytes": 3145728 }
  Response: 201 { "upload_id": "...", "key": "originals/{user_id}/{uuid}",
                  "presigned_part_urls": ["..."], "expires_in": 3600 }

POST /v1/uploads/{upload_id}/complete
  Idempotency-Key: <client-uuid>
  Body: { "parts": [{"part": 1, "etag": "..."}], "caption": "...", "tags": [...] }
  Response: 200 { "photo_id": "...", "status": "pending" }

GET /v1/photos/{photo_id}
  Accept: image/avif, image/webp, image/jpeg
  Response: 200 { "photo_id": "...", "owner_id": "...", "derivatives": {...},
                  "status": "visible", "like_count": 42 }

DELETE /v1/photos/{photo_id}
  Response: 204 No Content (soft-delete, recoverable for 30 days)

GET /v1/users/{user_id}/photos?cursor=...&limit=50
  Response: 200 { "items": [...], "next_cursor": "..." }

Pagination uses cursor-based keyset pagination on (created_at, photo_id) descending. Rate limiting: 100 uploads/hour per user, 1,000 reads/sec per IP.

Data Model#

Photo metadata is sharded by owner user_id; likes and comments are sharded by photo_id; follows are sharded by follower_id.

Instagram's 64-bit ID scheme encodes 41 bits timestamp + 13 bits shard + 10 bits sequence directly in the photo_id, enabling routing without a directory lookup ^[10].

High-Level Architecture#

The three decoupled subsystems (upload, transcode, read) share state only through S3 buckets and the metadata database.

Write path: The client obtains presigned URLs from the Upload API, streams bytes directly to S3 (bypassing the API tier entirely ^[11]), then signals completion. The API writes a photo_pending row and enqueues transcode + moderation jobs. Only after both pass does the photo become visible.

Read path: The client resolves a CDN URL. On edge hit (95% of requests), bytes return in < 50 ms. On miss, the request escalates to Origin Shield, which coalesces concurrent misses into a single S3 fetch ^[12]. The feed service returns photo IDs and metadata; the client fetches image bytes separately from the CDN.

Origin Shield coalesces N concurrent edge misses into a single S3 fetch, protecting the origin from viral-content stampedes that would exceed per-prefix limits.

Async path: Transcoder workers consume from SQS, download the original, produce 12 derivatives (4 sizes x 3 formats), write them to the derivatives bucket, and emit a photo_visible event to the feed service.

Deep Dives#

Deep dive 1: Storage tiering (Haystack to f4 to S3 Intelligent-Tiering)#

Photos follow a steep access curve: hundreds of reads in the first week, then roughly one order of magnitude drop per age band ^[4:2]. Yet only ~25% are deleted within a year ^[5:1], so storage grows monotonically.

The NFS problem (pre-2009). Facebook's original photo tier used NFS on commodity servers. Reading one photo required up to 3 disk I/Os just to walk directory metadata ^[13][14]. At 550,000 images/sec peak, the filesystem was the bottleneck.

Haystack (2010). A log-structured append-only store on XFS. Each storage node runs ~100 "physical volumes" (preallocated 100 GB files). An in-memory index maps photo_id to a byte offset, reducing reads to a single disk I/O ^[5:2]. Per the OSDI 2010 paper, Facebook stored 260 billion images (20 PB) this way, serving 1 million images/sec at peak ^[5:3].

f4 (2014). After 90 days in Haystack, blobs migrate to f4's warm tier. f4 uses Reed-Solomon (10,4) erasure coding within a datacenter and optionally XOR coding across datacenters, reducing the effective replication factor from 3.6x in Haystack to either 2.8x (cross-datacenter XOR) or 2.1x (single-datacenter Reed-Solomon only). As of the 2014 paper, f4 stored over 65 PB of logical BLOBs at these reduced replication factors ^[4:3][6:1].

For cloud-native services (Instagram on AWS/Meta infra): Use separate buckets with different lifecycles. Originals transition to S3 Intelligent-Tiering (automatic hot/warm/cold movement) after 30 days. Derivatives stay in S3 Standard, regeneratable from originals if expired ^[9:1].

Photos flow through temperature tiers; derivatives are regeneratable from originals and can be aggressively expired to save cost.

Deep dive 2: Thumbnail pipeline and format negotiation#

At 100M uploads/day producing 12 derivatives each, the transcoding pipeline processes 1.2 billion images daily. Throughput is the constraint.

Why libvips? It is a demand-driven, horizontally threaded library that benchmarks at roughly 7-8x higher throughput than ImageMagick on resize workloads on multi-core hardware (libvips 0.57s vs ImageMagick 4.44s on a 16-core benchmark), and still around 5-6x faster when libvips is restricted to a single worker thread ^[15][16]. At scale, that throughput difference is a directly visible line on the AWS bill. libvips powers Mastodon, sharp (Node.js), imgproxy, Rails Active Storage, and MediaWiki ^[16:1].

The derivative ladder:

AVIF compresses 20-30% smaller than WebP at equal visual quality ^[17]. The client negotiates format via the Accept header; the CDN varies the cache key on Accept.

EXIF GPS stripping is mandatory. Mobile phones embed GPS coordinates in every photo. If derivatives preserve this metadata, any viewer can extract the photographer's exact location. Strip all EXIF at the transcoder stage, preserving only orientation for correct display ^[16:2]. This is a privacy incident, not a feature request.

Two-phase publish: The feed never shows pending photos. A moderation worker processes the photo asynchronously (AWS Rekognition or equivalent). On pass, flip to visible and emit the feed event. On fail, flip to blocked and notify the user. This decouples upload latency from moderation latency.

Deep dive 3: Upload path with direct-to-S3 presigned URLs#

The upload API never touches photo bytes. This is the single most important bandwidth decision.

Why presigned URLs? AWS explicitly recommends this pattern: "By directly uploading these files to Amazon S3, you can avoid proxying these requests through your application server. This can significantly reduce network traffic and server CPU usage" ^[11:1]. At 100M uploads/day averaging 3 MB each, that is 300 TB/day of bandwidth the API tier never sees.

Resumable multipart upload: S3 supports parts from 5 MiB to 5 GiB (up to 10,000 parts), with a maximum object size of 48.8 TiB ^[18]. For a typical 3 MB photo, a single-part presigned PUT suffices. For larger files or unreliable networks, multipart upload with 5 MiB chunks allows resume on failure.

Multipart upload with resume: the client streams directly to S3, the API only orchestrates metadata and job dispatch.

Garbage collection: Clients that start but never finish leave orphaned parts. S3's AbortIncompleteMultipartUpload lifecycle rule cleans these after 7 days ^[19].

Idempotency: The client generates a UUID per upload attempt. If it retries the complete call after a timeout, the API detects the duplicate via the idempotency key and returns the existing photo_id.

Real-World Example#

Instagram: from Django on EC2 to three data centers serving 40 billion photos.

Instagram launched in 2010 as a pure Django + EC2 stack: PostgreSQL for metadata, Redis for feeds and sessions, Memcached for caching, S3 for photo bytes, and CloudFront as the CDN ^[20]. Photos went directly from the client to S3 via presigned URLs from day one. The Django app never handled image bytes.

By early 2012, Instagram had 30 million users and was sharding PostgreSQL by user_id using the 41+13+10 bit ID scheme, implemented as a PL/pgSQL function on each shard ^[10:1]. This avoided any central ID service (unlike Flickr's ticket-server approach ^[21]).

After the 2012 Facebook acquisition, Instagram migrated from AWS to Facebook data centers, swapped EC2 for Tupperware containers, and plugged into TAO, Scuba, and Haystack ^[2:2]. By 2015, they served over 1 million requests per second across three data centers with 40 billion photos stored ^[22].

Cross-region consistency used PgQ to invalidate Memcached entries, and a memcache lease mechanism collapsed thundering-herd cache misses on hot counters ^[1:2][2:3]. Instagram accepts a 60 ms cross-region latency penalty on writes ^[22:1].

The key architectural insight: the photo service owns bytes and metadata. The feed service owns timelines (see Design a Social Media Feed). Photo IDs flow through the feed; photo bytes never do. This decoupling is why Instagram could later slot Reels (a video service) into the same feed without rearchitecting ^[1:3].

Trade-offs#

The biggest meta-decision: managed object storage (S3) vs. custom blob store (Haystack). For any team that does not operate its own data centers, S3 with Intelligent-Tiering replaces the entire Haystack/f4 stack. The per-request cost at extreme scale is real, but the operational cost of running a custom storage system is higher for 99% of organizations.

Scaling and Failure Modes#

At 10x load (1B uploads/day): The transcoder pool saturates. Mitigation: auto-scale transcoder workers horizontally (stateless, queue-driven). S3 prefix limits become real; spread keys across more hash prefixes.

At 100x load (10B uploads/day): S3 request costs dominate the bill. CDN origin traffic exceeds Origin Shield capacity in a single region. Mitigation: multi-region origin shields, regional derivatives buckets with cross-region replication for hot content only.

At 1,000x load: The architecture shifts to CDN-first. Derivatives are pre-pushed to edge POPs for high-follower accounts. S3 becomes a cold origin only. Consider a Haystack-style custom store for warm data to eliminate per-request pricing.

Failure mode: Regional S3 outage. Reads degrade gracefully (CDN serves stale cached derivatives). Uploads fail in the affected region. Mitigation: multi-region upload endpoints with DNS failover; originals replicate cross-region within 15 minutes via S3 Cross-Region Replication.

Failure mode: Transcoder queue backup. Photos stay in pending state. Users see "processing" for minutes instead of seconds. Mitigation: dead-letter queue with alerting; auto-scale workers on queue depth; degrade gracefully by serving the original at reduced quality while derivatives generate.

Failure mode: Viral photo exceeds per-prefix S3 limits. S3 returns 503 Slow Down ^[23][24]. Mitigation: Origin Shield coalescing, hash-based prefix spreading, and pre-warming the shield cache for high-follower uploads.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Format migration backfill#

Your photo service has been running for two years with a 4-size JPEG-only derivative ladder. Product wants to add WebP and AVIF support for bandwidth savings. You have 40 billion stored photos. Design the migration strategy.

Key Takeaways#

• Three decoupled subsystems: Upload, transcode, and read scale independently. Conflating them is the fastest way to sound junior in an interview.
• Presigned direct-to-S3: The API never touches photo bytes. At 300 TB/day of uploads, this is a non-negotiable architectural decision ^[11:3].
• Pre-generate, do not resize on demand: A fixed derivative ladder (4 sizes x 3 formats) converts the read problem into a cache lookup. libvips benchmarks roughly 7-8x faster than ImageMagick on multi-core hardware ^[15:2].
• Origin Shield is the key CDN optimization: Request coalescing collapses viral stampedes into a single origin fetch, protecting S3 from exceeding per-prefix limits ^[12:2].
• Storage tiering is economic survival: Separate originals (archival) from derivatives (regeneratable). Tier aggressively. Facebook's f4 cut warm-BLOB effective replication from 3.6x to 2.1x using erasure coding, across a 65+ PB corpus ^[4:4].

Further Reading#

• Sharding and IDs at Instagram. The canonical blog post on the 41+13+10 bit ID scheme; explains why encoding the shard in the ID eliminates a directory lookup.
• Instagration Pt 2: Scaling to Multiple Data Centers. How Instagram handled cross-region Memcached invalidation via PgQ and scaled to 40B photos across three data centers.
• Finding a Needle in Haystack (OSDI 2010). Facebook's log-structured photo store that reduced per-read I/O from 3 disk seeks to 1.
• f4: Facebook's Warm BLOB Storage System (OSDI 2014). Erasure coding for warm photo data, reducing effective replication from 3.6x to 2.8x or 2.1x across a 65+ PB corpus.
• CloudFront Origin Shield documentation. The AWS feature that protects your origin from viral-content stampedes via request coalescing.
• AWS S3 performance guidelines. The per-prefix 3,500/5,500 RPS limits that drive origin-shield and prefix-spreading design decisions.
• libvips Speed and Memory Use benchmarks. The numbers behind the ~7-8x multi-core throughput advantage over ImageMagick.
• Flickr: Ticket Servers. The REPLACE INTO ID trick that Instagram explicitly considered and rejected in favor of shard-embedded IDs.

Flashcards#

References#