Blob and Object Storage: Storing the Big Stuff

TL;DR: Object storage is a flat, HTTP-addressable key-value store for opaque blobs. Amazon S3 holds over 500 trillion objects and serves 200+ million requests per second^[1]. It achieves 99.999999999% (eleven nines) durability through Reed-Solomon erasure coding spread across availability zones^[1:1]. Default to S3 Standard for hot data, tier to Glacier Deep Archive (23x cheaper) only after measuring access patterns, and always put a CDN in front of public buckets because egress costs will eat you alive.

Learning Objectives#

After this module, you will be able to:

• Explain the object-storage data model (bucket, key, object, metadata, versions)
• Design upload and download flows using presigned URLs and multipart uploads
• Use storage classes and lifecycle policies to control cost at scale
• Reason about S3 consistency guarantees (strong read-after-write since 2020)
• Decide when to store a blob in object storage vs a database vs a filesystem

Intuition#

Imagine a self-storage warehouse with millions of numbered lockers. You walk in, hand the clerk a labeled box (any size, from a postcard to a refrigerator), and get back a receipt with the locker number. To retrieve it, you show the receipt. There are no hallways, no floors, no nested rooms. Just a flat list of locker numbers. You cannot open a box and change one item inside; you swap the entire box or leave it alone.

Now imagine a record label's master archive. Every song, every take, every mix goes into a vault designed to survive earthquakes, floods, and human error. The vault stores each master on multiple tapes in multiple buildings. If one building burns, the others reconstruct the lost copies automatically. You never worry about capacity; the warehouse expands as you fill it.

Object storage is both of these things: a flat namespace of immutable blobs with planetary-scale durability. It trades POSIX semantics (no partial writes, no directories, no rename) for the ability to scale to exabytes without capacity planning. That trade-off is why every photo you upload, every video you stream, and every ML dataset you train on lives in an object store.

Theory#

Object vs file vs block#

A filesystem organizes data into a tree of directories and inodes. You can open(), seek(), write() arbitrary bytes at arbitrary offsets. A block device gives you raw numbered sectors. Object storage does neither.

An object store has three concepts: a bucket (namespace boundary), a key (UTF-8 string up to 1,024 bytes), and an object (the blob body plus metadata). The slash in photos/2025/cat.jpg is cosmetic; there is no photos/ directory^[2]. The API is intentionally tiny: PUT, GET, DELETE, HEAD, LIST, plus multipart upload variants.

File systems have a tree of directories and inodes; object stores have a flat keyspace where the slash is cosmetic.

Objects are immutable from the client's perspective. To change one byte of a 1 GB video, you PUT the entire 1 GB back (a single S3 object can be up to 48.8 TiB via multipart upload)^[2:1][3]. This constraint is what lets the system scale: no partial-write coordination, no lock managers, no directory rename serialization.

Internals: erasure coding and placement#

How does S3 promise eleven nines of durability without storing three full copies of every object? The answer is Reed-Solomon erasure coding.

Each object is split into k data shards. An encoder generates m parity shards using linear algebra over a finite field. Any k of the k+m shards can reconstruct the original object. Shards are placed across drives, racks, and availability zones so correlated failures never destroy more than m shards simultaneously^[4].

A (10, 4) code stores 14 shards for every 10 shards of real data: 1.4x raw overhead instead of 3x for triple replication, while surviving 4 simultaneous drive failures. Meta's f4 system used a similar erasure coding scheme to reduce its effective replication factor from 3.6x to as low as 2.1x, saving petabytes of disk^[5].

A (10, 4) Reed-Solomon code splits one object into 14 shards spread across racks and AZs; any 10 reconstruct the object, surviving 4 simultaneous failures.

On read, the frontend pulls from the nearest available k shards. On write, it waits for all k+m durable acks before returning HTTP 200. The spread placement means a single customer burst can pull from millions of disks simultaneously because aggregate heat across millions of tenants averages out^[4:1].

Durability vs availability#

These are different numbers. Durability is the probability your object still exists after a year. Availability is the probability you can read it right now.

S3 Standard: 99.999999999% durability (eleven nines), 99.99% availability^[1:2]. Translate eleven nines concretely: store 10 million objects, expect to lose one every 10,000 years. The durability comes from erasure coding across multiple AZs. The availability gap (99.99% = 52 minutes of downtime per year) comes from transient failures in the frontend fleet, metadata index, or network path.

S3 Express One Zone trades multi-AZ durability for single-digit millisecond latency at $0.16/GB-month^[6]. If the one AZ burns, your data is gone. Use it for ephemeral caches and scratch space, never for source-of-truth storage.

Storage classes and tiering#

S3 offers a spectrum from hot to frozen:

That is a 23x price spread from Standard to Deep Archive for the same API^[7].

Lifecycle policies automate transitions. A typical rule: "After 30 days, move to Standard-IA. After 120 days, move to Glacier Flexible. After 7 years, expire." These rules are free to configure and save 5x to 20x on stale data^[8].

A typical hot-to-cold lifecycle: Standard for the first 30 days, IA for the next 90, Glacier Flexible for a year, Deep Archive for legal-hold retention, eventual expire.

Consistency model#

Before December 1, 2020, S3 was eventually consistent for overwrites and deletes. A PUT followed by an immediate GET could return stale data. Analytics tools like Spark needed workarounds (S3Guard, EMRFS consistent view) to avoid reading phantom files.

On December 1, 2020, AWS shipped strong read-after-write consistency for all operations at no extra cost and no latency penalty^[9]. Every PUT now commits to a strongly consistent metadata index. Every GET resolves through the same index before reading shards. List-after-write is also strong.

The catch: strong consistency is per-key, not multi-object transactional. Two concurrent PUTs to the same key race; last-writer-wins. Use conditional writes (If-None-Match: * to fail if the key exists, If-Match: <etag> to fail if someone else changed it) to avoid lost updates^[10].

Access patterns: uploads and downloads#

Multipart upload breaks large objects into 5 MB to 5 GB parts. The client calls CreateMultipartUpload, PUTs each part in parallel, then calls CompleteMultipartUpload with the list of part ETags. AWS recommends multipart for anything over 100 MB^[3:1]. Benefits: parallelism, per-part retry, and resumability after network failures.

Presigned URLs let clients upload or download directly to S3 without proxying bytes through your backend. Your server signs a URL with time-limited HMAC-SHA256 credentials (SigV4, max 7 days). The client uses that URL to PUT bytes straight to the bucket^[11].

The backend signs a URL once; the mobile client uploads bytes directly to S3, bypassing backend bandwidth entirely.

Performance and costs#

S3 auto-scales to 3,500 PUT/sec and 5,500 GET/sec per partitioned prefix^[12]. There is no hard limit on the number of prefixes, so total throughput is effectively unbounded if you distribute keys well. If all your keys start with logs/2025-05-03/, you funnel writes into one partition and hit throttling within minutes.

Cost model:

At scale, egress dominates. Serving 10 TB/month of video directly from S3 costs $900 in egress vs $230 in storage^[7:1]. This is why CDN-origin architectures are standard: S3 to CloudFront is free, and CloudFront-to-internet rates are lower. For truly egress-heavy workloads, Cloudflare R2 ($0.015/GB-month, zero egress) or Backblaze B2 ($6.95/TB-month, approximately $0.00695/GB-month, with 3x free egress) are credible alternatives^[13][14][15].

Real-World Example#

Dropbox Magic Pocket: building an exabyte-scale private object store.

In 2012, Dropbox stored all user files on Amazon S3. By 2014, they were one of S3's largest customers, spending tens of millions annually on storage and egress. The economics did not scale: Dropbox's workload (large immutable files, heavy reads, predictable growth) was a perfect fit for purpose-built infrastructure.

Between February and October 2015, Dropbox migrated over 500 PB of user data from S3 into Magic Pocket, their custom object store^[16]. The system hit 90% cutover on October 7, 2015. Today it is multi-exabyte across three US regions.

Architecture: Files are split into immutable 4 MB blocks addressed by SHA-256 hash. Blocks live in 1 GB buckets, buckets in volumes, volumes in cells (~50 PB raw each). A centralized Master per cell coordinates repair and garbage collection but stays off the data plane. Cross-zone replication happens asynchronously within 1 second of upload^[17].

Key decisions:

• Immutable blocks eliminate concurrent-write coordination entirely.
• Sharded MySQL for the Block Index (leveraging existing operational expertise rather than inventing a new KV store).
• Erasure coding optimized for single-OSD reconstruction (the blog describes a scheme where "reconstruction from a smaller subset of OSDs" is possible under most failure scenarios)^[17:1].
• Per-cell Master caps cell size at ~100 PB rather than running Paxos, accepting the scaling limit for operational simplicity.

Design durability target: 99.9999999999% (twelve nines), one better than S3^[16:1]. The lesson: at sufficient scale, building your own object store is cheaper than renting one. Below ~100 PB, the operational cost of running your own storage fleet almost certainly exceeds S3's bill.

Trade-offs#

Common Pitfalls#

Exercise#

Design Challenge: You are building image storage for a social app with 100 million users. Each user has 10 photos on average (1 billion objects total). Average photo size is 500 KB. Access pattern: 10% of photos are "hot" (accessed daily), 90% are "cold" (accessed less than once per month). Design the storage architecture, including tiering strategy and serving path.

Key Takeaways#

• Object storage is a flat key-value store for blobs, exposed over HTTP. No directories, no partial writes, no POSIX. That simplicity is what lets it scale to exabytes.
• Eleven nines of durability comes from Reed-Solomon erasure coding spread across multiple availability zones, not from triple replication.
• S3 has been strongly consistent (read-after-write for all operations) since December 2020. But consistency is per-key, not multi-object transactional.
• Use presigned URLs for client uploads. Never proxy large files through your backend.
• Multipart upload is mandatory for files over 100 MB. Set an abort-incomplete lifecycle rule on every bucket from day one.
• Lifecycle policies are free to configure and save 5x to 20x on cold data. Default to Standard, tier down after measuring access patterns.
• Egress is the hidden cost. Put a CDN in front of public buckets. Evaluate R2 or B2 if egress exceeds storage cost.

Further Reading#

• Building and Operating a Pretty Big Storage System (Warfield, 2023) - the canonical insider account of S3's scale, heat management, and organizational architecture from an AWS distinguished engineer.
• Inside the Magic Pocket (Dropbox Engineering, 2016) - detailed architecture of an exabyte-scale private object store, from 4 MB blocks to 50 PB cells.
• f4: Facebook's Warm BLOB Storage System (OSDI 2014) - erasure coding for warm-tier blob storage; defines the effective-replication-factor metric that every storage engineer should know.
• Amazon S3 Strong Consistency (AWS Blog, 2020) - the announcement that eliminated eventual consistency from S3; pair with Vogels' deep dive for implementation details.
• S3 Performance Best Practices (AWS Docs) - AWS's own guide to prefix sharding, multipart uploads, and byte-range fetches; the reference for the 3,500/5,500 per-prefix limits.
• Announcing Cloudflare R2 (Cloudflare, 2021) - the zero-egress pitch that reshaped cloud-storage pricing; read to understand when R2 beats S3.
• Colossus Under the Hood (Google Cloud, 2021) - the file system behind GCS, YouTube, and Drive; scales to exabytes per cluster.
• MinIO Erasure Code Quickstart - clear, code-adjacent explanation of Reed-Solomon in a real S3-compatible implementation; useful for on-prem sizing decisions.

Flashcards#

References#