Design a Pastebin (Paste Sharing Service)

TL;DR. A pastebin looks like a URL shortener with a payload attached, but that payload (10 KB average, 1 MB at p99, 10 MB hard cap) changes everything. DynamoDB hard-caps items at 400 KB^[1], so blobs must live in object storage while metadata stays in a KV table. Expiration is a three-layer pipeline (S3 lifecycle rules, DynamoDB TTL, and a reconciler) because S3 lifecycle deletions can take a day or more^[2] and DynamoDB TTL deletes within a few days^[3]. Syntax highlighting belongs on the render path with caching. Cloudflare R2's zero-egress pricing makes it the cost-optimal blob store for a 10:1 read-heavy workload^[4].

Learning Objectives#

After this module, you will be able to:

• Design a blob/metadata storage split that handles payloads from 1 KB to 10 MB without hitting DynamoDB's 400 KB item limit
• Justify S3 lifecycle rules plus DynamoDB TTL as the expiration mechanism and explain the asynchronous-deletion trade-off
• Compare write-time, render-time, and client-side syntax highlighting with concrete cost and latency reasoning
• Estimate storage, compute, and egress costs for a 10M-paste/day service
• Implement abuse-prevention scanning as an async pipeline that does not block writes
• Trade off zero-knowledge encryption (PrivateBin model) against server-side scanning capability

Intuition#

A pastebin looks trivial. Accept text, return a URL, serve it back. A single Postgres table handles this for 100 users.

At 10 million pastes per day it collapses, and the reason is payload size. A URL shortener stores a 500-byte row. A pastebin stores blobs that run 10 KB on average, 1 MB at p99, and up to 10 MB for PRO users^[5]. That three-to-five orders of magnitude increase in record size means you cannot fit a paste in a DynamoDB item (400 KB cap^[1:1]), cannot afford to scan billions of rows nightly for expiration, and cannot run a CPU-intensive syntax highlighter synchronously on every write without burning 12 continuous cores.

The one insight that unlocks the design: split the system into a cheap, scalable blob store (S3/R2) for payloads and a fast, indexed metadata store (DynamoDB) for lookups. Then treat expiration as a data pipeline, not a database feature. Every other decision flows from this split.

Requirements#

Clarifying Questions#

• Q: What is the maximum paste size? Assume: 10 KB average, 1 MB at p99, 10 MB hard cap (PRO tier). Standard accounts cap at 512 KB^[5:1].

• Q: What TTL options do users get? Assume: 10 min, 1 hour, 1 day, 7 days (default), 30 days, never.

• Q: Authenticated users only, or anonymous? Assume: Both. Guests get 10 pastes per 24 hours; free accounts get 20; PRO gets 250^[5:2].

• Q: What visibility tiers exist? Assume: Public (indexed, searchable), unlisted (accessible only by URL), private (requires auth token).

• Q: Do we need syntax highlighting? Assume: Yes, server-rendered with caching. Support 100+ languages.

• Q: Multi-region required? Assume: Single-region at launch; design for multi-region expansion.

Functional Requirements#

• Create a paste (text content, language hint, TTL, visibility) and return a short URL.
• Retrieve a paste by ID with optional syntax-highlighted rendering.
• Delete a paste (owner or admin).
• List a user's pastes with cursor-based pagination.
• Auto-expire pastes after their TTL fires.

Non-Functional Requirements#

• Load: 10M writes/day (~116 avg QPS, ~1,000 peak); 100M reads/day (~1,160 avg QPS, ~10,000 peak).
• Latency: p99 < 200 ms on read; p99 < 500 ms on write.
• Availability: 99.99% read path, 99.9% write path.
• Consistency: eventual for read-after-write (CDN cache lag acceptable); strong for delete.
• Durability: 11 nines (S3 standard).

Capacity Estimation#

• Read:write ratio: 10:1. The CDN absorbs 95%+ of reads on popular pastes, so origin sees ~500 read QPS at steady state.
• Egress dominates cost. At 100M reads/day x 10 KB average = 1 TB/day egress. On S3 at $0.09/GB (US East), that is $90/day. On R2, it is $0. This single number justifies R2^[4:2].
• Metadata row size: ~200 bytes. At 10M rows/day, DynamoDB on-demand handles this without capacity planning.

API and Data Model#

API Design#

POST   /v1/pastes
       Idempotency-Key: <uuid>
       Body: { "content": "...", "language": "python", "ttl": "7d",
               "visibility": "unlisted", "password": null }
       Returns: 201 { "paste_id": "abc123xyz0", "url": "https://pb.io/abc123xyz0",
                       "expires_at": "2026-05-11T00:00:00Z" }
       Errors: 413 payload too large, 429 rate limited

GET    /v1/pastes/{paste_id}
       Query: ?format=raw|highlighted&theme=monokai
       Returns: 200 { "content": "...", "language": "python", "created_at": "...",
                       "expires_at": "...", "view_count": 42 }
       Errors: 404 not found or expired, 403 private paste

GET    /v1/pastes/{paste_id}/raw
       Returns: 200 text/plain (streaming for pastes > 1 MB)

DELETE /v1/pastes/{paste_id}
       Returns: 204 No Content

GET    /v1/users/{user_id}/pastes?cursor=...&limit=20
       Returns: 200 { "pastes": [...], "next_cursor": "..." }

Rate limiting: per-IP token bucket at the edge (Cloudflare WAF) plus per-account counters in Redis. Guests: 10/24h. Free: 20/24h. PRO: 250/24h^[5:3].

Data Model#

Metadata table (DynamoDB):

table paste_metadata (
  paste_id        String    -- PK, random 10-char base62
  owner_id        String    -- nullable for guest pastes
  created_at      Number    -- epoch seconds
  expires_at      Number    -- DynamoDB TTL attribute
  blob_key        String    -- S3 object key: "7d/abc123xyz0"
  content_hash    String    -- SHA-256 for dedup + integrity
  size_bytes      Number    -- for streaming decisions
  language_hint   String    -- lexer name
  visibility      String    -- public | unlisted | private
  scan_status     String    -- pending | clean | flagged
)

GSI: owner_id -> [paste_id, created_at]  (user's paste list)

Blob store (S3/R2): Objects keyed by <ttl-bucket>/<paste_id>. The TTL-bucket prefix (10m/, 1h/, 1d/, 7d/, 30d/, never/) enables S3 lifecycle rules to expire entire prefix cohorts^[7].

Each paste has one metadata row, one raw blob, and zero-or-one cached highlighted HTML blob.

High-Level Architecture#

The write path stores a blob then metadata and emits an event; the read path hits the CDN first, falls through to Redis, then DynamoDB and S3 on cache miss. Background workers handle highlighting and abuse scanning asynchronously.

Write path: Client POSTs content. The Paste Service generates a random 10-char base62 ID, PUTs the blob to S3 under the appropriate TTL prefix, PUTs metadata to DynamoDB, emits a PasteCreated event to SQS, and returns the URL. Total latency: ~100-200 ms (S3 PUT ~40 ms + DynamoDB PutItem ~10 ms + overhead).

Read path: The CDN serves cached public/unlisted pastes. On miss, the service checks Redis for metadata (5 ms), falls through to DynamoDB on Redis miss, validates expires_at < now(), fetches the blob from S3 via streaming, and returns it with Cache-Control: public, max-age=<remaining_ttl>.

Async path: The Highlight Worker renders syntax-highlighted HTML and stores it as a second blob. The Abuse Scanner checks for credential patterns (AWS keys with prefix AKIA^[8], Stripe sk_live_, SSH private keys). Matches set scan_status: flagged on the metadata row.

Deep Dives#

Deep dive 1: Storage split (metadata in DynamoDB, blob in S3)#

The fundamental constraint: DynamoDB hard-caps each item at 400 KB total (attribute names plus values)^[1:2]. A 1 MB paste physically cannot fit. Even a 512 KB paste (the standard-tier cap on pastebin.com^[5:4]) exceeds the limit. AWS's own Database Blog explicitly recommends storing large objects in S3 and keeping a pointer in DynamoDB^[9].

Write ordering matters. Write the blob first, then the metadata. If the blob PUT succeeds but the metadata PUT fails, you have an orphan blob that costs fractions of a cent and gets cleaned by the reconciler. If you wrote metadata first and the blob PUT failed, you would have a row pointing at nothing, causing user-visible 404s.

Deduplication via content hash. The content_hash (SHA-256) enables optional dedup: two identical pastes can share a blob. This saves storage but complicates TTL (the blob must outlive the longest-lived metadata row). Skip dedup at launch; revisit if storage costs grow.

Streaming large reads. A 10 MB paste must not be buffered entirely in memory. Use S3's streaming body piped directly to the HTTP response. At 100 concurrent reads of 10 MB pastes, buffering would consume 1 GB of process RSS. Streaming keeps memory constant.

Write the blob before metadata so a failure between steps leaves an orphan blob (cheap, reconcilable) rather than a dangling pointer (user-visible error).

Deep dive 2: TTL and the expiration pipeline#

Expiration is a data pipeline problem, not a database feature. Three layers work in parallel:

Layer 1: S3 lifecycle rules. Objects are written with a TTL-bucket prefix (10m/, 1h/, 1d/, 7d/, 30d/). A lifecycle rule on each prefix expires objects after the corresponding duration. This handles 90%+ of blob cleanup with zero custom code^[7:1]. The catch: S3 lifecycle expiration is asynchronous. AWS states that "Amazon S3 queues the object for removal and removes it asynchronously"^[7:2], with delays of a day or more^[2:1].

Layer 2: DynamoDB TTL. The expires_at attribute is the table's TTL attribute. DynamoDB auto-deletes expired items, but "automatically deletes expired items within a few days of their expiration time"^[3:1]. These deletions consume no write capacity^[10].

Layer 3: Reconciler. A daily background job handles orphans. It lists metadata rows whose blob HEAD returns 404 and deletes them. It also lists blobs older than a threshold whose paste_id has no metadata row and deletes those.

The critical insight: The service layer checks expires_at < now() on every read and returns 404 immediately, even if the blob and row still physically exist. Observable behavior is correct from the moment of expiry; physical deletion catches up within days.

Why not a synchronous cron? dpaste runs a cleanup_snippets Django management command on a cron^[11]. At dpaste's modest volume this works. At 10M pastes/day, you accumulate 3.65B pastes per year. A nightly scan of every row is untenable. The lifecycle + TTL + reconciler pattern scales because each layer handles its own domain without a full table scan.

TTL is three parallel pipelines: S3 lifecycle for blobs, DynamoDB TTL for rows, and a reconciler for orphans. The service layer returns 404 immediately on expiry without waiting for physical deletion.

Deep dive 3: Syntax highlighting placement#

Three options exist. Each has a clear best-fit scenario.

Option A: Write-time (server-side). Run Pygments (550+ lexers^[12]) on paste creation, store the HTML blob alongside the raw blob. dpaste uses this approach^[13]. Pro: reads serve pre-rendered HTML directly, CDN-friendly. Con: every paste pays CPU cost. At 10M pastes/day and ~100 ms per render, that is ~12 continuous CPU cores just for highlighting. Most pastes are never read again after the author checks the URL.

Option B: Render-time with cache (our pick). On first read requesting highlighted output, render the HTML, cache it in S3 as a second blob (highlighted/<paste_id>/<theme>), and serve the cached copy on subsequent reads. Pro: pastes never read cost zero CPU. Con: first reader pays a latency hit (50-200 ms for 10 KB; up to 2-5 seconds for 1 MB). Mitigation: the SQS event triggers a Highlight Worker that pre-renders asynchronously. If the worker finishes before the first read, the highlighted blob is already cached.

Option C: Client-side. Ship highlight.js (192 languages, 512 themes^[14]) to the browser. PrivateBin uses this because its zero-knowledge model means the server cannot see plaintext^[15]. Pro: zero server CPU, privacy-preserving. Con: depends on JavaScript, poor for SEO and accessibility.

We pick Option B for the general case. The event bus emits PasteCreated; the Highlight Worker renders HTML via Pygments and stores it in S3. The first read either hits the pre-rendered blob (if the worker finished) or falls back to plain text with a client-side highlighter as a progressive enhancement.

Real-World Example#

PrivateBin: zero-knowledge paste architecture#

PrivateBin is an open-source paste service (~8,300 GitHub stars^[16]) with a radically different trust model: the server never sees plaintext^[15:1]. The architecture demonstrates how a single design constraint (zero-knowledge) cascades through every component.

Encryption model: The browser generates a 256-bit AES key, encrypts the paste with AES-256-GCM via the Web Crypto API, and appends the key to the URL as a fragment identifier (the #key portion)^[15:2][17]. Because URL fragments are never sent in HTTP requests, even an HTTPS-terminating proxy or access log cannot leak the key. The server stores only opaque ciphertext.

Storage: PrivateBin's S3Storage backend stores each paste as a single S3 object keyed by <prefix>/<paste_id>, with the encrypted payload as JSON and selected metadata replicated into S3 object metadata for cheap HEAD lookups^[18]. Comments are stored under <prefix>/<paste_id>/discussion/<parent_id>/<comment_id>.

Expiration: Rather than relying on S3 lifecycle rules, PrivateBin implements a custom _getExpiredPastes method that lists all objects, reads their expire_date metadata via headObject, and batches expired keys for deletion^[18:1]. At scale this becomes a bottleneck (listing millions of objects), which is exactly why our recommended architecture uses TTL-bucket prefixes with lifecycle rules instead.

Cascading trade-offs from zero-knowledge:

• No server-side syntax highlighting (server cannot see plaintext). Client-side prettify.js is the only option^[15:3].
• No server-side search or indexing.
• No server-side abuse scanning. A compromised paste is invisible to the operator.
• A compromised server can inject malicious JavaScript that exfiltrates the decryption key. This is the residual trust model^[15:4].

PrivateBin proves that zero-knowledge is viable for privacy-focused deployments but incompatible with the abuse-scanning and server-side-highlighting requirements of a public service. For a general-purpose pastebin, store plaintext server-side and invest in the scanning pipeline.

Trade-offs#

The biggest meta-decision is the trust model. Zero-knowledge (PrivateBin) gives operators plausible deniability and users real privacy, but it eliminates server-side scanning, highlighting, and search. For a public service that must handle abuse reports and credential leaks, server-side plaintext is the pragmatic choice. For a privacy-focused deployment (healthcare, legal, whistleblower), zero-knowledge wins.

Scaling and Failure Modes#

At 10x (100M pastes/day, 1B reads/day):

• Write QPS hits ~10,000/sec peak. S3 handles 3,500 PUTs/sec per prefix; our 5+ TTL prefixes distribute writes. No bottleneck.
• DynamoDB on-demand auto-scales.
• The CDN absorbs read amplification. Origin sees <5% of reads.
• Abuse scanner queue depth grows. Add horizontal workers.

At 100x (1B pastes/day):

• Storage grows to ~1 TB/day ingest. Annual "never" pastes reach 100+ TB. Tiered storage (S3 Intelligent-Tiering) becomes necessary.
• The reconciler must shard by prefix to avoid listing billions of objects.
• Abuse scanning needs ML-based classification rather than regex matching.
• Move to multi-region: DynamoDB Global Tables for metadata, S3 Cross-Region Replication for blobs.

Failure modes:

• S3 regional outage: Reads fail for cache-miss pastes. Mitigation: CDN serves stale content for public pastes (set stale-while-revalidate). Writes fail entirely; return 503 with retry-after header.
• DynamoDB throttling on hot partition: A viral paste causes a hot key. Mitigation: Redis absorbs repeated metadata reads. DynamoDB adaptive capacity shifts throughput to hot partitions automatically.
• Orphan accumulation after reconciler failure: Blobs pile up without metadata rows, inflating storage cost. Detection: CloudWatch alarm on S3 object count vs DynamoDB item count divergence. Recovery: manual reconciler run.

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Scanner backpressure under attack#

Your abuse scanner takes 500 ms on a 10 MB paste. An attacker floods the service with 1,000 large pastes per minute from rotating IPs. The scanner queue backs up to 50,000 messages. Design the system behavior: what happens to new pastes, how do you protect legitimate users, and what is the failure mode if the scanner goes down entirely?

Key Takeaways#

• Split storage by nature: Blobs in object storage (S3/R2), metadata in DynamoDB. The 400 KB item limit is not negotiable.
• Write blob first, metadata second. Orphan blobs are cheap and reconcilable; dangling pointers cause user-visible errors.
• TTL is a pipeline, not a cron. S3 lifecycle + DynamoDB TTL + reconciler scales to billions without full table scans.
• Render-time highlighting wins. Most pastes are never read; write-time highlighting wastes CPU at scale.
• Random IDs are a security requirement. Sequential IDs enable enumeration of unlisted pastes.
• Zero-egress object storage changes the cost model. R2's free egress makes it optimal for read-heavy blob workloads.

Further Reading#

• AWS S3 Lifecycle Configuration. The canonical reference for TTL-based object expiration; read the "expiring objects" page for the asynchronous-delete caveat.
• AWS DynamoDB TTL. Explains the "within a few days" deletion window, epoch-seconds format, and when TTL processes ignore items.
• Large Object Storage Strategies for DynamoDB. The official AWS stance on S3 + DynamoDB for oversized items; the pattern this chapter is built on.
• Cloudflare R2 Pricing. Zero-egress pricing details; critical for cost comparisons on read-heavy workloads.
• PrivateBin. Production-quality zero-knowledge paste service; read lib/Data/S3Storage.php for the S3 integration pattern.
• dpaste Source Code. Smaller-scale reference with a Pygments pipeline, Django ORM metadata, and a cleanup_snippets cron for TTL.
• GitHub Secret Scanning Patterns. The gold-standard list of credential regex patterns (AWS, Stripe, GitHub PAT, and 500+ others); the right reference for building an abuse scanner.
• Syntax Highlighting on the Web. Thoughtful comparison of regex-based grammars (Pygments, highlight.js) vs tree-sitter for incremental rendering.

Flashcards#

References#