Design a Coding Agent (Claude Code / GitHub Copilot / Cursor)

TL;DR. A coding agent is three products sharing one backend: sub-200 ms autocomplete on a small model, conversational chat grounded in the repo, and an autonomous ReAct loop that edits files and runs shell commands for minutes per task. GitHub Copilot handles 400M+ completion requests per day at ~8,000 QPS peak^[1], Cursor crossed $2B ARR by early 2026^[2], and agent benchmarks on real GitHub issues have climbed rapidly: Devin scored 13.86% on the original SWE-bench in March 2024^[3], and Claude Opus 4.7 now leads SWE-bench Verified at 87.6% (April 2026)^[4]. The pivotal trade-off is model tiering: a cheap small model for autocomplete, a frontier model for chat, and frontier-plus-tools for the autonomous loop, yielding roughly an order-of-magnitude cost reduction over a single-model deployment based on published model pricing differences^[5].

Learning Objectives#

• Design a three-mode coding agent (autocomplete, chat, autonomous loop) with distinct latency and cost budgets for each mode
• Architect a hybrid repo-context retrieval layer combining embeddings, tree-sitter symbol graphs, and git-awareness
• Justify per-session Firecracker microVM isolation for untrusted agent tool calls and size the warm pool
• Estimate capacity for 1M concurrent sessions across three modes with different token economics
• Stream generated diffs to the editor with atomic apply and one-key rollback
• Build an eval harness on SWE-bench Verified that gates every model rollout before production

Intuition#

A coding agent looks like autocomplete with extra steps. Type a few characters, get a suggestion. Easy at 10 users. At one million concurrent sessions it collapses, and the reason is that you are actually running three different products behind one interface.

Autocomplete must return ghost text before the developer types past it. That means sub-200 ms end-to-end, which means a small model, aggressive prompt caching, and HTTP/2 stream-reset for the 45-50% of requests the developer cancels by typing through^[1:1]. Chat runs at conversational latency on a frontier model with retrieved repo context prepended. The autonomous loop is the hardest: a multi-step agent that reads files, edits them, runs shell commands, observes results, and iterates for minutes on a single prompt, all inside a sandbox that prevents it from destroying the host.

The naive approach (one frontier model for everything) fails on cost and latency simultaneously. A single GPT-4-class completion costs roughly 100x what a small-model autocomplete costs^[5:1]. And the autonomous loop introduces a new failure mode: the agent can write code that exploits its own sandbox, install malicious packages, or enter an infinite retry loop burning tokens until someone notices.

The insight that unlocks the design: treat each mode as a separate product with its own model tier, latency SLO, and safety envelope, but share the repo-context retrieval index, the diff-application path, and the eval harness across all three.

Requirements#

Clarifying Questions#

• Q: Which modes do we ship at launch? Assume: All three (autocomplete, chat, autonomous loop). Each has a separate latency budget.
• Q: Cloud service or local-first CLI? Assume: Cloud SaaS (like Copilot/Cursor) with enterprise option for on-prem indexing (like Claude Code).
• Q: Do we own the models or route to third-party providers? Assume: We operate inference for autocomplete (small tuned model) and route chat/autonomous to a frontier provider API.
• Q: Can the agent run arbitrary shell commands? Assume: Yes, inside a per-session sandbox with network allowlist and resource ceilings.
• Q: Do we index private repos server-side? Assume: Yes for SaaS; enterprise customers can opt for client-side-only indexing.
• Q: Multi-tenant SaaS or single-tenant? Assume: Multi-tenant with strict isolation. Enterprise tier gets dedicated sandbox pools.

Functional Requirements#

• Inline autocomplete: ghost-text completion triggered by typing, dismissable with Escape
• Chat: conversational panel grounded in the open file, selection, and repo index
• Autonomous loop: multi-step agent that reads, edits, runs tests, and iterates until done
• Diff review: every agent edit previewed as a diff the developer accepts or rejects per hunk
• Repo indexing: background job keeps embedding + symbol index in sync with the current branch

Non-Functional Requirements#

• Sessions: 1M concurrent across all modes
• Autocomplete latency: p99 < 200 ms from keystroke to ghost text^[1:2]
• Chat latency: first-token < 1 s; streaming at 40+ tokens/sec
• Autonomous task duration: up to 5 minutes; continuous progress visible
• Sandbox isolation: agent shell commands cannot escape to host, hit arbitrary internet, or exceed CPU/memory/time budgets
• Cost per autocomplete: < $0.0002 (supports ~$20/month subscription at 100K completions/month)

Capacity Estimation#

Key ratios: autocomplete dominates request count but is cheapest per request. Chat dominates daily cost. Prompt caching (90% savings on repeated prefixes^[6]) is the single biggest cost lever for chat and autonomous modes.

API and Data Model#

API Design#

POST /v1/complete
  Body: { "file": "src/main.ts", "cursor": 142, "prefix": "...", "suffix": "...", "lang": "typescript" }
  Returns: 200 { "suggestion": "...", "tokens": 20, "model": "small-v3" }
  Target: p99 200 ms

POST /v1/chat (SSE)
  Body: { "session_id": "uuid", "messages": [...], "context_refs": ["file://src/auth.ts"] }
  Returns: SSE stream of token deltas + tool_call events

POST /v1/task (SSE)
  Body: { "prompt": "Fix the failing test in auth.ts", "repo": "repo-id", "permissions": ["read", "edit", "run_tests"] }
  Returns: SSE stream of thought, tool_call, tool_result, diff, done events

POST /v1/diff/apply
  Body: { "hunks": [...], "workspace_id": "uuid" }
  Returns: 200 { "rollback_token": "uuid" }

POST /v1/diff/rollback
  Body: { "rollback_token": "uuid" }
  Returns: 204

Rate limiting enforces per-tenant token budgets in Redis. A runaway autonomous task hits a spend ceiling and pauses for human approval.

Data Model#

-- Sessions (Redis hot + PostgreSQL cold)
CREATE TABLE sessions (
  session_id    UUID PRIMARY KEY,
  user_id       UUID NOT NULL,
  repo_id       UUID,
  mode          TEXT NOT NULL,  -- autocomplete | chat | autonomous
  model         TEXT NOT NULL,
  token_spend   BIGINT DEFAULT 0,
  created_at    TIMESTAMPTZ DEFAULT now()
);

-- Task state (PostgreSQL + object storage for large payloads)
CREATE TABLE task_state (
  task_id       UUID PRIMARY KEY,
  session_id    UUID NOT NULL,
  status        TEXT NOT NULL,  -- running | waiting_approval | succeeded | failed
  scratchpad    TEXT,           -- agent chain-of-thought (summarized)
  sandbox_id    TEXT,
  created_at    TIMESTAMPTZ DEFAULT now()
);

-- Repo index chunks (vector DB + PostgreSQL metadata)
CREATE TABLE repo_chunks (
  chunk_id      UUID PRIMARY KEY,
  repo_id       UUID NOT NULL,
  commit_sha    TEXT NOT NULL,
  file_path     TEXT NOT NULL,
  symbol_name   TEXT,
  embedding     vector(1536),
  last_indexed  TIMESTAMPTZ
);

-- Diff log (training signal)
CREATE TABLE diff_log (
  diff_id       UUID PRIMARY KEY,
  task_id       UUID,
  hunks         JSONB,
  accepted      BOOLEAN,
  created_at    TIMESTAMPTZ DEFAULT now()
);

High-Level Architecture#

Three mode-specific services share a common retrieval index and sandbox fleet, with the API gateway routing by mode and enforcing per-tenant budgets.

Autocomplete path: The IDE debounces keystrokes, sends prefix + suffix to the gateway, which forwards to a warm small-model GPU pool. The model returns ghost text within 200 ms. HTTP/2 stream-reset handles cancellation without tearing down the TCP connection^[1:3].

Chat path: The gateway loads conversation history, queries the retrieval service for relevant repo chunks, prepends them to the prompt, and streams tokens from a frontier model over SSE.

Autonomous path: The orchestrator runs a ReAct loop^[7]: reason (LLM generates a tool call), act (execute in sandbox), observe (capture stdout/exit code), repeat. Each iteration updates the scratchpad. A risk gate requires human approval for destructive actions.

Autocomplete completes within a 200 ms budget; HTTP/2 stream-reset handles the 45-50% of requests cancelled by type-through without tearing down the connection.

Deep Dives#

Repo-context retrieval architecture#

The retrieval layer is the product differentiator. Embeddings alone miss structural questions like "where is authenticate called?" A hybrid retriever combines three signals.

Signal 1: Semantic embeddings. Code is chunked at roughly function-level granularity and embedded. On each commit, only changed files are re-embedded. The vector DB serves approximate nearest-neighbor queries against the developer's current query or cursor context.

Signal 2: Tree-sitter symbol graph. Aider parses every source file with tree-sitter to extract definition and reference tags, builds a directed multigraph where each file is a node and each referencer-to-definer edge carries a weight, then runs PageRank with personalization biased toward files in the active chat^[8]. Identifiers mentioned by the user get a 10x multiplier; identifiers in chat-active files get 50x^[9]. The top-ranked symbols render into a token-budgeted tree (default 1,024 tokens)^[8:1].

Signal 3: Git awareness. Recently changed files and files in the current PR get a score boost. This mirrors what human developers reach for: the code they are already thinking about.

The hybrid retriever merges semantic, structural, and recency signals into a single ranked context block that fits within the model's token budget.

Sourcegraph Cody uses the same pattern: code search, symbol intelligence, and embeddings layered in a hybrid retriever that can pull context from repositories the developer does not have open^[10]. The key trade-off is index freshness: a rebase invalidates thousands of chunks. Mitigation: prioritize reindex of files the developer has open, invalidate lazily by commit SHA, and expose index lag to the UI^[8:2].

Tool-use sandbox isolation#

Every agent-initiated shell command runs inside a per-session Firecracker microVM. Firecracker is a Rust VMM built on KVM with thread-specific seccomp filters and a jailer process that applies cgroup/namespace isolation then drops privileges^[11]. AWS Lambda and Fargate run on it. Practical boot latency is ~125-150 ms^[12], making a warm pool viable.

Sandbox anatomy:

• Read-only mount of the repo snapshot at task start
• tmpfs scratch for agent writes (destroyed on finish)
• Network allowlist: package registries + model API only
• cgroup-enforced CPU (1 vCPU), memory (2 GB), and wall-clock (60 s per command) ceilings
• Destroy-on-finish: no persistent state survives between tasks

Each autonomous task gets a dedicated microVM with hardware-enforced isolation; destroy-on-finish limits persistence of any compromise.

Why not containers? A container escape requires only a kernel CVE, not a hypervisor escape. Container-based sandboxes for AI agents remain vulnerable to kernel exploits, as multiple security researchers have documented^[12:1]. MicroVM-per-session gives hardware-enforced isolation where a kernel escape costs an attacker a fresh hypervisor CVE rather than a container breakout.

Cognition's Devin runs each task in a cloud sandbox with browser, editor, and shell^[3:1]. Cursor adapted its Background Agents VM scheduler to run "hundreds of thousands of concurrent sandboxed coding environments" during Composer RL training^[13].

Streaming diff application#

The model streams edits; the client renders them inline as hunks the developer accepts or rejects. The critical insight from Cursor: full-file rewrite outperforms unified diffs for files under 400 lines because models see more full-file code than diffs in pretraining and struggle to count line numbers^[14].

Cursor's fast-apply model: As described in their May 2024 blog post, a specialized 70B model (fine-tuned from Llama 3) rewrites the full file at ~1,000 tokens/sec (3,500 char/s) using speculative edits, a deterministic-draft variant of speculative decoding tuned for code^[14:1]. This yields a ~13x speedup over vanilla Llama-3-70b inference and a ~9x speedup over Cursor's previous GPT-4 speculative edits deployment.

Aider's search/replace format: Aider interprets unified diffs as search/replace operations: the - and space lines form the search text, the + and space lines form the replacement^[15]. Cursor adopted a variant of this format with redundant + and - markers that make the parser resilient to minor model failures^[14:2]. Both formats avoid line-number hallucinations because the model specifies what to find and what to replace, not where.

Apply semantics: Each apply is atomic. The client accumulates hunks, the developer approves per-hunk or batch, and the filesystem transaction either succeeds completely or rolls back. A rollback token enables single-keystroke undo.

For files over 400 lines, Cursor is training long-context variants to reach 2,500 lines^[14:3]. The trade-off: full-file rewrite costs more output tokens but eliminates positional errors that cause silent corruption.

Real-World Example#

GitHub Copilot: 1.3M paid subscribers and 400M+ completions per day.

Copilot reached 1.3 million paid subscribers by Q1 2024 with 30% quarter-over-quarter growth, and more than 50,000 organizations had adopted it at the time of the Accenture study^[16][17]. The service handles 400M+ completion requests daily, peaking at ~8,000 QPS during the overlap of European afternoon and US morning^[1:4].

The architecture centers on a global authenticating proxy written in Go. The IDE exchanges an OAuth credential for a signed 10-30 minute code-completion token. The proxy validates the signature locally, swaps in the real Azure API key, and forwards over a long-lived HTTP/2 connection. octoDNS does weighted geographic routing to the nearest healthy proxy region^[1:5].

HTTP/2 is load-bearing. Stream-reset cancels a single request without tearing down the TCP connection. This matters because 45-50% of completion requests are cancelled by type-through before a response arrives^[1:6]. Cloud-provider ALBs that downgrade to HTTP/1 on the backend would have made this impossible, so the team uses GitHub's internal HAProxy-based GLB^[1:7].

The team explicitly rejected a PoP model: since every completion requires model inference, edge proxies back to centralized models create traffic tromboning with no caching benefit^[1:8]. When a model upgrade emitted a bad EOF token, the team added negative token affinity at the proxy rather than waiting weeks for a client rollout^[1:9].

An Accenture enterprise study of Copilot found that developers accepted around 30% of suggestions, committed code from 90% of those accepted suggestions, and retained 88% of Copilot-generated characters; teams saw an 8.69% increase in pull requests and a 15% increase in merge rate^[17:1]. Separate GitHub lab studies have measured task-completion speedups of up to 55%^[17:2].

Trade-offs#

The meta-decision: model tiering. Without it, autocomplete at frontier-model prices costs ~$0.01 per request, making a $20/month subscription unprofitable at 100K completions/month. With tiering, autocomplete costs ~$0.00005 per request, and the subscription works.

Scaling and Failure Modes#

At 10x (6M autocomplete QPS): The warm GPU pool saturates. Mitigation: scale on queue depth (not CPU), request coalescing for identical prefixes within 20 ms, and aggressive prompt-prefix caching.

At 100x (60M QPS): The retrieval index becomes the bottleneck. Mitigation: shard the vector DB by repo, replicate hot repos, and move to a CDN-first pattern where the most common completions are cached at the edge.

At 1000x: The architecture shifts to federated regional inference fleets with a global control plane for routing and billing only.

Failure: Runaway autonomous loop. The agent retries a failing test forever, burning tokens. Detection: per-task step/token/wall-clock ceilings plus duplicate-tool-call detection. Recovery: circuit-breaker halts the task and notifies the developer^[19:2]. Anthropic's April 2026 post-mortem documents exactly this pattern: Claude Code "would continue executing, but increasingly without memory" during a broken-cache bug^[20].

Failure: Prompt-caching regression. A cache optimization silently corrupts agent memory; quality degrades for days. Anthropic's clear_thinking bug (March 26 to April 10, 2026) cleared thinking on every turn instead of once, making the agent "seem forgetful and repetitive"^[20:1]. Detection: per-request diagnostics exposing cache-hit vs fresh; eval suites on every system-prompt change.

Failure: Sandbox escape. Agent installs a malicious package that exploits a kernel CVE. Mitigation: Firecracker hypervisor isolation (not just containers), network allowlist, package-hash verification, destroy-on-finish^[11:2][12:4].

Common Pitfalls#

Follow-up Questions#

Exercise#

Exercise 1: Sandbox fleet sizing#

Your coding agent has 10,000 concurrent autonomous tasks. Each task holds a Firecracker microVM for an average of 3 minutes. Your warm pool boots VMs in 125 ms. Tasks arrive at a Poisson rate. How large should the warm pool be to ensure 99% of tasks get a VM within 500 ms?

Key Takeaways#

• Three modes, three systems. Autocomplete, chat, and autonomous loop share infrastructure but have fundamentally different latency budgets, model tiers, and safety envelopes.
• Retrieval quality is the product. Embeddings alone miss structural queries; pair them with a tree-sitter symbol graph and git-recency signal^[8:4][10:2].
• Sandbox isolation is non-negotiable. Any agent running shell commands on untrusted input needs per-session microVM isolation with network allowlist and destroy-on-finish^[11:3][12:5].
• The diff is the interface. Full-file rewrite at 1,000 tokens/sec beats unified diffs because models cannot count line numbers^[14:5].
• Evals gate the rollout. SWE-bench Verified, diff-log replay, and online canary metrics catch regressions that user-facing signals reveal only days later^[23][20:4].

Further Reading#

• GitHub Copilot: Building a Low-Latency Global Code Completion Service. The canonical reference for HTTP/2 proxy architecture, octoDNS routing, and the 200 ms latency budget.
• Cursor: Editing Files at 1000 Tokens per Second. Fast-apply model, speculative edits, and why full-file rewrite beats diffs for files under 400 lines.
• Anthropic: Building Effective Agents. Agent design patterns (prompt chaining, routing, orchestrator-workers) with the recommendation to start simple.
• Anthropic: An Update on Recent Claude Code Quality Reports. April 2026 post-mortem documenting three compounding regressions; canonical example of agent eval failures.
• Aider: Building a Better Repository Map with Tree Sitter. Tree-sitter + PageRank for code context retrieval; the open-source reference implementation.
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Yang et al., NeurIPS 2024; defines the ACI concept that every coding agent now uses.
• Anthropic: Prompt Caching with Claude. 90% cost and 85% latency reductions; the single biggest cost lever for agents that re-read repo context.
• Cursor: Composer - Building a Fast Frontier Model with RL. MoE + RL training on agent tasks at scale with hundreds of thousands of concurrent sandboxed environments.

Flashcards#

References#