# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Project Overview Wrenn Sandbox is a microVM-based code execution platform. Users create isolated sandboxes (Firecracker microVMs), run code inside them, and get output back via SDKs. Think E2B but with persistent sandboxes, pool-based pricing, and a single-binary deployment story. ## Build & Development Commands All commands go through the Makefile. Never use raw `go build` or `go run`. ```bash make build # Build all binaries → builds/ make build-cp # Control plane only (builds frontend first) make build-agent # Host agent only make build-envd # envd static binary (verified statically linked) make build-frontend # SvelteKit dashboard → internal/dashboard/static/ make dev # Full local dev: infra + migrate + control plane make dev-infra # Start PostgreSQL + Prometheus + Grafana (Docker) make dev-down # Stop dev infra make dev-cp # Control plane with hot reload (if air installed) make dev-frontend # Vite dev server with HMR (port 5173) make dev-agent # Host agent (sudo required) make dev-envd # envd in TCP debug mode make check # fmt + vet + lint + test (CI order) make test # Unit tests: go test -race -v ./internal/... make test-integration # Integration tests (require host agent + Firecracker) make fmt # gofmt both modules make vet # go vet both modules make lint # golangci-lint make migrate-up # Apply pending migrations make migrate-down # Rollback last migration make migrate-create name=xxx # Scaffold new goose migration (never create manually) make migrate-reset # Drop + re-apply all make generate # Proto (buf) + sqlc codegen make proto # buf generate for all proto dirs make tidy # go mod tidy both modules ``` Run a single test: `go test -race -v -run TestName ./internal/path/...` ## Architecture ``` User SDK → HTTPS/WS → Control Plane → Connect RPC → Host Agent → HTTP/Connect RPC over TAP → envd (inside VM) ``` **Three binaries, two Go modules:** | Binary | Module | Entry point | Runs as | |--------|--------|-------------|---------| | wrenn-cp | `git.omukk.dev/wrenn/sandbox` | `cmd/control-plane/main.go` | Unprivileged | | wrenn-agent | `git.omukk.dev/wrenn/sandbox` | `cmd/host-agent/main.go` | Root (NET_ADMIN + /dev/kvm) | | envd | `git.omukk.dev/wrenn/sandbox/envd` (standalone `envd/go.mod`) | `envd/main.go` | PID 1 inside guest VM | envd is a **completely independent Go module**. It is never imported by the main module. The only connection is the protobuf contract. It compiles to a static binary baked into rootfs images. **Key architectural invariant:** The host agent is **stateful** (in-memory `boxes` map is the source of truth for running VMs). The control plane is **stateless** (all persistent state in PostgreSQL). The reconciler (`internal/api/reconciler.go`) bridges the gap — it periodically compares DB records against the host agent's live state and marks orphaned sandboxes as "stopped". ### Control Plane **Packages:** `internal/api/`, `internal/dashboard/`, `internal/auth/`, `internal/scheduler/`, `internal/lifecycle/`, `internal/config/`, `internal/db/` Startup (`cmd/control-plane/main.go`) wires: config (env vars) → pgxpool → `db.Queries` (sqlc-generated) → Connect RPC client to host agent → `api.Server`. Everything flows through constructor injection. - **API Server** (`internal/api/server.go`): chi router with middleware. Creates handler structs (`sandboxHandler`, `execHandler`, `filesHandler`, etc.) injected with `db.Queries` and the host agent Connect RPC client. Routes under `/v1/sandboxes/*`. - **Reconciler** (`internal/api/reconciler.go`): background goroutine (every 30s) that compares DB records against `agent.ListSandboxes()` RPC. Marks orphaned DB entries as "stopped". - **Dashboard** (SvelteKit + Tailwind + Bits UI, statically built and embedded via `go:embed`, served as catch-all at root) - **Database**: PostgreSQL via pgx/v5. Queries generated by sqlc from `db/queries/sandboxes.sql`. Migrations in `db/migrations/` (goose, plain SQL). - **Config** (`internal/config/config.go`): purely environment variables (`DATABASE_URL`, `CP_LISTEN_ADDR`, `CP_HOST_AGENT_ADDR`), no YAML/file config. ### Host Agent **Packages:** `internal/hostagent/`, `internal/sandbox/`, `internal/vm/`, `internal/network/`, `internal/devicemapper/`, `internal/envdclient/`, `internal/snapshot/` Startup (`cmd/host-agent/main.go`) wires: root check → enable IP forwarding → clean up stale dm devices → `sandbox.Manager` (containing `vm.Manager` + `network.SlotAllocator` + `devicemapper.LoopRegistry`) → `hostagent.Server` (Connect RPC handler) → HTTP server. - **RPC Server** (`internal/hostagent/server.go`): implements `hostagentv1connect.HostAgentServiceHandler`. Thin wrapper — every method delegates to `sandbox.Manager`. Maps Connect error codes on return. - **Sandbox Manager** (`internal/sandbox/manager.go`): the core orchestration layer. Maintains in-memory state in `boxes map[string]*sandboxState` (protected by `sync.RWMutex`). Each `sandboxState` holds a `models.Sandbox`, a `*network.Slot`, and an `*envdclient.Client`. Runs a TTL reaper (every 10s) that auto-destroys timed-out sandboxes. - **VM Manager** (`internal/vm/manager.go`, `fc.go`, `config.go`): manages Firecracker processes. Uses raw HTTP API over Unix socket (`/tmp/fc-{sandboxID}.sock`), not the firecracker-go-sdk Machine type. Launches Firecracker via `unshare -m` + `ip netns exec`. Configures VM via PUT to `/boot-source`, `/drives/rootfs`, `/network-interfaces/eth0`, `/machine-config`, then starts with PUT `/actions`. - **Network** (`internal/network/setup.go`, `allocator.go`): per-sandbox network namespace with veth pair + TAP device. See Networking section below. - **Device Mapper** (`internal/devicemapper/devicemapper.go`): CoW rootfs via device-mapper snapshots. Shared read-only loop devices per base template (refcounted `LoopRegistry`), per-sandbox sparse CoW files, dm-snapshot create/restore/remove/flatten operations. - **envd Client** (`internal/envdclient/client.go`, `health.go`): dual interface to the guest agent. Connect RPC for streaming process exec (`process.Start()` bidirectional stream). Plain HTTP for file operations (POST/GET `/files?path=...&username=root`). Health check polls `GET /health` every 100ms until ready (30s timeout). ### envd (Guest Agent) **Module:** `envd/` with its own `go.mod` (`git.omukk.dev/wrenn/sandbox/envd`) Runs as PID 1 inside the microVM via `wrenn-init.sh` (mounts procfs/sysfs/dev, sets hostname, writes resolv.conf, then execs envd). Extracted from E2B (Apache 2.0), with shared packages internalized into `envd/internal/shared/`. Listens on TCP `0.0.0.0:49983`. - **ProcessService**: start processes, stream stdout/stderr, signal handling, PTY support - **FilesystemService**: stat/list/mkdir/move/remove/watch files - **Health**: GET `/health` ### Dashboard (Frontend) **Directory:** `frontend/` — standalone SvelteKit app (Svelte 5, runes mode) - **Stack**: SvelteKit + `adapter-static` + Tailwind CSS v4 + Bits UI (headless accessible components) - **Package manager**: pnpm - **Routing**: SvelteKit file-based routing under `frontend/src/routes/` - **Routing layout**: `/login` and `/signup` at root, authenticated pages under `/dashboard/*` (e.g. `/dashboard/capsules`, `/dashboard/keys`) - **Build output**: `frontend/build/` → copied to `internal/dashboard/static/` → embedded via `go:embed` into the control plane binary - **Serving**: `internal/dashboard/dashboard.go` registers a `NotFound` catch-all SPA handler with fallback to `index.html`. API routes (`/v1/*`, `/openapi.yaml`, `/docs`) are registered first and take priority - **Dev workflow**: `make dev-frontend` runs Vite dev server on port 5173 with HMR. API calls proxy to `http://localhost:8000` - **Fonts**: Manrope (UI), Instrument Serif (headings), JetBrains Mono (code), Alice (brand wordmark) — all self-hosted via `@fontsource` - **Dark mode**: class-based (`.dark` on ``) with system preference detection + localStorage persistence To add a new page: create `frontend/src/routes/your-page/+page.svelte`. ### Networking (per sandbox) Each sandbox gets its own Linux network namespace (`ns-{idx}`). Slot index (1-based, up to 65534) determines all addressing: ``` Host Namespace Namespace "ns-{idx}" Guest VM ────────────────────────────────────────────────────────────────────────────────────── veth-{idx} ←──── veth pair ────→ eth0 10.12.0.{idx*2}/31 10.12.0.{idx*2+1}/31 │ tap0 (169.254.0.22/30) ←── TAP ──→ eth0 (169.254.0.21) ↑ kernel ip= boot arg ``` - **Host-reachable IP**: `10.11.0.{idx}/32` — routed through veth to namespace, DNAT'd to guest - **Outbound NAT**: guest (169.254.0.21) → SNAT to vpeerIP inside namespace → MASQUERADE on host to default interface - **Inbound NAT**: host traffic to 10.11.0.{idx} → DNAT to 169.254.0.21 inside namespace - IP forwarding enabled inside each namespace - All details in `internal/network/setup.go` ### Sandbox State Machine ``` PENDING → STARTING → RUNNING → PAUSED → HIBERNATED │ │ ↓ ↓ STOPPED STOPPED → (destroyed) Any state → ERROR (on crash/failure) PAUSED → RUNNING (warm snapshot resume) HIBERNATED → RUNNING (cold snapshot resume, slower) ``` ### Key Request Flows **Sandbox creation** (`POST /v1/sandboxes`): 1. API handler generates sandbox ID, inserts into DB as "pending" 2. RPC `CreateSandbox` → host agent → `sandbox.Manager.Create()` 3. Manager: resolve base rootfs → acquire shared loop device → create dm-snapshot (sparse CoW file) → allocate network slot → `CreateNetwork()` (netns + veth + tap + NAT) → `vm.Create()` (start Firecracker with `/dev/mapper/wrenn-{id}`, configure via HTTP API, boot) → `envdclient.WaitUntilReady()` (poll /health) → store in-memory state 4. API handler updates DB to "running" with host_ip **Command execution** (`POST /v1/sandboxes/{id}/exec`): 1. API handler verifies sandbox is "running" in DB 2. RPC `Exec` → host agent → `sandbox.Manager.Exec()` → `envdclient.Exec()` 3. envd client opens bidirectional Connect RPC stream (`process.Start`), collects stdout/stderr/exit_code 4. API handler checks UTF-8 validity (base64-encodes if binary), updates last_active_at, returns result **Streaming exec** (`WS /v1/sandboxes/{id}/exec/stream`): 1. WebSocket upgrade, read first message for cmd/args 2. RPC `ExecStream` → host agent → `sandbox.Manager.ExecStream()` → `envdclient.ExecStream()` 3. envd client returns a channel of events; host agent forwards events through the RPC stream 4. API handler forwards stream events to WebSocket as JSON messages (`{type: "stdout"|"stderr"|"exit", ...}`) **File transfer**: Write uses multipart POST to envd `/files`; read uses GET. Streaming variants chunk in 64KB pieces through the RPC stream. ## REST API Routes defined in `internal/api/server.go`, handlers in `internal/api/handlers_*.go`. OpenAPI spec embedded via `//go:embed` and served at `/openapi.yaml` (Swagger UI at `/docs`). JSON request/response. API key auth via `X-API-Key` header. Error responses: `{"error": {"code": "...", "message": "..."}}`. ## Code Generation ### Proto (Connect RPC) Proto source of truth is `proto/envd/*.proto` and `proto/hostagent/*.proto`. Run `make proto` to regenerate. Three `buf.gen.yaml` files control output: | buf.gen.yaml location | Generates to | Used by | |---|---|---| | `proto/envd/buf.gen.yaml` | `proto/envd/gen/` | Main module (host agent's envd client) | | `proto/hostagent/buf.gen.yaml` | `proto/hostagent/gen/` | Main module (control plane ↔ host agent) | | `envd/spec/buf.gen.yaml` | `envd/internal/services/spec/` | envd module (guest agent server) | The envd `buf.gen.yaml` reads from `../../proto/envd/` (same source protos) but generates into envd's own module. This means the same `.proto` files produce two independent sets of Go stubs — one for each Go module. To add a new RPC method: edit the `.proto` file → `make proto` → implement the handler on both sides. ### sqlc Config: `sqlc.yaml` (project root). Reads queries from `db/queries/*.sql`, reads schema from `db/migrations/`, outputs to `internal/db/`. To add a new query: add it to the appropriate `.sql` file in `db/queries/` → `make generate` → use the new method on `*db.Queries`. ## Key Technical Decisions - **Connect RPC** (not gRPC) for all RPC communication between components - **Buf + protoc-gen-connect-go** for code generation (not protoc-gen-go-grpc) - **Raw Firecracker HTTP API** via Unix socket (not firecracker-go-sdk Machine type) - **TAP networking** (not vsock) for host-to-envd communication - **Device-mapper snapshots** for rootfs CoW — shared read-only loop device per base template, per-sandbox sparse CoW file, Firecracker gets `/dev/mapper/wrenn-{id}` - **PostgreSQL** via pgx/v5 + sqlc (type-safe query generation). Goose for migrations (plain SQL, up/down) - **Dashboard**: SvelteKit (Svelte 5, adapter-static) + Tailwind CSS v4 + Bits UI. Built to static files, embedded into the Go binary via `go:embed`, served as catch-all at root - **Lago** for billing (external service, not in this codebase) ## Coding Conventions - **Go style**: `gofmt`, `go vet`, `context.Context` everywhere, errors wrapped with `fmt.Errorf("action: %w", err)`, `slog` for logging, no global state - **Naming**: Sandbox IDs `sb-` + 8 hex, API keys `wrn_` + 32 chars, Host IDs `host-` + 8 hex - **Dependencies**: Use `go get` to add deps, never hand-edit go.mod. For envd deps: `cd envd && go get ...` (separate module) - **Generated code**: Always commit generated code (proto stubs, sqlc). Never add generated code to .gitignore - **Migrations**: Always use `make migrate-create name=xxx`, never create migration files manually - **Testing**: Table-driven tests for handlers and state machine transitions ### Two-module gotcha The main module (`go.mod`) and envd (`envd/go.mod`) are fully independent. `make tidy`, `make fmt`, `make vet` already operate on both. But when adding dependencies manually, remember to target the correct module (`cd envd && go get ...` for envd deps). `make proto` also generates stubs for both modules from the same proto sources. ## Rootfs & Guest Init - **wrenn-init** (`images/wrenn-init.sh`): the PID 1 init script baked into every rootfs. Mounts virtual filesystems, sets hostname, writes `/etc/resolv.conf`, then execs envd. - **Updating the rootfs** after changing envd or wrenn-init: `bash scripts/update-debug-rootfs.sh [rootfs_path]`. This builds envd via `make build-envd`, mounts the rootfs image, copies in the new binaries, and unmounts. Defaults to `/var/lib/wrenn/images/minimal.ext4`. - Rootfs images are minimal debootstrap — no systemd, no coreutils beyond busybox. Use `/bin/sh -c` for shell builtins inside the guest. ## Fixed Paths (on host machine) - Kernel: `/var/lib/wrenn/kernels/vmlinux` - Base rootfs images: `/var/lib/wrenn/images/{template}.ext4` - Sandbox clones: `/var/lib/wrenn/sandboxes/` - Firecracker: `/usr/local/bin/firecracker` (e2b's fork of firecracker) ## Web UI Styling ### Identity Warm, confident developer tool with industrial precision and crafted organic character. The feel is sharp and data-forward — not cold or sterile, but not soft either. Think: an engineer's favorite tool, built with care. --- ### Color Palette (Dark Mode) **Background scale (6 steps, near-black-green):** `#0a0c0b` (bg-0, page base) → `#0f1211` (bg-1, sidebar/topbar) → `#141817` (bg-2, cards/surfaces) → `#1a1e1c` (bg-3, hover states/elevated) → `#212624` (bg-4, inputs/avatars) → `#2a302d` (bg-5, active controls) **Text hierarchy (5 levels):** - Bright `#eae7e2` — page titles, metric values, active states - Primary `#d0cdc6` — body text, nav labels, readable content - Secondary `#9b9790` — supporting text, inactive nav, descriptions - Tertiary `#6b6862` — labels, section headers, timestamps - Muted `#454340` — ghost text, disabled states, grid labels **Sage green brand accent (3 tiers + 2 glows):** - Solid `#5e8c58` — primary accent, buttons, borders, active indicators - Mid `#89a785` — badges, chart lines, secondary accent - Bright `#a4c89f` — active nav text, live counts, chart dots - Glow `rgba(94,140,88,0.07)` — active nav backgrounds, subtle highlights - Glow Mid `rgba(94,140,88,0.14)` — live badges, status badge backgrounds **Borders (2 levels):** - Default `#1f2321` — card edges, dividers, sidebar borders - Mid `#2a2f2c` — hover states, interactive borders, stronger separation **Semantic status colors:** - Amber `#d4a73c` — warning, building, countdown timers - Red `#cf8172` — error, failed, destructive actions - Blue `#5a9fd4` — info, stopped (use sparingly) **Light mode:** (TBD — follow same warm-tinted approach. Background scale from `#f8f6f1` → `#dedbd5`. Text hierarchy inverts. Accent stays `#5e8c58` for solid.) --- ### Typography Four fonts, each with a clear role: | Font | Role | Weights | Where | |------|------|---------|-------| | **Manrope** (variable) | Body, UI | 400–700 | All body text, nav labels, buttons, descriptions, section headers | | **Instrument Serif** | Display, metrics | 400 | Page titles (h1), large metric values, empty-state headings only | | **JetBrains Mono** | Code, data | 400–600 | Status bar, time range buttons, search inputs, IDs, commit SHAs, countdown timers, log viewer, URL paths, code blocks | | **Alice** | Brand wordmark | 400 | Sidebar wordmark only — never used elsewhere | **Sizing:** - Base body: `14px` - Page title (h1): `24px` serif, `letter-spacing: -0.02em` - Card metric values: `36px` serif, `letter-spacing: -0.04em` - Chart inline metric: `30px` serif, `letter-spacing: -0.04em` - Nav items: `13px` body, weight 500 - Section/group labels: `11px` body, uppercase, `letter-spacing: 0.06em`, weight 600 - Chart section labels: `12px` body, uppercase, `letter-spacing: 0.05em`, weight 600 - Stat cell labels: `11px` body, uppercase, `letter-spacing: 0.05em`, weight 600 - Badge text: `10px`, uppercase, `letter-spacing: 0.04em`, weight 600 - Status bar / footer links: `11–12px` mono - Table headers: `11px` body, uppercase, `letter-spacing: 0.05em`, weight 600, color muted - Table body cells: `13px` **Key rule:** Instrument Serif is reserved exclusively for page-level titles and large numeric values. It provides warmth and character without softness. Everything else uses Manrope (UI) or JetBrains Mono (data/code). --- ### Spacing 4px base unit (Tailwind scale). Moderate density — functional and confident, never cramped. - Page content padding: `24–28px` - Card/surface internal padding: `18–20px` - Sidebar width: `230px` - Sidebar nav item padding: `8px 10px` - Sidebar brand area: `18px 16px 16px` - Tab bar items: `10px 16px` - Topbar: `16px 28px` - Metric strip cell: `18px 20px` - Chart header: `18px 20px` - Chart canvas: `14px 20px 12px` - Table header cells: `11px 16px` - Table body cells: `12px 16px` - Status bar: `6px 28px` - Between sections (cards): `20–24px` margin-bottom --- ### Borders & Depth **Flat aesthetic — no drop shadows.** Depth comes from background color stepping (bg-0 → bg-1 → bg-2 → bg-3), not shadows. `--shadow-sm: 0 0 #0000`. - All borders: `1px solid` in warm muted tones - Corner radii: cards/surfaces `8px`, inputs/buttons `5px`, logo mark `6px`, avatars `5px`, dots `50%` - Connected metric cells use shared border container with `border-left: 1px solid` between cells (no gap/grid trick) — creates the industrial panel look - Tables wrapped in `border-radius: 8px` container with overflow hidden --- ### Components **Sidebar navigation:** - Active items use `3px left-border` in sage solid (`#5e8c58`) with accent glow background (`rgba(94,140,88,0.07)`) - Active text color: accent-bright (`#a4c89f`) - Icons at `16px`, opacity 0.5 default, 1.0 on active - Group labels: `11px` uppercase with `0.06em` tracking, muted color **Status chip (live indicator):** - Rounded `8px` border, `bg-2` background, `border-mid` border - Pulsing dot: `7px`, accent-solid fill, `box-shadow: 0 0 8px rgba(94,140,88,0.5)` with glow animation - Count in mono at `14px` accent-bright, label in secondary text **Live badges (inline):** - `10px` text, uppercase, `3px` border-radius - Background: accent-glow-mid (`rgba(94,140,88,0.14)`), text: accent mid - Includes `5px` pulsing dot with box-shadow **Metric strip:** - 3-column grid, connected cells (single outer border, inner dividers) - Hover: background steps from bg-2 to bg-3 - Value: `36px` serif, bright text - Label: `11px` uppercase, tertiary - Sub-metadata row with `1px` divider between items **Chart cards:** - `8px` border-radius, bg-2 background, default border - Header: section label (12px uppercase) + large serif metric + live badge - Range group: segmented buttons with `1px` borders, mono text, active state uses bg-5 - Chart area: SVG with `0.5px` grid lines in border color, `10px` mono axis labels in muted - Data line: `1.5px` accent-solid stroke, `stroke-linejoin: round` - Area fill: gradient from `rgba(94,140,88,0.28)` → transparent - Data dot: accent-bright fill, `2.5px` bg-2 stroke, `4px` radius **Buttons hierarchy:** 1. Ghost (icon-btn): transparent bg, default border, tertiary color → border-mid + secondary on hover 2. Outline: no bg, border-mid border → accent-solid border + primary text on hover 3. Tool: bg-2 background, default border → border-mid + primary on hover 4. Filled/CTA: accent-solid background, white text → lighter green on hover, subtle `translateY(-1px)` lift **Tables:** - Container: `8px` border-radius, border, overflow hidden - Header: bg-3 background, `11px` uppercase muted text - Body: default bg, `1px` border-bottom between rows - Row hover: bg-3 **Empty states:** - Centered, `72px` vertical padding - Icon container: `56px` square, bg-3, border-mid border, `8px` radius - Heading: `20px` serif, bright text - Description: `13px` body, tertiary text - CTA button below **Inputs:** - bg-2 background, default border, `5px` radius - Mono font for search/filter inputs - Focus: `border-color: accent-solid` (clean single ring, no double-ring) - Placeholder: muted color **Focus rings:** Single accent-solid border-color change on focus. Clean and minimal — no double-ring outlines. --- ### Animation - **All interactive transitions:** `150ms ease` - **Page load / section entrance:** `fadeUp` — `opacity: 0, translateY(6px)` → visible, `0.35s ease`, staggered with `60–80ms` delays between elements - **Chart data animation:** SVG `` on path `d`, polyline `points`, and circle `cy` — `0.5–0.6s` duration, `0.2–0.35s` begin delay, `fill: freeze` - **Live status dot:** `glow` keyframe — `2.5s ease infinite` box-shadow bloom from `0 0 6px rgba(94,140,88,0.5)` → `0 0 14px rgba(94,140,88,0.2)` - **CTA buttons:** subtle `translateY(-1px)` on hover for lift feel --- ### Dark Mode Primary and default mode. Very dark near-black-green backgrounds (`#0a0c0b` base) with warm off-white text and desaturated sage accent. Completely flat — no card shadows anywhere. System preference detection + localStorage persistence. --- ### Overall Feel Sharp, warm, industrial-confident. Avoids cold grays entirely — palette leans slightly warm/brown-tinted throughout. The serif display type provides organic character and warmth on titles and metrics, while Manrope handles readable UI text and JetBrains Mono anchors the data-forward, developer-tool identity. Connected metric panels, tight chart cards, and uppercase section labels create engineering density without sacrificing readability. The result is a tool that feels crafted and precise — designed by someone who uses developer tools daily.