From cc63ed21976b9c8583824807d9745ba4b6b416ca Mon Sep 17 00:00:00 2001 From: pptx704 Date: Thu, 16 Apr 2026 18:14:50 +0600 Subject: [PATCH] Minor patch --- CLAUDE.md | 14 ++--- README.md | 133 ++++++++++++++++++++++++----------------- cmd/host-agent/main.go | 14 ++--- 3 files changed, 93 insertions(+), 68 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 33b1f06..56fdbbc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,10 +12,10 @@ All commands go through the Makefile. Never use raw `go build` or `go run`. ```bash make build # Build all binaries → builds/ -make build-cp # Control plane only (builds frontend first) +make build-cp # Control plane only make build-agent # Host agent only make build-envd # envd static binary (verified statically linked) -make build-frontend # SvelteKit dashboard → internal/dashboard/static/ +make build-frontend # SvelteKit dashboard → frontend/build/ (served by Caddy) make dev # Full local dev: infra + migrate + control plane make dev-infra # Start PostgreSQL + Prometheus + Grafana (Docker) @@ -64,7 +64,7 @@ envd is a **completely independent Go module**. It is never imported by the main ### Control Plane -**Internal packages:** `internal/api/`, `internal/dashboard/`, `internal/email/` +**Internal packages:** `internal/api/`, `internal/email/` **Public packages (importable by cloud repo):** `pkg/config/`, `pkg/db/`, `pkg/auth/`, `pkg/auth/oauth/`, `pkg/scheduler/`, `pkg/lifecycle/`, `pkg/channels/`, `pkg/audit/`, `pkg/service/`, `pkg/events/`, `pkg/id/`, `pkg/validate/` @@ -78,7 +78,7 @@ Startup (`cmd/control-plane/main.go`) is a thin wrapper: `cpserver.Run(cpserver. - **API Server** (`internal/api/server.go`): chi router with middleware. Creates handler structs (`sandboxHandler`, `execHandler`, `filesHandler`, etc.) injected with `db.Queries` and the host agent Connect RPC client. Routes under `/v1/capsules/*`. Accepts `[]cpextension.Extension` — each extension's `RegisterRoutes()` is called after all core routes are registered. - **Reconciler** (`internal/api/reconciler.go`): background goroutine (every 30s) that compares DB records against `agent.ListSandboxes()` RPC. Marks orphaned DB entries as "stopped". -- **Dashboard** (SvelteKit + Tailwind + Bits UI, statically built and embedded via `go:embed`, served as catch-all at root) +- **Dashboard** (SvelteKit + Tailwind + Bits UI, built to static files in `frontend/build/`, served by Caddy as a reverse proxy) - **Database**: PostgreSQL via pgx/v5. Queries generated by sqlc from `db/queries/*.sql` → `pkg/db/`. Migrations in `db/migrations/` (goose, plain SQL). `db/migrations/embed.go` exposes `migrations.FS` so the cloud repo can run OSS migrations via `go:embed`. - **Config** (`pkg/config/config.go`): purely environment variables (`DATABASE_URL`, `CP_LISTEN_ADDR`, `CP_HOST_AGENT_ADDR`), no YAML/file config. @@ -115,8 +115,8 @@ Runs as PID 1 inside the microVM via `wrenn-init.sh` (mounts procfs/sysfs/dev, s - **Package manager**: pnpm - **Routing**: SvelteKit file-based routing under `frontend/src/routes/` - **Routing layout**: `/login` and `/signup` at root, authenticated pages under `/dashboard/*` (e.g. `/dashboard/capsules`, `/dashboard/keys`) -- **Build output**: `frontend/build/` → copied to `internal/dashboard/static/` → embedded via `go:embed` into the control plane binary -- **Serving**: `internal/dashboard/dashboard.go` registers a `NotFound` catch-all SPA handler with fallback to `index.html`. API routes (`/v1/*`, `/openapi.yaml`, `/docs`) are registered first and take priority +- **Build output**: `frontend/build/` — static files served by Caddy +- **Serving**: Caddy reverse-proxies API requests to the control plane and serves the SvelteKit SPA directly. The control plane does not serve frontend assets. - **Dev workflow**: `make dev-frontend` runs Vite dev server on port 5173 with HMR. API calls proxy to `http://localhost:8000` - **Fonts**: Manrope (UI), Instrument Serif (headings), JetBrains Mono (code), Alice (brand wordmark) — all self-hosted via `@fontsource` - **Dark mode**: class-based (`.dark` on ``) with system preference detection + localStorage persistence @@ -211,7 +211,7 @@ To add a new query: add it to the appropriate `.sql` file in `db/queries/` → ` - **TAP networking** (not vsock) for host-to-envd communication - **Device-mapper snapshots** for rootfs CoW — shared read-only loop device per base template, per-sandbox sparse CoW file, Firecracker gets `/dev/mapper/wrenn-{id}` - **PostgreSQL** via pgx/v5 + sqlc (type-safe query generation). Goose for migrations (plain SQL, up/down) -- **Dashboard**: SvelteKit (Svelte 5, adapter-static) + Tailwind CSS v4 + Bits UI. Built to static files, embedded into the Go binary via `go:embed`, served as catch-all at root +- **Dashboard**: SvelteKit (Svelte 5, adapter-static) + Tailwind CSS v4 + Bits UI. Built to static files in `frontend/build/`, served by Caddy (not embedded in the Go binary) - **Lago** for billing (external service, not in this codebase) ## Coding Conventions diff --git a/README.md b/README.md index c312494..765be5a 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,16 @@ Secure infrastructure for AI -## Deployment - -### Prerequisites +## Prerequisites - Linux host with `/dev/kvm` access (bare metal or nested virt) - Firecracker binary at `/usr/local/bin/firecracker` - PostgreSQL - Go 1.25+ +- pnpm (for frontend) +- Docker (for dev infra and rootfs builds) -### Build +## Build ```bash make build # outputs to builds/ @@ -19,30 +19,77 @@ make build # outputs to builds/ Produces three binaries: `wrenn-cp` (control plane), `wrenn-agent` (host agent), `envd` (guest agent). -### Host setup +## Host setup -The host agent machine needs: +The host agent needs a kernel, a minimal rootfs image, and working directories on the host machine. -```bash -# Kernel for guest VMs -mkdir -p /var/lib/wrenn/kernels -# Place a vmlinux kernel at /var/lib/wrenn/kernels/vmlinux +### Directory structure -# Rootfs images -mkdir -p /var/lib/wrenn/images -# Build or place .ext4 rootfs images (e.g., minimal.ext4) - -# Sandbox working directory -mkdir -p /var/lib/wrenn/sandboxes - -# Snapshots directory -mkdir -p /var/lib/wrenn/snapshots - -# Enable IP forwarding -sysctl -w net.ipv4.ip_forward=1 +``` +/var/lib/wrenn/ +├── kernels/ +│ └── vmlinux # uncompressed Linux kernel (not bzImage) +├── images/ +│ └── minimal/ +│ └── rootfs.ext4 # base rootfs (all other templates snapshot from this) +├── sandboxes/ # per-sandbox CoW files (created at runtime) +└── snapshots/ # pause/hibernate snapshot files (created at runtime) ``` -### Configure +Create the directories: + +```bash +sudo mkdir -p /var/lib/wrenn/{kernels,images/minimal,sandboxes,snapshots} +``` + +### Kernel + +Place an uncompressed `vmlinux` kernel at `/var/lib/wrenn/kernels/vmlinux`. Versioned kernels (`vmlinux-{semver}`) are also supported — the agent picks the latest by semver. + +### Minimal rootfs + +The minimal rootfs is the base image that all other templates (Python, Node, etc.) are built on top of via device-mapper snapshots. It must contain: + +| Package | Why | +|---------|-----| +| `socat` | Bidirectional relay for port forwarding | +| `chrony` | Time sync from KVM PTP clock (`/dev/ptp0`) | +| `tini` | PID 1 zombie reaper (injected by build script, not apt) | +| `sudo` | User privilege management inside the guest | +| `wget` | HTTP fetching | +| `curl` | HTTP client | +| `ca-certificates` | TLS certificate verification | + +**To build a rootfs from a Docker container:** + +1. Create and configure a container with the required packages: + ```bash + docker run -it --name wrenn-minimal debian:bookworm bash + # Inside the container: + apt update && apt install -y socat chrony sudo wget curl ca-certificates + exit + ``` + +2. Export to a rootfs image (builds envd, injects wrenn-init + tini, shrinks to minimum size): + ```bash + sudo bash scripts/rootfs-from-container.sh wrenn-minimal minimal + ``` + +**To update an existing rootfs** after changing envd or `wrenn-init.sh`: + +```bash +bash scripts/update-minimal-rootfs.sh +``` + +This rebuilds envd via `make build-envd` and copies the fresh binaries into the mounted rootfs image. + +### IP forwarding + +```bash +sudo sysctl -w net.ipv4.ip_forward=1 +``` + +## Configure Copy `.env.example` to `.env` and edit: @@ -59,25 +106,21 @@ WRENN_HOST_LISTEN_ADDR=:50051 WRENN_DIR=/var/lib/wrenn ``` -### Run +## Development ```bash -# Apply database migrations -make migrate-up - -# Start control plane -./builds/wrenn-cp +make dev # Start PostgreSQL (Docker), run migrations, start control plane +make dev-agent # Start host agent (separate terminal, sudo) +make dev-frontend # Vite dev server with HMR (port 5173) +make check # fmt + vet + lint + test ``` -Control plane listens on `WRENN_CP_LISTEN_ADDR` (default `:8000`). - ### Host registration Hosts must be registered with the control plane before they can serve sandboxes. 1. **Create a host record** (via API or dashboard): ```bash - # As an admin (JWT auth) curl -X POST http://localhost:8000/v1/hosts \ -H "Authorization: Bearer $JWT_TOKEN" \ -H "Content-Type: application/json" \ @@ -87,17 +130,16 @@ Hosts must be registered with the control plane before they can serve sandboxes. 2. **Start the host agent** with the registration token and its externally-reachable address: ```bash - sudo WRENN_CP_URL=http://cp-host:8000 \ + sudo WRENN_CP_URL=http://localhost:8000 \ ./builds/wrenn-agent \ --register \ - --address 10.0.1.5:50051 + --address :50051 ``` On first startup the agent sends its specs (arch, CPU, memory, disk) to the control plane, receives a long-lived host JWT, and saves it to `$WRENN_DIR/host-token`. 3. **Subsequent startups** don't need `--register` — the agent loads the saved JWT automatically: ```bash - sudo WRENN_CP_URL=http://cp-host:8000 \ - ./builds/wrenn-agent --address 10.0.1.5:50051 + sudo ./builds/wrenn-agent --address :50051 ``` 4. **If registration fails** (e.g., network error after token was consumed), regenerate a token: @@ -107,23 +149,6 @@ Hosts must be registered with the control plane before they can serve sandboxes. ``` Then restart the agent with the new token. -The agent sends heartbeats to the control plane every 30 seconds. Host agent listens on `WRENN_HOST_LISTEN_ADDR` (default `:50051`). - -### Rootfs images - -envd must be baked into every rootfs image. After building: - -```bash -make build-envd -bash scripts/update-debug-rootfs.sh /var/lib/wrenn/images/minimal.ext4 -``` - -## Development - -```bash -make dev # Start PostgreSQL (Docker), run migrations, start control plane -make dev-agent # Start host agent (separate terminal, sudo) -make check # fmt + vet + lint + test -``` +The agent sends heartbeats to the control plane every 30 seconds. See `CLAUDE.md` for full architecture documentation. diff --git a/cmd/host-agent/main.go b/cmd/host-agent/main.go index 6300dcf..5896c2c 100644 --- a/cmd/host-agent/main.go +++ b/cmd/host-agent/main.go @@ -271,13 +271,13 @@ func checkPrivileges() error { bit uint name string }{ - {1, "CAP_DAC_OVERRIDE"}, // /dev/loop*, /dev/mapper/*, /dev/net/tun - {5, "CAP_KILL"}, // SIGTERM/SIGKILL to Firecracker processes - {12, "CAP_NET_ADMIN"}, // netlink, iptables, routing, TAP/veth - {13, "CAP_NET_RAW"}, // raw sockets (iptables) - {19, "CAP_SYS_PTRACE"}, // reading /proc/self/ns/net (netns.Get) - {21, "CAP_SYS_ADMIN"}, // netns, mount ns, losetup, dmsetup - {27, "CAP_MKNOD"}, // device-mapper node creation + {1, "CAP_DAC_OVERRIDE"}, // /dev/loop*, /dev/mapper/*, /dev/net/tun + {5, "CAP_KILL"}, // SIGTERM/SIGKILL to Firecracker processes + {12, "CAP_NET_ADMIN"}, // netlink, iptables, routing, TAP/veth + {13, "CAP_NET_RAW"}, // raw sockets (iptables) + {19, "CAP_SYS_PTRACE"}, // reading /proc/self/ns/net (netns.Get) + {21, "CAP_SYS_ADMIN"}, // netns, mount ns, losetup, dmsetup + {27, "CAP_MKNOD"}, // device-mapper node creation } var missing []string