forked from wrenn/wrenn
chore: update proto, scripts, and docs for CH migration
- Update hostagent proto: firecracker_version → vmm_version in metadata - Regenerate hostagent.pb.go - Update .env.example: WRENN_FIRECRACKER_BIN → WRENN_CH_BIN - Update Makefile: remove --isnotfc from dev-envd target - Update prepare-wrenn-user.sh: firecracker → cloud-hypervisor paths and capability assignments - Update wrenn-init.sh: disable write_zeroes on rootfs for dm-snapshot compatibility with CH - Update README.md and CLAUDE.md: Firecracker → Cloud Hypervisor throughout
This commit is contained in:
@ -16,7 +16,7 @@ WRENN_HOST_LISTEN_ADDR=:50051
|
||||
WRENN_HOST_INTERFACE=eth0
|
||||
WRENN_CP_URL=http://localhost:9725
|
||||
WRENN_DEFAULT_ROOTFS_SIZE=5Gi
|
||||
WRENN_FIRECRACKER_BIN=/usr/local/bin/firecracker
|
||||
WRENN_CH_BIN=/usr/local/bin/cloud-hypervisor
|
||||
|
||||
# Auth
|
||||
JWT_SECRET=
|
||||
|
||||
22
CLAUDE.md
22
CLAUDE.md
@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## Project Overview
|
||||
|
||||
Wrenn Sandbox is a microVM-based code execution platform. Users create isolated sandboxes (Firecracker microVMs), run code inside them, and get output back via SDKs. Think E2B but with persistent sandboxes, pool-based pricing, and a single-binary deployment story.
|
||||
Wrenn Sandbox is a microVM-based code execution platform. Users create isolated sandboxes (Cloud Hypervisor microVMs), run code inside them, and get output back via SDKs. Think E2B but with persistent sandboxes, pool-based pricing, and a single-binary deployment story.
|
||||
|
||||
## Build & Development Commands
|
||||
|
||||
@ -23,11 +23,11 @@ make dev-down # Stop dev infra
|
||||
make dev-cp # Control plane with hot reload (if air installed)
|
||||
make dev-frontend # Vite dev server with HMR (port 5173)
|
||||
make dev-agent # Host agent (sudo required)
|
||||
make dev-envd # envd in debug mode (--isnotfc, port 49983)
|
||||
make dev-envd # envd in debug mode (port 49983)
|
||||
|
||||
make check # fmt + vet + lint + test (CI order)
|
||||
make test # Unit tests: go test -race -v ./internal/...
|
||||
make test-integration # Integration tests (require host agent + Firecracker)
|
||||
make test-integration # Integration tests (require host agent + Cloud Hypervisor)
|
||||
make fmt # gofmt
|
||||
make vet # go vet
|
||||
make lint # golangci-lint
|
||||
@ -92,7 +92,7 @@ Startup (`cmd/host-agent/main.go`) wires: root/capabilities check → enable IP
|
||||
|
||||
- **RPC Server** (`internal/hostagent/server.go`): implements `hostagentv1connect.HostAgentServiceHandler`. Thin wrapper — every method delegates to `sandbox.Manager`. Maps Connect error codes on return.
|
||||
- **Sandbox Manager** (`internal/sandbox/manager.go`): the core orchestration layer. Maintains in-memory state in `boxes map[string]*sandboxState` (protected by `sync.RWMutex`). Each `sandboxState` holds a `models.Sandbox`, a `*network.Slot`, and an `*envdclient.Client`. Runs a TTL reaper (every 10s) that auto-destroys timed-out sandboxes.
|
||||
- **VM Manager** (`internal/vm/manager.go`, `fc.go`, `config.go`): manages Firecracker processes. Uses raw HTTP API over Unix socket (`/tmp/fc-{sandboxID}.sock`), not the firecracker-go-sdk Machine type. Launches Firecracker via `unshare -m` + `ip netns exec`. Configures VM via PUT to `/boot-source`, `/drives/rootfs`, `/network-interfaces/eth0`, `/machine-config`, then starts with PUT `/actions`.
|
||||
- **VM Manager** (`internal/vm/manager.go`, `ch.go`, `config.go`): manages Cloud Hypervisor processes. Uses raw HTTP API over Unix socket (`/tmp/ch-{sandboxID}.sock`). Launches Cloud Hypervisor via `unshare -m` + `ip netns exec` with `--api-socket path=...`. Configures and boots VM via `PUT /vm.create` + `PUT /vm.boot`. Snapshot restore uses `--restore source_url=file://...`.
|
||||
- **Network** (`internal/network/setup.go`, `allocator.go`): per-sandbox network namespace with veth pair + TAP device. See Networking section below.
|
||||
- **Device Mapper** (`internal/devicemapper/devicemapper.go`): CoW rootfs via device-mapper snapshots. Shared read-only loop devices per base template (refcounted `LoopRegistry`), per-sandbox sparse CoW files, dm-snapshot create/restore/remove/flatten operations.
|
||||
- **envd Client** (`internal/envdclient/client.go`, `health.go`): dual interface to the guest agent. Connect RPC for streaming process exec (`process.Start()` bidirectional stream). Plain HTTP for file operations (POST/GET `/files?path=...&username=root`). Health check polls `GET /health` every 100ms until ready (30s timeout).
|
||||
@ -109,7 +109,7 @@ Runs as PID 1 inside the microVM via `wrenn-init.sh` (mounts procfs/sysfs/dev, s
|
||||
- **HTTP endpoints**: GET `/health`, GET `/metrics`, POST `/init`, POST `/snapshot/prepare`, GET/POST `/files`
|
||||
- **Proto codegen**: `connectrpc-build` compiles `proto/envd/*.proto` at `cargo build` time via `build.rs` — no committed stubs
|
||||
- **Build**: `make build-envd` → static musl binary in `builds/envd`
|
||||
- **Dev**: `make dev-envd` → `cargo run -- --isnotfc --port 49983`
|
||||
- **Dev**: `make dev-envd` → `cargo run -- --port 49983`
|
||||
|
||||
### Dashboard (Frontend)
|
||||
|
||||
@ -164,7 +164,7 @@ HIBERNATED → RUNNING (cold snapshot resume, slower)
|
||||
**Sandbox creation** (`POST /v1/capsules`):
|
||||
1. API handler generates sandbox ID, inserts into DB as "pending"
|
||||
2. RPC `CreateSandbox` → host agent → `sandbox.Manager.Create()`
|
||||
3. Manager: resolve base rootfs → acquire shared loop device → create dm-snapshot (sparse CoW file) → allocate network slot → `CreateNetwork()` (netns + veth + tap + NAT) → `vm.Create()` (start Firecracker with `/dev/mapper/wrenn-{id}`, configure via HTTP API, boot) → `envdclient.WaitUntilReady()` (poll /health) → store in-memory state
|
||||
3. Manager: resolve base rootfs → acquire shared loop device → create dm-snapshot (sparse CoW file) → allocate network slot → `CreateNetwork()` (netns + veth + tap + NAT) → `vm.Create()` (start Cloud Hypervisor with `/dev/mapper/wrenn-{id}`, configure via `PUT /vm.create` + `PUT /vm.boot`) → `envdclient.WaitUntilReady()` (poll /health) → store in-memory state
|
||||
4. API handler updates DB to "running" with host_ip
|
||||
|
||||
**Command execution** (`POST /v1/capsules/{id}/exec`):
|
||||
@ -210,9 +210,9 @@ To add a new query: add it to the appropriate `.sql` file in `db/queries/` → `
|
||||
|
||||
- **Connect RPC** (not gRPC) for all RPC communication between components
|
||||
- **Buf + protoc-gen-connect-go** for Go code generation; **connectrpc-build** for Rust code generation in envd
|
||||
- **Raw Firecracker HTTP API** via Unix socket (not firecracker-go-sdk Machine type)
|
||||
- **Raw Cloud Hypervisor HTTP API** via Unix socket (`PUT /vm.create` + `PUT /vm.boot`)
|
||||
- **TAP networking** (not vsock) for host-to-envd communication
|
||||
- **Device-mapper snapshots** for rootfs CoW — shared read-only loop device per base template, per-sandbox sparse CoW file, Firecracker gets `/dev/mapper/wrenn-{id}`
|
||||
- **Device-mapper snapshots** for rootfs CoW — shared read-only loop device per base template, per-sandbox sparse CoW file, Cloud Hypervisor gets `/dev/mapper/wrenn-{id}`
|
||||
- **PostgreSQL** via pgx/v5 + sqlc (type-safe query generation). Goose for migrations (plain SQL, up/down)
|
||||
- **Dashboard**: SvelteKit (Svelte 5, adapter-static) + Tailwind CSS v4 + Bits UI. Built to static files in `frontend/build/`, served by Caddy (not embedded in the Go binary)
|
||||
- **Lago** for billing (external service, not in this codebase)
|
||||
@ -237,19 +237,19 @@ To add a new query: add it to the appropriate `.sql` file in `db/queries/` → `
|
||||
- Kernel: `/var/lib/wrenn/kernels/vmlinux`
|
||||
- Base rootfs images: `/var/lib/wrenn/images/{template}.ext4`
|
||||
- Sandbox clones: `/var/lib/wrenn/sandboxes/`
|
||||
- Firecracker: `/usr/local/bin/firecracker` (e2b's fork of firecracker)
|
||||
- Cloud Hypervisor: `/usr/local/bin/cloud-hypervisor`
|
||||
|
||||
## Design Context
|
||||
|
||||
### Users
|
||||
Developers across the full spectrum — solo engineers building side projects, startup teams integrating sandboxed execution into products, and platform/infra engineers at larger organizations running production workloads on Firecracker microVMs. They arrive with context: they know what a process is, what a rootfs is, what a TTY means. The interface must feel at home for all three: approachable enough not to intimidate a hacker, precise enough to earn the trust of a production ops team. Never condescend, never oversimplify. Trust the user to understand what they're looking at.
|
||||
Developers across the full spectrum — solo engineers building side projects, startup teams integrating sandboxed execution into products, and platform/infra engineers at larger organizations running production workloads on Cloud Hypervisor microVMs. They arrive with context: they know what a process is, what a rootfs is, what a TTY means. The interface must feel at home for all three: approachable enough not to intimidate a hacker, precise enough to earn the trust of a production ops team. Never condescend, never oversimplify. Trust the user to understand what they're looking at.
|
||||
|
||||
**Primary job to be done:** Understand what's running, act on it confidently, and get back to code.
|
||||
|
||||
### Brand Personality
|
||||
**Precise. Warm. Uncompromising.**
|
||||
|
||||
Wrenn is an engineer's favorite tool — built with visible care, not assembled from defaults. It runs real infrastructure (Firecracker microVMs), so the UI should reflect that seriousness without becoming cold or corporate. The warmth comes from the typography and color palette; the precision comes from hierarchy, density, and data fidelity.
|
||||
Wrenn is an engineer's favorite tool — built with visible care, not assembled from defaults. It runs real infrastructure (Cloud Hypervisor microVMs), so the UI should reflect that seriousness without becoming cold or corporate. The warmth comes from the typography and color palette; the precision comes from hierarchy, density, and data fidelity.
|
||||
|
||||
Emotional goal: **in control.** Users leave a session with full confidence in what's running, what happened, and what comes next. Nothing is hidden, nothing is ambiguous.
|
||||
|
||||
|
||||
4
Makefile
4
Makefile
@ -62,7 +62,7 @@ dev-frontend:
|
||||
cd frontend && bun run dev --port 5173 --host 0.0.0.0
|
||||
|
||||
dev-envd:
|
||||
cd envd-rs && cargo run -- --isnotfc --port 49983
|
||||
cd envd-rs && cargo run -- --port 49983
|
||||
|
||||
# ═══════════════════════════════════════════════════
|
||||
# Database (goose)
|
||||
@ -181,7 +181,7 @@ help:
|
||||
@echo " make dev-cp Control plane (hot reload if air installed)"
|
||||
@echo " make dev-frontend Vite dev server with HMR (port 5173)"
|
||||
@echo " make dev-agent Host agent (sudo required)"
|
||||
@echo " make dev-envd envd in debug mode (--isnotfc, port 49983)"
|
||||
@echo " make dev-envd envd in debug mode (port 49983)"
|
||||
@echo ""
|
||||
@echo " make build Build all binaries → builds/"
|
||||
@echo " make build-frontend Build SvelteKit dashboard → frontend/build/"
|
||||
|
||||
@ -5,7 +5,7 @@ Secure infrastructure for AI
|
||||
## Prerequisites
|
||||
|
||||
- Linux host with `/dev/kvm` access (bare metal or nested virt)
|
||||
- Firecracker binary at `/usr/local/bin/firecracker`
|
||||
- Cloud Hypervisor binary at `/usr/local/bin/cloud-hypervisor`
|
||||
- PostgreSQL
|
||||
- Go 1.25+
|
||||
- Rust 1.88+ with `x86_64-unknown-linux-musl` target (`rustup target add x86_64-unknown-linux-musl`)
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
#!/bin/sh
|
||||
# wrenn-init: minimal PID 1 init for Firecracker microVMs.
|
||||
# wrenn-init: minimal PID 1 init for Cloud Hypervisor microVMs.
|
||||
# Mounts virtual filesystems, starts chronyd for time sync, then execs tini + envd.
|
||||
|
||||
set -e
|
||||
@ -17,6 +17,11 @@ mkdir -p /sys/fs/cgroup
|
||||
mount -t cgroup2 cgroup2 /sys/fs/cgroup 2>/dev/null || true
|
||||
echo "+cpu +memory +io" > /sys/fs/cgroup/cgroup.subtree_control 2>/dev/null || true
|
||||
|
||||
# Disable write_zeroes on rootfs — dm-snapshot doesn't support BLKZEROOUT,
|
||||
# and CH advertises the feature anyway. Without this, every zeroing IO
|
||||
# hits EOPNOTSUPP and CH spams warnings. Only writable on kernel 6.6+.
|
||||
echo 0 > /sys/block/vda/queue/write_zeroes_max_bytes 2>/dev/null || true
|
||||
|
||||
# Set hostname and make it resolvable (sudo requires this).
|
||||
hostname capsule
|
||||
echo "127.0.0.1 capsule" >> /etc/hosts
|
||||
|
||||
@ -155,7 +155,7 @@ type CreateSandboxResponse struct {
|
||||
Status string `protobuf:"bytes,2,opt,name=status,proto3" json:"status,omitempty"`
|
||||
HostIp string `protobuf:"bytes,3,opt,name=host_ip,json=hostIp,proto3" json:"host_ip,omitempty"`
|
||||
// Runtime metadata collected during sandbox creation (e.g. envd_version,
|
||||
// kernel_version, firecracker_version, agent_version).
|
||||
// kernel_version, vmm_version, agent_version).
|
||||
Metadata map[string]string `protobuf:"bytes,4,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
|
||||
@ -146,7 +146,7 @@ message CreateSandboxResponse {
|
||||
string host_ip = 3;
|
||||
|
||||
// Runtime metadata collected during sandbox creation (e.g. envd_version,
|
||||
// kernel_version, firecracker_version, agent_version).
|
||||
// kernel_version, vmm_version, agent_version).
|
||||
map<string, string> metadata = 4;
|
||||
}
|
||||
|
||||
|
||||
@ -22,7 +22,7 @@
|
||||
# Prerequisites:
|
||||
# - wrenn-agent binary at /usr/local/bin/wrenn-agent
|
||||
# - wrenn-cp binary at /usr/local/bin/wrenn-cp
|
||||
# - firecracker binary at /usr/local/bin/firecracker
|
||||
# - cloud-hypervisor binary at /usr/local/bin/cloud-hypervisor
|
||||
# - libcap2-bin installed (for setcap)
|
||||
|
||||
set -euo pipefail
|
||||
@ -41,7 +41,7 @@ WRENN_GROUP="wrenn"
|
||||
WRENN_DIR="/var/lib/wrenn"
|
||||
AGENT_BIN="/usr/local/bin/wrenn-agent"
|
||||
CP_BIN="/usr/local/bin/wrenn-cp"
|
||||
FC_BIN="/usr/local/bin/firecracker"
|
||||
CH_BIN="/usr/local/bin/cloud-hypervisor"
|
||||
RESTORE_CAPS_SCRIPT="/etc/wrenn/restore-caps.sh"
|
||||
|
||||
# ── 1. Create system user ───────────────────────────────────────────────────
|
||||
@ -100,7 +100,7 @@ done
|
||||
# routing table manipulation
|
||||
# CAP_NET_RAW — raw socket access (needed by iptables internally)
|
||||
# CAP_SYS_PTRACE — reading /proc/self/ns/net (netns.Get)
|
||||
# CAP_KILL — sending SIGTERM/SIGKILL to Firecracker processes
|
||||
# CAP_KILL — sending SIGTERM/SIGKILL to Cloud Hypervisor processes
|
||||
# CAP_DAC_OVERRIDE — accessing /dev/loop*, /dev/mapper/*, /dev/net/tun,
|
||||
# /proc/sys/net/ipv4/ip_forward
|
||||
# CAP_MKNOD — creating device nodes (dm-snapshot)
|
||||
@ -120,12 +120,12 @@ else
|
||||
getcap "${AGENT_BIN}"
|
||||
fi
|
||||
|
||||
# Firecracker also needs capabilities when spawned by a non-root parent.
|
||||
# Cloud Hypervisor also needs capabilities when spawned by a non-root parent.
|
||||
# CAP_NET_ADMIN is required for network device access inside the netns.
|
||||
if [[ -f "${FC_BIN}" ]]; then
|
||||
setcap cap_net_admin,cap_sys_admin,cap_dac_override+ep "${FC_BIN}"
|
||||
echo " Capabilities set on ${FC_BIN}:"
|
||||
getcap "${FC_BIN}"
|
||||
if [[ -f "${CH_BIN}" ]]; then
|
||||
setcap cap_net_admin,cap_sys_admin,cap_dac_override+ep "${CH_BIN}"
|
||||
echo " Capabilities set on ${CH_BIN}:"
|
||||
getcap "${CH_BIN}"
|
||||
fi
|
||||
|
||||
# ── Helper: resolve binary path and apply setcap ────────────────────────────
|
||||
@ -191,13 +191,13 @@ setcap_binary() {
|
||||
setcap "$caps" "$bin" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# wrenn-agent and firecracker (only if present — they aren't package-managed).
|
||||
# wrenn-agent and cloud-hypervisor (only if present — they aren't package-managed).
|
||||
[[ -f /usr/local/bin/wrenn-agent ]] && \
|
||||
setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_sys_ptrace,cap_kill,cap_dac_override,cap_mknod+ep \
|
||||
/usr/local/bin/wrenn-agent 2>/dev/null || true
|
||||
[[ -f /usr/local/bin/firecracker ]] && \
|
||||
[[ -f /usr/local/bin/cloud-hypervisor ]] && \
|
||||
setcap cap_net_admin,cap_sys_admin,cap_dac_override+ep \
|
||||
/usr/local/bin/firecracker 2>/dev/null || true
|
||||
/usr/local/bin/cloud-hypervisor 2>/dev/null || true
|
||||
|
||||
# Child binaries (these are the ones wiped by apt).
|
||||
setcap_binary iptables "cap_net_admin,cap_net_raw+ep"
|
||||
@ -315,14 +315,14 @@ ExecStart=/usr/local/bin/wrenn-agent --address ${WRENN_ADVERTISE_ADDR}
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
# File descriptor limits (Firecracker + loop devices + sockets).
|
||||
# File descriptor limits (Cloud Hypervisor + loop devices + sockets).
|
||||
LimitNOFILE=65536
|
||||
LimitNPROC=4096
|
||||
|
||||
# Protect host filesystem — only allow access to what's needed.
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/var/lib/wrenn /tmp /run/netns /dev/mapper
|
||||
ReadOnlyPaths=/usr/local/bin/firecracker
|
||||
ReadOnlyPaths=/usr/local/bin/cloud-hypervisor
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
Reference in New Issue
Block a user