forked from wrenn/wrenn
v0.1.0 (#17)
This commit is contained in:
133
README.md
133
README.md
@ -2,16 +2,16 @@
|
||||
|
||||
Secure infrastructure for AI
|
||||
|
||||
## Deployment
|
||||
|
||||
### Prerequisites
|
||||
## Prerequisites
|
||||
|
||||
- Linux host with `/dev/kvm` access (bare metal or nested virt)
|
||||
- Firecracker binary at `/usr/local/bin/firecracker`
|
||||
- PostgreSQL
|
||||
- Go 1.25+
|
||||
- pnpm (for frontend)
|
||||
- Docker (for dev infra and rootfs builds)
|
||||
|
||||
### Build
|
||||
## Build
|
||||
|
||||
```bash
|
||||
make build # outputs to builds/
|
||||
@ -19,30 +19,77 @@ make build # outputs to builds/
|
||||
|
||||
Produces three binaries: `wrenn-cp` (control plane), `wrenn-agent` (host agent), `envd` (guest agent).
|
||||
|
||||
### Host setup
|
||||
## Host setup
|
||||
|
||||
The host agent machine needs:
|
||||
The host agent needs a kernel, a minimal rootfs image, and working directories on the host machine.
|
||||
|
||||
```bash
|
||||
# Kernel for guest VMs
|
||||
mkdir -p /var/lib/wrenn/kernels
|
||||
# Place a vmlinux kernel at /var/lib/wrenn/kernels/vmlinux
|
||||
### Directory structure
|
||||
|
||||
# Rootfs images
|
||||
mkdir -p /var/lib/wrenn/images
|
||||
# Build or place .ext4 rootfs images (e.g., minimal.ext4)
|
||||
|
||||
# Sandbox working directory
|
||||
mkdir -p /var/lib/wrenn/sandboxes
|
||||
|
||||
# Snapshots directory
|
||||
mkdir -p /var/lib/wrenn/snapshots
|
||||
|
||||
# Enable IP forwarding
|
||||
sysctl -w net.ipv4.ip_forward=1
|
||||
```
|
||||
/var/lib/wrenn/
|
||||
├── kernels/
|
||||
│ └── vmlinux # uncompressed Linux kernel (not bzImage)
|
||||
├── images/
|
||||
│ └── minimal/
|
||||
│ └── rootfs.ext4 # base rootfs (all other templates snapshot from this)
|
||||
├── sandboxes/ # per-sandbox CoW files (created at runtime)
|
||||
└── snapshots/ # pause/hibernate snapshot files (created at runtime)
|
||||
```
|
||||
|
||||
### Configure
|
||||
Create the directories:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /var/lib/wrenn/{kernels,images/minimal,sandboxes,snapshots}
|
||||
```
|
||||
|
||||
### Kernel
|
||||
|
||||
Place an uncompressed `vmlinux` kernel at `/var/lib/wrenn/kernels/vmlinux`. Versioned kernels (`vmlinux-{semver}`) are also supported — the agent picks the latest by semver.
|
||||
|
||||
### Minimal rootfs
|
||||
|
||||
The minimal rootfs is the base image that all other templates (Python, Node, etc.) are built on top of via device-mapper snapshots. It must contain:
|
||||
|
||||
| Package | Why |
|
||||
|---------|-----|
|
||||
| `socat` | Bidirectional relay for port forwarding |
|
||||
| `chrony` | Time sync from KVM PTP clock (`/dev/ptp0`) |
|
||||
| `tini` | PID 1 zombie reaper (injected by build script, not apt) |
|
||||
| `sudo` | User privilege management inside the guest |
|
||||
| `wget` | HTTP fetching |
|
||||
| `curl` | HTTP client |
|
||||
| `ca-certificates` | TLS certificate verification |
|
||||
|
||||
**To build a rootfs from a Docker container:**
|
||||
|
||||
1. Create and configure a container with the required packages:
|
||||
```bash
|
||||
docker run -it --name wrenn-minimal debian:bookworm bash
|
||||
# Inside the container:
|
||||
apt update && apt install -y socat chrony sudo wget curl ca-certificates
|
||||
exit
|
||||
```
|
||||
|
||||
2. Export to a rootfs image (builds envd, injects wrenn-init + tini, shrinks to minimum size):
|
||||
```bash
|
||||
sudo bash scripts/rootfs-from-container.sh wrenn-minimal minimal
|
||||
```
|
||||
|
||||
**To update an existing rootfs** after changing envd or `wrenn-init.sh`:
|
||||
|
||||
```bash
|
||||
bash scripts/update-minimal-rootfs.sh
|
||||
```
|
||||
|
||||
This rebuilds envd via `make build-envd` and copies the fresh binaries into the mounted rootfs image.
|
||||
|
||||
### IP forwarding
|
||||
|
||||
```bash
|
||||
sudo sysctl -w net.ipv4.ip_forward=1
|
||||
```
|
||||
|
||||
## Configure
|
||||
|
||||
Copy `.env.example` to `.env` and edit:
|
||||
|
||||
@ -59,25 +106,21 @@ WRENN_HOST_LISTEN_ADDR=:50051
|
||||
WRENN_DIR=/var/lib/wrenn
|
||||
```
|
||||
|
||||
### Run
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Apply database migrations
|
||||
make migrate-up
|
||||
|
||||
# Start control plane
|
||||
./builds/wrenn-cp
|
||||
make dev # Start PostgreSQL (Docker), run migrations, start control plane
|
||||
make dev-agent # Start host agent (separate terminal, sudo)
|
||||
make dev-frontend # Vite dev server with HMR (port 5173)
|
||||
make check # fmt + vet + lint + test
|
||||
```
|
||||
|
||||
Control plane listens on `WRENN_CP_LISTEN_ADDR` (default `:8000`).
|
||||
|
||||
### Host registration
|
||||
|
||||
Hosts must be registered with the control plane before they can serve sandboxes.
|
||||
|
||||
1. **Create a host record** (via API or dashboard):
|
||||
```bash
|
||||
# As an admin (JWT auth)
|
||||
curl -X POST http://localhost:8000/v1/hosts \
|
||||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
@ -87,17 +130,16 @@ Hosts must be registered with the control plane before they can serve sandboxes.
|
||||
|
||||
2. **Start the host agent** with the registration token and its externally-reachable address:
|
||||
```bash
|
||||
sudo WRENN_CP_URL=http://cp-host:8000 \
|
||||
sudo WRENN_CP_URL=http://localhost:8000 \
|
||||
./builds/wrenn-agent \
|
||||
--register <token-from-step-1> \
|
||||
--address 10.0.1.5:50051
|
||||
--address <host-ip>:50051
|
||||
```
|
||||
On first startup the agent sends its specs (arch, CPU, memory, disk) to the control plane, receives a long-lived host JWT, and saves it to `$WRENN_DIR/host-token`.
|
||||
|
||||
3. **Subsequent startups** don't need `--register` — the agent loads the saved JWT automatically:
|
||||
```bash
|
||||
sudo WRENN_CP_URL=http://cp-host:8000 \
|
||||
./builds/wrenn-agent --address 10.0.1.5:50051
|
||||
sudo ./builds/wrenn-agent --address <host-ip>:50051
|
||||
```
|
||||
|
||||
4. **If registration fails** (e.g., network error after token was consumed), regenerate a token:
|
||||
@ -107,23 +149,6 @@ Hosts must be registered with the control plane before they can serve sandboxes.
|
||||
```
|
||||
Then restart the agent with the new token.
|
||||
|
||||
The agent sends heartbeats to the control plane every 30 seconds. Host agent listens on `WRENN_HOST_LISTEN_ADDR` (default `:50051`).
|
||||
|
||||
### Rootfs images
|
||||
|
||||
envd must be baked into every rootfs image. After building:
|
||||
|
||||
```bash
|
||||
make build-envd
|
||||
bash scripts/update-debug-rootfs.sh /var/lib/wrenn/images/minimal.ext4
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
make dev # Start PostgreSQL (Docker), run migrations, start control plane
|
||||
make dev-agent # Start host agent (separate terminal, sudo)
|
||||
make check # fmt + vet + lint + test
|
||||
```
|
||||
The agent sends heartbeats to the control plane every 30 seconds.
|
||||
|
||||
See `CLAUDE.md` for full architecture documentation.
|
||||
|
||||
Reference in New Issue
Block a user