diff options
Diffstat (limited to 'USER_GUIDE.md')
| -rw-r--r-- | USER_GUIDE.md | 406 |
1 files changed, 406 insertions, 0 deletions
diff --git a/USER_GUIDE.md b/USER_GUIDE.md new file mode 100644 index 0000000..5ff93f1 --- /dev/null +++ b/USER_GUIDE.md @@ -0,0 +1,406 @@ +# Skiff User Guide + +This guide covers Skiff from two angles: **Developers**, who just want to +`git push` their app, and **System Administrators**, who provision and run +the FreeBSD host Skiff deploys onto. If you only need the quick reference, +see [README.md](README.md); this document goes deeper and includes +operational details discovered by actually standing up a host end-to-end. + +## Contents + +- [How Skiff works, in one picture](#how-skiff-works-in-one-picture) +- [Part 1: For Developers](#part-1-for-developers) +- [Part 2: For System Administrators](#part-2-for-system-administrators) +- [Troubleshooting](#troubleshooting) +- [Command reference](#command-reference) + +## How Skiff works, in one picture + +``` +you skiff host + | | + | git push skiff main | + |--------------------------->| post-receive hook -> `skiff deploy` + | | 1. git worktree add (checkout the sha) + | | 2. make -B (must produce ./app) + | | 3. zfs snapshot (pre-<sha>) + | | 4. optional Makefile `release` target + | | 5. flip `current` symlink, restart jail + | | 6. poll health path for up to 10s + | | healthy -> update nginx, done + | | unhealthy -> zfs rollback -r, restart + | <--------------------------| + | push succeeds/fails | +``` + +Every app lives in its own FreeBSD jail, cloned from a shared ZFS base +image. There's no scheduler, no dashboard, no separate metadata database - +`jail.conf.d/<app>.conf`, the ZFS dataset, and a `meta.json` file are the +only state, and the entire control plane is the one `skiff` binary. + +--- + +## Part 1: For Developers + +### What your app needs to provide + +Skiff has no opinion on your language - the entire build/run contract is a +`Makefile` in your repo root: + +- Running bare `make` (Skiff runs `make -B`, so a stale artifact left + over from a previous local run is never trusted - not that one would + exist anyway, since every deploy is a brand-new checkout) must leave an + executable file named `app` in the repo root. Skiff execs this file + directly via `daemon(8)` once it's deployed - the kernel handles + interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like + a native binary, so *any* language works as long as something ends up + executable at that one fixed path. A Go example: + + ```makefile + .PHONY: all + all: app + + app: main.go go.mod + go build -o app . + ``` + + A Python example, to show it's not just "compiled languages with an + extra step": + + ```makefile + .PHONY: all + all: app + + app: main.py requirements.txt + python3 -m venv .venv + .venv/bin/pip install -q -r requirements.txt + printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app + chmod +x app + ``` + +- That executable must listen on the port your sysadmin assigned when + they ran `new-app` (default `8080`), and answer the configured health + path (default `/healthz`) with a non-3xx status once it's ready to + serve traffic. Bind to `:<port>` (all interfaces), not + `127.0.0.1:<port>` - each jail has exactly one IP and binding to it + explicitly is unnecessary; the wildcard bind resolves correctly inside + the jail's own address space. + +- Optionally, a `release` target in the same Makefile, run once inside + the jail *before* cutover - against the **new** release's own checkout + (Skiff invokes it as `make -C <new-release-dir> release`, so relative + paths work), not whatever `current` still points at: + + ```makefile + release: + sqlite3 /app/data.db < migrations/0004_add_x.sql + ``` + + Skiff never interprets this target - it's entirely your own + convention. If it exits non-zero, the deploy aborts *before* touching + the live release, so a bad migration never takes your app down. Skiff + detects whether you've declared one by asking `make` itself + (`make -n release`), not by reading your Makefile's text - it doesn't + care how the target is defined, only whether bmake thinks it exists. + +- Persistent state: `/app/data.db` and `/app/env` (jail-relative paths, + siblings of `/app/current`) are meant to be your durable state, + untouched by any single release's checkout/build. In practice today + this is a *convention*, not something Skiff actively manages for you - + there's no automatic env-var injection or database provisioning. If you + want a SQLite file that survives every deploy, just open it at + `/app/data.db` yourself and it'll sit outside the `releases/<sha>` + directories that get replaced on every push. + +### Deploying + +Once your sysadmin has run `new-app` for you, they'll give you a remote +URL: + +``` +$ git remote add skiff root@host:/skiff/repos/myapp.git +$ git push skiff main +``` + +**Only pushes to `main` trigger a deploy.** Push to any other branch and +the hook no-ops (prints a message, exits 0) - handy for keeping a +`skiff` remote around without accidentally deploying WIP branches. + +The push blocks until the deploy finishes - build, snapshot, cutover, and +health check all stream straight back to your terminal: + +``` +remote: ==> checking out 1a36776... +remote: ==> building +remote: ==> snapshotting current state +remote: ==> no release target declared, skipping +remote: ==> cutting over +remote: Stopping jails: myapp. +remote: Starting jails: myapp. +remote: ==> healthy, updating routing +remote: ==> deployed myapp at 1a36776... +``` + +There's no dashboard - if it printed `healthy`, it's live; check the +terminal output, not a web UI. + +### If the health check fails + +Skiff automatically rolls the app's entire ZFS dataset back to right +before your deploy and restarts the jail on the old release: + +``` +remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776... +remote: Stopping jails: myapp. +remote: Starting jails: myapp. +remote: error: deploy failed health check and was rolled back: ... +``` + +The old version is back up by the time you see this error - you did not +just take the app down. Common causes, roughly in order of likelihood: + +1. Your app doesn't actually bind to the assigned port within 10 seconds + (that's the whole health-check budget - retried every 500ms). +2. It's listening on `127.0.0.1` instead of the wildcard address, or on + the wrong port. +3. The health path returns a 3xx/4xx/5xx instead of 2xx. +4. It panics/crashes immediately on start - check with your sysadmin, + who can `jexec <app> /app/current/app` to see stderr directly, or + inspect syslog inside the jail. + +**One git subtlety worth knowing:** even when the deploy fails and rolls +back, the git ref on the server *still advances* to your new commit - +git itself doesn't reject the push, only the app-level deploy failed +after the fact. So if you fix nothing and just run `git push` again, +you'll see `Everything up-to-date` and nothing will happen, because +there's no new commit to push. Either make a new commit (even +`git commit --allow-empty -m retry` while you investigate) or ask your +sysadmin to re-run the deploy directly on the host: +`skiff deploy --app myapp --new <sha> --ref refs/heads/main`. + +### Rolling back a bad release yourself + +If you have shell access to the host (or your sysadmin does it for you): + +``` +# skiff rollback --app myapp --list +# skiff rollback --app myapp --to pre-a1b2c3d +``` + +Read the sysadmin section's [rollback notes](#rollback) before you do +this in a hurry - it's more destructive than "undo." + +--- + +## Part 2: For System Administrators + +### Requirements + +- FreeBSD 15, ZFS root +- `nginx` and `git` - **neither ships in FreeBSD base.** Install them + with: + ``` + # pkg install nginx git-lite + ``` +- Whatever your apps' Makefiles actually invoke to build - `go`, + `python3`, `rustc`, `node`, etc. - installed on the **host** (builds + run outside the jail, via `make -B` in the checked-out worktree). Skiff + itself doesn't care which; `make` is already in FreeBSD base (bmake), + so it's never something you need to install separately. +- `jail_enable="YES"` in `/etc/rc.conf`: + ``` + # sysrc jail_enable=YES + ``` + Without this, `service jail start`/`restart` refuse to run at all, and + every deploy will fail at the cutover step with a jail that never + actually starts. + +### One-time host setup + +#### 1. ZFS layout + +``` +# zfs create zroot/skiff +# zfs create -o mountpoint=/skiff/base zroot/skiff/base +# bsdinstall jail /skiff/base +# zfs snapshot zroot/skiff/base@ready +# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps +``` + +`zroot/skiff/apps` must exist as its own dataset, **not a plain +directory** - `new-app` clones straight into it +(`zroot/skiff/apps/<name>`), and `zfs clone` requires its parent dataset +to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail +with `cannot create '...': parent does not exist`. + +`ready` is the snapshot every app's dataset gets cloned from. Re-snapshot +under a new tag after patching the base (e.g. `security-2026-08`) and +pass `--base-snapshot` to `new-app` to pick it up. + +**If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned +an interactive `dialog` (distribution-set vs. packages, mirror +selection) and has no terminal to answer it - common when driving this +over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and +re-run fully non-interactive instead: + +``` +# export nonInteractive=YES +# export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE +# bsdinstall jail /skiff/base +``` + +(match the release number to your host's own `uname -r`; check +`https://download.freebsd.org/releases/amd64/amd64/<version>/MANIFEST` +exists before relying on it.) You can watch real progress with +`zfs list zroot/skiff/base -o used` even while `dialog`'s own progress +bar is garbled without a real tty. + +#### 2. Repos directory + +``` +# mkdir -p /skiff/repos +``` + +#### 3. Nginx + +Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block: + +``` +include /usr/local/etc/nginx/skiff.d/*.conf; +``` + +``` +# mkdir -p /usr/local/etc/nginx/skiff.d +# sysrc nginx_enable=YES +# service nginx start +``` + +Skiff only ever writes files under `skiff.d/` and reloads - it never +touches `nginx.conf` itself. + +#### 4. Install skiff + +Cross-compile from any machine with Go, then copy the binary over: + +``` +$ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff +$ scp skiff root@host:/tmp/skiff +# install -m 755 /tmp/skiff /usr/local/sbin/skiff +``` + +### Provisioning a new app + +``` +# skiff new-app --name myapp --ip 10.0.0.5 --port 8080 --server-name myapp.example.com +``` + +This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and +the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside* +the jail's own filesystem, not the host's), creates the bare repo at +`/skiff/repos/myapp.git` with its `post-receive` hook, and starts the +(empty) jail. + +**Critical, undocumented-by-default step: alias the IP onto the host.** +Skiff's jails are classic (non-VNET) jails - `--ip` is an address the +jail binds to, but FreeBSD won't route to it unless that address already +exists on a host network interface. `new-app` does **not** do this for +you: + +``` +# ifconfig <your-interface> alias 10.0.0.5/24 +``` + +Without this, the jail starts fine but is completely unreachable - every +deploy for that app will fail its health check and roll back forever, +even though the app itself is working correctly inside the jail. This +was the single most confusing failure mode encountered testing Skiff: +the deploy pipeline, build, and app were all fine, and the jail was +simply not reachable. If you're provisioning many apps, plan a real IP +range up front and script the alias step - see [Known +gaps](#known-gaps) below. + +Make the alias persistent across reboots via `/etc/rc.conf` +(`ifconfig_<if>_alias0="inet 10.0.0.5/24"`) rather than a one-off +`ifconfig` command, or it won't survive a host restart. + +### Rollback + +``` +# skiff rollback --app myapp --list +# skiff rollback --app myapp --to pre-a1b2c3d +``` + +Rolls the app's entire ZFS dataset back to that snapshot (code and any +release-phase side effects together) and restarts the jail. + +This uses `zfs rollback -r`, which **destroys every snapshot newer than +the target** - ZFS can't roll back past one otherwise. Rolling back to +an old snapshot permanently discards the ability to return to anything +deployed in between; there's no separate "undo the rollback." Treat +`--to` as a one-way door, not a slider you can move back and forth. + +Because the bare git repo lives on a *separate* dataset from the app +(`/skiff/repos` vs. `/skiff/apps/<name>`), a rollback can delete a +release directory whose git worktree is still registered in the repo. +Current `skiff` versions self-heal this automatically (checkout runs +`git worktree prune` first); if you're running an older binary and see +`missing but already registered worktree`, run +`git -C /skiff/repos/<app>.git worktree prune` by hand and retry. + +### Inspecting a running app + +``` +# jls # list running jails +# jexec myapp ps aux # processes inside the jail +# jexec myapp sockstat -4 -l # what's actually listening +# jexec myapp service myapp status # the app's own rc.d status +# service jail restart myapp # what skiff itself calls on deploy +``` + +### Known gaps + +Carried over from the project's own README, plus what testing surfaced: + +- **No IP allocation.** `--ip` is operator-chosen and must be manually + aliased onto a host interface (see above) - fine for a handful of + apps on one host, but you'll want a small pool allocator and an + automated alias step beyond that. +- **Old `releases/<sha>` directories are never pruned.** The ZFS + snapshot is the real rollback mechanism; release directories are just + there for the instant symlink flip and are safe to clean up by hand + once you're confident you won't roll back to them. +- **Single host only.** Scaling to a second host is a `zfs + send`/`receive` of an app's dataset plus the same jail.conf/nginx + steps - not automated. +- **`skiff rollback -r` is destructive** (see above) - there is + currently no confirmation prompt or dry-run. + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` | +| `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` | +| Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig <if> alias <ip>/<mask>` (see [Provisioning](#provisioning-a-new-app)) | +| `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` | +| `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) | +| `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` | +| `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there | +| Health check fails and you're not sure why | - | `jexec <app> sockstat -4 -l` to check for a listener; `jexec <app> ps aux` to see if the process is even running; try `jexec <app> /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow | + +--- + +## Command reference + +``` +skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz] + [--server-name <name>] [--base-snapshot ready] + +skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>] + (normally invoked by the post-receive hook, not by hand) + +skiff rollback --app <app> --list +skiff rollback --app <app> --to <snapshot-tag> +``` |