Initial commit of skiff scaffolding

1 files changed, 406 insertions, 0 deletions

diff --git a/USER_GUIDE.md b/USER_GUIDE.md
new file mode 100644
index 0000000..5ff93f1
--- /dev/null
+++ b/USER_GUIDE.md
@@ -0,0 +1,406 @@
+# Skiff User Guide
+
+This guide covers Skiff from two angles: **Developers**, who just want to
+`git push` their app, and **System Administrators**, who provision and run
+the FreeBSD host Skiff deploys onto. If you only need the quick reference,
+see [README.md](README.md); this document goes deeper and includes
+operational details discovered by actually standing up a host end-to-end.
+
+## Contents
+
+- [How Skiff works, in one picture](#how-skiff-works-in-one-picture)
+- [Part 1: For Developers](#part-1-for-developers)
+- [Part 2: For System Administrators](#part-2-for-system-administrators)
+- [Troubleshooting](#troubleshooting)
+- [Command reference](#command-reference)
+
+## How Skiff works, in one picture
+
+```
+you                          skiff host
+ |                            |
+ | git push skiff main        |
+ |--------------------------->| post-receive hook -> `skiff deploy`
+ |                            |   1. git worktree add (checkout the sha)
+ |                            |   2. make -B (must produce ./app)
+ |                            |   3. zfs snapshot (pre-<sha>)
+ |                            |   4. optional Makefile `release` target
+ |                            |   5. flip `current` symlink, restart jail
+ |                            |   6. poll health path for up to 10s
+ |                            |        healthy -> update nginx, done
+ |                            |        unhealthy -> zfs rollback -r, restart
+ | <--------------------------|
+ | push succeeds/fails        |
+```
+
+Every app lives in its own FreeBSD jail, cloned from a shared ZFS base
+image. There's no scheduler, no dashboard, no separate metadata database -
+`jail.conf.d/<app>.conf`, the ZFS dataset, and a `meta.json` file are the
+only state, and the entire control plane is the one `skiff` binary.
+
+---
+
+## Part 1: For Developers
+
+### What your app needs to provide
+
+Skiff has no opinion on your language - the entire build/run contract is a
+`Makefile` in your repo root:
+
+- Running bare `make` (Skiff runs `make -B`, so a stale artifact left
+  over from a previous local run is never trusted - not that one would
+  exist anyway, since every deploy is a brand-new checkout) must leave an
+  executable file named `app` in the repo root. Skiff execs this file
+  directly via `daemon(8)` once it's deployed - the kernel handles
+  interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like
+  a native binary, so *any* language works as long as something ends up
+  executable at that one fixed path. A Go example:
+
+  ```makefile
+  .PHONY: all
+  all: app
+
+  app: main.go go.mod
+  	go build -o app .
+  ```
+
+  A Python example, to show it's not just "compiled languages with an
+  extra step":
+
+  ```makefile
+  .PHONY: all
+  all: app
+
+  app: main.py requirements.txt
+  	python3 -m venv .venv
+  	.venv/bin/pip install -q -r requirements.txt
+  	printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app
+  	chmod +x app
+  ```
+
+- That executable must listen on the port your sysadmin assigned when
+  they ran `new-app` (default `8080`), and answer the configured health
+  path (default `/healthz`) with a non-3xx status once it's ready to
+  serve traffic. Bind to `:<port>` (all interfaces), not
+  `127.0.0.1:<port>` - each jail has exactly one IP and binding to it
+  explicitly is unnecessary; the wildcard bind resolves correctly inside
+  the jail's own address space.
+
+- Optionally, a `release` target in the same Makefile, run once inside
+  the jail *before* cutover - against the **new** release's own checkout
+  (Skiff invokes it as `make -C <new-release-dir> release`, so relative
+  paths work), not whatever `current` still points at:
+
+  ```makefile
+  release:
+  	sqlite3 /app/data.db < migrations/0004_add_x.sql
+  ```
+
+  Skiff never interprets this target - it's entirely your own
+  convention. If it exits non-zero, the deploy aborts *before* touching
+  the live release, so a bad migration never takes your app down. Skiff
+  detects whether you've declared one by asking `make` itself
+  (`make -n release`), not by reading your Makefile's text - it doesn't
+  care how the target is defined, only whether bmake thinks it exists.
+
+- Persistent state: `/app/data.db` and `/app/env` (jail-relative paths,
+  siblings of `/app/current`) are meant to be your durable state,
+  untouched by any single release's checkout/build. In practice today
+  this is a *convention*, not something Skiff actively manages for you -
+  there's no automatic env-var injection or database provisioning. If you
+  want a SQLite file that survives every deploy, just open it at
+  `/app/data.db` yourself and it'll sit outside the `releases/<sha>`
+  directories that get replaced on every push.
+
+### Deploying
+
+Once your sysadmin has run `new-app` for you, they'll give you a remote
+URL:
+
+```
+$ git remote add skiff root@host:/skiff/repos/myapp.git
+$ git push skiff main
+```
+
+**Only pushes to `main` trigger a deploy.** Push to any other branch and
+the hook no-ops (prints a message, exits 0) - handy for keeping a
+`skiff` remote around without accidentally deploying WIP branches.
+
+The push blocks until the deploy finishes - build, snapshot, cutover, and
+health check all stream straight back to your terminal:
+
+```
+remote: ==> checking out 1a36776...
+remote: ==> building
+remote: ==> snapshotting current state
+remote: ==> no release target declared, skipping
+remote: ==> cutting over
+remote: Stopping jails: myapp.
+remote: Starting jails: myapp.
+remote: ==> healthy, updating routing
+remote: ==> deployed myapp at 1a36776...
+```
+
+There's no dashboard - if it printed `healthy`, it's live; check the
+terminal output, not a web UI.
+
+### If the health check fails
+
+Skiff automatically rolls the app's entire ZFS dataset back to right
+before your deploy and restarts the jail on the old release:
+
+```
+remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776...
+remote: Stopping jails: myapp.
+remote: Starting jails: myapp.
+remote: error: deploy failed health check and was rolled back: ...
+```
+
+The old version is back up by the time you see this error - you did not
+just take the app down. Common causes, roughly in order of likelihood:
+
+1. Your app doesn't actually bind to the assigned port within 10 seconds
+   (that's the whole health-check budget - retried every 500ms).
+2. It's listening on `127.0.0.1` instead of the wildcard address, or on
+   the wrong port.
+3. The health path returns a 3xx/4xx/5xx instead of 2xx.
+4. It panics/crashes immediately on start - check with your sysadmin,
+   who can `jexec <app> /app/current/app` to see stderr directly, or
+   inspect syslog inside the jail.
+
+**One git subtlety worth knowing:** even when the deploy fails and rolls
+back, the git ref on the server *still advances* to your new commit -
+git itself doesn't reject the push, only the app-level deploy failed
+after the fact. So if you fix nothing and just run `git push` again,
+you'll see `Everything up-to-date` and nothing will happen, because
+there's no new commit to push. Either make a new commit (even
+`git commit --allow-empty -m retry` while you investigate) or ask your
+sysadmin to re-run the deploy directly on the host:
+`skiff deploy --app myapp --new <sha> --ref refs/heads/main`.
+
+### Rolling back a bad release yourself
+
+If you have shell access to the host (or your sysadmin does it for you):
+
+```
+# skiff rollback --app myapp --list
+# skiff rollback --app myapp --to pre-a1b2c3d
+```
+
+Read the sysadmin section's [rollback notes](#rollback) before you do
+this in a hurry - it's more destructive than "undo."
+
+---
+
+## Part 2: For System Administrators
+
+### Requirements
+
+- FreeBSD 15, ZFS root
+- `nginx` and `git` - **neither ships in FreeBSD base.** Install them
+  with:
+  ```
+  # pkg install nginx git-lite
+  ```
+- Whatever your apps' Makefiles actually invoke to build - `go`,
+  `python3`, `rustc`, `node`, etc. - installed on the **host** (builds
+  run outside the jail, via `make -B` in the checked-out worktree). Skiff
+  itself doesn't care which; `make` is already in FreeBSD base (bmake),
+  so it's never something you need to install separately.
+- `jail_enable="YES"` in `/etc/rc.conf`:
+  ```
+  # sysrc jail_enable=YES
+  ```
+  Without this, `service jail start`/`restart` refuse to run at all, and
+  every deploy will fail at the cutover step with a jail that never
+  actually starts.
+
+### One-time host setup
+
+#### 1. ZFS layout
+
+```
+# zfs create zroot/skiff
+# zfs create -o mountpoint=/skiff/base zroot/skiff/base
+# bsdinstall jail /skiff/base
+# zfs snapshot zroot/skiff/base@ready
+# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps
+```
+
+`zroot/skiff/apps` must exist as its own dataset, **not a plain
+directory** - `new-app` clones straight into it
+(`zroot/skiff/apps/<name>`), and `zfs clone` requires its parent dataset
+to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail
+with `cannot create '...': parent does not exist`.
+
+`ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
+under a new tag after patching the base (e.g. `security-2026-08`) and
+pass `--base-snapshot` to `new-app` to pick it up.
+
+**If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned
+an interactive `dialog` (distribution-set vs. packages, mirror
+selection) and has no terminal to answer it - common when driving this
+over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and
+re-run fully non-interactive instead:
+
+```
+# export nonInteractive=YES
+# export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE
+# bsdinstall jail /skiff/base
+```
+
+(match the release number to your host's own `uname -r`; check
+`https://download.freebsd.org/releases/amd64/amd64/<version>/MANIFEST`
+exists before relying on it.) You can watch real progress with
+`zfs list zroot/skiff/base -o used` even while `dialog`'s own progress
+bar is garbled without a real tty.
+
+#### 2. Repos directory
+
+```
+# mkdir -p /skiff/repos
+```
+
+#### 3. Nginx
+
+Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block:
+
+```
+include /usr/local/etc/nginx/skiff.d/*.conf;
+```
+
+```
+# mkdir -p /usr/local/etc/nginx/skiff.d
+# sysrc nginx_enable=YES
+# service nginx start
+```
+
+Skiff only ever writes files under `skiff.d/` and reloads - it never
+touches `nginx.conf` itself.
+
+#### 4. Install skiff
+
+Cross-compile from any machine with Go, then copy the binary over:
+
+```
+$ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff
+$ scp skiff root@host:/tmp/skiff
+# install -m 755 /tmp/skiff /usr/local/sbin/skiff
+```
+
+### Provisioning a new app
+
+```
+# skiff new-app --name myapp --ip 10.0.0.5 --port 8080 --server-name myapp.example.com
+```
+
+This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and
+the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside*
+the jail's own filesystem, not the host's), creates the bare repo at
+`/skiff/repos/myapp.git` with its `post-receive` hook, and starts the
+(empty) jail.
+
+**Critical, undocumented-by-default step: alias the IP onto the host.**
+Skiff's jails are classic (non-VNET) jails - `--ip` is an address the
+jail binds to, but FreeBSD won't route to it unless that address already
+exists on a host network interface. `new-app` does **not** do this for
+you:
+
+```
+# ifconfig <your-interface> alias 10.0.0.5/24
+```
+
+Without this, the jail starts fine but is completely unreachable - every
+deploy for that app will fail its health check and roll back forever,
+even though the app itself is working correctly inside the jail. This
+was the single most confusing failure mode encountered testing Skiff:
+the deploy pipeline, build, and app were all fine, and the jail was
+simply not reachable. If you're provisioning many apps, plan a real IP
+range up front and script the alias step - see [Known
+gaps](#known-gaps) below.
+
+Make the alias persistent across reboots via `/etc/rc.conf`
+(`ifconfig_<if>_alias0="inet 10.0.0.5/24"`) rather than a one-off
+`ifconfig` command, or it won't survive a host restart.
+
+### Rollback
+
+```
+# skiff rollback --app myapp --list
+# skiff rollback --app myapp --to pre-a1b2c3d
+```
+
+Rolls the app's entire ZFS dataset back to that snapshot (code and any
+release-phase side effects together) and restarts the jail.
+
+This uses `zfs rollback -r`, which **destroys every snapshot newer than
+the target** - ZFS can't roll back past one otherwise. Rolling back to
+an old snapshot permanently discards the ability to return to anything
+deployed in between; there's no separate "undo the rollback." Treat
+`--to` as a one-way door, not a slider you can move back and forth.
+
+Because the bare git repo lives on a *separate* dataset from the app
+(`/skiff/repos` vs. `/skiff/apps/<name>`), a rollback can delete a
+release directory whose git worktree is still registered in the repo.
+Current `skiff` versions self-heal this automatically (checkout runs
+`git worktree prune` first); if you're running an older binary and see
+`missing but already registered worktree`, run
+`git -C /skiff/repos/<app>.git worktree prune` by hand and retry.
+
+### Inspecting a running app
+
+```
+# jls                                  # list running jails
+# jexec myapp ps aux                   # processes inside the jail
+# jexec myapp sockstat -4 -l           # what's actually listening
+# jexec myapp service myapp status     # the app's own rc.d status
+# service jail restart myapp           # what skiff itself calls on deploy
+```
+
+### Known gaps
+
+Carried over from the project's own README, plus what testing surfaced:
+
+- **No IP allocation.** `--ip` is operator-chosen and must be manually
+  aliased onto a host interface (see above) - fine for a handful of
+  apps on one host, but you'll want a small pool allocator and an
+  automated alias step beyond that.
+- **Old `releases/<sha>` directories are never pruned.** The ZFS
+  snapshot is the real rollback mechanism; release directories are just
+  there for the instant symlink flip and are safe to clean up by hand
+  once you're confident you won't roll back to them.
+- **Single host only.** Scaling to a second host is a `zfs
+  send`/`receive` of an app's dataset plus the same jail.conf/nginx
+  steps - not automated.
+- **`skiff rollback -r` is destructive** (see above) - there is
+  currently no confirmation prompt or dry-run.
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` |
+| `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` |
+| Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig <if> alias <ip>/<mask>` (see [Provisioning](#provisioning-a-new-app)) |
+| `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` |
+| `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) |
+| `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` |
+| `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there |
+| Health check fails and you're not sure why | - | `jexec <app> sockstat -4 -l` to check for a listener; `jexec <app> ps aux` to see if the process is even running; try `jexec <app> /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow |
+
+---
+
+## Command reference
+
+```
+skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz]
+              [--server-name <name>] [--base-snapshot ready]
+
+skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
+             (normally invoked by the post-receive hook, not by hand)
+
+skiff rollback --app <app> --list
+skiff rollback --app <app> --to <snapshot-tag>
+```