# Skiff User Guide This guide covers Skiff from two angles: **Developers**, who just want to `git push` their app, and **System Administrators**, who provision and run the FreeBSD host Skiff deploys onto. If you only need the quick reference, see [README.md](README.md); this document goes deeper and includes operational details discovered by actually standing up a host end-to-end. ## Contents - [How Skiff works, in one picture](#how-skiff-works-in-one-picture) - [Part 1: For Developers](#part-1-for-developers) - [Part 2: For System Administrators](#part-2-for-system-administrators) - [Troubleshooting](#troubleshooting) - [Command reference](#command-reference) ## How Skiff works, in one picture ``` you skiff host | | | git push skiff main | |--------------------------->| post-receive hook -> `skiff deploy` | | 1. git worktree add (checkout the sha) | | 2. make -B (must produce ./app) | | 3. zfs snapshot (pre-) | | 4. optional Makefile `release` target | | 5. flip `current` symlink, restart jail | | 6. poll health path for up to 10s | | healthy -> update nginx, done | | unhealthy -> zfs rollback -r, restart | <--------------------------| | push succeeds/fails | ``` Every app lives in its own FreeBSD jail, cloned from a shared ZFS base image. There's no scheduler, no dashboard, no separate metadata database - `jail.conf.d/.conf`, the ZFS dataset, and a `meta.json` file are the only state, and the entire control plane is the one `skiff` binary. --- ## Part 1: For Developers ### What your app needs to provide Skiff has no opinion on your language - the entire build/run contract is a `Makefile` in your repo root: - Running bare `make` (Skiff runs `make -B`, so a stale artifact left over from a previous local run is never trusted - not that one would exist anyway, since every deploy is a brand-new checkout) must leave an executable file named `app` in the repo root. Skiff execs this file directly via `daemon(8)` once it's deployed - the kernel handles interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like a native binary, so *any* language works as long as something ends up executable at that one fixed path. A Go example: ```makefile .PHONY: all all: app app: main.go go.mod go build -o app . ``` A Python example, to show it's not just "compiled languages with an extra step": ```makefile .PHONY: all all: app app: main.py requirements.txt python3 -m venv .venv .venv/bin/pip install -q -r requirements.txt printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app chmod +x app ``` - That executable must listen on the port your sysadmin assigned when they ran `new-app` (default `8080`), and answer the configured health path (default `/healthz`) with a non-3xx status once it's ready to serve traffic. Bind to `:` (all interfaces), not `127.0.0.1:` - each jail has exactly one IP and binding to it explicitly is unnecessary; the wildcard bind resolves correctly inside the jail's own address space. - Optionally, a `release` target in the same Makefile, run once inside the jail *before* cutover - against the **new** release's own checkout (Skiff invokes it as `make -C release`, so relative paths work), not whatever `current` still points at: ```makefile release: sqlite3 /app/data.db < migrations/0004_add_x.sql ``` Skiff never interprets this target - it's entirely your own convention. If it exits non-zero, the deploy aborts *before* touching the live release, so a bad migration never takes your app down. Skiff detects whether you've declared one by asking `make` itself (`make -n release`), not by reading your Makefile's text - it doesn't care how the target is defined, only whether bmake thinks it exists. - Persistent state: `/app/data.db` and `/app/env` (jail-relative paths, siblings of `/app/current`) are meant to be your durable state, untouched by any single release's checkout/build. In practice today this is a *convention*, not something Skiff actively manages for you - there's no automatic env-var injection or database provisioning. If you want a SQLite file that survives every deploy, just open it at `/app/data.db` yourself and it'll sit outside the `releases/` directories that get replaced on every push. ### Deploying Once your sysadmin has run `new-app` for you, they'll give you a remote URL: ``` $ git remote add skiff root@host:/skiff/repos/myapp.git $ git push skiff main ``` **Only pushes to `main` trigger a deploy.** Push to any other branch and the hook no-ops (prints a message, exits 0) - handy for keeping a `skiff` remote around without accidentally deploying WIP branches. The push blocks until the deploy finishes - build, snapshot, cutover, and health check all stream straight back to your terminal: ``` remote: ==> checking out 1a36776... remote: ==> building remote: ==> snapshotting current state remote: ==> no release target declared, skipping remote: ==> cutting over remote: Stopping jails: myapp. remote: Starting jails: myapp. remote: ==> healthy, updating routing remote: ==> deployed myapp at 1a36776... ``` There's no dashboard - if it printed `healthy`, it's live; check the terminal output, not a web UI. ### If the health check fails Skiff automatically rolls the app's entire ZFS dataset back to right before your deploy and restarts the jail on the old release: ``` remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776... remote: Stopping jails: myapp. remote: Starting jails: myapp. remote: error: deploy failed health check and was rolled back: ... ``` The old version is back up by the time you see this error - you did not just take the app down. Common causes, roughly in order of likelihood: 1. Your app doesn't actually bind to the assigned port within 10 seconds (that's the whole health-check budget - retried every 500ms). 2. It's listening on `127.0.0.1` instead of the wildcard address, or on the wrong port. 3. The health path returns a 3xx/4xx/5xx instead of 2xx. 4. It panics/crashes immediately on start - check with your sysadmin, who can `jexec /app/current/app` to see stderr directly, or inspect syslog inside the jail. **One git subtlety worth knowing:** even when the deploy fails and rolls back, the git ref on the server *still advances* to your new commit - git itself doesn't reject the push, only the app-level deploy failed after the fact. So if you fix nothing and just run `git push` again, you'll see `Everything up-to-date` and nothing will happen, because there's no new commit to push. Either make a new commit (even `git commit --allow-empty -m retry` while you investigate) or ask your sysadmin to re-run the deploy directly on the host: `skiff deploy --app myapp --new --ref refs/heads/main`. ### Rolling back a bad release yourself If you have shell access to the host (or your sysadmin does it for you): ``` # skiff rollback --app myapp --list # skiff rollback --app myapp --to pre-a1b2c3d ``` Read the sysadmin section's [rollback notes](#rollback) before you do this in a hurry - it's more destructive than "undo." --- ## Part 2: For System Administrators ### Requirements - FreeBSD 15, ZFS root - `nginx` and `git` - **neither ships in FreeBSD base.** Install them with: ``` # pkg install nginx git-lite ``` - Whatever your apps' Makefiles actually invoke to build - `go`, `python3`, `rustc`, `node`, etc. - installed on the **host** (builds run outside the jail, via `make -B` in the checked-out worktree). Skiff itself doesn't care which; `make` is already in FreeBSD base (bmake), so it's never something you need to install separately. - `jail_enable="YES"` in `/etc/rc.conf`: ``` # sysrc jail_enable=YES ``` Without this, `service jail start`/`restart` refuse to run at all, and every deploy will fail at the cutover step with a jail that never actually starts. ### One-time host setup #### 1. ZFS layout ``` # zfs create zroot/skiff # zfs create -o mountpoint=/skiff/base zroot/skiff/base # bsdinstall jail /skiff/base # zfs snapshot zroot/skiff/base@ready # zfs create -o mountpoint=/skiff/apps zroot/skiff/apps ``` `zroot/skiff/apps` must exist as its own dataset, **not a plain directory** - `new-app` clones straight into it (`zroot/skiff/apps/`), and `zfs clone` requires its parent dataset to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail with `cannot create '...': parent does not exist`. `ready` is the snapshot every app's dataset gets cloned from. Re-snapshot under a new tag after patching the base (e.g. `security-2026-08`) and pass `--base-snapshot` to `new-app` to pick it up. **If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned an interactive `dialog` (distribution-set vs. packages, mirror selection) and has no terminal to answer it - common when driving this over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and re-run fully non-interactive instead: ``` # export nonInteractive=YES # export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE # bsdinstall jail /skiff/base ``` (match the release number to your host's own `uname -r`; check `https://download.freebsd.org/releases/amd64/amd64//MANIFEST` exists before relying on it.) You can watch real progress with `zfs list zroot/skiff/base -o used` even while `dialog`'s own progress bar is garbled without a real tty. #### 2. Repos directory ``` # mkdir -p /skiff/repos ``` #### 3. Nginx Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block: ``` include /usr/local/etc/nginx/skiff.d/*.conf; ``` ``` # mkdir -p /usr/local/etc/nginx/skiff.d # sysrc nginx_enable=YES # service nginx start ``` Skiff only ever writes files under `skiff.d/` and reloads - it never touches `nginx.conf` itself. #### 4. Install skiff Cross-compile from any machine with Go, then copy the binary over: ``` $ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff $ scp skiff root@host:/tmp/skiff # install -m 755 /tmp/skiff /usr/local/sbin/skiff ``` ### Provisioning a new app ``` # skiff new-app --name myapp --ip 10.0.0.5 --port 8080 --server-name myapp.example.com ``` This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside* the jail's own filesystem, not the host's), creates the bare repo at `/skiff/repos/myapp.git` with its `post-receive` hook, and starts the (empty) jail. **Critical, undocumented-by-default step: alias the IP onto the host.** Skiff's jails are classic (non-VNET) jails - `--ip` is an address the jail binds to, but FreeBSD won't route to it unless that address already exists on a host network interface. `new-app` does **not** do this for you: ``` # ifconfig alias 10.0.0.5/24 ``` Without this, the jail starts fine but is completely unreachable - every deploy for that app will fail its health check and roll back forever, even though the app itself is working correctly inside the jail. This was the single most confusing failure mode encountered testing Skiff: the deploy pipeline, build, and app were all fine, and the jail was simply not reachable. If you're provisioning many apps, plan a real IP range up front and script the alias step - see [Known gaps](#known-gaps) below. Make the alias persistent across reboots via `/etc/rc.conf` (`ifconfig__alias0="inet 10.0.0.5/24"`) rather than a one-off `ifconfig` command, or it won't survive a host restart. ### Rollback ``` # skiff rollback --app myapp --list # skiff rollback --app myapp --to pre-a1b2c3d ``` Rolls the app's entire ZFS dataset back to that snapshot (code and any release-phase side effects together) and restarts the jail. This uses `zfs rollback -r`, which **destroys every snapshot newer than the target** - ZFS can't roll back past one otherwise. Rolling back to an old snapshot permanently discards the ability to return to anything deployed in between; there's no separate "undo the rollback." Treat `--to` as a one-way door, not a slider you can move back and forth. Because the bare git repo lives on a *separate* dataset from the app (`/skiff/repos` vs. `/skiff/apps/`), a rollback can delete a release directory whose git worktree is still registered in the repo. Current `skiff` versions self-heal this automatically (checkout runs `git worktree prune` first); if you're running an older binary and see `missing but already registered worktree`, run `git -C /skiff/repos/.git worktree prune` by hand and retry. ### Inspecting a running app ``` # jls # list running jails # jexec myapp ps aux # processes inside the jail # jexec myapp sockstat -4 -l # what's actually listening # jexec myapp service myapp status # the app's own rc.d status # service jail restart myapp # what skiff itself calls on deploy ``` ### Known gaps Carried over from the project's own README, plus what testing surfaced: - **No IP allocation.** `--ip` is operator-chosen and must be manually aliased onto a host interface (see above) - fine for a handful of apps on one host, but you'll want a small pool allocator and an automated alias step beyond that. - **Old `releases/` directories are never pruned.** The ZFS snapshot is the real rollback mechanism; release directories are just there for the instant symlink flip and are safe to clean up by hand once you're confident you won't roll back to them. - **Single host only.** Scaling to a second host is a `zfs send`/`receive` of an app's dataset plus the same jail.conf/nginx steps - not automated. - **`skiff rollback -r` is destructive** (see above) - there is currently no confirmation prompt or dry-run. --- ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` | | `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` | | Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig alias /` (see [Provisioning](#provisioning-a-new-app)) | | `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/.git worktree prune` | | `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) | | `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app --new --ref refs/heads/main` | | `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there | | Health check fails and you're not sure why | - | `jexec sockstat -4 -l` to check for a listener; `jexec ps aux` to see if the process is even running; try `jexec /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow | --- ## Command reference ``` skiff new-app --name --ip [--port 8080] [--health-path /healthz] [--server-name ] [--base-snapshot ready] skiff deploy --app --new [--ref refs/heads/main] [--old ] (normally invoked by the post-receive hook, not by hand) skiff rollback --app --list skiff rollback --app --to ```