path: root/USER_GUIDE.md

# Skiff User Guide

This guide covers Skiff from two angles: **Developers**, who just want to
`git push` their app, and **System Administrators**, who provision and run
the FreeBSD host Skiff deploys onto. If you only need the quick reference,
see [README.md](README.md); this document goes deeper and includes
operational details discovered by actually standing up a host end-to-end.

## Contents

- [How Skiff works, in one picture](#how-skiff-works-in-one-picture)
- [Part 1: For Developers](#part-1-for-developers)
- [Part 2: For System Administrators](#part-2-for-system-administrators)
- [Troubleshooting](#troubleshooting)
- [Command reference](#command-reference)

## How Skiff works, in one picture

```
you                          skiff host
 |                            |
 | git push skiff main        |
 |--------------------------->| post-receive hook -> `skiff deploy`
 |                            |   1. git worktree add (checkout the sha)
 |                            |   2. make -B (must produce ./app)
 |                            |   3. zfs snapshot (pre-<sha>)
 |                            |   4. optional Makefile `release` target
 |                            |   5. flip `current` symlink, restart jail
 |                            |   6. poll health path for up to 10s
 |                            |        healthy -> update nginx, done
 |                            |        unhealthy -> zfs rollback -r, restart
 | <--------------------------|
 | push succeeds/fails        |
```

Every app lives in its own FreeBSD jail, cloned from a shared ZFS base
image. There's no scheduler, no dashboard, no separate metadata database -
`jail.conf.d/<app>.conf`, the ZFS dataset, and a `meta.json` file are the
only state, and the entire control plane is the one `skiff` binary.

---

## Part 1: For Developers

### What your app needs to provide

Skiff has no opinion on your language - the entire build/run contract is a
`Makefile` in your repo root:

- Running bare `make` (Skiff runs `make -B`, so a stale artifact left
  over from a previous local run is never trusted - not that one would
  exist anyway, since every deploy is a brand-new checkout) must leave an
  executable file named `app` in the repo root. Skiff execs this file
  directly via `daemon(8)` once it's deployed - the kernel handles
  interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like
  a native binary, so *any* language works as long as something ends up
  executable at that one fixed path. A Go example:

  ```makefile
  .PHONY: all
  all: app

  app: main.go go.mod
  	go build -o app .
  ```

  A Python example, to show it's not just "compiled languages with an
  extra step":

  ```makefile
  .PHONY: all
  all: app

  app: main.py requirements.txt
  	python3 -m venv .venv
  	.venv/bin/pip install -q -r requirements.txt
  	printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app
  	chmod +x app
  ```

- That executable must listen on the port your sysadmin assigned when
  they ran `new-app` (default `8080`), and answer the configured health
  path (default `/healthz`) with a non-3xx status once it's ready to
  serve traffic. Bind to `:<port>` (all interfaces), not
  `127.0.0.1:<port>` - each jail has exactly one IP and binding to it
  explicitly is unnecessary; the wildcard bind resolves correctly inside
  the jail's own address space.

- Optionally, a `release` target in the same Makefile, run once inside
  the jail *before* cutover - against the **new** release's own checkout
  (Skiff invokes it as `make -C <new-release-dir> release`, so relative
  paths work), not whatever `current` still points at:

  ```makefile
  release:
  	sqlite3 /app/data.db < migrations/0004_add_x.sql
  ```

  Skiff never interprets this target - it's entirely your own
  convention. If it exits non-zero, the deploy aborts *before* touching
  the live release, so a bad migration never takes your app down. Skiff
  detects whether you've declared one by asking `make` itself
  (`make -n release`), not by reading your Makefile's text - it doesn't
  care how the target is defined, only whether bmake thinks it exists.

- Persistent state: `/app/data.db` and `/app/env` (jail-relative paths,
  siblings of `/app/current`) are your durable state, untouched by any
  single release's checkout/build. If you want a SQLite file that
  survives every deploy, just open it at `/app/data.db` yourself and
  it'll sit outside the `releases/<sha>` directories that get replaced on
  every push.

  `/app/env` is actively managed by Skiff via `skiff config`:

  ```
  # skiff config set --app myapp GREETING=Howdy SECRET=abc123
  # skiff config list --app myapp
  # skiff config unset --app myapp SECRET
  ```

  `set` merges into the existing config (keys you don't mention are left
  alone); `unset` removes the keys you name. Multiple `KEY=VALUE` pairs
  in one `set` call are applied atomically. Every var is plumbed in as a
  plain environment variable - `os.Getenv("GREETING")` in Go, no bespoke
  file format for your app to parse - and reaches both the running app
  and any release-phase Makefile target.

  A config change carries the exact same safety guarantee as a code
  deploy: Skiff snapshots the app's dataset first, restarts the jail,
  waits for the health check, and automatically rolls back to the
  pre-change snapshot if the app doesn't come back healthy. There's no
  weaker, second-tier safety story for config versus code - `skiff
  rollback --list` shows both `pre-<sha>` (deploy) and `precfg-<epoch>`
  (config) snapshots interleaved, and `skiff rollback --to` reverts
  either kind.

### Deploying

Once your sysadmin has run `new-app` for you, they'll give you a remote
URL:

```
$ git remote add skiff root@host:/skiff/repos/myapp.git
$ git push skiff main
```

**Only pushes to `main` trigger a deploy.** Push to any other branch and
the hook no-ops (prints a message, exits 0) - handy for keeping a
`skiff` remote around without accidentally deploying WIP branches.

The push blocks until the deploy finishes - build, snapshot, cutover, and
health check all stream straight back to your terminal:

```
remote: ==> checking out 1a36776...
remote: ==> building
remote: ==> snapshotting current state
remote: ==> no release target declared, skipping
remote: ==> cutting over
remote: Stopping jails: myapp.
remote: Starting jails: myapp.
remote: ==> healthy, updating routing
remote: ==> deployed myapp at 1a36776...
```

There's no dashboard - if it printed `healthy`, it's live; check the
terminal output, not a web UI.

### If the health check fails

Skiff automatically rolls the app's entire ZFS dataset back to right
before your deploy and restarts the jail on the old release:

```
remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776...
remote: Stopping jails: myapp.
remote: Starting jails: myapp.
remote: error: deploy failed health check and was rolled back: ...
```

The old version is back up by the time you see this error - you did not
just take the app down. Common causes, roughly in order of likelihood:

1. Your app doesn't actually bind to the assigned port within 10 seconds
   (that's the whole health-check budget - retried every 500ms).
2. It's listening on `127.0.0.1` instead of the wildcard address, or on
   the wrong port.
3. The health path returns a 3xx/4xx/5xx instead of 2xx.
4. It panics/crashes immediately on start - check with your sysadmin,
   who can `jexec <app> /app/current/app` to see stderr directly, or
   inspect syslog inside the jail.

**One git subtlety worth knowing:** even when the deploy fails and rolls
back, the git ref on the server *still advances* to your new commit -
git itself doesn't reject the push, only the app-level deploy failed
after the fact. So if you fix nothing and just run `git push` again,
you'll see `Everything up-to-date` and nothing will happen, because
there's no new commit to push. Either make a new commit (even
`git commit --allow-empty -m retry` while you investigate) or ask your
sysadmin to re-run the deploy directly on the host:
`skiff deploy --app myapp --new <sha> --ref refs/heads/main`.

### Rolling back a bad release yourself

If you have shell access to the host (or your sysadmin does it for you):

```
# skiff rollback --app myapp --list
# skiff rollback --app myapp --to pre-a1b2c3d
```

Read the sysadmin section's [rollback notes](#rollback) before you do
this in a hurry - it's more destructive than "undo."

---

## Part 2: For System Administrators

### Requirements

- FreeBSD 15, ZFS root
- `nginx` and `git` - **neither ships in FreeBSD base.** Install them
  with:
  ```
  # pkg install nginx git-lite
  ```
- Whatever your apps' Makefiles actually invoke to build - `go`,
  `python3`, `rustc`, `node`, etc. - installed on the **host** (builds
  run outside the jail, via `make -B` in the checked-out worktree). Skiff
  itself doesn't care which; `make` is already in FreeBSD base (bmake),
  so it's never something you need to install separately.
- `jail_enable="YES"` in `/etc/rc.conf`:
  ```
  # sysrc jail_enable=YES
  ```
  Without this, `service jail start`/`restart` refuse to run at all, and
  every deploy will fail at the cutover step with a jail that never
  actually starts.

### One-time host setup

#### 1. ZFS layout

```
# zfs create zroot/skiff
# zfs create -o mountpoint=/skiff/base zroot/skiff/base
# bsdinstall jail /skiff/base
# zfs snapshot zroot/skiff/base@ready
# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps
```

`zroot/skiff/apps` must exist as its own dataset, **not a plain
directory** - `new-app` clones straight into it
(`zroot/skiff/apps/<name>`), and `zfs clone` requires its parent dataset
to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail
with `cannot create '...': parent does not exist`.

`ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
under a new tag after patching the base (e.g. `security-2026-08`) and
pass `--base-snapshot` to `new-app` to pick it up.

**If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned
an interactive `dialog` (distribution-set vs. packages, mirror
selection) and has no terminal to answer it - common when driving this
over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and
re-run fully non-interactive instead:

```
# export nonInteractive=YES
# export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE
# bsdinstall jail /skiff/base
```

(match the release number to your host's own `uname -r`; check
`https://download.freebsd.org/releases/amd64/amd64/<version>/MANIFEST`
exists before relying on it.) You can watch real progress with
`zfs list zroot/skiff/base -o used` even while `dialog`'s own progress
bar is garbled without a real tty.

#### 2. Repos directory

```
# mkdir -p /skiff/repos
```

#### 3. Jail config

FreeBSD doesn't ship an `/etc/jail.conf`, and at boot `service jail
start` (invoked with no jail named) only creates jails it finds in that
one file - it never looks inside `jail.conf.d/` on its own. Point it
there once:

```
# echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf
```

Skip this and everything still *appears* to work: `skiff new-app` starts
the new jail immediately, and a manual `service jail start <app>` after
a reboot works too - both read `jail.conf.d/<app>.conf` directly,
bypassing `/etc/jail.conf` entirely. Only the boot-time, no-name form of
`service jail start` actually depends on the include line, so the gap
stays invisible until the host reboots and no app jail comes back up on
its own - see the [Troubleshooting](#troubleshooting) entry below.

#### 4. Nginx

Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block:

```
include /usr/local/etc/nginx/skiff.d/*.conf;
```

```
# mkdir -p /usr/local/etc/nginx/skiff.d
# sysrc nginx_enable=YES
# service nginx start
```

Skiff only ever writes files under `skiff.d/` and reloads - it never
touches `nginx.conf` itself.

#### 5. Install skiff

Cross-compile from any machine with Go, then copy the binary over:

```
$ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff
$ scp skiff root@host:/tmp/skiff
# install -m 755 /tmp/skiff /usr/local/sbin/skiff
```

### Provisioning a new app

```
# skiff new-app --name myapp --ip 10.0.0.5 --port 8080 --server-name myapp.example.com
```

This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and
the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside*
the jail's own filesystem, not the host's), creates the bare repo at
`/skiff/repos/myapp.git` with its `post-receive` hook, and starts the
(empty) jail.

**Critical, undocumented-by-default step: alias the IP onto the host.**
Skiff's jails are classic (non-VNET) jails - `--ip` is an address the
jail binds to, but FreeBSD won't route to it unless that address already
exists on a host network interface. `new-app` does **not** do this for
you:

```
# ifconfig <your-interface> alias 10.0.0.5/24
```

Without this, the jail starts fine but is completely unreachable - every
deploy for that app will fail its health check and roll back forever,
even though the app itself is working correctly inside the jail. This
was the single most confusing failure mode encountered testing Skiff:
the deploy pipeline, build, and app were all fine, and the jail was
simply not reachable. If you're provisioning many apps, plan a real IP
range up front and script the alias step - see [Known
gaps](#known-gaps) below.

Make the alias persistent across reboots via `/etc/rc.conf`
(`ifconfig_<if>_alias0="inet 10.0.0.5/24"`) rather than a one-off
`ifconfig` command, or it won't survive a host restart.

### Rollback

```
# skiff rollback --app myapp --list
# skiff rollback --app myapp --to pre-a1b2c3d
```

Rolls the app's entire ZFS dataset back to that snapshot (code and any
release-phase side effects together) and restarts the jail.

This uses `zfs rollback -r`, which **destroys every snapshot newer than
the target** - ZFS can't roll back past one otherwise. Rolling back to
an old snapshot permanently discards the ability to return to anything
deployed in between; there's no separate "undo the rollback." Treat
`--to` as a one-way door, not a slider you can move back and forth.

Because the bare git repo lives on a *separate* dataset from the app
(`/skiff/repos` vs. `/skiff/apps/<name>`), a rollback can delete a
release directory whose git worktree is still registered in the repo.
Current `skiff` versions self-heal this automatically (checkout runs
`git worktree prune` first); if you're running an older binary and see
`missing but already registered worktree`, run
`git -C /skiff/repos/<app>.git worktree prune` by hand and retry.

### Inspecting a running app

```
# jls                                  # list running jails
# jexec myapp ps aux                   # processes inside the jail
# jexec myapp sockstat -4 -l           # what's actually listening
# jexec myapp service myapp status     # the app's own rc.d status
# service jail restart myapp           # what skiff itself calls on deploy
```

### Known gaps

Carried over from the project's own README, plus what testing surfaced:

- **No IP allocation.** `--ip` is operator-chosen and must be manually
  aliased onto a host interface (see above) - fine for a handful of
  apps on one host, but you'll want a small pool allocator and an
  automated alias step beyond that.
- **Old `releases/<sha>` directories are never pruned.** The ZFS
  snapshot is the real rollback mechanism; release directories are just
  there for the instant symlink flip and are safe to clean up by hand
  once you're confident you won't roll back to them.
- **Single host only.** Scaling to a second host is a `zfs
  send`/`receive` of an app's dataset plus the same jail.conf/nginx
  steps - not automated.
- **`skiff rollback -r` is destructive** (see above) - there is
  currently no confirmation prompt or dry-run.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` |
| `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` |
| App jails don't come back up after a host reboot, even with `jail_enable=YES` and manual `service jail start <app>` working fine | `/etc/jail.conf` missing or doesn't `.include` `jail.conf.d/*.conf` - boot-time `service jail start` only creates jails it finds in `/etc/jail.conf` itself | `echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf` (see [Jail config](#3-jail-config)) |
| Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig <if> alias <ip>/<mask>` (see [Provisioning](#provisioning-a-new-app)) |
| `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` |
| `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) |
| `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` |
| `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there |
| Health check fails and you're not sure why | - | `jexec <app> sockstat -4 -l` to check for a listener; `jexec <app> ps aux` to see if the process is even running; try `jexec <app> /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow |

---

## Command reference

```
skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz]
              [--server-name <name>] [--base-snapshot ready]

skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
             (normally invoked by the post-receive hook, not by hand)

skiff rollback --app <app> --list
skiff rollback --app <app> --to <snapshot-tag>

skiff config set --app <app> KEY=VALUE [KEY=VALUE...]
skiff config unset --app <app> KEY [KEY...]
skiff config list --app <app>
```