path: root/USER_GUIDE.md

# Skiff User Guide

This guide covers Skiff from two angles: **Developers**, who just want to
`git push` their app, and **System Administrators**, who provision and run
the FreeBSD host Skiff deploys onto. If you only need the quick reference,
see [README.md](README.md); this document goes deeper and includes
operational details discovered by actually standing up a host end-to-end.

## Contents

- [How Skiff works, in one picture](#how-skiff-works-in-one-picture)
- [Part 1: For Developers](#part-1-for-developers)
- [Part 2: For System Administrators](#part-2-for-system-administrators)
- [Troubleshooting](#troubleshooting)
- [Command reference](#command-reference)

## How Skiff works, in one picture

```
you                          skiff host
 |                            |
 | git push skiff main        |
 |--------------------------->| post-receive hook -> `skiff deploy`
 |                            |   1. git worktree add (checkout the sha)
 |                            |   2. make -B (must produce ./app)
 |                            |   3. zfs snapshot (pre-<sha>)
 |                            |   4. optional Makefile `release` target
 |                            |   5. flip `current` symlink, restart jail
 |                            |   6. poll health path for up to 10s
 |                            |        healthy -> update nginx, done
 |                            |        unhealthy -> zfs rollback -r, restart
 | <--------------------------|
 | push succeeds/fails        |
```

Every app lives in its own FreeBSD jail, cloned from a shared ZFS base
image. There's no scheduler, no dashboard, no separate metadata database -
`jail.conf.d/<app>.conf`, the ZFS dataset, and a `meta.json` file are the
only state, and the entire control plane is the one `skiff` binary.

---

## Part 1: For Developers

### What your app needs to provide

Skiff has no opinion on your language - the entire build/run contract is a
`Makefile` in your repo root:

- Running bare `make` (Skiff runs `make -B`, so a stale artifact left
  over from a previous local run is never trusted - not that one would
  exist anyway, since every deploy is a brand-new checkout) must leave an
  executable file named `app` in the repo root. Skiff execs this file
  directly via `daemon(8)` once it's deployed - the kernel handles
  interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like
  a native binary, so *any* language works as long as something ends up
  executable at that one fixed path. A Go example:

  ```makefile
  .PHONY: all
  all: app

  app: main.go go.mod
  	go build -o app .
  ```

  A Python example, to show it's not just "compiled languages with an
  extra step":

  ```makefile
  .PHONY: all
  all: app

  app: main.py requirements.txt
  	python3 -m venv .venv
  	.venv/bin/pip install -q -r requirements.txt
  	printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app
  	chmod +x app
  ```

- That executable must listen on the port your sysadmin assigned when
  they ran `new-app` (default `8080`), and answer the configured health
  path (default `/healthz`) with a non-3xx status once it's ready to
  serve traffic. Bind to `:<port>` (all interfaces), not
  `127.0.0.1:<port>` - each jail has exactly one IP and binding to it
  explicitly is unnecessary; the wildcard bind resolves correctly inside
  the jail's own address space.

- Optionally, a `release` target in the same Makefile, run once inside
  the jail *before* cutover - against the **new** release's own checkout
  (Skiff invokes it as `make -C <new-release-dir> release`, so relative
  paths work), not whatever `current` still points at:

  ```makefile
  release:
  	sqlite3 /app/data.db < migrations/0004_add_x.sql
  ```

  Skiff never interprets this target - it's entirely your own
  convention. If it exits non-zero, the deploy aborts *before* touching
  the live release, so a bad migration never takes your app down. Skiff
  detects whether you've declared one by asking `make` itself
  (`make -n release`), not by reading your Makefile's text - it doesn't
  care how the target is defined, only whether bmake thinks it exists.

- Persistent state: `/app/data.db` and `/app/env` (jail-relative paths,
  siblings of `/app/current`) are your durable state, untouched by any
  single release's checkout/build. If you want a SQLite file that
  survives every deploy, just open it at `/app/data.db` yourself and
  it'll sit outside the `releases/<sha>` directories that get replaced on
  every push.

  `/app/env` is actively managed by Skiff via `skiff config`:

  ```
  # skiff config set --app myapp GREETING=Howdy SECRET=abc123
  # skiff config list --app myapp
  # skiff config unset --app myapp SECRET
  ```

  `set` merges into the existing config (keys you don't mention are left
  alone); `unset` removes the keys you name. Multiple `KEY=VALUE` pairs
  in one `set` call are applied atomically. Every var is plumbed in as a
  plain environment variable - `os.Getenv("GREETING")` in Go, no bespoke
  file format for your app to parse - and reaches both the running app
  and any release-phase Makefile target.

  A config change carries the exact same safety guarantee as a code
  deploy: Skiff snapshots the app's dataset first, restarts the jail,
  waits for the health check, and automatically rolls back to the
  pre-change snapshot if the app doesn't come back healthy. There's no
  weaker, second-tier safety story for config versus code - `skiff
  rollback --list` shows both `pre-<sha>` (deploy) and `precfg-<epoch>`
  (config) snapshots interleaved, and `skiff rollback --to` reverts
  either kind.

### Deploying

Once your sysadmin has run `new-app` for you, they'll give you a remote
URL:

```
$ git remote add skiff root@host:/skiff/repos/myapp.git
$ git push skiff main
```

**Only pushes to `main` trigger a deploy.** Push to any other branch and
the hook no-ops (prints a message, exits 0) - handy for keeping a
`skiff` remote around without accidentally deploying WIP branches.

The push blocks until the deploy finishes - build, snapshot, cutover, and
health check all stream straight back to your terminal:

```
remote: ==> checking out 1a36776...
remote: ==> building
remote: ==> snapshotting current state
remote: ==> no release target declared, skipping
remote: ==> cutting over
remote: Stopping jails: myapp.
remote: Starting jails: myapp.
remote: ==> healthy, updating routing
remote: ==> deployed myapp at 1a36776...
```

There's no dashboard - if it printed `healthy`, it's live; check the
terminal output, not a web UI.

### If the health check fails

Skiff automatically rolls the app's entire ZFS dataset back to right
before your deploy and restarts the jail on the old release:

```
remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776...
remote: Stopping jails: myapp.
remote: Starting jails: myapp.
remote: error: deploy failed health check and was rolled back: ...
```

The old version is back up by the time you see this error - you did not
just take the app down. Common causes, roughly in order of likelihood:

1. Your app doesn't actually bind to the assigned port within 10 seconds
   (that's the whole health-check budget - retried every 500ms).
2. It's listening on `127.0.0.1` instead of the wildcard address, or on
   the wrong port.
3. The health path returns a 3xx/4xx/5xx instead of 2xx.
4. It panics/crashes immediately on start - check with your sysadmin,
   who can `jexec <app> /app/current/app` to see stderr directly, or
   inspect syslog inside the jail.

**One git subtlety worth knowing:** even when the deploy fails and rolls
back, the git ref on the server *still advances* to your new commit -
git itself doesn't reject the push, only the app-level deploy failed
after the fact. So if you fix nothing and just run `git push` again,
you'll see `Everything up-to-date` and nothing will happen, because
there's no new commit to push. Either make a new commit (even
`git commit --allow-empty -m retry` while you investigate) or ask your
sysadmin to re-run the deploy directly on the host:
`skiff deploy --app myapp --new <sha> --ref refs/heads/main`.

### Rolling back a bad release yourself

If you have shell access to the host (or your sysadmin does it for you):

```
# skiff rollback --app myapp --list
# skiff rollback --app myapp --to pre-a1b2c3d
```

Read the sysadmin section's [rollback notes](#rollback) before you do
this in a hurry - it's more destructive than "undo."

---

## Part 2: For System Administrators

### Requirements

- FreeBSD 15, ZFS root
- `nginx` and `git` - **neither ships in FreeBSD base.** Install them
  with:
  ```
  # pkg install nginx git-lite
  ```
- `certbot` - only needed if you'll route real domains via `skiff domains`:
  ```
  # pkg install py312-certbot
  ```
  (the package name tracks the default Python version, so this may drift)
- Whatever your apps' Makefiles actually invoke to build - `go`,
  `python3`, `rustc`, `node`, etc. - baked into the base jail image
  (see [ZFS layout](#1-zfs-layout) below); builds run *inside* the jail
  via `jexec`, not on the host. Skiff itself doesn't care which; `make`
  is already in FreeBSD base (bmake), so it's never something you need
  to install separately. Watch for versioned port names - FreeBSD's
  `go125`, `python312`, etc. install as that literal binary, not a bare
  `go`/`python3`. Check for an unversioned meta-port first (`go` itself
  is one - see [ZFS layout](#1-zfs-layout)); only symlink a default by
  hand if no meta-port exists for what you need. The build step also
  needs real outbound network access from inside the jail
  (module/package downloads) - see the `--ip` note under [Provisioning a
  new app](#provisioning-a-new-app).
- `jail_enable="YES"` in `/etc/rc.conf`:
  ```
  # sysrc jail_enable=YES
  ```
  Without this, `service jail start`/`restart` refuse to run at all, and
  every deploy will fail at the cutover step with a jail that never
  actually starts.

### One-time host setup

#### 1. ZFS layout

```
# zfs create zroot/skiff
# zfs create -o mountpoint=/skiff/base zroot/skiff/base
# bsdinstall jail /skiff/base
```

`bsdinstall jail` only lays down the bare base system - it ships `pkg(8)`'s
bootstrap stub but never actually runs it, so a jail cloned from this alone
has no working `pkg` of its own (files can still be `pkg -c`-installed into
it from the host, but nothing *inside* the jail can manage them, and
`jexec <app> pkg ...` fails with "the package management tool is not yet
installed"). Bootstrap pkg into the image itself, before snapshotting:

```
# mount -t devfs devfs /skiff/base/dev
# env ASSUME_ALWAYS_YES=yes pkg -c /skiff/base bootstrap -f
```

(`-f` because the stub already counts as "pkg is present" otherwise; devfs
is needed for pkg's own network/TLS calls inside a bare chroot - `fetch(1)`
wants `/dev/null` and `/dev/random`.) Now install whatever your apps'
Makefiles actually need to build:

```
# pkg -c /skiff/base install -y go
```

FreeBSD's versioned language ports (`go125`, `python312`, ...) install as
that literal binary name, not a bare `go`/`python3` - a Makefile invoking
the unversioned command needs one. Check for an unversioned meta-port
first: `go` is exactly that - it depends on whichever `goNNN` is current
and installs the `go`/`gofmt` symlinks itself (`pkg install -y go`, not
`go125` directly, is the whole fix). If no meta-port exists for what you
need, symlink a default into `/usr/local/bin` by hand instead.

Unmount devfs and snapshot once the image can actually build and run apps:

```
# umount /skiff/base/dev
# zfs snapshot zroot/skiff/base@ready
# zfs create -o mountpoint=none zroot/skiff/appbase
# zfs create -o mountpoint=none zroot/skiff/appdata
# mkdir -p /skiff/apps
```

`zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their
own dataset, **not a plain directory** - `new-app` clones/creates
straight into them (`zroot/skiff/appbase/<name>`,
`zroot/skiff/appdata/<name>`), and `zfs clone`/`zfs create` require the
parent dataset to already exist. A plain `mkdir` in their place will let
`new-app` fail with `cannot create '...': parent does not exist`.

Their own mountpoint doesn't matter (`none` is fine) - every per-app
dataset sets its own explicit mountpoint under `/skiff/apps/<name>`:
`appbase/<name>` (the OS layer: `/etc`, `/usr`, `/bin`, `jail.conf`, the
jail's rc.d script, `dma.conf`) mounts at `/skiff/apps/<name>` itself,
the jail's own root; `appdata/<name>` (the app's persistent data:
releases, current, env, `meta.json`, `data.db`) mounts nested inside it,
at `/skiff/apps/<name>/app`. `/skiff/apps` itself is just a plain
directory, like `/skiff/repos` - nothing needs to be a dataset at that
exact path.

Keeping the OS layer and the app's data on separate datasets is what
lets `skiff rebuild` (see [Rebuilding a jail](#rebuilding-a-jail))
destroy and reclone the former without any risk to the latter.

`ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
under a new tag after patching the base (e.g. `security-2026-08`) and
pass `--base-snapshot` to `new-app` to pick it up.

**If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned
an interactive `dialog` (distribution-set vs. packages, mirror
selection) and has no terminal to answer it - common when driving this
over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and
re-run fully non-interactive instead:

```
# export nonInteractive=YES
# export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE
# bsdinstall jail /skiff/base
```

(match the release number to your host's own `uname -r`; check
`https://download.freebsd.org/releases/amd64/amd64/<version>/MANIFEST`
exists before relying on it.) You can watch real progress with
`zfs list zroot/skiff/base -o used` even while `dialog`'s own progress
bar is garbled without a real tty.

#### 2. Repos directory

```
# mkdir -p /skiff/repos
```

#### 3. Jail config

FreeBSD doesn't ship an `/etc/jail.conf`, and at boot `service jail
start` (invoked with no jail named) only creates jails it finds in that
one file - it never looks inside `jail.conf.d/` on its own. Point it
there once:

```
# echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf
```

Skip this and everything still *appears* to work: `skiff new-app` starts
the new jail immediately, and a manual `service jail start <app>` after
a reboot works too - both read `jail.conf.d/<app>.conf` directly,
bypassing `/etc/jail.conf` entirely. Only the boot-time, no-name form of
`service jail start` actually depends on the include line, so the gap
stays invisible until the host reboots and no app jail comes back up on
its own - see the [Troubleshooting](#troubleshooting) entry below.

#### 4. Nginx

Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block:

```
include /usr/local/etc/nginx/skiff.d/*.conf;
```

```
# mkdir -p /usr/local/etc/nginx/skiff.d
# sysrc nginx_enable=YES
# service nginx start
```

Skiff only ever writes files under `skiff.d/` and reloads - it never
touches `nginx.conf` itself.

#### 5. TLS (Let's Encrypt)

Skiff issues certificates itself via `certbot certonly` (webroot
validation, never certbot's own `--nginx` plugin - Skiff owns
`skiff.d/*.conf` and regenerates it on every domain/deploy change, and
would fight any other writer of that file). Register the certbot account
once:

```
# certbot register --agree-tos -m you@example.com
```

Renewal rides on FreeBSD's own `periodic(8)`, the same as any other
certbot install - Skiff runs no daemon or scheduler of its own for this,
and `certbot renew` is host-wide by design, so one cron job covers every
app's certificates regardless of when each was issued:

```
# sysrc -f /etc/periodic.conf weekly_certbot_enable="YES"
```

`certonly` mode doesn't reload nginx after a renewal the way certbot's
installer plugins do, so add that as a deploy hook once - certbot runs
every script under this directory after any successful renewal:

```
# mkdir -p /usr/local/etc/letsencrypt/renewal-hooks/deploy
# printf '#!/bin/sh\nnginx -s reload\n' > /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx
# chmod +x /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx
```

#### 6. Install skiff

Cross-compile from any machine with Go, then copy the binary over:

```
$ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff
$ scp skiff root@host:/tmp/skiff
# install -m 755 /tmp/skiff /usr/local/sbin/skiff
```

### Provisioning a new app

```
# skiff new-app --name myapp --ip 192.168.1.50 --port 8080
```

This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and
the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside*
the jail's own filesystem, not the host's), creates the bare repo at
`/skiff/repos/myapp.git` with its `post-receive` hook, and starts the
(empty) jail.

Skiff's jails are classic (non-VNET) jails - `--ip` is an address the
jail binds to, but FreeBSD won't route to it unless that address exists
on a host network interface. `jail.conf.d/myapp.conf` includes an
`interface` line (set from `--iface`, default `vtnet0`), which makes
`jail(8)` itself alias and unalias the IP on that interface as the jail
starts and stops - no manual `ifconfig` step, and it survives a host
reboot the same way `jail_enable="YES"` does. Pass `--iface` if your
host's network interface isn't `vtnet0` (check with `ifconfig`).

`new-app` also appends a `MAILNAME` line to the jail's `dma.conf`,
inheriting `SMARTHOST`/`PORT`/`AUTHPATH`/`SECURETRANSFER`/`STARTTLS`
from the base image, so mail sent from inside the jail (e.g. by cron or
the app itself) is attributed to `myapp.skiff.local` instead of the
base image's own hostname.

**`--ip` must actually route on the host's `--iface` interface** - in
practice, an address in the same subnet as the host's own primary
address on that interface, not an arbitrary block like `10.0.0.0/8`.
Classic (non-VNET) jails share the host's routing table, so an
out-of-subnet address can still send packets - they just may never get a
response, silently, if whatever sits upstream (a NAT gateway, a
virtualized bridge) only answers its own subnet. This didn't matter
before builds moved inside the jail (see [Requirements](#requirements));
now every deploy's build step needs real outbound network access from
that exact address (module downloads, `pkg install`, etc), not just
inbound reachability for the app itself.

### Domains and TLS

`new-app` only provisions the jail - nothing is routed until you add a
domain:

```
# skiff domains add --app myapp --primary myapp.example.com
# skiff domains add --app myapp www.myapp.example.com
# skiff domains list --app myapp
# skiff domains remove --app myapp www.myapp.example.com
```

`add` writes the hostname to the app's `meta.json`, routes it over plain
HTTP immediately (so the ACME HTTP-01 challenge below has somewhere to
land), then requests or expands a single certificate covering every
domain currently registered for the app, named after whichever is
primary - matching how a real multi-domain vhost is normally issued (one
SAN cert), not a cert per hostname.

If DNS isn't pointed at the app's `--ip` yet, certificate issuance fails
and the command exits non-zero - the app stays reachable over HTTP, and
`add` is safe to re-run once DNS is fixed, exactly like retrying a failed
deploy (see [If the health check
fails](#if-the-health-check-fails)). Nothing about a failed `add` is
destructive.

The first domain added for an app becomes primary automatically;
`--primary` reassigns it explicitly. Every non-primary domain
301-redirects to the primary, over HTTP and HTTPS alike, once its own
hostname is covered by the certificate.

`remove` only updates routing - unlike `add`, it never calls out to
certbot, so a DNS or network problem can't make it fail. It's refused if
you try to remove the current primary while other domains remain -
reassign primary first (`skiff domains add --primary <other-hostname>`).

Domain/TLS changes never restart the jail and never run a health check -
unlike a code deploy or `skiff config`, they don't touch the running app
at all, only nginx and the certificate.

### Rollback

```
# skiff rollback --app myapp --list
# skiff rollback --app myapp --to pre-a1b2c3d
```

Rolls the app's entire ZFS dataset back to that snapshot (code and any
release-phase side effects together) and restarts the jail.

This uses `zfs rollback -r`, which **destroys every snapshot newer than
the target** - ZFS can't roll back past one otherwise. Rolling back to
an old snapshot permanently discards the ability to return to anything
deployed in between; there's no separate "undo the rollback." Treat
`--to` as a one-way door, not a slider you can move back and forth.

Because the bare git repo lives on a *separate* dataset from the app
(`/skiff/repos` vs. `/skiff/apps/<name>`), a rollback can delete a
release directory whose git worktree is still registered in the repo.
Current `skiff` versions self-heal this automatically (checkout runs
`git worktree prune` first); if you're running an older binary and see
`missing but already registered worktree`, run
`git -C /skiff/repos/<app>.git worktree prune` by hand and retry.

### Rebuilding a jail

```
# skiff rebuild --app myapp [--base-snapshot ready] [--iface vtnet0]
# skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]
```

Every app's jail is really two independent ZFS datasets: an **OS
layer** (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script,
`dma.conf` - everything `new-app` clones from `zroot/skiff/base`) and a
**data layer** (`/app`: releases, current, env, `meta.json`,
`data.db`). `skiff rebuild` discards the OS layer and reclones it fresh
from `--base-snapshot`, while the data layer - a separate dataset,
never touched - survives untouched. Use it to:

- pick up a patched/upgraded base image on an app that already exists
  (re-snapshot the base under a new tag first - see [ZFS
  layout](#1-zfs-layout) - then pass it as `--base-snapshot`);
- regenerate stale `jail.conf`/rc.d/`dma.conf` on an app provisioned by
  an older `skiff` binary, without touching its data.

Guarded exactly like a deploy or config change: the old OS layer is
kept until the rebuilt jail passes its health check, and automatically
swapped back if it doesn't - `--app`'s rebuild fails loudly, and the
app keeps serving on the OS layer it had before you ran the command.
`--all` rebuilds every app sequentially and stops at the first failure,
so a bad `--base-snapshot` can't cascade through every app on the host.

### Inspecting a running app

```
# jls                                  # list running jails
# jexec myapp ps aux                   # processes inside the jail
# jexec myapp sockstat -4 -l           # what's actually listening
# jexec myapp service myapp status     # the app's own rc.d status
# service jail restart myapp           # what skiff itself calls on deploy
```

### Known gaps

Carried over from the project's own README, plus what testing surfaced:

- **No IP allocation.** `--ip` is operator-chosen - fine for a handful
  of apps on one host, but you'll want a small pool allocator beyond
  that.
- **Old `releases/<sha>` directories are never pruned.** The ZFS
  snapshot is the real rollback mechanism; release directories are just
  there for the instant symlink flip and are safe to clean up by hand
  once you're confident you won't roll back to them.
- **Single host only.** Scaling to a second host is a `zfs
  send`/`receive` of an app's dataset plus the same jail.conf/nginx
  steps - not automated.
- **`skiff rollback -r` is destructive** (see above) - there is
  currently no confirmation prompt or dry-run.
- **`skiff domains remove` doesn't shrink the shared certificate's SAN.**
  A removed hostname may still be listed on the app's certificate until
  the next `add` re-syncs the domain set. Harmless, since nothing routes
  to it anymore.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/appbase` (or `appdata`) was created as a directory, not a dataset | `zfs create -o mountpoint=none zroot/skiff/appbase` (and same for `appdata`) |
| `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` |
| App jails don't come back up after a host reboot, even with `jail_enable=YES` and manual `service jail start <app>` working fine | `/etc/jail.conf` missing or doesn't `.include` `jail.conf.d/*.conf` - boot-time `service jail start` only creates jails it finds in `/etc/jail.conf` itself | `echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf` (see [Jail config](#3-jail-config)) |
| Deploy always health-check-fails with `connection refused`, even though the app builds fine | `--iface` on `new-app`/`rebuild` doesn't name a real interface on this host, so `jail(8)` couldn't alias the IP | Check `ifconfig` for the right interface name, then `skiff rebuild --app <app> --iface <if>` (see [Provisioning](#provisioning-a-new-app)) |
| `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` |
| `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) |
| `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` |
| `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there |
| Health check fails and you're not sure why | - | `jexec <app> sockstat -4 -l` to check for a listener; `jexec <app> ps aux` to see if the process is even running; try `jexec <app> /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow |
| Build fails inside the jail with a missing command (`go: not found`, etc.) even though it's installed on the host | The base jail image itself is missing it, or FreeBSD installed it under a versioned name (`go125`, not `go`) | `jexec <app> which <cmd>`; if it's there under a versioned name, check for an unversioned meta-port (`pkg -c /skiff/base install -y go`, not `go125`) before symlinking one by hand; re-snapshot and `skiff rebuild --app <app> --base-snapshot <new-tag>` either way |
| Build fails inside the jail on any network call (module download, `pkg install`), or `pkg`/`fetch`/`go mod` just time out | The app's `--ip` doesn't actually route from this host - classic jails share the host's routing table, but an out-of-subnet address (e.g. `10.0.0.5` on a host whose real subnet is `192.168.1.0/24`) can send packets that silently never get a response behind some NAT/bridged setups | `jexec <app> fetch -T5 -o /dev/null https://<some-public-ip>` to confirm; if it hangs, re-provision with an `--ip` in the same subnet as the host's own address on `--iface` (see [Provisioning a new app](#provisioning-a-new-app)) |
| `skiff domains add` fails with a certbot/ACME error | Most likely DNS isn't pointed at the app's `--ip` yet; less commonly, the app's `--ip` doesn't route outbound from this host (see above) | Fix DNS, then re-run `skiff domains add` with the same hostname - safe to retry, the app stays on plain HTTP in the meantime and nothing is left half-configured |

---

## Command reference

```
skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz]
              [--base-snapshot ready] [--iface vtnet0]

skiff deploy --app <app> --new <sha> [--ref refs/heads/main]
             (normally invoked by the post-receive hook, not by hand)

skiff rollback --app <app> --list
skiff rollback --app <app> --to <snapshot-tag>

skiff rebuild --app <app> [--base-snapshot ready] [--iface vtnet0]
skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]

skiff config set --app <app> KEY=VALUE [KEY=VALUE...]
skiff config unset --app <app> KEY [KEY...]
skiff config list --app <app>

skiff domains add --app <app> [--primary] <hostname>
skiff domains remove --app <app> <hostname>
skiff domains list --app <app>
```