# Skiff User Guide This guide covers Skiff from two angles: **Developers**, who just want to `git push` their app, and **System Administrators**, who provision and run the FreeBSD host Skiff deploys onto. If you only need the quick reference, see [README.md](README.md); this document goes deeper and includes operational details discovered by actually standing up a host end-to-end. ## Contents - [How Skiff works, in one picture](#how-skiff-works-in-one-picture) - [Part 1: For Developers](#part-1-for-developers) - [Part 2: For System Administrators](#part-2-for-system-administrators) - [Troubleshooting](#troubleshooting) - [Command reference](#command-reference) ## How Skiff works, in one picture ``` you skiff host | | | git push skiff main | |--------------------------->| post-receive hook -> `skiff deploy` | | 1. git worktree add (checkout the sha) | | 2. make -B (must produce ./app) | | 3. zfs snapshot (pre-) | | 4. optional Makefile `release` target | | 5. flip `current` symlink, restart jail | | 6. poll health path for up to 10s | | healthy -> update nginx, done | | unhealthy -> zfs rollback -r, restart | <--------------------------| | push succeeds/fails | ``` Every app lives in its own FreeBSD jail, cloned from a shared ZFS base image. There's no scheduler, no dashboard, no separate metadata database - `jail.conf.d/.conf`, the ZFS dataset, and a `meta.json` file are the only state, and the entire control plane is the one `skiff` binary. --- ## Part 1: For Developers ### What your app needs to provide Skiff has no opinion on your language - the entire build/run contract is a `Makefile` in your repo root: - Running bare `make` (Skiff runs `make -B`, so a stale artifact left over from a previous local run is never trusted - not that one would exist anyway, since every deploy is a brand-new checkout) must leave an executable file named `app` in the repo root. Skiff execs this file directly via `daemon(8)` once it's deployed - the kernel handles interpreter shebang lines (`#!/usr/bin/env python3`, etc.) exactly like a native binary, so *any* language works as long as something ends up executable at that one fixed path. A Go example: ```makefile .PHONY: all all: app app: main.go go.mod go build -o app . ``` A Python example, to show it's not just "compiled languages with an extra step": ```makefile .PHONY: all all: app app: main.py requirements.txt python3 -m venv .venv .venv/bin/pip install -q -r requirements.txt printf '#!/bin/sh\nexec %s/.venv/bin/python3 %s/main.py\n' "$$(pwd)" "$$(pwd)" > app chmod +x app ``` - That executable must listen on the port your sysadmin assigned when they ran `new-app` (default `8080`), and answer the configured health path (default `/healthz`) with a non-3xx status once it's ready to serve traffic. Bind to `:` (all interfaces), not `127.0.0.1:` - each jail has exactly one IP and binding to it explicitly is unnecessary; the wildcard bind resolves correctly inside the jail's own address space. - Optionally, a `release` target in the same Makefile, run once inside the jail *before* cutover - against the **new** release's own checkout (Skiff invokes it as `make -C release`, so relative paths work), not whatever `current` still points at: ```makefile release: sqlite3 /app/data.db < migrations/0004_add_x.sql ``` Skiff never interprets this target - it's entirely your own convention. If it exits non-zero, the deploy aborts *before* touching the live release, so a bad migration never takes your app down. Skiff detects whether you've declared one by asking `make` itself (`make -n release`), not by reading your Makefile's text - it doesn't care how the target is defined, only whether bmake thinks it exists. - Persistent state: `/app/data.db` and `/app/env` (jail-relative paths, siblings of `/app/current`) are your durable state, untouched by any single release's checkout/build. If you want a SQLite file that survives every deploy, just open it at `/app/data.db` yourself and it'll sit outside the `releases/` directories that get replaced on every push. `/app/env` is actively managed by Skiff via `skiff config`: ``` # skiff config set --app myapp GREETING=Howdy SECRET=abc123 # skiff config list --app myapp # skiff config unset --app myapp SECRET ``` `set` merges into the existing config (keys you don't mention are left alone); `unset` removes the keys you name. Multiple `KEY=VALUE` pairs in one `set` call are applied atomically. Every var is plumbed in as a plain environment variable - `os.Getenv("GREETING")` in Go, no bespoke file format for your app to parse - and reaches both the running app and any release-phase Makefile target. A config change carries the exact same safety guarantee as a code deploy: Skiff snapshots the app's dataset first, restarts the jail, waits for the health check, and automatically rolls back to the pre-change snapshot if the app doesn't come back healthy. There's no weaker, second-tier safety story for config versus code - `skiff rollback --list` shows both `pre-` (deploy) and `precfg-` (config) snapshots interleaved, and `skiff rollback --to` reverts either kind. ### Deploying Once your sysadmin has run `new-app` for you, they'll give you a remote URL: ``` $ git remote add skiff root@host:/skiff/repos/myapp.git $ git push skiff main ``` **Only pushes to `main` trigger a deploy.** Push to any other branch and the hook no-ops (prints a message, exits 0) - handy for keeping a `skiff` remote around without accidentally deploying WIP branches. The push blocks until the deploy finishes - build, snapshot, cutover, and health check all stream straight back to your terminal: ``` remote: ==> checking out 1a36776... remote: ==> building remote: ==> snapshotting current state remote: ==> no release target declared, skipping remote: ==> cutting over remote: Stopping jails: myapp. remote: Starting jails: myapp. remote: ==> healthy, updating routing remote: ==> deployed myapp at 1a36776... ``` There's no dashboard - if it printed `healthy`, it's live; check the terminal output, not a web UI. ### If the health check fails Skiff automatically rolls the app's entire ZFS dataset back to right before your deploy and restarts the jail on the old release: ``` remote: ==> health check failed (never became healthy at http://10.0.0.5:8080/healthz: ...), rolling back to pre-1a36776... remote: Stopping jails: myapp. remote: Starting jails: myapp. remote: error: deploy failed health check and was rolled back: ... ``` The old version is back up by the time you see this error - you did not just take the app down. Common causes, roughly in order of likelihood: 1. Your app doesn't actually bind to the assigned port within 10 seconds (that's the whole health-check budget - retried every 500ms). 2. It's listening on `127.0.0.1` instead of the wildcard address, or on the wrong port. 3. The health path returns a 3xx/4xx/5xx instead of 2xx. 4. It panics/crashes immediately on start - check with your sysadmin, who can `jexec /app/current/app` to see stderr directly, or inspect syslog inside the jail. **One git subtlety worth knowing:** even when the deploy fails and rolls back, the git ref on the server *still advances* to your new commit - git itself doesn't reject the push, only the app-level deploy failed after the fact. So if you fix nothing and just run `git push` again, you'll see `Everything up-to-date` and nothing will happen, because there's no new commit to push. Either make a new commit (even `git commit --allow-empty -m retry` while you investigate) or ask your sysadmin to re-run the deploy directly on the host: `skiff deploy --app myapp --new --ref refs/heads/main`. ### Rolling back a bad release yourself If you have shell access to the host (or your sysadmin does it for you): ``` # skiff rollback --app myapp --list # skiff rollback --app myapp --to pre-a1b2c3d ``` Read the sysadmin section's [rollback notes](#rollback) before you do this in a hurry - it's more destructive than "undo." --- ## Part 2: For System Administrators ### Requirements - FreeBSD 15, ZFS root - `nginx` and `git` - **neither ships in FreeBSD base.** Install them with: ``` # pkg install nginx git-lite ``` - `certbot` - only needed if you'll route real domains via `skiff domains`: ``` # pkg install py312-certbot ``` (the package name tracks the default Python version, so this may drift) - Whatever your apps' Makefiles actually invoke to build - `go`, `python3`, `rustc`, `node`, etc. - baked into the base jail image (see [ZFS layout](#1-zfs-layout) below); builds run *inside* the jail via `jexec`, not on the host. Skiff itself doesn't care which; `make` is already in FreeBSD base (bmake), so it's never something you need to install separately. Watch for versioned port names - FreeBSD's `go125`, `python312`, etc. install as that literal binary, not a bare `go`/`python3`. Check for an unversioned meta-port first (`go` itself is one - see [ZFS layout](#1-zfs-layout)); only symlink a default by hand if no meta-port exists for what you need. The build step also needs real outbound network access from inside the jail (module/package downloads) - see the `--ip` note under [Provisioning a new app](#provisioning-a-new-app). - `jail_enable="YES"` in `/etc/rc.conf`: ``` # sysrc jail_enable=YES ``` Without this, `service jail start`/`restart` refuse to run at all, and every deploy will fail at the cutover step with a jail that never actually starts. ### One-time host setup #### 1. ZFS layout ``` # zfs create zroot/skiff # zfs create -o mountpoint=/skiff/base zroot/skiff/base # bsdinstall jail /skiff/base ``` `bsdinstall jail` only lays down the bare base system - it ships `pkg(8)`'s bootstrap stub but never actually runs it, so a jail cloned from this alone has no working `pkg` of its own (files can still be `pkg -c`-installed into it from the host, but nothing *inside* the jail can manage them, and `jexec pkg ...` fails with "the package management tool is not yet installed"). Bootstrap pkg into the image itself, before snapshotting: ``` # mount -t devfs devfs /skiff/base/dev # env ASSUME_ALWAYS_YES=yes pkg -c /skiff/base bootstrap -f ``` (`-f` because the stub already counts as "pkg is present" otherwise; devfs is needed for pkg's own network/TLS calls inside a bare chroot - `fetch(1)` wants `/dev/null` and `/dev/random`.) Now install whatever your apps' Makefiles actually need to build: ``` # pkg -c /skiff/base install -y go ``` FreeBSD's versioned language ports (`go125`, `python312`, ...) install as that literal binary name, not a bare `go`/`python3` - a Makefile invoking the unversioned command needs one. Check for an unversioned meta-port first: `go` is exactly that - it depends on whichever `goNNN` is current and installs the `go`/`gofmt` symlinks itself (`pkg install -y go`, not `go125` directly, is the whole fix). If no meta-port exists for what you need, symlink a default into `/usr/local/bin` by hand instead. Unmount devfs and snapshot once the image can actually build and run apps: ``` # umount /skiff/base/dev # zfs snapshot zroot/skiff/base@default # zfs create -o mountpoint=none zroot/skiff/appbase # zfs create -o mountpoint=none zroot/skiff/appdata # mkdir -p /skiff/apps ``` `zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their own dataset, **not a plain directory** - `new-app` clones/creates straight into them (`zroot/skiff/appbase/`, `zroot/skiff/appdata/`), and `zfs clone`/`zfs create` require the parent dataset to already exist. A plain `mkdir` in their place will let `new-app` fail with `cannot create '...': parent does not exist`. Their own mountpoint doesn't matter (`none` is fine) - every per-app dataset sets its own explicit mountpoint under `/skiff/apps/`: `appbase/` (the OS layer: `/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script, `dma.conf`) mounts at `/skiff/apps/` itself, the jail's own root; `appdata/` (the app's persistent data: releases, current, env, `meta.json`, `data.db`) mounts nested inside it, at `/skiff/apps//app`. `/skiff/apps` itself is just a plain directory, like `/skiff/repos` - nothing needs to be a dataset at that exact path. Keeping the OS layer and the app's data on separate datasets is what lets `skiff rebuild` (see [Rebuilding a jail](#rebuilding-a-jail)) destroy and reclone the former without any risk to the latter. `default` is the snapshot every app's dataset gets cloned from unless `--base-snapshot` says otherwise. Re-snapshot under a new tag after patching the base (e.g. `security-2026-08`) and pass `--base-snapshot` to `new-app` to pick it up. **If `bsdinstall jail` hangs at 100% CPU with no output:** it's spawned an interactive `dialog` (distribution-set vs. packages, mirror selection) and has no terminal to answer it - common when driving this over a non-interactive SSH session. Kill it (`pkill -9 bsddialog`) and re-run fully non-interactive instead: ``` # export nonInteractive=YES # export BSDINSTALL_DISTSITE=https://download.freebsd.org/releases/amd64/amd64/15.0-RELEASE # bsdinstall jail /skiff/base ``` (match the release number to your host's own `uname -r`; check `https://download.freebsd.org/releases/amd64/amd64//MANIFEST` exists before relying on it.) You can watch real progress with `zfs list zroot/skiff/base -o used` even while `dialog`'s own progress bar is garbled without a real tty. #### 2. Repos directory ``` # mkdir -p /skiff/repos ``` #### 3. Jail config FreeBSD doesn't ship an `/etc/jail.conf`, and at boot `service jail start` (invoked with no jail named) only creates jails it finds in that one file - it never looks inside `jail.conf.d/` on its own. Point it there once: ``` # echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf ``` Skip this and everything still *appears* to work: `skiff new-app` starts the new jail immediately, and a manual `service jail start ` after a reboot works too - both read `jail.conf.d/.conf` directly, bypassing `/etc/jail.conf` entirely. Only the boot-time, no-name form of `service jail start` actually depends on the include line, so the gap stays invisible until the host reboots and no app jail comes back up on its own - see the [Troubleshooting](#troubleshooting) entry below. #### 4. Jail networking on a single public IP Classic (non-VNET) jails share the host's network stack - `--ip` just adds an address to the host's existing interface (see [Provisioning a new app](#provisioning-a-new-app)) - so on a host with only one public address, jails shouldn't get public IPs at all. Only nginx needs one, listening on 80/443; every app's vhost `proxy_pass`es to its jail's own private address. Alias a private range onto the same interface as the public IP, persisted so it survives a reboot: ``` # sysrc ifconfig_vtnet0_alias0="inet 10.0.0.1/24" # ifconfig vtnet0 inet 10.0.0.1/24 alias ``` and hand out jail `--ip`s from within it (`10.0.0.2`, `10.0.0.3`, ...). Inbound just works from there - nginx reaching a jail is on-link delivery on the same NIC, no gateway involved, the same same-subnet rule covered above. Outbound doesn't, without one more step. A jail's build step needs real internet access (module/package downloads - see [Requirements](#requirements)), and a private range has nothing upstream translating it to the public internet the way, say, a NAT'd virtual network would - the host itself has to do that translation for its own jails. A full `/etc/pf.conf` for a skiff host looks like this - modeled on a plain single-app host's firewall (default-deny inbound, explicit allowed services), with one addition for the jail subnet: ``` ext_if = "vtnet0" jail_net = "10.0.0.0/24" tcp_services = "{ ssh, http, https }" nat on $ext_if from $jail_net to any -> 203.0.113.5 block in all set skip on lo pass inet proto icmp all pass inet6 proto icmp6 all pass in on $ext_if proto tcp to port $tcp_services pass out on $ext_if from any to any keep state ``` Write the host's actual public IP into the `nat` line (`203.0.113.5` above is a placeholder) - not `-> ($ext_if)`. Once the interface carries more than one address (the public one and the private jail range both live on `$ext_if`), pf's dynamic-interface syntax can silently pick the wrong one and NAT jail traffic to the *private* address instead, which is just as unreachable from the internet as no NAT at all. `sysrc pf_enable="YES"; service pf start` is enough on its own - the rc.d script loads the kernel module itself, no manual `kldload` needed. Only reach for `kldload pf` if you want to dry-run check the syntax with `pfctl -nf` *before* committing to `start` - `pfctl` (unlike the rc.d script) doesn't load the module for you, dry run or not: ``` # kldload pf # pfctl -nf /etc/pf.conf ``` ``` # sysrc pf_enable="YES" # service pf start ``` Skip the NAT step and jail builds fail exactly like the out-of-subnet case above - packets leave the jail fine, `tcpdump` shows them, nothing ever answers - just for a different reason (no return NAT, instead of no route at all). #### 5. Nginx Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block: ``` include /usr/local/etc/nginx/skiff.d/*.conf; ``` ``` # mkdir -p /usr/local/etc/nginx/skiff.d # sysrc nginx_enable=YES # service nginx start ``` Skiff only ever writes files under `skiff.d/` and reloads - it never touches `nginx.conf` itself. #### 6. TLS (Let's Encrypt) Skiff issues certificates itself via `certbot certonly` (webroot validation, never certbot's own `--nginx` plugin - Skiff owns `skiff.d/*.conf` and regenerates it on every domain/deploy change, and would fight any other writer of that file). Register the certbot account once: ``` # certbot register --agree-tos -m you@example.com ``` Renewal rides on FreeBSD's own `periodic(8)`, the same as any other certbot install - Skiff runs no daemon or scheduler of its own for this, and `certbot renew` is host-wide by design, so one cron job covers every app's certificates regardless of when each was issued: ``` # sysrc -f /etc/periodic.conf weekly_certbot_enable="YES" ``` `certonly` mode doesn't reload nginx after a renewal the way certbot's installer plugins do, so add that as a deploy hook once - certbot runs every script under this directory after any successful renewal: ``` # mkdir -p /usr/local/etc/letsencrypt/renewal-hooks/deploy # printf '#!/bin/sh\nnginx -s reload\n' > /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx # chmod +x /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx ``` #### 7. Install skiff Cross-compile from any machine with Go, then copy the binary over: ``` $ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff $ scp skiff root@host:/tmp/skiff # install -m 755 /tmp/skiff /usr/local/sbin/skiff ``` ### Provisioning a new app ``` # skiff new-app --name myapp --ip 192.168.1.50 --port 8080 ``` This clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and the jail's own `/usr/local/etc/rc.d/myapp` script (installed *inside* the jail's own filesystem, not the host's), creates the bare repo at `/skiff/repos/myapp.git` with its `post-receive` hook, and starts the (empty) jail. Skiff's jails are classic (non-VNET) jails - `--ip` is an address the jail binds to, but FreeBSD won't route to it unless that address exists on a host network interface. `jail.conf.d/myapp.conf` includes an `interface` line (set from `--iface`, default `vtnet0`), which makes `jail(8)` itself alias and unalias the IP on that interface as the jail starts and stops - no manual `ifconfig` step, and it survives a host reboot the same way `jail_enable="YES"` does. Pass `--iface` if your host's network interface isn't `vtnet0` (check with `ifconfig`). `new-app` also appends a `MAILNAME` line to the jail's `dma.conf`, inheriting `SMARTHOST`/`PORT`/`AUTHPATH`/`SECURETRANSFER`/`STARTTLS` from the base image, so mail sent from inside the jail (e.g. by cron or the app itself) is attributed to `myapp.skiff.local` instead of the base image's own hostname. **`--ip` must actually route on the host's `--iface` interface** - in practice, an address in the same subnet as the host's own primary address on that interface, not an arbitrary block like `10.0.0.0/8`. Classic (non-VNET) jails share the host's routing table, so an out-of-subnet address can still send packets - they just may never get a response, silently, if whatever sits upstream (a NAT gateway, a virtualized bridge) only answers its own subnet. This didn't matter before builds moved inside the jail (see [Requirements](#requirements)); now every deploy's build step needs real outbound network access from that exact address (module downloads, `pkg install`, etc), not just inbound reachability for the app itself. ### Domains and TLS `new-app` only provisions the jail - nothing is routed until you add a domain: ``` # skiff domains add --app myapp --primary myapp.example.com # skiff domains add --app myapp www.myapp.example.com # skiff domains list --app myapp # skiff domains remove --app myapp www.myapp.example.com ``` `add` writes the hostname to the app's `meta.json`, routes it over plain HTTP immediately (so the ACME HTTP-01 challenge below has somewhere to land), then requests or expands a single certificate covering every domain currently registered for the app, named after whichever is primary - matching how a real multi-domain vhost is normally issued (one SAN cert), not a cert per hostname. If DNS isn't pointed at the app's `--ip` yet, certificate issuance fails and the command exits non-zero - the app stays reachable over HTTP, and `add` is safe to re-run once DNS is fixed, exactly like retrying a failed deploy (see [If the health check fails](#if-the-health-check-fails)). Nothing about a failed `add` is destructive. The first domain added for an app becomes primary automatically; `--primary` reassigns it explicitly. Every non-primary domain 301-redirects to the primary, over HTTP and HTTPS alike, once its own hostname is covered by the certificate. `remove` only updates routing - unlike `add`, it never calls out to certbot, so a DNS or network problem can't make it fail. It's refused if you try to remove the current primary while other domains remain - reassign primary first (`skiff domains add --primary `). Domain/TLS changes never restart the jail and never run a health check - unlike a code deploy or `skiff config`, they don't touch the running app at all, only nginx and the certificate. ### Rollback ``` # skiff rollback --app myapp --list # skiff rollback --app myapp --to pre-a1b2c3d ``` Rolls the app's entire ZFS dataset back to that snapshot (code and any release-phase side effects together) and restarts the jail. This uses `zfs rollback -r`, which **destroys every snapshot newer than the target** - ZFS can't roll back past one otherwise. Rolling back to an old snapshot permanently discards the ability to return to anything deployed in between; there's no separate "undo the rollback." Treat `--to` as a one-way door, not a slider you can move back and forth. Because the bare git repo lives on a *separate* dataset from the app (`/skiff/repos` vs. `/skiff/apps/`), a rollback can delete a release directory whose git worktree is still registered in the repo. Current `skiff` versions self-heal this automatically (checkout runs `git worktree prune` first); if you're running an older binary and see `missing but already registered worktree`, run `git -C /skiff/repos/.git worktree prune` by hand and retry. ### Rebuilding a jail ``` # skiff rebuild --app myapp [--base-snapshot default] [--iface vtnet0] # skiff rebuild --all [--base-snapshot default] [--iface vtnet0] ``` Every app's jail is really two independent ZFS datasets: an **OS layer** (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script, `dma.conf` - everything `new-app` clones from `zroot/skiff/base`) and a **data layer** (`/app`: releases, current, env, `meta.json`, `data.db`). `skiff rebuild` discards the OS layer and reclones it fresh from `--base-snapshot`, while the data layer - a separate dataset, never touched - survives untouched. Use it to: - pick up a patched/upgraded base image on an app that already exists (re-snapshot the base under a new tag first - see [ZFS layout](#1-zfs-layout) - then pass it as `--base-snapshot`); - regenerate stale `jail.conf`/rc.d/`dma.conf` on an app provisioned by an older `skiff` binary, without touching its data. Guarded exactly like a deploy or config change: the old OS layer is kept until the rebuilt jail passes its health check, and automatically swapped back if it doesn't - `--app`'s rebuild fails loudly, and the app keeps serving on the OS layer it had before you ran the command. `--all` rebuilds every app sequentially and stops at the first failure, so a bad `--base-snapshot` can't cascade through every app on the host. ### Inspecting a running app ``` # jls # list running jails # jexec myapp ps aux # processes inside the jail # jexec myapp sockstat -4 -l # what's actually listening # jexec myapp service myapp status # the app's own rc.d status # service jail restart myapp # what skiff itself calls on deploy ``` ### Known gaps Carried over from the project's own README, plus what testing surfaced: - **No IP allocation.** `--ip` is operator-chosen - fine for a handful of apps on one host, but you'll want a small pool allocator beyond that. - **Old `releases/` directories are never pruned.** The ZFS snapshot is the real rollback mechanism; release directories are just there for the instant symlink flip and are safe to clean up by hand once you're confident you won't roll back to them. - **Single host only.** Scaling to a second host is a `zfs send`/`receive` of an app's dataset plus the same jail.conf/nginx steps - not automated. - **`skiff rollback -r` is destructive** (see above) - there is currently no confirmation prompt or dry-run. - **`skiff domains remove` doesn't shrink the shared certificate's SAN.** A removed hostname may still be listed on the app's certificate until the next `add` re-syncs the domain set. Harmless, since nothing routes to it anymore. --- ## Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/appbase` (or `appdata`) was created as a directory, not a dataset | `zfs create -o mountpoint=none zroot/skiff/appbase` (and same for `appdata`) | | `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` | | App jails don't come back up after a host reboot, even with `jail_enable=YES` and manual `service jail start ` working fine | `/etc/jail.conf` missing or doesn't `.include` `jail.conf.d/*.conf` - boot-time `service jail start` only creates jails it finds in `/etc/jail.conf` itself | `echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf` (see [Jail config](#3-jail-config)) | | Deploy always health-check-fails with `connection refused`, even though the app builds fine | `--iface` on `new-app`/`rebuild` doesn't name a real interface on this host, so `jail(8)` couldn't alias the IP | Check `ifconfig` for the right interface name, then `skiff rebuild --app --iface ` (see [Provisioning](#provisioning-a-new-app)) | | `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/.git worktree prune` | | `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) | | `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app --new --ref refs/heads/main` | | `build: \`make\` did not produce ./app` | Your Makefile's default target ran (or was skipped as "up to date") without ultimately leaving a file named `app` in the repo root | Run `make -B` locally in a clean checkout and confirm `./app` exists after; check the target actually named `app` (or a dependency of the default target) writes there | | Health check fails and you're not sure why | - | `jexec sockstat -4 -l` to check for a listener; `jexec ps aux` to see if the process is even running; try `jexec /app/current/app` directly to see startup errors that daemon(8) would otherwise swallow | | Build fails inside the jail with a missing command (`go: not found`, etc.) even though it's installed on the host | The base jail image itself is missing it, or FreeBSD installed it under a versioned name (`go125`, not `go`) | `jexec which `; if it's there under a versioned name, check for an unversioned meta-port (`pkg -c /skiff/base install -y go`, not `go125`) before symlinking one by hand; re-snapshot and `skiff rebuild --app --base-snapshot ` either way | | Build fails inside the jail on any network call (module download, `pkg install`), or `pkg`/`fetch`/`go mod` just time out | The app's `--ip` doesn't actually route from this host - classic jails share the host's routing table, but an out-of-subnet address (e.g. `10.0.0.5` on a host whose real subnet is `192.168.1.0/24`) can send packets that silently never get a response behind some NAT/bridged setups | `jexec fetch -T5 -o /dev/null https://` to confirm; if it hangs, re-provision with an `--ip` in the same subnet as the host's own address on `--iface` (see [Provisioning a new app](#provisioning-a-new-app)) | | `skiff domains add` fails with a certbot/ACME error | Most likely DNS isn't pointed at the app's `--ip` yet; less commonly, the app's `--ip` doesn't route outbound from this host (see above) | Fix DNS, then re-run `skiff domains add` with the same hostname - safe to retry, the app stays on plain HTTP in the meantime and nothing is left half-configured | --- ## Command reference ``` skiff new-app --name --ip [--port 8080] [--health-path /healthz] [--base-snapshot default] [--iface vtnet0] skiff deploy --app --new [--ref refs/heads/main] (normally invoked by the post-receive hook, not by hand) skiff rollback --app --list skiff rollback --app --to skiff rebuild --app [--base-snapshot default] [--iface vtnet0] skiff rebuild --all [--base-snapshot default] [--iface vtnet0] skiff config set --app KEY=VALUE [KEY=VALUE...] skiff config unset --app KEY [KEY...] skiff config list --app skiff domains add --app [--primary] skiff domains remove --app skiff domains list --app ```