# Skiff A minimal PaaS for FreeBSD + Nginx apps, in any language with a Makefile. `git push` triggers a build, an optional app-defined release phase, a ZFS snapshot, a cutover, and a health check - with automatic rollback to the pre-deploy snapshot if the new release fails its health check. One binary, no daemon, no separate metadata database - jail.conf, ZFS datasets, and a couple of flat files per app are the only state. Skiff has no opinion on your language or runtime - the only thing it requires of an app is a `Makefile` that produces an executable named `app`, and that `app` accepts TCP traffic on an assigned port. See [What an app needs to provide](#what-an-app-needs-to-provide). ## Requirements - FreeBSD 15, ZFS root - `nginx` and `git` - neither ships in base; install with `pkg install nginx git-lite` - `certbot` - only needed if you'll route real domains via `skiff domains`; install with `pkg install py312-certbot` (package name tracks the default Python version, so this may drift) - `jail_enable="YES"` in `/etc/rc.conf` (`sysrc jail_enable=YES`) - `service jail start`/`restart` refuse to run without it - Whatever toolchain your apps' Makefiles invoke to build - `go`, `python`, `rustc`, `node`, etc. - baked into the base jail image (see [One-time host setup](#one-time-host-setup)); builds run inside the jail via `jexec`, not on the host itself. `make` itself is already in FreeBSD base (bmake), so it's the one build tool Skiff never asks you to install. Watch for versioned port names (`go125`, `python312`, ...) - FreeBSD installs the literal versioned binary, not a bare `go`/`python3`. Check for an unversioned meta-port first (`go` itself is one - see [ZFS layout](#zfs-layout)); only symlink a default by hand if no meta-port exists for what you need. - The build step needs real outbound network access from inside the jail (module/package downloads) - see the `--ip` note under [First deploy](#first-deploy). ## One-time host setup ### ZFS layout ``` # zfs create zroot/skiff # zfs create -o mountpoint=/skiff/base zroot/skiff/base # bsdinstall jail /skiff/base ``` `bsdinstall jail` only lays down the bare base system - it ships `pkg(8)`'s bootstrap stub but never actually runs it, so a jail cloned from this alone has no working `pkg` of its own (files can still be `pkg -c`-installed into it from the host, but nothing *inside* the jail can manage them, and `jexec pkg ...` fails with "the package management tool is not yet installed"). Bootstrap pkg into the image itself, before snapshotting: ``` # mount -t devfs devfs /skiff/base/dev # env ASSUME_ALWAYS_YES=yes pkg -c /skiff/base bootstrap -f ``` (`-f` because the stub already counts as "pkg is present" otherwise; devfs is needed for pkg's own network/TLS calls inside a bare chroot - `fetch(1)` wants `/dev/null` and `/dev/random`.) Now install whatever your apps' Makefiles actually need to build: ``` # pkg -c /skiff/base install -y go ``` FreeBSD's versioned language ports (`go125`, `python312`, ...) install as that literal binary name, not a bare `go`/`python3` - a Makefile invoking the unversioned command needs one. Check for an unversioned meta-port first: `go` is exactly that - it depends on whichever `goNNN` is current and installs the `go`/`gofmt` symlinks itself (`pkg install -y go`, not `go125` directly, is the whole fix). If no meta-port exists for what you need, symlink a default into `/usr/local/bin` by hand instead. Unmount devfs and snapshot once the image can actually build and run apps: ``` # umount /skiff/base/dev # zfs snapshot zroot/skiff/base@default # zfs create -o mountpoint=none zroot/skiff/appbase # zfs create -o mountpoint=none zroot/skiff/appdata # mkdir -p /skiff/apps ``` `default` is the snapshot every app's dataset gets cloned from unless `--base-snapshot` says otherwise. Re-snapshot under a new tag after patching the base (e.g. `security-2026-08`) and pass `--base-snapshot` to `new-app` to pick it up. `zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their own dataset, not a plain directory - `new-app` clones/creates straight into them (`zroot/skiff/appbase/`, `zroot/skiff/appdata/`), and `zfs clone`/`zfs create` require the parent dataset to already exist. Their own mountpoint doesn't matter (`none` is fine) since every per-app dataset sets its own explicit mountpoint under `/skiff/apps/` - `appbase/` at `/skiff/apps/` itself (the jail's root), `appdata/` at `/skiff/apps//app`. `/skiff/apps` itself is a plain directory, like `/skiff/repos` - nothing needs to be a dataset at that exact path. Keeping the OS layer (`appbase`) and the app's persistent data (`appdata`) on separate datasets is what lets `skiff rebuild` destroy and reclone the former without any risk to the latter. ### Repos directory ``` # mkdir -p /skiff/repos ``` ### Jail config FreeBSD doesn't ship an `/etc/jail.conf`, and at boot `service jail start` (no jail named) only creates jails it finds in that one file - it never looks at `jail.conf.d/` on its own. Point it there once: ``` # echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf ``` Without this, `skiff new-app` and manual `service jail start ` still work fine - both read `jail.conf.d/.conf` directly - so the gap stays hidden until the host reboots and no app jail comes back up. ### Jail networking on a single public IP Classic (non-VNET) jails share the host's network stack - `--ip` just adds an address to the host's existing interface (see [First deploy](#first-deploy)) - so on a host with only one public address, jails shouldn't get public IPs at all. Only nginx needs one, listening on 80/443; every app's vhost `proxy_pass`es to its jail's own private address. Alias a private range onto the same interface as the public IP, persisted so it survives a reboot: ``` # sysrc ifconfig_vtnet0_alias0="inet 10.0.0.1/24" # ifconfig vtnet0 inet 10.0.0.1/24 alias ``` and hand out jail `--ip`s from within it (`10.0.0.2`, `10.0.0.3`, ...). Inbound just works from there - nginx reaching a jail is on-link delivery on the same NIC, no gateway involved, the same same-subnet rule as always. Outbound doesn't, without one more step. A jail's build step needs real internet access (module/package downloads - see [Requirements](#requirements)), and a private range has nothing upstream translating it to the public internet the way, say, a NAT'd virtual network would - the host itself has to do that translation for its own jails. A full `/etc/pf.conf` for a skiff host looks like this - modeled on a plain single-app host's firewall (default-deny inbound, explicit allowed services), with one addition for the jail subnet: ``` ext_if = "vtnet0" jail_net = "10.0.0.0/24" tcp_services = "{ ssh, http, https }" nat on $ext_if from $jail_net to any -> 203.0.113.5 block in all set skip on lo pass inet proto icmp all pass inet6 proto icmp6 all pass in on $ext_if proto tcp to port $tcp_services pass out on $ext_if from any to any keep state ``` Write the host's actual public IP into the `nat` line (`203.0.113.5` above is a placeholder) - not `-> ($ext_if)`. Once the interface carries more than one address (the public one and the private jail range both live on `$ext_if`), pf's dynamic-interface syntax can silently pick the wrong one and NAT jail traffic to the *private* address instead, which is just as unreachable from the internet as no NAT at all. `sysrc pf_enable="YES"; service pf start` is enough on its own - the rc.d script loads the kernel module itself, no manual `kldload` needed. Only reach for `kldload pf` if you want to dry-run check the syntax with `pfctl -nf` *before* committing to `start` - `pfctl` (unlike the rc.d script) doesn't load the module for you, dry run or not: ``` # kldload pf # pfctl -nf /etc/pf.conf ``` ``` # sysrc pf_enable="YES" # service pf start ``` Skip the NAT step and jail builds fail exactly like the out-of-subnet case in Requirements - packets leave the jail fine, `tcpdump` shows them, nothing ever answers - just for a different reason (no return NAT, instead of no route at all). ### Nginx Add once to `/usr/local/etc/nginx/nginx.conf`, inside the `http` block: ``` include /usr/local/etc/nginx/skiff.d/*.conf; ``` ``` # mkdir -p /usr/local/etc/nginx/skiff.d ``` Skiff only ever writes files under `skiff.d/` and reloads - it never touches `nginx.conf` itself. ### TLS (Let's Encrypt) Skiff issues certificates itself via `certbot certonly` (webroot validation, never certbot's own `--nginx` plugin - Skiff owns `skiff.d/*.conf` and would fight any other writer of it). Register the certbot account once: ``` # certbot register --agree-tos -m you@example.com ``` Renewal rides on FreeBSD's own `periodic(8)`, same as any other certbot install - Skiff runs no daemon or scheduler of its own for this: ``` # sysrc -f /etc/periodic.conf weekly_certbot_enable="YES" ``` `certonly` mode doesn't reload nginx after a renewal the way certbot's installer plugins do, so add that as a deploy hook once: ``` # mkdir -p /usr/local/etc/letsencrypt/renewal-hooks/deploy # printf '#!/bin/sh\nnginx -s reload\n' > /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx # chmod +x /usr/local/etc/letsencrypt/renewal-hooks/deploy/reload-nginx ``` ### Install skiff ``` $ GOOS=freebsd GOARCH=amd64 go build -o skiff ./cmd/skiff # install -m 755 skiff /usr/local/sbin/skiff ``` ## Deploying an app ### First deploy ``` # skiff new-app --name myapp --ip 192.168.1.50 --port 8080 ``` Clones the base dataset, writes `/etc/jail.conf.d/myapp.conf` and the jail's own rc.d service, creates the bare repo at `/skiff/repos/myapp.git` with its `post-receive` hook installed, and starts the (empty) jail. Prints the remote URL to add locally. `--ip` must actually route on the host's `--iface` interface - in practice, an address in the same subnet as the host's own primary address on that interface, not an arbitrary block like `10.0.0.0/8`. Classic (non-VNET) jails share the host's routing table, so an out-of-subnet address can still send packets - they just may never get a response, silently, if whatever sits upstream (a NAT gateway, a virtualized bridge) only answers its own subnet. This didn't matter before builds moved inside the jail; now every deploy's build step needs real outbound network access from that exact address (module downloads, `pkg install`, etc). ### Domains & TLS `new-app` only provisions the jail - nothing is routed until you add a domain: ``` # skiff domains add --app myapp --primary myapp.example.com # skiff domains add --app myapp www.myapp.example.com # skiff domains list --app myapp # skiff domains remove --app myapp www.myapp.example.com ``` `add` routes the hostname over plain HTTP immediately, then requests or expands a single certificate covering every domain currently registered for the app (named after whichever is primary). If DNS isn't pointed at the app's `--ip` yet, issuance fails and the command exits non-zero - the app stays reachable over HTTP, and `add` is safe to re-run once DNS is fixed, same as retrying a failed deploy. The first domain added becomes primary automatically; every other domain 301-redirects to it, over HTTP and HTTPS alike. `remove` only updates routing - it never calls out to certbot, so a DNS or network problem can't make it fail. ### Every deploy after ``` $ git remote add skiff root@host:/skiff/repos/myapp.git $ git push skiff main ``` The push blocks until the deploy finishes - success or failure prints straight to your terminal, no dashboard to check. ### Rollback ``` # skiff rollback --app myapp --list # skiff rollback --app myapp --to pre-a1b2c3d ``` Rolls the app's entire ZFS dataset back to that snapshot (code and any release-phase side effects together) and restarts the jail. This uses `zfs rollback -r`, which **destroys every snapshot newer than the target** - ZFS can't roll back past one otherwise. Rolling back to an old snapshot permanently discards the ability to return to anything deployed in between; there's no separate "undo the rollback." ### Rebuilding a jail ``` # skiff rebuild --app myapp # skiff rebuild --all ``` Discards the app's OS layer (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script, `dma.conf`) and reclones it fresh from the base image, leaving `/app` (releases, current, env, `meta.json`, `data.db`) completely untouched - the two live on separate ZFS datasets. Use this to pick up a patched/upgraded base image (`--base-snapshot`) on an app that already exists, or to regenerate stale jail config after a Skiff upgrade, without touching the app's own data. Guarded exactly like a deploy: the old OS layer is kept until the rebuilt jail passes its health check, and swapped back automatically if it doesn't. ## What an app needs to provide A `Makefile` in the repo root, and nothing else Skiff cares about - no required language, framework, or file layout beyond this contract: - Running bare `make` (Skiff runs `make -B`, so it never trusts a stale build artifact left in the checkout; every deploy is a fresh worktree anyway) must leave an executable file named `app` in the repo root. What produces it is entirely up to you - `go build -o app .`, a `cp target/release/myapp app` after `cargo build --release`, a shell script with a `#!/usr/bin/env python3` shebang that `chmod +x`'d itself into place, whatever. Skiff execs this file directly (via `daemon(8)`), and the kernel doesn't care whether it's a native binary or a script - only that it's marked executable and has a valid interpreter line if it's not. - That executable listens on the port given to `new-app` (bind the wildcard address, not `127.0.0.1`), and answers the configured health path with a non-3xx status once ready. - Optionally, a `release` target in the same Makefile - run once inside the jail *before* cutover (against the new release's own checkout, not whatever `current` still points at), gating the deploy on its exit code. Skiff doesn't interpret this target; it's entirely the app's own convention (`sqlite3` migrations, cache warming, whatever). Detected by asking `make` itself via `make -n release`, not by parsing the Makefile - it can be defined however you like (directly, via an include, a pattern rule). ```makefile .PHONY: all release all: app app: main.go go.mod go build -o app . release: sqlite3 /app/data.db < migrations/0004_add_x.sql ``` See [USER_GUIDE.md](USER_GUIDE.md) for a full walkthrough with a real example app. Runtime dependencies (a Python interpreter, the Node runtime, etc.) must be present in the jail's base image if your `app` isn't a self-contained static binary - that's a base-jail provisioning concern, not something the Makefile handles. ## Known gaps - No IP allocation - `--ip` is operator-chosen. Fine for a handful of apps on one host; would want a small pool allocator beyond that. - Old `releases/` directories are never pruned. The ZFS snapshot is the real rollback mechanism; the release dirs are just there for the instant symlink flip and are safe to clean up by hand. - Single host only. Scaling to a second host is a `zfs send`/`receive` of an app's dataset plus the same jail.conf/nginx steps - not yet automated here. - `skiff domains remove` doesn't shrink the shared certificate's SAN - a removed hostname may still be listed on it until the next `add` re-syncs the domain set. Harmless, since nothing routes to it anymore.