Split per-app dataset into OS/data layers, add skiff rebuild

Each app now gets two ZFS datasets - a disposable OS layer
(zroot/skiff/appbase/<app>) cloned from base, and a persistent data
layer (zroot/skiff/appdata/<app>) holding releases/current/env/
meta.json/data.db. `skiff rebuild --app|--all` discards and reclones
the OS layer to pick up an OS/toolchain/config-template upgrade,
guarded by the same snapshot-and-rollback safety net as deploy and
config changes; the data layer is never touched.

Also fixes stale docs (builds already run via jexec inside the jail,
not on the host; --iface already auto-aliases the jail's IP, so the
manual ifconfig step and its troubleshooting entry are gone) found
while updating the same sections for this feature.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

1 files changed, 77 insertions, 37 deletions

diff --git a/USER_GUIDE.md b/USER_GUIDE.md
index fa8449c..fe78bd0 100644
--- a/USER_GUIDE.md
+++ b/USER_GUIDE.md
@@ -225,10 +225,11 @@ this in a hurry - it's more destructive than "undo."
   # pkg install nginx git-lite
   ```
 - Whatever your apps' Makefiles actually invoke to build - `go`,
-  `python3`, `rustc`, `node`, etc. - installed on the **host** (builds
-  run outside the jail, via `make -B` in the checked-out worktree). Skiff
-  itself doesn't care which; `make` is already in FreeBSD base (bmake),
-  so it's never something you need to install separately.
+  `python3`, `rustc`, `node`, etc. - baked into the base jail image
+  (see [ZFS layout](#1-zfs-layout) below); builds run *inside* the jail
+  via `jexec`, not on the host. Skiff itself doesn't care which; `make`
+  is already in FreeBSD base (bmake), so it's never something you need
+  to install separately.
 - `jail_enable="YES"` in `/etc/rc.conf`:
   ```
   # sysrc jail_enable=YES
@@ -246,14 +247,31 @@ this in a hurry - it's more destructive than "undo."
 # zfs create -o mountpoint=/skiff/base zroot/skiff/base
 # bsdinstall jail /skiff/base
 # zfs snapshot zroot/skiff/base@ready
-# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps
+# zfs create -o mountpoint=none zroot/skiff/appbase
+# zfs create -o mountpoint=none zroot/skiff/appdata
+# mkdir -p /skiff/apps
 ```
 
-`zroot/skiff/apps` must exist as its own dataset, **not a plain
-directory** - `new-app` clones straight into it
-(`zroot/skiff/apps/<name>`), and `zfs clone` requires its parent dataset
-to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail
-with `cannot create '...': parent does not exist`.
+`zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their
+own dataset, **not a plain directory** - `new-app` clones/creates
+straight into them (`zroot/skiff/appbase/<name>`,
+`zroot/skiff/appdata/<name>`), and `zfs clone`/`zfs create` require the
+parent dataset to already exist. A plain `mkdir` in their place will let
+`new-app` fail with `cannot create '...': parent does not exist`.
+
+Their own mountpoint doesn't matter (`none` is fine) - every per-app
+dataset sets its own explicit mountpoint under `/skiff/apps/<name>`:
+`appbase/<name>` (the OS layer: `/etc`, `/usr`, `/bin`, `jail.conf`, the
+jail's rc.d script, `dma.conf`) mounts at `/skiff/apps/<name>` itself,
+the jail's own root; `appdata/<name>` (the app's persistent data:
+releases, current, env, `meta.json`, `data.db`) mounts nested inside it,
+at `/skiff/apps/<name>/app`. `/skiff/apps` itself is just a plain
+directory, like `/skiff/repos` - nothing needs to be a dataset at that
+exact path.
+
+Keeping the OS layer and the app's data on separate datasets is what
+lets `skiff rebuild` (see [Rebuilding a jail](#rebuilding-a-jail))
+destroy and reclone the former without any risk to the latter.
 
 `ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
 under a new tag after patching the base (e.g. `security-2026-08`) and
@@ -341,28 +359,20 @@ the jail's own filesystem, not the host's), creates the bare repo at
 `/skiff/repos/myapp.git` with its `post-receive` hook, and starts the
 (empty) jail.
 
-**Critical, undocumented-by-default step: alias the IP onto the host.**
 Skiff's jails are classic (non-VNET) jails - `--ip` is an address the
-jail binds to, but FreeBSD won't route to it unless that address already
-exists on a host network interface. `new-app` does **not** do this for
-you:
+jail binds to, but FreeBSD won't route to it unless that address exists
+on a host network interface. `jail.conf.d/myapp.conf` includes an
+`interface` line (set from `--iface`, default `vtnet0`), which makes
+`jail(8)` itself alias and unalias the IP on that interface as the jail
+starts and stops - no manual `ifconfig` step, and it survives a host
+reboot the same way `jail_enable="YES"` does. Pass `--iface` if your
+host's network interface isn't `vtnet0` (check with `ifconfig`).
 
-```
-# ifconfig <your-interface> alias 10.0.0.5/24
-```
-
-Without this, the jail starts fine but is completely unreachable - every
-deploy for that app will fail its health check and roll back forever,
-even though the app itself is working correctly inside the jail. This
-was the single most confusing failure mode encountered testing Skiff:
-the deploy pipeline, build, and app were all fine, and the jail was
-simply not reachable. If you're provisioning many apps, plan a real IP
-range up front and script the alias step - see [Known
-gaps](#known-gaps) below.
-
-Make the alias persistent across reboots via `/etc/rc.conf`
-(`ifconfig_<if>_alias0="inet 10.0.0.5/24"`) rather than a one-off
-`ifconfig` command, or it won't survive a host restart.
+`new-app` also appends a `MAILNAME` line to the jail's `dma.conf`,
+inheriting `SMARTHOST`/`PORT`/`AUTHPATH`/`SECURETRANSFER`/`STARTTLS`
+from the base image, so mail sent from inside the jail (e.g. by cron or
+the app itself) is attributed to `myapp.skiff.local` instead of the
+base image's own hostname.
 
 ### Rollback
 
@@ -388,6 +398,34 @@ Current `skiff` versions self-heal this automatically (checkout runs
 `missing but already registered worktree`, run
 `git -C /skiff/repos/<app>.git worktree prune` by hand and retry.
 
+### Rebuilding a jail
+
+```
+# skiff rebuild --app myapp [--base-snapshot ready] [--iface vtnet0]
+# skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]
+```
+
+Every app's jail is really two independent ZFS datasets: an **OS
+layer** (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script,
+`dma.conf` - everything `new-app` clones from `zroot/skiff/base`) and a
+**data layer** (`/app`: releases, current, env, `meta.json`,
+`data.db`). `skiff rebuild` discards the OS layer and reclones it fresh
+from `--base-snapshot`, while the data layer - a separate dataset,
+never touched - survives untouched. Use it to:
+
+- pick up a patched/upgraded base image on an app that already exists
+  (re-snapshot the base under a new tag first - see [ZFS
+  layout](#1-zfs-layout) - then pass it as `--base-snapshot`);
+- regenerate stale `jail.conf`/rc.d/`dma.conf` on an app provisioned by
+  an older `skiff` binary, without touching its data.
+
+Guarded exactly like a deploy or config change: the old OS layer is
+kept until the rebuilt jail passes its health check, and automatically
+swapped back if it doesn't - `--app`'s rebuild fails loudly, and the
+app keeps serving on the OS layer it had before you ran the command.
+`--all` rebuilds every app sequentially and stops at the first failure,
+so a bad `--base-snapshot` can't cascade through every app on the host.
+
 ### Inspecting a running app
 
 ```
@@ -402,10 +440,9 @@ Current `skiff` versions self-heal this automatically (checkout runs
 
 Carried over from the project's own README, plus what testing surfaced:
 
-- **No IP allocation.** `--ip` is operator-chosen and must be manually
-  aliased onto a host interface (see above) - fine for a handful of
-  apps on one host, but you'll want a small pool allocator and an
-  automated alias step beyond that.
+- **No IP allocation.** `--ip` is operator-chosen - fine for a handful
+  of apps on one host, but you'll want a small pool allocator beyond
+  that.
 - **Old `releases/<sha>` directories are never pruned.** The ZFS
   snapshot is the real rollback mechanism; release directories are just
   there for the instant symlink flip and are safe to clean up by hand
@@ -422,10 +459,10 @@ Carried over from the project's own README, plus what testing surfaced:
 
 | Symptom | Likely cause | Fix |
 |---|---|---|
-| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` |
+| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/appbase` (or `appdata`) was created as a directory, not a dataset | `zfs create -o mountpoint=none zroot/skiff/appbase` (and same for `appdata`) |
 | `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` |
 | App jails don't come back up after a host reboot, even with `jail_enable=YES` and manual `service jail start <app>` working fine | `/etc/jail.conf` missing or doesn't `.include` `jail.conf.d/*.conf` - boot-time `service jail start` only creates jails it finds in `/etc/jail.conf` itself | `echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf` (see [Jail config](#3-jail-config)) |
-| Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig <if> alias <ip>/<mask>` (see [Provisioning](#provisioning-a-new-app)) |
+| Deploy always health-check-fails with `connection refused`, even though the app builds fine | `--iface` on `new-app`/`rebuild` doesn't name a real interface on this host, so `jail(8)` couldn't alias the IP | Check `ifconfig` for the right interface name, then `skiff rebuild --app <app> --iface <if>` (see [Provisioning](#provisioning-a-new-app)) |
 | `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` |
 | `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) |
 | `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` |
@@ -438,7 +475,7 @@ Carried over from the project's own README, plus what testing surfaced:
 
 ```
 skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz]
-              [--server-name <name>] [--base-snapshot ready]
+              [--server-name <name>] [--base-snapshot ready] [--iface vtnet0]
 
 skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
              (normally invoked by the post-receive hook, not by hand)
@@ -446,6 +483,9 @@ skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
 skiff rollback --app <app> --list
 skiff rollback --app <app> --to <snapshot-tag>
 
+skiff rebuild --app <app> [--base-snapshot ready] [--iface vtnet0]
+skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]
+
 skiff config set --app <app> KEY=VALUE [KEY=VALUE...]
 skiff config unset --app <app> KEY [KEY...]
 skiff config list --app <app>