Split per-app dataset into OS/data layers, add skiff rebuild

Each app now gets two ZFS datasets - a disposable OS layer
(zroot/skiff/appbase/<app>) cloned from base, and a persistent data
layer (zroot/skiff/appdata/<app>) holding releases/current/env/
meta.json/data.db. `skiff rebuild --app|--all` discards and reclones
the OS layer to pick up an OS/toolchain/config-template upgrade,
guarded by the same snapshot-and-rollback safety net as deploy and
config changes; the data layer is never touched.

Also fixes stale docs (builds already run via jexec inside the jail,
not on the host; --iface already auto-aliases the jail's IP, so the
manual ifconfig step and its troubleshooting entry are gone) found
while updating the same sections for this feature.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

8 files changed, 359 insertions, 49 deletions

diff --git a/README.md b/README.md
index ba51aba..4373008 100644
--- a/README.md
+++ b/README.md
@@ -20,10 +20,10 @@ needs to provide](#what-an-app-needs-to-provide).
 - `jail_enable="YES"` in `/etc/rc.conf` (`sysrc jail_enable=YES`) - `service
   jail start`/`restart` refuse to run without it
 - Whatever toolchain your apps' Makefiles invoke to build - `go`, `python`,
-  `rustc`, `node`, etc. - installed on the **host** (builds run outside the
-  jail; see [One-time host setup](#one-time-host-setup)). `make` itself is
-  already in FreeBSD base (bmake), so it's the one build tool Skiff never
-  asks you to install.
+  `rustc`, `node`, etc. - baked into the base jail image (see [One-time host
+  setup](#one-time-host-setup)); builds run inside the jail via `jexec`, not
+  on the host itself. `make` itself is already in FreeBSD base (bmake), so
+  it's the one build tool Skiff never asks you to install.
 
 ## One-time host setup
 
@@ -34,16 +34,28 @@ needs to provide](#what-an-app-needs-to-provide).
 # zfs create -o mountpoint=/skiff/base zroot/skiff/base
 # bsdinstall jail /skiff/base
 # zfs snapshot zroot/skiff/base@ready
-# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps
+# zfs create -o mountpoint=none zroot/skiff/appbase
+# zfs create -o mountpoint=none zroot/skiff/appdata
+# mkdir -p /skiff/apps
 ```
 
 `ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
 under a new tag after patching the base (e.g. `security-2026-08`) and pass
 `--base-snapshot` to `new-app` to pick it up.
 
-`zroot/skiff/apps` must exist as its own dataset, not a plain directory -
-`new-app` clones straight into it (`zroot/skiff/apps/<name>`), and `zfs
-clone` requires the parent dataset to already exist.
+`zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their own
+dataset, not a plain directory - `new-app` clones/creates straight into them
+(`zroot/skiff/appbase/<name>`, `zroot/skiff/appdata/<name>`), and `zfs
+clone`/`zfs create` require the parent dataset to already exist. Their own
+mountpoint doesn't matter (`none` is fine) since every per-app dataset sets
+its own explicit mountpoint under `/skiff/apps/<name>` - `appbase/<name>` at
+`/skiff/apps/<name>` itself (the jail's root), `appdata/<name>` at
+`/skiff/apps/<name>/app`. `/skiff/apps` itself is a plain directory, like
+`/skiff/repos` - nothing needs to be a dataset at that exact path.
+
+Keeping the OS layer (`appbase`) and the app's persistent data (`appdata`)
+on separate datasets is what lets `skiff rebuild` destroy and reclone the
+former without any risk to the latter.
 
 ### Repos directory
 
@@ -125,6 +137,23 @@ target** - ZFS can't roll back past one otherwise. Rolling back to an old
 snapshot permanently discards the ability to return to anything deployed
 in between; there's no separate "undo the rollback."
 
+### Rebuilding a jail
+
+```
+# skiff rebuild --app myapp
+# skiff rebuild --all
+```
+
+Discards the app's OS layer (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's
+rc.d script, `dma.conf`) and reclones it fresh from the base image, leaving
+`/app` (releases, current, env, `meta.json`, `data.db`) completely
+untouched - the two live on separate ZFS datasets. Use this to pick up a
+patched/upgraded base image (`--base-snapshot`) on an app that already
+exists, or to regenerate stale jail config after a Skiff upgrade, without
+touching the app's own data. Guarded exactly like a deploy: the old OS
+layer is kept until the rebuilt jail passes its health check, and swapped
+back automatically if it doesn't.
+
 ## What an app needs to provide
 
 A `Makefile` in the repo root, and nothing else Skiff cares about - no
diff --git a/USER_GUIDE.md b/USER_GUIDE.md
index fa8449c..fe78bd0 100644
--- a/USER_GUIDE.md
+++ b/USER_GUIDE.md
@@ -225,10 +225,11 @@ this in a hurry - it's more destructive than "undo."
   # pkg install nginx git-lite
   ```
 - Whatever your apps' Makefiles actually invoke to build - `go`,
-  `python3`, `rustc`, `node`, etc. - installed on the **host** (builds
-  run outside the jail, via `make -B` in the checked-out worktree). Skiff
-  itself doesn't care which; `make` is already in FreeBSD base (bmake),
-  so it's never something you need to install separately.
+  `python3`, `rustc`, `node`, etc. - baked into the base jail image
+  (see [ZFS layout](#1-zfs-layout) below); builds run *inside* the jail
+  via `jexec`, not on the host. Skiff itself doesn't care which; `make`
+  is already in FreeBSD base (bmake), so it's never something you need
+  to install separately.
 - `jail_enable="YES"` in `/etc/rc.conf`:
   ```
   # sysrc jail_enable=YES
@@ -246,14 +247,31 @@ this in a hurry - it's more destructive than "undo."
 # zfs create -o mountpoint=/skiff/base zroot/skiff/base
 # bsdinstall jail /skiff/base
 # zfs snapshot zroot/skiff/base@ready
-# zfs create -o mountpoint=/skiff/apps zroot/skiff/apps
+# zfs create -o mountpoint=none zroot/skiff/appbase
+# zfs create -o mountpoint=none zroot/skiff/appdata
+# mkdir -p /skiff/apps
 ```
 
-`zroot/skiff/apps` must exist as its own dataset, **not a plain
-directory** - `new-app` clones straight into it
-(`zroot/skiff/apps/<name>`), and `zfs clone` requires its parent dataset
-to already exist. A plain `mkdir /skiff/apps` will let `new-app` fail
-with `cannot create '...': parent does not exist`.
+`zroot/skiff/appbase` and `zroot/skiff/appdata` must each exist as their
+own dataset, **not a plain directory** - `new-app` clones/creates
+straight into them (`zroot/skiff/appbase/<name>`,
+`zroot/skiff/appdata/<name>`), and `zfs clone`/`zfs create` require the
+parent dataset to already exist. A plain `mkdir` in their place will let
+`new-app` fail with `cannot create '...': parent does not exist`.
+
+Their own mountpoint doesn't matter (`none` is fine) - every per-app
+dataset sets its own explicit mountpoint under `/skiff/apps/<name>`:
+`appbase/<name>` (the OS layer: `/etc`, `/usr`, `/bin`, `jail.conf`, the
+jail's rc.d script, `dma.conf`) mounts at `/skiff/apps/<name>` itself,
+the jail's own root; `appdata/<name>` (the app's persistent data:
+releases, current, env, `meta.json`, `data.db`) mounts nested inside it,
+at `/skiff/apps/<name>/app`. `/skiff/apps` itself is just a plain
+directory, like `/skiff/repos` - nothing needs to be a dataset at that
+exact path.
+
+Keeping the OS layer and the app's data on separate datasets is what
+lets `skiff rebuild` (see [Rebuilding a jail](#rebuilding-a-jail))
+destroy and reclone the former without any risk to the latter.
 
 `ready` is the snapshot every app's dataset gets cloned from. Re-snapshot
 under a new tag after patching the base (e.g. `security-2026-08`) and
@@ -341,28 +359,20 @@ the jail's own filesystem, not the host's), creates the bare repo at
 `/skiff/repos/myapp.git` with its `post-receive` hook, and starts the
 (empty) jail.
 
-**Critical, undocumented-by-default step: alias the IP onto the host.**
 Skiff's jails are classic (non-VNET) jails - `--ip` is an address the
-jail binds to, but FreeBSD won't route to it unless that address already
-exists on a host network interface. `new-app` does **not** do this for
-you:
+jail binds to, but FreeBSD won't route to it unless that address exists
+on a host network interface. `jail.conf.d/myapp.conf` includes an
+`interface` line (set from `--iface`, default `vtnet0`), which makes
+`jail(8)` itself alias and unalias the IP on that interface as the jail
+starts and stops - no manual `ifconfig` step, and it survives a host
+reboot the same way `jail_enable="YES"` does. Pass `--iface` if your
+host's network interface isn't `vtnet0` (check with `ifconfig`).
 
-```
-# ifconfig <your-interface> alias 10.0.0.5/24
-```
-
-Without this, the jail starts fine but is completely unreachable - every
-deploy for that app will fail its health check and roll back forever,
-even though the app itself is working correctly inside the jail. This
-was the single most confusing failure mode encountered testing Skiff:
-the deploy pipeline, build, and app were all fine, and the jail was
-simply not reachable. If you're provisioning many apps, plan a real IP
-range up front and script the alias step - see [Known
-gaps](#known-gaps) below.
-
-Make the alias persistent across reboots via `/etc/rc.conf`
-(`ifconfig_<if>_alias0="inet 10.0.0.5/24"`) rather than a one-off
-`ifconfig` command, or it won't survive a host restart.
+`new-app` also appends a `MAILNAME` line to the jail's `dma.conf`,
+inheriting `SMARTHOST`/`PORT`/`AUTHPATH`/`SECURETRANSFER`/`STARTTLS`
+from the base image, so mail sent from inside the jail (e.g. by cron or
+the app itself) is attributed to `myapp.skiff.local` instead of the
+base image's own hostname.
 
 ### Rollback
 
@@ -388,6 +398,34 @@ Current `skiff` versions self-heal this automatically (checkout runs
 `missing but already registered worktree`, run
 `git -C /skiff/repos/<app>.git worktree prune` by hand and retry.
 
+### Rebuilding a jail
+
+```
+# skiff rebuild --app myapp [--base-snapshot ready] [--iface vtnet0]
+# skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]
+```
+
+Every app's jail is really two independent ZFS datasets: an **OS
+layer** (`/etc`, `/usr`, `/bin`, `jail.conf`, the jail's rc.d script,
+`dma.conf` - everything `new-app` clones from `zroot/skiff/base`) and a
+**data layer** (`/app`: releases, current, env, `meta.json`,
+`data.db`). `skiff rebuild` discards the OS layer and reclones it fresh
+from `--base-snapshot`, while the data layer - a separate dataset,
+never touched - survives untouched. Use it to:
+
+- pick up a patched/upgraded base image on an app that already exists
+  (re-snapshot the base under a new tag first - see [ZFS
+  layout](#1-zfs-layout) - then pass it as `--base-snapshot`);
+- regenerate stale `jail.conf`/rc.d/`dma.conf` on an app provisioned by
+  an older `skiff` binary, without touching its data.
+
+Guarded exactly like a deploy or config change: the old OS layer is
+kept until the rebuilt jail passes its health check, and automatically
+swapped back if it doesn't - `--app`'s rebuild fails loudly, and the
+app keeps serving on the OS layer it had before you ran the command.
+`--all` rebuilds every app sequentially and stops at the first failure,
+so a bad `--base-snapshot` can't cascade through every app on the host.
+
 ### Inspecting a running app
 
 ```
@@ -402,10 +440,9 @@ Current `skiff` versions self-heal this automatically (checkout runs
 
 Carried over from the project's own README, plus what testing surfaced:
 
-- **No IP allocation.** `--ip` is operator-chosen and must be manually
-  aliased onto a host interface (see above) - fine for a handful of
-  apps on one host, but you'll want a small pool allocator and an
-  automated alias step beyond that.
+- **No IP allocation.** `--ip` is operator-chosen - fine for a handful
+  of apps on one host, but you'll want a small pool allocator beyond
+  that.
 - **Old `releases/<sha>` directories are never pruned.** The ZFS
   snapshot is the real rollback mechanism; release directories are just
   there for the instant symlink flip and are safe to clean up by hand
@@ -422,10 +459,10 @@ Carried over from the project's own README, plus what testing surfaced:
 
 | Symptom | Likely cause | Fix |
 |---|---|---|
-| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/apps` was created as a directory, not a dataset | `zfs create -o mountpoint=/skiff/apps zroot/skiff/apps` |
+| `zfs clone ...: parent does not exist` on `new-app` | `zroot/skiff/appbase` (or `appdata`) was created as a directory, not a dataset | `zfs create -o mountpoint=none zroot/skiff/appbase` (and same for `appdata`) |
 | `Cannot 'start' jail. Set jail_enable to YES...` | `jail_enable` unset | `sysrc jail_enable=YES` |
 | App jails don't come back up after a host reboot, even with `jail_enable=YES` and manual `service jail start <app>` working fine | `/etc/jail.conf` missing or doesn't `.include` `jail.conf.d/*.conf` - boot-time `service jail start` only creates jails it finds in `/etc/jail.conf` itself | `echo '.include "/etc/jail.conf.d/*.conf";' > /etc/jail.conf` (see [Jail config](#3-jail-config)) |
-| Deploy always health-check-fails with `connection refused`, even though the app builds fine | Jail's IP was never aliased onto a host interface | `ifconfig <if> alias <ip>/<mask>` (see [Provisioning](#provisioning-a-new-app)) |
+| Deploy always health-check-fails with `connection refused`, even though the app builds fine | `--iface` on `new-app`/`rebuild` doesn't name a real interface on this host, so `jail(8)` couldn't alias the IP | Check `ifconfig` for the right interface name, then `skiff rebuild --app <app> --iface <if>` (see [Provisioning](#provisioning-a-new-app)) |
 | `git worktree add ...: missing but already registered worktree` on redeploy | Stale worktree entry after a rollback deleted a release dir (older skiff binaries only) | `git -C /skiff/repos/<app>.git worktree prune` |
 | `bsdinstall jail` spins at 100% CPU forever | Interactive `dialog` waiting on a terminal that isn't there | Kill it, re-run with `nonInteractive=YES` and `BSDINSTALL_DISTSITE` set (see setup section) |
 | `git push` says `Everything up-to-date` but you wanted to retry a failed deploy | The git ref already advanced despite the deploy failing - nothing new to push | New commit (`git commit --allow-empty`), or on the host: `skiff deploy --app <app> --new <sha> --ref refs/heads/main` |
@@ -438,7 +475,7 @@ Carried over from the project's own README, plus what testing surfaced:
 
 ```
 skiff new-app --name <app> --ip <ip> [--port 8080] [--health-path /healthz]
-              [--server-name <name>] [--base-snapshot ready]
+              [--server-name <name>] [--base-snapshot ready] [--iface vtnet0]
 
 skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
              (normally invoked by the post-receive hook, not by hand)
@@ -446,6 +483,9 @@ skiff deploy --app <app> --new <sha> [--ref refs/heads/main] [--old <sha>]
 skiff rollback --app <app> --list
 skiff rollback --app <app> --to <snapshot-tag>
 
+skiff rebuild --app <app> [--base-snapshot ready] [--iface vtnet0]
+skiff rebuild --all [--base-snapshot ready] [--iface vtnet0]
+
 skiff config set --app <app> KEY=VALUE [KEY=VALUE...]
 skiff config unset --app <app> KEY [KEY...]
 skiff config list --app <app>
diff --git a/cmd/skiff/main.go b/cmd/skiff/main.go
index 8ca1406..3286da7 100644
--- a/cmd/skiff/main.go
+++ b/cmd/skiff/main.go
@@ -7,6 +7,7 @@ import (
 	"flag"
 	"fmt"
 	"os"
+	"path/filepath"
 	"sort"
 	"strings"
 	"time"
@@ -17,6 +18,7 @@ import (
 	"skiff/internal/jailmgr"
 	"skiff/internal/layout"
 	"skiff/internal/lock"
+	"skiff/internal/rebuild"
 	"skiff/internal/sh"
 	"skiff/internal/tmpl"
 	"skiff/internal/zfsutil"
@@ -24,7 +26,7 @@ import (
 
 func main() {
 	if len(os.Args) < 2 {
-		fmt.Fprintln(os.Stderr, "usage: skiff <deploy|new-app|rollback|config> [flags]")
+		fmt.Fprintln(os.Stderr, "usage: skiff <deploy|new-app|rollback|rebuild|config> [flags]")
 		os.Exit(1)
 	}
 
@@ -36,6 +38,8 @@ func main() {
 		err = cmdNewApp(os.Args[2:])
 	case "rollback":
 		err = cmdRollback(os.Args[2:])
+	case "rebuild":
+		err = cmdRebuild(os.Args[2:])
 	case "config":
 		err = cmdConfig(os.Args[2:])
 	default:
@@ -109,6 +113,11 @@ func cmdNewApp(args []string) error {
 		return fmt.Errorf("clone base dataset: %w", err)
 	}
 
+	fmt.Fprintf(out, "==> creating data dataset %s\n", cfg.AppDataDataset(*name))
+	if err := zfsutil.Create(ctx, out, cfg.AppDataDataset(*name), cfg.AppDir(*name)); err != nil {
+		return fmt.Errorf("create data dataset: %w", err)
+	}
+
 	if err := os.MkdirAll(cfg.ReleasesDir(*name), 0755); err != nil {
 		return fmt.Errorf("create app dir: %w", err)
 	}
@@ -165,7 +174,7 @@ func cmdRollback(args []string) error {
 	}
 	ctx := context.Background()
 	cfg := layout.Default()
-	dataset := cfg.AppDataset(*app)
+	dataset := cfg.AppDataDataset(*app)
 
 	if *list {
 		snaps, err := zfsutil.Snapshots(ctx, dataset)
@@ -191,6 +200,64 @@ func cmdRollback(args []string) error {
 	return jailmgr.Restart(ctx, os.Stdout, *app)
 }
 
+func cmdRebuild(args []string) error {
+	fs := flag.NewFlagSet("rebuild", flag.ExitOnError)
+	app := fs.String("app", "", "app name")
+	all := fs.Bool("all", false, "rebuild every app")
+	baseSnapshot := fs.String("base-snapshot", "ready", "snapshot on the base dataset to clone from")
+	iface := fs.String("iface", layout.DefaultJailIface, "host interface the jail's IP is aliased onto")
+	fs.Parse(args)
+
+	if (*app == "") != *all {
+		return fmt.Errorf("exactly one of --app or --all is required")
+	}
+
+	cfg := layout.Default()
+	if *app != "" {
+		return rebuildOne(cfg, *app, *baseSnapshot, *iface)
+	}
+
+	apps, err := listApps(cfg)
+	if err != nil {
+		return fmt.Errorf("list apps: %w", err)
+	}
+	var done []string
+	for _, a := range apps {
+		fmt.Printf("==> rebuilding %s\n", a)
+		if err := rebuildOne(cfg, a, *baseSnapshot, *iface); err != nil {
+			fmt.Printf("==> rebuilt %d/%d apps before %s failed\n", len(done), len(apps), a)
+			return fmt.Errorf("rebuild %s: %w", a, err)
+		}
+		done = append(done, a)
+	}
+	fmt.Printf("==> rebuilt %d/%d apps\n", len(done), len(apps))
+	return nil
+}
+
+func rebuildOne(cfg layout.Config, app, baseSnapshot, iface string) error {
+	unlock, err := lock.Acquire(cfg.LockPath(app))
+	if err != nil {
+		return fmt.Errorf("acquire app lock: %w", err)
+	}
+	defer unlock.Release()
+	return rebuild.Run(context.Background(), os.Stdout, cfg, app, baseSnapshot, iface)
+}
+
+// listApps has no other caller today - every other command already knows
+// its app name via --app.
+func listApps(cfg layout.Config) ([]string, error) {
+	matches, err := filepath.Glob(filepath.Join(cfg.JailConfDir, "*.conf"))
+	if err != nil {
+		return nil, err
+	}
+	apps := make([]string, len(matches))
+	for i, m := range matches {
+		apps[i] = strings.TrimSuffix(filepath.Base(m), ".conf")
+	}
+	sort.Strings(apps)
+	return apps, nil
+}
+
 func cmdConfig(args []string) error {
 	if len(args) < 1 {
 		return fmt.Errorf("usage: skiff config <set|unset|list> --app <name> [args...]")
diff --git a/internal/deploy/deploy.go b/internal/deploy/deploy.go
index 30846ac..8d32613 100644
--- a/internal/deploy/deploy.go
+++ b/internal/deploy/deploy.go
@@ -113,7 +113,7 @@ func Run(ctx context.Context, out io.Writer, cfg layout.Config, app, sha string)
 // as the other if it goes wrong, so both are guarded the same way. label
 // is used only in the final error message (e.g. "deploy", "config change").
 func GuardedApply(ctx context.Context, out io.Writer, cfg layout.Config, app, tag, label string, apply func() error) error {
-	dataset := cfg.AppDataset(app)
+	dataset := cfg.AppDataDataset(app)
 
 	existing, err := zfsutil.Snapshots(ctx, dataset)
 	if err != nil {
diff --git a/internal/jailmgr/jailmgr.go b/internal/jailmgr/jailmgr.go
index 9ddf9b8..5da5663 100644
--- a/internal/jailmgr/jailmgr.go
+++ b/internal/jailmgr/jailmgr.go
@@ -68,6 +68,10 @@ func Start(ctx context.Context, out io.Writer, app string) error {
 	return sh.Run(ctx, out, "", "service", "jail", "start", app)
 }
 
+func Stop(ctx context.Context, out io.Writer, app string) error {
+	return sh.Run(ctx, out, "", "service", "jail", "stop", app)
+}
+
 // Restart restarts an already-running jail, or starts it if this is its
 // first deploy and it isn't up yet.
 func Restart(ctx context.Context, out io.Writer, app string) error {
diff --git a/internal/layout/layout.go b/internal/layout/layout.go
index 34edc32..5ac6c26 100644
--- a/internal/layout/layout.go
+++ b/internal/layout/layout.go
@@ -38,8 +38,19 @@ func Default() Config {
 	}
 }
 
+// AppDataset is the OS layer, cloned from BaseDataset and mounted at
+// JailRoot(app) - the jail's own root. Named "appbase" (not "apps") to
+// read clearly alongside AppDataDataset now that the two are siblings.
 func (c Config) AppDataset(app string) string {
-	return fmt.Sprintf("%s/apps/%s", c.ZFSPool, app)
+	return fmt.Sprintf("%s/appbase/%s", c.ZFSPool, app)
+}
+
+// AppDataDataset is the persistent data layer, mounted at AppDir(app):
+// releases/, current, env, meta.json, and the app's own data.db. It is a
+// separate dataset from AppDataset so `skiff rebuild` can destroy and
+// reclone the OS layer without touching it.
+func (c Config) AppDataDataset(app string) string {
+	return fmt.Sprintf("%s/appdata/%s", c.ZFSPool, app)
 }
 
 func (c Config) JailRoot(app string) string {
diff --git a/internal/rebuild/rebuild.go b/internal/rebuild/rebuild.go
new file mode 100644
index 0000000..f58c67f
--- /dev/null
+++ b/internal/rebuild/rebuild.go
@@ -0,0 +1,126 @@
+// Package rebuild implements `skiff rebuild`: discard an app's OS layer
+// (everything cloned from base) and reclone it fresh, leaving the data layer
+// (a separate dataset) untouched. Guarded the same way as
+// deploy.GuardedApply: the old OS dataset is kept until the new one is
+// confirmed healthy, and swapped back automatically on failure.
+package rebuild
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"os"
+
+	"skiff/internal/appmeta"
+	"skiff/internal/health"
+	"skiff/internal/jailmgr"
+	"skiff/internal/layout"
+	"skiff/internal/zfsutil"
+)
+
+func Run(ctx context.Context, out io.Writer, cfg layout.Config, app, baseSnapshot, iface string) error {
+	osDataset := cfg.AppDataset(app)
+	osDatasetPrev := osDataset + "-prev"
+	jailRoot := cfg.JailRoot(app)
+	jailRootPrev := jailRoot + "-prev"
+	dataDataset := cfg.AppDataDataset(app)
+
+	// jail.conf.d/<app>.conf lives on the host, outside both per-app
+	// datasets, so swapping OS-layer datasets back on a failed rebuild
+	// doesn't revert it on its own - back it up so rollback can.
+	oldConf, err := os.ReadFile(cfg.JailConfPath(app))
+	if err != nil {
+		return fmt.Errorf("read current jail.conf: %w", err)
+	}
+
+	fmt.Fprintln(out, "==> stopping jail")
+	if err := jailmgr.Stop(ctx, out, app); err != nil {
+		return fmt.Errorf("stop jail: %w", err)
+	}
+	if err := zfsutil.Unmount(ctx, out, dataDataset); err != nil {
+		return fmt.Errorf("unmount data dataset: %w", err)
+	}
+
+	fmt.Fprintf(out, "==> setting aside current OS layer as %s\n", osDatasetPrev)
+	if err := zfsutil.Rename(ctx, out, osDataset, osDatasetPrev); err != nil {
+		return fmt.Errorf("rename OS dataset: %w", err)
+	}
+	// Rename doesn't move the mountpoint - relocate it explicitly, or the
+	// clone below silently mount-stacks on top of it instead of at jailRoot.
+	if err := zfsutil.SetMountpoint(ctx, out, osDatasetPrev, jailRootPrev); err != nil {
+		return fmt.Errorf("relocate old OS dataset mountpoint: %w", err)
+	}
+
+	fmt.Fprintf(out, "==> cloning fresh OS layer from %s@%s\n", cfg.BaseDataset, baseSnapshot)
+	if err := zfsutil.Clone(ctx, out, cfg.BaseDataset, baseSnapshot, osDataset, jailRoot); err != nil {
+		return fmt.Errorf("clone fresh OS dataset: %w", err)
+	}
+	if err := zfsutil.Mount(ctx, out, dataDataset); err != nil {
+		return fmt.Errorf("remount data dataset: %w", err)
+	}
+
+	if err := provisionAndStart(ctx, out, cfg, app, iface); err != nil {
+		fmt.Fprintf(out, "==> %v, rolling back to previous OS layer\n", err)
+		return rollback(ctx, out, cfg, app, osDataset, osDatasetPrev, jailRoot, dataDataset, oldConf, err)
+	}
+
+	fmt.Fprintf(out, "==> healthy, discarding previous OS layer %s\n", osDatasetPrev)
+	if err := zfsutil.Destroy(ctx, out, osDatasetPrev); err != nil {
+		return fmt.Errorf("rebuild succeeded but cleanup failed, destroy %s manually: %w", osDatasetPrev, err)
+	}
+	return nil
+}
+
+func provisionAndStart(ctx context.Context, out io.Writer, cfg layout.Config, app, iface string) error {
+	meta, err := appmeta.Read(cfg.MetaPath(app))
+	if err != nil {
+		return fmt.Errorf("read app metadata: %w", err)
+	}
+	if err := jailmgr.WriteConf(cfg, app, meta.IP, iface); err != nil {
+		return fmt.Errorf("write jail config: %w", err)
+	}
+	if err := jailmgr.EnableMail(cfg, app); err != nil {
+		return fmt.Errorf("configure mail: %w", err)
+	}
+	if err := jailmgr.Start(ctx, out, app); err != nil {
+		return fmt.Errorf("start jail: %w", err)
+	}
+	if err := health.WaitHealthy(ctx, meta); err != nil {
+		return fmt.Errorf("rebuilt jail failed health check: %w", err)
+	}
+	return nil
+}
+
+// rollback restores the previous OS layer, mirroring GuardedApply's failure
+// shape: the app ends up running on the dataset it was healthy on before
+// the attempt, and cause is always reported even if a rollback step fails.
+func rollback(ctx context.Context, out io.Writer, cfg layout.Config, app, osDataset, osDatasetPrev, jailRoot, dataDataset string, oldConf []byte, cause error) error {
+	fail := func(step string, err error) error {
+		return fmt.Errorf("rebuild failed AND rollback failed: %v / %s: %w", cause, step, err)
+	}
+	if err := jailmgr.Stop(ctx, out, app); err != nil {
+		return fail("stop jail", err)
+	}
+	if err := zfsutil.Unmount(ctx, out, dataDataset); err != nil {
+		return fail("unmount data dataset", err)
+	}
+	if err := zfsutil.Destroy(ctx, out, osDataset); err != nil {
+		return fail("destroy failed OS dataset", err)
+	}
+	if err := zfsutil.Rename(ctx, out, osDatasetPrev, osDataset); err != nil {
+		return fail("restore previous OS dataset", err)
+	}
+	if err := zfsutil.SetMountpoint(ctx, out, osDataset, jailRoot); err != nil {
+		return fail("restore OS dataset mountpoint", err)
+	}
+	if err := os.WriteFile(cfg.JailConfPath(app), oldConf, 0644); err != nil {
+		return fail("restore jail.conf", err)
+	}
+	if err := zfsutil.Mount(ctx, out, dataDataset); err != nil {
+		return fail("remount data dataset", err)
+	}
+	if err := jailmgr.Start(ctx, out, app); err != nil {
+		return fail("restart on previous OS layer", err)
+	}
+	return fmt.Errorf("rebuild failed and was rolled back to the previous OS layer: %w", cause)
+}
diff --git a/internal/zfsutil/zfsutil.go b/internal/zfsutil/zfsutil.go
index 918289c..5e68e30 100644
--- a/internal/zfsutil/zfsutil.go
+++ b/internal/zfsutil/zfsutil.go
@@ -41,6 +41,39 @@ func DestroySnapshot(ctx context.Context, out io.Writer, dataset, tag string) er
 	return sh.Run(ctx, out, "", "zfs", "destroy", snapRef(dataset, tag))
 }
 
+// Create makes a new, empty dataset mounted at mountpoint - unlike Clone,
+// it has no origin snapshot to inherit from.
+func Create(ctx context.Context, out io.Writer, dataset, mountpoint string) error {
+	return sh.Run(ctx, out, "", "zfs", "create", "-o", "mountpoint="+mountpoint, dataset)
+}
+
+// Rename renames a dataset. It deliberately does not touch the dataset's
+// mountpoint property, which zfs(8) leaves unchanged on a rename - callers
+// that care where the renamed dataset mounts must call SetMountpoint too.
+func Rename(ctx context.Context, out io.Writer, oldName, newName string) error {
+	return sh.Run(ctx, out, "", "zfs", "rename", oldName, newName)
+}
+
+func SetMountpoint(ctx context.Context, out io.Writer, dataset, mountpoint string) error {
+	return sh.Run(ctx, out, "", "zfs", "set", "mountpoint="+mountpoint, dataset)
+}
+
+func Mount(ctx context.Context, out io.Writer, dataset string) error {
+	return sh.Run(ctx, out, "", "zfs", "mount", dataset)
+}
+
+func Unmount(ctx context.Context, out io.Writer, dataset string) error {
+	return sh.Run(ctx, out, "", "zfs", "unmount", dataset)
+}
+
+// Destroy removes an entire dataset and any snapshots on it, as opposed to
+// DestroySnapshot's single dataset@tag. Recursive because callers use this
+// to discard a whole discarded OS-layer clone - an operator's own manual
+// snapshot on it shouldn't be able to block that.
+func Destroy(ctx context.Context, out io.Writer, dataset string) error {
+	return sh.Run(ctx, out, "", "zfs", "destroy", "-r", dataset)
+}
+
 // Snapshots lists snapshot names (without the dataset@ prefix), newest last.
 func Snapshots(ctx context.Context, dataset string) ([]string, error) {
 	raw, err := sh.Output(ctx, "", "zfs", "list", "-H", "-t", "snapshot", "-o", "name", "-s", "creation", "-r", dataset)