Debugging a sandbox that denies everything

Writing a deny-by-default sandbox is easy. (deny default) is one line. Everything after that line is debugging, and the debugging is strange because a sandbox fails by making other programs fail, in their own words, for reasons they do not know.

These are the failures I hit while confining agent steps in cadence, on macOS with sandbox-exec and on Linux with bubblewrap. Each is a few lines of fix and cost far more than a few lines of time. I am not a kernel engineer, and it shows in the order I found them.

1. The process dies before main

First profile, first run: the child aborts instantly. No output, no error from my command, because my command never started. The dynamic linker could not read the dyld shared cache, and on macOS almost nothing can start without it.

The fix is two parts. Import the system base profile, which carries what every process needs to exist, and allow reads on the cache:

(version 1)
(import "bsd.sb")
(deny default)
(allow file-read* (subpath "/private/var/db/dyld"))

(version 1)
(import "bsd.sb")
(deny default)
(allow file-read* (subpath "/private/var/db/dyld"))

Lesson: under deny-by-default, "my program crashed" and "my program was never born" look the same from outside. Check for the second first.

2. The rule that matches nothing

The step could not write inside its own jail, which the profile explicitly allowed. The jail lived under /tmp.

On macOS /tmp is a symlink to /private/tmp, and /var to /private/var. A sandbox profile is matched against the real path. A rule for /tmp/run/work is valid, loads without complaint, and matches nothing at all, because no file has ever had that path as far as the kernel is concerned.

if real, err := filepath.EvalSymlinks(p); err == nil {
    return real
}

if real, err := filepath.EvalSymlinks(p); err == nil {
    return real
}

Every path is resolved before it goes into a rule. There is a test for exactly this, because it is the kind of bug that comes back.

3. DNS works on my machine

On Linux, with /etc bound read-only into the namespace, agent steps could not resolve any hostname. curl by IP worked.

On a systemd-resolved host, /etc/resolv.conf is a symlink into /run/systemd/resolve. Binding /etc gives the sandbox the symlink and not its target. The file is present, readable, and points at nothing.

for _, p := range []string{"/run/systemd/resolve", "/etc/resolv.conf"} {
    args = append(args, "--ro-bind-try", p, p)
}

for _, p := range []string{"/run/systemd/resolve", "/etc/resolv.conf"} {
    args = append(args, "--ro-bind-try", p, p)
}

--ro-bind-try and not --ro-bind, because on a host without systemd-resolved the path does not exist and a plain bind of a missing source makes bubblewrap refuse to start. Which is failure number four.

4. Binding a path that is not there

Bubblewrap fails outright if a bind source does not exist. Distributions disagree about /lib64, /sbin and /opt, so a fixed list of system directories works on one distro and dies on the next. Every bind is now guarded by a stat, or uses the -try variant. The profile is generated per machine, not written once.

5. The private tmp that was not private

Each step gets its own /tmp as a tmpfs. Steps can also be granted extra read-write paths by policy. One day a policy included /tmp itself.

Bubblewrap applies its arguments in order. --tmpfs /tmp followed later by --bind /tmp /tmp mounts the host directory straight over the private one. Every step now shares the real /tmp, with every other step and every other process on the machine. Nothing fails. Isolation is simply gone, and all the tests that check "can the step write to its tmp" still pass.

if filepath.Clean(p) == "/tmp" {
    continue
}

if filepath.Clean(p) == "/tmp" {
    continue
}

This is the one that worries me in retrospect. A sandbox bug that produces an error gets fixed the same day. A sandbox bug that produces success can live for years.

6. The terminal is a way out

Bubblewrap's manual has a note next to --new-session. It is worth reading. A sandboxed process that shares the controlling terminal can push characters into its input queue with the TIOCSTI ioctl. Those characters are then read by whatever reads the terminal next, which is your shell, outside the sandbox. The filesystem is sealed and the escape goes through the keyboard.

One flag closes it. I would not have found it by thinking about files.

7. The test that only passed here

The regression test for failure five asserted that a specific path under /tmp was never bound. I used a directory that happened to exist on my machine. Locally: green. On CI: red, on every run, because the path did not exist there, and the code under test (correctly, see failure four) skips paths that do not exist.

So a test for one portability bug failed because of the fix for another, on a machine I could not see. The test now creates its own directory. And since Linux-only behaviour never shows up in a test run on a Mac, a push is not finished until the pipeline has been looked at.

What the seven have in common

None of these were logic errors in the usual sense. Every one was a wrong belief about the platform: that /tmp is a directory, that /etc/resolv.conf is a file, that mount order does not matter, that a terminal is just output.

Three habits came out of it.

Test by escaping, not by inspecting. Asserting on the generated profile text checks that I wrote what I meant. Running a process that tries to read a neighbour's file checks that what I meant is true.

Fear the silent success. After failure five, every isolation property has a test that tries to violate it, because the dangerous bugs are the ones where everything keeps working.

Report the degraded mode. When no enforcing backend is available the tool says so, on every run, and a workflow can refuse to start. A sandbox that might be off is, for planning purposes, off.

The code that came out of all this is short: about seventy lines per backend. Most of what a sandbox knows is not in the rules. It is in the comments explaining why each rule is there.