isopod

Dogfood findings

Gaps found by using isopod, per the standing rule: dogfooding is the primary gap-discovery mechanism. Every entry gets a severity and a fix-or-file decision. Oldest first — the sessions read as a chronology. Format: [status] severity — finding → decision.

Nine sessions over four days, each one a gauntlet thrown at whatever had just landed:

timeline
    title Dogfood sessions
    section 2026-07-21
        M2 surface gauntlet : exit codes, streams, truncation, timeout, concurrency
        M4 networking : live, after sudo isopod setup
        M4 acceptance : pip and git through isopod on the Alpine base
    section 2026-07-22
        Self-build : isopod builds its own workspace : M6 warm-pool verification
        MCP v2 gauntlet : all-MCP, including the self-build via MCP
        Findings-fix wave : 15 to 25 closed : proto v3 : images rebuilt
        Proto-v3 verification : the next-gauntlet checklist, post-restart
    section 2026-07-23
        Fix wave : 26 and 27, built in-sandbox and verified live
    section 2026-07-28
        0.12.0 dogfood : self-build on HEAD : the base-skew feature under real use

2026-07-21 — M2 surface gauntlet (exit codes, streams, truncation, timeout, env/cwd, binary, errors, concurrency)

What was thrown at isopod run: exit 42; stderr-only output; 200 KB stdout (> 64 KiB cap); sleep 30 under --timeout-s 3; --env/--cwd; 4 KiB of /dev/urandom; a nonexistent binary; two concurrent runs; 25-run directory accumulation. All core behaviors correct (truncation exact, full logs retained, timeout kill in 3.05 s wall, signal 9 reported, concurrent runs isolated with distinct vanity names).

  1. [fixed 9ecbd79] HIGH — vanity names exist but nothing lists or resolves them. Names are persisted in each VM dir's meta.json, but there is no isopod vm ls, so a user/model cannot look up resilient-legionary after the fact — which defeats the point of memorable handles. → FIX at M3 integration: isopod vm ls (id, name, flavor, created, status) reading the meta.json files; name→vm resolution helper shared with stages.

  2. [fixed 9ecbd79] MEDIUM — ~/.isopod/vms/ grows without bound. 25 dirs / 600 KB after one day of testing; harmless now (logs only), but every run adds one and nothing prunes. → FIX at M3 integration: isopod vm gc [--keep-last N] [--older-than 7d] with sane defaults; consider auto-gc on run.

  3. [fixed 9ecbd79] MEDIUM — command-not-found is indistinguishable from infra failure. isopod run -- /bin/nonexistent yields {ok:false, error:"exec over vsock: guest agent reported an error: exec: No such file or directory"} — same shape as a genuine sandbox/transport failure, and no exit code. Callers (especially the future MCP tool) need to tell "your command is wrong" from "the sandbox broke". → FIX before M5: spawn-failure becomes a structured outcome (exit_code:127-style or an error_kind field), reserving ok:false for infrastructure faults.

  4. [fixed M5] LOW — no stdin plumbing. The proto supports stdin_b64 end-to-end but isopod run has no --stdin/--stdin-file, so piping data into a sandboxed command requires a file-put dance that doesn't exist yet either. → file for M5 (MCP file_put + a --stdin-file flag land together).

  5. [fixed 9ecbd79] MEDIUM — guest rootfs has no /tmp. Found by probing the guest environment: echo t > /tmp/x fails on a fresh dev-agent VM (the dir simply isn't in the image; mkdir -p works). A large fraction of real scripts and tools assume /tmp. → FIX at M3 integration: add /tmp (mode 1777) and /var/tmp to every flavor's populate step (dev-busybox, dev-agent, base-sqfs). Guest-env facts for the record: 235 MB usable RAM of 256 configured, ~53 MB free rootfs, 302 busybox applets, uid 0.

2026-07-21 — M4 networking (live, post-sudo isopod setup)

Egress works (ICMP + DNS through the NAT), concurrency lands on distinct slots (0/1), host isolation holds (guest can't reach the host tap — the iifname input-drop fix), --no-network attaches no NIC. Two findings:

  1. [fixed f332743+] HIGH — a leaked firecracker holding a tap breaks its slot until manually killed. A VMM that outlived its run (here dev-85eddd65 from an earlier failed attempt) kept isopod-tap0 open, so every later slot-0 run died with EBUSY at PUT /network-interfaces — a confusing, persistent failure with no self-recovery. The stale-lock sweep only reclaims locks whose pid is dead; a live-but-orphaned VMM defeats it. → FIX: (a) claim should verify the tap is actually openable (or that no firecracker holds it) and either reclaim or skip to the next slot; (b) harden run teardown so a VMM is never orphaned (audit every error path between spawn and shutdown; the FcProcess Drop guard should cover it — find why this one escaped); (c) isopod vm gc / a --kill-stale should reap orphaned VMMs. Worth fixing before M5 (an MCP client hitting a wedged slot would be baffling).

  2. [note] MINOR — HTTP-by-IP to 1.1.1.1:80 is an unreliable egress probe. wget http://1.1.1.1 doesn't cleanly return 200 (Cloudflare redirects to HTTPS), so it's a poor liveness check even though egress works. Use ICMP + DNS (both confirmed) in the runbook; drop the plain-HTTP-by-IP check.

2026-07-21 — M4 acceptance (pip/git through isopod, Alpine base)

The marquee test passed: bare pip install requests into an Alpine stage → commit → fork BY VANITY NAME → import requests with no reinstall → parent byte-identical. Three fixes fell out of running it:

  1. [fixed 0a37865+] HIGH — bare pip install failed (PEP 668). Alpine's Python 3.14 ships an EXTERNALLY-MANAGED marker, so pip install errored and an agent would have to know --break-system-packages. In a disposable sandbox that protection is pure friction. → FIXED: the base-alpine build removes every pythonX.Y/EXTERNALLY-MANAGED marker; bare pip install now works.

  2. [fixed] HIGH — --commit-as committed a stage even when the command FAILED. The first pip run errored (PEP 668) yet still committed a stage (lucent-crucible) missing the package — a silent footgun for anyone who later forks it. → FIXED: --commit-as now commits only on exit_code == 0, logging a clear skip reason otherwise.

  3. [fixed] HIGH — a stage didn't record which base it was built on. Meta hardcoded base: base-sqfs regardless; forking an Alpine-built stage without remembering --base base-alpine would mount alpine layers over a busybox base (site-packages but no interpreter) — a silent broken merge. → FIXED: stage::commit records the true base flavor; a fork auto-uses the recorded base (verified: forking with no --base runs Python 3.14), and stacking enforces a single base per chain.

  4. [note] DOC — --timeout-s budget includes boot. --timeout-s 3 gives the command ~2.6 s of real exec time (boot consumes ~0.4 s of the budget). Reasonable semantics for an outer wall clock, but must be documented in the CLI help and the eventual MCP tool description (whose default timeout should account for it).

2026-07-22 — self-build dogfood (isopod builds its own workspace) + M6 warm-pool verification

Headline positive: isopod compiled its own full 6-crate workspace — 182 crates including rustls / aws-lc-sys / reqwest / tokio / rmcp — in 96 s inside an isopod sandbox, and the freshly-built isopod binary ran (isopod 0.1.0) inside the sandbox. Recipe exercised the stage-fork model end to end: stage the toolchain once (rustup stable-musl 1.97.1 → rust-stable stage in 22 s), then fork it and build with target/ on a guest tmpfs. Proof the stage/fork model + M5.5 flex resources carry a real heavy workload. Five gaps fell out.

  1. [fixed] HIGH — sandbox networking doesn't survive a WSL2/host restart, and the failure is cryptic. The user-owned isopod-tap0..7 are non-persistent; after the WSL utility VM recycles they vanish and every networked run/sandbox_run fails with a raw Firecracker string — Open tap device failed: Operation not permitted ... Invalid TUN/TAP Backend provided by isopod-tap0 — with no hint the fix is sudo isopod setup. Hit both the MCP path and the M6 agent this session. → FIXED: require_network_setup now, when a manifest exists, also checks the provisioned taps are actually present (net::provisioned_taps_present → /sys/class/net/isopod-tap<i>) and fails fast — before any disk work — with an actionable "networking was provisioned but its tap devices are missing — the host was most likely restarted … re-run sudo isopod setup (or --no-network)" message. (Runtime is unprivileged, so auto-reprovision isn't possible; the clear message is the fix. Unit-tested via an injected presence predicate.) (PLAN networking risk #3, made concrete.)

  2. [fixed] MED — writable scratch (overlay upper) was fixed at ~1 GiB with no size knob. A minimal rustup toolchain (799 MiB) nearly fills it (98 MiB free, 89 %); a real build (target/ reached 1.5 GiB) can't fit at all. Workaround that unblocked the self-build: mount a tmpfs in the guest and point CARGO_TARGET_DIR/RUSTUP_HOME/CARGO_HOME at it (trades RAM for space). → FIXED: added --scratch-mib (CLI) / scratch_mib (MCP sandbox_run), bounded 128..=65536 MiB, validated before boot (clear range error, no VM launched). The image is sparse, so a large apparent size costs little host disk until written. Verified live: 4096 → 3.9 G overlay, 8192 → 7.8 G. Passing it forces the cold ext4 path (a warm resume uses a RAM tmpfs upper), so the requested size always takes effect; --mem-mib remains the lever for a bigger RAM upper on warm runs.

  3. [fixed] MED — no Rust toolchain in any base, and base-alpine has no apk. The squashfs base bakes in python/node/git/gcc/make/cmake but strips the package manager, so you can't apk add a toolchain at runtime. curl/xz/bash also absent (wget/gzip/tar/base64 present). rustup-over-wget works fine. → for a system whose own dogfood is "build yourself", consider a toolchain-bearing base flavor, or keep apk available in base-alpine. → FIXED (2026-07-22 wave): base-alpine retains the verified static apk + signing keys (in-guest apk add jq verified live) and adds cmake + GNU coreutils; Rust toolchains stay stages (rust-stable), which the self-build proved out.

  4. [fixed] LOW/footgun — --base X without --stage silently boots the legacy dev-agent ext4 rootfs, ignoring --base. The flag appears to do nothing — and, mid-M6, that legacy rootfs still carried the old proto v1 guest, surfacing a confusing "guest 1 does not match host 2" until you realise --base only applies with --stage. → warn/error when --base is passed without --stage, or make --base imply the squashfs/overlay topology. → FIXED (2026-07-22 wave): a lone --base is now a hard error naming both valid spellings.

  5. [fixed] proto-version skew across guest images after a PROTO_VERSION bump. Bumping to v2 for ConfigureNet (M6) requires rebuilding every guest image (base-sqfs, base-alpine squashfs, legacy dev-agent) and restarting any long-lived isopod-mcp server, or the guest baked into one image (or the stale server binary) mismatches. Credit: the error is clear and names both versions. → build tooling should rebuild all guest images together + stamp their proto version; surface per-image proto version in a status command. → FIXED (2026-07-22 wave): every build stamps <image>.meta.json (flavor, proto, agent sha256, image sha256); run paths refuse a stale image pre-boot naming the fix; isopod image build-all force-rebuilds every flavor together and image ls shows per-image proto + stale/unstamped. Exercised for real by the v2→v3 bump.

Hypothesis retracted: I expected aws-lc-sys to fail for want of cmake — it built cleanly (base-alpine ships cmake for node-gyp, and aws-lc-sys has a cc path). Not a finding.

Concurrency stress (positive, no gap). 6 networked run --stage base launched in parallel all warm-resumed from the one shared 512 MiB snapshot, each claimed a distinct slot (0–5) with its own /30, all exited 0 with NET-OK, and left zero leaks (no firecracker procs, no held slot locks). The O_EXCL slot-claim held under this contention and concurrent resume from a single read-only memfile is safe — the core multi-agent model holds under load.

Corrected 2026-07-26. This was read at the time as "the slot claim is race-free", which it was not. Six simultaneous launches is the one shape the O_EXCL-plus-staleness claim got right; runs staggered by more than a few seconds is the shape it got wrong, and neither this stress nor the unit tests ever ran that. The claim is now an flock held for the run's lifetime, which does not depend on timing at all. See the 0.11.0 changelog entry.

2026-07-22 — MCP v2 gauntlet (post-restart, all-MCP) + self-build via MCP

Session restart picked up the proto-v2 isopod-mcp binary; the whole surface was re-verified through the MCP tools alone: a 6-agent workflow ran ~37 scenarios (exec semantics, stage lifecycle, network/F1, resource caps, toolchain, warm pool) with an adversarial coverage critic, plus inline probes. Headline: the full workspace now builds inside an isopod sandbox driven end-to-end over MCP — clean debug build 2 m 05.9 s (4 vcpu / 3072 MiB / 8 GiB scratch, crates.io downloads included), committed as stage isopod-build (1.53 GiB layer, +34.3 s commit), incremental rebuild 6.93 s from a fork, cargo test -p isopod-proto -p isopod-fc green in 39.6 s, release isopod in 2 m 35 s, and the binary extracted byte-exact to the host (stdout log 14,300,661 B complete; decoded 10,585,920 B, sha256 match) where it runs directly — file: static-pie musl, no loader dependency. Chain: rust-stable → isopod-src → isopod-build, both new stages retained for future builds (see docs/sandbox-build.md).

Corrections to the 2026-07-22 self-build entry above: - cmake was never in base-alpine. ALPINE_PACKAGES (rootfs.rs) never listed it and git log -S cmake on the file is empty; cmake --version in-guest fails. #15's "python/node/git/gcc/make/cmake" list and the retracted-hypothesis note ("ships cmake for node-gyp") were wrong — aws-lc-sys built via its cc path with no cmake present. - The warm pool DOES engage via MCP — but invisibly, and two gauntlet agents misdiagnosed it as broken (a warm-eligible run's console.log is 121 B: agent re-IP line only, no kernel boot). What looked like a "provably cold" comparator (2 c/1024 MiB) had silently built its own snapshot on first use (5.4 s) and warm-resumed afterwards. End-to-end: warm ≈ 430 ms, cold ext4 path ≈ 570–700 ms, first-use-of-a-shape snapshot build ≈ 5.4 s. The M6 "resume 52–72 ms" figure is the restore step, not wall time. See #20.

  1. [fixed] MED — a bad cwd fails blaming /bin/sh, not the missing directory. sandbox_run with cwd="/no/such/dir" returns exit 127 with stderr isopod-exec: /bin/sh: No such file or directory (os error 2) — the natural read is "the image has no shell" (and 127 usually means command-not-found), a wrong-way debugging lead. → isopod-exec should check/chdir the cwd first and report cwd '/no/such/dir': No such file or directory.

  2. [fixed¹] MED — stale-proto guest failure is masked by a tap error on the networked path. base-sqfs (still proto v1, #17) with default networking fails as Open tap device failed: … Device or resource busy … Invalid TUN/TAP Backend provided by isopod-tap0 (reproduced twice; base-alpine on the same slot works immediately after) — pointing at host networking instead of the real cause. Only network=false surfaces the correct guest agent protocol version 1 does not match host 2. → surface the proto mismatch before/instead of the tap error; also worth checking whether any cold boot can transiently collide with a warm-pool-held tap.

  3. [fixed] LOW — MCP result JSON omits boot-path and commit-cost observability. The CLI result has path: "cold"|"warm"; the MCP result doesn't, which is exactly why the warm pool was misdiagnosed mid-gauntlet. commit_as runs also fold commit time into total_ms (1612 ms total vs 41 ms exec on a trivial commit; ~34 s for a 1.53 GiB layer ≈ 20 s/GiB). → add path (incl. distinguishing "snapshot-build" from plain cold) and commit_ms to the MCP result.

  4. [fixed] LOW — no first-class host↔guest file channel. Payload-in: MCP stdin transits model context twice, so the 290 KB source tarball (~75 k tokens) is unusable over MCP — injection had to use CLI --stdin-file (which worked perfectly). Artifact-out: base64 over stdout is lossless (log file byte-exact at 14.3 MB) but floods the tool result with a truncated blob. → sandbox_run stdin_file (host path) + a copy-out parameter (guest path → host file); a git remote will also fix source-in.

  5. [documented] — parallel sandbox_run tool calls in ONE Claude message execute serially. Six batched calls all ran on slot 0 at ~3.3 s each (a genuine overlap would force distinct slots). Client-side behavior, not a server bug: concurrent requests from separate agent processes interleaved fine during the gauntlet (slots 0/1 held simultaneously), matching the 6-way CLI proof. Guidance for agents: fan out via subagents for parallel sandboxes.

  6. [fixed] — guest hostname is (none). $(hostname) in-guest prints (none); setting it to the vanity VM name (e.g. lucent-cryptarch) would improve log/prompt ergonomics.

  7. [fixed] — rootfs.rs comment implies in-guest apk add, but no apk ships. The keep-parent-dirs comment says "so an online guest can apk add more packages later", yet command -v apk apk.static finds nothing in-guest (re-verified). Align the comment with however #15 is resolved.

Positive re-verifications (all via MCP): - F1 egress proven against a live service: host connects to its own LAN listener (<host-lan-ip>:3478) instantly; the guest gets a filtered timeout on that same listening port, and on gateway:80, and on RFC1918/link-local probes — while DNS through the gateway works. Drop-not-refused + live-listener evidence closes the "maybe nothing was there" gap. - Truncation: 600 KB stdout → in-band string capped, stdout_truncated=true, stdout_bytes exact, log file complete (verified byte-exact at 14.3 MB during extraction). Binary stdout is lossy U+FFFD in JSON but byte-accurate in the log; stdout_bytes counts raw bytes. - Resource-cap errors are uniformly self-serve (vcpus 1-or-even w/ examples; host CPU cap; 128 MiB mem floor; over-mem shows the full headroom arithmetic; scratch range 128..=65536). - Stage model: commit-on-zero only (exit 3 → no stage), chain/parent info correct, whiteouts work, parents immutable, stage_rm protection names every dependent by id+label (excellent), child-first removal clean; label + vanity-name + full-id resolution work (id prefix is not supported — fine per docs, but docker/git-style unique-prefix ids would be nicer). - Timeout shape: timed_out=true, exit_code=null, signal=9, partial stdout preserved. - network=false: no NIC (fields absent from result), exec fine over vsock; offline forks of a pip-carrying stage import the package with no network. - Quoting/UTF-8/env/cwd/stdin (12 B and 8 KB) all exact.

Next-gauntlet checklist (from the adversarial coverage critic — none ever covered): duplicate + concurrent commit_as labels; timeout during boot/commit and whether commit_as fires on timed_out; stderr truncation + dual-stream flood (pipe-deadlock class); unconsumed stdin (EPIPE) and 64 KB–1 MB stdin; hostile labels (unicode, ../x, very long) and env names (=, empty, PATH/HOME override); opaque-dir whiteouts (rm -rf + recreate) and 8–16-layer chains; cwd into stage-created/whiteouted dirs; VM-record/exec-log retention under a long-lived MCP server (vm_gc semantics, dangling *_log_path); ICMP egress; nonexistent command via MCP (#3 regression probe).

2026-07-22 — findings-fix wave: #15–#25 closed, proto v3, images rebuilt

One coordinated pass (plan-mode designed, code-explorer-mapped) closed every open finding. Host-only wins first (#20 observability fields, #16 --base hard error, MCP stdin_file, auto-GC at startup + every 20 runs, #22 docs), then a proto-v3 wave: SetHostname (#23), streamed CopyOut (#21, CLI --copy-out / MCP copy_out), #18 cwd error fix, apk + cmake + coreutils in base-alpine (#15/#24), image sidecars + image build-all/image ls + pre-boot skew guard (#17, unmasks #19), and SnapshotKey v2 keyed on the base image's content id:

  1. [fixed] MED — SnapshotKey ignored image content, so warm snapshots survived image rebuilds as silent stale resumes (surfaced by plan-mode exploration; almost certainly bit the v1→v2 bump too). Key material v2 adds the sidecar-recorded image sha256, cheap to read per run. A rebuilt base now simply keys to fresh snapshots.

Verified live post-cutover (all four images rebuilt + stamped proto v3, warmpool cleared): guest hostname == vanity name on cold boot and warm resume; first warm-eligible run reports snapshot_built:true with the ~4 s build visible in total_ms, next run path:"warm", resume_ms:56, 406 ms total; bad cwd → isopod-exec: cwd '/no/such/dir': No such file or directory; --copy-out extracted a 0755 artifact byte-exact; in-guest apk add jq → jq 1.8.1; cmake 4.2.3 + GNU coreutils 9.11 present; base-sqfs boots networked and warm-resumes (the #19 mask scenario is gone); image ls shows all images proto 3, none stale. Milestone: the full workspace test suite now runs inside a sandbox — 132/132 core tests in-guest (GNU cp closed the last host-only gap).

¹ #19's masking is fixed and NIC errors now name slot + tap; the original tap-busy collision itself was never reproduced (static analysis cleared the slot claim, FC restore override, and shutdown ordering) — a live repro attempt is queued for the next gauntlet.

Caveat: the long-lived isopod-mcp server must restart to pick up proto v3 — until then the MCP tools fail fast against v3 guests (by design, and now pre-boot). Full MCP-side re-verification + the checklist gauntlet run after the restart.

2026-07-22 (post-restart) — proto-v3 MCP verification gauntlet (the "Next-gauntlet checklist")

The isopod-mcp server was restarted onto proto v3 and the whole never-covered checklist was run through the MCP tools alone, orchestrated as a phased workflow: a reachability canary → 8 parallel scenario buckets → an isolated concurrent-same-label commit race → the queued #19 tap-busy live repro (4 agents) → vm_gc/retention → adversarial verification of every candidate finding (to kill probe artifacts) → a coverage critic. 49 scenarios, 23 agents, ~843 k tokens, ~17 min. Result: 1 HIGH and 2 LOW confirmed; 2 candidate findings refuted on verification. Everything else PASS/INFO, including the full F1 re-verification.

  1. [fixed] HIGH — forking a stage with ≥10 overlay layers silently breaks and boots on the wrong rootfs. (Fix + live verification: see the 2026-07-23 fix-wave section below.) sandbox_run happily commits a 10th layer (exit 0), but forking any depth-≥10 stage boots onto the read-only squashfs base root with all committed layer state invisible, writes failing — and the MCP result returns a normal exit 0 with no error field (a command that happens to exit 0 could even commit_as a bogus stage). Live repro: gaunt-chain-10 fork → /bin/sh: can't create /root/chain/l11: nonexistent directory; /proc/mounts shows /dev/root / squashfs ro (no overlay), /dev/vdb..vdj mounted at /layers/1..9, but /layers/10 absent; console: [isopod-agent] overlay: FAILED to assemble stage root (layers=10, upper=drive): mount layer /dev/vdk at /layers/10: No such file or directory (os error 2); continuing on the read-only base root. Root cause (crates/guest-agent/src/overlay.rs:189-193): the layer-mount loop mounts each drive at /layers/<i+1> but never create_dir_alls the mountpoint (unlike UPPER_DIR/WORK_DIR at :185-186); it relies on mountpoints pre-baked into the base image, which only ship /layers/0..9, so the 10th layer's /layers/10 doesn't exist → mount(2) ENOENT. Practical cap is 9 committed layers. The unit tests (overlay.rs:354-355) only assert layer_mountpoint(1) and (9) — never 10, so the cap was never caught. The silent half (overlay.rs:65, 87-96): an assembly failure is logged to the console only and boot proceeds on the base root ("best-effort by design"); nothing propagates to the host / MCP result. → FIX (two parts): (a) one-liner — std::fs::create_dir_all(&mnt)?; before the mount in the :189 loop, which removes the pre-baked-mountpoint dependency and lifts the cap entirely (add an integration test forking a 16-layer chain); (b) surface overlay-assembly failure as a run error (or at least a overlay_degraded: true flag in the exec result) instead of silently booting on the wrong rootfs — otherwise a broken deep fork masquerades as a healthy exit-0 run. (b) is a design call — making it fatal changes boot behavior — so worth an explicit decision. Requires a guest-agent rebuild + re-stamp of all images and a gauntlet re-run to close.

  2. [fixed] LOW — env keys are forwarded to the guest execve environment without validation. (Fix + live verification: see the 2026-07-23 fix-wave section below.) env={"FO=O":"bar"} is accepted silently and lands in the guest environ as the ambiguous entry FO=O=bar (any parser reads it as FO="O=bar", not the requested name); env={"":"bar"} lands as a nameless =bar entry. Verified at the raw /proc/self/environ level (not a busybox env display quirk). Neither crashes or wedges the guest agent, and later runs are unaffected — POSIX env names must be nonempty and =-free, but execve doesn't enforce it and the passthrough is faithful, so this is a minor input-validation gap, not a malfunction. Source: crates/guest-agent/src/exec.rs:84 (for (k, v) in &req.env { cmd.env(k, v); } — no key check). → FIX: reject keys matching /=|^$/ with a clear pre-boot error (host-side in the MCP/CLI param validation, mirroring the stdin/stdin_file -32602 style), or skip+warn in exec.rs.

Refuted on verification (recorded so they aren't re-raised): - Commit runs return stage_id/stage_name, not the "documented" commit_id (G5, first raised LOW) — refuted: the string commit_id exists nowhere in the repo; stage_id + stage_name are the interface. No drift. (The workflow schema I wrote carried the wrong field name in from the checklist prose — the verifier caught it.) - sandbox_run doesn't return the committed stage's vanity name, so callers misreport it (R2, first raised LOW) — refuted: the result does carry stage_name; both race agents simply reported vm_name by mistake. The committed name is directly available and can't be confused with the VM name.

Positive re-verifications (all via MCP, all PASS): - F1 egress hardening holds after the restart — all 7 RFC1918/link-local probes (incl. 169.254.169.254) DROP with a full ~3 s socket timeout (no CONNECTED, no fast ConnectionRefused = pure drop semantics), while public ICMP (1.1.1.1), DNS, and HTTP (example.com → 200) all work — destination-scoped, not a blanket kill. network=false is airtight (no guest_ip/slot fields, instant OSError, exec still over vsock). - #3 nonexistent-command and #18 cwd-error stay closed (structured exit 127, correct isopod-exec: cwd '…': No such file or directory). - stderr truncation (first stderr-side probe): 200 000 B → 64 KiB in-band cap, stderr_truncated=true, stderr_bytes exact, on-disk log complete at 200 000 B. - dual-stream 8 MB+8 MB interleaved flood (the pipe-deadlock probe): no deadlock — completed in 176 ms, both streams truncated in-band with exact *_bytes, both logs complete at 8 000 000 B each. - stdin: unconsumed 1 MiB stdin_file → clean exit 0 (host writer tolerates EPIPE); 64 KiB and 1 MiB delivered byte-exact (wc -c = 65536 / 1048576); stdin+stdin_file together and stdin_file="-" both rejected pre-boot (-32602). - hostile labels (unicode 😀-café-Ω, ../../../etc/passwd-pwned, 410-char) all safe: labels are pure metadata, stage dirs are always st-<hex>, nothing written outside ~/.isopod/stages (host-checked). Duplicate labels → clean ambiguity error naming all candidates (-32603), no silent pick. - opaque-dir whiteout (rm -rf /data + recreate): only the new file visible, no lower-layer bleed. cwd into a stage-created / whiteout-recreated dir: works. - proto-v3 fixes: hostname == vanity name on cold and warm; copy_out byte-exact (sha match, mode 0755); in-guest apk add jq → jq 1.8.1; cmake 4.2.3 + GNU coreutils cp 9.11; observability fields (path/resume_ms/snapshot_built/commit_ms) all present. - timeout: exec-timeout → timed_out=true, exit_code=null, signal=9, partial stdout kept; commit_as on a timed-out run correctly commits NOTHING (the key never-tested interaction); boot-timeout edge leaks no slot. - concurrent same-label commit race: store fully consistent — both stages present, all 29 vanity names unique, both forkable with distinct uncontaminated content, metadata survived the server restart. (Commits landed 2 s apart — batched MCP calls serialize — so a truly simultaneous index write wasn't forced; behavior nonetheless correct.) - #19 tap-busy: NOT reproduced across 12 networked base-sqfs runs (cold→warm→warm on slot 0, egress live each time). Caveat — under-powered: the workflow cap is 2 agents and each agent's runs were sequential, so the concurrent / crashed-owner tap-reclaim paths (crates/core/src/net.rs slot allocator) were not exercised. Absence here ≠ absence. - vm_gc/retention: keep_last=5 keeps exactly the 5 newest ∪ sub-60 s records (the 62 s record was correctly pruned — the 1-minute grace is a hard cutoff), disk physically freed (39→5 dirs), no over-prune, and pruned runs' *_log_path become dangling by design (matches the docstring warning).

Note — a security-heuristic false positive during the run: the R2 verifier was flagged for stage_rm "solar-psion" "with no evidence it was created this session". Investigated and cleared: solar-psion was that agent's own throwaway — it ran commit_as="r2-verify-commit-name-probe" (which returned stage_name=solar-psion) to check whether the result exposes the stage name, then deleted its own test stage. All 9 pre-existing stages remained intact; no real stage was lost.

Next-gauntlet checklist (from the coverage critic — genuinely open): - Post-#26-fix boundary sweep: chains at exactly 9/10/11/16 layers; assert layer-1 content is visible at layer 16 and writes land. Plus a silent-fallback guard probe that asserts overlay-assembly failure surfaces as a run ERROR, not exit-0 on the base root. - cwd into a WHITEOUTED dir (fork a stage where the cwd target was rm -rf'd in a later layer) — #18-style clean spawn error expected; only the stage-created case was covered. - Deep opaque-dir whiteout (recreate at layer ~8 through a long lowerdir chain) — blocked by #26 today. - Deterministic timeout-DURING-commit (dirty 2–4 GB of incompressible scratch so the commit genuinely runs multi-second; assert timed_out, no partial/orphan stage, store integrity). H4 sparsified to ~632 KB so this stayed unobserved. - Concurrent tap gauntlet: N>slot-count simultaneous networked runs (expect graceful slot-exhaustion, not EBUSY) and SIGKILL-a-networked-VM-then-relaunch (crashed-owner tap reclaim) — the actual #19 failure class, still unexercised. - Warm-vs-cold matrix: record the path field per scenario and force both; the RAM-upper (warm) vs drive-scratch (cold) overlay paths diverge, and big-write behavior on the RAM upper (ENOSPC/OOM within mem_mib) is uncovered. - vm_gc racing a live run: invoke vm_gc keep_last=1 mid-sandbox_run; assert the live record + its *_log_path survive. - Regression probes once #26/#27 land (env-name rejection; the deep-chain fix).

2026-07-23 — #26/#27 fix wave (built in-sandbox, verified live)

Root-cause refinement for #26: the intended chain-depth cap was always 10 (stage.rs MAX_CHAIN_DEPTH, derived from Firecracker's virtio-MMIO IRQ slot budget and enforced at both commit and chain_paths) — the bug was an off-by-one between the baked mountpoints and the 1-based layer indexing: base images shipped /layers/0..9 while a depth-10 chain needs /layers/1..10, so exactly the last permitted depth broke. (The gauntlet's "≥10 layers" phrasing was thus really "the depth-10 boundary"; depths >10 were always refused loudly by the cap.)

Fixes landed: - #26a (guest, overlay.rs) — layer mountpoints now live on a tmpfs mounted over /layers and are create_dir_all'd per layer. The base root is a read-only squashfs, so the naive create_dir_all fix would have EROFS'd; the tmpfs removes the baked-mountpoint dependency for any depth the cap permits. - #26b (proto + guest + host) — overlay-assembly failure is no longer silent: the guest records it and reports it in every Pong (additive overlay_error field, proto stays v3); the host's ping() turns it into a fatal AgentError::OverlayDegraded (exactly parallel to ProtoMismatch), so all readiness paths — run, snapshot build, warm resume — refuse to proceed on a wrong rootfs instead of returning exit 0. A degraded snapshot can never be cached. The guest still boots to the base root for serial-log diagnosability (PID 1 must not die), but no exec is served by a run. - #27 (host + guest) — env validation at two levels: core::vm::validate_env rejects empty/=/NUL names and NUL values pre-boot (the shared choke point covering the MCP map, which parse_env_kv never sees), and the guest agent independently rejects the same shapes before execve (defense in depth, exit 127 + isopod-exec: stderr). - Readiness error contexts reworded ("readiness check failed" instead of "did not answer a ping") so an OverlayDegraded/ProtoMismatch cause isn't wrapped in a misleading message. - Not landed: a redundant 24-layer "vdz naming ceiling" guard drafted during the fix was removed on review — MAX_CHAIN_DEPTH = 10 already governs strictly tighter at both commit and resolve, so the device-naming ceiling is unreachable.

Build: full workspace built + tested inside an isopod sandbox (isopod-build stage, offline — taps were down, see below; stdin_file source injection + copy_out extraction). All tests green (135 core + 42 + 39 + guest/proto/cli suites, 0 failures) including new units: layer_mountpoint(10), Pong overlay_error additive-shape round-trip, host ping_rejects_degraded_overlay_root, validate_env, guest validate_env_pair. Warmed build cache committed as stage isopod-build/2026-07-23-fix26. All four images re-stamped (proto 3, agent 6b7d85db52c3…); stale warm-pool snapshots (old base ids) cleared.

Live verification (new CLI + images): - Depth matrix at the real boundary: depth 9 fork OK (regression); depth-10 fork of gaunt-chain-10 now works — all 10 markers visible, writes land, /proc/mounts shows the 10-deep overlay root (lowerdir=/layers/10/upper:…:/layers/1/upper:/); depth-11 commit refused pre-emptively with the clear MMIO-budget error. (The checklist's "16-layer" probe is moot — 16 was never bootable by design.) - MCP path, no server restart needed for the guest side: sandbox_run forking gaunt-chain-10 through the running server → 10 markers, overlay root, exit 0 (images are read per run). env={"FO=O":"bar"} → exit 127, invalid environment variable name "FO=O"; env={"":"bar"} → exit 127, name must not be empty (the guest-side defense; host-side pre-boot rejection activates on the next MCP server restart). - The silent-fallback guard (#26b) is covered by unit + proto-shape tests; a live forced-assembly-failure probe would need a deliberately broken image and stays on the checklist.

Environment notes: WSL2 was restarted since the gauntlet — tap slots are gone, so networked runs fail with the correct #13 guidance until sudo isopod setup is re-run (all of the above verified with --no-network; warm-pool paths therefore unexercised this wave). The long-lived MCP server still runs the pre-fix host code: guest-side fixes are already effective through it (images re-read per run); validate_env, OverlayDegraded, and the reworded contexts engage on its next restart (binaries at target/release/{isopod-mcp,isopod-jail} are already the fixed builds).

2026-07-28 — 0.12.0 dogfood (self-build on HEAD, then the base-skew feature under real use)

The first session against 0.12.0 after the base-skew wave and the adversarial review of it. Two halves: build isopod's own HEAD inside isopod, then use the new machinery the way an operator would — rebuild a base and find out what happens to the store.

The self-build still works, and the tree is green in a clean guest. Forked isopod-build/2026-07-23-fix26, injected HEAD's source as a tar over stdin_file: cargo build --workspace 29.91 s; cargo test --workspace 502 passed, 0 failed, 11 ignored across 20 suites, the same totals the host reports; cargo build --release -p isopod-cli 52.69 s with copy_out streaming 11 466 472 bytes to the host, where the static-pie musl binary ran and read the real stage store. Both builds ran under default-deny egress: the ledger recorded exactly the two crates.io flows the build needed, denied: [], nothing else attempted.

The base-skew feature does what it claims. A stamped two-deep chain, a base rebuild, and the fork was refused before boot on both surfaces with identical text, naming the stale ancestor rather than the stage asked for. A pre-0.12.0 unstamped chain booted clean and silent on the rebuilt base — the legacy stages survive. The rebuilt image layout (no /rom, an empty /layers) mounts and stacks correctly at depth 2, and the guest builds its mountpoints on the tmpfs as designed.

What the session found is what surrounds it. One image rebuild sets off a cascade that no command reports:

flowchart TB
    R["isopod image build-rootfs --force<br/>or image build-all"] --> ID["new content id<br/>even when the tree did not change"]
    ID --> S["every stamped stage on that flavor<br/>now refuses to fork"]
    ID --> W["the warm-pool key changes<br/>a fresh 512 MiB snapshot is minted"]
    S --> Q1["which stages?<br/>no command answers"]
    W --> Q2["the old snapshot<br/>is never retired"]
    Q1 --> D["found one refusal at a time"]
    Q2 --> D2["3072 of 3584 MiB orphaned on this host"]

  1. [open] MEDIUM — no surface derives base staleness, and stage info on a stacked tip is a false all-clear. stage list and stage info serialize StageMeta verbatim (cli/src/main.rs:527, mcp/src/main.rs:958); neither reads an image sidecar, so no listing output can change when an image is rebuilt. Replaying check_base_chain_in over this host's store found 3 RebuiltBase among 41 rows — all 41 printed identically. Worse for a stacked stage: its own base_sha256 is the current image's while an ancestor's is not, so the record reads clean and the chain does not. The pub check_base_chain wrapper (stage.rs:357) that would serve a listing has zero callers. → FIX: a derived StageEntry beside StageMeta (mirroring image::ImageEntry) carrying base_state / base_stale / base_reason, computed once from an in-memory index of the listing rather than re-reading each ancestor. Derived, never persisted — a stamped verdict would be a stale on-disk lie at the next rebuild.

  2. [open] MEDIUM — the rebuild that invalidates stages never looks at the stage store. build_rootfs (image/rootfs.rs:195) publishes and returns; nothing warns before or after. image build-all is documented as required after a PROTO_VERSION bump, so the mandatory operation silently makes every stamped stage on the host unforkable. → FIX: after a successful publish, count the stages whose chain references the outgoing id and report them in the result JSON and on stderr. It must never fail the rebuild.

  3. [fixed] MEDIUM — the squashfs pack is not timestamp-pinned, so a rebuild over an unchanged tree mints a new base identity. Measured: two build-rootfs --force runs 4 s apart over an identical tree gave 86f20abd… and 6398c829…. (Three runs inside the same second gave one id — the stamp has one-second granularity, which is what made this look reproducible at first.) This is the root cause of #28 and #29: most invalidation is spurious. run_mksquashfs (image/rootfs.rs:1280) passes -all-root -noappend -quiet -no-progress; squashfs-tools 4.6.1 already defaults to -reproducible and accepts -mkfs-time / -all-time. Measured with those pinned: byte-identical output across a 3 s gap with the tree touched in between, and a genuine content change still moved the id. → FIX: pin both. Verify first that epoch-0 mtimes on base files upset nothing that reads them (everything a run writes lands in the overlay upper, not the base), and note that base-alpine pulls packages, so its inputs can vary for real — pinning removes the spurious churn, not the genuine kind. Fixed, to 1980-01-01 rather than the epoch so a DOS/ZIP date field can hold it. Two things the finding did not anticipate turned up in the doing. The timestamps a guest might have cared about are not read: Alpine ships hash-based bytecode caches (flags=3, PEP 552), so a pinned mtime cannot invalidate them. And squashfs-tools treats an ambient SOURCE_DATE_EPOCH as competing with the flags rather than as a default — with both present it exits SOURCE_DATE_EPOCH and command line options can't be used at the same time to set timestamp(s) and builds nothing, so the naive pin would have broken image build-rootfs outright on any host that exports it. The variable is removed from the packer's environment.

  4. [open] MEDIUM — warm-pool snapshots accumulate and nothing retires an orphan. The key embeds base_sha256, so a rebuilt base can never resume a stale snapshot — that part is right, and ensure self-heals by minting a new one on the first eligible run (measured: 3.87 s to build, 220 ms to resume after). But the one it replaces stays. This host: 7 snapshots, 3.6 GiB, 3072 MiB of it orphaned. The memfile is non-sparse by construction — Firecracker's Full-snapshot dump set_lens to the full guest RAM and writes every byte, zero pages included. warmpool list cannot tell you which are orphaned, warmpool rm takes a keyhash or --all, vm gc resolves only paths::vms_dir(), and the MCP surface exposes no warm-pool tool at all. → FIX: sweep orphans where the new id is already known — publish_image runs on every rebuild — or add warmpool rm --orphaned; and surface an orphaned flag in warmpool list.

  5. [fixed] HIGH — two concurrent runs of the same warm shape build into one directory on fixed .partial names, with no lock. snapshot::ensure creates artifacts.dir and writes vmstate.partial / memfile.partial (snapshot.rs:551); there is no flock anywhere in the module (the one at :429 claims a network slot). Two racers both see is_complete() false, both dump into the same two paths, both rename. The window is the several-second memory dump, and it opens on exactly the shape the product invites — default sandbox_runs right after any rebuild empties the pool for every shape, while the tool description tells the model to issue concurrent sandboxes from separate agents. → FIX: an exclusive flock per keyhash around ensure, in the shape net::claim_lock already uses; second arrival waits and then finds the snapshot complete. Failing that, per-process unique staging names so a loser can only lose its own bytes. FIXED: both halves. An exclusive per-keyhash flock (build.lock, inside the keyhash directory so warmpool rm — which removes directories and skips plain files — reclaims it) is taken in ensure before any build. A second arrival waits up to 90 s, notices the moment the winner publishes, and reuses that snapshot instead of building a second one over the top; if the wait expires the run cold-boots, which is what a cache miss does anyway. Staging names carry the pid now, so even with the lock gone a loser can only destroy its own bytes. Tested at the primitive and at the call site: ensure grew an ensure_at seam so the wait-then-reuse path runs without booting a VM or touching the process-global $ISOPOD_HOME, and all three tests fail if the flock is removed. Three mutations.

  6. [open] MEDIUM — a warm resume that fails never retires the snapshot. The run falls back to a cold boot (vm/mod.rs:2392) and reports path: "cold" with nothing to say why. One bad snapshot — from #32, a full disk, a killed builder — poisons that shape permanently: every later run of it pays a failed resume before its cold boot. → FIX: on resume failure, unlink the snapshot's meta.json, which is the file is_complete() reads; the next run rebuilds it. Surface the reason in the report.

  7. [open] MEDIUM — the stage store's label uniqueness is a TOCTOU, and four places say the state is file-locked. commit_in reads the whole store to check the label (stage.rs:418) and writes later; stage.rs contains no lock of any kind. Meanwhile README.md:120 and its architecture diagram, docs/getting-started.md:542, and mcp/src/main.rs:678 all describe ~/.isopod as file-locked for concurrent sessions — true of network slots and the VM registry, not of this. Two agents committing the same label produce two stages carrying it, after which every reference to that label is ambiguous. → FIX: a blocking exclusive lock on a .commit.lock in the store root around the read-decide-write window; and narrow the three prose claims to the state that is actually locked.

  8. [open] MEDIUM — commit_as is validated only after the run, and the scratch is deleted either way. Both refusals commit_in can raise — a label already in use, and a chain at MAX_CHAIN_DEPTH — are computable before boot, but the check runs at vm/mod.rs:1615, after the exec. The failure is deliberately reported rather than thrown (so the exit code, output and logs survive) but cleanup_disk still removes the scratch unless --keep, and sandbox_run has no keep parameter, so on the MCP surface a long build refused at commit time is unrecoverable. → FIX: a pure precheck_commit_in called from the preflight block before resolve_stage_plan, raising the same three bails while nothing has booted.

  9. [open] MEDIUM — the depth-cap refusal prescribes stage flatten, which does not exist. stage.rs:507 ends with "flatten the chain first"; there is no such subcommand, MCP tool, or documented procedure anywhere in the repo. An ordinary afternoon of use reached depth 7 of 10 in one lineage, and this store already holds gaunt-chain-10 at exactly the cap — a lineage that can never accept another commit. → FIX: until a flatten operation exists, name a remedy that does (rebuild the lineage from --stage base), in the shape of the sibling refusals in that file.

  10. [open] LOW — the skew override leaves no trace in the result payload. Both warnings are eprintln! (vm/mod.rs:1424, stage.rs:496); RunReport and the MCP result carry no warning field, so a run that booted across a rebuilt base returns an ordinary success to a program. The operator who set the variable is the one reading stderr, which is why this is LOW rather than MEDIUM. → FIX: a warnings: Vec<String> on the report, populated from the same strings.

  11. [open] LOW — the WrongFlavor remediation is CLI-shaped and cannot apply where that verdict fires. stage.rs:199 advises re-running from --stage base --base <flavor>, but a flavor mismatch means the layers belong to a different root; rebuilding on this flavor is not the fix, and the syntax is wrong for the MCP surface that receives the string verbatim. (The Unverifiable arm's isopod image build-rootfs --force advice is correctly CLI-shaped — that one is an operator action.) → FIX: reword to describe the remedy rather than spell a command.

  12. [fixed 2148e6c] MEDIUM — the documented way to "rebase" a stage does not rebase. Three files said forking with ISOPOD_ALLOW_BASE_SKEW=1 and committing is how a stage is moved onto a rebuilt image. The check ranks the worst verdict across every link, and the commit stacks a child without touching its ancestors, so the result refuses the next fork exactly as the original did — verified live end to end. The getting-started diagram drew the override landing on the same node as a clean fork; it now loops back to the stage it started from.

  13. [open] MEDIUM — "a failed setup command never silently produces a broken stage" is false for a pipeline. Happened this session: cargo build … | tail -25 failed, the shell returned tail's status, and commit_as committed a stage from a broken build. The MCP surface forces /bin/sh -c with no argv escape hatch, and the adjacent documentation sells pipes. Nothing in crates/core or crates/mcp can detect this — the shell reports what POSIX says it reports. → FIX (docs): say that a pipeline's status is its last command's, and put set -o pipefail in the recipes that pipe.

  14. [open] LOW — the refusal prints twelve hex digits and no documented command prints them back. ImageEntry (image/rootfs.rs:519) carries proto_version/unstamped/stale/built_unix but drops meta.sha256, which list_images has already read. (build-rootfs without --force short-circuits and reports the sha, which is an undocumented route to it.) → FIX: add sha256 to ImageEntry.

  15. [open] LOW — docs that undersell the blast radius. The troubleshooting table prescribes isopod image build-all for three symptoms without saying it invalidates every stamped stage; sandbox-build.md's stage-chain table omits the base rebuild from its "rebuild trigger" column; and it calls label reuse "untested" when the store refuses it outright with two distinct errors. Its everyday check/test loop also never says the crates.io cache only survives if you commit_as — a fresh fork needed the network twice this session for dependencies a previous uncommitted run had already fetched.

  16. [fixed] LOW — the changelog's "three cases deliberately do not refuse" lists one that always refuses. The third bullet is the flavor mismatch, which is refused in every case including under the override. The 0.12.0 entry also omits the base-image layout change (/rom dropped, ten baked mountpoints replaced by one) that shipped in the same release.

  17. [open] LOW — ~/.isopod/m0/ holds 891 MiB that no command lists or cleans. The M0 spike scratch — logs, spike images, a Firecracker binary. It is not dead: resolve_fc_bin still falls back to ~/.isopod/m0/bin/firecracker as a last resort, so an operator chasing disk pressure who guesses and deletes it removes that fallback. → FIX: a row in the layout table naming it and the condition under which it is safe to delete.

  18. [open] LOW — an interrupted commit strands a full-size layer.ext4.partial. commit_in stages the layer beside its destination and renames (stage.rs:526); a kill in between leaves up to a full scratch (1 GiB by default) in the largest store on the host, which list_in skips silently and stage rm cannot name. → FIX: report the residue from list_in's skip branch instead of continuing past it.

Checked and found sound (recorded because each was a live suspicion this session, and the answer is load-bearing): the warm pool self-heals after a rebuild rather than going permanently cold — the first probe's cold path was caused by --no-network, which disqualifies a run from the warm shape, not by the rebuild. A stale snapshot can never be resumed onto a new base, because base_sha256 is a keyed field. publish_image deliberately clears the old sidecar before the rename, so a failed stamp never leaves an old sidecar vouching for new bytes. stage rm of a mid-chain stage leaves nothing behind because it refuses, naming every dependent — removing this session's three probe stages took a leaf-first walk, which is the documented behaviour and not a gap. The unstamped-stage exemption and the whole-chain check both behave exactly as getting-started.md describes. sandbox_run's env is guest environment and cannot reach the host override. dns_queries: [] is correct in filtered mode: the guest resolves nothing, because it is handed a proxy and the broker resolves host-side. And the MCP host-I/O root confining stdin_file/copy_out to the server's working directory is deliberate and documented — payloads went under target/, which is gitignored.

Store state at the end: 38 stages (3 stamped, all the isopod-build/0.12.0* chain on base-alpine), 6.9 GiB allocated; base.sqfs rebuilt several times in the course of the measurements and now 6398c829…; base-alpine.sqfs untouched, so it still ships the pre-0.12.0 /rom + /layers/0..9 layout — harmless, since the guest mounts a tmpfs over /layers, but it should be rebuilt for consistency once #30 lands, so the rebuild costs nothing.

2026-07-28 (later) — a test-fix wave verified in-sandbox, and the ceiling that found

The work was a one-commit fix to screen_resolved's test (a refusal that fired correctly could name the wrong address and the suite stayed green). Everything that verifies it ran inside isopod, which is the point of this entry: the negative control is deliberately broken source, and running that on the host means editing the working tree and trusting a sed to put it back.

The verification loop works and is fast. Forked isopod-build/0.12.0-tested, overlaid HEAD's source as a tar over stdin_file: cargo test -p isopod-oci-registry 31 passed in 28.8 s wall including the incremental rebuild. Then the control in a second throwaway VM — same fork, sed the mutation in, 30 passed / 1 failed, naming the intended test. Host tree never touched. Then the whole mutation harness in a third: git init over the overlaid source is enough for git archive HEAD, and scripts/mutation-check.py --only oci-registry-refusal-names-the-wrong-address reported 1/1 mutations caught in 155 s from cold. A destructive-by-design harness runs unsupervised in a sandbox, which is the case it was written for.

What it cost to get there is the finding. The bytes-in channel is the narrow one:

flowchart TB
    HF["host file"] -->|"PutFile: base64 inside one JSON frame"| CAP["MAX_FRAME_LEN 8 MiB<br/>so about 6 MiB of raw input"]
    CAP --> MCP["MCP: refuses at 4 MiB, before boot,<br/>naming the limit"]
    CAP --> CLI["CLI: no check at all —<br/>boots a VM, then dies at the frame layer"]
    GF["guest file"] -->|"CopyOut: FileChunk x N, then FileDone"| NOCAP["streamed since proto v3,<br/>no ceiling"]
    NOCAP --> HP["host path, 16 GiB per file"]

  1. [open] MEDIUM — the host→guest file channel is single-frame and capped while guest→host is streamed and unbounded. Proto v3 gave CopyOut a streamed channel (FileChunk × N + FileDone) explicitly so artifacts out have "no MAX_FRAME_LEN size ceiling" (proto/src/lib.rs:45). The reverse direction never got the same treatment: PutFile is documented "single-frame; fits within MAX_FRAME_LEN" (proto/src/msg.rs:58), and since binary rides base64 inside the JSON, an 8 MiB frame cap is a ~6 MiB raw input ceiling. Measured: a 50 995 061-byte tarball produced a 67 993 510-byte frame, exactly the 4/3 inflation plus JSON. That is what stopped this session putting the repo with its .git (49 MiB) into a guest, so mutation-check.py — which is built around git archive HEAD — had to be handed a synthesized single-commit repo instead of the real history. Nothing above ~6 MiB gets in without the network: no dataset, no wheelhouse, no repo with history. → FIX: mirror v3's CopyOut with a streamed PutFile (chunks + done). Until then the MCP's own error advises "Copy it into a stage instead", which names no command isopod has — the only honest workaround is its second half, splitting the payload across runs and commit_as-ing between them.

  2. [open] LOW — the CLI boots a VM before discovering the payload will not fit, and reports the failure in post-base64 bytes. crates/cli/src/main.rs:506 reads --stdin-file with an unbounded std::fs::read; the MCP path checks against MAX_STDIN_FILE_BYTES first (mcp/src/main.rs:84). So the same oversized file is a clean pre-boot refusal naming the limit on one surface, and on the other: a booted VM, a wasted boot, and frame length 67993510 exceeds cap 8388608 — printed twice, from an error chain that formats its own source. Neither number is one the operator chose; they passed a 49 MiB file and are told about 67 993 510 bytes. → FIX: give the CLI the same pre-flight check and message, expressed in raw input bytes.

  3. [open] LOW — docs/sandbox-build.md base64s a payload the channel already carries raw, spending a third of a ceiling it never mentions. The "Getting source in" recipe pipes the tar through base64 -w0 and decodes it guest-side. Unnecessary on both surfaces — binary is base64'd inside the frame either way, so doing it again first is pure inflation, and it moves the effective ceiling from ~6 MiB down to ~4.5 MiB. Verified raw both ways this session: sandbox_run with stdin_file and a plain .tgz (31/31 tests green in-guest), and isopod run --stdin-file with the same file, 644 ms total. → FIX: drop base64 from the recipe and state the ceiling next to it.

This entry's own diagram was proved the same way. Rendering every diagram in the built site needs jsdom, mermaid, python-markdown and pygments, none of them installed on this host and none of them wanted there; a sandbox took the install and the render and was thrown away — 29 diagrams checked, 0 failures, 0 tag warnings, site built with no missing sources and no unresolved links. It is the smallest possible version of the argument for the whole tool: a check that would otherwise not have been run, because running it meant polluting a laptop.

Checked and found sound: stdin_file is binary-safe on both the MCP and CLI surfaces — the doc's encoding step was defensive, not required. The host-I/O root confinement refused a scratchpad path outside the server's working directory — deliberate, and already recorded as sound by the previous session; payloads went under target/ again. Forking the same stage three times concurrently-in-sequence left the parent untouched, and none of the three throwaway VMs was committed, so the store gained nothing from a session that ran a full mutation harness inside it.

2026-07-29 — 0.12.2 verified in-sandbox, and a guest that cannot talk to itself

The wave-closing bump had to be run on the exact tree being tagged. The host's taps were gone — WSL2 had restarted, which destroys them — so the only way to boot at all was network: false. That is where this entry starts, because the suite came back 18 failed, and all 18 were broker tests.

A uniform failure is a harness result until something proves otherwise, and the control was already in the same output: 363 tests passed beside the 18, and those same 18 are green on CI. The cause is not the code:

flowchart LR
    NF["sandbox_run with network false"] --> NIC["no NIC attached — intended"]
    NF --> LO["lo left DOWN — not intended"]
    LO --> B["bind on 127.0.0.1 SUCCEEDS<br/>a bind does not need the link up"]
    LO --> C["connect on 127.0.0.1 reaches nothing"]
    B --> T["a test that binds, then dials itself,<br/>fails far from the actual fault"]
    C --> T

  1. [fixed in 0.12.3] MEDIUM — network: false leaves the loopback interface DOWN, so a guest cannot talk to itself. Booted with network: false, the guest has exactly one interface and it is down: 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN. The failure mode is bad because it is partial — bind() on 127.0.0.1 succeeds, since binding never required the link to be up, so a workload gets a socket and a port number and only fails later when something tries to reach it. Measured against the workspace suite: network: false gives isopod-core 363 passed / 18 failed, every failure a broker test that listens and then dials itself; a single ip link set lo up first gives 381 passed / 0 failed, and the whole workspace goes to 604 passed / 0 failed / 14 ignored. The flag's documentation says it runs code "with no network at all", which an operator reads as no egress — loopback is not egress, it is the guest's own plumbing, and a test suite, a local server, or a database that binds a port all need it. → FIX: bring lo up unconditionally at guest-agent start. It costs nothing on a networked run and removes a class of failure that presents as the workload's bug rather than the sandbox's.

  2. [open] LOW — the bare isopod-build stage is older than the dependency set, so the label the docs tell you to fork no longer builds offline. docs/sandbox-build.md says every build "forks isopod-build". That stage was committed before flate2 entered the tree, so its crates.io cache does not contain it, and an offline build there dies with no matching package named flate2 found — a resolution error that reads like a broken lockfile rather than a stale cache. The versioned siblings are fine: isopod-build/0.12.0-tested carries the current dependency set and ran the whole suite offline. → FIX: the doc now names the newest isopod-build/* stage rather than the bare label, and stage_list is the way to find it. The deeper issue is that a mutable-looking label pinned to an immutable stage silently rots; label-reuse semantics are still untested (noted in docs/sandbox-build.md).

  3. [fixed in 0.14.0] HIGH — a coexisting Docker install silently breaks all NAT egress, because its FORWARD policy drops every guest packet. Found by running isopod in CI, and it is not a CI problem: any host with Docker on it is affected. Docker sets the iptables ip filter FORWARD chain policy to DROP and jumps to a DOCKER-USER chain that, by default, contains only RETURN — so guest→WAN traffic falls straight through to the drop. isopod setup reports complete success: 12 taps, an inet isopod table, ip_forward=1, and the guest gets an address. Then nothing works, and the error the workload sees is whatever its own first network call happens to be. Measured on a hosted runner: a guest handed a literal IP — no DNS anywhere in the path — could not open TLS to it, while the host reached both 1.1.1.1 and 8.8.8.8 successfully. Host traffic goes through OUTPUT; guest traffic goes through FORWARD, and only one of those is dropped.

    The failure mode is bad in a specific way: it is invisible to isopod. Nothing in setup looks at whether another tool has already claimed the forward hook, so every diagnostic isopod prints says the network is fine. The first symptom is a DNS timeout inside a guest, which reads as a resolver problem — and did: this was diagnosed as a resolver bug first, confidently and wrongly, before the literal-IP control ruled DNS out.

    Reproduced and settled in throwaway network namespaces (scripts/diagnose-forward-hook.sh), three namespaces wired client → router → server so traffic is genuinely forwarded both ways:

    case result
    plain forwarding (harness control) REACHED
    Docker shape: FORWARD policy DROP BLOCKED
    + isopod ACCEPT in DOCKER-USER REACHED
    isopod's own nft DROP alone BLOCKED
    + isopod's nft DROP on top of the ACCEPT BLOCKED

    The last row is the one that decides whether the fix is allowed to exist. An ACCEPT verdict ends traversal of that base chain only; every other base chain registered at the hook still runs. So inserting an accept rule into Docker's chain removes Docker's drop without bypassing isopod's own egress enforcement, which stays authoritative. → FIX: isopod setup inserts an accept rule for isopod-tap* into DOCKER-USER when that chain exists — two rules in fact, because a single inbound accept leaves the reply to die on the same policy DROP, so not even a handshake completes. setup now reports a docker_user field so the answer is visible instead of inferred from whether the network happens to work, and --no-docker-user opts out for anyone curating that chain themselves.

    Known residue, documented rather than fixed: Docker offers no persistence contract for DOCKER-USER, and a daemon restart or network creation may flush or reorder it. The failure is fail-closed — guest egress stops, nothing opens — and the remedy is re-running sudo isopod setup, which is the same doctrine already published for a flushed nftables ruleset.

  4. [fixed in 0.18.0] HIGH — every submount of a read-only jail bind was writable. The jail binds ~/.isopod read-only, and that tree holds the stage store and the guest images. bind_mount uses MS_REC, so the bind carries submounts with it — but MS_REMOUNT | MS_RDONLY applies to exactly one mount, so every submount stayed writable. Measured in a disposable guest rather than reasoned about: bind a tree containing a tmpfs, remount the top read-only, and a write to the top is refused while a write to the submount succeeds and the file's contents change.
    write to the top of a read-only bind refused
    write to a submount of it succeeded

    A submount under ~/.isopod is not exotic — a separate disk for images, a tmpfs, an encrypted volume. The jail exists to contain a compromised Firecracker, and that process could have written to a stage layer every later run forks. → FIX: remount_readonly_recursive uses mount_setattr(2) with AT_RECURSIVE — the kernel walks the tree itself, in one atomic call — and falls back on a pre-5.12 kernel to reading /proc/self/mountinfo and remounting every mount at or beneath the target, deepest first so a parent cannot shadow a child. Fails closed either way. See #53 for what the first version of that fallback got wrong.

    Found because the jail's syscall layer had no unit tests at all — 21 unsafe blocks covered only by one #[ignore]d integration test that, until the day before, ran nowhere but a maintainer's laptop. The parser is now pure and unit-tested, including the two ways it can be subtly wrong: matching on raw prefix (which would remount unrelated host mounts read-only) and comparing mountinfo's octal-escaped paths without decoding them (which would miss a submount under any directory with a space in its name).

  5. [fixed in 0.18.0] HIGH — the fix for #52 could not start a jail on a host whose mount table held a nosuid/nodev/noexec mount. Caught by the live suite on the branch — not by review, not by the local gate, and not by the eight unit tests written for #52, every one of which passed.

    isopod-jail: child setup failed: remount <root>/ read-only: remounting <root>/proc/sys/fs/binfmt_misc read-only: Operation not permitted

    A mount inherited into a new user namespace has its nosuid, nodev and noexec bits locked (mount_namespaces(7)). A remount that does not name them again reads as an attempt to clear them, and the kernel answers EPERM. The top-level bind remounts fine because the jail created it and it carries no locked bits — which is exactly why this passed on a maintainer's laptop and failed on a hosted runner. It is a property of the host's mount table, not of the code.

    Measured in a nested user namespace rather than reasoned about:

    remount of a nosuid,nodev,noexec submount result
    MS_RDONLY alone EPERM
    MS_RDONLY + the three flags passed back ok, and the mount is ro

    → FIX: mount_setattr(2) with AT_RECURSIVE only ever adds MOUNT_ATTR_RDONLY and clears nothing, so it cannot engage the rule at all. Atime needs no help either way — mount(2) preserves it for a remount that names no atime flag, so repeating it could only add a way to get it wrong.

    The pre-5.12 fallback then failed the same way a second time, on the same mount, for a different reason — and the second reason is the interesting one. Reading each mount's flags out of its mountinfo line looks obviously correct and is not: two mounts can share one mount point, and a remount by path reaches only the topmost. /proc/sys/fs/binfmt_misc is exactly that shape on a hosted runner — a systemd autofs with the real filesystem mounted over it — so the walk applied the lower mount's flags to the upper mount:

    /tmp/dst/sub rw,relatime keep=0 <- lower /tmp/dst/sub rw,nosuid,nodev,noexec,relatime keep=14 <- topmost remount /tmp/dst/sub keep=0 -> EPERM remount /tmp/dst/sub keep=14 -> ok

    The fallback now asks statvfs(2) for the flags instead. It resolves a path exactly as mount(2) does, so it always describes the mount about to be remounted — the correlation problem cannot arise. mountinfo is left to do the one job it can do reliably: enumerate which paths exist.

    Reproduced locally by stacking two tmpfs mounts at one path inside a user namespace and running the real jail binary against it: the committed code fails at that path, the fix passes.

    Then the fallback was deleted. Three rounds, three defects, every one of them found on a hosted runner and none on a developer's machine — because whether the walk fails is a property of the host's mount table, not of the code. It dropped locked flags; it read flags off the wrong line where two mounts share a mount point; and it could never have reached a shadowed mount at all. mount_setattr(2) was correct on the first attempt and structurally cannot hit any of the three. The jail now requires Linux 5.12 and refuses to start below it, naming the requirement, this host's kernel, and the fact that dropping ISOPOD_JAIL=1 starts an unjailed VM. The jail is opt-in, so this floor applies to nothing else.

    The lesson worth keeping is not about mounts. A second implementation of a security boundary, on a path that only runs where nobody tests, is a liability priced as a feature — it accumulates defects at full rate and reveals them at the rate the untested hosts are exercised.

    The fallback is now forced by an environment variable in the live probe, so both paths run on every host that can run the probe at all, instead of the older one running only on machines nobody tests on. The probe also gained the assertion #52 never had: that a submount of the read-only bind is read-only (/dev/shm, which is both a tmpfs under / and nosuid,nodev, so it covers the recursion and the locked-flag rule at once). Both mechanisms were then broken on purpose to confirm the probe fails — dropping the flags reproduces the EPERM above, and dropping AT_RECURSIVE reproduces #52's writable hole.

Checked and found sound: the tap teardown was reported exactly right — the error named WSL2 as the likely cause, named sudo isopod setup as the fix, and offered --no-network as the alternative, which is what made the offline route obvious. Forking isopod-build/0.12.0-tested three times left the parent untouched and nothing was committed, so a session that ran the full suite twice and a probe VM added nothing to the store.

Rendered from docs/dogfood-findings.md on the main branch.