isopod

Changelog

All notable changes to isopod. The format follows Keep a Changelog loosely; versions follow Semantic Versioning with pre-1.0 semantics (minor = features or breaking changes, patch = fixes). See CONTRIBUTING.md § Versioning for the policy.

[0.20.0] — 2026-08-03

Two findings from a competitive audit against another Firecracker sandbox, both against isopod rather than the competitor. The audit's headline was that isopod escapes the identical-entropy-after-fork bug that sandbox ships — but only by a property it inherits and never asserted, and one that would disappear silently.

Added — the guest kernel must be able to reseed after a fork

isopod's warm pool resumes one memory image many times. What stops those sandboxes sharing a CSPRNG is not isopod's code — there is none — but Firecracker's VMGenID device plus a CONFIG_VMGENID guest kernel, which reseeds when the generation counter changes. Measured across three warm resumes of a five-day-old snapshot: boot_id, /dev/urandom, stack ASLR, and Python's random, os.urandom and ssl.RAND_bytes all differed.

The problem was that nothing said so. A kernel without the option would resume every warm sandbox with the CSPRNG state frozen into the snapshot, every resume would still succeed and no test would fail — and fetch-kernel --allow-unpinned selects whatever the CI bucket currently offers, so this was a live path.

fetch-kernel now refuses to install a kernel with no VMGenID reseed path, naming CONFIG_VMGENID and the consequence. A live test asserts it for every installed kernel and carries a control that must be rejected — a substring search over a 40 MB binary is exactly the shape that quietly matches everything. SECURITY.md gains a Warm resume section stating what one memory image resumed many times does and does not share.

Fixed — a deep $ISOPOD_HOME failed as an unexplained ten-second timeout

A Unix socket path cannot exceed 107 bytes, and isopod puts Firecracker's API socket and the guest-agent vsock inside $ISOPOD_HOME/vms/<vm-id>/. Past that depth bind fails inside Firecracker, the process exits 1, and isopod waited the full ten seconds for a socket that would never appear before reporting a timeout that named the path but not the reason:

Firecracker API socket …/vms/dev-96965fa0/api.sock did not become ready within
10000 ms (process exited: exit status: 1)

Nothing in that says "your home directory is too long", and the run costs ten seconds to find out. Both spawn paths now check before creating anything and refuse in 6 ms with the length, the limit, and how many bytes to remove:

the VM directory … is too deep for a Unix socket: its vsock path is 170 bytes and
the kernel's limit is 107 … Shorten $ISOPOD_HOME by at least 63 bytes (it is
currently 142 bytes) or leave it unset to use ~/.isopod

The limit was measured rather than read off a header — binding at successively deeper paths fails at exactly 108 — and the test asserts both sides of the boundary. Found by the competitive audit's dogfood pass.

Added — snapshots are integrity-checked before they are resumed

Stages were content-addressed with blake3; snapshots recorded sizes and nothing else, and the resume path checked only that three files existed. On the unjailed default path, anything able to write ~/.isopod/snapshots/*/memfile had code execution in every later warm run.

meta.json now records a blake3 of both artifacts. vmstate — which carries the vCPU register state the guest resumes at — is digested in full on every resume. The memory file is checked for identity rather than content, and the reason is measured, not assumed: blake3 runs at 1.59 GiB/s here, so digesting a 512 MiB memory file costs 315 ms and a 3 GiB one about 1.9 s, against a ~49 ms resume and the ~410 ms cold boot that resume exists to beat. Verifying every byte on the hot path would make the warm path slower than the path it replaces.

ISOPOD_VERIFY_SNAPSHOT=1 digests both files in full for anyone who wants to pay that. The honest boundary is asserted rather than described: one test proves a restored mtime defeats the fast check while the full check still catches it.

Verification failures are not run failures. An unverifiable or legacy snapshot is rebuilt and the run cold-boots, so the fail-closed direction costs one cold boot rather than an error.

[0.19.0] — 2026-08-03

Releases 0.15.0 through 0.18.0 were never tagged or published. They landed on main as steps of one piece of work and are kept below as the record of what changed when. This is the version that ships them, so upgrading from 0.14.0 means taking all of it — read those sections too, they describe behaviour in this release.

0.18.0 in particular shipped a fallback that let the jail run on kernels older than 5.12. It is removed here, and because 0.18.0 was never published, no released version ever offered it.

Changed — the rootless jail requires Linux 5.12 (ISOPOD_JAIL=1 only)

Breaking for jail users on older kernels; nothing else changes. The jail is opt-in, so isopod itself gains no kernel floor — an unjailed run is unaffected, and a host below 5.12 keeps everything except the optional second isolation layer.

mount_setattr(2) is the only mechanism that makes a bind read-only including every mount beneath it and actually holds: the kernel walks the tree in one call, so nothing can appear between reading a mount table and acting on it; it adds MOUNT_ATTR_RDONLY and clears nothing, so it cannot trip the rule that a nosuid/nodev/noexec bit locked into a user namespace may not be cleared; and it reaches a mount that another mount is stacked over, which nothing driven by path can do.

0.18.0 shipped a hand-rolled walk over /proc/self/mountinfo for older kernels. It is removed. It failed all three ways above, in three successive rounds, each one found only by the live suite on a hosted runner and never on a developer's machine — because whether it fails at all is a property of the host's mount table, not of the code. It dropped the locked flags and earned EPERM; it then read flags off the wrong mountinfo line where two mounts share a mount point (/proc/sys/fs/binfmt_misc is a systemd autofs with the real filesystem mounted over it) and earned the same EPERM again; and it could never have reached a shadowed mount at all, since a remount by path only ever finds the topmost.

A second implementation of a security boundary, on a path that only runs where nobody tests, accumulates defects at full rate and reveals them at the rate the untested hosts are exercised. It is worth less than the hosts it buys. Dogfood finding #53.

On an older kernel the jail now refuses to start, naming the requirement, this host's kernel release, and the fact that dropping ISOPOD_JAIL=1 starts an unjailed VM.

Changed — the jail's tests follow the code they protected

The mountinfo parser and its unit tests are gone with the fallback they served. What remains is tested where the guarantee now lives: the live probe's assertion that a submount of a read-only bind is read-only, and the unsupported-kernel path — both the message and, separately, the decision to use it. That second test exists because a mutation proved the first was not enough: the message was covered, the call site was not, so deleting the call left every test passing.

[0.18.0] — 2026-07-30

Fixed — every submount of a read-only jail bind was writable

The rootless jail binds ~/.isopod read-only, and that tree holds the stage store and the guest images. bind_mount uses MS_REC, so the bind carries its submounts with it — but MS_REMOUNT | MS_RDONLY applies to exactly one mount, so every submount stayed writable.

Measured rather than reasoned about, in a disposable guest: bind a tree containing a tmpfs, remount the top read-only, and a write to the top is refused while a write to the submount succeeds and the file's contents change.

A submount under ~/.isopod is not exotic — a separate disk for images, a tmpfs, an encrypted volume. The jail exists to contain a compromised Firecracker, and that process could have written to a stage layer every later run forks.

remount_readonly_recursive now makes the whole tree read-only, and fails closed: a jail that cannot prove the tree is read-only must not report that it is. It prefers mount_setattr(2) with AT_RECURSIVE, where the kernel walks the tree itself in one atomic call. On a kernel older than 5.12 it falls back to reading /proc/self/mountinfo and remounting each mount in turn, deepest first. Dogfood finding #52.

That fallback has to hand each mount its own flags back. A mount inherited into a new user namespace has nosuid, nodev and noexec locked (mount_namespaces(7)), and a remount that does not name them again reads as an attempt to clear them — EPERM. The first version of this fix omitted them and could not start a jail at all on a host whose mount table happened to contain such a mount under the bind; it was caught by the live suite on the branch, on a hosted runner, after passing on a maintainer's laptop and in the pull-request gate. mount_setattr sidesteps the rule entirely by only ever adding the read-only attribute. Dogfood finding #53.

The flags come from statvfs(2), not from the mount's line in /proc/self/mountinfo. Those are not the same thing: two mounts can share one mount point — a systemd autofs with the real filesystem mounted over it, which is exactly what /proc/sys/fs/binfmt_misc is on a hosted runner — and a remount by path reaches only the topmost. Reading flags off a mountinfo line applied the lower mount's flags to the upper mount and earned the same EPERM a second time. statvfs resolves a path the way mount does, so it always describes the mount about to be remounted; mountinfo is left to enumerate paths, which is the one job it can do reliably.

ISOPOD_JAIL_FORCE_REMOUNT_WALK=1 forces the fallback path. It exists so the live probe can exercise both implementations on any host, rather than leaving the older one to run only where nobody tests; it selects which implementation makes the tree read-only, never whether it is made read-only.

Fixed — the pinned Alpine bootstrap package stopped existing

Building the base-alpine guest image began failing with a bare 404 for apk-tools-static-3.0.6-r0.apk. Nothing here changed: Alpine's CDN serves only the current revision of each package, so a pin goes 404 the moment upstream rebuilds it — here, 3.0.7-r0 shipped and took 3.0.6-r0 with it.

That is the pin doing its job. A digest that could quietly follow a moving file would be no pin at all, so the fix is to re-pin, never to stop verifying. Bumped to 3.0.7-r0 and its sha256, with the new apk.static checked to still be a static x86_64 ELF at the expected path in the package.

Both bootstrap downloads now explain this when they fail, instead of reporting an HTTP status that reads like a network fault or a compromised mirror: the error names the listing to check and the two constants to re-pin.

Added — the jail's syscall layer has unit tests

It had none. Twenty-one unsafe blocks, covered only by a single #[ignore]d integration test which — until the day before this — ran nowhere but a maintainer's laptop and needs a privilege CI can withdraw. A security boundary in a sandbox for untrusted code has to be checked in the pull-request gate, on every change, without needing privilege.

The mountinfo parser is pure and tested, including the ways it can be subtly wrong. Matching on raw prefix would make ~/.isopod-other and ~/.isopodx read-only too — unrelated host mounts, outside the jail, silently losing write access. mountinfo octal-escapes its mount points, so comparing the raw text would miss a submount under any directory with a space in its name. And a mount point can appear twice, so it is listed once — the duplicate names the same reachable mount.

The live probe now asserts what it never did: that a submount of the read-only bind is read-only, checked against /dev/shm — a tmpfs under / that is also nosuid,nodev, so one assertion covers both the recursion and the locked-flag rule. It runs the jail twice, once per implementation.

[0.17.0] — 2026-07-30

Fixed — a guest that never took its resolver said nothing about it

The guest agent logged a failed /etc/resolv.conf write and carried on, then logged net: eth0 up … dns [1.1.1.1,8.8.8.8] on the next line. So the failure did not merely go unnoticed — it reported success. The guest went on resolving perfectly well through whatever its image or stage layer happened to carry.

That was survivable while the baked value was the same public resolvers the host would have sent. It stops being survivable now that the host sends a gateway address instead: the public-slot redirect is deliberately pinned to the gateway, so a guest that kept its baked resolvers takes its own masqueraded path and resolves everything through a third party — while the host, the operator and the run's own egress record all report that gateway DNS policy is in force.

The resolver write now propagates. It is the one secondary step in apply that is not best-effort, and the reason is that it is the only one whose failure is invisible: a broken route or a missing NIC breaks the workload's first call in front of whoever reads the log, and a wrong resolver breaks nothing at all.

Added — Pong reports a resolver the guest never took, and the host refuses the run

Boot-time application stays best-effort for addressing — a no-NIC boot must not stop exec over vsock — so the failure is recorded and surfaced in every Pong as an additive resolv_error, exactly as a failed overlay assembly already is. The host treats it as fatal: a run whose DNS policy is not in force is not the run that was asked for.

The record is made by reading the file back, not inferred from the error. apply returns one error type for several steps, and a no-NIC boot fails at the address ioctl long before the resolver is touched — reporting a resolver problem there would point at the wrong subsystem.

Changed — guest images bake a loopback tombstone instead of public resolvers

/etc/resolv.conf now ships as nameserver 127.53.53.53 with options timeout:1 attempts:1 and a header comment naming the serial line to grep for. Anything in 127/8 terminates on the guest's own loopback, which the agent raises unconditionally — so in the failure state not one DNS packet leaves the VM, even for a regression nobody has predicted.

Not 127.0.0.1: that is indistinguishable from what a libc does with a missing file, and it collides with a workload-installed dnsmasq or unbound, which would forward the queries to its own public upstream — rebuilding the leak inside the sentinel meant to expose it.

The options line is doing real work. Measured in a musl guest: a lookup against an unreachable resolver takes ~5 s without it and ~1 s with it, because musl's resolver uses an unconnected socket and never sees the ICMP refusal.

This needs a guest image rebuild (isopod image build-all --force) to take effect. Existing committed stages carry their own frozen copy of the file and are unaffected either way.

The test that covered this asserted only that the file "has a nameserver", which passed just as happily on the public resolvers it was meant to guard.

Added — scripts/diagnose-forward-hook.sh

The experiment that established finding #51 and the fix for it, now shipped rather than cited from a path outside the repository. If isopod is installed alongside Docker and guests cannot reach the network, run it: it reproduces the failure and verifies isopod's own enforcement still applies, entirely inside throwaway network namespaces.

[0.16.0] — 2026-07-30

Part two of the gateway DNS resolver; 0.15.0 provisioned the rules, this builds the resolver that answers on them. Still nothing points a guest at it — that is part three — so no guest behaves differently yet.

Added — a DNS resolver for NAT slots, and an honest failure mode

DnsForwarder answers on a slot's gateway through the host's own resolution path, so a guest resolves exactly what the host resolves — including split-horizon and internal names a public resolver can never see, and without sending every lookup to a third party regardless of the operator's DNS policy. Same lifecycle as the egress broker: two tokio tasks in the supervisor's process, aborted on drop, dead with the run. No new process, no new port.

It never falls back to a public resolver. A host that cannot resolve gives its guests SERVFAIL. The alternative — quietly retrying against 1.1.1.1 — would defeat the privacy property on exactly the networks where nobody expects it, and would do so invisibly.

SERVFAIL rather than "no records" is the point of a new Resolution type. resolve_v4 collapsed "the resolver failed" and "the name has no A record" into an empty vector, which is right for a caller about to dial something and wrong for one about to synthesise a reply: answering NOERROR-empty tells a guest the domain exists but has no addresses, which is terminal — resolvers stop retrying. A broken host resolver would have been indistinguishable from a missing domain, which is the exact confusion that sent this project chasing a resolver bug for half a day when the real fault was a packet filter.

The resolver answers one sandbox. It binds a host address, so without the peer gate any local process could use it as a general-purpose resolver — and, on a filtered slot, learn what that sandbox looked up.

Changed — DNS over TCP no longer competes with proxy traffic

The filtered broker served DNS-TCP through the same accept loop and the same MAX_CONCURRENT_CONNS permits as SOCKS and HTTP tunnels, so a run that saturated its tunnels could starve its own name resolution — a stall that presents as a DNS timeout and points nowhere near the cause. DNS now has its own accept loop.

The responder is one type with a mode switch rather than two servers: the transport, parser and encoder were already policy-free, and only two points in answer_dns ever cared. Filtered mode is unchanged in behaviour — allowlist, ledger event, remembered answers for attribution — and its 41 tests still pass untouched.

[0.15.0] — 2026-07-30

Added — provisioning for a gateway DNS resolver (inert until the runtime half lands)

isopod setup now installs, for every public (NAT) slot, a daddr-pinned redirect of gateway-addressed DNS to port 5353, plus the matching input accept. The manifest records the port as gateway_dns_port.

Nothing uses these rules yet, and nothing changes for any guest. They only match traffic aimed at a slot's own gateway on port 53, and no guest is pointed there — every guest still resolves through the public resolvers in its image. The rules are installed first, deliberately, because provisioning happens once as root and the runtime cannot open a hole for itself later. The forwarder that answers on 5353, and the plumbing that points guests at it, are the next change.

Why bother. The hardcoded 1.1.1.1 / 8.8.8.8 in every guest image work on a permissive network and fail on a restrictive one, and either way they send every guest lookup to a third party regardless of the operator's own DNS policy — and internal or split-horizon names can never resolve. A resolver on the slot gateway, answering through the host's own resolution path, fixes all three. This was briefly mistaken for the cause of a CI egress failure; it was not (that was Docker's FORWARD policy, fixed in 0.14.0), so this is a portability and privacy improvement rather than a bug fix.

The daddr pin is load-bearing twice. It keeps NAT semantics — a guest that deliberately queries 1.1.1.1:53 still takes its direct masqueraded path — and it makes the rollout order-independent: an older binary on a newer provisioning keeps using public resolvers, and the new rules are completely inert for it rather than hijacking its queries into a port with no listener. gateway_dns_port is read the same fail-closed way as broker_tcp_ports: unrecorded means "no resolver here", and the runtime falls back to the public list rather than guessing.

Public slots open only 5353, never the broker's four-port set — no broker listens on a NAT gateway, and the input chain is the only thing between a guest and host services.

Changed — the provisioning format is versioned, and its regression gate with it

no_filtered_slots_is_byte_identical_to_0_8_1 asserted that an install with no filtered slots emitted the pre-0.9 table verbatim. That is deliberately no longer true, so it is replaced by the_ruleset_matches_the_checked_in_provisioning_fixture against new 0.15 fixtures. The 0.8.1 fixtures stay in the tree as the record of what the old format was.

The fixtures are generated by build_nft_ruleset itself, through an #[ignore]d regenerate_the_provisioning_fixtures, never typed out — a fixture hand-written to match what the author believed the writer emits pins the belief, not the behaviour. It stays #[ignore]d so CI can never rewrite the thing it compares against.

[0.14.0] — 2026-07-30

Fixed — a coexisting Docker install silently swallowed all guest egress

Docker sets the iptables ip filter FORWARD policy to DROP and jumps to a DOCKER-USER chain containing only RETURN, so every guest→WAN packet fell through to that drop. Any host running Docker had broken NAT egress, and nothing said so: isopod setup reported complete success throughout — taps created, nft table installed, ip_forward=1, guest addressed — because nothing in setup looked at whether another tool had already claimed the forward hook. Dogfood finding #51.

The first symptom is a timeout inside a guest, usually a DNS lookup, which reads as a resolver problem. It read as one here, and was diagnosed as one — a hardcoded-public-resolver bug — confidently and wrongly, until a guest handed a literal IP with no DNS anywhere in the path failed too, while the host reached both 1.1.1.1 and 8.8.8.8. Host traffic goes through OUTPUT and guest traffic through FORWARD; only one of those was dropped.

setup now inserts two accept rules into DOCKER-USER when that chain exists. Two, not one: the reply arrives on the WAN interface and dies on the same policy DROP, so a single inbound accept never completes a TCP handshake.

Why this cannot weaken the sandbox. Per nft(8), an accept verdict ends evaluation of the current base chain and the packet advances to the next base chain, whereas a drop ends the whole ruleset. inet isopod's forward chain is a separate base chain at the same hook, so accepting in Docker's table removes Docker's drop and none of isopod's — tap↔tap isolation, anti-spoof, the IPv6 deny, the RFC1918 guard, the filtered-slot drop and the closing iifname "isopod-tap*" drop default-deny all still apply. Measured in throwaway network namespaces rather than assumed: with the accepts live, a drop in a separate inet chain still blocked the connection and its counter showed the packets arriving. The rules are accept-only and scoped to isopod's own taps and its own 10.107.0.0/16.

Docker publishes no persistence contract for that chain, so a daemon restart or a network creation may flush it. That is fail-closed — egress stops, nothing opens — and the remedy is re-running sudo isopod setup, the same doctrine already published for a flushed nftables ruleset.

Added — setup reports what it did about the forward hook

isopod setup's JSON gains docker_user: installed, already-present, chain-absent, iptables-missing, lock-busy, skipped or removed. The failure this addresses is invisible to every other field in that report, so the answer is stated rather than left to be inferred from whether the network happens to work. A lock timeout reports lock-busy and not chain-absent, because conflating them would read as "nothing to do" on precisely the busy Docker hosts where there is most to do.

--no-docker-user declines the mechanism for anyone curating that chain themselves. A kernel without xt_comment falls back to unmarked rules rather than failing a setup that succeeds today, and teardown matches both spellings so neither can orphan.

[0.13.3] — 2026-07-30

Added — CI reports what it learned, on the pull request, while it can still matter

Every pull request now gets a single digest comment, rewritten in place rather than appended to: the version-guard verdict in words, the third-party dependency delta, and line coverage. Coverage previously ran only on pushes to main — a number nobody saw until after the merge cannot inform the merge.

The digest is written to the run summary first and mirrored into the comment second, so it still renders for pull requests from forks, whose tokens are read-only by design. It is deliberately absent from ci-ok's needs: a report must not be able to block a merge whose code is fine.

Workspace crates are excluded from the dependency delta. They carry the workspace version, so without that filter every release bump reported nine "dependency changes" and buried the one line that mattered.

Added — a coverage badge, labelled for what it actually measures

coverage (unit), not coverage. A badge cannot carry a caveat, so the qualification lives in the label where it cannot be separated from the number. docs/coverage.md is new and says the rest: which fifteen #[ignore]d tests are excluded, why the number is lowest exactly where the code is most security-relevant, and why nothing gates on it.

The badge endpoint is rendered by the same parser that produces the digest, so the two cannot disagree, and it is written only when the summary actually parsed — a failed coverage run leaves the previous badge standing rather than replacing a real number with a wrong one.

Fixed — a false claim about hosted runners, in four places

coverage.yml, ci.yml, and the CI research notes all stated that the #[ignore]d suite cannot run on GitHub-hosted runners. That was false when written: it repeated a pre-April-2024 assumption without checking one. A free ubuntu-latest runner has /dev/kvm, and the full-boot probe booted a real guest on one in 84 ms and completed a privileged isopod setup. The suite is absent from the pull-request path because reaching it costs a vendored Firecracker build and four guest images — a scheduling constraint, not a capability one.

[0.13.2] — 2026-07-29

Fixed — a tag on the wrong commit published packages declaring the wrong version

v0.13.1 was created with a bare git tag -a v0.13.1, which tags whatever HEAD happens to be. Work at the time spanned several git worktrees and the primary checkout was on another branch, so the tag landed on v0.13.0's commit. The release built from that tree and published isopod_0.13.0-1_amd64.deb and isopod-0.13.0-1.x86_64.rpm inside a tarball named isopod-0.13.1. A binary that reports a version it is not is a support problem, so the release and the tag were deleted and re-cut at the commit carrying the bump.

The Version guard was right and was disbelieved. It failed four consecutive pushes with "code changed since v0.13.1 but the workspace version was not bumped", which was true — it diffs against the tag's commit, and the tag was a commit early. The message sent the reader looking for a missing bump that had already been made. The guard now checks tag placement directly: if the workspace version is already tagged, the tagged commit's own Cargo.toml must declare that version, and the failure names the offending sha and what it actually declares.

Added — scripts/bump-version.py and a release skill, so this is mechanical

Three things the guard enforces were being done by hand and by memory: the workspace version and .claude-plugin/plugin.json moving together, the version going up by one step of the right level, and the tag landing on the commit that carries the bump.

scripts/bump-version.py patch|minor|major performs the first two and refreshes Cargo.lock. --tag performs the third: it refuses unless HEAD's own Cargo.toml declares the version being tagged, passes an explicit sha rather than relying on HEAD, and re-reads the tag afterwards to confirm where it landed. --check verifies the whole state, including tag placement, and is what to run first when the guard goes red.

A pre-release resolves to its own release rather than stepping past it — 0.13.0-preview.1 --patch gives 0.13.0, not 0.13.1 — because the preview line depends on that and anything else skips the version it was previewing.

The release skill in .claude/skills/ documents the sequence, which level to pick, and the two commands that diagnose a red Version guard. What neither buys: the script does not commit, does not push, and does not write the changelog entry — those stay deliberate.

[0.13.1] — 2026-07-29

Fixed — 0.13.0 shipped unformatted

One Attr::Bucket call in the new copy-out instrumentation exceeded the line width, so cargo fmt --check failed on the release commit. The tag built and released green — Release and CI are separate workflows, and formatting has no bearing on a binary — but main was red from the moment 0.13.0 landed.

The gap was procedural: the branch was rebased onto a main that had moved, and the merge was verified with clippy and the test suite but not with cargo fmt --check. Rebasing reflows nothing by itself; the offending line was inside a hunk the rebase carried across, and only the formatter's line-width rule noticed.

[0.13.0] — 2026-07-29

Added — spans over the phases nobody could see, and six report fields to match

Internal tracing instrumentation over the run path, and six additive RunReport fields: boot_ms, teardown_ms, copy_out_ms, snapshot_build_ms, and the split of the existing commit_ms into commit_hash_ms + commit_copy_ms. Each field is absent when its phase did not happen, so every existing consumer sees the exact JSON it always did. resume_ms and commit_ms themselves are unchanged.

What the numbers showed once they existed:

What this does NOT buy: it is instrumentation, not telemetry. There is no exporter, no network path, and no opentelemetry dependency in any build. With RUST_LOG unset the binaries install no subscriber and write zero bytes; RUST_LOG=isopod=debug prints the spans to stderr and nowhere else. Span attributes pass through a sealed Attr type whose only string carriers are &'static str and the host-minted vm_id, and guest-influenced magnitudes appear only as log2 buckets. exec_ms still folds vsock output streaming into compute time. The two spans written blind while the host's taps were down — isopod.run.snapshot_ensure and the warm isopod.run.resume — have since run live: a first-warm run reported snapshot_build_ms: 4238 / resume_ms: 72 with both spans on stderr, and a second run of the same shape reported resume_ms: 61 with snapshot_build_ms correctly absent.

[0.12.5] — 2026-07-29

Fixed — the commit hash pass read a gigabyte 8 KiB at a time

stage_id_for streamed layer.ext4 through std::io::copy, whose stack buffer is 8 KiB: 131 072 read() calls per apparent GiB, each also zero-filling its slice wherever the sparse scratch has a hole. It now reads through a 4 MiB heap buffer — 256 reads per GiB — retrying EINTR as io::copy did.

The measurement that motivated this did not reproduce, and the claim is corrected rather than repeated. One instrumented run showed the hash pass at 10.81 s against 0.38 s for the sparse copy of the same file, and that state did not appear again. Real end-to-end commits of the same shape ran 987/881/1018 ms before and 923/888/992 ms after: indistinguishable, and dominated by writeback of the scratch that had just been written.

What survives measurement, on a real committed layer of 1 GiB apparent and 64 MiB allocated, warm cache, isolated pass: 0.90 s before, 0.47 s after. So the claim is 1.9× on the pass, plus a 512× cut in read() calls — which is what bounds the pathological case. Per-call overhead has only to reach ~80 µs for the old loop to cost 10 s/GiB, where the new loop stays under a second. That is worth having on a contended machine even though today's commits do not show it.

The digest does not change, and that is the whole constraint. Stage ids are content-addressed, and the id is BLAKE3 over the file's full apparent bytes, holes included as the zeros they read back as. Skipping the holes would have been faster again — and would have silently re-identified every stage in every existing store, breaking forks with no error anywhere. A new test keeps the old std::io::copy implementation in place as the definition and asserts the buffered pass is byte-identical to it, over a fixture shaped for every loop boundary and over a real make_scratch_ext4 image.

A plain revert to io::copy is invisible to any test by design, since identical bytes is the contract; so mutation stage-hash-feeds-the-whole-buffer breaks the seam the loop actually has — feeding the hasher its whole buffer rather than the bytes read — and two tests catch it from different directions.

blake3's mmap and rayon features cut the isolated pass a further ~4×, measured, and are declined: six crates and a thread pool in a sandbox tool, none of them in the guest build stage's cargo cache, so offline in-guest builds would stop working. The stake is ~0.35 s per apparent GiB that writeback absorbs today.

[0.12.4] — 2026-07-29

Security — the S3 XML parser carried two denial-of-service defects in its dependency

quick-xml 0.39.4 is subject to two RustSec advisories, and isopod's one use of it — quick_xml::de::from_str parsing S3 ListObjectsV2 responses during kernel selection (crates/core/src/image/s3.rs) — reaches both defective paths. That was established by reading the 0.39.4 source, not by assuming the serde surface was insulated:

What reachable means here, and what it does not: the only bytes that parser ever sees come from https://s3.amazonaws.com/spec.ccfc.min — Firecracker's public kernel bucket — over TLS, on the blocking one-shot image import path. Exploiting either defect requires that endpoint, or the TLS path to it, to turn hostile, and the blast radius is a hung or OOM-killed isopod image command in the operator's terminal. No sandbox, no guest, and no long-lived process parses this XML. This was a real defect in a parser of remote input, not a reachable compromise of anything isopod isolates.

quick-xml moves to 0.41.0, which fixes both: a hash pre-filter replaces the quadratic scan, and a start tag declaring more than 256 namespace bindings is rejected instead of allocated. No isopod source changed — the crate crosses two 0.x minors, but the API churn was elsewhere; the de surface and the serialize feature are intact, and the lockfile holds exactly one copy of the crate. cargo deny check advisories fails on the tree before this commit and passes after it.

The upgrade's own risk is behavioural drift in deserialization, and the fixtures in s3.rs stand guard over every shape isopod parses — the pagination fields (including the Option continuation token), the CommonPrefixes roll-up, the Contents keys, each under the real bucket's namespace declaration. All pass unchanged. The one behaviour 0.41 adds — rejecting more than 256 namespace declarations on one element — cannot fire on a well-formed S3 listing, which declares one.

[0.12.3] — 2026-07-29

Fixed — a guest booted with no NIC left loopback down, so it could not talk to itself

The only loopback bring-up lived inside the network-config apply(), which is reached only after configure_if_requested finds an isopod.net token on the kernel command line — and the whole point of --no-network / network: false is that there is none. The guest's one interface stayed state DOWN, so lo came up in every boot except the one that had nothing else.

The failure is expensive because it is partial. bind() on 127.0.0.1 succeeds — binding never required the link to be up — so a workload gets a socket and a port number and fails later, far from the cause, when something dials it. Measured with isopod's own suite as the workload (finding #49): network: false gave isopod-core 363 passed / 18 failed, every failure a broker test that listens and then dials itself; one ip link set lo up first gave 381 / 0.

Loopback is now a boot duty, not network configuration: the agent's main() brings lo up unconditionally (net::ensure_loopback_up), before any network decision, and apply() shares the helper so a runtime reconfigure is still a full replacement on its own. configure_if_requested keeps its contract of being a no-op absent the token. --no-network still means what it says about egress — no NIC is attached and nothing leaves the guest; loopback is the guest's own plumbing, not a way out.

This changes the agent binary, not the protocol: PROTO_VERSION stays 3, and an existing guest image keeps the old agent until rebuilt. The agent-hash freshness check exists for exactly this shape of change — once the host binaries are rebuilt, every agent-carrying image reports stale and the run path refuses it, naming the fix: isopod image build-all for the built flavors, a re-import (local, from cached blobs) for OCI bases.

Mutation loopback-left-down-without-a-nic deletes the unconditional call and pairs with the boot-order assertion in the agent's tests, so the duty cannot be refactored away in silence.

[0.12.2] — 2026-07-29

Fixed — the refusal named an address, but nothing made it name the right one

screen_resolved refuses a name when any address it resolves to is floored, and that function's own doc calls naming the address load-bearing: the message reaches an operator's terminal about their own machine, and it is the only thing that tells them whether they hit a rebinding payload or their own split-horizon DNS.

Nothing pinned it. Replacing bad.ip() with addrs[0].ip() — a refusal that fires correctly and then points at the wrong record — left all 31 tests green. That was established by applying the change and running the suite, not by reading it.

The unit level is the only level that reaches it. The integration test asserts against the addresses the host actually resolved, which is right for what it covers and cannot cover this: localhost answers with loopback and nothing else, so every address it resolves is itself an offender and the match holds under either implementation. A record that passes and a record that is floored only coexist in the unit test's pairs.

The assertion now has two halves and the second is the load-bearing one: the message must name the offender, and must not name the record that passed. Naming addrs[0] satisfies the first half in three of the four pairs — only the good-record-first case catches it, and only the negative half catches it there. Put in operator terms, what addrs[0] does is send someone debugging a split-horizon resolver to look at the one record that is fine.

Addresses are compared by parsing the literal and printing it back, rather than by searching for the spelling the table typed, because the two differ: std prints an IPv4-mapped address in mixed notation and a NAT64 one in hex, so a table's 64:ff9b::169.254.169.254 would never be found in a message that says 64:ff9b::a9fe:a9fe.

Mutation oci-registry-refusal-names-the-wrong-address pairs with the assertion so it cannot be refactored away in silence.

Fixed — the in-sandbox build recipe encoded a payload that was already encoded

docs/sandbox-build.md told you to base64 -w0 a source tarball before handing it to --stdin-file. The channel is binary-safe on both the CLI and the MCP surface — the bytes are base64'd inside the protocol frame either way — so encoding them first only inflated the payload by a third, against a ceiling that is already the binding constraint. The recipe now sends the tarball raw and states that ceiling: PutFile is a single frame capped at MAX_FRAME_LEN, so roughly 6 MiB of raw input, and there is no inbound equivalent to the streamed, unbounded copy_out.

[0.12.1] — 2026-07-28

Fixed — a test asserted the host's resolver configuration, not the floor

the_client_dials_only_addresses_the_floor_allows checked that the refusal message named 127.0.0.1. Which loopback address localhost answers with is the host's business: this developer machine says 127.0.0.1, GitHub's runners say ::1 first. The floor behaved correctly on both — the assertion did not, and the test failed the first time it ran anywhere but the machine it was written on.

It now asserts against the addresses the host actually resolved rather than against hardcoded spellings: it asserts the property instead of the host. That is not strictly stronger than the literal it replaced — it deliberately accepts one thing the old assertion rejected, namely a correct refusal naming ::1, which was the false positive. It is unambiguously stronger than the two-way disjunction that would have been the lazy fix, since that is merely two host assumptions where there was one.

The old assertion was worse than brittle: it was inverted with respect to the v6 floor. Had the IPv6 loopback branch of address_is_allowed been broken, the first floored record would have been 127.0.0.1, the message would have named it, and the test would have passed on the very runner where correct code made it fail.

The listener now binds to whichever address localhost answers with first. That is hardening, not a fix: happy-eyeballs falls back to the other family, and three sibling tests bind v4 and dial by name on a ::1-first runner without trouble. What it buys is that the connector reaches the listener on the family it tries first, so the test's 500 ms "was the socket connected anyway?" check catches a broken floor immediately rather than after a fallback.

[0.12.0] — 2026-07-27

Added — image ls lists imported bases, and image rm removes them

image ls enumerated the built flavors and nothing else, so an imported base was invisible to it — while BaseRef::parse's refusal for an unknown base told the operator to run that very command. A doc and a surface disagreeing is a defect here, and this was one.

One list, not two: image ls answers "what can I pass to --base?", and that is a single namespace — BaseRef::parse takes either spelling and a stage records either in one string. Rows gain kind (builtin/imported) and source_ref, and an imported base gets the identical freshness computation a built one gets, which matters more for imports because every guest-agent rebuild invalidates them.

isopod image rm <name> [--force] implements design decision 5: it refuses while a stage records that base, names the stages, and a forced removal reports what it broke. The sidecar is removed after the image — the opposite order to publishing, because here the bytes never change, so the dangerous failure is a live image left unstamped rather than a new image vouched for by an old stamp.

Over MCP, image_list is new and read-only. Import and rm stay CLI-only: nothing a model asks for can pull bytes onto the host or take a base out from under a stage.

The blob cache key was a real defect, not just untidy. It was slug_for(reference), which maps every run of non-alphanumerics to one dash, so a/b:c and a-b-c collided. A cache directory is an OCI layout — the blobs are content-addressed and safely shared, but the single index.json is not. Two concurrent imports of colliding references could interleave into an image packed from the other reference's manifest, while its sidecar recorded the reference that was asked for. Every digest still verified, because nothing was substituted at the blob level: digests answer "are these the bytes that were named", and the key has to answer "whose layout is this". Now keyed on a readable prefix plus 16 hex of the reference's sha256. Existing oci-blobs/<slug> directories are orphaned but inert; a re-import re-downloads once.

Fixed — a registry name is judged by the address it resolves to

The destination floor screened URLs and IP literals, so https://blob.evil/ resolving to 169.254.169.254 walked straight through, and a name that checked out could be re-resolved before the socket opened. SECURITY.md carried that as an explicit non-claim.

The client now installs a reqwest DNS resolver that applies the address rules to every address a name answers with — one floored record refuses the whole name rather than filtering it, because unlike the broker's operator-written allowlist, this name is as likely as not the registry's own text. The check and the lookup are one act: the connector dials what the resolver returned and performs no lookup of its own, so there is no interval for an answer to change in. dns_resolver rather than resolve_to_addrs precisely because the latter takes its map at client-build time, and half the hosts a pull dials — redirect targets, the token realm — arrive mid-pull. TLS still verifies against the name.

A registry named as a floored address literal never reaches a resolver at all, so isopod image import 169.254.169.254/x/y is now refused at construction. The floor also gained fec0::/10, which the guest broker already refused — the two floors disagreeing was the thing being fixed.

SECURITY.md's non-claim was replaced with three narrower ones, not deleted; the honest remaining gap is the system-proxy path, where the host is handed to a proxy and nothing resolves locally.

Reference::is_local was also wrong for IPv6: it split the authority on :, so [::1]:5000 yielded "[" and a bare ::1 yielded "" — meaning the "::1" arm was unreachable, a case listed as supported that could never match. It failed closed, so an IPv6 loopback registry was refused rather than over-trusted.

Fixed — two concurrent warm-pool builds could publish each other's bytes

Dogfood finding #32, the only open HIGH and the only one that could publish corrupt state rather than merely waste work. snapshot::ensure created the snapshot directory and wrote vmstate.partial / memfile.partial — fixed names, no lock. Two runs of the same warm shape both saw an incomplete snapshot, both dumped several hundred MiB into those two paths, and both renamed, so one could publish a memory file the other was still writing. Every later resume of that shape would then get it.

This is not a contrived race. Any image rebuild empties the pool for every shape at once, and the MCP tool description actively invites concurrent sandboxes from separate agents — so two runs arriving together on a cold pool is the ordinary case, and the window is the several-second memory dump.

An exclusive per-keyhash flock now guards the build, in the shape net::claim_lock already uses (O_NOFOLLOW, regular-file check, LOCK_EX | LOCK_NB, so a crashed owner's lock is simply gone and needs no staleness heuristic). A second arrival waits up to 90 s, notices the moment the winner publishes, and reuses that snapshot rather than building a second one; if the wait expires it cold-boots, which is what a cache miss does anyway. The lock lives inside the keyhash directory, because warmpool rm removes directories and skips plain files — a lock beside them would survive every prune forever.

Staging names now carry the pid as well, which the finding named as the fallback: with the lock gone, a loser can still only destroy its own bytes.

Tested at the primitive and at the call site. ensure grew an ensure_at seam so the wait-then-reuse path is exercised without booting a VM and without touching the process-global $ISOPOD_HOME — a trap this codebase has fallen into twice. All three tests fail if the flock is removed; three mutations.

Measured — an imported image boots like a base isopod built itself

The wave-2 exit benchmark, in BENCHMARKS.md. 30 samples per cell, one shadow $ISOPOD_HOME, same host, same guest agent, same 1 vCPU / 512 MiB, same warm/cold path.

Base Origin On disk warm p50 resume_ms p50
base-sqfs built (busybox) 1.54 MB 238–254 ms 43–50 ms
oci:alpine-3.20 imported 3.82 MB 230–235 ms 43–45 ms
oci:python-3.12-alpine imported 17.11 MB 238 ms 43 ms
base-alpine built (py/node/gcc) 150.72 MB 318 ms 48 ms

The like-for-like pair — the two minimal bases — tie. They are reported as ranges because they were measured twice and the first base-sqfs sample was an outlier with a fat tail (254 ms, p90 320) that did not reproduce (238 ms, p90 262). At this sample size importing costs nothing at boot; it is not faster, and the changelog does not claim it is.

What moves is content, not origin: the 150 MB toolchain base pays ~80 ms more on a warm run, and nearly all of it is exec_ms (106 ms vs 34) rather than resume. resume_ms is flat at 43–50 ms across all four — the snapshot restore does not care what the base is or how big it is.

Import cost, which is the number an operator feels first: alpine:3.20 1.7 s cold and 1.0 s with blobs cached; python:3.12-alpine (4 layers) 3.5 s and 2.4 s. The cached figure is what a re-import costs after a guest-agent rebuild invalidates every imported base.

scripts/bench.py gains --base, so a built base and an imported one are measured by one harness rather than two.

Fixed — five credential and SSRF defects in the registry client

An adversarial pass scoped to isopod-oci-registry, the wave-2 exit criterion. The crate had never had one. Six attackers over separate surfaces, every serious finding then handed to a skeptic told to refute it. Each defect below was reproduced before it was fixed.

The Docker Hub credential was sent to every registry. docker_config_auth tried three keys and returned the first hit; the third was Docker Hub's legacy https://index.docker.io/v1/ key, tried unconditionally. So any operator who had ever run docker login sent that credential to whatever registry they named next — isopod image import evil.example.com/x/y handed it over on the first request, before any challenge, with no redirect and no hostile-registry behaviour required. For a localhost reference the scheme is http, so it went in the clear. Measured with a config containing only the Hub key: all six references tested came back with the Hub credential. The legacy key is now consulted only when the reference actually names Hub, and keys are matched to the host they name — ghcr.io.evil.com is not ghcr.io.

A token realm had no destination floor. Challenge::parse required only that the realm be https or loopback — and the loopback half was ungated, while the identical exemption on the redirect path was gated on the operator having named a local registry. So a remote registry could answer 401 with realm="http://localhost:5000/token" and have the client post the operator's credential, in the clear, to whatever was listening on their own machine; or name https://169.254.169.254/, which is https and is the cloud metadata endpoint. The realm now goes through the same predicate a redirect target does. One floor, not two, because the second one was the weaker one and it was on the path that carries a credential.

A credential dropped at one hop came back at the next. carry_credential was recomputed per hop from may_carry_credential(current, next), which compares the two ends of one hop. Once a redirect had taken the client to the attacker's origin, the next hop compared their host to their host, said yes, and re-attached the token the first hop correctly dropped. Two redirects instead of one defeated the origin rule entirely. It is now a latch. The predicate was never wrong — the loop threw its answer away — so this is caught by a fake-registry test that redirects twice, not by a unit test of the predicate.

A host reached by redirect could start the token dance. The 401 branch did not consult carry_credential, so a CDN the registry redirected to could challenge the client, name a realm, and be paid in the operator's ~/.docker/config.json credential. A CDN has no business challenging us; the pull now fails with its 401.

IPv6 spellings walked through the SSRF floor. The floor screened 169.254.169.254 and matched IPv6 by prefix, so [::ffff:169.254.169.254] — the same address, IPv4-mapped — was allowed, as were the IPv4-compatible and NAT64 (64:ff9b::) spellings, and CGNAT space. An IPv6 literal is now reduced to the IPv4 address it names before the rules are applied. The crate's own doc had claimed this was "the same destination floor the guest egress broker applies"; the broker handles mapped spellings and this did not, so the claim was false as well as the code being wrong.

A pinned digest went unverified when the blob was already cached. repo@sha256:X is a promise about exact bytes, and the by-digest fetch inside the index branch checked it — the top-level one did not. It only showed on a re-pull: the write path skips a blob that is already present and correct, so a substituted body was never hashed, while the config and layer descriptors driving the rest of the pull were parsed straight out of it. The same rule now applies to both, before anything reads the bytes.

SECURITY.md gains an import section stating what holds and what is not claimed — in particular that this floor judges the URL, not the resolved address, so it is not equivalent to the broker's resolved-address gate.

One existing test asserted a vulnerable behaviour (http://localhost:5000 as an acceptable realm) and now asserts the fix, which is the second time this session a test had encoded the defect it was named for.

Added — a run can boot an imported base, and its config becomes run defaults

--base oci:<name> (and the same over MCP) boots an imported image. Verified live: alpine:3.20 pulled from Docker Hub, imported and cold-booted in ~0.8 s running a command as root, then committed as a stage and forked — the stage records oci:alpine-3.20 with the image's content id, and the fork resolves it back without --base.

Base selection is now a BaseRef: a closed type at the CLI and MCP edge, plain (slug, digest) strings everywhere the choice is persisted. StageMeta::base, BaseId and SnapshotKey::base were already strings, so nothing stored changed and no stage needed migrating — a RootfsFlavor::Imported variant would have rippled through every match on the enum instead. The oci: prefix is required: a bare name would collide with the flavor slugs the moment somebody imported an image called base-alpine, and an unknown base now says how an imported one is spelled.

The image config becomes defaults, never behaviour. Env is merged under the run's own environment and WorkingDir is used only when the run names no cwd, so a python:3.12 base finds python on PATH without the caller restating it, while a run that sets PATH still wins. Verified both ways, with a built-in base as the control: it keeps the agent's baseline PATH and picks up nothing.

The defaults come from the base the run actually resolved, not from the --base field. A fork boots the base its stage recorded and ignores --base, so reading the caller's field would apply one image's environment to a run booting a different image. A WorkingDir of / or "" is treated as "no opinion" rather than forced onto the run, and an Env entry with no = is skipped rather than half-guessed.

Added — isopod image import

Three ways in, one path after that:

isopod image import alpine:3.20
isopod image import --oci-layout ./layout --name my-base
isopod image import --docker-save ./saved.tar --name my-base

Pull (or read) an image layout, verify every blob, unpack the layers through the confined extractor, adapt the tree, pack it and stamp it. Verified live against Docker Hub: alpine:3.20 and busybox:1.36 both import and pack, and the same image imported from a registry, from a local layout and from a tarball produces the same content id — three sources converging on one image, which is what makes the layout and tarball paths worth having rather than merely present.

Blobs are cached under ~/.isopod/images/oci-blobs/, so a re-import is local. That is not a nicety: an imported base is stamped with the guest-agent hash it was built against, and every agent rebuild invalidates every imported base.

A legacy docker save archive — the pre-OCI format, a top-level manifest.json naming <hash>/layer.tar — is refused by name, with the skopeo command that converts it. It is not an image layout, and the layout reader's own "no oci-layout" is accurate and useless for the operator holding one. Failures name the path the operator actually typed, not the temporary directory a tarball was extracted into.

Not yet wired: an imported base cannot be selected with --base. The run path resolves bases through the built-in flavor enum, and widening that is the next step, with the image config becoming run defaults at the same time.

Added — an unpacked OCI tree becomes a base isopod can boot

The adaptation half of an image import. It takes the directory tree the extractor produces and adds the few things the guest agent needs in order to be PID 1: the agent at /.isopod/init with /init pointing at it relatively, the three empty overlay mountpoints (/overlay, /mnt, and a /layers that must stay empty), the pseudo-filesystem mountpoints an image happens not to ship, and a /tmp if there is none. The image's own /sbin/init is left alone — on a Debian-derived image that is systemd, and the kernel boots init=/init, so /init is the only path isopod has to own. An image that ships its own /init has it replaced, and the sidecar records that it did.

isopod runs your image's filesystem, with isopod's init — not "isopod runs your container". An imported image's ENTRYPOINT can never be PID 1, because PID 1 is the agent that does the overlay mounts, the pivot and the RPC. The entrypoint, command and USER are recorded and never acted on, and the ones that are ignored rather than merely unused say so in the command's own output.

An image with no /bin/sh is refused by name at import time, with the shape of image that does work. Distroless and scratch images cannot be run by a surface whose exec is /bin/sh -c, and the alternative to refusing is an exit 127 inside a VM, long after the import looked like it worked. The check resolves /bin/sh within the image: Path::exists() follows an absolute link like /bin/sh -> /bin/busybox against the host's root, so on an ordinary machine a distroless image would have passed.

The pack is the built-in flavors' pinned mksquashfs invocation plus a pseudo-file carrying the setuid, setgid and sticky bits — which is the only place those bits are ever applied. They are never written to the host tree, where they would sit on attacker-authored files in the operator's home before any VM exists. Every path in that pseudo-file is quoted and escaped: the format is space-delimited with a type field in second position, so a tar entry named evil c 0666 0 0 1 3 would otherwise render a line that reads "create a character device". And because mksquashfs silently ignores a pseudo-file line naming a path it cannot find — exit 0, no diagnostic — the pack verifies that as many special-mode entries came out of the image as went in, rather than treating a successful exit as evidence.

The sidecar gains an oci section recording the reference, the resolved platform, the manifest and config digests and every layer digest, so a re-import is a local operation. That matters more than it sounds: the freshness check compares the agent hash, not only the protocol version, so every guest-agent rebuild invalidates every imported base.

No CLI surface yet — isopod image import is the next step, and the documentation lands with the command rather than ahead of it.

Fixed — a setuid bit does not survive its own removal

Report::setuid_paths is what the pack step reapplies inside an imported image, and it was the union of what every layer set rather than a description of the finished tree. Three ways an image could get back a privilege it had given up:

The set is now resolved as the layers are applied: a bit is recorded when an entry carries it and dropped when an entry replaces that path without it, and the deletions are reconciled at finish() by asking the tree rather than by reimplementing the whiteout and opaque rules a second time — a second implementation of a deletion rule is a second chance to disagree with the first. setuid_paths is now documented as a snapshot rather than a per-layer delta, sorted by path, and both the per-layer report and the running total carry the same resolved answer instead of two different meanings for one field.

Found while building the pack step that consumes it, which is the first code to ever read the field for its stated purpose. Two of the suite's own assertions had encoded the defect — one expected a whited-out file to stay in the list.

Stages now record which build of the base image their layers were made over, and a fork refuses a base that has been rebuilt since. Existing stages are unaffected: they carry no stamp, so there is nothing to disagree with, and they fork exactly as they did.

Fixed — a base image built twice from one tree is the same image

mksquashfs wrote the current time into the superblock and copied each file's mtime out of the assembly directory, which is created fresh on every build. So isopod image build-rootfs --force over an unchanged tree minted a new content id: measured, two runs four seconds apart produced two different images of the same root filesystem. (Three runs inside one second produced one, which is what made it look reproducible.)

Nothing about that id is cosmetic. It is what a stage records as the base its layers were made over and what the warm pool keys a snapshot on, so a rebuild that changed nothing retired every stamped stage on the flavor and orphaned a 512 MiB snapshot — and image build-all is documented as required after a PROTO_VERSION bump, which made the mandatory operation the expensive one.

The pack now pins both halves of the clock, -mkfs-time for the superblock and -all-time for the files. Each alone still moves the id; together the same tree packs to the same bytes across any gap, while a real content change still moves it. 1980-01-01 rather than the epoch, because it is the earliest instant a DOS/ZIP date field can represent and a guest that archives something it copied out of the base should get a valid date. Nothing in the guest reads base timestamps: base-alpine ships hash-based bytecode caches (PEP 552), and every file a run writes lands in the overlay upper with a real, strictly newer mtime.

SOURCE_DATE_EPOCH is removed from the packer's environment rather than honoured. squashfs-tools reads it itself and treats it as competing with the flags — with both present it exits SOURCE_DATE_EPOCH and command line options can't be used at the same time to set timestamp(s) and builds nothing, so an operator whose shell exports it, which is the ordinary reproducible-build environment, could not build an image at all. It also settles what the variable would otherwise raise: an image id that moves with the ambient environment is the defect the pin exists to close.

This is a behaviour change to a guarantee, not only a speedup: a rebuild is no longer proof that a base has moved, so the prose that said "any rebuild counts" is now wrong and has been corrected in README.md, docs/getting-started.md, docs/mcp-usage.md and the Unverifiable refusal — which used to promise that re-stamping a sidecar-less image would always mint a new build, and now says it restores the same id when the tree has not changed. The ext4 dev flavors are deliberately untouched: they are not stage bases, so no content id is keyed on them.

Added — a stage records the base build, and a fork checks it

StageMeta gains base_sha256: the content id (the image sidecar's sha256) of the base image the run actually booted. Every commit stamps it; isopod stage list / stage info and the MCP stage_list / stage_info surface it.

The flavor slug was never enough to identify a base. isopod image build-all replaces base-alpine with a different root under the same name — new Alpine packages, a new guest agent — and a stage's layers are overlay upperdirs over that build. They still mount: the merge succeeds, the run starts, and the breakage arrives later as a chain whose contents no longer match what is beneath them (site-packages whose interpreter moved is the usual shape). Nothing in the 0.11.0 run path could tell the two apart.

isopod run --stage <ref> now compares the stamp against the image on this host before anything boots, and refuses a mismatch, naming both content ids and both ways out. Every stage in the chain is checked, not just the one named: the layers of every ancestor are mounted too, and checking only the tip let a single unstamped link launder everything behind it. The same comparison runs in the store, so a stacked commit cannot record a chain that mixes known-different builds unless the operator opts in below.

Two cases deliberately do not refuse:

One case refuses in every case, including under the override below: a flavor mismatch. Those layers are not stale, they belong to a different root.

ISOPOD_ALLOW_BASE_SKEW=1 overrides the refusal, loudly. It covers the commit as well as the boot on purpose: rebuilding the guest images changes the base of every stage at once, and an escape hatch that boots the fork but then refuses to save what it produced would strand exactly the work it exists for. It is an escape for a run, not a repair: the layer it commits records the new image while its ancestors keep their own stamps, and since the check walks every link, the stage it produces still needs the variable to boot. Rebuilding the stage on the current image is what clears it; the store keeps the evidence either way.

Eight mutations were added to scripts/mutation-check.py covering the new guards: accepting any content id, dropping the stamp at commit time (which fails nothing at the time and silently disables the check for good), letting the override excuse a flavor mismatch, the run path never consulting the check at all, the check seeing only the chain's tip, the commit path always allowing skew, and a rebuild leaving a stamp that outlives the image it describes.

Fixed — in this release's own new code, before it shipped

An adversarial pass over the above found five defects in it. They are listed because the pattern is the finding: three of them were places where the tests covered the policy and nothing covered the code that calls it.

Changed — the base image ships one mountpoint, not ten

Base images carried /rom and ten numbered /layers/0..9 directories. Since the guest started mounting a tmpfs over /layers and creating /layers/<i> per layer at boot (0.11.x, dogfood finding #26), the baked directories have decided nothing — the off-by-one they caused is gone, and any depth the chain cap permits works without them. /rom had no reader at all. Both are dropped: a base image now ships overlay, mnt, and an empty layers.

An image only picks this up when it is rebuilt, and an image still carrying the old directories is harmless — the tmpfs masks them. There is no need to rebuild for this reason alone.

Included from the unreleased 0.11.1

scripts/mutation-check.py itself, which asserts that a fixed set of deliberate breakages makes the suite fail — CI proves the tests pass, not that they check anything. It failed on its first run: the copy_out staging-name test sampled a single name where the clamp point depends on the pid's digit count, so whether it caught a mid-character split was decided by luck. That test now sweeps every UTF-8 width at every suffix length the real caller can produce.

Added — isopod-oci-unpack, the layer extractor, on its own

A new workspace crate that applies OCI image layer tars onto a directory tree. Nothing depends on it yet, and it is wired into no command — that is the point. It is the one component whose failure writes attacker-authored bytes into the operator's home directory before any VM exists, so it exists and gets attacked on its own before anything dials out to a registry.

Confinement is a directory-fd walk from the destination root with O_NOFOLLOW on every open, rather than a check against a resolved path. The difference is the cross-layer symlink: layer 1 ships foo -> /home/you, layer 2 ships foo/.bashrc, and each layer is innocent read alone. It is also why a dangling link — the shape that escaped copy_out in 0.11.0 — needs no special case here: nothing ever looks at a link's target.

Also enforced, each with a test and a mutation: .. and absolute names refused rather than normalised; hard-link targets confined by the same walk, and linked without AT_SYMLINK_FOLLOW so a link-to-a-link cannot share a host inode; device and FIFO entries skipped and reported; setuid, setgid and sticky bits recorded for the pack step and never written to the host; .wh. and .wh..wh..opq whiteouts applied so that nothing an image author deleted survives; cumulative anti-bomb ceilings measured on the decompressed stream; and a staging directory that is discarded on drop, so a refused image leaves nothing behind at all.

Attacking the crate during development found one escape that the design did not anticipate: a whiteout marker spelled .wh... yields the delete target .., which is the only name that reaches the delete walk without going through the entry-name component loop. At the top of the tree that names the staging root's parent — the caller's own destination directory — and the recursive delete emptied it. Refused now in both the name planner and the syscall layer, with a mutation for each so the pairing cannot quietly become decorative.

A second pass, scoped to the crate alone before anything is wired to it — the wave-1 exit criterion — found no escape. The directory-fd walk, the hard-link confinement, the whiteout and opaque rules and the teardown all held against the inputs built against them, including the delete paths, which refuse a planted link exactly as the write path does. What it did find were two defects either side of the confinement, both the shape the previous three passes kept finding: a fix that covered the branch that worked.

The umask test cannot set a umask — it is per-process and Rust runs tests as threads — so it re-executes the test binary under umask 077 and asserts on what that child produced.

[0.11.0] — 2026-07-26

A hardening release. An adversarial review of shipped 0.10.0 — 34 agents, no design framing supplied, findings verified by exploiting them rather than by reading — turned up eleven real defects. Two of them invalidated 0.10.0's stated threat model outright, so they are fixed here rather than deferred. Existing hosts must re-provision before their next filtered run (see the kernel guard below); public-slot runs are unaffected.

A second adversarial pass — this one over the fixes themselves, before they were pushed — found nine more defects, several of them in the new code. Those are folded in below rather than listed separately, but three are worth calling out because the first version of this release claimed they were closed and they were not:

A third pass, over the second pass's fixes, found four more. Two of them were in the second pass's own new code, and both are stated here rather than folded in, because the pattern is the finding: a fix that closes the case it was shown is not the same thing as a fix that closes the class.

The fix is in two places on purpose. The destination is normalised before any guard runs — trailing separators and . components dropped, a .. below the deepest existing directory refused outright and any other .. resolved against the existing prefix and re-tested against the root — so no guard can be shown a different final component from the one that will be opened. And the write itself now opens with O_NOFOLLOW, so it refuses to traverse a link whatever the check concluded — because a path check and a File::create are two lookups of one name, and a symlink planted between them defeats any amount of checking. O_NONBLOCK goes on with it, so a FIFO destination fails instead of blocking a thread forever. This is on the shared path, so the CLI gets it too: isopod run --copy-out g:/host/p now writes to /host/p or fails, rather than through /host/p to wherever it points. That is a statement about the final component; a symlink among the parent directories is still followed, which SECURITY.md records as an explicit non-claim. The read path (stdin_file) was verified unaffected — it canonicalises, and both spellings were already refused. - The pinned-host floor checked a different parser than the dialer uses. The guard added earlier in this release classified a credential's pinned host with host.parse::<IpAddr>() and skipped anything that failed, reasoning that a name goes through the floored resolver. But the upstream leg is format!("https://{host}{path}") handed to reqwest, whose URL crate uses the WHATWG host parser — which reads decimal, hex and short-form IPv4 as addresses and normalises them to a dotted quad before hyper sees the authority, so no resolver is ever consulted. 2852039166, 0xa9fea9fe, 2130706433, 127.1, 0177.0.0.1 and 0x0a6b0801 all failed IpAddr::from_str, all skipped the guard, and all dial 169.254.169.254, 127.0.0.1 or a sibling run's broker gateway. The guard now classifies the host with the same parser the dialer uses, and additionally refuses any pinned host the parser rewrites at all — egress.denied recorded host: "2852039166", which an operator cannot grep for 169.254.169.254, so the store, the log and the destination are now required to be one string. A pinned name that the URL parser accepts is untouched and still floored at resolution; one whose last label is all digits or 0x-hex (api.123) takes the parser's address branch, fails there, and is refused at startup — reqwest could not have dialled it either, so the credential could never have been spent. - Three lockfile shapes the flock claim stopped handling. Replacing O_CREAT|O_EXCL with O_CREAT (necessary — the lockfile is now durable) incidentally dropped the refusal O_EXCL was providing. A FIFO at slot-<i>.lock made the claim block in open forever, with no timeout on that path, which under the MCP server wedges a blocking-pool thread for good; a symlink was followed, putting the flock on an inode outside the 0700 state directory; a directory failed the entire pool with every other slot free. The claim now opens with O_NOFOLLOW|O_NONBLOCK and fstats the descriptor, refusing anything that is not a regular file, and a slot that cannot be opened is skipped with a warning instead of failing the scan — the exhaustion error says how many were skipped and why. Not guest- or MCP-reachable (the directory is 0700 and hostio refuses ~/.isopod for every root), so this is robustness rather than a boundary crossing, but the hang was new in this release. - SECURITY.md claimed a property the code does not have. The flock bullet said the slot was "reclaimable immediately with nothing to sweep and no liveness guess". The lock is — the kernel drops it, kill -9 included, verified. The slot is not: a kill -9 of the supervisor leaves its Firecracker holding the tap, and the next run's registry::reap_orphans has to SIGKILL it first, on a pid-and-start-time liveness test. Reproduced: the lock read free within 2 s while the orphaned VMM still held isopod-tap0. Both that bullet and the CHANGELOG's "correct the instant after" now say what actually happens.

A fourth pass, over the third pass's fixes, found no way to escape the confinement — three independent reviews, 42 end-to-end escape attempts all refused with no filesystem delta, a 2,770,211-host fuzz of the pinned-host classifier with zero invariant violations, and 36 concurrent slot claimants with zero double-claims. It found one behavioural defect and a cluster of prose that overstated the code, all fixed here:

A fifth pass, over the staging fix itself, found the pattern had not stopped. The new "a failed copy leaves nothing behind" invariant had one hole: a malformed base64 chunk returned straight out of the stream loop past the cleanup, and since every attempt takes a fresh sequence number, a guest repeating it leaked one staging file per attempt rather than reusing one name — unbounded host disk, in a function whose neighbouring comment exists because "a malicious agent could stream forever and fill the host disk". The staging path now owns its own Drop, so the invariant holds for every early return rather than the ones written so far. Also fixed: the staging suffix could push a legal destination name past NAME_MAX; the publish half of the copy had no test at all, so deleting the mode mask kept the suite green; the flock line added to docs/m4-verify.md two commits earlier described flags that do not do what it said (flock(1) always creates the file, and -E 1 is the default); and an eleventh contract, on the claim_network() wrapper, still said crash recovery needs no step.

A sixth pass found no defect in the drop guard — 24 concurrent copies with 12 aimed at one destination left every success byte-intact and no residue, and the pass-5 leak is gone (five malformed chunks left five staging files before, zero now). What it found was one operational hazard and two false claims:

Four mutants that survived a mutation survey are now killed: publishing before the rename rather than after, classifying a device as staged, dropping the character-boundary walk-back in the name clamp (a reachable panic — the destination is caller-supplied and may be multibyte), and widening the staging file's in-flight mode. The staging error hint no longer tells every failure that the directory must be writable when the cause was a leftover file or a missing directory. And a guest-chosen mode is no longer applied to a Direct destination: carrying the exec bit exists so an artifact isopod created arrives runnable, and a device or FIFO the operator already owns is not that.

Pass 4 also produced one new non-claim rather than a fix. copy_out keeps the executable bit by design — a binary built in the sandbox should arrive runnable — and the default host-I/O root is the server's working directory, i.e. a project containing .git/hooks/. A copy-out to .git/hooks/pre-commit therefore lands executable and runs on the operator's next git commit, outside any VM; demonstrated against a running server. Stripping the bit by default would break artifact extraction, and enumerating which files a project treats as code is not something isopod can do — .envrc, a Makefile, a CI config and a node_modules/.bin entry are all the same shape. SECURITY.md states the limit plainly instead: nothing in a writable root is safe from being made executable.

Fixed — the MCP surface could read and write arbitrary host files

sandbox_run's two arguments that name a host path, stdin_file (read) and copy_out[].host (write), were used verbatim. That was demonstrated end to end: stdin_file: "~/.isopod/credentials.json" returned the credential store in stdout, and copy_out wrote guest-authored bytes anywhere, creating parent directories. Either one undoes the premise the credential design rests on — that the caller is a model whose context the sandboxed code may have written, and so may name an alias but never a secret.

Both paths are now resolved and confined to a host-I/O root, defaulting to the server's working directory, with symlinks resolved before the check so a link planted inside cannot reach out (including one whose target does not exist yet — see the note above). ISOPOD_MCP_HOST_IO_ROOT moves the root (/ restores the old behaviour, explicitly); ISOPOD_MCP_HOST_IO, ISOPOD_MCP_STDIN_FILE and ISOPOD_MCP_COPY_OUT set to off refuse the arguments outright. The startup log line says which is in force. The CLI is deliberately unaffected — there the caller is the operator.

The confinement is a path check, so it is also given the things a path check cannot infer. A dangling symlink is refused rather than written through (where the write would land cannot be resolved, so it cannot be checked), and a multiply-linked file is refused outright (a hard link is a second name for an inode and resolves to itself, so no prefix test can see it). The destination is normalised before any guard runs — trailing separators and . components dropped, .. still refused — so no guard can be shown a different final component from the one that will be opened. And the write opens the final component with O_NOFOLLOW, so it refuses to traverse a symlink whatever the check concluded; a check and an open are two lookups of one name, and only the syscall can close the gap between them. isopod's own state directory is refused whatever the root is, including with the confinement off. An empty ISOPOD_MCP_HOST_IO_ROOT no longer reads as a root — Path::starts_with("") is true for every path, so it admitted everything while the log said "confined to " — and a switch set to anything not recognisably affirmative now reads as off rather than staying open.

stdin_file also gained the guards the credential loader already had: regular files only (a FIFO blocks the read forever, /dev/zero grows the buffer until the host is out of memory) and a 4 MiB ceiling enforced on the read, not just on the stat.

copy_out no longer applies the guest's reported mode verbatim. The exec bit still travels — that is why the mode travels at all — but setuid, setgid, sticky and group/other write are cleared, and owner read/write is always granted. A sandbox could otherwise land a setuid-to-the-operator binary in the project directory with chmod 6777.

Fixed — the broker dialled from the host with no destination floor

The broker resolves and connects on the guest's behalf as a host process, so the packet filter's public-only-egress rule — which governs forwarded traffic — never applied to it. An allowlist entry that resolved inward was a confused deputy with host-level reach: allow_cidrs: ["169.254.169.254/32"] reached cloud instance metadata, allow_hosts: ["localhost"] reached the host's own services, and 10.107.0.0/16 reached sibling runs' brokers — including their credential endpoints.

Every destination is now checked after resolution: loopback, link-local, multicast, broadcast, IPv4-mapped spellings of those, and isopod's own slot supernet are refused outright; private and CGNAT ranges follow the host's --allow-lan-egress, which until now was recorded in the manifest and never consulted. The same floor applies to the DNS answers the broker synthesises and to the credential endpoint's upstream leg (via a custom reqwest resolver — that leg does its own resolution, so a pinned host whose DNS answered 127.0.0.1 would otherwise have received an Authorization header). Refusals are recorded as non_public_address, and the explanation names --allow-lan-egress.

Fixed — every broker listener served every process on the host

All four listeners bind the slot gateway, which is a host address: packets a host process sends there are delivered locally and never cross the iifname "isopod-tap<i>" input rule that gates guest access. So while a run with --inject was live, any local account could curl http://10.107.8.1:3129/github/user and spend the operator's token, with the calls landing in the flight recorder as though the sandbox had made them. Each listener now serves only its own slot's guest address, TCP and UDP alike, and closes anything else before reading a byte. The first refusal explains itself on the supervisor's stderr.

Added — a filtered slot's enforcement is verified at run time

Nothing checked that the nftables ruleset was still loaded. slots.json existing and the taps existing were the whole test, and taps created with ip tuntap add outlive nft flush ruleset — which a firewalld reload performs — so a "filtered" run would boot on a wide-open slot while its broker held a live token.

sudo isopod setup now also clears net.ipv4.conf.isopod-tap<i>.forwarding for every filtered tap. That makes the kernel refuse to forward what arrives there independently of the ruleset, and — the point — it is world-readable, so the unprivileged runtime can confirm it. Every filtered run checks the whole filtered pool before anything boots, and the claimed slot again after, and fails closed with the re-provisioning command. This is why an existing host must re-run sudo isopod setup: its taps predate the flag.

SECURITY.md now states plainly what this does not cover — the guest→host half of the guarantee still rests on the nftables input chain, and an unprivileged process cannot read the live ruleset to confirm it.

Fixed — resource and lifecycle defects

owner.pid now records the pid and that pid's start time. A pid alone is not an identity: these records outlive their runs by design, so on a busy host a finished run's pid is reused and a bare /proc/<pid> test reports that run as live forever. Records written by an earlier isopod carry only a pid and keep the old, weaker check — reading them as dead would collect live runs, which is the failure this path exists to prevent. - vm_gc deleted by a path from meta.json. The record's vm_id came from the file rather than the directory, so "vm_id": "../../.." in any writable meta.json made remove_dir_all traverse out of the vms root — reachable through a copy_out destination. The directory name is now authoritative. - A network slot is now claimed with flock, and the staleness heuristic is gone. A run holds an exclusive flock on ~/.isopod/net/slot-<i>.lock for its whole lifetime. That deletes the entire class of bug this release spent three attempts on: there is no staleness to decide, no write grace, and no unlink — the kernel releases the lock when the owning process dies, kill -9 included, so the lock is free the instant its owner dies, with nothing to reconcile. The slot is a separate question and always was: a kill -9 of the supervisor leaves its Firecracker alive and still holding the tap, so the next run's registry::reap_orphans still has to SIGKILL it before it can boot, and that check is a pid-and-start-time liveness test. What is gone is deciding occupancy by guessing; what remains is killing a process that is demonstrably still there. flock also belongs to the open file description rather than the process, so two concurrent runs under one MCP server are told apart for free; fcntl record locking would have handed the second one a lock the first still needed. The lockfiles now carry no contents at all, and are no longer unlinked. net::sweep_stale is removed.

This release introduced a regression that this change closes. Earlier in 0.11.0 the claim started writing a <pid> <nonce> token into the lockfile so a release could tell a sibling run's lock from its own — but the staleness parser was left reading the file as a bare pid. Every real lockfile therefore failed to parse, took the "unparseable" branch, and was declared stale five seconds after it was written, with its owner alive and its VM on the tap. Since every claim swept first, the next run unlinked a live lock and took the occupied slot; the loser's firecracker died on Open tap device failed … Device or resource busy cold, or Failed to restore devices … Net: on a warm resume. Two runs more than five seconds apart collided reliably. It failed closed — firecracker will not open a tap twice — so it was a reliability defect and not a crossing of the isolation boundary, but it was a live one. The tests missed it because they claim and release in milliseconds, inside the grace period, and because the one test that built a lockfile by hand wrote the bare pid the code had stopped writing. No fixture stands in for a held lock any more — the regression test ages a real lock, taken through the claiming path, past the old grace before re-claiming. The two tests that still write a lockfile by hand do so precisely to prove an unheld file's contents are never read.

Also fixed with it, and previously listed here as not fixed: deciding staleness and unlinking were two operations, so two claimants reading the same dead pid could both proceed, the second unlinking what the first had just written. - A run outlived its own timeout. getaddrinfo has no cancellation, and dropping the run's tokio runtime waits for every blocking thread with no bound — so a name whose nameserver never answered held the sandbox_run request (and a server thread) for the full resolver budget after the report was built. Both resolution paths are now bounded at 5 s and the teardown at 250 ms. - The state tree was re-chmodded to 0755 on every call. Not at create time — every call, so chmod 700 ~/.isopod/vms was silently undone by the next run, and every run's console.log, exec-*.log and egress.jsonl were readable by any local account. Directories are now created 0700 and tightened if found looser, never loosened.

Fixed — smaller correctness

[0.10.0] — 2026-07-25

The endpoint is not a reverse proxy: the guest does not compose the request. It reads a stated intent and constructs a new request from its own parts — its own Host, its own Authorization, a normalised path, and at most two allowlisted headers. Redirects are disabled on the upstream leg, because following one would carry the credential to a host the operator never named, chosen by a party who is not the operator. The client also ignores the host's own *_PROXY variables: those are exactly what isopod exports into a filtered guest, so an isopod running inside an isopod sandbox would otherwise route its credential leg through its parent's broker.

--inject lives inside the egress policy rather than beside it, so naming a credential switches the run to a filtered slot. As a sibling field, --inject github with no --allow-host would have left the run on a public slot with full NAT egress and nothing enforcing the credential's allow list — the one combination the feature must never produce.

The pinned host is deliberately not allowlisted, and a direct connection to it is refused as pinned_credential_host rather than a generic denial: the fix is to use the endpoint, not to widen a list.

[0.9.1] — 2026-07-25

[0.9.0] — 2026-07-25

[0.8.1] — 2026-07-23

[0.8.0] — 2026-07-23

[0.7.3] — 2026-07-23

[0.7.2] — 2026-07-23

[0.7.1] — 2026-07-23

[0.7.0] — 2026-07-23

The post-v1 hardening and findings-fix wave; adopts the versioning policy (versions 0.2.0–0.6.0 below were tagged retroactively at their milestone-close commits).

[0.6.0] — 2026-07-22

[0.5.0] — 2026-07-21

[0.4.0] — 2026-07-21

[0.3.0] — 2026-07-21

[0.2.0] — 2026-07-21

[0.1.0] — 2026-07-21

Rendered from CHANGELOG.md on the main branch.