Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.
Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.
A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.
Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.
The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.
Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.
Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
Determine node health at start completion instead of unconditionally
reaching Running. Zero transports up is now Failed (fatal): start()
tears down cleanly and returns an error, and the daemon exits. Any
configured optional child that failed to start - a transport beyond the
first, Nostr, mDNS, TUN, DNS, or a worker pool - leaves the node
Degraded but serving, with an operator warning naming what failed. All
configured children up is Full. A child the node was never asked to run
does not count against health.
The published NodeState gains Degraded and Failed variants, both visible
via control queries; Degraded is operational, Failed is not. The
lifecycle FSM gains the health states plus the PublishState action that
drives them - a health fork cannot be a single direct state write, which
is why the earlier commits deferred it to here.
Runtime child-exit health re-evaluation (a running child dying) is a
separate liveness-monitoring mechanism left for a follow-up; this commit
is start-time health only.
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.
- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
namespace every docker resource to it: a fipsci_<run>_<suite> compose project
per suite and per parallel chaos child, and per-run image tags
(fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
parallel chaos children, then force-remove this run's docker resources via
the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
SIGKILL regardless of network name: direct docker run/network resources, the
sidecar compose services, and every per-suite compose network (acl-allowlist,
boringtun, firewall, nat, static, both tor suites, and the chaos generator
template). Parametrize the static/sidecar compose image refs so the per-run
tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
override on the sim CLI, assigned per-child in ci-local.sh) so parallel
children never collide on a shared docker subnet, and a chaos net can never
span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
docker's default-address-pool range, so an auto-assigned net cannot land on
it either; node IPs derive from the subnet, so no scenario config changes.
The previous default configured systemd-resolved with `resolvectl dns
fips0 [<fips0_addr>]:5354`, intended to bypass an Ubuntu 22 systemd 249
interface-scoping bug. That target collides with the daemon's
mesh-interface filter on Linux: when an IPv6 packet's destination
belongs to a non-loopback interface, the kernel attributes the packet
to that interface in IPV6_PKTINFO (ipi6_ifindex == fips0) even though
loopback delivery is used (tcpdump shows lo). The mesh-interface filter
sees arrival_ifindex == mesh_ifindex and silently drops every query at
trace level — invisible to operators at the default debug level.
Net effect on stock deployments: every .fips query on systemd-resolved
hosts was silently dropped.
Daemon side
-----------
- Default `dns.bind_addr` changes from "::" to "::1" (IPv6 loopback
only). The mesh-interface filter is then defanged on the default
path because loopback isn't reachable from mesh peers. The filter
remains in place defensively for operators who explicitly bind "::"
to expose a mesh-reachable responder.
fips-dns-setup backend unification
----------------------------------
- New `try_global_drop_in` backend writes
/etc/systemd/resolved.conf.d/fips.conf with DNS=[::1]:5354 and
Domains=~fips. Inserted ahead of `try_resolvectl` in the dispatch
chain. The standard loopback path has no interface scoping, so
ipi6_ifindex reports lo and the filter passes.
- All other backends now target [::1]:5354 to match the daemon's
default IPv6-loopback bind:
- try_dns_delegate writes DNS=[::1]:5354
- try_dnsmasq writes server=/fips/::1#5354
- try_nm_dnsmasq writes server=/fips/::1#5354
- Fixed dns-delegate file path: was /etc/systemd/dns-delegate/, must
be /etc/systemd/dns-delegate.d/ (with .d suffix). systemd-resolved
silently ignored the previous path.
- fips-dns-teardown handles the new global-drop-in backend in cleanup.
- The legacy resolvectl per-link backend stays as a fallback,
documented to require careful daemon bind_addr coordination.
fips-gateway upstream pairing
-----------------------------
- gateway.dns.upstream default changes from 127.0.0.1:5354 to
[::1]:5354 to match the daemon's default bind. Linux IPv6 sockets
bound to explicit ::1 do not accept v4-mapped traffic, so the old
default would have caused the gateway's startup DNS reachability
probe to time out and systemd to restart-loop the service.
- Operators who set a non-default daemon `dns.bind_addr` must also
set `gateway.dns.upstream` to match — documented inline.
Documentation
-------------
- packaging/common/fips.yaml and packaging/openwrt-ipk fips.yaml
examples updated; rationale for the bind_addr choice and the
daemon/gateway pairing recorded inline.
Test coverage
-------------
- testing/dns-resolver/test.sh: real-fipsd end-to-end scenario added.
Builds fipsd in a Debian 12 builder image (cached), runs the daemon
with a real TUN in a privileged container, configures DNS via the
setup script, and asserts `dig @127.0.0.53 AAAA <npub>.fips` returns
AAAA. Refactored as a parameterized helper running across Debian
12/13 and Ubuntu 22/24/26 (5 e2e scenarios). Backend-aware
assertions: on systemd >= 258 the expected backend is dns-delegate;
on older systemd it's global-drop-in. Strict content checks fail CI
on any [::1]:5354 drift. fips-gateway also exercised in the
debian12 scenario to lock the gateway-upstream pairing. Renamed all
"fipsd" references to "fips" (project convention).
- testing/deb-install/ (new harness): builds the actual .deb via
cargo-deb in a Debian 12 builder image (cached), installs via apt
across each target distro, verifies maintainer scripts, conffile
placement, binary placement, and end-to-end .fips resolution after
start. Also exercises fips-gateway against the installed daemon to
verify the gateway/daemon default pairing on a real .deb path.
- This is the test layer that was missing — the previous harness only
verified config files were written, never that queries reached the
daemon.
Verified: dns-resolver 78/78 assertions, deb-install 55/55 assertions
across all 5 distros (debian:12, debian:trixie, ubuntu:22.04,
ubuntu:24.04, ubuntu:26.04).
Replace the resolvectl-only fips-dns.service with a detection script
that configures whichever DNS resolver is available:
1. systemd dns-delegate (systemd >= 258, declarative drop-in)
2. systemd-resolved via resolvectl (most systemd distros)
3. dnsmasq (standalone)
4. NetworkManager with dnsmasq plugin
5. Warning with manual instructions if none found
Service reloads are non-fatal — config is written and the backend
is recorded even if the reload fails, preventing state file cleanup
issues under set -e.
Teardown reads the recorded backend from /run/fips/dns-backend and
reverses the configuration, or cleans up all possible backends if
the state file is missing.
Includes a Docker-based test harness (testing/dns-resolver/test.sh)
covering all five backends across Debian 12, Debian 13, Fedora,
and bare systems.
Fixes#52.
