ArjenandJohnathan Corgan
7c8cf01905
InterfaceUnavailable existed to make absence branchable, and then stopped
being branchable at the transport boundary: every non-MTU error was flattened
into NodeError::SendFailed { reason: format!(...) }, so no caller downstream
could tell a two-second interface flap from a permanent fault. Both got the
same treatment, which for a half-built handshake means being torn down and
filed as peer misbehaviour.
The classification belongs on the error rather than at each call site, and the
question worth asking is not what went wrong but whether waiting fixes it: a
transient failure was refused by a condition the daemon is already working to
resolve, so the state built around it — a half-finished handshake, a route, a
queued packet — is worth keeping. TransportError::is_transient answers that
once, and NodeError::SendUnavailable carries the answer across the node
boundary instead of discarding it.
Deliberately narrow: only InterfaceUnavailable. Timeout and ConnectionRefused
describe a remote that did not answer, which is a statement about the peer
rather than about this node's ability to transmit, and their retry paths sit
at a different layer. The test pins that narrowness in both directions,
because the failure mode of this abstraction is someone adding a variant to
the transient list and quietly making callers hold state open for a fault that
will never clear.
No behaviour change yet. This is the plumbing half; the callers that should
act on it — route withdrawal on detach, and not counting a local interface
flap as a handshake reject — are recorded in reference/ and deferred, because
both are routing changes that want their own test story.
feat(node): withdraw a transport's peers when its interface goes away
Losing an interface withdrew nothing. The peers stayed in the registry, the
routes through them stayed selectable, and this node kept advertising
reachability it no longer had — so transit traffic was dropped in silence and
other nodes kept routing toward us for those destinations, until the liveness
reaper noticed up to link_dead_timeout_secs (30 s) later.
Measured on real hardware: a dongle detached at 07:37:19 took the node's parent
with it, and no new parent was chosen until 07:37:46. Twenty-seven seconds
routing through a link that had already gone, with four alternative peers
available the whole time. The alternatives are the point — a mesh that can
route around a dead link should not be the last to hear the link is dead.
The detach edge is both earlier and more certain than inactivity, so it is the
better trigger. reap_peers_on_transport routes through the same
route_link_dead the liveness reaper uses rather than open-coding a second
teardown: every consequence of losing a peer — sessions, path MTU release,
session indices, decrypt-worker unregistration, the link, the control machine,
tree cleanup and re-announce, bloom withdrawal — already hangs off that one
path, and a parallel one would drift from it.
Not policy-filtered. Whether an interface's absence is normal is a statement
about node *health*; it says nothing about whether the routes over it still
work. An optional interface's peers are exactly as unreachable.
PathBroken needs no new wiring. Once the peers are gone resolve_next_hop
returns None, which takes the NoRoute path — and that one already synthesises
the routing error, rate limiting included. The cure for the silent drop was to
stop having a route, not to add a second error path.
Deliberately undamped. A flapping interface cannot drive a reap storm through
here: ChurnGuard suppresses `announce` after three short-lived bindings, which
leaves `announced` false, which makes `detached()` return `retract: false` —
so no presence edge is published at all during churn. The edges this reacts to
are already rate-limited at the source, reaping an already-reaped transport is
a no-op, and a second damper would only add a way for the two to disagree.
The trade taken: immediate reaping costs a re-peer for an absence shorter than
the dead timeout that then recovers — a `wifi reload` returns in ~5 s and today
costs nothing, where this costs ~15 s of re-peering. Accepted, because
black-holing is silent, poisons other nodes' routing and needs the full timeout
to clear, where a re-peer is bounded, visible and self-healing. A grace period
remains available if that proves wrong; reference/ records its shape.
The integration assertion is the one that proves the wiring rather than the
unit: link_dead_timeout_secs is left at its 30 s default, so a withdrawal
inside 15 s can only have come from the detach edge. Verified against the
defect — with the reap disabled, that assertion fails and every other case in
the suite still passes.
fix(node): keep a half-built link when msg2 hits a transient transport
A send refused because the interface is absent or mid-rebind was treated as a
failed handshake: the link was removed, the reverse-address entry dropped, the
session index freed, the control machine torn down, the queued PromoteToActive
aborted — and the whole thing recorded as
RejectReason::Handshake(HandshakeReject::BadState).
That counter means "the remote sent something invalid". A local interface flap
is not the remote's fault, and an operator reading the rejects would conclude
it was. The initiator, meanwhile, resends msg1 into a link that no longer
exists and has to rebuild from nothing.
The binder is already working to bring the interface back, so the half-built
link is now left exactly where it is for that resend to land on. Nothing leaks
by staying: an initiator that never resends leaves a stale connection, which
`check_timeouts` reaps at `handshake_timeout_secs` like every other abandoned
handshake. Only a genuinely terminal error still tears down.
This is the first consumer of `TransportError::is_transient`, which is what
the central classification was for — before it, the distinction did not
survive as far as this call site.
The rekey msg1 send site gets the severity half only. Its teardown was already
benign: it returns before `set_rekey_state`, so the cycle simply does not
start and is retried when rekey next comes due, with nothing torn down and
nothing charged to the peer. Only the `warn!` was wrong for a local,
self-clearing condition the presence machine has already reported.
The test drives a real absent Ethernet transport rather than a stub, so the
error under test is the one production raises, from the code path that raises
it. Verified against the defect: with the transient branch disabled, the link
is destroyed and the assertion fails.
The deferral gives the epoch-mismatch restart arm in handle_msg1 a third
outcome, and its post-promote debug_assert! did not admit it. That arm's
assertion required the machine to be Established or absent; a transient msg2
failure returns before PromoteToActive and leaves it registered at
Handshaking{ReceivedMsg1}, so a debug build panics there. The assertion is
widened to name that phase exactly, which keeps it red for any other state,
and a_transient_msg2_failure_on_the_restart_path_leaves_the_fresh_leg_pending
covers the arm the existing transient test does not reach. Three comments
around those two arms claimed a send failure always removes the machine; each
now names the transient case as well. Release builds were never affected: the
tail is gated on Established, so a deferred machine simply skips it.
The route_link_dead doc comment is put back on route_link_dead. Inserting
reap_peers_on_transport between the comment and the function it described left
both blocks running together, so rustdoc attached the whole thing to the new
function and route_link_dead lost its documentation. The restored text also
names the second caller this commit adds, and generalises the sentence about
where now_ms comes from, since both callers now hoist it once per batch.
The transport-layer design's grace-period paragraph is rewritten. It argued
that no linger timer was needed because peers survive a detach untouched and
the liveness reaper is the effective bound. The detach reap makes both halves
false, so the paragraph now states the trade it actually makes: peers go at
the edge, a half-built link is still held, and a short absence that recovers
costs a re-peer, which is preferred to silent black-holing.
Changelog entries for both user-visible halves.
The insert_transport_for_test helper is no longer added here. Nothing used it
until two commits later, so cargo clippy --all-targets -- -D warnings failed
on dead code at this point in the history; it now lands with its caller.
FIPS Testing
Integration and simulation test harnesses for FIPS, using Docker containers running the full protocol stack.
Test Harnesses
static/ -- Static Docker Network
Fixed topologies with manual scripts for building, config generation, connectivity tests (ping, iperf), and network impairment (netem). Useful for deterministic debugging and validating specific topology configurations.
| Topology | Nodes | Transport | Description |
|---|---|---|---|
| mesh | 5 | UDP | Sparse mesh, 6 links, multi-hop |
| chain | 5 | UDP | Linear chain, max 4-hop paths |
| rekey | 5 | UDP | Rekey integration test topology |
tor/ -- Tor Transport Integration
End-to-end Tor transport testing with Docker containers running real Tor daemons. Requires internet access for Tor bootstrapping.
| Scenario | Description |
|---|---|
| socks5-outbound | Outbound SOCKS5 connections through Tor to clearnet peer |
| directory-mode | Inbound via HiddenServiceDir onion service (co-located) |
nat/ -- NAT Traversal Lab
Real Docker NAT traversal tests for the Nostr/STUN bootstrap path,
using router containers with iptables-based NAT, a local Nostr relay,
and a local STUN responder.
| Scenario | Description |
|---|---|
| cone | Two NATed peers establish a UDP traversal path |
| symmetric | UDP traversal fails under symmetric NAT, TCP fallback wins |
| lan | Peers on the same LAN prefer local addresses over reflexive |
chaos/ -- Stochastic Simulation
Automated network testing with configurable node counts, topology algorithms (random geometric, Erdos-Renyi, chain, explicit), and fault injection (netem mutation, link flaps, traffic generation, node churn). 10 scenarios covering general stress and node churn, discovery over sparse topologies, spanning-tree and bloom-propagation regression, transport-specific validation (UDP, TCP, Ethernet), and ECN/congestion testing. Scenarios are defined in YAML and executed via a Python harness that manages the full lifecycle: topology generation, Docker orchestration, fault scheduling, log collection, and analysis.
interop/ -- Mixed-Version Interop Harness
On-demand harness that runs an N-node full mesh from a node-spec where
each node can run a different build of the FIPS daemon, then attributes
every FMP/FSP/rekey/connectivity failure to a specific version pair
(same-version vs MIXED). Used to catch interop regressions between
builds, not as a per-commit CI gate; not part of ci-local.sh.
mesh-lab/ -- Mesh Reliability Lab
On-demand harness that runs a chosen integration suite N times under a
configurable host-pressure profile (idle / light / github-runner-
equivalent / heavy via stress-ng), per-container netem impairment,
and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
mechanism-match summary across the run. Used for statistical reliability
characterization of known flake classes under calibrated stress, not as
a per-commit gate; not part of ci-local.sh.
sidecar/ -- Network Sidecar Isolation
FIPS running as a sidecar container that owns the network namespace of
a companion application container, with iptables/ip6tables rules
confining the app to the mesh. scripts/test-sidecar.sh boots a
three-node chain of such pairs and asserts both connectivity and
isolation.
firewall/ -- nftables Baseline
End-to-end exercise of the production fips0 nftables baseline at
packaging/common/fips.nft, covering the default-deny, conntrack and
drop-in semantics.
iface-binding/ -- Dynamic Interface Binding
Two nodes whose only transports are interface-bound, started before the
interface they name exists. Asserts the boot race (the daemon comes up
Degraded and binds when the interface appears, with no restart), the flap
(down/up in both directions), destroy-and-recreate, that an optional
interface's absence never moves node health, and that absence is logged once
on the edge rather than once per retry.
acl-allowlist/ -- Peer ACL Enforcement
Six nodes with per-node allowlist files mounted at the runtime ACL paths, exercising insiders, outsiders and allowed remotes at once to check which peer pairs are admitted and which are rejected.
native-api/ -- Native Datagram API
Checks the experimental native datagram API: a client process opens a flow to a remote pubkey over a Unix socket, receives a file descriptor, and exchanges datagrams on it with no TUN device and no IPv6 emulation.
medium-change/ -- Transport-Medium Change
A multi-homed node whose default route moves between two live access paths
while mesh traffic is in flight, with the far peer reachable only through a
router so the path to it actually follows that default route. Asserts the
peering survives without a re-handshake (link_id and authenticated_at_ms
unchanged) and that the far side re-pins to the new source address.
Includes a negative control that runs the same move with
node.netmon.enabled: false and requires the outage, so a topology that
has stopped exercising the bug fails rather than passing quietly.
dns-resolver/ -- fips-dns-setup Backends
Runs fips-dns-setup against each supported Linux resolver backend in
systemd containers, verifying backend detection, generated config and
teardown, plus an end-to-end scenario that resolves a .fips name
through the configured backend.
deb-install/ -- Debian Package Install
Installs the built .deb in privileged systemd containers for each
target distro and verifies unit enablement, conffile placement and
end-to-end .fips resolution as a user would meet it.
boringtun/ -- WireGuard Throughput Baseline
Two userspace WireGuard peers running Cloudflare BoringTun, measured
with iperf3, as a comparison baseline for FIPS tunnel throughput.
ble/ -- BLE L2CAP Spike
Standalone cargo project (ble_spike) that validates the bluer API
assumptions behind the BleIo trait against real adapters on two
machines. Not a Docker harness.
Running CI locally (ci-local.sh)
ci-local.sh runs the full local CI pipeline — build,
clippy, unit tests, and the integration suites (including the chaos
scenarios) — mirroring the GitHub ci.yml integration matrices. Run
./ci-local.sh --help for the full option list and --list for the
available suites. Every run starts with a parity check that verifies the
local suite set covers the same work as the GitHub matrix, per scenario for
chaos and per distro for deb-install, across every job that carries a
matrix; a divergence fails the run. GitHub
runs the same check as its own ci-parity job. --check-parity runs it
alone (see check-ci-parity.sh).
Note that ci-local.sh covers the integration suites and the glibc unit
tests. GitHub additionally runs the library tests on macOS, Windows and
musl (built for the musl target and run natively), and a --features profiling pass; the musl
leg exists because interface presence is built on getifaddrs/ifa_flags,
which musl reimplements independently, and OpenWrt is a musl target. A local
green run does not certify those four.
The Linux and musl legs also create an address-less dummy interface
(fips-probe0) and pass its name to the tests as
FIPS_TEST_ADDRLESS_IFACE. That is the one assumption the interface-binding
mechanism rests on that no ordinary test can reach: loopback has addresses, so
probing it asks whether getifaddrs works rather than whether it reports an
interface that has none — which is exactly what fips-mesh0 and fips-ap0
are on OpenWrt. Set the variable by hand to run the assertion locally against
an interface you have created; leave it unset and the assertion does not run.
Per-run isolation and the FIPS_CI_RUN_ID override
Every invocation derives a run id and scopes all of its Docker resources to it, so two simultaneous runs on the same host (for example, one per git worktree, or an operator testing by hand while CI is in flight) never collide:
- Compose projects are named
fipsci_<run-id>_<suite>, so container, network, and volume names are all prefixed per run. - Build images are tagged
fips-test:<run-id>andfips-test-app:<run-id>, exported asFIPS_TEST_IMAGE/FIPS_TEST_APP_IMAGE, and every compose file and suite script reads those. The run does not writefips-test:latestat all: a bridge back to that shared mutable name would let a consumer that had been missed keep working while resolving whichever concurrent run wrote the tag last.:lateststays the hand-build name, produced bytesting/scripts/build.sh, and remains the default every consumer falls back to when the variables are unset. - The build context is a per-run copy at
testing/docker-<run-id>/, exported asFIPS_BUILD_CONTEXT. It is absolute because compose resolves a relative build context against the compose file's own directory rather than the working directory.testing/docker/is the hand-run context and a CI run does not write to it. Without this, two runs race on the contents of one directory and either can build a correctly-per-run-tagged image from the other's binaries. - Each parallel chaos child gets a unique, non-overlapping
/24in10.30.x(via the sim--subnetoverride).10.30.xsits outside Docker's default address pool and the fixed-subnet suites'172.xranges, so neither a sibling chaos child nor an auto-assigned network can swallow a pinned subnet.
By default the run id is <short-git-sha>-<random> — the SHA portion
records what code a container is testing, the random suffix keeps
simultaneous runs of the same SHA disjoint. Override it for a
reproducible, attach-by-name debug session:
FIPS_CI_RUN_ID=mydebug ./ci-local.sh --only static-mesh
# containers are named fipsci_mydebug_static_fips-node-a, etc.
Preemption-safety and exit codes
ci-local.sh is safe to cancel mid-run. A signal trap tears down every
compose project the run started (not just the current suite) and reaps
any in-flight parallel chaos children, bounded by a timeout so a stuck
compose down cannot wedge the trap. Exit codes distinguish a cancelled
run from a failing one:
| Code | Meaning |
|---|---|
0 |
all stages passed |
1 |
one or more stages failed |
130 |
interrupted by SIGINT — cancelled, not a failure |
143 |
terminated by SIGTERM — cancelled, not a failure |
A preempting CI worker (the push-triggered, CI-gated build pipeline that
kills an in-flight run when a newer same-branch tip arrives) maps
130/143 → cancelled (discard, do not record a failing commit), 0
→ green, any other non-zero → red.
Cleaning up leftover resources
Every CI-created container, network, and volume carries the label
com.corganlabs.fips-ci=1. If a run is hard-killed (SIGKILL, OOM, crash)
and leaves resources behind, reap them with:
./ci-local.sh --reap # or: ./ci-cleanup.sh
ci-cleanup.sh force-removes everything bearing the CI
label or a fipsci_ compose-project prefix; it is safe to run when there
is nothing to reap and safe to run repeatedly. Pass --project-prefix to
scope the sweep to a single run.
It also removes the chaos simulation's leftover host-namespace veth
interfaces (vh…a/vh…b), the one resource it touches that is neither a
docker object nor labelled — a host interface can carry neither a label
nor a compose project, so it is matched by name shape alone. That makes
the reach here asymmetric with everything above, and worth stating
plainly:
- A bare
chaos.shrun's containers survive a broad reap. Its compose project is notfipsci_, and the simulation labels only the network, not the services. - A bare
chaos.shrun's veth interfaces do not. An unscoped reap deletes them while they are in use, severing the Ethernet links of a live simulation and leaving its containers running.
So do not run a broad --reap while a bare simulation is up. Scope the
interface sweep with --veth-suffixes (which is what ci-local.sh's own
teardown passes) or wait for the simulation to finish. --project-prefix
does not help here: it scopes only the compose-project sweep.