fr34aky
eaca285ad2
The suite pinned three fixed /24s — 172.31.60, .61 and .62. Compose
project names are unique per run, so two concurrent runs got distinct
container and network *names*, but the address pools are constants and
both runs asked for the same ones. Whichever created a network first
won; the other died at topology start with
failed to create network ..._mc-far: invalid pool request:
Pool overlaps with other one on this address space
having tested nothing. Pushing maint, master and next within a second of
each other is enough to hit it, and the branch that loses looks broken
when it is fine.
The three `MC_*_PREFIX` overrides existed from the start but nothing
ever set them, so the defaults were the only values ever used.
This does what the nat suite already does. A free /24 per network is
claimed under 10.42.0.0/16 before the lab starts, with docker's own
`network create` as the atomic arbiter of who owns what — the run-id
derived offset is deliberately not used here for the same reason it was
rejected there: it makes a collision unlikely rather than impossible,
and a collision is the failure being removed. Both CI labels are stamped
so ci-cleanup.sh's label sweep recovers the networks when a run is
SIGKILLed, which no inline removal can cover.
Since the claim creates the networks, compose has to attach rather than
create, so `docker-compose.external-net.yml` declares the three
external, applied through a new `MC_EXTRA_COMPOSE` hook. Nothing else
sets it: the GitHub matrix runs one job per runner and invokes
`test.sh` directly, and a bare `docker compose up` is a single lab, so
both keep the fixed defaults and the addresses in the README stay
literal. The hook appends to the whole COMPOSE array rather than to the
`up` alone, so teardown addresses the same project — a `down` without
the overlay would not know the networks are external.
The claim lives in ci-local.sh rather than the suite script for the
reason the nat comment gives: the workflow invokes the script directly
and tears down with the base file only, and does not want a claim.
Release does a `compose down` before removing the networks. That order
is load-bearing rather than tidy: `docker network rm` silently no-ops on
a network that still has endpoints attached and reports success, so
without the `down` the removal fails exactly on the path it exists for.
A network left behind would not merely leak — the next invocation in the
run would hit `network with name ... already exists`, which is not a
pool overlap, so the allocator correctly refuses to advance and fails.
Also gives the suite its own compose project. It set none, so it
inherited whichever COMPOSE_PROJECT_NAME the previous suite exported —
which is why the failure named a medium-change network under the nat
project, `fipsci_<runid>_nat_mc-far`. That is not what caused the
overlap, but filing one suite's resources under another's project is a
teardown hazard: `down --remove-orphans` on either would consider the
other's containers orphans.
The /24 claim loop is now shared rather than copied a third time.
`ci_claim_nat_net` keeps its name, its `[nat]` log tag and its exported
prefix, and becomes a two-line caller.
Verified rather than assumed. The lab runs on the claimed prefixes, not
just alongside them: `node-b re-pinned to 10.42.1.10:2121`. With the
first three candidates occupied by squatter networks — the collision
path itself — the allocator advances to 10.42.3/4/5 and the suite passes
on those. Run standalone with no overlay it still renders 172.31.6x and
passes 8/8, so the workflow path is untouched. `nat-cone` passes and
still claims 10.41.0/1, so the shared loop did not disturb it. Networks
are gone after each run. `ci-local.sh --only medium-change` is 17/17.
Not exercised: the partial-claim rollback, which needs a /16 exhausted
part-way to reach. It mirrors ci_claim_nat_networks' shape.
Transport-Medium Change Lab
A node whose network attachment moves under it — WLAN to LAN, Wi-Fi to cellular — while its peers stay where they are.
node-a ──┬── mc-primary ───┐
│ ├── router ── mc-far ── node-b
└── mc-secondary ─┘
node-a is multi-homed with two equally usable paths to the router. node-b
sits beyond the router and is reachable only through it. That last part
carries the whole design: because node-b is off-link, the route to it
follows node-a's default route, which is what the suite moves. Put
node-b on a bridge shared with node-a and the directly-connected route
wins, the source address never changes, and there is nothing left to test.
Both of node-a's interfaces stay up throughout. Nothing is unplugged.
The only thing that changes is which of them the default route points at,
which is what makes this a medium change rather than a link failure — and
which is why link-state watching alone does not see it.
Running
./testing/medium-change/scripts/test.sh
# or
./testing/ci-local.sh --only medium-change
What it asserts
Traffic returning after the move is a weak signal: a peering that was torn
down by the liveness timeout and rebuilt by a re-dial also ends with traffic
flowing. The suite therefore checks continuity, from fipsctl show peers on
both nodes:
| Observation | Meaning |
|---|---|
link_id unchanged on node-a |
the link was never rebuilt |
authenticated_at_ms unchanged on node-a |
no second handshake ran |
transport_addr changed on node-b |
the far side re-pinned to the new source |
| longest ping gap within budget | the data plane genuinely carried through |
The third is what stops the first two from passing vacuously on a topology where nothing actually moved.
Phases 1 and 2 move the route in each direction, since the two are not symmetric — one direction leaves the old interface holding an address the routing table has abandoned, the other returns to it.
The negative control
Phase 3 repeats the move with node.netmon.enabled: false and requires
the outage. If traffic survives with detection off, this topology is not
exercising the code path and every assertion above is vacuous — so the suite
fails rather than passing quietly.
This is deliberate. A regression test that has never been seen to fail is a claim, not a test, and the claim is cheap to make and expensive to trust.
Knobs
| Variable | Default | Meaning |
|---|---|---|
MC_MAX_GAP_SECS |
5 |
longest tolerated break in traffic across a move |
MC_CONTROL_DARK_SECS |
12 |
how long the control must stay dark |
MC_PRIMARY_PREFIX |
172.31.60 |
first access path /24 |
MC_SECONDARY_PREFIX |
172.31.61 |
second access path /24 |
MC_FAR_PREFIX |
172.31.62 |
far segment /24 |
MC_EXTRA_COMPOSE |
unset | extra compose overlays, colon-separated |
The gap budget sits far below the 30 s liveness timeout on purpose: a pass must mean the move was absorbed, not that the reaper was quick.
Running two of these at once
The three prefixes above are fixed defaults, and two runs that both take them
do not both get them: docker refuses the second with Pool overlaps with other one on this address space, and the suite fails at topology start having tested
nothing. The compose project name is unique per run, so the containers and the
networks get distinct names — it is only the address pools that are shared.
testing/ci-local.sh avoids that by claiming a free /24 per network under
10.42.0.0/16 before it starts the lab, letting docker's own network create
be the arbiter of who owns what, and pointing compose at the result with
docker-compose.external-net.yml via MC_EXTRA_COMPOSE. It exports the
claimed prefixes as the three MC_*_PREFIX variables, so everything the suite
renders from them follows.
Nothing else applies that overlay. The GitHub matrix runs one job per runner and the README invocation above is a single lab, so both keep the fixed defaults and the addresses in this document stay literal. If you want to run two by hand on one host, set the three prefixes yourself.