mirror of https://github.com/jmcorgan/fips.git synced 2026-10-30 07:37:36 +00:00

Files

T

Johnathan Corgan 2ea5f88d94 chaos: bound and retry host ip commands; mark setup aborts NOT SCORED

chaos: bound host ip commands at 90 s and retry a timed-out one once.

Veth setup runs each host-namespace ip command in a privileged helper
container with a fixed 30 s timeout, and a command that timed out aborted
setup. On a GitHub runner one netns move ran past 30 s while the six docker
calls before it in the same pair took 0.64 s together; across 964 pairs in
241 GitHub logs the slowest pair took 2.26 s. The run aborted with exit 4
before the mesh was wired.

The bound is now a named HOST_CMD_TIMEOUT_SECS of 90 s. A plain retry is
not safe: the timeout kills only the docker client, and the --rm helper goes
on running the command. So each helper gets a unique name, and on a timeout
its docker state is logged, it is removed with docker rm -f, and the link is
looked for. If the command took effect it is not repeated; otherwise it is
tried once more. A retry that fails is checked the same way before it
raises, since a helper docker did not know about at removal can still start
and do the work first. Two timeouts still abort setup.

The helper's state and where the link was found go to runner.log, so the
next stall says whether docker or the kernel held it up.

chaos: print NOT SCORED instead of assertion verdicts when setup aborts.

A run whose setup raised after its containers started already exits 4 and
records status=aborted, but teardown went on to settle, snapshot and assert
against the mesh as it stood. When veth setup timed out on a GitHub runner,
the log ended in FAIL min_traffic and FAIL baseline with four roots, for a
ring whose links were never wired, which reads as a mesh failure.

Teardown for such a run now collects and analyses the node logs, writes the
metadata and removes the veth pairs, but restores, settles, snapshots and
asserts nothing. In place of the verdicts it prints, logs and writes to
assertions.txt "NOT SCORED: run aborted in setup" followed by the class and
first line of the exception that ended setup. The line names what raised,
not whose fault it was, since a node container that dies during setup ends
there too. An abort after setup is unchanged.

chaos: self-test the veth host-command bound against a stand-in docker.

testing/chaos/veth-host-test.sh drives veth setup, and the whole runner for
one case, against a stand-in docker that keeps links in a state directory and
runs each host ip command in a detached process standing for the helper
container, so a client killed by a timeout leaves its helper running as
docker does. The stalled command is the second end's netns move, the one
that stalled on GitHub. Cases:

- no stall: the pair completes in the same 15 docker calls as before;
- a 31 s stall at the shipped bound completes (the old 30 s bound aborts);
- attempt 1 times out before its move: its helper is removed by name, never
  moves the link, and attempt 2 does;
- the move lands, then the helper hangs past the bound: found in place and
  not repeated;
- the helper does not exist until after docker rm -f: attempt 2 moves the
  link, and the late helper's move fails rather than undoing it;
- the late helper moves the link while attempt 2 is starting: attempt 2's
  failure is checked and the link found where it belongs;
- both attempts time out: the runner exits 4 and prints NOT SCORED naming
  the timeout, with no assertion verdicts.

The other cases shorten the bound to 2 or 3 s so they do not wait out 90 s.
It runs in the CI parity job, which now installs jinja2 for the runner, and
in ci-local.sh's guards, and takes about a minute and a half.
testing/chaos/README.md describes the bound and the test.
check-image-scoping.sh exempts the stand-in's path by its one reference: it
is a file named docker, which the guard's context pattern would otherwise
read as the shared build directory.

2026-10-22 07:20:26 +00:00

1.9 KiB

Raw Permalink Blame History

veth-host-test

Support files for ../veth-host-test.sh, the self-test of the chaos runner's host ip command handling in ../sim/veth.py and of the runner's NOT SCORED verdict in ../sim/runner.py. Nothing here runs on its own.

fake_docker.py stands in for the docker CLI. The test puts a wrapper named docker first on PATH that runs it. It keeps a table of links and the namespace each is in, under a state directory named by FAKE_DOCKER_STATE. docker exec runs the runner's own shell script under bash, with ip and cat /sys/class/net/... replaced by functions acting on that table. A host helper (docker run --rm ... --entrypoint ip) runs in a detached process standing for the container, so killing the client leaves it running and only docker rm -f NAME stops it, as with docker. A plan file (FAKE_DOCKER_PLAN) makes chosen attempts at chosen host commands stall before their effect, stall after it, or start late. Every call goes to calls.log, every change to the links to effects.log, and every helper's life to helpers.log.
drive.py runs the code under test against the fake: one Ethernet pair through VethManager.setup_all, or the whole runner through its own entry point. HOST_TIMEOUT in its environment replaces HOST_CMD_TIMEOUT_SECS, so cases that time a command out more than once do not wait out the real bound each time.

Rules for changing these:

The fake models only what veth setup and the runner's setup and teardown call. A new docker call in that path needs a matching answer here, or the healthy case and the call count in ../veth-host-test.sh fail.
The call count in the healthy case is the number of docker calls one pair makes when nothing stalls. A change to it is a change in what setup does on every run, and should be deliberate.
A case's stall must be shaped like the defect it stands for: a helper that outlives its client, not a client that returns early.

1.9 KiB Raw Permalink Blame History

veth-host-test

1.9 KiB

Raw Permalink Blame History