mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Johnathan Corgan
2ea5f88d94
chaos: bound host ip commands at 90 s and retry a timed-out one once. Veth setup runs each host-namespace ip command in a privileged helper container with a fixed 30 s timeout, and a command that timed out aborted setup. On a GitHub runner one netns move ran past 30 s while the six docker calls before it in the same pair took 0.64 s together; across 964 pairs in 241 GitHub logs the slowest pair took 2.26 s. The run aborted with exit 4 before the mesh was wired. The bound is now a named HOST_CMD_TIMEOUT_SECS of 90 s. A plain retry is not safe: the timeout kills only the docker client, and the --rm helper goes on running the command. So each helper gets a unique name, and on a timeout its docker state is logged, it is removed with docker rm -f, and the link is looked for. If the command took effect it is not repeated; otherwise it is tried once more. A retry that fails is checked the same way before it raises, since a helper docker did not know about at removal can still start and do the work first. Two timeouts still abort setup. The helper's state and where the link was found go to runner.log, so the next stall says whether docker or the kernel held it up. chaos: print NOT SCORED instead of assertion verdicts when setup aborts. A run whose setup raised after its containers started already exits 4 and records status=aborted, but teardown went on to settle, snapshot and assert against the mesh as it stood. When veth setup timed out on a GitHub runner, the log ended in FAIL min_traffic and FAIL baseline with four roots, for a ring whose links were never wired, which reads as a mesh failure. Teardown for such a run now collects and analyses the node logs, writes the metadata and removes the veth pairs, but restores, settles, snapshots and asserts nothing. In place of the verdicts it prints, logs and writes to assertions.txt "NOT SCORED: run aborted in setup" followed by the class and first line of the exception that ended setup. The line names what raised, not whose fault it was, since a node container that dies during setup ends there too. An abort after setup is unchanged. chaos: self-test the veth host-command bound against a stand-in docker. testing/chaos/veth-host-test.sh drives veth setup, and the whole runner for one case, against a stand-in docker that keeps links in a state directory and runs each host ip command in a detached process standing for the helper container, so a client killed by a timeout leaves its helper running as docker does. The stalled command is the second end's netns move, the one that stalled on GitHub. Cases: - no stall: the pair completes in the same 15 docker calls as before; - a 31 s stall at the shipped bound completes (the old 30 s bound aborts); - attempt 1 times out before its move: its helper is removed by name, never moves the link, and attempt 2 does; - the move lands, then the helper hangs past the bound: found in place and not repeated; - the helper does not exist until after docker rm -f: attempt 2 moves the link, and the late helper's move fails rather than undoing it; - the late helper moves the link while attempt 2 is starting: attempt 2's failure is checked and the link found where it belongs; - both attempts time out: the runner exits 4 and prints NOT SCORED naming the timeout, with no assertion verdicts. The other cases shorten the bound to 2 or 3 s so they do not wait out 90 s. It runs in the CI parity job, which now installs jinja2 for the runner, and in ci-local.sh's guards, and takes about a minute and a half. testing/chaos/README.md describes the bound and the test. check-image-scoping.sh exempts the stand-in's path by its one reference: it is a file named docker, which the guard's context pattern would otherwise read as the shared build directory.
veth-host-test
Support files for ../veth-host-test.sh, the self-test of the chaos runner's
host ip command handling in ../sim/veth.py and of the runner's NOT SCORED
verdict in ../sim/runner.py. Nothing here runs on its own.
fake_docker.pystands in for thedockerCLI. The test puts a wrapper nameddockerfirst onPATHthat runs it. It keeps a table of links and the namespace each is in, under a state directory named byFAKE_DOCKER_STATE.docker execruns the runner's own shell script under bash, withipandcat /sys/class/net/...replaced by functions acting on that table. A host helper (docker run --rm ... --entrypoint ip) runs in a detached process standing for the container, so killing the client leaves it running and onlydocker rm -f NAMEstops it, as with docker. A plan file (FAKE_DOCKER_PLAN) makes chosen attempts at chosen host commands stall before their effect, stall after it, or start late. Every call goes tocalls.log, every change to the links toeffects.log, and every helper's life tohelpers.log.drive.pyruns the code under test against the fake: one Ethernet pair throughVethManager.setup_all, or the whole runner through its own entry point.HOST_TIMEOUTin its environment replacesHOST_CMD_TIMEOUT_SECS, so cases that time a command out more than once do not wait out the real bound each time.
Rules for changing these:
- The fake models only what veth setup and the runner's setup and teardown
call. A new docker call in that path needs a matching answer here, or the
healthycase and the call count in../veth-host-test.shfail. - The call count in the
healthycase is the number of docker calls one pair makes when nothing stalls. A change to it is a change in what setup does on every run, and should be deliberate. - A case's stall must be shaped like the defect it stands for: a helper that outlives its client, not a client that returns early.