mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Johnathan Corgan
2ea5f88d94
chaos: bound host ip commands at 90 s and retry a timed-out one once. Veth setup runs each host-namespace ip command in a privileged helper container with a fixed 30 s timeout, and a command that timed out aborted setup. On a GitHub runner one netns move ran past 30 s while the six docker calls before it in the same pair took 0.64 s together; across 964 pairs in 241 GitHub logs the slowest pair took 2.26 s. The run aborted with exit 4 before the mesh was wired. The bound is now a named HOST_CMD_TIMEOUT_SECS of 90 s. A plain retry is not safe: the timeout kills only the docker client, and the --rm helper goes on running the command. So each helper gets a unique name, and on a timeout its docker state is logged, it is removed with docker rm -f, and the link is looked for. If the command took effect it is not repeated; otherwise it is tried once more. A retry that fails is checked the same way before it raises, since a helper docker did not know about at removal can still start and do the work first. Two timeouts still abort setup. The helper's state and where the link was found go to runner.log, so the next stall says whether docker or the kernel held it up. chaos: print NOT SCORED instead of assertion verdicts when setup aborts. A run whose setup raised after its containers started already exits 4 and records status=aborted, but teardown went on to settle, snapshot and assert against the mesh as it stood. When veth setup timed out on a GitHub runner, the log ended in FAIL min_traffic and FAIL baseline with four roots, for a ring whose links were never wired, which reads as a mesh failure. Teardown for such a run now collects and analyses the node logs, writes the metadata and removes the veth pairs, but restores, settles, snapshots and asserts nothing. In place of the verdicts it prints, logs and writes to assertions.txt "NOT SCORED: run aborted in setup" followed by the class and first line of the exception that ended setup. The line names what raised, not whose fault it was, since a node container that dies during setup ends there too. An abort after setup is unchanged. chaos: self-test the veth host-command bound against a stand-in docker. testing/chaos/veth-host-test.sh drives veth setup, and the whole runner for one case, against a stand-in docker that keeps links in a state directory and runs each host ip command in a detached process standing for the helper container, so a client killed by a timeout leaves its helper running as docker does. The stalled command is the second end's netns move, the one that stalled on GitHub. Cases: - no stall: the pair completes in the same 15 docker calls as before; - a 31 s stall at the shipped bound completes (the old 30 s bound aborts); - attempt 1 times out before its move: its helper is removed by name, never moves the link, and attempt 2 does; - the move lands, then the helper hangs past the bound: found in place and not repeated; - the helper does not exist until after docker rm -f: attempt 2 moves the link, and the late helper's move fails rather than undoing it; - the late helper moves the link while attempt 2 is starting: attempt 2's failure is checked and the link found where it belongs; - both attempts time out: the runner exits 4 and prints NOT SCORED naming the timeout, with no assertion verdicts. The other cases shorten the bound to 2 or 3 s so they do not wait out 90 s. It runs in the CI parity job, which now installs jinja2 for the runner, and in ci-local.sh's guards, and takes about a minute and a half. testing/chaos/README.md describes the bound and the test. check-image-scoping.sh exempts the stand-in's path by its one reference: it is a file named docker, which the guard's context pattern would otherwise read as the shared build directory.
108 lines
3.1 KiB
Python
108 lines
3.1 KiB
Python
#!/usr/bin/env python3
|
|
"""Drive the chaos runner's veth setup against the stand-in docker.
|
|
|
|
Two modes, both run from veth-host-test.sh with the fake docker first on PATH:
|
|
|
|
pair SIM_PARENT set up one Ethernet pair, n01--n02, through
|
|
VethManager.setup_all, and print RESULT ok or
|
|
RESULT VethSetupError with the elapsed time
|
|
runner SIM_PARENT SCENARIO
|
|
run the whole runner on SCENARIO, as `python -m sim`
|
|
does, and print RESULT exit=N
|
|
|
|
SIM_PARENT is the directory holding the `sim` package under test. When
|
|
HOST_TIMEOUT is set in the environment it replaces the module's bound on a
|
|
host command, so the cases that time a command out twice need not wait for
|
|
the real bound twice. A tree with no such bound ignores it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import sys
|
|
import time
|
|
import types
|
|
|
|
|
|
class PairTopology:
|
|
"""The parts of SimTopology that VethManager reads, for one pair."""
|
|
|
|
veth_token = ""
|
|
|
|
def __init__(self):
|
|
self.nodes = {
|
|
n: types.SimpleNamespace(ethernet_macs={}) for n in ("n01", "n02")
|
|
}
|
|
|
|
def ethernet_edges(self):
|
|
"""The one Ethernet edge."""
|
|
return [("n01", "n02")]
|
|
|
|
def container_name(self, node_id):
|
|
"""Container name for a node, as the runner renders it."""
|
|
return f"fips-node-{node_id}"
|
|
|
|
def veth_host_name(self, a, b, end):
|
|
"""Temporary host-side name for one end of a pair."""
|
|
return f"vh{a[1:]}{b[1:]}{end}"
|
|
|
|
|
|
def apply_timeout(veth) -> None:
|
|
"""Shorten the host-command bound when the case asks for it."""
|
|
value = os.environ.get("HOST_TIMEOUT")
|
|
if value:
|
|
veth.HOST_CMD_TIMEOUT_SECS = float(value)
|
|
|
|
|
|
def run_pair() -> int:
|
|
"""Set up the pair and report how it ended."""
|
|
from sim import veth
|
|
|
|
apply_timeout(veth)
|
|
mgr = veth.VethManager(PairTopology())
|
|
start = time.monotonic()
|
|
try:
|
|
mgr.setup_all()
|
|
except veth.VethSetupError as e:
|
|
print(f"RESULT VethSetupError after {time.monotonic() - start:.1f}s: {e}")
|
|
return 0
|
|
print(f"RESULT ok after {time.monotonic() - start:.1f}s pairs={len(mgr._host_pairs)}")
|
|
return 0
|
|
|
|
|
|
def run_runner(scenario: str) -> int:
|
|
"""Run the runner's own entry point and report its exit status."""
|
|
from sim import veth
|
|
from sim.__main__ import main
|
|
|
|
apply_timeout(veth)
|
|
sys.argv = ["sim", scenario]
|
|
try:
|
|
main()
|
|
code = 0
|
|
except SystemExit as e:
|
|
code = e.code if isinstance(e.code, int) else 1
|
|
print(f"RESULT exit={code}", flush=True)
|
|
return 0
|
|
|
|
|
|
def entry() -> int:
|
|
"""Parse the mode and run it."""
|
|
mode, sim_parent = sys.argv[1], sys.argv[2]
|
|
sys.path.insert(0, sim_parent)
|
|
if mode == "pair":
|
|
logging.basicConfig(
|
|
level=logging.INFO,
|
|
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
|
)
|
|
return run_pair()
|
|
if mode == "runner":
|
|
return run_runner(sys.argv[3])
|
|
print(f"drive.py: unknown mode {mode}", file=sys.stderr)
|
|
return 2
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(entry())
|