mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Johnathan Corgan
1ce3f578e4
The node-synthetic group is an allow-list of test modules, and an allow-list goes stale the day the next socket test lands: the new module runs beside the others and the first sign is a dropped handshake on a loaded runner, far from the cause. testing/check-node-test-groups.py recomputes the membership. It flags every function under src/node/tests/ (and in src/testutil.rs) that uses a socket primitive or calls a flagged function, to a fixpoint, so modules that reach a socket only through a helper are caught. Module names come from mod.rs's declarations. It then evaluates both profiles' overrides in order, as nextest does, and fails when a test in a flagged module is not in node-synthetic or a lock_large_network_test() holder is not in node-large. It also checks the nextest pin in the workflows matches the config. It exits 2 rather than pass when it finds no test modules, no flagged module, no lock holder, or a filter it cannot evaluate, so a moved directory or a reformatted filter cannot pass vacuously. Uses that open no socket are exempted one function and primitive at a time, each with its reason, in testing/node-test-groups/exemptions.toml. The fixtures beside it cover each way a test reaches a socket and each check the guard makes; every case names the output it expects, so a case that fails for the wrong reason fails the runner. Both run from local CI's guards and from the ci-parity job.
2.8 KiB
2.8 KiB
testing/node-test-groups
Data and fixture tests for testing/check-node-test-groups.py, the guard that
keeps the node test groups in .config/nextest.toml in step with the source.
nextest runs each test in its own process, so it keeps tests apart with test groups rather than locks. Two groups run their tests one at a time:
node-largeholds the tests that takelock_large_network_test(), the 100-node tests.node-syntheticholds the test modules undersrc/node/tests/that open a real socket, directly or through a helper.
Every other node test runs in parallel. The guard recomputes both memberships from the source and fails when a socket module or a lock holder is outside its group, so a new socket test cannot land outside the group unnoticed. Its docstring states what counts as a socket use, how calls are resolved, the two filter shapes it accepts, and its exit codes (0 pass, 1 finding, 2 cannot check).
Files
exemptions.toml: primitive uses that open no socket, one entry per function and primitive, each with its reason. The guard reads it from here by default. One file servesmaint,masterandnext, so an entry naming a function a branch lacks is reported as a note, not a failure.test.sh: the fixture runner. It runs from local CI's guards (run_node_test_groupsintesting/ci-local.sh) and from the ci-parity job in.github/workflows/ci.yml; the two are kept in step by hand.base.fixture: a small tree that must pass. It is one file holding several: after its header, each=== <path>line starts a file at that path. Its modules cover each way a test can reach a socket (directly, through an aliased import, two helpers away, through asrc/testutil.rstype, through a#[path]module, through amod.rshelper that opens a connected UDP socket, through the rx loop's shutdown variant) and each way text only looks like one (comments, strings, raw strings, aVec's.drain().*.case: one case each, in the same format. The runner writes a case's files over a copy of the base tree. The header's#lines say what the case breaks;expect exit Nis the status the guard must return, and eachexpect <text>orexpect-absent <text>is a string its output must or must not contain, so a case that fails for the wrong reason fails the runner.
Rules for changing it
- A new way for a test to reach a socket gets a primitive in the guard, a
module in
base.fixturethat uses it, and a case that leaves that module out of the group. - Each check the guard makes is shown to fire by a case. A change that removes a check, or makes one pass on input it cannot read, should turn a case red; if no case turns red, add one.
- The fixture trees are never compiled. They need only the text the guard reads, written as real code would be.