mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Johnathan Corgan
9ebc80c395
churn-mixed-10 failed its baseline on GitHub with seven distinct roots against a ceiling of six. The final snapshot was taken two to four seconds after teardown restarted the nodes the churn had stopped, and on GitHub's runners the churn stops the pinned root, n01, in recent runs. A restored root is its own root until the mesh re-elects it, so the snapshot read a tree still re-converging: across recent GitHub runs it saw two or three roots in most, where a settled read sees one. Teardown now waits, before the final snapshot, until every node answers and three tree reads taken five seconds apart agree on each node's root and parent, bounded at ninety seconds. Running out is logged and is not a failure in itself; the snapshot is taken and the assertions judge it, so a node that never comes back is still reported absent and a mesh that collapsed still fails its floor. A read that any node did not answer never counts toward agreement, so the settle also covers what the control-socket wait did, and that wait is removed rather than kept as a second wait in front of it. The churn-mixed calibration comment claimed that seed 42 gives an identical chaos schedule. It does not: events are drawn as the run goes, so host timing can move them. The comment now says so, and that its numbers were measured on the unsettled snapshot and need re-measuring before any floor is tightened. The floors themselves are unchanged.
145 lines
5.0 KiB
YAML
145 lines
5.0 KiB
YAML
# Parameterized churn test with mixed transports (UDP, Ethernet, TCP)
|
|
#
|
|
# Default: 20 nodes. Override num_nodes with --nodes flag in chaos.sh,
|
|
# or copy this file and adjust topology.num_nodes directly.
|
|
#
|
|
# Tests spanning tree convergence, link flaps, node churn, and traffic
|
|
# across heterogeneous transports at configurable scale.
|
|
|
|
scenario:
|
|
name: "churn-mixed"
|
|
seed: 42
|
|
duration_secs: 600
|
|
|
|
topology:
|
|
num_nodes: 20
|
|
algorithm: erdos_renyi
|
|
params:
|
|
p: 0.3
|
|
ensure_connected: true
|
|
subnet: "172.20.0.0/16"
|
|
ip_start: 10
|
|
transport_mix:
|
|
udp: 0.6
|
|
ethernet: 0.2
|
|
tcp: 0.2
|
|
|
|
netem:
|
|
enabled: true
|
|
default_policy:
|
|
delay_ms: { min: 5, max: 50 }
|
|
jitter_ms: { min: 1, max: 10 }
|
|
loss_pct: { min: 0, max: 2 }
|
|
mutation:
|
|
interval_secs: { min: 20, max: 45 }
|
|
fraction: 0.3
|
|
policies:
|
|
normal:
|
|
delay_ms: [5, 20]
|
|
loss_pct: [0, 1]
|
|
degraded:
|
|
delay_ms: [50, 100]
|
|
jitter_ms: [10, 30]
|
|
loss_pct: [3, 8]
|
|
|
|
link_flaps:
|
|
enabled: true
|
|
interval_secs: { min: 30, max: 60 }
|
|
max_down_links: 3
|
|
down_duration_secs: { min: 10, max: 30 }
|
|
protect_connectivity: true
|
|
|
|
traffic:
|
|
enabled: true
|
|
max_concurrent: 10
|
|
interval_secs: { min: 0, max: 30 }
|
|
duration_secs: { min: 5, max: 90 }
|
|
parallel_streams: 4
|
|
|
|
node_churn:
|
|
enabled: true
|
|
interval_secs: { min: 60, max: 90 }
|
|
max_down_nodes: 5
|
|
down_duration_secs: { min: 30, max: 90 }
|
|
protect_connectivity: false
|
|
|
|
bandwidth:
|
|
enabled: true
|
|
tiers_mbps: [1, 10, 100, 1000]
|
|
|
|
# Baseline: the mesh came up, agreed on a root, and took parents. This
|
|
# asserts nothing about which nodes survive the churn; it exists so that
|
|
# a run in which the mesh never formed cannot report success, which
|
|
# until now it could, because this scenario carried no assertions at
|
|
# all.
|
|
#
|
|
# This one is calibrated rather than derived, because a scenario that
|
|
# stops and starts nodes on purpose does not hold a single spanning
|
|
# tree.
|
|
#
|
|
# Calibrated 2026-07-26 against fourteen runs that recorded a baseline
|
|
# verdict, read from their assertions.txt files, all with this file's
|
|
# seed 42. The seed fixes the random draws but not the schedule: events
|
|
# are drawn as the run goes, so host timing can change which node is
|
|
# stopped and when.
|
|
#
|
|
# distinct roots 1 2 3 4 5 -> 2 3 5 3 1 runs
|
|
# nodes parented 9 8 7 6 5 -> the exact complement, in all 14
|
|
# sessions 12 to 20, minimum 12
|
|
#
|
|
# Those runs predate the settle that now holds the final snapshot until
|
|
# every node answers and the tree has stopped moving. Their snapshot came
|
|
# a few seconds after teardown restarted the nodes the churn had stopped,
|
|
# probably the mesh's root among them: GitHub's runners stop it in recent
|
|
# runs with this seed, but which node the July runs stopped is not
|
|
# recorded. So the spread of roots above most likely measures a tree
|
|
# still re-converging rather than a settled one. That is inferred from the
|
|
# order of the changes and was not re-run.
|
|
# The numbers must be re-measured against settled runs before any floor
|
|
# here is tightened.
|
|
#
|
|
# The previous ceiling of 4 roots sat at roughly the 93rd percentile of
|
|
# that distribution: one run in fourteen exceeded it and four sat at or
|
|
# above it, so it reddened a share of runs whatever the daemon did. It
|
|
# was set from six runs whose observed maximum was 3, which is how a
|
|
# threshold one step outside a small sample ends up inside the real one.
|
|
#
|
|
# The ceiling is now 6, one step beyond the observed maximum of 5, and
|
|
# the parented floor is its complement at 4. That still fails a mesh
|
|
# that collapsed — seven or more of ten nodes islanded — which is the
|
|
# only thing this assertion was ever meant to catch. Do not read a pass
|
|
# as convergence: four of the six gating scenarios assert a floor of
|
|
# this kind, and it means the mesh formed and nobody errored.
|
|
#
|
|
# min_sessions stays at 10 against an observed minimum of 12. It has
|
|
# never fired and there is no evidence it is mis-set, but the margin is
|
|
# thin and a future failure there should be read as calibration before
|
|
# it is read as a defect.
|
|
#
|
|
# Tighten any of these only against a larger sample of settled runs,
|
|
# gathered at the invocation CI actually runs.
|
|
#
|
|
# READ THIS BEFORE RETUNING: the numbers above describe the invocation CI
|
|
# gates on, which is not this file's own defaults. ci-local runs it as
|
|
# "churn-mixed --nodes 10 --duration 120", and --nodes sed-patches
|
|
# num_nodes in a copy of this file before the scenario is loaded, so the
|
|
# gating run is a 10-node 120-second one while a bare `chaos.sh
|
|
# churn-mixed` runs the 20-node 600-second scenario written here. This is
|
|
# the only scenario CI overrides that way.
|
|
#
|
|
# The floors hold for both but are calibrated for the smaller. A bare
|
|
# 20-node run clears them easily: 20 answering, 1 root, 19 parented, 69
|
|
# sessions, measured 2026-07-23. Retuning against numbers like those would
|
|
# put them past what the 10-node gating run can reach, and CI would go red
|
|
# while a manual run stayed green.
|
|
assertions:
|
|
baseline:
|
|
min_nodes_reporting: 10
|
|
max_roots: 6
|
|
min_nodes_parented: 4
|
|
min_sessions: 10
|
|
|
|
logging:
|
|
rust_log: "debug"
|
|
output_dir: "./sim-results"
|