mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Arjen
f673a18b96
A peer reachable over more than one transport keeps one Noise session and moves its traffic between transports on failure or degradation. Implements docs/design/fips-multi-path-switchover.md §4-§10 and closes the two-interface case in #143. Three inner link messages next to Heartbeat: `0x52 PathProbe` and `0x53 PathAck`, carrying a probe id, the sender's path id and a `remote_active` bit, and `0x54 PathClose`, naming the receiver's path id and a reason. A probe is an ordinary encrypted frame sent on a candidate transport; the receiver, having decrypted it against the session found by index, adds the path as `Probing`, marks it `rx_live` and answers on that same path. The prober's receipt of the ack marks the path `Live`, `tx_live`, and takes an RTT sample. No handshake, no key material, no index allocation. Old nodes drop the unknown types at debug, so a path to one stays `Probing` and never becomes eligible. The discovery gate changes shape: a live peer beaconing on a transport we hold no path to it over becomes a path candidate rather than being skipped, and the heartbeat tick probes it. The active path's first probe is small — the handshake proved it and seeded its MTU — while a standby's discovery probes are padded to the link MTU, as is one a minute on every path, so a medium that passes small frames and drops large ones never proves itself. A standby the peer never answers on is given up after eight probes; the active path never is. Detection is per path and takes each medium's own failure signal: a carrier edge, an unreachable-on-send (`ENETUNREACH`/`EHOSTUNREACH`), an interface going away, or two unanswered heartbeats on a path the peer is also silent on. Any of them marks the path `Suspect` and selection leaves it at once, because the standby is warm: heartbeats run at `node.path.active_heartbeat_ms` where either side sends and `standby_heartbeat_ms` elsewhere, both stretched by the path's own round trip so a Tor or Nym path is neither flooded nor declared dead every round trip. A peer holding one live path is not heartbeated here at all — selection has nothing to move to, and the link heartbeat keeps its liveness. Soft signals (the peer's `remote_active` flipping away, silence here while a standby hears the peer) trigger a probe, never `Suspect`: reading them as a verdict forces both sides onto one path and loops under a one-way failure. A node that loses a path tells the peer with a `PathClose` on a surviving one, so the peer moves at once rather than after its own timeout. A transport that returns inside the five-minute grace revives its dead paths as `Probing` with their RTT window and ETX intact. Selection is measured, not configured: each path scores `quality_index(etx, min_rtt)`, and traffic moves when the active path is no longer eligible (mandatory) or when a standby beats it by `switch_margin` for `switch_dwell_secs` (discretionary). Min RTT over a window rather than SRTT, because SRTT inflates under load on the path carrying traffic while an idle standby looks pristine — a ping-pong generator. A `role: backup` transport carries a peer's traffic only while no normal path is eligible, and yields outright when one becomes eligible. `fipsctl path pin` overrides both while its path is eligible. A switch re-seeds the path MTU from the new path, tightens the session MTUs and refreshes the MSS ceiling, so the first frames after a switch are not black-holed. The link record and `addr_to_link` follow the active path. The link cost the tree sees is held at its pre-switch value for the dwell, and until the two receiver reports that span the switch have arrived — the first counts every frame in flight on the old path as lost and spikes the per-report ETX for one interval, the second replaces it — so neither a short flap nor that spike ripples mesh-wide through parent selection or the next-hop order. Operator surface: `role: backup` on any transport, `node.path.*` (`switch_margin`, validated finite and at least 1.0, `switch_dwell_secs`, `min_samples`, `active_heartbeat_ms`, `standby_heartbeat_ms`), and `fipsctl path show|pin|unpin` over the `path_show`, `path_pin` and `path_unpin` control commands. Two chaos scenarios calibrate the defaults and are wired into both runners: `dual-path-flap` (a raw-Ethernet veth as the cable, the Docker bridge over UDP as the wifi) and `dual-udp-flap` (two interface-bound UDP instances). Each flaps one path under iperf and carries detectors that can fail on a switchover that did not carry traffic — a per-node ceiling on "Peer promoted to active" (a second is a re-peering), a one-second ceiling from link-down to the first switch, and a two-second ceiling on any zero-byte iperf interval run — alongside the `path_switches` band. Neither has been run to calibrate; the defaults are chosen, not derived, and the design doc says so. Refs #143
122 lines
4.3 KiB
YAML
122 lines
4.3 KiB
YAML
# One peer, two UDP paths: switchover between two interface-bound UDP instances
|
||
#
|
||
# The all-IP twin of dual-path-flap. Two nodes joined twice, both by UDP: a
|
||
# dedicated veth carrying IP (`10.222.0.0/24`), on which each node runs a
|
||
# UDP instance bound to that interface (`transports.udp.<veth>.interface`),
|
||
# and the Docker bridge on the ordinary wildcard instance. Both are static
|
||
# peer addresses on the dial owner (`udp/main` first, `udp/<veth>` second),
|
||
# so both handshakes run at startup; the first to complete makes the
|
||
# session, the second settles as a cross-connection and creates no path
|
||
# state — the dialled address is a candidate, and the probe exchange under
|
||
# the session proves it as a path at both ends.
|
||
#
|
||
# What this exercises that dual-path-flap cannot: two UDP instances as two
|
||
# paths, `udp.interface` binding on the listen socket *and* on the per-peer
|
||
# connected sockets (Linux `SO_BINDTODEVICE`), and a configured address on a
|
||
# new transport becoming a path. The flap and fail-back mechanics are the
|
||
# same: `link_flaps` sets 100% loss on the veth, the interface stays up, and
|
||
# the heartbeat echo timeout is the detector.
|
||
#
|
||
# What to read from a run: as dual-path-flap. "Path live: the peer answers
|
||
# on this transport" in both node logs is the second instance being proven
|
||
# as a path; `fipsctl path show` on either node should list both instances
|
||
# live before the first flap, and "Peer promoted to active" appears once
|
||
# per node (`max_promotions` fails the run otherwise).
|
||
#
|
||
# n01 ===udp/veth=== n02 (flapped)
|
||
# n01 ----udp/main--- n02 (never touched)
|
||
|
||
scenario:
|
||
name: "dual-udp-flap"
|
||
seed: 42
|
||
duration_secs: 180
|
||
|
||
topology:
|
||
algorithm: explicit
|
||
num_nodes: 2
|
||
default_transport: udp
|
||
params:
|
||
adjacency:
|
||
- [n01, n02, udp-veth+udp]
|
||
|
||
# The veth is mild; the bridge half carries 40 ms each way. Score is
|
||
# etx × (1 + min_rtt/100): about 1.0 on the veth against about 1.8 over the
|
||
# bridge, well past the 1.3 margin, so the pair moves onto the veth soon
|
||
# after the runner adds it and returns to it after each flap once it is
|
||
# proven again. Every flap therefore hits the *active* path, and the
|
||
# fail-back is exercised too.
|
||
netem:
|
||
enabled: true
|
||
default_policy:
|
||
delay_ms: [1, 3]
|
||
jitter_ms: [0, 1]
|
||
loss_pct: [0, 0.2]
|
||
dual_udp_policy:
|
||
delay_ms: [40, 40]
|
||
jitter_ms: [0, 1]
|
||
loss_pct: [0, 0.2]
|
||
|
||
# The mechanism. Only the veth flaps; the bridge path is the standby that
|
||
# has to be warm when it does. `protect_connectivity` is off because the
|
||
# harness counts the dual edge once and would refuse to flap the "only"
|
||
# link; the bridge half keeps the pair connected throughout.
|
||
link_flaps:
|
||
enabled: true
|
||
only_transport: udp-veth
|
||
interval_secs: {min: 35, max: 45}
|
||
max_down_links: 1
|
||
down_duration_secs: {min: 12, max: 18}
|
||
protect_connectivity: false
|
||
|
||
traffic:
|
||
enabled: true
|
||
max_concurrent: 1
|
||
interval_secs: {min: 5, max: 10}
|
||
duration_secs: {min: 20, max: 30}
|
||
parallel_streams: 2
|
||
|
||
node_churn:
|
||
enabled: false
|
||
|
||
assertions:
|
||
baseline:
|
||
min_nodes_reporting: 2
|
||
max_roots: 1
|
||
min_nodes_parented: 1
|
||
|
||
# Traffic crossed the pair while the cable was being taken away.
|
||
min_traffic:
|
||
min_sessions_ok: 2
|
||
|
||
# Three to four flaps in 180 s at the interval above. Each flap moves both
|
||
# nodes off the veth (2) and its return moves them back (2): four per
|
||
# flap, plus the initial move onto the veth (2). Floor: the first flap
|
||
# moved something. Ceiling: four flaps, both directions, both nodes, the
|
||
# initial move, and nothing else.
|
||
path_switches:
|
||
min_total: 4
|
||
max_total: 18
|
||
|
||
# The detectors that can fail on a switchover that did not carry. A
|
||
# promotion past the first is a re-peering: the session was lost to the
|
||
# link-dead reaper and rebuilt, which standby probes alone cannot show.
|
||
# The switch must land inside a second of the runner taking the link
|
||
# down. And no iperf3 session may sit at zero bytes for longer than two
|
||
# of its one-second intervals: min_traffic passes on bytes either side
|
||
# of a hole, this fails on the hole.
|
||
max_promotions:
|
||
per_node: 1
|
||
switch_latency:
|
||
max_ms: 1000
|
||
max_stall:
|
||
max_secs: 2
|
||
|
||
max_errors:
|
||
max_total: 0
|
||
|
||
logging:
|
||
# The path handler at debug: "Path suspect" and "Sent path probe" are what
|
||
# the switch latency is read from.
|
||
rust_log: "info,fips::node::handlers::path=debug,fips::node::lifecycle=debug"
|
||
output_dir: "./sim-results"
|