Files
fips/testing/medium-change/docker-compose.yml
Johnathan Corgan 992432c2db medium-change: wait for traffic before each move and measure from the move
The negative control once failed with 2 replies where it needs 5. Each phase
started a ping, slept a fixed 3 s and required 5 replies, and the control
also slept a fixed 5 s after restarting node-a, and the phases 5 s between
them. Nothing waited for the condition those sleeps stood for, so anything
that delayed the first replies failed the run, and nothing in the log showed
how close any run came.

Before each move the suite now waits, each with a deadline, for:

- node-a's connected UDP socket to node-b to be pinned to the path the
  phase starts from, read from /proc/net/udp. That socket is what a move
  strands; a move made before the tick that opens it goes out through the
  wildcard socket, which follows the route by itself. This replaces the 5 s
  sleeps after the control's restart and between Phases 1 and 2.
- 5 replies, with a 15 s deadline, replacing the fixed 3 s window. Every
  phase prints how long the wait took, when the first and fifth replies came
  and the fifth's icmp_seq, and notes a start slower than the old 3 s window.

The reply count is now always one integer. grep -c printed 0 and the
fallback printed another, so with no replies at all the baseline test
errored, was taken as false, and the phase moved the route anyway.

Gaps are measured from the last reply before the move. A gap before the
move could otherwise let the control pass on an outage the move did not
cause, or fail a netmon phase for one.

On failure the suite now prints node-a's UDP socket table, every phase's
ping log and both nodes' whole logs, not the last 60 lines. For diagnosis,
MC_CONTROL_RUNS repeats Phase 3, MC_KEEP_DIR keeps every ping log and both
node logs, and MC_RUST_LOG replaces both nodes' log filter.
2026-10-22 07:20:26 +00:00

133 lines
4.7 KiB
YAML

# Transport-medium change lab.
#
# node-a ──┬── mc-primary ───┐
# │ ├── router ── mc-far ── node-b
# └── mc-secondary ─┘
#
# node-a is multi-homed with two equally usable paths to the router; node-b
# sits beyond it and is reachable only through the router. That last part is
# the whole design: node-b has to be off-link so the route to it follows
# node-a's *default* route, which is what the suite moves. Put node-b on a
# shared bridge instead and the directly-connected route wins, the source
# address never changes, and the bug under test cannot reproduce.
networks:
mc-primary:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: ${MC_PRIMARY_PREFIX:-172.31.60}.0/24
mc-secondary:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: ${MC_SECONDARY_PREFIX:-172.31.61}.0/24
mc-far:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: ${MC_FAR_PREFIX:-172.31.62}.0/24
x-fips-common: &fips-common
image: ${FIPS_TEST_IMAGE:-fips-test:latest}
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
restart: "no"
entrypoint:
- /usr/local/bin/mc-node-entrypoint.sh
environment:
- RUST_LOG=info,fips::node::netmon=debug,fips::node::handlers::netmon=debug
- PRIMARY_PREFIX=${MC_PRIMARY_PREFIX:-172.31.60}
- SECONDARY_PREFIX=${MC_SECONDARY_PREFIX:-172.31.61}
- FAR_PREFIX=${MC_FAR_PREFIX:-172.31.62}
- ROUTER_OCTET=254
services:
router:
build:
context: ./router
container_name: fips-mc-router${FIPS_CI_NAME_SUFFIX:-}
cap_add:
- NET_ADMIN
sysctls:
- net.ipv4.ip_forward=1
# Strict reverse-path filtering, and the suite does not work without it.
#
# It is what makes a stale source address *hurt*. Both of node-a's
# interfaces stay up and both stay routable, so a packet still sourced
# from the old path is otherwise forwarded and answered quite happily —
# the pin is stale but harmless, the bug does not bite, and the negative
# control passes, which would make every assertion in this suite vacuous.
#
# A real gateway drops that packet as spoofed, because the reverse route
# for its source points out a different interface. That is the actual
# reason a medium change black-holes traffic in the field, so it is the
# thing the lab has to model.
- net.ipv4.conf.all.rp_filter=1
- net.ipv4.conf.default.rp_filter=1
restart: "no"
networks:
mc-primary:
ipv4_address: ${MC_PRIMARY_PREFIX:-172.31.60}.254
mc-secondary:
ipv4_address: ${MC_SECONDARY_PREFIX:-172.31.61}.254
mc-far:
ipv4_address: ${MC_FAR_PREFIX:-172.31.62}.254
# The node under test. Two paths up at once, default route on the primary,
# and the suite moves it to the secondary mid-traffic.
node-a:
<<: *fips-common
container_name: fips-mc-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-mc-node-a
depends_on:
- router
environment:
# MC_RUST_LOG replaces both nodes' filters, for a diagnosis run that
# wants session-level logs; see the README.
- RUST_LOG=${MC_RUST_LOG:-info,fips::node::netmon=debug,fips::node::handlers::netmon=debug}
- PRIMARY_PREFIX=${MC_PRIMARY_PREFIX:-172.31.60}
- SECONDARY_PREFIX=${MC_SECONDARY_PREFIX:-172.31.61}
- ROUTER_OCTET=254
- DEFAULT_VIA=primary
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./node/entrypoint.sh:/usr/local/bin/mc-node-entrypoint.sh:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
mc-primary:
ipv4_address: ${MC_PRIMARY_PREFIX:-172.31.60}.10
mc-secondary:
ipv4_address: ${MC_SECONDARY_PREFIX:-172.31.61}.10
# The far peer. Single-homed and stationary — it never moves, so anything
# the suite observes at this end is a consequence of node-a's move.
node-b:
<<: *fips-common
container_name: fips-mc-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-mc-node-b
depends_on:
- router
environment:
- RUST_LOG=${MC_RUST_LOG:-info}
- FAR_PREFIX=${MC_FAR_PREFIX:-172.31.62}
- ROUTER_OCTET=254
- DEFAULT_VIA=far
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./node/entrypoint.sh:/usr/local/bin/mc-node-entrypoint.sh:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
mc-far:
ipv4_address: ${MC_FAR_PREFIX:-172.31.62}.20