mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 15:48:34 +00:00
The cone and symmetric starts failed now and then with
`Invalid "netns" value`, a node log ending "Timed out waiting for IPv4
on eth0" and, often, a router that had exited with an empty log. Both
entrypoints gave the harness 30 s from container start to deliver their
interface, and setup-topology.sh delivered it with ten serial helper
containers, five per pair. Starting a container is normally 0.16 s, but
while another lab starts on the same Docker daemon it stalls for 3 to
15 s at a time (measured in create and start; exec, inspect and logs
stay fast, and the kernel and disk were not the bottleneck). Ten such
starts overran the window, so the move landed on an exited container.
Reproduced before the change, with nothing injected: two NAT labs
started together on one daemon failed in 4 of 4 starts:
Error: argument "2219471" is wrong: Invalid "netns" value
LAB START FAILED: start_cone exited non-zero on attempt 1; ...
fips-nat-cone-b: Timed out waiting for IPv4 on eth0
fips-nat-router-b: (empty log)
A docker shim that delays each `docker run ... --pid=host` by 8 s also
failed both of two runs the same way. A single idle lab passes either
way.
The change:
- setup-topology.sh runs one helper container per pair (delete, add and
both moves in one `sh -c`, each step echoed) instead of five, checks
that both containers are still running before it, and after a failed
move or configure step names any container that exited, with its exit
code and last 20 log lines. wait_for_pid stops early once a container
has exited.
- Both entrypoints wait IFACE_WAIT_SECS (default 120 s) for their
interface, separate from WAIT_TIMEOUT_SECS (30 s), which still bounds
the relay and STUN waits. The default is set in the scripts, not the
compose file, so the change applies unchanged to the other branches.
- The router entrypoint now says which interface it timed out on and
lists its links, instead of exiting silently under set -e.
- testing/nat/README.md describes the start order and the bound.
The longer wait bounds harness setup, not product behaviour; a stuck
setup still fails, naming the container.
After the change: the two-labs-together start passed 4 of 4, with
router B configured at 31 to 34 s after compose up in the symmetric
lab, past the old 30 s window, so both halves of the change are
needed. The 8 s shim passed twice, and the symmetric scenario alone
passed. Breaking the guard with a 130 s shim delay fails with
"fips-nat-cone-a exited (code 1) before its interface was moved",
followed by its log, "Timed out after 120s waiting for IPv4 on eth0".
There is no committed regression test: reproducing this needs Docker
and a second lab or a delaying shim, so it stays a captured run.
145 lines
4.4 KiB
Bash
Executable File
145 lines
4.4 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
set -euo pipefail
|
|
|
|
NAT_MODE="${NAT_MODE:-cone}"
|
|
LAN_HOST="${LAN_HOST:?LAN_HOST is required}"
|
|
LAN_SUBNET="${LAN_SUBNET:?LAN_SUBNET is required}"
|
|
WAN_SUBNET="${WAN_SUBNET:?WAN_SUBNET is required}"
|
|
TCP_FORWARD_PORTS="${TCP_FORWARD_PORTS:-8443}"
|
|
WAN_GATEWAY="${WAN_GATEWAY:-}"
|
|
# The LAN interface is not there at start: setup-topology.sh moves it in after
|
|
# `docker compose up` returns, and that waits on container starts that have
|
|
# taken seconds each on a busy daemon. Set here rather than in
|
|
# docker-compose.yml so the compose file stays as it is on every branch.
|
|
IFACE_WAIT_SECS="${IFACE_WAIT_SECS:-120}"
|
|
|
|
find_iface_for_subnet() {
|
|
local subnet="$1"
|
|
while read -r idx iface _ cidr _; do
|
|
case "$cidr" in
|
|
${subnet%0/24}*)
|
|
echo "$iface"
|
|
return 0
|
|
;;
|
|
esac
|
|
done < <(ip -o -4 addr show)
|
|
}
|
|
|
|
# Wait for an interface by name. On a timeout, say so and list the links
|
|
# before failing: the caller runs under `set -e`, so a bare failure here
|
|
# would exit the router with an empty log.
|
|
wait_for_iface() {
|
|
local if_name="$1"
|
|
local timeout_secs="${2:-$IFACE_WAIT_SECS}"
|
|
local deadline=$((SECONDS + timeout_secs))
|
|
|
|
while [ "$SECONDS" -lt "$deadline" ]; do
|
|
if ip link show dev "$if_name" >/dev/null 2>&1; then
|
|
return 0
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
|
|
echo "Timed out after ${timeout_secs}s waiting for interface ${if_name}" >&2
|
|
ip -o link >&2 || true
|
|
return 1
|
|
}
|
|
|
|
wait_for_subnet_iface() {
|
|
local subnet="$1"
|
|
local timeout_secs="${2:-$IFACE_WAIT_SECS}"
|
|
local deadline=$((SECONDS + timeout_secs))
|
|
local iface=""
|
|
|
|
while [ "$SECONDS" -lt "$deadline" ]; do
|
|
iface="$(find_iface_for_subnet "$subnet" || true)"
|
|
if [ -n "$iface" ]; then
|
|
echo "$iface"
|
|
return 0
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
|
|
return 1
|
|
}
|
|
|
|
if [ -n "${LAN_IF:-}" ]; then
|
|
wait_for_iface "$LAN_IF"
|
|
else
|
|
LAN_IF="$(wait_for_subnet_iface "$LAN_SUBNET")"
|
|
fi
|
|
|
|
if [ -n "${WAN_IF:-}" ]; then
|
|
wait_for_iface "$WAN_IF"
|
|
else
|
|
WAN_IF="$(wait_for_subnet_iface "$WAN_SUBNET")"
|
|
fi
|
|
|
|
if [ -z "${LAN_IF:-}" ] || [ -z "${WAN_IF:-}" ]; then
|
|
echo "Failed to detect LAN/WAN interfaces"
|
|
ip -o -4 addr show
|
|
exit 1
|
|
fi
|
|
|
|
if [ -z "$WAN_GATEWAY" ]; then
|
|
WAN_GATEWAY="$(echo "$WAN_SUBNET" | awk -F. '{print $1 "." $2 "." $3 ".1"}')"
|
|
fi
|
|
|
|
WAN_ADDR="$(ip -o -4 addr show dev "$WAN_IF" | awk '{print $4}' | cut -d/ -f1 | head -1)"
|
|
|
|
if [ -z "$WAN_ADDR" ]; then
|
|
echo "Failed to determine WAN IPv4 address for ${WAN_IF}"
|
|
ip -o -4 addr show dev "$WAN_IF" || true
|
|
exit 1
|
|
fi
|
|
|
|
sysctl -w net.ipv4.ip_forward=1 >/dev/null || true
|
|
sysctl -w net.ipv4.conf.all.rp_filter=0 >/dev/null || true
|
|
sysctl -w "net.ipv4.conf.${LAN_IF}.rp_filter=0" >/dev/null || true
|
|
sysctl -w "net.ipv4.conf.${WAN_IF}.rp_filter=0" >/dev/null || true
|
|
|
|
ip route replace default via "$WAN_GATEWAY" dev "$WAN_IF"
|
|
|
|
iptables -F
|
|
iptables -t nat -F
|
|
iptables -P FORWARD DROP
|
|
|
|
iptables -A FORWARD -i "$LAN_IF" -o "$WAN_IF" -j ACCEPT
|
|
iptables -A FORWARD -i "$WAN_IF" -o "$LAN_IF" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
|
|
|
|
for port in $TCP_FORWARD_PORTS; do
|
|
iptables -t nat -A PREROUTING -i "$WAN_IF" -p tcp --dport "$port" \
|
|
-j DNAT --to-destination "${LAN_HOST}:8443"
|
|
iptables -A FORWARD -i "$WAN_IF" -o "$LAN_IF" -p tcp -d "$LAN_HOST" --dport 8443 -j ACCEPT
|
|
done
|
|
|
|
case "$NAT_MODE" in
|
|
cone)
|
|
# Full-cone emulation for the single LAN node in this harness:
|
|
# preserve the UDP source port on egress and forward any inbound UDP
|
|
# on the WAN address back to the lone LAN host on the same port.
|
|
iptables -t nat -A PREROUTING -i "$WAN_IF" -p udp -j DNAT --to-destination "$LAN_HOST"
|
|
iptables -A FORWARD -i "$WAN_IF" -o "$LAN_IF" -p udp -d "$LAN_HOST" -j ACCEPT
|
|
iptables -t nat -A POSTROUTING -s "$LAN_SUBNET" -o "$WAN_IF" -p udp \
|
|
-j SNAT --to-source "$WAN_ADDR"
|
|
iptables -t nat -A POSTROUTING -s "$LAN_SUBNET" -o "$WAN_IF" ! -p udp -j MASQUERADE
|
|
;;
|
|
symmetric)
|
|
iptables -t nat -A POSTROUTING -s "$LAN_SUBNET" -o "$WAN_IF" -p udp \
|
|
-j MASQUERADE --random-fully
|
|
iptables -t nat -A POSTROUTING -s "$LAN_SUBNET" -o "$WAN_IF" ! -p udp -j MASQUERADE
|
|
;;
|
|
*)
|
|
echo "Unknown NAT_MODE: $NAT_MODE"
|
|
exit 1
|
|
;;
|
|
esac
|
|
|
|
echo "Router ready: mode=${NAT_MODE} lan_if=${LAN_IF} wan_if=${WAN_IF}"
|
|
ip route
|
|
iptables -S
|
|
iptables -t nat -S
|
|
|
|
exec tail -f /dev/null
|