fips_playground/mesh-scan

FIPS Mesh Reachability Scanner

A standalone tool that answers: "From a fresh node peered with only node X, can I actually reach node Y through the mesh?" — producing a reachability matrix across the real public FIPS mesh.

See ../plans/mesh-reachability-scanner.md for the full design rationale.

Why a standalone script

The browser attaches to a single long-lived FIPS daemon. The scanner needs ephemeral daemons peered one-at-a-time with different seed nodes, which the browser architecture can't do. This tool drives real fipsd processes via their Unix control sockets, reusing patterns from ../../fips/testing/chaos/sim/control.py and the browser's handle_fips_directory_json.

Files

File	Purpose
mesh_scan.py	Main scanner / orchestrator
control.py	Control-socket client (line-delimited JSON over Unix socket)
directory.py	Nostr Kind 37195 advert fetcher + npub→IPv6 derivation
fipsd.py	Ephemeral daemon lifecycle (config, spawn, poll, teardown, TUN cleanup)
reachability.py	ping6 / TCP reachability checks

Requirements

• Python 3.10+
• websocket-client, pyyaml (see requirements.txt)
• fips / fipsctl binaries in PATH
• iproute2 (ip command), ping6
• CAP_NET_ADMIN — fipsd needs it to create TUN devices and network namespaces. The scanner auto-detects whether the current user has it; if not, it spawns daemons via sudo -n (passwordless sudo). Alternatively, set the capability on the binary once: sudo setcap cap_net_admin+ep /usr/local/bin/fips.
• Network namespace support — ip netns is used to isolate each ephemeral daemon (see How it works below).

Install Python deps:

pip install -r mesh-scan/requirements.txt

Quick start

1. Smoke test (two ephemeral nodes, no public mesh)

Verifies the whole toolchain — daemon spawn, control socket, peering, reachability — without touching the public mesh:

cd mesh-scan
python3 mesh_scan.py --self-test -v

You should see two ephemeral daemons (fips-scan-900, fips-scan-901) peer over loopback and a ping6 succeed from one TUN to the other.

2. Full scan

# Random sample of 5 seeds, default 45s convergence, ping6 checks:
python3 mesh_scan.py --seeds-sample 5 -v

# Specific seeds, specific targets:
python3 mesh_scan.py --seeds npub1abc,npub1def --targets npub1ghi,npub1jkl

# All seeds (O(N) daemon lifecycles — slow):
python3 mesh_scan.py --seeds all

# Seeds near the tree root (smallest NodeAddr — best reachability if mesh is healthy):
python3 mesh_scan.py --seeds-near-root 5

3. Cleanup

If the scanner crashes, fips-scan-* network namespaces may leak:

python3 mesh_scan.py --cleanup-stale

Inputs / flags

Flag	Default	Description
--seeds all	—	Every node in the directory
--seeds npub1a,npub1b	—	Explicit seed list
--seeds-sample K	5	Random sample of K seeds
--seeds-near-root K	—	K seeds with smallest NodeAddr (tree-root neighborhood)
--targets npub1a,...	all except seed	Restrict targets
--converge-secs N	45	Convergence budget after peering
--ping-timeout N	5	Per-target reachability timeout (seconds)
--method ping6|tcp	ping6	Primary check method
--tcp-port N	80	TCP fallback port
--delay N	0	Seconds between target checks (politeness)
--relay wss://...	damus/nos.lol/offchain	Advert relay (repeatable)
--out-dir DIR	mesh-scan-out	Output directory

Output

matrix.csv

Rows = seeds, columns = targets, cells = ok/<rtt>ms / fail:

seed            npub1abc   npub1def   npub1ghi
npub1root       ok/42ms    ok/55ms    fail
npub1hub        ok/38ms    ok/41ms    ok/67ms

diagnostics.jsonl

One JSON record per (seed, target) pair:

{
  "seed": "npub1abc", "target": "npub1def",
  "reachable": true, "rtt_ms": 55, "loss_pct": 0, "method": "ping6",
  "target_in_identity_cache": true, "target_in_seed_bloom": true,
  "cached_coords": true, "tree_distance": 3,
  "seed_depth": 2, "seed_parent": "npub1root", "mesh_root": "npub1root",
  "converge_secs": 12
}

Summary (stdout)

Total/overall reachability, per-seed rates (good vs bad hubs), per-target rates (universally unreachable = NAT/firewall suspects), and failure-cause correlation (identity-cache miss / bloom miss / coord-cache miss).

Interpreting results

The per-pair diagnostics map directly to FIPS mesh-routing failure modes (see plans/mesh-reachability-scanner.md "Background" section):

Diagnostic	Failure mode
target_in_identity_cache=false	Identity-cache gate (session.rs:1144) — discovery skipped
target_in_seed_bloom=false	Bloom propagation gap — target's existence not propagated to your seed
mesh_root differs	Tree partition — target is in a different spanning tree
cached_coords=false	Coord-cache miss (cold path) — find_next_hop returns None

Direct peering hits find_next_hop branch 2 (direct peer), bypassing all of: coord cache, bloom discovery, tree routing. That's why it always works when mesh routing fails — and why a target that's fail from all seeds but reachable by direct peering is the symptom this scanner diagnoses.

How it works

Each ephemeral daemon runs in its own network namespace (ip netns). This is necessary because the FIPS daemon hardcodes a fd00::/8 route on its TUN device (fips/src/upper/tun.rs). On a host that already runs a long-lived FIPS daemon (fips0), that route already exists, so a second daemon's TUN init fails with File exists (os error 17). A fresh netns gives each ephemeral daemon its own routing table where fd00::/8 is free.

The control socket is a Unix domain socket in /tmp (shared filesystem), so it remains reachable from the host process. Reachability checks (ping6/TCP) are executed inside the daemon's netns via ip netns exec, so the fd00::/8 route resolves to the daemon's TUN rather than the host's fips0.

Each netns also gets a veth pair with NAT masquerading so the ephemeral daemon can reach the public internet (to peer with remote seed nodes over UDP/TCP). Without this, the netns would be isolated to loopback-only and couldn't connect to any public seed. The veth setup, default route, and iptables masquerade rules are created automatically and torn down on daemon stop / --cleanup-stale.

For the --self-test, both ephemeral daemons share a single netns (fips-scan-selftest) so they can peer over loopback (127.0.0.1).

Notes / caveats

• Heuristics: target_in_seed_bloom and cached_coords are best-effort. The control-socket show_bloom / show_routing queries expose aggregate stats, not per-target membership, so these fields use the identity cache as a proxy. Treat them as directional indicators, not ground truth.
• NAT: the scanner node itself may be behind NAT. FIPS Nostr discovery
  • STUN hole-punching should handle this, but running on a publicly-addressable host removes the confound.
• Politeness: scanning N×M pairs generates real mesh traffic. Use --delay and a small --seeds-sample for production meshes.
• Coexistence with sys-fips: each ephemeral daemon runs in its own network namespace with unique ports and a /tmp control socket, so it does not collide with the host's long-lived fips0 daemon.
• Directional asymmetry (observed in self-test): a freshly-peered two-node pair can show asymmetric ping6 reachability (one direction succeeds, the other fails) before coord caches warm up. The self-test tries both directions and passes if either succeeds. On the real mesh, convergence (--converge-secs) mitigates this; the per-pair diagnostics capture cached_coords / target_in_identity_cache to explain any residual asymmetry.
• IPv6 host connectivity: the scanner works even on hosts without public IPv6 (e.g. Qubes OS qubes), because the FIPS overlay uses fd00::/8 IPv6 inside the TUN, which is contained in the netns and tunneled over the daemon's IPv4 UDP transport.