84 Commits

Author SHA1 Message Date
Arjen
3798ba853c fix(openwrt): ship mesh transports commented, uncomment on setup
The default-on mesh0/mesh1 transport entries logged a per-boot
"interface missing" bind warning on installs that never create
fips-mesh* (PR #123 review). Ship them commented out instead, and have
fips-mesh-setup uncomment the matching meshN block in fips.yaml when it
creates the interface (re-commenting it on remove) — a stock install
stays quiet while the flash-and-drop-in flow still needs no manual
config edit.

Also note in the how-to that both the create and remove paths run
`wifi reload`, which briefly drops client APs on all radios.

The shipped-config regression test now checks both states parse: as
shipped (mesh inactive) and after the uncomment the helper performs.
2026-07-15 16:09:30 +02:00
Arjen
cdf384d730 docs(openwrt): document STA-on-same-radio channel capture; warn in fips-mesh-setup
Field-found third silent-non-peering cause: a client (sta) interface
must follow its upstream AP's channel, the whole radio follows the
STA, and a mesh pinned to a different channel never joins — visible
as 'type mesh point' with no channel line and an empty station dump.
It also does not recover when the STA disconnects (wifi reload + fips
restart needed), so a roaming uplink is fundamentally incompatible
with a fixed-channel mesh on the same radio. Add to the guide's
triage and constraints, and make fips-mesh-setup warn when the target
radio carries a STA.
2026-07-15 16:09:30 +02:00
Arjen
47ce287360 docs(openwrt): clarify dual-band mesh is failover, not multipath
FIPS keeps one active link per peer (cross-connection resolution picks
one winner; the second authenticated path replaces or is torn down),
so a second meshed band is a standby that re-establishes the peer
after keepalive timeout — traffic never uses both bands at once.
Reword helper, shipped config, and guide to say so.
2026-07-15 16:09:30 +02:00
Arjen
c0fcbb6963 style(openwrt): grep -q in the free-index probe (shellcheck SC2143) 2026-07-15 16:09:30 +02:00
Arjen
0284f55405 feat(openwrt): dual-band mesh backhaul — one fips-mesh-setup instance per radio
fips-mesh-setup previously managed a single hardcoded instance
(fips_mesh/fips-mesh0), so running it for a second radio silently
switched the mesh instead of adding a band. Now each radio gets its
own section and netdev (radio0 -> fips-mesh0, radio1 -> fips-mesh1,
trailing-digit derivation with a free-index fallback and a collision
guard), 'remove' takes an optional radio and otherwise removes all
instances, and the shipped fips.yaml carries mesh0+mesh1 entries so
dual-band routers get two complementary backhaul paths — longer reach
on 2.4 GHz, more capacity on 5 GHz — that FIPS's spanning tree treats
as redundant links. Guide and regression test updated to match.
2026-07-15 16:09:30 +02:00
Arjen
e5c484b990 fix(openwrt): surface the same-channel requirement in fips-mesh-setup and the backhaul guide
Field-tested on real hardware: two routers with default channel
settings silently never peer, because mesh points only peer on the
same channel and 'auto' lets each radio pick its own. The script now
prints the radio's band/channel, warns loudly on 'auto' with the
exact uci command to pin one, and the how-to gains an ordered no-peers
triage (channel mismatch first, then an on-air scan check, DFS CAC
wait, and regdomain).
2026-07-15 16:09:30 +02:00
Arjen
413cb8216a feat(openwrt): 802.11s open-mesh backhaul — fips-mesh-setup helper, default transport binding, how-to
Router-to-router radio backhaul over an open 802.11s mesh interface,
with FIPS providing all encryption (Noise IK), authentication, and
routing on top of bare L2 neighbor links:

- fips-mesh-setup: opt-in UCI helper that creates the 'fips-mesh0'
  mesh-point interface (encryption none, mesh_fwding 0 — FIPS is the
  routing layer, SAE would duplicate the Noise layer and force ath10k
  raw mode). 'remove' subcommand undoes it. Radio setup stays opt-in;
  a package must not commandeer radios on install.
- fips.yaml: ship the 'mesh' Ethernet-transport entry enabled by
  default — inert until the interface exists, since a transport whose
  interface is missing at startup is skipped with a warning.
- Regression test: the shipped OpenWrt fips.yaml must parse via the
  real Config deserializer and keep the mesh entry.
- Packaging: install the helper in ipk/apk/buildroot (three synced
  copies), extend CI structural checks and shellcheck targets.
- docs/how-to/set-up-80211s-mesh-backhaul.md: full guide, including
  the radio-first-then-restart ordering constraint.
2026-07-15 16:09:30 +02:00
Johnathan Corgan
c8077967cd peer: add a round-trip contract test for the action message type
Prove the per-peer machine's action type is a runtime-agnostic message
contract: it is Send + Sync + 'static, and every variant round-trips
unchanged through a single-threaded async channel (a current-thread runtime,
sender and receiver on one thread). A wildcard-free match over the variants
makes any future action a compile error here until it is added to the sample,
so a variant that embedded a runtime handle could not slip through unproven.
2026-07-14 23:36:18 +00:00
Johnathan Corgan
5dfa571908 node: pin promote-failure warnings to the handshake tracing target
The two promote-failure warn! sites in the peer-action executor emit under
this module's own tracing target rather than the handshake target the rest of
the establish-path logging uses. Pin both (outbound "Failed to promote
connection" and inbound "Failed to promote inbound connection") to
fips::node::handlers::handshake, matching the sibling max-peers reject log and
keeping all establish-path diagnostics grouped under one target regardless of
which module physically emits them.
2026-07-14 23:23:54 +00:00
Johnathan Corgan
6bebca88ac node: rewrite planning-phase locators out of source comments
Comments across the per-peer machine, executor, lifecycle supervisor,
and peering reconciler carried internal rollout labels and design-note
section references. Rewrite them to describe the code's behavior
directly. Comment-text only; no code changes.
2026-07-14 01:38:52 +00:00
Johnathan Corgan
5d5da69a5b node: drive the link-dead peer reap through the per-peer machine
Route each link-dead peer that the tick sweep's plan_heartbeats decides
to reap through the per-peer machine and executor, replacing the inline
reap body in check_link_heartbeats. The batch decision, the liveness
snapshots (read from the hot-path-written receive clock), and the
heartbeat-send arm stay shell-side and byte-unchanged; the machine only
consumes the decided LinkDeadSuspected, tearing the peer down via
remove_active_peer and reporting the loss to the reconciler exactly as
before, on the same tick with the same wall-clock timestamp. The reap
log stays shell-side.

The machine's link-dead handler no longer emits a decrypt-session
unregister keyed by its shadow index (the full peer teardown already
unregisters the real index; the shadow could have drifted to a reused
index), and its guard now covers the Established state a freshly
promoted peer sits in.

Handshake-timeout, retransmit, and stale-connection cleanup stay inline:
they act on pre-promotion legs that have no machine, and the loss reflex
they use differs from the link-dead one.
2026-07-13 17:08:35 +00:00
Johnathan Corgan
e05b868cf8 node: drive rekey cadence cutover and drain through the peer machine
Route each Cutover and Drain that the shell-side batch poll_rekey decides
through the per-peer machine and the executor, replacing the inline
effect bodies in check_rekey. The batch decision and its per-peer
snapshots stay shell-side and byte-unchanged: poll_rekey phase-groups all
cutovers, then all drains, then all initiations across the peer set, and
that ordering governs the shared index allocator's free-then-allocate
sequence that appears on the wire, so the machine only consumes the
already-decided actions (a new RekeyConsume event) without re-deciding.
InitiateRekey stays inline (its Noise msg1 build is a shell-side leaf)
with a RekeyInitiated observation feeding the machine so its control
state stays coherent for the next tick's cutover.

The cutover and drain logs, which relocated into the executor in the
prior commit, are pinned back to the fips::node::handlers::rekey tracing
target so they stay visible under the operator's module log filter.

Also clears the machine's shadow draining_index on drain so a later
cross-connection resolution cannot double-free the already-freed index.
2026-07-13 16:20:45 +00:00
Johnathan Corgan
0ebd1b44c0 node: add the initiator-cutover and drain executor actions (unwired)
Prepare the establish executor to drive FMP rekey by activating the
SwapSendState action (initiator K-bit cutover via cutover_to_new_session,
with the gated decrypt-worker re-registration) and adding a CompleteDrain
action (erase the drained previous session: free its index, drop its
peers_by_index entry, unregister its decrypt session). Both reproduce the
current inline rekey.rs cutover and drain bodies. The machine's drain
mapping now emits CompleteDrain, using the real drained index rather than
a shadow copy.

Unwired: nothing drives the machine's rekey path yet (live rekey still
runs inline), so these arms are unreachable and the change is
behavior-neutral. The cadence fold that routes cutover and drain through
the machine follows.
2026-07-13 14:51:11 +00:00
Johnathan Corgan
800cfb23e3 node: relocate decrypt-session registration to the establish executor
Move register_decrypt_worker_session out of promote_connection into the
executor's PromoteToActive handler, gated on a promoted or
cross-connection-won result. Every live promote now flows through that
one executor path, so registration still fires exactly once at the same
synchronous point; the direct test callers of promote_connection spawn
no worker pool, so the call was already a no-op for them.

Add the cross-connection loser-link teardown (close the losing
transport, remove its link, re-point addr_to_link at the winner) to the
executor as a guarded follow-up. It is unreachable on the current driven
establish paths, which only promote net-new peers, and asserts so, but
keeps the executor complete for when that case is driven.

Remove the now-dead drive_promote_to_active and ConnAction::PromoteToActive.
2026-07-13 14:20:56 +00:00
Johnathan Corgan
e9112cc1bb node: drive net-new outbound establish through the per-peer machine
Cut the net-new outbound handshake completion (a received msg2 that
promotes a fresh outbound leg to a new peer) over to the per-peer
control machine, mirroring the inbound cutover. handle_msg2 still runs
the msg2 prologue, the ACL check, and the cross-connection swap/keep
arms inline; for the net-new promote it now builds a transient machine,
steps it, and drives promote_connection through the executor. The
session index was already allocated at dial, so there is no two-phase
authorize here.

The wire, index sequence, and peer registry state after promote are
byte-neutral. To keep the promote-failure path neutral, the executor's
cleanup now distinguishes inbound from outbound: an outbound promote
failure records the reject only, matching the prior handler, rather
than the inbound path's link and index teardown.

Cross-connection swap/keep, rekey-msg2, and the dial path stay inline;
the loser-link surgery for the currently unreachable driven
cross-connection case lands with the register relocation next.
2026-07-13 13:42:08 +00:00
Johnathan Corgan
c80a7fdea5 node: drive restart inbound establish through the per-peer machine
Cut the restart handshake (an inbound msg1 from a peer that reconnected
with a new epoch) over to the per-peer control machine, completing the
inbound establish cutover. The old peer is torn down and the fresh leg
promotes through the same two-step authorize-then-allocate path as the
net-new case: the machine's first step emits the old-peer teardown
(invalidate send-state, report loss), the shell interposes the ACL
check, and the second step allocates the new index and sends msg2. The
old index is freed before the new one is allocated and the msg2 wire
bytes are unchanged, so the sequence stays byte-neutral.

With restart driven through the machine, the shared inline establish
tail that only the restart arm reached is deleted.

Also bounds the peer_machines map (remove_active_peer now drops the
peer's machine entry) and restores the msg2-send, promote, and
index-allocation failure warnings the cutover had dropped.
2026-07-13 12:46:23 +00:00
Johnathan Corgan
0bf031dd32 node: drive net-new inbound establish through the per-peer machine
Cut the net-new inbound handshake (a fresh msg1 that promotes to a new
peer, plus the at-capacity reject) over from the inline handle_msg1 logic
to the per-peer control machine. handle_msg1 still classifies via
establish_inbound and still owns the Noise wire step, the late ACL check,
and the promote_connection registry surgery; for the net-new path it now
builds the machine, steps it, and executes the returned actions.

Authorization is interposed between two machine steps so the session
index is allocated only after the ACL check passes: a rejected or
unauthorized msg1 consumes no index, matching the prior order exactly.
The msg2 wire bytes, the index-allocation sequence, and the reject
metrics are all byte-neutral. Restart, resend, rekey-respond, and the
other reject arms stay inline unchanged; they move to the machine once
outbound establish is cut over and every promoted peer has a machine.

Also fills in the executor's send-failure and promote-failure cleanup so
a mid-establish error tears the leg down and frees its index exactly as
before.
2026-07-13 12:05:17 +00:00
Johnathan Corgan
4a0584a5e9 node/dataplane: add the per-peer machine home and action executor (unwired)
Add Node.peer_machines (a LinkId-keyed map from the stable link handle to
the per-peer control machine) as the home for the machines, and a new
dataplane/peer_actions.rs holding execute_peer_actions / advance_peer_machine:
the executor that maps each PeerAction the machine emits to its shell call —
frame and send a handshake via build_msg2, drive promote_connection and feed
the PromotionResult back through the machine, tear a peer down via
remove_active_peer, free session indices, report loss via note_link_dead.
Actions for the rekey, connected-UDP, and timer paths are stubbed with notes
for the commits that fold those mechanisms in.

Unwired: nothing drives the machine yet — no live handler path calls the
executor and peer_machines is never populated — so this is behavior-neutral;
the inbound and outbound establish paths still run their existing inline
logic. The executor is cut over path-by-path in the following commits.
2026-07-13 11:11:20 +00:00
Johnathan Corgan
59155df4e3 peer: split ActivePeer send-state into PeerSendState (two-tier boundary)
Draw the control/published-send-state boundary inside ActivePeer by
grouping the send-critical fields — the three epoch session slots
{current, previous, pending}, the K-bit flag and session-start, the
transport target, the connected-UDP handles, and the hot counters —
into a new co-located PeerSendState struct. The control-tier fields
(identity, connectivity, declaration/ancestry, filter and tree-announce
groups, remote_epoch, the rekey-negotiation sub-machine, and the rest)
stay on ActivePeer.

Behavior-neutral: a pure field regrouping. Every accessor signature is
unchanged (bodies now read/write self.send.*), so the hot path and the
handlers are byte-untouched; both K-bit cutovers still rotate the three
slots atomically with the same control-tier updates. No Arc/ArcSwap —
the fields are co-located and read by plain borrow; publishing behind a
shared cell is later plumbing for a sharded data plane.
2026-07-13 10:36:15 +00:00
Johnathan Corgan
fcaee74ec0 peer: add the per-peer FMP control state machine (unwired)
Introduce src/peer/machine.rs: a sans-IO per-peer control FSM that
consolidates the scattered handshake/rekey/timeout driver logic now
spread across node/handlers. The machine is a pure reducer —
step(event, now, index_allocator) -> [action] — that reuses the
existing FMP decision cores (establish_inbound/establish_outbound/
cross_connection_winner/poll_*) rather than reimplementing any
decision, and returns runtime-agnostic actions the driver executes.

Control-tier state only; the published send-state boundary and the
driver wiring land in following commits. The machine is terminal at
Closed — re-dial is the reconciler's, so it holds no cross-attempt
retry state.

Includes eight unit tests: inbound and outbound establish, N:1
identity crystallization, the dual-initiation tie-break,
restart-override, rekey initiator cutover, the data-plane-owned
responder cutover boundary, and liveness -> link-dead -> report-lost.
Unwired — nothing calls it yet.
2026-07-13 09:58:45 +00:00
Johnathan Corgan
56e3d56c25 peer/connected_udp: own the connected-socket fd with OwnedFd
Replace the hand-rolled RawFd field and the unsafe Drop on
ConnectedPeerSocket with an OwnedFd, whose own drop glue closes the
fd. from_fd now stores the OwnedFd it already receives instead of
stripping ownership through into_raw_fd, and the manual libc::close is
gone, shrinking the unsafe surface.

Behavior-neutral: the fd still closes exactly once at last-Arc-drop and
as_raw_fd returns the same underlying fd while the socket is alive.
2026-07-13 09:20:04 +00:00
Johnathan Corgan
7b0590f70e node/lifecycle: wire the runtime child-exit producer
Wire exit detection for the four directly-observable optional children so
the supervisor FSM's ChildExited edge fires at runtime. A runtime
child-liveness mpsc channel carries a Child on exit: the DNS task and the
two TUN threads self-report when their body returns, and a 2s poll monitor
reports mDNS and Nostr via new is_finished accessors. The rx_loop gains a
select arm that steps the FSM and republishes health (Degraded, since a
running node always has at least one transport up). Transports and worker
pools expose no runtime-exit signal and are left for a follow-up.

Adds LanRendezvous::is_finished and NostrRendezvous::is_finished; the
latter treats a shutdown-taken connect_task as finished so the monitor
terminates after a stop instead of polling forever.
2026-07-13 08:48:51 +00:00
Johnathan Corgan
b93a127623 node/lifecycle: add runtime ChildExited health routing to the supervisor FSM
Add the sans-IO half of runtime child-liveness monitoring: a ChildExited
event and an on_child_exited handler that routes a runtime task or thread
exit the same way a start failure does. An optional child exiting degrades
the node (Degraded, with the child recorded in the health reasons); the
last transport exiting publishes Failed. There is no restart, and the
handler is inert outside Running (startup, drain, and teardown own their
own child bookkeeping via the pending/up sets). Failed here is a published
health signal only, not a teardown.

The start and runtime paths share the health classification, extracted
from resolve_start_health into classify_health.

The producer that emits the event (the exit-detection wiring) lands in a
following commit; the event variant carries a temporary allow(dead_code)
until then.
2026-07-13 08:25:45 +00:00
Johnathan Corgan
85a4983dbe node/peering: restore the open-discovery sweep summary log
The overlay-discovery cutover to the sans-IO reconciler dropped the
operator-facing "open-discovery sweep complete" summary, including its
per-reason skip breakdown, because the sweep logic moved into the
log-free core. Have the core accumulate the enqueue/skip tally as it
reconciles and return it as data; the driver adds the two values only it
holds (the raw cache size and the self-advert filter) and emits the
summary, reproducing the old startup-vs-per-tick summarize gate and the
budget-zero debug path.

The self-advert precedence matches the pre-cutover sweep: a stale or
self-configured own advert is attributed to the age/configured buckets,
not skipped_self.

Behavior-neutral: the tally is pure side-counting; no dial or enqueue
decision changes.
2026-07-13 08:00:46 +00:00
Johnathan Corgan
5090ab7851 node/peering: drive opportunistic transport-neighbor growth through the reconciler
Cut the last scattered peering mechanism — opportunistic growth from
transport-neighbor beacons and LAN mDNS — over to the reconciler's
opportunistic layer via a gate-checked reconcile_opportunistic wrapper,
one call per tick slot (transport and LAN stay separate slots so their
per-tick budget and per-peer cap are not shared). The driver keeps the
beacon/mDNS I/O and the path-granular prefilters that read live state
(self, fresh-enough-to-skip, connecting-on-path) and executes the emitted
Connect intents; the core owns the connected/budget/per-peer-cap
decisions. A first-wins per-peer dedup on the LAN path reproduces the old
inline once-per-peer dial now that the snapshot core cannot observe the
intra-tick connecting feedback.

Delete the now-dead discovery_connect_budget helper. With this, all three
scattered peering mechanisms (auto-connect and retry, overlay discovery,
neighbor growth) are unified in one sans-IO reconciler. Behavior-neutral;
unit test count unchanged (1607).
2026-07-13 06:02:21 +00:00
Johnathan Corgan
03ced618ce node/peering: drive Nostr open-discovery through the reconciler overlay layer
Cut the overlay (Nostr open-discovery) enqueue over to the sans-IO
reconciler. A gate-checked reconcile_overlay wrapper runs the overlay
layer alone at the discovery tick slot; the monolithic reconcile would
re-fire the always-on retry-dial and double the per-tick dial cap, so the
overlay slot must call only its layer. The driver builds the candidate
pool from the overlay advert cache (self excluded), the cooldown and
configured-npub sets, and the startup/steady max-age, then feeds the
reconciler; the emitted enqueue set drives the per-enqueue identity-cache
and alias pre-seed exactly as before. Enqueued entries are dialed at the
retry slot, preserving the two-phase cadence.

Delete the now-dead enqueue-budget and expiry helpers. Behavior-neutral:
same candidates enqueued in the same order with the same budget, cap, and
expiry. Opportunistic transport-neighbor growth remains imperative and is
cut over next. Unit test count unchanged (1607).
2026-07-13 05:34:28 +00:00
Johnathan Corgan
bf81f422ea node/peering: drive the mandatory-floor and retry mechanism through the reconciler
Route the auto-connect floor and the connection-retry mechanism through
the sans-IO reconciler instead of the imperative Node methods. A thin
peering driver (note_handshake_timeout / note_link_dead) centralizes the
reflex path with the gate guard and the already-connected check; the
per-tick retry-dial and the startup floor build reconciler inputs and
execute the emitted Connect intents. The three imperative retry methods
are removed and their call sites rerouted.

Wire the drain and startup gates. Entering a bounded drain now clears the
retry schedule and suppresses the peer-loss reconnect reflex (the drain
window runs under the Suspended gate), so the drain no longer fights a
reconnect for the peers it just closed. The startup floor runs under an
explicit Reconciling gate at the existing peer-connect seam, preserving
the current dial position.

Behavior-neutral except the intended drain change. Overlay discovery and
opportunistic transport-neighbor growth remain imperative for now; they
observe the relocated retry schedule and are cut over next. Unit tests
migrated to the driver API with assertions unchanged (1607).
2026-07-13 05:00:10 +00:00
Johnathan Corgan
a0cf593580 node: relocate the peering fields off Node into the Peering owner
Move retry_pending and pending_connects from Node into the Peering owner
struct (Node.peering), the home introduced with the reconciler core.
Pure mechanical relocation: the two collections now live behind
self.peering, with roughly seventy accessor sites repointed. No decision
logic is touched and the unit-test count is unchanged (1607). The
imperative peering methods still run; the reconciler is wired in the
following commit.
2026-07-13 04:24:34 +00:00
Johnathan Corgan
5d08d27d3c node/peering: add the sans-IO peering reconciler core (unwired)
Introduce PeeringReconciler, the synchronous decision core for peer
desired-state, with its input/action vocabulary (Gate, Budget, Observed,
Candidate, DiscoveryPools, Policy, PeeringAction) and the Peering owner
struct. reconcile() computes connect/retry intents across four layers:
the auto-connect mandatory floor, the ceiling-only overlay enqueue, the
opportunistic transport-neighbor growth, and the node.limits ceiling
enforced inline. It mirrors the supervisor sans-IO shape: no I/O, no
clock reads (time enters as a parameter), no runtime handles.

The core owns its own cross-attempt retry schedule so escalating backoff
survives the fresh connection created per re-dial. Gate-guarded reflexes
(on_handshake_timeout / on_link_dead) reproduce the current backoff math
and no-op while draining. Overlay is strictly ceiling-only (no peer-count
set-point). Disconnect is defined but never emitted (no shedding).

Unwired here: the driver still runs the imperative peering methods and
Node holds the live retry map. 11 unit tests drive the pure core with
synthetic inputs. No behavior change.
2026-07-13 04:10:36 +00:00
Johnathan Corgan
b676c9d83a node: rewrap over-width retry_state_iter signature
Homing the retry schedule under peering/ lengthened the RetryState
module path, pushing the retry_state_iter return-type line past the
100-column max_width. Rewrap to satisfy rustfmt. No behavior change.
2026-07-13 04:00:41 +00:00
Johnathan Corgan
a45eefb58a node: home the peering concept; relocate the connection-retry schedule
Create src/node/peering/ as the home for the peer desired-state
(homeostatic reconciler) concept and move the cross-attempt connection
retry schedule into it: RetryState plus schedule_retry /
schedule_reconnect / process_pending_retries.

Mechanical relocation only. The methods remain inherent on Node; the
retry_count must persist across re-dials (a fresh connection is created
each attempt), which is why the schedule belongs in the peering home
rather than on a per-connection type. The one-level-deeper module path
requires pub(super) -> pub(in crate::node) to preserve the prior scope,
plus module-path fixups at the reference sites. No behavior change; unit
test count unchanged (1596).
2026-07-13 03:30:52 +00:00
Johnathan Corgan
d61d189572 node: split Running into Full/Degraded, add Failed health state
Determine node health at start completion instead of unconditionally
reaching Running. Zero transports up is now Failed (fatal): start()
tears down cleanly and returns an error, and the daemon exits. Any
configured optional child that failed to start - a transport beyond the
first, Nostr, mDNS, TUN, DNS, or a worker pool - leaves the node
Degraded but serving, with an operator warning naming what failed. All
configured children up is Full. A child the node was never asked to run
does not count against health.

The published NodeState gains Degraded and Failed variants, both visible
via control queries; Degraded is operational, Failed is not. The
lifecycle FSM gains the health states plus the PublishState action that
drives them - a health fork cannot be a single direct state write, which
is why the earlier commits deferred it to here.

Runtime child-exit health re-evaluation (a running child dying) is a
separate liveness-monitoring mechanism left for a follow-up; this commit
is start-time health only.
2026-07-13 00:46:31 +00:00
Johnathan Corgan
d6ca632251 node: add bounded graceful-shutdown drain phase
Add an operator-visible Draining phase on daemon shutdown. On the
shutdown signal the node broadcasts Disconnect to all peers, then keeps
serving for a bounded window - up to node.drain_timeout_secs (default 2s),
exiting early once all peers are gone - before tearing down. This lets
in-flight traffic settle and peers observe the disconnect before the
transports close, rather than the previous immediate teardown.

The lifecycle FSM gains a Draining state plus Drain/DrainDeadlineElapsed
events; the run loop observes the shutdown signal and transitions to
draining in place - one continuous loop, so the channel receivers are
never destructively cancelled. The published NodeState gains a Draining
variant, visible via control queries during the window. The immediate
stop() path used by tests and non-daemon callers is unchanged: it still
tears down immediately with no drain wait.

The reconciler-gate actions the drain emits are no-ops until the peering
reconciler lands and consumes them.
2026-07-12 23:11:28 +00:00
Johnathan Corgan
6c5fd3f4b0 node: extract lifecycle supervisor FSM, migrate substrate fields (behavior-neutral)
Introduce a sans-IO lifecycle supervisor: a synchronous step(event) ->
[action] state machine (SupervisorFsm) that authors the substrate-child
spawn and teardown order, plus an owner struct (Supervisor) holding the
substrate runtime fields and embedding the FSM. The substrate-lifecycle
fields leave Node's flat list into the owner - state, packet_tx, the TUN
reader/writer handles + shutdown fd + channels, the DNS task + identity
channel, the Nostr/LAN rendezvous drivers, and the encrypt/decrypt worker
pools; the dataplane keeps packet_rx.

start()/stop() become the driver executing the FSM's SpawnChild/StopChild
actions: same children, same order, same warn/debug-and-continue on
optional failures, same logs, same NodeState transitions. Behavior- and
wire-neutral. The only determinism change is that transports now tear
down in ascending-id order, previously nondeterministic HashMap iteration.

The bounded Draining phase and the Running{Full|Degraded} health split
land as separate follow-on commits.
2026-07-12 21:03:46 +00:00
Johnathan Corgan
434b9726aa node: establish dataplane/ and session/ concept homes (behavior-neutral)
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).

Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
  -> node/dataplane/ — the whole RX hot path (the select! run loop,
  transit/local forwarding, the link-message router, the RX decrypt
  path with responder K-bit cutover + roam writes, and connected-UDP
  fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
  concept home for the data/state types. The message-behavior file
  handlers/session.rs stays put for now (folds in with the later FSP
  session step).

The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.

Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
2026-07-12 19:22:08 +00:00
Johnathan Corgan
26d70ebb59 Merge branch 'maint' into master
Forward-merge three maint fixes, hand-relocated into master's
post-sans-IO / discovery-to-lookup structure:
- drop the redundant TreeMetrics parent_switched counter (keep
  parent_switches), reconciled across the refactored tree/mmp sites and
  the spanning-tree test reads
- keep the tighter path_mtu when applying a LookupResponse, now in
  handlers/lookup.rs after the discovery module rename
- reuse one shared secp256k1 context in the identity module
2026-07-12 16:51:01 +00:00
Johnathan Corgan
9b46b6fa85 fipstop: rename routing-stats "Discovery" labels to "Lookup"
The routing-stats pane's Discovery Requests/Responses sections show the FMP
overlay coordinate-lookup counters. Rename the section labels and the nested
JSON key they read from "discovery" to "lookup" to match the metric family's
canonical name. The daemon dual-emits both keys, so this reads the current
name and no longer depends on the deprecated "discovery" alias.
2026-07-12 01:52:54 +00:00
Johnathan Corgan
cbc089b820 transport/ethernet: rename config discovery flag to listen
Rename the Ethernet per-interface config flag from discovery to listen,
so the receive/transmit toggle pair reads as the symmetric announce
(transmit) / listen (receive) neighbor-beacon vocabulary. The old
discovery: key is still accepted via a serde alias, so deployed configs
load unchanged; to_yaml re-emits it under the canonical listen: name.
Marked deprecated for removal at the v2 cutover.

Updates the config field + accessor, the transport listen_enabled local,
the one struct-literal test consumer, the chaos sim config generator,
packaged fips.yaml examples, and the classified operator-facing docs
(ethernet neighbor-beacon subsystem prose; the generic Transport
discovery capability prose is left unchanged). Adds a compat parse test
asserting the legacy alias, the new key, and that deny_unknown_fields
still rejects unknown keys. Behavior-neutral.
2026-07-11 23:13:28 +00:00
Johnathan Corgan
4c95be0000 transport/ble: rename discovery module to neighbor
Mirror the ethernet neighbor rename in the BLE transport: the internal
peer-detection buffer becomes NeighborBuffer (from DiscoveryBuffer) and
the module is renamed discovery -> neighbor. The BlueR/bluez API terms
(DiscoveryFilter, set_discovery_filter) and the BLE advertise/scan
mechanism vocabulary are unchanged, as is BleConfig. Behavior-neutral.
2026-07-11 23:01:48 +00:00
Johnathan Corgan
e362ab67a6 transport/ethernet: rename discovery module to neighbor
Rename the ethernet link-local neighbor-beacon subsystem from the
overloaded "discovery" vocabulary to a neighbor umbrella. The beacon
frame vocabulary is retained (build_beacon/parse_beacon/BEACON_SIZE/
FRAME_TYPE_BEACON/FRAME_TYPE_DATA); only the subsystem and buffer
identifiers change: DiscoveryBuffer becomes NeighborBuffer,
discovery_buffer becomes neighbor_buffer, and DISCOVERY_VERSION becomes
BEACON_VERSION (wire value 0x01 unchanged). Config toggles are left for
a follow-up commit. Behavior-neutral.
2026-07-11 22:56:05 +00:00
Johnathan Corgan
6c9f55ea80 transport: rename darwin_sockopts to sockopts_macos
The module is gated target_os="macos" (not the broader Darwin/iOS family),
so the name now tracks the cfg and matches the *_macos.rs file convention
(io_macos.rs). Drops the redundant inner #![cfg(target_os="macos")] (the
decl gate already covers it) and corrects a stale top comment that claimed
the module stays visible on Linux — it is macos-decl-gated, so it never was.
Mechanical rename; no logic change.
2026-07-11 21:37:00 +00:00
Johnathan Corgan
89a31fd555 transport: move the connected-UDP fast-path handles into the peer module
ConnectedPeerSocket and PeerRecvDrain are node/peer-side logic: the Transport
trait never touches them, ActivePeer stores them, and node's encrypt worker
drives them. They only happened to live under transport/udp. Relocate the
handle types to a new src/peer/connected_udp/ module (socket.rs + drain.rs),
which the node handler and encrypt worker reach via node -> peer (no new edge;
a node home would have forced a peer -> node cycle).

The udp transport keeps only the kernel-construction seam: open_connected_fd,
now folded into udp/io.rs (the byte-layer home) behind a linux/macos-gated
submodule and re-exported as transport::udp::open_connected_fd. The node
handler builds the fd through it and adopts it via ConnectedPeerSocket::from_fd.

Behavior-neutral: no wire/config/metric/log change; the per-packet hot path
(bare-RawFd send_batch_gso/raw) is untouched. Preserves the Arc multi-owner
contract, the drop-drain-before-socket ordering, and the drain's detach-on-Drop
deadlock avoidance. Reconciles the old cfg(unix)/any(linux,macos) double-gate
onto the single any(linux,macos) predicate.
2026-07-11 21:30:36 +00:00
Johnathan Corgan
ab0a46f2c0 transport: extract udp open_connected_fd syscall body into a standalone fn
Pulls the connected-UDP socket construction (socket/REUSEADDR/REUSEPORT/
BUFFORCE/bind/connect + darwin tuning) out of ConnectedPeerSocket::open into
a pub(crate) open_connected_fd returning an OwnedFd; open() now delegates and
adopts the fd. Error paths still close the fd via the temporary's Drop; the
success tail transfers ownership through OwnedFd. Behavior identical. Prepares
the fast-path handle types to move to the peer module while the socket
construction stays with the udp transport.
2026-07-11 21:08:29 +00:00
Johnathan Corgan
e839aead7a transport: extract tcp connection-pool types into pool.rs
Moves the inline TcpConnection/ConnectingEntry structs, the Direction
enum, and the ConnectionPool/ConnectingPool type aliases out of the
1147-line tcp/mod.rs into a dedicated pool.rs, matching the canonical
per-transport layout. Struct fields are pub(crate) so mod.rs can still
construct and read them across the module boundary. Pure relocation; no
logic change.
2026-07-11 20:26:42 +00:00
Johnathan Corgan
6d6889d0f6 transport: give ethernet a canonical addr.rs home for MAC parsing
Moves parse_mac_string into ethernet/addr.rs and re-exports it at the
ethernet module root so the ethernet::parse_mac_string path stays
byte-identical for its one external consumer (zero churn there). Its
unit tests stay in mod.rs, calling through the re-export. Pure
relocation; no logic change.
2026-07-11 20:19:08 +00:00
Johnathan Corgan
196d9492da transport: rename ethernet byte-layer module from socket to io
Normalizes the ethernet transport onto the canonical byte-layer name
(io.rs), including the per-OS files io_linux.rs/io_macos.rs and their
#[path] wiring. Pure file rename plus module-path and doc-comment
updates; no logic, wire, config, metric, or log change.
2026-07-11 20:14:37 +00:00
Johnathan Corgan
0f2e91b479 transport: rename udp byte-layer module from socket to io
Normalizes the udp transport onto the canonical per-transport module
layout (ble is the reference: io/pool/addr/stats). Pure file rename plus
module-path updates at the use sites; no logic, wire, config, metric, or
log change. Behavior identical.
2026-07-11 20:10:19 +00:00
Johnathan Corgan
32475d859e transport: consolidate tor and nym SOCKS5 dialing into a shared module
The Nym transport was a near-clone of the Tor transport's outbound
SOCKS5 path. Extract the shared logic into src/transport/socks5/ so both
transports drive one implementation instead of two maintained copies:

- share one SOCKS5 mock server between the tor and nym tests
- extract the common send/receive/connect counters into a shared
  ProxiedStatsBase (snapshot structs and emitted metrics unchanged)
- add a shared Socks5Dialer that collapses all six connect variants;
  the sole dialing difference (tor's per-destination circuit-isolation
  auth vs nym's no-auth) is an enum on the dialer
- route both tor and nym dialing through the shared dialer
- share the proxied connection pool, generic over a per-connection meta
  type that carries tor's inbound/outbound direction counting (nym uses
  the unit type)
- share the proxied receive loop

Behavior-neutral: no wire-format, config-key, emitted-metric, log, or
error-variant change. Tor keeps its control-port, inbound/onion, and
directory surface and its own address validation.
2026-07-11 18:45:41 +00:00
Johnathan Corgan
f2e6b8befb transport: promote FMP stream framing to transport::framing
The FMP frame-boundary reader lived in transport/tcp/stream.rs, but the
Tor and Nym transports both reached across module boundaries to
'use crate::transport::tcp::stream::read_fmp_packet', a layering smell:
the reader is a shared stream-framing utility, not a TCP-private one.
Move the file to transport/framing.rs and repoint the tcp/tor/nym use
sites, removing the tor->tcp and nym->tcp dependencies. Behavior
unchanged.
2026-07-11 04:32:38 +00:00
Johnathan Corgan
1aacdfa086 transport: share connection-pool counters via PoolCounters
TcpStats and TorStats each carried an identical pair of pool_inbound/
pool_outbound atomics plus the same five record_pool_* / pool_inbound_count
methods over them. Move the counter logic into a shared PoolCounters
struct (new transport/stats_common.rs) embedded as a 'pool' field in both.
The public record_pool_* methods stay as thin delegators so all call
sites and the flat pool_inbound/pool_outbound snapshot fields are
unchanged; only the duplicated atomic bookkeeping is now single-sourced.
2026-07-11 04:30:19 +00:00
Johnathan Corgan
a7dfe47663 transport: derive Serialize for EthernetStatsSnapshot
EthernetStatsSnapshot derived only Clone/Debug/Default, unlike every
other transport's *StatsSnapshot which also derives Serialize. That gap
forced a hand-rolled serde_json::json!{} arm in TransportHandle::
transport_stats() that re-listed all ten fields by hand. Add the
Serialize derive and collapse the arm to the same one-line
serde_json::to_value(...) form the other transports use. The emitted
JSON keys and values are unchanged.
2026-07-11 04:27:50 +00:00
Johnathan Corgan
8aab71af86 bench: add routing next-hop microbench
Measures the per-forwarded-packet cost of routing candidate assembly
(routing_candidates over a synthetic RoutingView) against a zero-alloc
reference across 8/32/128/256 peers, with per-call allocation counts via
a counting allocator. Criterion harness; no production code change.
2026-07-11 01:48:34 +00:00
Johnathan Corgan
2cffc10520 Merge branch 'maint' (bloom single-digest double-hashing)
Forward-merges the bloom SHA-256-once fix from maint. The bloom filter was
relocated to proto/bloom/ by the sans-IO refactor, so the fix applied cleanly
to proto/bloom/core.rs; the behavior-neutral test was re-homed into
proto/bloom/tests/core.rs (maint carried it in the pre-split bloom/tests.rs).
2026-07-11 01:32:48 +00:00
Johnathan Corgan
1c1ed0d939 Relocate PromotionResult into proto/fmp
Move the PromotionResult enum (and its impl) out of peer::mod and into
proto/fmp/core.rs, alongside the cross_connection_winner tie-break helper
that was relocated the same way. This is FMP connection-lifecycle result
vocabulary, so it belongs in the FMP subsystem home rather than the peer
module.

Behavior-neutral pure type relocation: consumers import it from
crate::proto::fmp, and the crate-root public path crate::PromotionResult
is preserved via a re-export in lib.rs (mirroring cross_connection_winner).
Full lib suite green at baseline.
2026-07-10 16:52:18 +00:00
Johnathan Corgan
1c41f73931 Relocate FMP link wire codec into proto/fmp
Move the FMP mesh-layer wire format (common prefix, encrypted/msg1/msg2
headers, and the build_*/inner-header codec fns) out of node/wire.rs and
into proto/fmp/wire.rs, so the whole FMP wire surface lives with its
subsystem, matching the proto/fsp/wire.rs layout. The wire module becomes
pub(crate) mod wire; callers reach it via crate::proto::fmp::wire.

Behavior-neutral: pure relocation plus import-path rewrites across the
node/peer consumers; no logic change. Full lib suite green at baseline.
2026-07-10 16:42:17 +00:00
Johnathan Corgan
39ad4d2e67 Extract the nostr rendezvous decision logic into sans-IO state machines
Carve the nostr rendezvous engine's decision logic out of the async
NostrRendezvous driver into synchronous, clock-injected cores, following
the failure_state.rs pattern: sync state behind std::sync::Mutex, time
passed in as now_ms, no .await and no I/O in the core. The driver performs
all relay/socket I/O and executes the returned plans.

- AdvertMachine (src/nostr/advert.rs): advert publish/cache/fetch/prune
  decision logic; the driver executes a returned PublishPlan.
- TraversalMachine (src/nostr/traversal_machine.rs): the engine-scoped
  cross-session decision state -- in-flight-initiator dedup, the dual-init
  responder suppression election, and the replay/seen-session cache.
- classify_punch_packet (src/nostr/traversal.rs): a pure classifier for
  the punch recv loop's packet branch table.

The driver keeps all I/O, the NAT punch send cadence, the offer-slot
admission semaphore, and the pending-answer oneshot routing. Behavior-
neutral: no wire, config-key, or metric change. Adds unit coverage for
the advert plans, election ordering, replay eviction, initiator dedup,
and punch classification.
2026-07-10 03:28:11 +00:00
Johnathan Corgan
4d2504f59d Update trace-target references for the relocated rendezvous modules
Moving the nostr rendezvous engine to src/nostr/ (and mDNS to src/mdns/)
changed the module-path-derived tracing targets from
fips::discovery::nostr::* to fips::nostr::*. Update the RUST_LOG filters
in the NAT and mesh-lab test compose files and the resolve-peers-via-nostr
tutorial's debug recipe to the new targets so trace configs and the
documented journal-watch command keep emitting the intended lines.

Without this, the stun-faults suite's Phase-0 pre-flight (which greps the
daemon journal for the debug-level "STUN observation succeeded" /
"traversal: initiator STUN observed" lines) saw those lines suppressed —
the daemon behaved correctly, but the stale RUST_LOG target hid the
evidence. Test-harness and docs only; no source or behavior change.
2026-07-09 23:20:28 +00:00
Johnathan Corgan
1208f6a5c2 Consolidate the nostr rendezvous driver into src/nostr
Pull the rendezvous driver state and the movable driver logic out of the
Node struct into the src/nostr home. A new RendezvousDriver owns the
engine handle and the four bookkeeping fields (traversal start time,
startup-sweep latch, adopted bootstrap-transport set and their npubs)
that previously sat loose on Node; Node holds a single driver field and
reaches the same data through thin accessors, including the rx-loop
hot-path protocol-mismatch hook.

The advert build/refresh, the via_nostr fallback-address resolve, the
overlay-endpoint-to-PeerAddress mapping, and the bootstrap request move
onto the driver, taking their Node inputs explicitly (a transport-
endpoint snapshot for advert building) rather than reading Node fields
directly. Transport/connection-table-bound work (traversal adoption,
bootstrap-transport cleanup, the open-discovery sweep, and the outbound
budget calculators) stays on Node as thin glue that calls into the
driver.

Behavior-neutral relocation: statements moved verbatim, no logic, wire,
config-key, or metric changes. lifecycle.rs shrinks ~230 lines. cargo
fmt/build/clippy clean; lib suite 1547 passing (baseline unchanged).
2026-07-09 22:13:56 +00:00
Johnathan Corgan
3f80530cc5 Move nostr peer rendezvous and mDNS into dedicated module homes
Relocate the overlay peer-rendezvous subsystem out of the overloaded
src/discovery/ tree into two focused, independent homes: src/nostr/
(relay-mediated overlay endpoint advertise/resolve/auto-mesh plus NAT
traversal) and src/mdns/ (link-local DNS-SD rendezvous). The two
subsystems are independent, so they get separate homes rather than
sharing one.

Drop the ambiguous "Discovery" stem from their identifiers in favor of
"Rendezvous": NostrDiscovery -> NostrRendezvous, LanDiscovery ->
LanRendezvous, and the matching config, policy, field, and method names.
The former src/discovery.rs handoff types (EstablishedTraversal,
BootstrapHandoffResult, the punch-packet helpers) fold into
src/nostr/handoff and stay reachable via the crate-root re-exports.

Pure relocation and rename: no logic, wire-format, config-key, metric,
or tracing-target changes. The operator-facing node.rendezvous.nostr.*
and node.rendezvous.lan.* config keys and the fips-overlay-v1 advert
namespace are byte-identical. cargo fmt/build/clippy clean; lib test
suite 1547 passing (baseline unchanged).
2026-07-09 21:51:54 +00:00
Johnathan Corgan
3b401a0cbd Disambiguate the mesh-lookup "discovery" name across the source tree
The identifier "discovery" named three unrelated subsystems; the FMP
overlay coordinate-lookup subsystem is now consistently "lookup". This
finishes the concept-#1 rename across the shell, config, and metric
layers left after the earlier proto-layer rename:

- Handler module node::handlers::discovery -> node::handlers::lookup, and
  reset_discovery_backoff -> reset_lookup_backoff.
- The Node lookup-engine field Node.discovery -> Node.lookup, renamed by
  resolved binding so the metrics().discovery and node.discovery config
  paths are left untouched.
- The lookup metric types DiscoveryMetrics -> LookupMetrics,
  DiscoveryStatsSnapshot -> LookupStatsSnapshot, and Metrics.discovery ->
  Metrics.lookup.

Two surfaces cross a stability boundary and ship behind a compatibility
window, both marked in-code for removal at the v2 cutover:

- The control-socket metric family is dual-emitted under both "discovery"
  (deprecated alias) and "lookup" so existing dashboards keep working.
- The node.discovery.* config table is split into node.lookup.* (mesh
  lookup scalars) and node.rendezvous.* (nostr/lan peer rendezvous).
  NodeConfig does not deny unknown fields, so a naive rename would make a
  deployed node.discovery: block deserialize into nothing and silently
  revert every setting to default. A deprecated all-Option
  DiscoveryConfigCompat field captures a legacy block and a new post-parse
  Config::normalize_deprecated_keys pass folds it into the new tables with
  a one-time deprecation warning.

Flip the packaged fips.yaml templates to the new keys, add legacy/new/
scalar compat parse tests, and record the split and deprecations in the
CHANGELOG. Behavior-neutral; fmt/clippy clean, lib suite green.
2026-07-09 15:43:31 +00:00
Johnathan Corgan
0b2212e1e8 proto: bound powi test drift by relative tolerance, not ULP
The previous portability fix bounded the powi test's drift from std::powi
at an absolute ULP count, but Windows/MSVC drift grows with the exponent
(3 ULP by exp=14), so no fixed ULP bound is portable.

Replace the ULP oracle with a loose relative-tolerance sanity sweep
(1e-11), which any sane libm clears by ~1000x regardless of exponent
while still catching a grossly wrong impl. Add a golden-bit pin for the
exp=14 case that exposed the growth. The powi function is unchanged; the
golden-bit determinism pins already passed on Windows.
2026-07-09 01:30:18 +00:00
Johnathan Corgan
b2ce7cd3c8 proto: make the powi guard test portable across platforms
The powi guard test asserted our square-and-multiply result was
bit-for-bit equal to f64::powi, which failed the Windows unit-test job by
one ULP. f64::powi is not portably bit-stable: Linux and macOS lower it
to compiler-rt's __powidf2, but Windows/MSVC rounds differently.

Our square-and-multiply is pure IEEE-754 f64 multiplication and is
therefore deterministic across every platform, which is the property that
actually matters for mesh nodes to agree on bloom FPR and backoff timing
regardless of OS. Replace the std bit-equality assertion with golden-bit
pins on our own output (locking that cross-platform determinism) plus a
bounded-drift sanity sweep against std::powi. The powi function itself is
unchanged.
2026-07-09 01:19:49 +00:00
Johnathan Corgan
e3e03f6a5d proto: rename the mesh discovery subsystem module from discovery to lookup
Rename src/proto/discovery to src/proto/lookup and bring the module's naming
onto the lookup stem, matching its concept: a mesh lookup of a node's
coordinates from its pubkey, sent into the mesh as a bloom-filter-guided
multicast request that returns a unicast response with the coordinates.

- the module directory and its declaration (proto::discovery -> proto::lookup)
- exported types: DiscoveryAction -> LookupAction, DiscoveryBackoff ->
  LookupBackoff, DiscoveryForwardRateLimiter -> LookupForwardRateLimiter,
  MAX_RECENT_DISCOVERY_REQUESTS -> MAX_RECENT_LOOKUP_REQUESTS, and the Discovery
  state struct -> Lookup
- internal terminology: doc comments, the "overlay-lookup" phrasing, the
  empty_discovery/suppressing_discovery test helpers, and the disc
  parameter/variable all take the lookup names
- the already-lookup-named wire types (LookupRequest/LookupResponse) are unchanged

Behavior-neutral: no wire bytes or decision logic change. Two references are
intentionally kept as "discovery": the still-named node::handlers::discovery
shell module and the node.discovery.* config keys, which belong to the broader
disambiguation of the shell, config, and metric surfaces still to come.
2026-07-08 21:54:24 +00:00
Johnathan Corgan
2b009196b5 proto: no_std hygiene for the fmt/alloc imports and the filter math
Bring the proto tree closer to a std+alloc-only posture, behavior unchanged on
every decision path:

- Sweep the remaining std::fmt and std::collections imports over to core::fmt
  and alloc::collections across the wire codecs and their tests. Subsystem files
  that shadow the core name with a child core module use the leading-colon
  ::core::fmt form. Imports only.

- Replace the two f64::powi calls (bloom false-positive-rate, FMP backoff timer)
  with a shared core-only square-and-multiply helper in proto/math, bit-identical
  to std::powi (a guard test pins this against every exponent the codecs reach),
  and route the diagnostic estimated-count natural log through libm::log. The FPR
  reject decision and the backoff timer are bit-for-bit unchanged; only the
  debug-only count estimate may differ by at most one ULP.
2026-07-08 19:00:25 +00:00
Johnathan Corgan
5d13090d8f proto: add a bounds-checked byte reader/writer and adopt it across the wire codecs
Introduce a shared proto/codec module with a cursor Reader (short reads fail with
MessageTooShort { expected: position + needed, got: total }) and an append Writer,
and adopt them across the seven subsystem wire codecs, replacing the repetitive
manual slicing, try_into, and from_le_bytes extraction. Each existing length check
maps to the reader (an up-front minimum becomes require at position zero, so the
per-field expected values, including the tree-announce expected 99, are reproduced
exactly); the bloom exact-length check stays explicit since it also rejects
over-long payloads. Encoded bytes and decode decisions are unchanged.
2026-07-08 19:00:25 +00:00
Johnathan Corgan
6538731176 proto: replace the string Malformed error with typed variants and drop thiserror
Replace Malformed(String) with a no_std-clean set: Malformed(&static str)
for the static decode diagnostics, BadSizeClass { got, max } for the two
bloom size-class checks, and typed sources BadCoord(CoordError) /
BadBloom(BloomError) for the coordinate and bloom construction failures. Add a
dedicated CoordError so proto/coord no longer depends upward on stp TreeError,
resolving the temporary inversion from the coordinate relocation. Drop the
thiserror derive from the four proto error types (Error, TreeError, BloomError,
CoordError) in favor of hand-rolled core::fmt::Display and core::error::Error
impls. thiserror remains in use elsewhere in the crate. Wire decode decisions
are unchanged; only the error type and its diagnostic text change.
2026-07-08 19:00:25 +00:00
Johnathan Corgan
9697026c81 proto: share a per-address rate limiter and backoff helper across subsystems
Hoist the two line-for-line-identical per-destination minimum-interval limiters
(discovery forward, routing error) into a shared PerAddrRateLimiter, and fold
the duplicated exponential-backoff math (discovery originator, FMP retry) into a
shared backoff_ms helper, both in a new proto/rate_limit module. The subsystem
limiters become thin delegating newtypes so should_forward/should_send and the
backoff call conventions are preserved. Behavior is byte-identical.
2026-07-08 19:00:25 +00:00
Johnathan Corgan
a2400d823f proto/stp: extract ParentDeclaration and relocate the coordinate type to a shared proto/coord module
Two behavior-neutral relocations, wire bytes unchanged:

- Move the ParentDeclaration type and its impls out of state.rs into a dedicated
  declaration.rs, re-exported at the same path, and relocate the inline wire.rs
  test module into tests/wire.rs alongside the other stp unit tests (reusing the
  shared test helpers). wire.rs drops to non-test code only.

- Move TreeCoordinate, CoordEntry, and the coordinate wire codec out of
  proto/stp into a shared proto/coord module (a peer of proto/link), re-exported
  from proto::stp so every existing import path keeps resolving unchanged.
  Coordinate is a shared addressing primitive with many non-stp consumers, so it
  no longer belongs under the spanning-tree subsystem. The codec carries a
  documented temporary dependency on stp::TreeError until a dedicated CoordError
  is introduced.
2026-07-08 19:00:25 +00:00
Johnathan Corgan
dc9334e725 proto/stp: hoist the parent flap/hold-down veto to the shell for a clock-free classify core
evaluate_parent no longer reads the injected clock: it returns a ParentEval of
Mandatory, Discretionary, or None, and the flap/hold-down veto is applied at the
edge via a new TreeState::is_switch_suppressed(now_ms), gating only the
discretionary arm. classify_announce/classify_periodic take the pre-computed
switch_suppressed bool instead of now_ms, so the whole classify ladder is
clock-free; the shell callers (tree announce/periodic re-eval, MMP first-RTT
re-eval, handle_parent_lost) compute the veto verdict. The no-coords parent
case stays discretionary (veto-gated) exactly as before. Behavior unchanged;
the veto tests now assert the moved responsibility.
2026-07-08 19:00:24 +00:00
Johnathan Corgan
309a91d293 proto/mmp: split state into per-role modules and dissolve the vestigial mmp shell
Reorganize the MMP subsystem, behavior unchanged throughout:

- Restore the pre-migration role split: move SenderState into sender.rs,
  ReceiverState (with its GapTracker helper) into receiver.rs, MmpMetrics and
  RrLog into metrics.rs, and PathMtuState into path_mtu.rs, leaving the Mmp
  aggregate plus the peer/session state in state.rs (1339 to 196 lines). Home
  the module constants in a new limits.rs and re-export them at the same paths.
  Pure code-motion with module rewiring; visibility unchanged.

- Dissolve the 97-line src/mmp shell, which held only MmpConfig and the
  monotonic mono_ms() clock: move MmpConfig into config/node.rs (re-exported as
  crate::config::MmpConfig), move mono_ms() into a new top-level src/time.rs
  shell time seam, repoint all callers, and delete src/mmp/. Also correct stale
  src/mmp/ doc-comment path labels to proto/mmp/ where they name the protocol
  primitives.
2026-07-08 19:00:24 +00:00
Johnathan Corgan
c1ddbf053c proto/discovery: home the recent-request cap and inject request_id from the shell
Two behavior-neutral discovery cleanups:

- Move MAX_RECENT_DISCOVERY_REQUESTS out of the node discovery handler into the
  discovery subsystem limits module and re-export it, keeping the two use sites
  unchanged. Same value and semantics; only its home moves.

- Remove the ambient rand draw from LookupRequest by deleting generate() and
  having the shell draw the random request_id and pass it into the existing
  new() constructor. Same per-request u64 draw, now at the shell, leaving the
  discovery codec free of ambient RNG reads.
2026-07-08 19:00:24 +00:00
Johnathan Corgan
b53db662c3 proto/bloom: add BloomFilter::fpr() and use it for the false-positive-rate
The false-positive-rate primitive (fill ratio raised to the hash count) was
inlined at three sites. Hoist it into a BloomFilter::fpr() method and call it
from all three; the threshold/policy logic stays at the call sites. Behavior
and result bytes are unchanged.
2026-07-08 19:00:24 +00:00
Johnathan Corgan
4ad5940114 proto/fsp: extract FSP session protocol into sans-IO layout, retire src/protocol
Migrate the FSP end-to-end session subsystem into src/proto/fsp/ following the
established sans-IO shape, and retire the src/protocol grab-bag now that FSP was
its last occupant.

Relocate the FSP session wire (node/session_wire.rs plus the FSP message types
from protocol/session.rs) into proto/fsp/wire.rs. Hoist the pure decision logic
into proto/fsp/core.rs over plain-data SessionSnapshots returning an ordered
FspAction list the shell drives: session-rekey policy, msg3-resend
classification, post-decrypt epoch reaction, setup/dual-init tie-break,
coords/path-MTU emit-policy, bounded pending-queue, and IPv6 ECN. The
crypto-owning SessionEntry stays shell-side in node/session.rs (matching the FMP
ActivePeer pattern); proto/fsp is wire + core + limits only, with no proto->noise
dependency and no crypto.

Move the coords helpers to proto/stp/ (they serialize TreeCoordinate), and split
SessionMessageType: the encrypted-inner 0x10-0x1F variants stay in proto/fsp/wire.rs
while the 0x20-0x2F routing signals become a new RoutingSignalType in
proto/routing/wire.rs. Migrate the session-MMP shell adapter, which continues to
drive proto/mmp/.

Retire src/protocol: LinkMessageType and SessionDatagram move to a new shared
proto/link.rs, ProtocolError becomes proto::Error (relocated verbatim), the
deprecated MessageType alias and the unimported PROTOCOL_VERSION are dropped, and
src/protocol/ is deleted along with its lib.rs module declaration.

Behavior-neutral: wire bytes unchanged, oracle tests pass unedited except
mod-path relocation; adds rekey/epoch characterization tests and pure
poll/emit-policy core tests.
2026-07-08 04:57:00 +00:00
Johnathan Corgan
4ed674ea8b proto/bloom: relocate bloom filter into sans-IO layout
Migrate the v1 bloom filter subsystem into src/proto/bloom/, matching the
discovery/routing/fmp/mmp/stp reference layout and completing the proto/
relocation series for the data/wire subsystems.

- wire.rs: the FilterAnnounce (0x20) codec, moved from protocol/filter.rs
- core.rs: the pure BloomFilter algorithm (hash/insert/contains/merge/
  as_bytes/from_bytes/estimated_count), moved from bloom/filter.rs
- state.rs: BloomState (per-peer inbound store, compute_outgoing_filter,
  the injected-clock send debounce), moved from bloom/state.rs
- limits.rs: the v1 sizing constants
- mod.rs: module wiring + the BloomError enum
- tests/: the unit suite split by target (core/state/wire), no inline tests

no_std+alloc hygiene: core::fmt over std::fmt, the tracing dependency
dropped from the pure filter, and std collections replaced with
BTreeMap/BTreeSet (NodeAddr: Ord) for deterministic iteration. The pure
filter combination stays a BloomState method; the two irreducible shell
gathers (peer_inbound_filters, build_filter_announce) remain in the async
shell. Wire bytes and observable behavior are unchanged; full local CI
green (36/36) including the bloom-storm chaos gate.
2026-07-07 23:10:12 +00:00
Johnathan Corgan
a67801099d proto/stp: sans-IO spanning-tree state machine
Migrate the full non-async spanning-tree surface into proto/stp/, mirroring the
discovery/routing/fmp/mmp conversions. The classification ladder (parent-switch /
self-root / loop-drop / ancestry-update / periodic-rebroadcast / parent-lost) moves
out of the async node handlers into a pure Stp classify layer returning a
TreeDecision the shell drives, with effect ordering and per-arm invalidation
preserved verbatim. src/tree/ relocates wholesale: TreeState + ParentDeclaration data
+ coordinates into proto/stp/{state,coordinate}, the flap-dampening / hold-down
cluster into a FlapDampener in limits.rs, and the wire codec into wire.rs. The clock
is injected as u64 (wall-clock secs for the escaping declaration timestamp, monotonic
ms for the dampening timers via mmp::mono_ms); declaration crypto is field-partitioned
so sign/verify/hash run in the shell while the in-core modules carry data +
signing_bytes only. Peer maps/sets move to BTree; core/state/coordinate/limits are
core+alloc clean, with wire.rs the one std-tethered file. Behavior-neutral:
characterization tests added for the handler decision arms; convergence suite and
ci-local (36/36) green.
2026-07-07 17:07:33 +00:00
Johnathan Corgan
50a595a0ed proto/mmp: sans-IO metrics-reporting state machine 2026-07-07 09:24:17 +00:00
Johnathan Corgan
4802792e38 proto/fmp: sans-IO connection-lifecycle state machine 2026-07-07 06:20:02 +00:00
Johnathan Corgan
9ea57b483a proto/routing: sans-IO transit + hop-selection state machine 2026-07-07 04:12:05 +00:00
Johnathan Corgan
e03b206f62 proto/discovery: sans-IO state-machine migration + no_std reductions
Migrate the FMP discovery decision logic out of the async handlers into
synchronous, runtime-agnostic sans-IO state machines owned by the protocol
structs, with I/O pushed to the edges. Pulls the full decision surface into a
pure core (backoff, rate-limit, planners, response routing), consolidates the
tests into a per-module tree with a shared crate testutil, and injects a u64
wall-clock so the core is free of Instant and std time.

Also brings the module toward no_std+alloc: the four discovery maps use
alloc::collections::BTreeMap (HashMap's RandomState is std-only), Arc is spelled
alloc::sync::Arc, the backoff-reset log lives in the shell (the core returns the
cleared count so observability stays out of the pure core), and the crate root
names alloc directly. The one remaining tether is ProtocolError's
std::error::Error coupling in the wire codec.

First subsystem of the broader sans-IO refactor; establishes the extraction
patterns and conventions carried forward to the remaining protocols.
2026-07-05 21:59:21 +00:00
Johnathan Corgan
1dbfefc9d0 Merge branch 'maint' 2026-07-02 22:03:41 +00:00
Johnathan Corgan
793f844448 Merge branch 'maint' 2026-06-29 16:47:43 +00:00
Johnathan Corgan
243bd7985a Merge branch 'maint' 2026-06-29 14:23:57 +00:00
Johnathan Corgan
30c5808e09 Merge maint into master after the v0.4.0 rollover
maint carries only its 0.4.1-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
2026-06-27 18:35:41 +00:00
Johnathan Corgan
3c9a629ad4 Open 0.5.0-dev cycle on master after v0.4.0 release
Bump the version to 0.5.0-dev and update the status badge.
2026-06-27 18:28:52 +00:00
237 changed files with 33351 additions and 15632 deletions

View File

@@ -285,6 +285,7 @@ jobs:
"$FILES_DIR/etc/fips/firewall.sh"
"$FILES_DIR/etc/hotplug.d/net/99-fips"
"$FILES_DIR/etc/uci-defaults/90-fips-setup"
"$FILES_DIR/usr/bin/fips-mesh-setup"
)
fail=0
for f in "${TARGETS[@]}"; do
@@ -404,6 +405,7 @@ jobs:
./usr/bin/fipsctl
./usr/bin/fipstop
./usr/bin/fips-gateway
./usr/bin/fips-mesh-setup
./etc/init.d/fips
./etc/init.d/fips-gateway
./etc/fips/fips.yaml
@@ -717,6 +719,7 @@ jobs:
for path in \
usr/bin/fips usr/bin/fipsctl usr/bin/fipstop usr/bin/fips-gateway \
usr/bin/fips-mesh-setup \
etc/init.d/fips etc/init.d/fips-gateway \
etc/fips/fips.yaml etc/fips/firewall.sh etc/dnsmasq.d/fips.conf \
etc/sysctl.d/fips-gateway.conf etc/sysctl.d/fips-bridge.conf \

View File

@@ -11,6 +11,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- The Ethernet transport's per-interface `discovery` flag was renamed to
`listen` (`transports.ethernet.*`) to match the symmetric `announce`
(transmit) / `listen` (receive) neighbor-beacon vocabulary. The old
`discovery:` key is still accepted via a serde alias, so deployed configs
continue to load unchanged; `Config::to_yaml()` re-emits it under the
canonical `listen:` name. Update your `fips.yaml` to `listen:`.
- The mesh-lookup control-metrics family is now emitted under the key
`lookup` in `fipsctl stats metrics` and `show routing`. The former key
`discovery` is still emitted as a deprecated alias carrying identical
counters; update dashboards and alerts to read `lookup`.
- The overloaded `node.discovery.*` config table was split into
`node.lookup.*` (mesh-lookup scalars: `ttl`, `attempt_timeouts_secs`,
`recent_expiry_secs`, `backoff_base_secs`, `backoff_max_secs`,
`forward_min_interval_secs`) and `node.rendezvous.*` (peer rendezvous:
`nostr.*`, `lan.*`). A deployed `node.discovery:` block still loads and is
folded into the new tables with a one-time deprecation warning; migrate your
`fips.yaml` to the new keys.
### Deprecated
- The `discovery` metric-family key (control-socket JSON). It is dual-emitted
alongside the new `lookup` key during a migration window and will be removed.
Migrate dashboards/alerts from `discovery.*` to `lookup.*`.
- The `node.discovery.*` config table. Its keys were split into `node.lookup.*`
(mesh-lookup) and `node.rendezvous.*` (peer rendezvous). A legacy
`node.discovery:` block still applies for now with a deprecation warning and
will be removed; migrate to `node.lookup.*` / `node.rendezvous.*`.
- The Ethernet `transports.ethernet.discovery` flag, renamed to
`transports.ethernet.listen`. The old key is still accepted via a serde
alias and will be removed at the v2 cutover; migrate to `listen`.
### Fixed
## [0.4.0] - 2026-06-27

3
Cargo.lock generated
View File

@@ -1074,7 +1074,7 @@ checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
[[package]]
name = "fips"
version = "0.4.1-dev"
version = "0.5.0-dev"
dependencies = [
"arc-swap",
"bech32",
@@ -1087,6 +1087,7 @@ dependencies = [
"hex",
"hkdf",
"libc",
"libm",
"mdns-sd",
"nostr",
"nostr-sdk",

View File

@@ -1,6 +1,6 @@
[package]
name = "fips"
version = "0.4.1-dev"
version = "0.5.0-dev"
edition = "2024"
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
license = "MIT"
@@ -17,6 +17,7 @@ secp256k1 = { version = "0.30", features = ["rand", "global-context"] }
sha2 = "0.10"
hkdf = "0.12"
ring = "0.17"
libm = "0.2"
rand = "0.10.1"
crossbeam-channel = "0.5"
thiserror = "2.0"
@@ -108,3 +109,8 @@ path = "src/bin/fips-gateway.rs"
[[bin]]
name = "fipstop"
path = "src/bin/fipstop/main.rs"
[[bench]]
name = "routing_next_hop"
path = "benches/routing_next_hop.rs"
harness = false

View File

@@ -3,7 +3,7 @@
![banner](docs/logos/fips_banner.png)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)
[![Status](https://img.shields.io/badge/status-v0.4.1--dev-green.svg)](#status--roadmap)
[![Status](https://img.shields.io/badge/status-v0.5.0--dev-green.svg)](#status--roadmap)
A self-organizing encrypted mesh network built on Nostr identities,
capable of operating over arbitrary transports without central
@@ -210,10 +210,10 @@ testing/ Docker-based integration test harnesses + chaos simulation
## Status & roadmap
FIPS is at **v0.4.1-dev** on the `maint` branch.
FIPS is at **v0.5.0-dev** on the `master` branch.
[v0.4.0](https://github.com/jmcorgan/fips/releases/tag/v0.4.0) has
shipped; this line carries patch-level fixes for the 0.4.x series. The
core protocol works end-to-end over
shipped; this development line continues the testing-and-polishing
track toward v0.5.0. The core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
mesh of thousands of nodes. v0.4.0 added the Nym mixnet transport and
mDNS LAN discovery alongside the existing Nostr-mediated peer discovery,

365
benches/routing_next_hop.rs Normal file
View File

@@ -0,0 +1,365 @@
//! Micro-benchmark quantifying the per-forwarded-packet heap-allocation cost
//! of the routing next-hop candidate-assembly path.
//!
//! `find_next_hop` runs once per forwarded data packet. Its sans-IO core
//! assembles a `Vec<Candidate>` by enumerating every peer through the
//! `RoutingView` seam: `peer_addrs()` materializes a `Vec<NodeAddr>` of all
//! peers, the survivors are snapshotted (each cloning its `TreeCoordinate`),
//! and the result is collected into a second `Vec`. This bench measures that
//! per-call allocation against a fused zero-alloc reference that iterates the
//! peer map directly and borrows coordinates instead of cloning.
//!
//! Visibility caveat: the production `routing_candidates` / `select_best_candidate`
//! / `RoutingView` / `Candidate` are `pub(crate)` (src/proto/routing/core.rs)
//! and are not re-exported at the crate root, so an external bench crate cannot
//! name them. Rather than change production visibility, this file reproduces
//! that path verbatim over the real public `NodeAddr` / `TreeCoordinate` /
//! `CoordEntry` / `BloomFilter` types with the same iterator chain and the same
//! `HashMap`-backed view the shell uses (src/node/mod.rs NodeRoutingView). The
//! allocation behavior is therefore identical to production by construction;
//! only the symbol identity differs.
use std::alloc::{GlobalAlloc, Layout, System};
use std::collections::HashMap;
use std::hint::black_box;
use std::sync::atomic::{AtomicUsize, Ordering};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use fips::{BloomFilter, NodeAddr, TreeCoordinate};
// ---------------------------------------------------------------------------
// Counting global allocator: bumps a process-global counter on every heap
// allocation operation (alloc / alloc_zeroed / realloc). Sampled tightly and
// single-threaded in `report_allocs` so no unrelated allocations are captured.
// ---------------------------------------------------------------------------
struct CountingAlloc;
static ALLOCS: AtomicUsize = AtomicUsize::new(0);
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
const PEER_COUNTS: [usize; 4] = [8, 32, 128, 256];
/// Fraction of peers whose bloom filter reports the destination reachable.
const REACH_NUMERATOR: usize = 1;
const REACH_DENOMINATOR: usize = 2;
/// Tree depth for synthetic coordinates (self..root), a realistic mesh depth.
const COORD_DEPTH: usize = 8;
// ---------------------------------------------------------------------------
// Reproduction of the pub(crate) routing seam (src/proto/routing/core.rs).
// ---------------------------------------------------------------------------
trait RoutingView {
fn peer_addrs(&self) -> Vec<NodeAddr>;
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool;
fn peer_can_send(&self, peer: &NodeAddr) -> bool;
fn peer_link_cost(&self, peer: &NodeAddr) -> f64;
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate>;
}
struct Candidate {
addr: NodeAddr,
can_send: bool,
link_cost: f64,
coords: Option<TreeCoordinate>,
}
/// Verbatim from `routing::routing_candidates` (core.rs). Allocates the
/// `peer_addrs` Vec, clones each survivor's coords, and collects into a Vec.
fn routing_candidates(rv: &impl RoutingView, dest: &NodeAddr) -> Vec<Candidate> {
rv.peer_addrs()
.into_iter()
.filter(|peer| rv.peer_may_reach(peer, dest))
.map(|peer| Candidate {
can_send: rv.peer_can_send(&peer),
link_cost: rv.peer_link_cost(&peer),
coords: rv.peer_coords(&peer),
addr: peer,
})
.collect()
}
/// Verbatim from `routing::select_best_candidate` (core.rs). Pure, no alloc.
fn select_best_candidate(
candidates: &[Candidate],
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(&Candidate, f64, usize)> = None;
for candidate in candidates {
if !candidate.can_send {
continue;
}
let cost = candidate.link_cost;
let dist = candidate
.coords
.as_ref()
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((_, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost
&& dist == *best_dist
&& candidate.addr < best.as_ref().unwrap().0.addr)
}
};
if dominated {
best = Some((candidate, cost, dist));
}
}
best.map(|(candidate, _, _)| candidate.addr)
}
// ---------------------------------------------------------------------------
// Bench-local view, HashMap-backed exactly like src/node/mod.rs NodeRoutingView.
// ---------------------------------------------------------------------------
struct BenchPeer {
bloom: BloomFilter,
can_send: bool,
link_cost: f64,
}
struct BenchView {
peers: HashMap<NodeAddr, BenchPeer>,
coords: HashMap<NodeAddr, TreeCoordinate>,
}
impl RoutingView for BenchView {
fn peer_addrs(&self) -> Vec<NodeAddr> {
self.peers.keys().copied().collect()
}
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.bloom.contains(dest))
}
fn peer_can_send(&self, peer: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.can_send)
}
fn peer_link_cost(&self, peer: &NodeAddr) -> f64 {
self.peers.get(peer).map_or(f64::INFINITY, |p| p.link_cost)
}
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate> {
self.coords.get(peer).cloned()
}
}
/// Zero-alloc reference: what an iterator/visitor seam would do. Iterates the
/// peer map directly, fuses the may_reach + can_send filters, borrows coords
/// instead of cloning, and tracks the best hop inline. No Vec, no coord clone.
fn resolve_next_hop_zeroalloc(
view: &BenchView,
dest: &NodeAddr,
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(NodeAddr, f64, usize)> = None;
for (addr, peer) in &view.peers {
if !peer.bloom.contains(dest) {
continue;
}
if !peer.can_send {
continue;
}
let cost = peer.link_cost;
let dist = view
.coords
.get(addr)
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((best_addr, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost && dist == *best_dist && *addr < *best_addr)
}
};
if dominated {
best = Some((*addr, cost, dist));
}
}
best.map(|(addr, _, _)| addr)
}
// ---------------------------------------------------------------------------
// Scenario construction.
// ---------------------------------------------------------------------------
fn addr(tag: u8, i: u16) -> NodeAddr {
let mut b = [0u8; 16];
b[0] = tag;
b[1..3].copy_from_slice(&i.to_le_bytes());
NodeAddr::from_bytes(b)
}
/// A depth-`COORD_DEPTH` coordinate whose leaf is `leaf`, sharing a fixed
/// interior path and root with `shared_tag`. Peers built with the dest's
/// shared_tag sit close to the destination (distance 2); a distinct shared_tag
/// sits far (near the root), modeling our own position.
fn coord(leaf: NodeAddr, shared_tag: u8) -> TreeCoordinate {
let mut path = Vec::with_capacity(COORD_DEPTH);
path.push(leaf);
for level in 1..(COORD_DEPTH - 1) {
path.push(addr(shared_tag, level as u16));
}
path.push(addr(9, 0)); // common root
TreeCoordinate::from_addrs(path).expect("valid coord path")
}
struct Scenario {
view: BenchView,
dest: NodeAddr,
dest_coords: TreeCoordinate,
my_coords: TreeCoordinate,
}
impl Scenario {
fn new(n: usize) -> Self {
let dest = addr(2, 0);
// Destination path uses interior tag 4; peers reuse tag 4 so survivors
// are close to the destination. Our own coords use tag 5 (far).
let dest_coords = coord(dest, 4);
let my_coords = coord(addr(6, 0), 5);
let mut peers = HashMap::new();
let mut coords = HashMap::new();
for i in 0..n {
let paddr = addr(1, i as u16);
let mut bloom = BloomFilter::new();
// Realistic fill: a handful of unrelated reachable addrs.
for f in 0..4u16 {
bloom.insert(&addr(7, i as u16 * 4 + f));
}
// A controlled fraction advertise the destination as reachable.
if (i % REACH_DENOMINATOR) < REACH_NUMERATOR {
bloom.insert(&dest);
}
peers.insert(
paddr,
BenchPeer {
bloom,
can_send: true,
link_cost: 1.0 + (i as f64) * 0.01,
},
);
// Peers share the destination's interior path (tag 4) → close.
coords.insert(paddr, coord(paddr, 4));
}
Self {
view: BenchView { peers, coords },
dest,
dest_coords,
my_coords,
}
}
fn survivors(&self) -> usize {
self.view
.peers
.values()
.filter(|p| p.bloom.contains(&self.dest))
.count()
}
}
// ---------------------------------------------------------------------------
// Allocation-per-call report (printed once, before criterion timing).
// ---------------------------------------------------------------------------
fn count_allocs<T>(iters: usize, mut f: impl FnMut() -> T) -> f64 {
for _ in 0..8 {
black_box(f());
}
let start = ALLOCS.load(Ordering::Relaxed);
for _ in 0..iters {
black_box(f());
}
let end = ALLOCS.load(Ordering::Relaxed);
(end - start) as f64 / iters as f64
}
fn report_allocs() {
const ITERS: usize = 2000;
println!("\n=== allocations per call (heap alloc ops: alloc+alloc_zeroed+realloc) ===");
println!(
"{:>6} {:>10} {:>16} {:>16}",
"peers", "survivors", "current/call", "zero-alloc/call"
);
for &n in &PEER_COUNTS {
let s = Scenario::new(n);
let survivors = s.survivors();
let current = count_allocs(ITERS, || {
let cands = routing_candidates(&s.view, &s.dest);
select_best_candidate(&cands, &s.dest_coords, &s.my_coords)
});
let zero = count_allocs(ITERS, || {
resolve_next_hop_zeroalloc(&s.view, &s.dest, &s.dest_coords, &s.my_coords)
});
println!("{n:>6} {survivors:>10} {current:>16.2} {zero:>16.2}");
}
println!();
}
fn bench_next_hop(c: &mut Criterion) {
report_allocs();
let mut group = c.benchmark_group("find_next_hop");
for &n in &PEER_COUNTS {
let scenario = Scenario::new(n);
group.bench_with_input(BenchmarkId::new("current_alloc", n), &n, |b, _| {
b.iter(|| {
let cands = routing_candidates(&scenario.view, &scenario.dest);
black_box(select_best_candidate(
&cands,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
group.bench_with_input(BenchmarkId::new("zero_alloc_ref", n), &n, |b, _| {
b.iter(|| {
black_box(resolve_next_hop_zeroalloc(
&scenario.view,
&scenario.dest,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
}
group.finish();
}
criterion_group! {
name = benches;
config = Criterion::default().sample_size(50);
targets = bench_next_hop
}
criterion_main!(benches);

View File

@@ -477,7 +477,7 @@ The TXT record carries three keys (`src/discovery/lan/mod.rs:47-55`):
Once per node tick, the node drains browser events and acts on them in
`poll_lan_discovery()` (`src/node/lifecycle.rs:907`, called from
`src/node/handlers/rx_loop.rs:266`). For each discovered peer it finds
`src/node/dataplane/rx_loop.rs:266`). For each discovered peer it finds
a UDP transport whose family matches the peer address, parses the
`npub` into a `PeerIdentity`, skips peers it is already connected to or
currently connecting to, and otherwise initiates a connection.

View File

@@ -262,7 +262,7 @@ UDP (1500 vs 1472 MTU).
- **No IP dependency**: Operates below the IP layer. Nodes on the same
Ethernet segment can communicate without IP addresses or routing
infrastructure
- **Broadcast discovery**: Nodes discover each other via periodic beacon
- **Broadcast neighbor detection**: Nodes discover each other via periodic beacon
broadcasts on the shared medium, with no static peer configuration required
- **Higher MTU**: Standard Ethernet frames carry 1500 bytes of payload,
yielding an effective FIPS MTU of 1499 after the frame type prefix
@@ -293,7 +293,7 @@ socket.
| Addressing | 6-byte MAC address |
| Platform | Linux only (`CAP_NET_RAW` required) |
### Beacon Discovery
### Neighbor Beacons
Ethernet nodes discover peers via broadcast beacons sent to
ff:ff:ff:ff:ff:ff. Each beacon is a 34-byte frame containing the sender's
@@ -301,7 +301,7 @@ x-only public key. Receiving nodes extract the MAC source address from the
frame and the public key from the payload, then report the discovered peer
to FMP.
Four configuration flags control discovery behavior — `discovery`
Four configuration flags control neighbor behavior — `listen`
(listen for beacons), `announce` (broadcast beacons), `auto_connect`
(initiate handshakes to discovered peers), and `accept_connections`
(accept inbound handshakes). The flag table and per-flag defaults
@@ -310,13 +310,13 @@ under `transports.ethernet.*`.
A typical discoverable node sets `announce`, `auto_connect`, and
`accept_connections` all true. A passive listener uses just
`discovery: true` to observe the network without announcing itself.
`listen: true` to observe the network without announcing itself.
### WiFi Compatibility
WiFi interfaces in infrastructure (managed) mode work transparently for
unicast — the mac80211 subsystem handles frame translation between 802.11
and 802.3. Broadcast beacon discovery is unreliable in managed mode because
and 802.3. Broadcast neighbor detection is unreliable in managed mode because
access points commonly isolate clients from each other's broadcast traffic.
Startup logging:
@@ -895,7 +895,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| --------- | ------ | ----- |
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
| TCP/IP | **Implemented** | FMP header-based framing, non-blocking connect, per-connection MSS MTU |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, neighbor beacons, Linux only |
| WiFi | **Implemented** (via Ethernet transport, infrastructure mode) | mac80211 translates 802.11↔802.3; broadcast beacons unreliable through APs |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| Nym | **Implemented** | Outbound-only SOCKS5 through nym-socks5-client, mixnet anonymity, IP/hostname addressing |

View File

@@ -25,4 +25,5 @@ X" to "X is done".
| [persistent-identity.md](persistent-identity.md) | Provision a stable Nostr keypair so the node keeps the same npub across restarts |
| [host-aliases.md](host-aliases.md) | Use shortnames (`test-us01.fips`, `my-laptop.fips`) instead of full npubs by editing `/etc/fips/hosts` or setting peer aliases |
| [set-up-bluetooth-peer.md](set-up-bluetooth-peer.md) | Configure a Bluetooth Low Energy peer link |
| [set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md) | Link OpenWrt FIPS routers over an open 802.11s radio backhaul (FIPS provides encryption, authentication, and routing) |
| [diagnose-mtu-issues.md](diagnose-mtu-issues.md) | Triage MTU-shaped failures and rule out their imposters (bufferbloat, transport saturation) |

View File

@@ -0,0 +1,247 @@
# Set Up an 802.11s Mesh Backhaul (OpenWrt)
Link FIPS routers over radio — no cables, no APs, no shared
infrastructure — by running the Ethernet transport on an open 802.11s
mesh interface. The radio layer provides nothing but L2 frames to
direct neighbors; FIPS provides everything else: encryption and
authentication (Noise IK), peer discovery (Ethernet beacons), and
routing (the spanning tree).
For the transport design, see
[../design/fips-transport-layer.md](../design/fips-transport-layer.md).
For all `transports.ethernet.*` configuration keys, see
[../reference/configuration.md](../reference/configuration.md).
## Why open, why forwarding off
Two deliberate choices distinguish this from a stock 802.11s setup:
- **`encryption none`** — the mesh is open on purpose. Every FIPS peer
link is already authenticated and encrypted by the Noise IK
handshake, so SAE at L2 would duplicate that work, add a shared
credential to provision across routers, and (on ath10k) force the
firmware into its slower raw Tx/Rx mode. A stranger can form an
802.11s peering with your router, but their frames die at the FIPS
handshake — the same security model as mDNS and BLE discovery, where
the advert is only a hint and the handshake is the authentication.
What you concede: L2 metadata (MAC addresses, frame sizes) is
visible in the air, and a hostile radio can burn airtime — both true
of any radio link regardless of L2 encryption.
- **`mesh_fwding 0`** — disables 802.11s's own HWMP routing so each
mesh link is a plain neighbor link. FIPS is the routing layer; two
routing layers would fight, and broadcast discovery beacons would
flood the whole mesh instead of reaching direct neighbors only.
The interface is **not** bridged into `br-lan` — the FIPS Ethernet
transport binds it directly.
## When to use
- Two or more OpenWrt FIPS routers within radio range of each other,
where running cable is impractical.
- You want the mesh segment to keep working with zero shared
credentials or per-site configuration ("flash and drop in").
It is **not** for connecting phones or laptops — client devices
cannot join an 802.11s mesh. They enter the mesh through a normal AP
on the same router (see constraints below), or over BLE.
## Requirements
- OpenWrt 22.03+ with the FIPS package installed.
- A radio whose driver supports mesh point interfaces. Check with:
```sh
iw list | grep -A 10 "Supported interface modes" | grep "mesh point"
```
The mainstream OpenWrt chips (ath9k, ath10k, mt76) all qualify.
- Ideally a dual- or tri-band router, so one band can be dedicated to
the backhaul (see constraints).
## Step 1 — create the mesh interface(s)
On **each** router, run the helper once per radio you want in the
backhaul:
```sh
fips-mesh-setup radio1
```
This creates an open 802.11s interface with mesh ID `fips-mesh` and
HWMP forwarding off, attaches it to an unmanaged netifd interface (no
IP configuration — none is needed), uncomments the matching `meshN`
transport entry in `/etc/fips/fips.yaml` (see Step 2), and reloads the
radio. Interfaces are named by radio index: `radio0` → `fips-mesh0`,
`radio1` → `fips-mesh1`. Pass a second argument to use a different
mesh ID.
Note: the helper runs `wifi reload`, which re-applies the whole
wireless config and so briefly drops every client AP on all radios for
a few seconds. `fips-mesh-setup remove` reloads the same way. Expect
the blip if clients are connected.
On dual-band routers, meshing **both** bands is worth it: 2.4 GHz
reaches further at lower rates, 5 GHz carries more over shorter
links. Note this is **failover, not multipath**: FIPS keeps one
active link per peer, so traffic uses one band at a time — the other
is a standby that re-establishes the peer if the active link dies
(detection via keepalive timeout, so a cutover takes seconds, not
milliseconds):
```sh
fips-mesh-setup radio0
fips-mesh-setup radio1
```
**Pin the same channel on every backhaul router, per band.** Mesh
points only peer on the same channel, and the mesh inherits whatever
the radio is set to — with `channel 'auto'` (the default on many
devices) each router picks its own and the mesh silently never forms.
The script prints the radio's current band and channel and warns on
`auto`:
```sh
uci set wireless.radio1.channel='36'
uci commit wireless && wifi reload
```
Prefer a non-DFS channel (3648 on 5 GHz): on DFS channels the radio
must wait ~60 s in CAC before transmitting after every reload.
Equivalent manual UCI (per radio), if you prefer to see what it does:
```sh
uci batch <<'EOF'
set wireless.fips_mesh_radio1=wifi-iface
set wireless.fips_mesh_radio1.device='radio1'
set wireless.fips_mesh_radio1.mode='mesh'
set wireless.fips_mesh_radio1.mesh_id='fips-mesh'
set wireless.fips_mesh_radio1.encryption='none'
set wireless.fips_mesh_radio1.mesh_fwding='0'
set wireless.fips_mesh_radio1.ifname='fips-mesh1'
set wireless.fips_mesh_radio1.network='fips_mesh_radio1'
set network.fips_mesh_radio1=interface
set network.fips_mesh_radio1.proto='none'
EOF
uci commit
wifi reload
```
## Step 2 — check the FIPS transport binding
The `fips.yaml` shipped in the OpenWrt package carries one transport
entry per radio, but **commented out** — so a stock install that never
runs this helper logs no per-boot "interface missing" warning.
`fips-mesh-setup` uncommented the matching `meshN` entry in Step 1, so
there is normally nothing to do here. If you maintain your own config
(or ran the manual UCI above instead of the helper), make sure the
entries are present and uncommented:
```yaml
transports:
ethernet:
mesh0:
interface: "fips-mesh0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
mesh1:
interface: "fips-mesh1"
discovery: true
announce: true
auto_connect: true
accept_connections: true
```
## Step 3 — restart the daemon (order matters)
```sh
/etc/init.d/fips restart
```
Restart fips **after** the mesh interface is up. A transport whose
interface is missing at startup is logged and skipped, not retried —
so if the daemon comes up before the radio, the mesh transport stays
dead until the next restart. (An interface that *vanishes and
returns* after startup is recovered automatically; only the missing-
at-startup case needs this ordering.)
## Verify
L2 first — the 802.11s peering, with a second configured router in
range:
```sh
iw dev fips-mesh0 station dump
```
You should see one station entry per neighbor router, with signal
levels. No entries means a radio problem, not a FIPS problem — triage
in this order:
1. **Channel mismatch** (the most common cause): compare
`iw dev fips-mesh0 info` on both routers — mesh ID *and* channel
must match exactly.
2. **The mesh interface never joined** — `iw dev fips-meshX info`
shows `type mesh point` but **no channel line**, and `station dump`
is empty. Usual cause: a client (`sta`) interface on the same
radio. A STA must follow its upstream AP's channel, the whole
radio follows the STA, and a mesh pinned to a different channel
silently stays down. Check for a STA sharing the radio
(`iw dev`, look for `type managed` on the same phy), compare
`iw dev <sta-iface> info | grep channel`, and re-pin the mesh
channel to match — on every backhaul router.
3. **Is the other router transmitting at all?**
```sh
iw dev fips-mesh0 scan | grep -i -B4 "MESH ID"
```
Its mesh ID visible → transmission works, peering is failing
(mesh ID typo, or one side has encryption set). Nothing visible →
check `wifi status` on the other router, remember the ~60 s DFS
CAC wait, and confirm the country code is set
(`uci get wireless.radio1.country`) — an unset regdomain can
block channels entirely.
4. `logread | grep -iE "mesh|fips-mesh0"` on both sides.
Then the FIPS layer on top:
```sh
logread | grep -i beacon # beacons flowing on the new transport
fipsctl show peers # neighbor authenticated and connected
fipsctl show links # link on the 'ethernet' transport
```
Discovery is automatic: each node beacons its pubkey every few
seconds, and `auto_connect` initiates the Noise handshake on first
sight.
## Constraints
- **Airtime is shared per radio.** All virtual interfaces on one
radio (AP + mesh) share one channel, and multi-hop forwarding on a
single radio roughly halves throughput per hop. On dual/tri-band
hardware, dedicate one band to `fips-mesh0` and serve clients on
the others.
- **AP + mesh coexistence is driver-dependent.** It works on the
mainstream chips (this is the standard Freifunk/Gluon setup), but
check `iw list` under "valid interface combinations" for your
hardware.
- **Clients can't join.** Phones and laptops reach the mesh through
the router's normal AP or via BLE — never through the 802.11s
interface.
- **Radio links are lossy.** A neighbor at the edge of range will
form an 802.11s peering yet deliver a fraction of its frames.
Expect link-quality effects that don't exist on wired Ethernet.
- **A client (STA) uplink on the same radio owns the channel.** The
STA must follow whatever channel its upstream AP uses; every other
interface on that radio follows the STA. A mesh pinned to a
different channel silently never joins, and it does **not** recover
when the STA disconnects — a `wifi reload` (plus a fips restart) is
needed. A *roaming* uplink (travel-router / hotspot-chasing setups)
is fundamentally incompatible with a fixed-channel mesh on the same
radio: dedicate the mesh to the radio the STA never uses, and treat
any mesh sharing a STA radio as best-effort.

View File

@@ -436,7 +436,7 @@ Requires `CAP_NET_RAW` or running as root. Linux only.
| `mtu` | u16 | *(auto)* | Override MTU. Default: interface MTU minus 3 (for frame type + length prefix) |
| `recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `discovery` | bool | `true` | Listen for discovery beacons from other nodes |
| `listen` | bool | `true` | Listen for neighbor beacons from other nodes |
| `announce` | bool | `false` | Broadcast announcement beacons on the LAN |
| `auto_connect` | bool | `false` | Auto-connect to discovered peers |
| `accept_connections` | bool | `false` | Accept incoming connection attempts from discovered peers |
@@ -450,7 +450,7 @@ transports:
ethernet:
lan:
interface: "eth0"
discovery: true
listen: true
announce: true
backbone:
interface: "eth1"
@@ -458,7 +458,7 @@ transports:
```
Each named instance operates independently with its own socket and
discovery state. The instance name is used in log messages and the
neighbor state. The instance name is used in log messages and the
`name()` method on the Transport trait.
### TCP (`transports.tcp.*`)
@@ -840,7 +840,7 @@ peers:
### Mixed UDP + Ethernet Example
A node bridging internet peers (UDP) and a local Ethernet segment with
beacon discovery:
neighbor beacons:
```yaml
node:
@@ -856,7 +856,7 @@ transports:
mtu: 1472
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
@@ -999,7 +999,7 @@ transports:
# mtu: null # null = interface MTU - 3 (typically 1497)
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# discovery: true # listen for beacons
# listen: true # listen for beacons
# announce: false # broadcast beacons
# auto_connect: false # connect to discovered peers
# accept_connections: false # accept inbound handshakes

View File

@@ -209,7 +209,7 @@ for the metadata-privacy model and the rejection of onion routing.
| --------- | --------------- | ------------ | ------ |
| UDP | None until `bind_addr` set | `0.0.0.0:2121` typical | Operator sets `transports.udp.bind_addr` |
| TCP | None until `bind_addr` set | None — outbound-only without bind | Operator sets `transports.tcp.bind_addr` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `discovery`, `announce`, `auto_connect`, `accept_connections` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `listen`, `announce`, `auto_connect`, `accept_connections` |
| Tor | None until `directory_service` configured | `127.0.0.1:8443` (loopback only) | Operator sets `transports.tor.directory_service` and configures `HiddenServiceDir` in `torrc` |
| BLE | Off by default | n/a | Operator enables `transports.ble.*` |
| Nostr discovery | Off by default | n/a (relay client, not a listener) | Operator sets `node.discovery.nostr.enabled: true` |

View File

@@ -71,7 +71,7 @@ supplies the rest:
- **Addressing**: the `fips0` adapter takes an `fd97:...` ULA
derived from the npub. No DHCP. No SLAAC. The address is
cryptographically tied to the identity.
- **Discovery**: each daemon broadcasts a small beacon on the
- **Neighbor detection**: each daemon broadcasts a small beacon on the
link advertising its npub; the other daemon's listener picks
it up and dials in over the same link.
- **Routing**: the FIPS mesh layer builds its own spanning tree
@@ -177,7 +177,7 @@ different interface names — that is normal.
Edit `/etc/fips/fips.yaml` on **both** nodes. Under
`transports:`, add an `ethernet:` block. The key settings are
the four discovery flags — both nodes must opt in to all four,
the four neighbor flags — both nodes must opt in to all four,
and they default to off:
```yaml
@@ -185,7 +185,7 @@ transports:
ethernet:
interface: "<eth>" # the name from Step 1
announce: true # broadcast our beacon on the link
discovery: true # listen for beacons (default; shown for clarity)
listen: true # listen for beacons (default; shown for clarity)
auto_connect: true # dial peers we discover
accept_connections: true # accept dial-ins from peers we discover
```
@@ -194,7 +194,7 @@ Each flag does one thing:
- `announce: true` — emit a small beacon every
`beacon_interval_secs` (default 30s) carrying our npub.
- `discovery: true` — listen for incoming beacons; populate a
- `listen: true` — listen for incoming beacons; populate a
candidate-peer list keyed by source MAC and observed npub.
- `auto_connect: true` — when we see a beacon from an npub
we have not yet peered with, initiate the outbound Noise
@@ -218,7 +218,7 @@ is "all four flags on both ends."
> lan:
> interface: "eth0"
> announce: true
> discovery: true
> listen: true
> auto_connect: true
> accept_connections: true
> dongle:
@@ -227,7 +227,7 @@ is "all four flags on both ends."
> # ...
> ```
>
> Each named instance runs its own socket and discovery state.
> Each named instance runs its own socket and neighbor state.
> A single ground-up link only needs the flat form shown
> first; named instances become useful when the same node
> bridges multiple physical segments.
@@ -390,7 +390,7 @@ What you do need on the AP side:
networks and "secure" enterprise APs ship with it on.
When client isolation is on, the AP refuses to forward
station-to-station frames — the broadcast beacons never
arrive at the other node, and discovery fails silently.
arrive at the other node, and neighbor detection fails silently.
If beacons aren't crossing, this is the first thing to
check.
@@ -401,7 +401,7 @@ adapter name.
### Bluetooth LE (experimental but works)
BLE is a separate transport (`transports.ble.*`) with its own
discovery model — L2CAP advertisements rather than raw L2
neighbor-detection model — L2CAP advertisements rather than raw L2
broadcasts. The shape of the tutorial is the same (advertise +
scan + auto-connect + accept), but the prerequisites are
different: BlueZ, `bluetoothd`, an HCI adapter, and the
@@ -424,7 +424,7 @@ Windows builds skip it.
a radio link), `CAP_NET_RAW`, and a few config flags on each
end are sufficient. The mesh supplies its own identity,
addressing, discovery, and routing.
- **Discovery is a four-flag opt-in.** `announce`, `discovery`,
- **Neighbor detection is a four-flag opt-in.** `announce`, `listen`,
`auto_connect`, and `accept_connections` each control one
thing; both ends must agree before a link will form.
- **The two modes coexist.** Overlay peers and ground-up peers

View File

@@ -178,7 +178,7 @@ The resolution itself happens at debug-log level, so you will
not see it in the default-level journal. The user-facing way to
confirm everything worked is `fipsctl show peers` in the next
step. (To watch the resolution in the journal, run the daemon
manually with `RUST_LOG=fips::discovery::nostr=debug`; not
manually with `RUST_LOG=fips::nostr=debug`; not
necessary for this tutorial.)
## Step 5: Verify the resolved endpoint

View File

@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -34,17 +43,17 @@ node:
# - "stun:stun.cloudflare.com:3478"
# - "stun:global.stun.twilio.com:3478"
#
# Optional mDNS-based LAN discovery for sub-second same-LAN pairing.
# Optional mDNS-based LAN rendezvous for sub-second same-LAN pairing.
# Opt-in (default false): default-off avoids a per-LAN identity
# broadcast on nodes that have deliberately disabled other discovery
# broadcast on nodes that have deliberately disabled other rendezvous
# channels, and avoids any multicast surprise on upgrade. Requires an
# operational UDP transport (the advertised port is the one peers dial).
# lan:
# enabled: false
# # Optional application/network scope carried in the LAN-only TXT
# # record. Browsers that set a scope ignore adverts for other scopes.
# # Kept separate from the Nostr discovery `app` tag so relay-visible
# # adverts can stay generic while LAN discovery stays per-private-network.
# # Kept separate from the Nostr rendezvous `app` tag so relay-visible
# # adverts can stay generic while LAN rendezvous stays per-private-network.
# # scope: "lab-floor-3"
# # Advanced: overrides the mDNS service type. Leave unset in normal
# # use — only needed to run multiple isolated services on one
@@ -89,7 +98,7 @@ transports:
# Ethernet transport — uncomment and set your interface name.
# ethernet:
# interface: "eth0"
# discovery: true
# listen: true
# announce: true
# auto_connect: true
# accept_connections: true
@@ -147,5 +156,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect

View File

@@ -182,6 +182,7 @@ install -m 0755 "$RELEASE_DIR/fips" "$STAGE_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$STAGE_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$STAGE_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$STAGE_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$STAGE_DIR/usr/bin/fips-mesh-setup"
install -d "$STAGE_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$STAGE_DIR/etc/init.d/fips"

View File

@@ -96,6 +96,9 @@ define Package/fips/install
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fipstop $(1)/usr/bin/fipstop
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fips-gateway $(1)/usr/bin/fips-gateway
# 802.11s mesh backhaul setup helper
$(INSTALL_BIN) $(CURDIR)/files/usr/bin/fips-mesh-setup $(1)/usr/bin/fips-mesh-setup
# procd init script
$(INSTALL_DIR) $(1)/etc/init.d
$(INSTALL_BIN) $(CURDIR)/files/etc/init.d/fips $(1)/etc/init.d/fips

View File

@@ -14,6 +14,7 @@ For ad-hoc deployment without the build system, see
| `/usr/bin/fipsctl` | CLI control tool (`fipsctl show peers`, `fipsctl show links`, …) |
| `/usr/bin/fipstop` | Live TUI dashboard |
| `/usr/bin/fips-gateway` | Outbound LAN gateway service (not started by default) |
| `/usr/bin/fips-mesh-setup` | Opt-in helper — creates an open 802.11s mesh interface for router↔router backhaul |
| `/etc/init.d/fips` | procd service for the daemon (auto-start, crash respawn) |
| `/etc/init.d/fips-gateway` | procd service for the gateway (disabled by default) |
| `/etc/fips/fips.yaml` | Node configuration (edit before first start) |

View File

@@ -161,6 +161,7 @@ install -m 0755 "$RELEASE_DIR/fips" "$DATA_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$DATA_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$DATA_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$DATA_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$DATA_DIR/usr/bin/fips-mesh-setup"
install -d "$DATA_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$DATA_DIR/etc/init.d/fips"

View File

@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -74,23 +83,49 @@ transports:
ethernet:
wan:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
wwan:
interface: "phy0-sta0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
lan:
interface: "br-lan"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
# 802.11s mesh backhaul between FIPS routers. These entries ship
# commented out so a stock install that never creates fips-mesh*
# logs no per-boot "interface missing" bind warning. Running
# 'fips-mesh-setup <radio>' creates the interface AND uncomments the
# matching block here (once per radio; radio0 -> fips-mesh0, radio1 ->
# fips-mesh1); 'fips-mesh-setup remove' re-comments it. Restart fips
# after — a transport whose interface is missing at startup is skipped,
# not retried. Dual-band routers can mesh on both bands at once —
# failover, not multipath: FIPS keeps one active link per peer, the
# other band stands by. The mesh runs OPEN (no SAE) with 802.11s
# forwarding off: FIPS's Noise handshake is the encryption and
# authentication, and FIPS is the routing layer. See
# docs/how-to/set-up-80211s-mesh-backhaul.md.
# mesh0:
# interface: "fips-mesh0"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# mesh1:
# interface: "fips-mesh1"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# Bluetooth Low Energy transport — requires BlueZ and the 'ble' feature.
# ble:
# adapter: "hci0"
@@ -121,5 +156,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect

View File

@@ -0,0 +1,259 @@
#!/bin/sh
# fips-mesh-setup — configure open 802.11s mesh interfaces for FIPS backhaul.
#
# Usage:
# fips-mesh-setup <radio> [mesh-id] e.g. fips-mesh-setup radio1
# fips-mesh-setup remove [radio] no radio: remove all instances
#
# Creates a mesh-point interface on the given radio and leaves everything
# above L2 to FIPS. Run once per radio: dual-band routers can mesh on both
# bands at once (2.4 GHz reaches further, 5 GHz carries more). Note this is
# failover, not multipath — FIPS keeps one active link per peer; the other
# band stands by and reconnects the peer if the active link dies.
#
# - encryption 'none' — the mesh is OPEN on purpose. FIPS's Noise IK
# handshake authenticates and encrypts every peer link, so SAE would
# only duplicate that (and on ath10k it forces the slower raw Tx/Rx
# firmware mode). A stranger can form an 802.11s peering but cannot
# pass the FIPS handshake.
# - mesh_fwding '0' — disables 802.11s HWMP forwarding so each mesh
# link is a plain L2 neighbor link. FIPS is the routing layer; two
# routing layers would fight.
#
# Interfaces are named per radio index (radio0 -> fips-mesh0, radio1 ->
# fips-mesh1) and are intentionally NOT bridged into br-lan: the FIPS
# Ethernet transport binds each directly and runs discovery beacons over it.
#
# The shipped /etc/fips/fips.yaml carries 'mesh0' and 'mesh1' entries under
# 'transports.ethernet' bound to these names, but commented out — a stock
# install that never creates fips-mesh* then logs no bind warning. This
# helper uncomments the matching entry when it creates an interface and
# re-comments it on remove, so the daemon binds the transport without a
# manual config edit. After an interface is up, restart fips.
# See docs/how-to/set-up-80211s-mesh-backhaul.md for the full guide.
DEFAULT_MESH_ID="fips-mesh"
CONFIG="/etc/fips/fips.yaml"
# Replace $CONFIG with the rewritten $CONFIG.tmp. Force mode 0600 first: the
# package installs fips.yaml 0600 (it may hold an inline 'nsec' private key),
# and a fresh tmp file would otherwise land world-readable after the move.
mesh_config_write() {
chmod 600 "$CONFIG.tmp" && mv "$CONFIG.tmp" "$CONFIG"
}
# Uncomment the 'mesh<idx>' transports.ethernet block in $CONFIG (created by
# 'fips-mesh-setup'). Reversible with mesh_config_disable. Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
mesh_config_enable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" && return 0
grep -q "^ # mesh$idx:" "$CONFIG" || return 2
awk -v idx="$idx" '
$0 ~ ("^ # mesh" idx ":[ \t]*$") { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
# Re-comment the 'mesh<idx>' block so the daemon stops binding it (and stops
# warning about the now-missing interface). Inverse of mesh_config_enable.
mesh_config_disable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" || return 0
awk -v idx="$idx" '
$0 ~ ("^ mesh" idx ":[ \t]*$") { blk = 1; sub(/^ /, " # "); print; next }
blk && /^ / { sub(/^ /, " # "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
usage() {
echo "Usage: fips-mesh-setup <radio> [mesh-id]" >&2
echo " fips-mesh-setup remove [radio]" >&2
echo "Radios on this device:" >&2
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)=wifi-device$/ \1/p" >&2
exit 1
}
# List the UCI section names of fips-managed mesh wifi-ifaces.
mesh_sections() {
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\(fips_mesh[^.=]*\)=wifi-iface$/\1/p"
}
# ---------------------------------------------------------------------------
# remove [radio] — delete the wireless and network sections created below
# ---------------------------------------------------------------------------
if [ "$1" = "remove" ]; then
if [ -n "$2" ]; then
SECTIONS="fips_mesh_$(printf '%s' "$2" | tr -c 'a-zA-Z0-9_' '_')"
else
SECTIONS="$(mesh_sections)"
fi
[ -n "$SECTIONS" ] || {
echo "No fips mesh instances configured."
exit 0
}
for section in $SECTIONS; do
ifname="$(uci -q get "wireless.$section.ifname")"
uci -q delete "wireless.$section"
uci -q delete "network.$section"
# Re-comment the matching mesh<N> transport in fips.yaml so the
# daemon stops warning about the interface we just removed.
idx="$(printf '%s' "$ifname" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$idx" ] && mesh_config_disable "$idx"
echo "Removed ${ifname:-$section}."
done
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected on remove.
wifi reload
echo "Restart fips: /etc/init.d/fips restart"
exit 0
fi
RADIO="$1"
MESH_ID="${2:-$DEFAULT_MESH_ID}"
[ -n "$RADIO" ] || usage
if [ "$(uci -q get "wireless.$RADIO")" != "wifi-device" ]; then
echo "Error: '$RADIO' is not a wifi-device in /etc/config/wireless." >&2
usage
fi
# One instance per radio: section fips_mesh_<radio>, netdev fips-mesh<N>
# where N is the radio's trailing index (radio0 -> fips-mesh0). For radios
# named without a trailing number, fall back to the first free index.
SECTION="fips_mesh_$(printf '%s' "$RADIO" | tr -c 'a-zA-Z0-9_' '_')"
IDX="$(printf '%s' "$RADIO" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$IDX" ] || IDX="$(printf '%s' "$RADIO" | sed -n 's/^\([0-9]\{1,\}\)$/\1/p')"
if [ -z "$IDX" ]; then
IDX=0
while uci show wireless 2>/dev/null | grep -q "\.ifname='fips-mesh$IDX'"; do
IDX=$((IDX + 1))
done
fi
MESH_IFNAME="fips-mesh$IDX"
# Refuse a name collision from another radio's instance (e.g. two radios
# whose names end in the same digit) rather than silently hijacking it.
OWNER="$(uci show wireless 2>/dev/null \
| sed -n "s/^wireless\.\(fips_mesh[^.=]*\)\.ifname='$MESH_IFNAME'$/\1/p")"
if [ -n "$OWNER" ] && [ "$OWNER" != "$SECTION" ]; then
echo "Error: $MESH_IFNAME is already used by section '$OWNER'." >&2
echo "Remove it first: fips-mesh-setup remove" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Driver capability check (advisory — config below is harmless either way)
# ---------------------------------------------------------------------------
if command -v iw >/dev/null 2>&1; then
if ! iw list 2>/dev/null | grep -q "\* mesh point"; then
echo "Warning: no radio on this device advertises 'mesh point' support" >&2
echo "(iw list | grep 'mesh point'). The interface may fail to come up." >&2
fi
fi
# ---------------------------------------------------------------------------
# Wireless: open 802.11s mesh point, HWMP forwarding off
# ---------------------------------------------------------------------------
uci -q delete "wireless.$SECTION"
uci set "wireless.$SECTION=wifi-iface"
uci set "wireless.$SECTION.device=$RADIO"
uci set "wireless.$SECTION.mode=mesh"
uci set "wireless.$SECTION.mesh_id=$MESH_ID"
uci set "wireless.$SECTION.encryption=none"
uci set "wireless.$SECTION.mesh_fwding=0"
uci set "wireless.$SECTION.ifname=$MESH_IFNAME"
uci set "wireless.$SECTION.network=$SECTION"
# Radios ship disabled on fresh OpenWrt installs; a disabled radio would
# leave the mesh interface down with no error anywhere visible.
if [ "$(uci -q get "wireless.$RADIO.disabled")" = "1" ]; then
echo "Note: enabling $RADIO (was disabled)."
uci -q delete "wireless.$RADIO.disabled"
fi
# The mesh inherits the radio's channel, and mesh points only peer on the
# same channel. 'auto' lets each router pick its own — the classic silent
# non-peering cause — so surface the setting loudly.
CHANNEL="$(uci -q get "wireless.$RADIO.channel")"
BAND="$(uci -q get "wireless.$RADIO.band")"
if [ -z "$CHANNEL" ] || [ "$CHANNEL" = "auto" ]; then
echo "Warning: $RADIO channel is '${CHANNEL:-unset}' — each router may" >&2
echo "auto-select a different channel and mesh points only peer on the" >&2
echo "same one. Pin the same channel on every backhaul router, e.g.:" >&2
echo " uci set wireless.$RADIO.channel='36' && uci commit wireless && wifi reload" >&2
fi
# A client (sta) interface on the same radio follows its upstream AP's
# channel and drags every other interface with it — a mesh pinned to a
# different channel silently never joins, and does not recover when the
# STA disconnects.
for s in $(uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)\.mode='sta'$/\1/p"); do
if [ "$(uci -q get "wireless.$s.device")" = "$RADIO" ]; then
echo "Warning: $RADIO also carries client interface '$s' (mode 'sta')." >&2
echo "The whole radio follows that STA's upstream channel — a mesh" >&2
echo "pinned to a different channel stays down silently. Align the" >&2
echo "mesh channel with the upstream AP, or put the mesh on a radio" >&2
echo "without a STA (a roaming uplink is incompatible with a" >&2
echo "fixed-channel mesh on the same radio)." >&2
fi
done
# ---------------------------------------------------------------------------
# Network: unmanaged interface so netifd brings the netdev up. No IP config —
# the FIPS Ethernet transport speaks raw frames on it.
# ---------------------------------------------------------------------------
uci -q delete "network.$SECTION"
uci set "network.$SECTION=interface"
uci set "network.$SECTION.proto=none"
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected when adding a mesh.
wifi reload
# Enable the matching mesh<N> transport in the shipped fips.yaml (it ships
# commented out). Tailor the restart hint to what we could do.
mesh_config_enable "$IDX"
case $? in
0) TRANSPORT_NOTE="The mesh$IDX transport in $CONFIG that binds '$MESH_IFNAME' is
now uncommented and enabled." ;;
1) TRANSPORT_NOTE="No $CONFIG found — add a transports.ethernet entry binding
interface '$MESH_IFNAME' by hand." ;;
*) TRANSPORT_NOTE="No 'mesh$IDX' entry in $CONFIG — add a transports.ethernet
entry binding interface '$MESH_IFNAME' by hand (copy the mesh0 block)." ;;
esac
cat <<EOF
Created open 802.11s mesh '$MESH_ID' as $MESH_IFNAME on $RADIO \
(band ${BAND:-?}, channel ${CHANNEL:-auto}).
ALL routers in this backhaul must share this mesh ID AND channel
(per band). On a dual-band router, run fips-mesh-setup for the other
radio too — second band is a standby path (failover, not multipath).
Next steps:
1. $TRANSPORT_NOTE
Restart the daemon AFTER the interface is up — a transport whose
interface is missing at startup is skipped, not retried:
/etc/init.d/fips restart
2. Verify L2 peering with a second FIPS router in range:
iw dev $MESH_IFNAME station dump
and the FIPS link on top of it:
fipsctl show peers
Run 'fips-mesh-setup remove' to undo all instances, or
'fips-mesh-setup remove $RADIO' for just this one.
EOF

View File

@@ -68,7 +68,7 @@ and set the interface name:
transports:
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true

View File

@@ -8,7 +8,7 @@ use fips::config::{IdentitySource, resolve_identity};
use fips::version;
use fips::{Config, Node};
use std::path::PathBuf;
use tracing::{debug, error, info, warn};
use tracing::{debug, error, info};
use tracing_subscriber::{EnvFilter, fmt};
/// FIPS mesh network daemon
@@ -157,26 +157,23 @@ async fn run_daemon(
info!("FIPS running");
// Run the RX event loop until shutdown signal.
// stop() drops the packet channel, causing run_rx_loop to exit.
tokio::select! {
result = node.run_rx_loop() => {
match result {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
}
_ = shutdown_signal => {
info!("Shutdown signal received");
}
// Serve until the shutdown signal, then drain in place before returning.
// The rx loop observes the signal directly, so its channels are never
// destructively cancelled — they live in the loop's locals across serve and
// drain, and are dropped only on clean exit (after which teardown does not
// need them). On the signal the loop broadcasts a shutdown Disconnect and
// waits (bounded by node.drain_timeout_secs) for peers to clear.
match node.run_rx_loop_with_shutdown(shutdown_signal).await {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
info!("FIPS shutting down");
// Stop the node (shuts down transports, TUN, I/O threads)
if let Err(e) = node.stop().await {
warn!("Error during shutdown: {}", e);
}
// Close the drain window (if the loop drained) and tear down. A drained
// loop tears down without re-broadcasting; a loop that exited some other
// way falls back to the immediate stop().
node.finish_shutdown().await;
info!("FIPS shutdown complete");
}

View File

@@ -134,8 +134,8 @@ fn draw_routing_stats(
let cols =
Layout::horizontal([Constraint::Percentage(50), Constraint::Percentage(50)]).split(inner);
// Shorthand for a nested counter value (e.g. discovery.req_received).
let disc = |key: &str| helpers::nested_u64(data, "discovery", key);
// Shorthand for a nested counter value (e.g. lookup.req_received).
let lookup = |key: &str| helpers::nested_u64(data, "lookup", key);
let err = |key: &str| helpers::nested_u64(data, "error_signals", key);
let cong = |key: &str| helpers::nested_u64(data, "congestion", key);
@@ -174,32 +174,32 @@ fn draw_routing_stats(
));
left.push(Line::from(""));
left.extend(section(
"Discovery Requests",
"Lookup Requests",
&[
("Received", disc("req_received")),
("Forwarded", disc("req_forwarded")),
("Initiated", disc("req_initiated")),
("Deduplicated", disc("req_deduplicated")),
("Target Is Us", disc("req_target_is_us")),
("Duplicate", disc("req_duplicate")),
("Bloom Miss", disc("req_bloom_miss")),
("Backoff Suppressed", disc("req_backoff_suppressed")),
("Fwd Rate Limited", disc("req_forward_rate_limited")),
("TTL Exhausted", disc("req_ttl_exhausted")),
("Decode Error", disc("req_decode_error")),
("Received", lookup("req_received")),
("Forwarded", lookup("req_forwarded")),
("Initiated", lookup("req_initiated")),
("Deduplicated", lookup("req_deduplicated")),
("Target Is Us", lookup("req_target_is_us")),
("Duplicate", lookup("req_duplicate")),
("Bloom Miss", lookup("req_bloom_miss")),
("Backoff Suppressed", lookup("req_backoff_suppressed")),
("Fwd Rate Limited", lookup("req_forward_rate_limited")),
("TTL Exhausted", lookup("req_ttl_exhausted")),
("Decode Error", lookup("req_decode_error")),
],
));
left.push(Line::from(""));
left.extend(section(
"Discovery Responses",
"Lookup Responses",
&[
("Received", disc("resp_received")),
("Accepted", disc("resp_accepted")),
("Forwarded", disc("resp_forwarded")),
("Timed Out", disc("resp_timed_out")),
("Identity Miss", disc("resp_identity_miss")),
("Proof Failed", disc("resp_proof_failed")),
("Decode Error", disc("resp_decode_error")),
("Received", lookup("resp_received")),
("Accepted", lookup("resp_accepted")),
("Forwarded", lookup("resp_forwarded")),
("Timed Out", lookup("resp_timed_out")),
("Identity Miss", lookup("resp_identity_miss")),
("Proof Failed", lookup("resp_proof_failed")),
("Decode Error", lookup("resp_decode_error")),
],
));

View File

@@ -1,60 +0,0 @@
//! Bloom Filter Implementation
//!
//! 1KB Bloom filters for reachability in FIPS routing. Each node
//! maintains filters that summarize which destinations are reachable
//! through each peer, enabling efficient routing decisions without
//! global network knowledge.
//!
//! ## v1 Parameters
//!
//! - Size: 1 KB (8,192 bits) - sized for actual ~400-800 entry occupancy
//! - Hash functions: k=5 - optimal at ~1,200 entries, good for 800-1,600
//! - Bandwidth: 1 KB/announce (75% reduction from original 4KB design)
//!
//! These parameters are right-sized for typical network occupancy of
//! ~250-800 entries per node.
mod filter;
mod state;
use thiserror::Error;
pub use filter::BloomFilter;
pub use state::BloomState;
/// Default filter size in bits (1KB = 8,192 bits).
///
/// Sized for ~800-1,600 entries. FPR ~0.05% at 400 entries, ~0.9% at 800.
/// This is v1 protocol default (size_class=1).
pub const DEFAULT_FILTER_SIZE_BITS: usize = 8192;
/// Default filter size in bytes (1KB).
pub const DEFAULT_FILTER_SIZE_BYTES: usize = DEFAULT_FILTER_SIZE_BITS / 8;
/// Default number of hash functions.
///
/// k=5 is optimal at ~1,200 entries and a good compromise for 800-1,600.
/// At 400 entries: FPR ~0.05%. At 800 entries: FPR ~0.9%.
pub const DEFAULT_HASH_COUNT: u8 = 5;
/// Size class for v1 protocol (1 KB filters).
pub const V1_SIZE_CLASS: u8 = 1;
/// Filter sizes by size_class: bytes = 512 << size_class
pub const SIZE_CLASS_BYTES: [usize; 4] = [512, 1024, 2048, 4096];
/// Errors related to Bloom filter operations.
#[derive(Debug, Error)]
pub enum BloomError {
#[error("invalid filter size: expected {expected} bits, got {got}")]
InvalidSize { expected: usize, got: usize },
#[error("filter size must be a multiple of 8, got {0}")]
SizeNotByteAligned(usize),
#[error("hash count must be positive")]
ZeroHashCount,
}
#[cfg(test)]
mod tests;

View File

@@ -9,7 +9,7 @@ use std::collections::HashMap;
use super::CacheStats;
use super::entry::CacheEntry;
use crate::NodeAddr;
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// Default maximum entries in coordinate cache.
pub const DEFAULT_COORD_CACHE_SIZE: usize = 50_000;

2
src/cache/entry.rs vendored
View File

@@ -1,6 +1,6 @@
//! Cache entry with TTL and LRU tracking.
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// A cached coordinate entry.
#[derive(Clone, Debug)]

View File

@@ -33,9 +33,9 @@ use thiserror::Error;
#[cfg(target_os = "linux")]
pub use gateway::{ConntrackConfig, GatewayConfig, GatewayDnsConfig, PortForward, Proto};
pub use node::{
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, DiscoveryConfig, LimitsConfig,
NodeConfig, NostrDiscoveryConfig, NostrDiscoveryPolicy, RateLimitConfig, RekeyConfig,
RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, LimitsConfig, LookupConfig, MmpConfig,
NodeConfig, NostrRendezvousConfig, NostrRendezvousPolicy, RateLimitConfig, RekeyConfig,
RendezvousConfig, RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
};
pub use peer::{ConnectPolicy, PeerAddress, PeerConfig};
pub use transport::{
@@ -489,10 +489,59 @@ impl Config {
source: e,
})?;
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})
let mut config: Config =
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})?;
config.normalize_deprecated_keys();
Ok(config)
}
/// COMPAT (drop at the v2 cutover): fold a deprecated `node.discovery:`
/// block into the `node.lookup.*` (mesh-lookup scalars) and
/// `node.rendezvous.*` (nostr/LAN peer rendezvous) tables that replaced it.
///
/// Runs at every deserialize boundary (see `load_file`). A present legacy
/// field fills the corresponding new-table field, so a config that predates
/// the split keeps behaving identically. When a legacy block is seen, a
/// one-time deprecation warning names the old→new key moves. Exposed to the
/// crate so config tests that deserialize directly can invoke it.
pub(crate) fn normalize_deprecated_keys(&mut self) {
let Some(compat) = self.node.discovery.take() else {
return;
};
tracing::warn!(
target: "fips::config",
"`node.discovery.*` is deprecated and will be removed: mesh-lookup \
scalars moved to `node.lookup.*`, and peer-rendezvous keys moved to \
`node.rendezvous.nostr.*` / `node.rendezvous.lan.*`. Please migrate; \
a legacy `node.discovery` block still applies for now."
);
if let Some(v) = compat.ttl {
self.node.lookup.ttl = v;
}
if let Some(v) = compat.attempt_timeouts_secs {
self.node.lookup.attempt_timeouts_secs = v;
}
if let Some(v) = compat.recent_expiry_secs {
self.node.lookup.recent_expiry_secs = v;
}
if let Some(v) = compat.backoff_base_secs {
self.node.lookup.backoff_base_secs = v;
}
if let Some(v) = compat.backoff_max_secs {
self.node.lookup.backoff_max_secs = v;
}
if let Some(v) = compat.forward_min_interval_secs {
self.node.lookup.forward_min_interval_secs = v;
}
if let Some(v) = compat.nostr {
self.node.rendezvous.nostr = v;
}
if let Some(v) = compat.lan {
self.node.rendezvous.lan = v;
}
}
/// Get the standard search paths in priority order (lowest to highest).
@@ -600,7 +649,7 @@ impl Config {
/// Validate cross-field configuration invariants.
pub fn validate(&self) -> Result<(), ConfigError> {
let nostr = &self.node.discovery.nostr;
let nostr = &self.node.rendezvous.nostr;
let any_transport_advertises_on_nostr = self
.transports
@@ -620,13 +669,13 @@ impl Config {
if any_transport_advertises_on_nostr && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one transport has `advertise_on_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one transport has `advertise_on_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
if self.peers.iter().any(|peer| peer.via_nostr) && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one peer has `via_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one peer has `via_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
@@ -648,12 +697,12 @@ impl Config {
if nostr.enabled && has_nat_udp_advert {
if nostr.dm_relays.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.dm_relays` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.dm_relays` to be non-empty".to_string(),
));
}
if nostr.stun_servers.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.stun_servers` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.stun_servers` to be non-empty".to_string(),
));
}
}
@@ -721,6 +770,73 @@ node:
assert!(config.has_identity());
}
/// The fips.yaml shipped in the OpenWrt package must keep parsing as the
/// config schema evolves. The 802.11s mesh backhaul entries (one per
/// radio, so dual-band routers can mesh on both bands) ship commented
/// out — a stock install that never creates fips-mesh* logs no per-boot
/// bind warning; `fips-mesh-setup` uncomments the matching block when it
/// creates the interface (docs/how-to/set-up-80211s-mesh-backhaul.md).
/// Verify both states parse: as shipped (mesh inactive), and after the
/// uncomment `fips-mesh-setup` performs.
#[test]
fn shipped_openwrt_config_parses() {
let yaml = include_str!("../../packaging/openwrt-ipk/files/etc/fips/fips.yaml");
// As shipped: parses, and the mesh entries are commented out (a
// running daemon binds no fips-mesh* transport, so no bind warning).
let config: Config = serde_yaml::from_str(yaml).expect("shipped OpenWrt fips.yaml");
for name in ["mesh0", "mesh1"] {
assert!(
!config
.transports
.ethernet
.iter()
.any(|(n, _)| n == Some(name)),
"{name} must ship commented out, not active, in fips.yaml"
);
}
// What `fips-mesh-setup` produces: uncomment each mesh block, which
// must still parse into a transport entry bound to the right netdev.
let config: Config = serde_yaml::from_str(&uncomment_mesh_blocks(yaml))
.expect("fips.yaml with mesh transports uncommented");
for (name, interface) in [("mesh0", "fips-mesh0"), ("mesh1", "fips-mesh1")] {
assert!(
config
.transports
.ethernet
.iter()
.any(|(n, eth)| n == Some(name) && eth.interface == interface),
"{name} backhaul entry missing after uncommenting shipped fips.yaml"
);
}
}
/// Mirror `fips-mesh-setup`'s block uncomment: strip the ` # ` prefix
/// from each `# mesh<N>:` header and its ` # ` continuation lines,
/// leaving every other comment untouched.
fn uncomment_mesh_blocks(yaml: &str) -> String {
let mut out = String::new();
let mut in_block = false;
for line in yaml.lines() {
let is_header = line
.strip_prefix(" # mesh")
.and_then(|r| r.strip_suffix(':'))
.is_some_and(|n| !n.is_empty() && n.bytes().all(|b| b.is_ascii_digit()));
if is_header {
in_block = true;
out.push_str(&line.replacen(" # ", " ", 1));
} else if in_block && line.starts_with(" # ") {
out.push_str(&line.replacen(" # ", " ", 1));
} else {
in_block = false;
out.push_str(line);
}
out.push('\n');
}
out
}
#[test]
fn test_parse_yaml_with_hex() {
let yaml = r#"
@@ -1261,7 +1377,9 @@ peers:
}
#[test]
fn test_parse_nostr_discovery_config() {
fn test_parse_legacy_discovery_nostr_config_compat() {
// COMPAT (drop at the v2 cutover): a deprecated `node.discovery.nostr`
// block must fold into `node.rendezvous.nostr` via normalize.
let yaml = r#"
node:
discovery:
@@ -1285,26 +1403,27 @@ peers:
- transport: udp
addr: "nat"
"#;
let config: Config = serde_yaml::from_str(yaml).unwrap();
assert!(config.node.discovery.nostr.enabled);
assert!(!config.node.discovery.nostr.advertise);
assert_eq!(config.node.discovery.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.discovery.nostr.signal_ttl_secs, 45);
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert!(config.node.rendezvous.nostr.enabled);
assert!(!config.node.rendezvous.nostr.advertise);
assert_eq!(config.node.rendezvous.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.rendezvous.nostr.signal_ttl_secs, 45);
assert_eq!(
config.node.discovery.nostr.policy,
NostrDiscoveryPolicy::ConfiguredOnly
config.node.rendezvous.nostr.policy,
NostrRendezvousPolicy::ConfiguredOnly
);
assert_eq!(config.node.discovery.nostr.open_discovery_max_pending, 12);
assert_eq!(config.node.rendezvous.nostr.open_discovery_max_pending, 12);
assert_eq!(
config.node.discovery.nostr.advert_relays,
config.node.rendezvous.nostr.advert_relays,
vec!["wss://relay-a.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.dm_relays,
config.node.rendezvous.nostr.dm_relays,
vec!["wss://relay-b.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.stun_servers,
config.node.rendezvous.nostr.stun_servers,
vec!["stun:stun.example.org:3478".to_string()]
);
assert_eq!(
@@ -1314,6 +1433,55 @@ peers:
assert!(config.peers[0].via_nostr);
}
#[test]
fn test_parse_lookup_and_rendezvous_new_keys() {
// The post-split keys parse directly, with no deprecated block and no
// normalize warning.
let yaml = r#"
node:
lookup:
ttl: 7
attempt_timeouts_secs: [3, 6]
forward_min_interval_secs: 9
rendezvous:
nostr:
enabled: true
app: "fips.new.keys.v1"
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 7);
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![3, 6]);
assert_eq!(config.node.lookup.forward_min_interval_secs, 9);
// Unset scalar keeps its default.
assert_eq!(config.node.lookup.recent_expiry_secs, 10);
assert!(config.node.rendezvous.nostr.enabled);
assert_eq!(config.node.rendezvous.nostr.app, "fips.new.keys.v1");
assert!(config.node.discovery.is_none());
}
#[test]
fn test_legacy_discovery_lookup_scalars_compat() {
// COMPAT (drop at the v2 cutover): legacy `node.discovery` mesh-lookup
// scalars must fold into `node.lookup`; unset keys keep their defaults.
let yaml = r#"
node:
discovery:
ttl: 5
backoff_base_secs: 4
backoff_max_secs: 30
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 5);
assert_eq!(config.node.lookup.backoff_base_secs, 4);
assert_eq!(config.node.lookup.backoff_max_secs, 30);
// Unset legacy scalar leaves the new-table default intact.
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![1, 2, 4, 8]);
// The compat block is consumed by normalize.
assert!(config.node.discovery.is_none());
}
#[test]
fn test_validate_transport_advert_requires_nostr_enabled() {
let mut config = Config::default();
@@ -1321,7 +1489,7 @@ peers:
advertise_on_nostr: Some(true),
..Default::default()
});
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("advertise_on_nostr"));
@@ -1337,7 +1505,7 @@ peers:
}],
..Default::default()
};
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("via_nostr"));
@@ -1358,7 +1526,7 @@ peers:
// Empty addresses + via_nostr=true + nostr.enabled=true → ok.
config.peers[0].via_nostr = true;
config.node.discovery.nostr.enabled = true;
config.node.rendezvous.nostr.enabled = true;
config
.validate()
.expect("via_nostr should allow empty addresses");
@@ -1367,8 +1535,8 @@ peers:
#[test]
fn test_validate_nat_udp_advert_requires_relays_and_stun() {
let mut config = Config::default();
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.dm_relays.clear();
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.dm_relays.clear();
config.transports.udp = TransportInstances::Single(UdpConfig {
advertise_on_nostr: Some(true),
public: Some(false),
@@ -1378,8 +1546,8 @@ peers:
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("dm_relays"));
config.node.discovery.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.discovery.nostr.stun_servers.clear();
config.node.rendezvous.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.rendezvous.nostr.stun_servers.clear();
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("stun_servers"));
}

View File

@@ -7,7 +7,7 @@
use serde::{Deserialize, Serialize};
use super::IdentityConfig;
use crate::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpConfig, MmpMode};
use crate::proto::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpMode};
// ============================================================================
// Node Configuration Subsections
@@ -186,48 +186,42 @@ impl CacheConfig {
}
}
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`): the overlay coordinate-lookup
/// engine (address → coordinates). The peer-rendezvous keys that used to
/// share this table (`nostr`/`lan`) now live under [`RendezvousConfig`]
/// (`node.rendezvous.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DiscoveryConfig {
/// Hop limit for LookupRequest flood (`node.discovery.ttl`).
#[serde(default = "DiscoveryConfig::default_ttl")]
pub struct LookupConfig {
/// Hop limit for LookupRequest flood (`node.lookup.ttl`).
#[serde(default = "LookupConfig::default_ttl")]
pub ttl: u8,
/// Per-attempt timeouts in seconds (`node.discovery.attempt_timeouts_secs`).
/// Per-attempt timeouts in seconds (`node.lookup.attempt_timeouts_secs`).
/// Each entry is the time to wait for a response before sending the next
/// LookupRequest (with a fresh request_id). Sequence length determines the
/// total number of attempts before declaring the destination unreachable.
/// Default `[1, 2, 4, 8]` gives 4 attempts and a 15s total budget.
#[serde(default = "DiscoveryConfig::default_attempt_timeouts_secs")]
#[serde(default = "LookupConfig::default_attempt_timeouts_secs")]
pub attempt_timeouts_secs: Vec<u64>,
/// Dedup cache expiry in seconds (`node.discovery.recent_expiry_secs`).
#[serde(default = "DiscoveryConfig::default_recent_expiry_secs")]
/// Dedup cache expiry in seconds (`node.lookup.recent_expiry_secs`).
#[serde(default = "LookupConfig::default_recent_expiry_secs")]
pub recent_expiry_secs: u64,
/// Base backoff after lookup failure in seconds (`node.discovery.backoff_base_secs`).
/// Base backoff after lookup failure in seconds (`node.lookup.backoff_base_secs`).
/// Doubles per consecutive failure up to `backoff_max_secs`. Defaults to 0
/// (no post-failure suppression); the per-attempt sequence in
/// `attempt_timeouts_secs` provides the only retry pacing.
#[serde(default = "DiscoveryConfig::default_backoff_base_secs")]
#[serde(default = "LookupConfig::default_backoff_base_secs")]
pub backoff_base_secs: u64,
/// Maximum backoff cap in seconds (`node.discovery.backoff_max_secs`).
#[serde(default = "DiscoveryConfig::default_backoff_max_secs")]
/// Maximum backoff cap in seconds (`node.lookup.backoff_max_secs`).
#[serde(default = "LookupConfig::default_backoff_max_secs")]
pub backoff_max_secs: u64,
/// Minimum interval between forwarded lookups for the same target in seconds
/// (`node.discovery.forward_min_interval_secs`).
/// (`node.lookup.forward_min_interval_secs`).
/// Defense-in-depth against misbehaving nodes.
#[serde(default = "DiscoveryConfig::default_forward_min_interval_secs")]
#[serde(default = "LookupConfig::default_forward_min_interval_secs")]
pub forward_min_interval_secs: u64,
/// Nostr-mediated overlay endpoint discovery.
#[serde(default = "DiscoveryConfig::default_nostr")]
pub nostr: NostrDiscoveryConfig,
/// mDNS / DNS-SD peer discovery on the local link. Identity surface
/// is a strict subset of what `nostr.advertise` already publishes
/// publicly, so there's no marginal privacy cost; the latency win
/// for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default = "DiscoveryConfig::default_lan")]
pub lan: crate::discovery::lan::LanDiscoveryConfig,
}
impl Default for DiscoveryConfig {
impl Default for LookupConfig {
fn default() -> Self {
Self {
ttl: 64,
@@ -236,13 +230,11 @@ impl Default for DiscoveryConfig {
backoff_base_secs: 0,
backoff_max_secs: 0,
forward_min_interval_secs: 2,
nostr: NostrDiscoveryConfig::default(),
lan: crate::discovery::lan::LanDiscoveryConfig::default(),
}
}
}
impl DiscoveryConfig {
impl LookupConfig {
fn default_ttl() -> u8 {
64
}
@@ -261,12 +253,45 @@ impl DiscoveryConfig {
fn default_forward_min_interval_secs() -> u64 {
2
}
fn default_nostr() -> NostrDiscoveryConfig {
NostrDiscoveryConfig::default()
}
fn default_lan() -> crate::discovery::lan::LanDiscoveryConfig {
crate::discovery::lan::LanDiscoveryConfig::default()
}
}
/// Peer rendezvous (`node.rendezvous.*`): how the node finds peers to connect
/// to at all — Nostr-mediated overlay endpoints and mDNS/DNS-SD on the local
/// link. Distinct from mesh lookup ([`LookupConfig`]), which finds coordinates
/// for an already-known mesh address.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct RendezvousConfig {
/// Nostr-mediated overlay endpoint rendezvous (`node.rendezvous.nostr.*`).
#[serde(default)]
pub nostr: NostrRendezvousConfig,
/// mDNS / DNS-SD peer rendezvous on the local link (`node.rendezvous.lan.*`).
/// Identity surface is a strict subset of what `nostr.advertise` already
/// publishes publicly, so there's no marginal privacy cost; the latency
/// win for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default)]
pub lan: crate::mdns::LanRendezvousConfig,
}
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:` block.
///
/// The `node.discovery.*` table was split into `node.lookup.*` (mesh-lookup
/// scalars) and `node.rendezvous.*` (nostr/LAN peer rendezvous). Because
/// `NodeConfig` does not deny unknown fields, a still-deployed `node.discovery:`
/// block would otherwise deserialize into nothing and silently revert every
/// lookup/rendezvous setting to its default. This all-`Option` mirror captures
/// it so [`Config::normalize_deprecated_keys`] can fold it into the new tables
/// with a one-time deprecation warning; unset legacy keys stay `None` and leave
/// the new-table defaults intact.
#[derive(Debug, Clone, Deserialize)]
pub(crate) struct DiscoveryConfigCompat {
pub ttl: Option<u8>,
pub attempt_timeouts_secs: Option<Vec<u64>>,
pub recent_expiry_secs: Option<u64>,
pub backoff_base_secs: Option<u64>,
pub backoff_max_secs: Option<u64>,
pub forward_min_interval_secs: Option<u64>,
pub nostr: Option<NostrRendezvousConfig>,
pub lan: Option<crate::mdns::LanRendezvousConfig>,
}
/// Nostr advert discovery policy.
@@ -278,33 +303,33 @@ impl DiscoveryConfig {
/// - `open`: also consider adverts for non-configured peers
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NostrDiscoveryPolicy {
pub enum NostrRendezvousPolicy {
Disabled,
#[default]
ConfiguredOnly,
Open,
}
/// Nostr-mediated overlay endpoint discovery (`node.discovery.nostr.*`).
/// Nostr-mediated overlay endpoint discovery (`node.rendezvous.nostr.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct NostrDiscoveryConfig {
pub struct NostrRendezvousConfig {
/// Enable Nostr-signaled traversal bootstrap.
#[serde(default)]
pub enabled: bool,
/// Publish service advertisements so remote peers can bootstrap inbound.
#[serde(default = "NostrDiscoveryConfig::default_advertise")]
#[serde(default = "NostrRendezvousConfig::default_advertise")]
pub advertise: bool,
/// Relay URLs used for service advertisements.
#[serde(default = "NostrDiscoveryConfig::default_advert_relays")]
#[serde(default = "NostrRendezvousConfig::default_advert_relays")]
pub advert_relays: Vec<String>,
/// Relay URLs used for encrypted signaling events.
#[serde(default = "NostrDiscoveryConfig::default_dm_relays")]
#[serde(default = "NostrRendezvousConfig::default_dm_relays")]
pub dm_relays: Vec<String>,
/// STUN servers used for local reflexive address discovery.
/// Outbound observation uses only this local list; peer-advertised STUN
/// values are informational and are not treated as egress targets.
#[serde(default = "NostrDiscoveryConfig::default_stun_servers")]
#[serde(default = "NostrRendezvousConfig::default_stun_servers")]
pub stun_servers: Vec<String>,
/// Whether to advertise local (RFC 1918 / ULA) interface addresses as
/// host candidates in the traversal offer.
@@ -318,85 +343,85 @@ pub struct NostrDiscoveryConfig {
#[serde(default)]
pub share_local_candidates: bool,
/// Traversal application namespace and advert identifier suffix.
#[serde(default = "NostrDiscoveryConfig::default_app")]
#[serde(default = "NostrRendezvousConfig::default_app")]
pub app: String,
/// Signaling TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_signal_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_signal_ttl_secs")]
pub signal_ttl_secs: u64,
/// Policy for advert-derived endpoint discovery.
#[serde(default)]
pub policy: NostrDiscoveryPolicy,
pub policy: NostrRendezvousPolicy,
/// Max number of open-discovery peers queued for outbound retry/connection
/// at once. Prevents unbounded queue growth from ambient advert traffic.
#[serde(default = "NostrDiscoveryConfig::default_open_discovery_max_pending")]
#[serde(default = "NostrRendezvousConfig::default_open_discovery_max_pending")]
pub open_discovery_max_pending: usize,
/// Max concurrent inbound traversal offers processed at once.
/// Acts as a rate limit against offer spam from relays.
#[serde(default = "NostrDiscoveryConfig::default_max_concurrent_incoming_offers")]
#[serde(default = "NostrRendezvousConfig::default_max_concurrent_incoming_offers")]
pub max_concurrent_incoming_offers: usize,
/// Max cached overlay adverts retained from relay traffic.
/// Bounds memory under ambient advert volume.
#[serde(default = "NostrDiscoveryConfig::default_advert_cache_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_advert_cache_max_entries")]
pub advert_cache_max_entries: usize,
/// Max seen-session IDs retained for replay detection.
/// Oldest entries are evicted when the cap is exceeded.
#[serde(default = "NostrDiscoveryConfig::default_seen_sessions_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_seen_sessions_max_entries")]
pub seen_sessions_max_entries: usize,
/// Overall punch attempt timeout in seconds.
#[serde(default = "NostrDiscoveryConfig::default_attempt_timeout_secs")]
#[serde(default = "NostrRendezvousConfig::default_attempt_timeout_secs")]
pub attempt_timeout_secs: u64,
/// Replay tracking retention window in seconds.
#[serde(default = "NostrDiscoveryConfig::default_replay_window_secs")]
#[serde(default = "NostrRendezvousConfig::default_replay_window_secs")]
pub replay_window_secs: u64,
/// Delay before punch traffic starts.
#[serde(default = "NostrDiscoveryConfig::default_punch_start_delay_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_start_delay_ms")]
pub punch_start_delay_ms: u64,
/// Interval between punch packets.
#[serde(default = "NostrDiscoveryConfig::default_punch_interval_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_interval_ms")]
pub punch_interval_ms: u64,
/// How long to keep punching before failure.
#[serde(default = "NostrDiscoveryConfig::default_punch_duration_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_duration_ms")]
pub punch_duration_ms: u64,
/// Advert TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_ttl_secs")]
pub advert_ttl_secs: u64,
/// How often adverts are refreshed in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_refresh_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_refresh_secs")]
pub advert_refresh_secs: u64,
/// Settle delay in seconds after Nostr discovery starts before the
/// one-shot startup sweep of cached adverts runs. Allows the relay
/// subscription backlog to populate the in-memory advert cache.
/// Only used under `policy: open`. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_delay_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_delay_secs")]
pub startup_sweep_delay_secs: u64,
/// Maximum age in seconds for cached adverts considered by the
/// one-shot startup sweep. Adverts whose `created_at` is older than
/// `now - startup_sweep_max_age_secs` are skipped. Only used under
/// `policy: open`. Default: 3600 (1 hour).
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_max_age_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_max_age_secs")]
pub startup_sweep_max_age_secs: u64,
/// Number of consecutive NAT-traversal failures against a peer before
/// an extended cooldown is applied to throttle further offer publishes.
/// At this threshold the daemon also actively re-fetches the peer's
/// advert from `advert_relays` to evict cache entries for peers that
/// have gone away. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_failure_streak_threshold")]
#[serde(default = "NostrRendezvousConfig::default_failure_streak_threshold")]
pub failure_streak_threshold: u32,
/// Cooldown applied to a peer once `failure_streak_threshold` is hit.
/// Suppresses both open-discovery sweep enqueues and per-attempt
/// retry firings until elapsed. Default: 1800 (30 minutes).
#[serde(default = "NostrDiscoveryConfig::default_extended_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_extended_cooldown_secs")]
pub extended_cooldown_secs: u64,
/// Minimum interval between `NAT traversal failed` WARN log lines for
/// the same peer. Subsequent failures inside the window log at DEBUG.
/// Reduces log spam on public-test nodes with many cache-learned
/// peers. Default: 300 (5 minutes).
#[serde(default = "NostrDiscoveryConfig::default_warn_log_interval_secs")]
#[serde(default = "NostrRendezvousConfig::default_warn_log_interval_secs")]
pub warn_log_interval_secs: u64,
/// Maximum entries retained in the per-npub failure-state map.
/// Bounds memory under high cache turnover. Oldest entries (by last
/// failure time) evicted when the cap is exceeded. Default: 4096.
#[serde(default = "NostrDiscoveryConfig::default_failure_state_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_failure_state_max_entries")]
pub failure_state_max_entries: usize,
/// Cooldown applied after observing a fatal protocol mismatch on a
/// Nostr-adopted bootstrap transport (e.g. `Unknown FMP version`
@@ -404,11 +429,11 @@ pub struct NostrDiscoveryConfig {
/// of `extended_cooldown_secs` and much longer because the mismatch
/// is structural — re-traversing the peer is wasted effort until one
/// side upgrades. Default: 86400 (24 hours).
#[serde(default = "NostrDiscoveryConfig::default_protocol_mismatch_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_protocol_mismatch_cooldown_secs")]
pub protocol_mismatch_cooldown_secs: u64,
}
impl Default for NostrDiscoveryConfig {
impl Default for NostrRendezvousConfig {
fn default() -> Self {
Self {
enabled: false,
@@ -419,7 +444,7 @@ impl Default for NostrDiscoveryConfig {
share_local_candidates: false,
app: Self::default_app(),
signal_ttl_secs: Self::default_signal_ttl_secs(),
policy: NostrDiscoveryPolicy::default(),
policy: NostrRendezvousPolicy::default(),
open_discovery_max_pending: Self::default_open_discovery_max_pending(),
max_concurrent_incoming_offers: Self::default_max_concurrent_incoming_offers(),
advert_cache_max_entries: Self::default_advert_cache_max_entries(),
@@ -442,7 +467,7 @@ impl Default for NostrDiscoveryConfig {
}
}
impl NostrDiscoveryConfig {
impl NostrRendezvousConfig {
fn default_advertise() -> bool {
true
}
@@ -726,6 +751,41 @@ impl SessionConfig {
}
}
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
/// Session-layer Metrics Measurement Protocol (`node.session_mmp.*`).
///
/// Separate from link-layer `node.mmp.*` to allow independent mode/interval
@@ -970,6 +1030,19 @@ pub struct NodeConfig {
#[serde(default = "NodeConfig::default_link_dead_timeout_secs")]
pub link_dead_timeout_secs: u64,
/// Graceful-shutdown drain deadline in seconds (`node.drain_timeout_secs`).
/// The bounded `Draining` phase broadcasts a shutdown `Disconnect` and then
/// waits up to this long for peers to clear before tearing down, early-
/// exiting as soon as all peers are gone. `None` selects the 2-second
/// default (see [`NodeConfig::drain_timeout`]).
///
/// Kept `Option` deliberately: `NodeConfig` has no `deny_unknown_fields`, so
/// a naive non-`Option` add with a `default` fn would silently rewrite the
/// value into deployed configs on the next serialize. The `Option` +
/// `skip_serializing_if` keeps absent configs absent.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub drain_timeout_secs: Option<u64>,
/// Resource limits (`node.limits.*`).
#[serde(default)]
pub limits: LimitsConfig,
@@ -986,9 +1059,19 @@ pub struct NodeConfig {
#[serde(default)]
pub cache: CacheConfig,
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`).
#[serde(default)]
pub discovery: DiscoveryConfig,
pub lookup: LookupConfig,
/// Peer rendezvous (`node.rendezvous.*`).
#[serde(default)]
pub rendezvous: RendezvousConfig,
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:`
/// block, folded into `lookup`/`rendezvous` by
/// [`Config::normalize_deprecated_keys`]. Never re-serialized.
#[serde(default, skip_serializing)]
pub(crate) discovery: Option<DiscoveryConfigCompat>,
/// Spanning tree (`node.tree.*`).
#[serde(default)]
@@ -1041,11 +1124,14 @@ impl Default for NodeConfig {
base_rtt_ms: 100,
heartbeat_interval_secs: 10,
link_dead_timeout_secs: 30,
drain_timeout_secs: None,
limits: LimitsConfig::default(),
rate_limit: RateLimitConfig::default(),
retry: RetryConfig::default(),
cache: CacheConfig::default(),
discovery: DiscoveryConfig::default(),
lookup: LookupConfig::default(),
rendezvous: RendezvousConfig::default(),
discovery: None,
tree: TreeConfig::default(),
bloom: BloomConfig::default(),
session: SessionConfig::default(),
@@ -1089,12 +1175,72 @@ impl NodeConfig {
fn default_link_dead_timeout_secs() -> u64 {
30
}
/// Graceful-shutdown drain deadline as a `Duration`.
///
/// Returns the configured `drain_timeout_secs`, or the 2-second default
/// when unset. Used by the daemon's bounded `Draining` phase.
pub fn drain_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(self.drain_timeout_secs.unwrap_or(2))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
#[test]
fn test_drain_timeout_default_and_override() {
// Unset → the 2-second default.
let c = NodeConfig::default();
assert_eq!(c.drain_timeout_secs, None);
assert_eq!(c.drain_timeout(), std::time::Duration::from_secs(2));
// Explicit override is honored.
let c2 = NodeConfig {
drain_timeout_secs: Some(10),
..NodeConfig::default()
};
assert_eq!(c2.drain_timeout(), std::time::Duration::from_secs(10));
// A zero override is a valid (immediate) drain, not the default.
let c3 = NodeConfig {
drain_timeout_secs: Some(0),
..NodeConfig::default()
};
assert_eq!(c3.drain_timeout(), std::time::Duration::from_secs(0));
}
#[test]
fn test_ecn_config_defaults() {
let c = EcnConfig::default();
@@ -1123,27 +1269,27 @@ mod tests {
}
#[test]
fn test_nostr_discovery_startup_sweep_defaults() {
let c = NostrDiscoveryConfig::default();
fn test_nostr_rendezvous_startup_sweep_defaults() {
let c = NostrRendezvousConfig::default();
assert_eq!(c.startup_sweep_delay_secs, 5);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}
#[test]
fn test_nostr_discovery_startup_sweep_yaml_override() {
fn test_nostr_rendezvous_startup_sweep_yaml_override() {
let yaml = "enabled: true\npolicy: open\nstartup_sweep_delay_secs: 10\nstartup_sweep_max_age_secs: 1800\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert!(c.enabled);
assert_eq!(c.policy, NostrDiscoveryPolicy::Open);
assert_eq!(c.policy, NostrRendezvousPolicy::Open);
assert_eq!(c.startup_sweep_delay_secs, 10);
assert_eq!(c.startup_sweep_max_age_secs, 1_800);
}
#[test]
fn test_nostr_discovery_startup_sweep_partial_yaml_uses_defaults() {
fn test_nostr_rendezvous_startup_sweep_partial_yaml_uses_defaults() {
// Only override delay; max_age should fall back to default.
let yaml = "enabled: true\nstartup_sweep_delay_secs: 30\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(c.startup_sweep_delay_secs, 30);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}

View File

@@ -282,9 +282,10 @@ pub struct EthernetConfig {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub send_buf_size: Option<usize>,
/// Listen for discovery beacons from other nodes. Default: true.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub discovery: Option<bool>,
/// Listen for neighbor beacons from other nodes. Default: true.
/// (Renamed from `discovery`; the old key is still accepted.)
#[serde(default, alias = "discovery", skip_serializing_if = "Option::is_none")]
pub listen: Option<bool>,
/// Broadcast announcement beacons on the LAN. Default: false.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -319,9 +320,9 @@ impl EthernetConfig {
self.send_buf_size.unwrap_or(DEFAULT_ETHERNET_SEND_BUF)
}
/// Whether to listen for discovery beacons. Default: true.
pub fn discovery(&self) -> bool {
self.discovery.unwrap_or(true)
/// Whether to listen for neighbor beacons. Default: true.
pub fn listen(&self) -> bool {
self.listen.unwrap_or(true)
}
/// Whether to broadcast announcement beacons. Default: false.
@@ -1046,4 +1047,22 @@ mod tests {
assert_eq!(parse_bind_port("[::]:443"), Some(443));
assert_eq!(parse_bind_port("not-a-socket-addr"), None);
}
#[test]
fn ethernet_listen_accepts_legacy_discovery_alias_and_rejects_unknown() {
// (a) The legacy `discovery:` key is still accepted via serde alias.
let legacy: EthernetConfig =
serde_yaml::from_str("interface: eth0\ndiscovery: true\n").unwrap();
assert_eq!(legacy.listen, Some(true));
// (b) The new canonical `listen:` key parses into the renamed field.
let renamed: EthernetConfig =
serde_yaml::from_str("interface: eth0\nlisten: true\n").unwrap();
assert_eq!(renamed.listen, Some(true));
// (c) `deny_unknown_fields` still rejects an unknown ethernet key.
let bogus: Result<EthernetConfig, _> =
serde_yaml::from_str("interface: eth0\nbogus: true\n");
assert!(bogus.is_err());
}
}

View File

@@ -236,7 +236,7 @@ pub fn show_peers(node: &Node) -> Value {
// Per-npub Nostr-traversal failure-state snapshot, indexed by npub
// for O(1) per-peer lookup. Empty if Nostr discovery is disabled.
let nostr_state: std::collections::HashMap<String, _> = node
.nostr_discovery_handle()
.nostr_rendezvous_handle()
.map(|d| {
d.failure_state_snapshot()
.into_iter()
@@ -1487,7 +1487,9 @@ pub fn show_routing(node: &Node) -> Value {
"recent_requests": node.recent_request_count(),
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -1543,7 +1545,9 @@ pub(crate) fn show_routing_from_handle(handle: &super::read_handle::ControlReadH
"recent_requests": view.recent_requests,
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -2328,7 +2332,9 @@ pub(crate) fn show_metrics_from_handle(handle: &super::read_handle::ControlReadH
let m = handle.metrics();
json!({
"forwarding": m.forwarding.snapshot(),
"discovery": m.discovery.snapshot(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": m.lookup.snapshot(),
"lookup": m.lookup.snapshot(),
"tree": m.tree.snapshot(),
"bloom": m.bloom.snapshot(),
"congestion": m.congestion.snapshot(),
@@ -2761,12 +2767,12 @@ mod tests {
/// Structural confirmation that the rx_loop no longer dispatches `show_*`:
/// the rx_loop source carries no `queries::dispatch` call and no
/// `starts_with("show_")` routing branch. Reads the committed source of
/// `src/node/handlers/rx_loop.rs` and asserts both markers are absent. This
/// `src/node/dataplane/rx_loop.rs` and asserts both markers are absent. This
/// is the milestone's "remove `show_*` from the data-plane dispatch path"
/// invariant, guarded against regression.
#[test]
fn rx_loop_has_no_show_dispatch() {
let src = include_str!("../node/handlers/rx_loop.rs");
let src = include_str!("../node/dataplane/rx_loop.rs");
assert!(
!src.contains("queries::dispatch"),
"rx_loop must not call queries::dispatch (show_* served off-loop)"
@@ -2794,7 +2800,9 @@ mod tests {
let expected_families = [
("forwarding", "received_packets"),
// `discovery` is the deprecated dual-emit alias for `lookup`; drop at the v2 cutover.
("discovery", "req_received"),
("lookup", "req_received"),
("tree", "accepted"),
("bloom", "accepted"),
("congestion", "ce_forwarded"),

View File

@@ -63,6 +63,30 @@
"ttl_exhausted_packets": 0
},
"identity_cache_entries": 0,
"lookup": {
"req_backoff_suppressed": 0,
"req_bloom_miss": 0,
"req_decode_error": 0,
"req_dedup_cache_full": 0,
"req_deduplicated": 0,
"req_duplicate": 0,
"req_fallback_forwarded": 0,
"req_forward_rate_limited": 0,
"req_forwarded": 0,
"req_initiated": 0,
"req_no_tree_peer": 0,
"req_received": 0,
"req_target_is_us": 0,
"req_ttl_exhausted": 0,
"resp_accepted": 0,
"resp_decode_error": 0,
"resp_forwarded": 0,
"resp_identity_miss": 0,
"resp_no_route": 0,
"resp_proof_failed": 0,
"resp_received": 0,
"resp_timed_out": 0
},
"pending_lookups": [],
"pending_tun_destinations": 0,
"pending_tun_packets": 0,

View File

@@ -3,22 +3,29 @@
//! A distributed, decentralized network routing protocol for mesh nodes
//! connecting over arbitrary transports.
pub mod bloom;
// Name the `alloc` crate directly so the sans-IO protocol cores can spell their
// heap-type imports in `no_std`-forward form (`alloc::sync::Arc`,
// `alloc::collections::BTreeMap`). The crate remains `std`; this only reduces the
// distance to extracting the pure cores into a `no_std` crate later.
extern crate alloc;
pub mod cache;
pub mod config;
pub mod control;
pub mod discovery;
#[cfg(target_os = "linux")]
pub mod gateway;
pub mod identity;
pub mod mmp;
pub mod mdns;
pub mod node;
pub mod noise;
pub mod nostr;
pub mod peer;
pub mod perf_profile;
pub mod protocol;
pub(crate) mod proto;
#[cfg(test)]
pub(crate) mod testutil;
mod time;
pub mod transport;
pub mod tree;
pub mod upper;
pub mod utils;
pub mod version;
@@ -33,14 +40,16 @@ pub use identity::{
pub use config::{Config, ConfigError, IdentityConfig, NymConfig, TorConfig, UdpConfig};
pub use upper::config::{DnsConfig, TunConfig};
// Re-export discovery types
pub use discovery::{BootstrapHandoffResult, EstablishedTraversal};
// Re-export nostr rendezvous handoff types
pub use nostr::{BootstrapHandoffResult, EstablishedTraversal, is_punch_packet};
// Re-export tree types
pub use tree::{CoordEntry, ParentDeclaration, TreeCoordinate, TreeError, TreeState};
// Re-export tree types (relocated from tree:: to proto::stp)
pub use proto::stp::{
CoordEntry, CoordError, ParentDeclaration, TreeCoordinate, TreeError, TreeState,
};
// Re-export bloom filter types
pub use bloom::{BloomError, BloomFilter, BloomState};
// Re-export bloom filter types (relocated from bloom:: to proto::bloom)
pub use proto::bloom::{BloomError, BloomFilter, BloomState};
// Re-export transport types
pub use transport::udp::UdpTransport;
@@ -50,20 +59,41 @@ pub use transport::{
TransportState, TransportType, packet_channel,
};
// Re-export protocol types
pub use protocol::{
CoordsRequired, FilterAnnounce, HandshakeMessageType, LinkMessageType, LookupRequest,
LookupResponse, PathBroken, ProtocolError, SessionAck, SessionDatagram, SessionFlags,
SessionMessageType, SessionSetup, TreeAnnounce,
// Re-export link-layer types (relocated from protocol:: to proto::link)
pub use proto::link::{LinkMessageType, SessionDatagram};
// Re-export the shared protocol error (relocated from protocol:: to proto::Error)
pub use proto::Error;
// Re-export FSP session wire types (relocated from protocol:: to proto::fsp)
pub use proto::fsp::{SessionAck, SessionFlags, SessionMessageType, SessionSetup};
// Re-export STP wire types (relocated from protocol:: to proto::stp)
pub use proto::stp::TreeAnnounce;
// Re-export bloom wire types (relocated from protocol:: to proto::bloom)
pub use proto::bloom::FilterAnnounce;
// Re-export discovery wire types (relocated from protocol:: to proto::lookup)
pub use proto::lookup::{LookupRequest, LookupResponse};
// Re-export routing wire types (relocated from protocol:: to proto::routing)
pub use proto::routing::{
COORDS_REQUIRED_SIZE, CoordsRequired, MTU_EXCEEDED_SIZE, MtuExceeded, PathBroken,
};
// Re-export FMP link-framing wire type (relocated from protocol:: to proto::fmp)
pub use proto::fmp::HandshakeMessageType;
// Re-export cache types
pub use cache::{CacheEntry, CacheError, CacheStats, CoordCache};
// Re-export FMP tie-break helper and promotion result (relocated from peer:: to proto::fmp)
pub use proto::fmp::{PromotionResult, cross_connection_winner};
// Re-export peer types
pub use peer::{
ActivePeer, ConnectivityState, HandshakeState, PeerConnection, PeerError, PeerSlot,
PromotionResult, cross_connection_winner,
};
// Re-export node types

View File

@@ -54,8 +54,12 @@ pub const TXT_KEY_SCOPE: &str = "scope";
/// `PROTOCOL_VERSION`).
pub const TXT_KEY_VERSION: &str = "v";
/// FIPS protocol version advertised in the mDNS TXT `v` key. Kept in sync
/// with the Nostr rendezvous `PROTOCOL_VERSION` (same value, `"1"`).
const TXT_PROTOCOL_VERSION: &str = "1";
#[derive(Debug, Error)]
pub enum LanDiscoveryError {
pub enum LanRendezvousError {
#[error("mDNS daemon init failed: {0}")]
Daemon(String),
#[error("mDNS register failed: {0}")]
@@ -79,7 +83,7 @@ pub struct LanDiscoveredPeer {
pub observed_at: Instant,
}
/// Browser-side events surfaced by `LanDiscovery::drain_events`.
/// Browser-side events surfaced by `LanRendezvous::drain_events`.
#[derive(Debug, Clone)]
pub enum LanEvent {
Discovered(LanDiscoveredPeer),
@@ -87,17 +91,17 @@ pub enum LanEvent {
/// Runtime configuration for the mDNS responder + browser.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct LanDiscoveryConfig {
pub struct LanRendezvousConfig {
/// Master switch. Default: `false` — LAN discovery is opt-in. Operators
/// who want sub-second same-LAN pairing enable it via
/// `node.discovery.lan.enabled: true`. Default-off avoids reintroducing
/// `node.rendezvous.lan.enabled: true`. Default-off avoids reintroducing
/// a per-LAN identity broadcast on nodes that have deliberately disabled
/// other discovery channels, and avoids any multicast surprise on upgrade.
#[serde(default = "LanDiscoveryConfig::default_enabled")]
#[serde(default = "LanRendezvousConfig::default_enabled")]
pub enabled: bool,
/// Overridable service type, primarily so integration tests can run
/// multiple isolated services on the same loopback interface.
#[serde(default = "LanDiscoveryConfig::default_service_type")]
#[serde(default = "LanRendezvousConfig::default_service_type")]
pub service_type: String,
/// Optional application/network scope carried in the LAN-only TXT
/// record. Browsers that set a scope ignore adverts for other scopes.
@@ -109,7 +113,7 @@ pub struct LanDiscoveryConfig {
pub scope: Option<String>,
}
impl Default for LanDiscoveryConfig {
impl Default for LanRendezvousConfig {
fn default() -> Self {
Self {
enabled: Self::default_enabled(),
@@ -119,7 +123,7 @@ impl Default for LanDiscoveryConfig {
}
}
impl LanDiscoveryConfig {
impl LanRendezvousConfig {
fn default_enabled() -> bool {
false
}
@@ -129,7 +133,7 @@ impl LanDiscoveryConfig {
}
/// Running mDNS responder + browser bound to the node's UDP advert port.
pub struct LanDiscovery {
pub struct LanRendezvous {
daemon: ServiceDaemon,
own_npub: String,
instance_fullname: String,
@@ -137,7 +141,12 @@ pub struct LanDiscovery {
event_pump: tokio::task::JoinHandle<()>,
}
impl LanDiscovery {
impl LanRendezvous {
/// Whether the mDNS event-pump task has exited (runtime liveness).
pub fn is_finished(&self) -> bool {
self.event_pump.is_finished()
}
/// Start the mDNS responder and browser.
///
/// `advertised_port` is the UDP port the operational UDP transport
@@ -148,16 +157,16 @@ impl LanDiscovery {
identity: &Identity,
scope: Option<String>,
advertised_port: u16,
config: LanDiscoveryConfig,
) -> Result<Arc<Self>, LanDiscoveryError> {
config: LanRendezvousConfig,
) -> Result<Arc<Self>, LanRendezvousError> {
if !config.enabled {
return Err(LanDiscoveryError::Disabled);
return Err(LanRendezvousError::Disabled);
}
if advertised_port == 0 {
return Err(LanDiscoveryError::NoAdvertisedPort);
return Err(LanRendezvousError::NoAdvertisedPort);
}
let daemon = ServiceDaemon::new().map_err(|e| LanDiscoveryError::Daemon(e.to_string()))?;
let daemon = ServiceDaemon::new().map_err(|e| LanRendezvousError::Daemon(e.to_string()))?;
let npub = identity.npub();
// mDNS DNS labels are capped at 63 bytes. 16 bech32 chars of npub
@@ -176,7 +185,7 @@ impl LanDiscovery {
}
props.insert(
TXT_KEY_VERSION.to_string(),
super::nostr::PROTOCOL_VERSION.to_string(),
TXT_PROTOCOL_VERSION.to_string(),
);
// host_ipv4 is set to "127.0.0.1" *and* enable_addr_auto() is
@@ -193,18 +202,18 @@ impl LanDiscovery {
advertised_port,
Some(props),
)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?
.map_err(|e| LanRendezvousError::Register(e.to_string()))?
.enable_addr_auto();
let instance_fullname = service_info.get_fullname().to_string();
daemon
.register(service_info)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?;
.map_err(|e| LanRendezvousError::Register(e.to_string()))?;
let browse_rx = daemon
.browse(&config.service_type)
.map_err(|e| LanDiscoveryError::Browse(e.to_string()))?;
.map_err(|e| LanRendezvousError::Browse(e.to_string()))?;
let (events_tx, events_rx) = tokio::sync::mpsc::unbounded_channel();
let own_npub = npub.clone();

View File

@@ -4,7 +4,7 @@ use std::time::Duration;
use crate::Identity;
use mdns_sd::ScopedIp;
use super::{LanDiscovery, LanDiscoveryConfig, LanEvent};
use super::{LanEvent, LanRendezvous, LanRendezvousConfig};
/// Distinct service type per test run so concurrent cargo-test workers
/// on the same machine don't cross-feed each other's adverts via the
@@ -15,8 +15,8 @@ fn isolated_service_type(tag: &str) -> String {
format!("_fipstest-{tag}-{rand:08x}._udp.local.")
}
fn config_for(service_type: String) -> LanDiscoveryConfig {
LanDiscoveryConfig {
fn config_for(service_type: String) -> LanRendezvousConfig {
LanRendezvousConfig {
enabled: true,
service_type,
scope: None,
@@ -47,7 +47,7 @@ fn non_link_local_ipv6_advert_is_preserved() {
}
async fn wait_for_peer(
discovery: &LanDiscovery,
discovery: &LanRendezvous,
expected_npub: &str,
timeout: Duration,
) -> Option<super::LanDiscoveredPeer> {
@@ -64,7 +64,7 @@ async fn wait_for_peer(
None
}
/// Two LanDiscovery instances on isolated service types — `a` browses
/// Two LanRendezvous instances on isolated service types — `a` browses
/// only its own type and never sees `b`, and vice versa. Sanity check
/// that the scope-isolation defense works (we'd lose isolation if mdns-
/// sd ever leaked across service types).
@@ -76,7 +76,7 @@ async fn isolated_service_types_do_not_cross_feed() {
let service_a = isolated_service_type("isolated-a");
let service_b = isolated_service_type("isolated-b");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-x".to_string()),
61001,
@@ -84,7 +84,7 @@ async fn isolated_service_types_do_not_cross_feed() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-x".to_string()),
61002,
@@ -118,7 +118,7 @@ async fn isolated_service_types_do_not_cross_feed() {
assert!(!saw_a_from_b, "isolated service types must not cross-feed");
}
/// Two LanDiscovery instances on the same service type and the same
/// Two LanRendezvous instances on the same service type and the same
/// scope: each should observe the other's advert within a few seconds.
/// Exercises the responder + browser + TXT plumbing end-to-end.
///
@@ -136,7 +136,7 @@ async fn matched_scope_peers_observe_each_other() {
let service = isolated_service_type("matched");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-shared".to_string()),
61101,
@@ -144,7 +144,7 @@ async fn matched_scope_peers_observe_each_other() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-shared".to_string()),
61102,
@@ -181,7 +181,7 @@ async fn cross_scope_advert_is_filtered() {
let service = isolated_service_type("cross-scope");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-a".to_string()),
61201,
@@ -189,7 +189,7 @@ async fn cross_scope_advert_is_filtered() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-b".to_string()),
61202,

View File

@@ -1,556 +0,0 @@
//! MMP derived metrics.
//!
//! `MmpMetrics` processes incoming ReceiverReports (from our peer) and
//! maintains derived metrics: SRTT, loss rate, goodput, ETX, and dual
//! EWMA trend indicators. Updated by the sender side when it receives
//! a ReceiverReport about its own traffic.
use crate::mmp::algorithms::{DualEwma, SrttEstimator, compute_etx};
use crate::mmp::report::ReceiverReport;
use std::time::Instant;
use tracing::trace;
/// Derived MMP metrics, updated from incoming ReceiverReports.
///
/// This lives on the sender side: when we receive a ReceiverReport from
/// our peer describing what they observed about our traffic, we process
/// it here to compute RTT, loss, goodput, and trend indicators.
pub struct MmpMetrics {
/// Smoothed RTT from timestamp echo.
pub srtt: SrttEstimator,
/// Dual EWMA trend detectors.
pub rtt_trend: DualEwma,
pub loss_trend: DualEwma,
pub goodput_trend: DualEwma,
pub jitter_trend: DualEwma,
pub etx_trend: DualEwma,
/// Forward delivery ratio (what fraction of our frames the peer received).
pub delivery_ratio_forward: f64,
/// Reverse delivery ratio (set when we compute from our own receiver state).
pub delivery_ratio_reverse: f64,
/// ETX computed from bidirectional delivery ratios.
pub etx: f64,
/// Smoothed goodput in bytes/sec (forward direction: what the peer received from us).
pub goodput_bps: f64,
// --- State for delta computation ---
/// Previous ReceiverReport's cumulative counters (for computing interval deltas).
prev_rr_cum_packets: u64,
prev_rr_cum_bytes: u64,
prev_rr_highest_counter: u64,
prev_rr_ecn_ce: u32,
prev_rr_reorder: u32,
/// Time of previous ReceiverReport (for goodput rate computation).
prev_rr_time: Option<Instant>,
/// Whether we have a previous ReceiverReport for delta computation.
has_prev_rr: bool,
// --- State for reverse delivery ratio delta computation ---
/// Previous reverse-side cumulative packets received (our receiver state).
prev_reverse_packets: u64,
/// Previous reverse-side highest counter (our receiver state).
prev_reverse_highest: u64,
/// Whether we have a previous reverse-side snapshot for delta computation.
has_prev_reverse: bool,
}
impl MmpMetrics {
/// Reset state derived from ReceiverReport counters for rekey cutover.
///
/// The new session starts with counter 0, so the prev_rr deltas must
/// be reset to avoid computing bogus loss/goodput from the counter
/// discontinuity. RTT (SRTT) is preserved since it remains valid.
pub fn reset_for_rekey(&mut self) {
self.prev_rr_cum_packets = 0;
self.prev_rr_cum_bytes = 0;
self.prev_rr_highest_counter = 0;
self.prev_rr_ecn_ce = 0;
self.prev_rr_reorder = 0;
self.prev_rr_time = None;
self.has_prev_rr = false;
self.delivery_ratio_forward = 1.0;
self.prev_reverse_packets = 0;
self.prev_reverse_highest = 0;
self.has_prev_reverse = false;
// Keep srtt, etx, trends, goodput_bps — they'll refresh from data
}
pub fn new() -> Self {
Self {
srtt: SrttEstimator::new(),
rtt_trend: DualEwma::new(),
loss_trend: DualEwma::new(),
goodput_trend: DualEwma::new(),
jitter_trend: DualEwma::new(),
etx_trend: DualEwma::new(),
delivery_ratio_forward: 1.0,
delivery_ratio_reverse: 1.0,
etx: 1.0,
goodput_bps: 0.0,
prev_rr_cum_packets: 0,
prev_rr_cum_bytes: 0,
prev_rr_highest_counter: 0,
prev_rr_ecn_ce: 0,
prev_rr_reorder: 0,
prev_rr_time: None,
has_prev_rr: false,
prev_reverse_packets: 0,
prev_reverse_highest: 0,
has_prev_reverse: false,
}
}
/// Process an incoming ReceiverReport (from the peer about our traffic).
///
/// `our_timestamp_ms` is the current session-relative time in ms (for RTT).
/// `now` is the current monotonic time (for goodput rate computation).
///
/// Returns `true` if this report produced the first SRTT measurement
/// (transition from uninitialized to initialized).
pub fn process_receiver_report(
&mut self,
rr: &ReceiverReport,
our_timestamp_ms: u32,
now: Instant,
) -> bool {
let had_srtt = self.srtt.initialized();
if self.has_prev_rr {
let counters_regressed = rr.highest_counter < self.prev_rr_highest_counter
|| rr.cumulative_packets_recv < self.prev_rr_cum_packets
|| rr.cumulative_bytes_recv < self.prev_rr_cum_bytes
|| rr.ecn_ce_count < self.prev_rr_ecn_ce
|| rr.cumulative_reorder_count < self.prev_rr_reorder;
let duplicate_counters = rr.highest_counter == self.prev_rr_highest_counter
&& rr.cumulative_packets_recv == self.prev_rr_cum_packets
&& rr.cumulative_bytes_recv == self.prev_rr_cum_bytes
&& rr.ecn_ce_count == self.prev_rr_ecn_ce
&& rr.cumulative_reorder_count == self.prev_rr_reorder;
// Safe to drop: reports are only built after interval data, so
// a fresh report always advances at least one cumulative counter.
if counters_regressed || duplicate_counters {
trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = self.prev_rr_highest_counter,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = self.prev_rr_cum_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = self.prev_rr_cum_bytes,
"Ignoring stale MMP ReceiverReport"
);
return false;
}
}
// --- RTT from timestamp echo ---
// RTT = now - echoed_timestamp - dwell_time
if rr.timestamp_echo > 0 {
let echo_ms = rr.timestamp_echo;
let dwell_ms = u32::from(rr.dwell_time);
let rtt_sample_ms = echo_ms
.checked_add(dwell_ms)
.and_then(|send_done_ms| our_timestamp_ms.checked_sub(send_done_ms));
match rtt_sample_ms {
Some(rtt_ms) if rtt_ms > 0 => {
let rtt_us = (rtt_ms as i64) * 1000;
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
rtt_ms = rtt_ms,
srtt_ms = self.srtt.srtt_us() as f64 / 1000.0,
"RTT sample from timestamp echo"
);
self.srtt.update(rtt_us);
self.rtt_trend.update(rtt_us as f64);
}
_ => {
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
"Ignoring invalid MMP RTT sample"
);
}
}
}
// --- Loss rate from cumulative counters ---
// Delta: frames the peer should have received vs. actually received
if self.has_prev_rr {
let counter_span = rr
.highest_counter
.saturating_sub(self.prev_rr_highest_counter);
let packets_delta = rr
.cumulative_packets_recv
.saturating_sub(self.prev_rr_cum_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_forward = delivery.clamp(0.0, 1.0);
let loss_rate = 1.0 - self.delivery_ratio_forward;
self.loss_trend.update(loss_rate);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
// --- Goodput from cumulative bytes + time delta ---
if self.has_prev_rr {
let bytes_delta = rr
.cumulative_bytes_recv
.saturating_sub(self.prev_rr_cum_bytes);
self.goodput_trend.update(bytes_delta as f64);
// Compute bytes/sec if we have a time reference
if let Some(prev_time) = self.prev_rr_time {
let elapsed = now.duration_since(prev_time);
let secs = elapsed.as_secs_f64();
if secs > 0.0 {
let bps = bytes_delta as f64 / secs;
// EWMA smoothing: α = 1/4
if self.goodput_bps == 0.0 {
self.goodput_bps = bps;
} else {
self.goodput_bps += (bps - self.goodput_bps) * 0.25;
}
}
}
}
// --- Jitter trend ---
self.jitter_trend.update(rr.jitter as f64);
// --- Save for next delta ---
self.prev_rr_cum_packets = rr.cumulative_packets_recv;
self.prev_rr_cum_bytes = rr.cumulative_bytes_recv;
self.prev_rr_highest_counter = rr.highest_counter;
self.prev_rr_ecn_ce = rr.ecn_ce_count;
self.prev_rr_reorder = rr.cumulative_reorder_count;
self.prev_rr_time = Some(now);
self.has_prev_rr = true;
!had_srtt && self.srtt.initialized()
}
/// Update the reverse delivery ratio from our own receiver state.
///
/// Computes a per-interval delta (same as forward ratio) rather than
/// a lifetime cumulative ratio, so ETX responds to recent conditions.
pub fn update_reverse_delivery(&mut self, our_recv_packets: u64, peer_highest: u64) {
if self.has_prev_reverse {
let counter_span = peer_highest.saturating_sub(self.prev_reverse_highest);
let packets_delta = our_recv_packets.saturating_sub(self.prev_reverse_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_reverse = delivery.clamp(0.0, 1.0);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
self.prev_reverse_packets = our_recv_packets;
self.prev_reverse_highest = peer_highest;
self.has_prev_reverse = true;
}
/// Current smoothed RTT in milliseconds, or `None` if not yet measured.
pub fn srtt_ms(&self) -> Option<f64> {
if self.srtt.initialized() {
Some(self.srtt.srtt_us() as f64 / 1000.0)
} else {
None
}
}
/// Current loss rate (0.0 = no loss, 1.0 = total loss).
pub fn loss_rate(&self) -> f64 {
1.0 - self.delivery_ratio_forward
}
/// Smoothed loss rate (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_loss(&self) -> Option<f64> {
if self.loss_trend.initialized() {
Some(self.loss_trend.long())
} else {
None
}
}
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_etx(&self) -> Option<f64> {
if self.etx_trend.initialized() {
Some(self.etx_trend.long())
} else {
None
}
}
/// Current smoothed goodput in bytes/sec, or 0 if not yet measured.
pub fn goodput_bps(&self) -> f64 {
self.goodput_bps
}
/// Cumulative ECN CE count from the most recent ReceiverReport.
pub fn last_ecn_ce_count(&self) -> u32 {
self.prev_rr_ecn_ce
}
}
impl Default for MmpMetrics {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
fn make_rr(
highest_counter: u64,
cum_packets: u64,
cum_bytes: u64,
timestamp_echo: u32,
dwell: u16,
jitter: u32,
) -> ReceiverReport {
ReceiverReport {
highest_counter,
cumulative_packets_recv: cum_packets,
cumulative_bytes_recv: cum_bytes,
timestamp_echo,
dwell_time: dwell,
max_burst_loss: 0,
mean_burst_loss: 0,
jitter,
ecn_ce_count: 0,
owd_trend: 0,
burst_loss_count: 0,
cumulative_reorder_count: 0,
interval_packets_recv: 0,
interval_bytes_recv: 0,
}
}
#[test]
fn test_rtt_from_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
// Peer echoes timestamp 1000ms, dwell=5ms, our current time=1050ms
let rr = make_rr(10, 10, 5000, 1000, 5, 0);
m.process_receiver_report(&rr, 1050, now);
assert!(m.srtt.initialized());
// RTT = 1050 - 1000 - 5 = 45ms
let srtt_ms = m.srtt_ms().unwrap();
assert!((srtt_ms - 45.0).abs() < 1.0, "srtt={srtt_ms}, expected ~45");
}
#[test]
fn test_ignores_duplicate_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5_000, 1_000, 5, 0);
m.process_receiver_report(&rr1, 1_050, t0);
let rr2 = make_rr(20, 18, 14_000, 1_100, 5, 0);
m.process_receiver_report(&rr2, 1_150, t0 + Duration::from_secs(1));
let baseline_srtt_ms = m.srtt_ms().unwrap();
let baseline_loss = m.loss_rate();
let baseline_goodput = m.goodput_bps();
assert!(baseline_loss > 0.0);
assert!(baseline_goodput > 0.0);
// A duplicate of the same counters arriving later would be a 4.895s
// RTT sample if accepted. It is stale and must not move metrics.
m.process_receiver_report(&rr2, 6_000, t0 + Duration::from_secs(5));
assert_eq!(m.srtt_ms().unwrap(), baseline_srtt_ms);
assert_eq!(m.loss_rate(), baseline_loss);
assert_eq!(m.goodput_bps(), baseline_goodput);
}
#[test]
fn test_ignores_out_of_order_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let valid_rr = make_rr(20, 20, 10000, 1000, 5, 0);
m.process_receiver_report(&valid_rr, 1050, now);
let baseline_srtt_ms = m.srtt_ms().unwrap();
let old_rr = make_rr(10, 10, 5000, 1000, 0, 0);
m.process_receiver_report(&old_rr, 6000, now + Duration::from_secs(5));
let srtt_ms = m.srtt_ms().unwrap();
assert_eq!(srtt_ms, baseline_srtt_ms);
}
#[test]
fn test_ignores_wrapped_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let wrapped_rr = make_rr(10, 10, 5000, u32::MAX - 10, 20, 0);
m.process_receiver_report(&wrapped_rr, 15, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_ignores_future_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let future_rr = make_rr(10, 10, 5_000, 2_000, 5, 0);
m.process_receiver_report(&future_rr, 1_000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_loss_rate_computation() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline
let rr1 = make_rr(100, 100, 50000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
// Second report: 200 counters sent, 190 received (5% loss)
let rr2 = make_rr(300, 290, 145000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
let loss = m.loss_rate();
assert!((loss - 0.05).abs() < 0.01, "loss={loss}, expected ~0.05");
}
#[test]
fn test_etx_updates() {
let mut m = MmpMetrics::new();
assert_eq!(m.etx, 1.0); // initial: perfect
// Simulate some loss via forward ratio
m.delivery_ratio_forward = 0.9;
// First call establishes the baseline (no ETX update yet)
m.update_reverse_delivery(100, 100);
assert_eq!(m.etx, 1.0); // still perfect — baseline only
// Second call: 190 of 200 frames received (5% loss)
m.update_reverse_delivery(290, 300);
assert!(m.etx > 1.0);
assert!(m.etx < 2.0);
}
#[test]
fn test_no_rtt_without_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let rr = make_rr(10, 10, 5000, 0, 0, 0);
m.process_receiver_report(&rr, 1000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_jitter_trend() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5000, 0, 0, 100);
m.process_receiver_report(&rr1, 0, t0);
let rr2 = make_rr(20, 20, 10000, 0, 0, 500);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(m.jitter_trend.initialized());
// Short-term should be closer to 500 than long-term
assert!(m.jitter_trend.short() > m.jitter_trend.long());
}
#[test]
fn test_goodput_bps() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline (50KB received)
let rr1 = make_rr(100, 100, 50_000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
assert_eq!(m.goodput_bps(), 0.0); // no rate yet (first report)
// Second report 1s later: 150KB total (100KB delta in 1s = 100KB/s)
let rr2 = make_rr(300, 290, 150_000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(
m.goodput_bps() > 90_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
assert!(
m.goodput_bps() < 110_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
}
#[test]
fn test_reverse_delivery_delta() {
let mut m = MmpMetrics::new();
// First call: baseline only, no ratio update
m.update_reverse_delivery(100, 100);
assert_eq!(m.delivery_ratio_reverse, 1.0); // unchanged from default
// Second call: perfect delivery (200 new frames, all received)
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Third call: 50% loss (100 frames sent, 50 received)
m.update_reverse_delivery(350, 400);
assert!(
(m.delivery_ratio_reverse - 0.5).abs() < 0.001,
"reverse={}, expected 0.5",
m.delivery_ratio_reverse
);
}
#[test]
fn test_reverse_delivery_rekey_reset() {
let mut m = MmpMetrics::new();
// Establish baseline and one measurement
m.update_reverse_delivery(100, 100);
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Rekey resets reverse state
m.reset_for_rekey();
// First call after rekey: baseline only
m.update_reverse_delivery(50, 50);
// delivery_ratio_reverse was reset to 1.0 by reset_for_rekey's
// clearing of delivery_ratio_forward; reverse is not explicitly
// reset — but the delta state is, so next call computes fresh.
assert_eq!(m.delivery_ratio_reverse, 1.0);
// Second call after rekey: 80% delivery
m.update_reverse_delivery(90, 100);
assert!(
(m.delivery_ratio_reverse - 0.8).abs() < 0.001,
"reverse={}, expected 0.8",
m.delivery_ratio_reverse
);
}
}

View File

@@ -1,555 +0,0 @@
//! Metrics Measurement Protocol (MMP) — link-layer instantiation.
//!
//! Measures link quality between adjacent peers: RTT, loss, jitter,
//! throughput, one-way delay trend, and ETX. Operates on the per-frame
//! hooks (counter, timestamp, flags) introduced by the FMP wire format
//! revision.
//!
//! Three operating modes trade measurement fidelity for overhead:
//! - **Full**: sender + receiver reports at RTT-adaptive intervals
//! - **Lightweight**: receiver reports only (infer loss from counters)
//! - **Minimal**: spin bit + CE echo only, no reports
use serde::{Deserialize, Serialize};
use std::fmt::{self, Debug};
use std::time::{Duration, Instant};
// Sub-modules
pub mod algorithms;
pub mod metrics;
pub mod receiver;
pub mod report;
pub mod sender;
// Re-exports
pub use algorithms::{
DualEwma, JitterEstimator, OwdTrendDetector, SpinBitState, SrttEstimator, compute_etx,
};
pub use metrics::MmpMetrics;
pub use receiver::ReceiverState;
pub use report::{ReceiverReport, SenderReport};
pub use sender::SenderState;
// Session-layer re-exports
// MmpSessionState and PathMtuState are defined in this file
// ============================================================================
// Constants
// ============================================================================
/// SenderReport body size (after msg_type byte): 3 reserved + 44 payload = 47.
pub const SENDER_REPORT_BODY_SIZE: usize = 47;
/// ReceiverReport body size (after msg_type byte): 3 reserved + 64 payload = 67.
pub const RECEIVER_REPORT_BODY_SIZE: usize = 67;
/// SenderReport total wire size including inner header: 5 + 47 = 52.
pub const SENDER_REPORT_WIRE_SIZE: usize = 52;
/// ReceiverReport total wire size including inner header: 5 + 67 = 72.
pub const RECEIVER_REPORT_WIRE_SIZE: usize = 72;
// --- EWMA parameters (as shift amounts for integer arithmetic) ---
/// Jitter EWMA: α = 1/16 (RFC 3550 §6.4.1).
pub const JITTER_ALPHA_SHIFT: u32 = 4;
/// SRTT: α = 1/8 (Jacobson, RFC 6298).
pub const SRTT_ALPHA_SHIFT: u32 = 3;
/// RTTVAR: β = 1/4 (Jacobson, RFC 6298).
pub const RTTVAR_BETA_SHIFT: u32 = 2;
/// Dual EWMA short-term: α = 1/4.
pub const EWMA_SHORT_ALPHA: f64 = 0.25;
/// Dual EWMA long-term: α = 1/32.
pub const EWMA_LONG_ALPHA: f64 = 1.0 / 32.0;
// --- Timing defaults (milliseconds) ---
/// Default report interval before SRTT is available (cold start).
pub const DEFAULT_COLD_START_INTERVAL_MS: u64 = 200;
/// Minimum report interval (SRTT clamp floor).
///
/// Raised from 100ms to 1000ms: parent re-evaluation runs every 60s,
/// so 60 samples/cycle is more than sufficient for EWMA convergence (~10).
/// The cold-start phase uses `DEFAULT_COLD_START_INTERVAL_MS` (200ms) for
/// fast initial SRTT convergence before transitioning to this floor.
pub const MIN_REPORT_INTERVAL_MS: u64 = 1_000;
/// Maximum report interval (SRTT clamp ceiling).
pub const MAX_REPORT_INTERVAL_MS: u64 = 5_000;
/// Number of SRTT samples before transitioning from cold-start to normal floor.
///
/// During cold-start, report intervals use `DEFAULT_COLD_START_INTERVAL_MS` as
/// the floor to gather SRTT samples quickly. After this many updates, the floor
/// switches to `MIN_REPORT_INTERVAL_MS`.
pub const COLD_START_SAMPLES: u32 = 5;
/// Default OWD ring buffer capacity.
pub const DEFAULT_OWD_WINDOW_SIZE: usize = 32;
/// Default operator log interval in seconds.
pub const DEFAULT_LOG_INTERVAL_SECS: u64 = 30;
// --- Session-layer timing defaults ---
// Session reports are routed end-to-end (bandwidth cost on every transit link),
// so intervals are higher than link-layer.
/// Session-layer minimum report interval.
pub const MIN_SESSION_REPORT_INTERVAL_MS: u64 = 500;
/// Session-layer maximum report interval.
pub const MAX_SESSION_REPORT_INTERVAL_MS: u64 = 10_000;
/// Session-layer cold-start report interval (before SRTT is available).
pub const SESSION_COLD_START_INTERVAL_MS: u64 = 1_000;
// ============================================================================
// Operating Mode
// ============================================================================
/// MMP operating mode.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MmpMode {
/// Sender + receiver reports at RTT-adaptive intervals. Maximum fidelity.
#[default]
Full,
/// Receiver reports only. Loss inferred from counter gaps.
Lightweight,
/// Spin bit + CE echo only. No reports exchanged.
Minimal,
}
impl fmt::Display for MmpMode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
MmpMode::Full => write!(f, "full"),
MmpMode::Lightweight => write!(f, "lightweight"),
MmpMode::Minimal => write!(f, "minimal"),
}
}
}
// ============================================================================
// Configuration
// ============================================================================
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
// ============================================================================
// Per-Peer MMP State
// ============================================================================
/// Combined MMP state for a single peer link.
///
/// Wraps sender, receiver, metrics, and spin bit state. One instance
/// per `ActivePeer`.
pub struct MmpPeerState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
}
impl MmpPeerState {
/// Create MMP state for a new peer link.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &MmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new(),
receiver: ReceiverState::new(config.owd_window_size),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
// ============================================================================
// Per-Session MMP State (session-layer instantiation)
// ============================================================================
/// Combined MMP state for a single end-to-end session.
///
/// Wraps sender, receiver, metrics, spin bit, and path MTU state.
/// One instance per established `SessionEntry`.
pub struct MmpSessionState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
pub path_mtu: PathMtuState,
}
impl MmpSessionState {
/// Create MMP state for a new session.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &crate::config::SessionMmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new_with_cold_start(SESSION_COLD_START_INTERVAL_MS),
receiver: ReceiverState::new_with_cold_start(
config.owd_window_size,
SESSION_COLD_START_INTERVAL_MS,
),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
path_mtu: PathMtuState::new(),
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
impl Debug for MmpSessionState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpSessionState")
.field("mode", &self.mode)
.field("path_mtu", &self.path_mtu.current_mtu())
.finish_non_exhaustive()
}
}
// ============================================================================
// Path MTU State (session-layer only)
// ============================================================================
/// Path MTU tracking for a single session.
///
/// Destination side: observes `path_mtu` from incoming SessionDatagram envelopes
/// and generates PathMtuNotification messages back to the source.
///
/// Source side: applies received PathMtuNotification to limit outbound datagram
/// size. Decrease is immediate; increase requires 3 consecutive notifications.
pub struct PathMtuState {
/// Current effective path MTU (what we use for sending).
current_mtu: u16,
/// Last observed path MTU from incoming datagrams (destination-side).
last_observed_mtu: u16,
/// Whether the observed MTU has changed since the last notification.
observed_changed: bool,
/// Last time a PathMtuNotification was sent.
last_notification_time: Option<Instant>,
/// Notification interval: max(10s, 5 * SRTT). Default 10s.
notification_interval: Duration,
/// For source-side increase tracking: consecutive higher-value notifications.
consecutive_increase_count: u8,
/// Time of the first notification in the current increase sequence.
first_increase_time: Option<Instant>,
/// The MTU value being proposed for increase.
pending_increase_mtu: u16,
}
impl PathMtuState {
/// Create path MTU state with no initial measurement.
pub fn new() -> Self {
Self {
current_mtu: u16::MAX,
last_observed_mtu: u16::MAX,
observed_changed: false,
last_notification_time: None,
notification_interval: Duration::from_secs(10),
consecutive_increase_count: 0,
first_increase_time: None,
pending_increase_mtu: 0,
}
}
/// Current effective path MTU (source-side, for sending).
pub fn current_mtu(&self) -> u16 {
self.current_mtu
}
/// Last observed incoming path MTU (destination-side).
pub fn last_observed_mtu(&self) -> u16 {
self.last_observed_mtu
}
/// Update notification interval from SRTT: max(10s, 5 * SRTT).
pub fn update_interval_from_srtt(&mut self, srtt_ms: f64) {
let five_srtt = Duration::from_millis((srtt_ms * 5.0) as u64);
self.notification_interval = five_srtt.max(Duration::from_secs(10));
}
/// Seed source-side current_mtu from outbound transport MTU.
///
/// Called on each send. Only decreases (never increases) the current_mtu
/// so the destination's PathMtuNotification can still raise it later.
/// Ensures current_mtu doesn't stay at u16::MAX before any notification
/// arrives from the destination.
pub fn seed_source_mtu(&mut self, outbound_mtu: u16) {
if outbound_mtu < self.current_mtu {
self.current_mtu = outbound_mtu;
}
}
// --- Destination side ---
/// Observe the path_mtu from an incoming SessionDatagram envelope.
///
/// Called on the destination (receiver) side for every session message.
pub fn observe_incoming_mtu(&mut self, path_mtu: u16) {
if path_mtu != self.last_observed_mtu {
self.observed_changed = true;
self.last_observed_mtu = path_mtu;
}
}
/// Check if a PathMtuNotification should be sent.
///
/// Send on first measurement, on decrease (immediate), or periodic
/// confirmation at the notification interval.
pub fn should_send_notification(&self, now: Instant) -> bool {
if self.last_observed_mtu == u16::MAX {
return false; // No measurement yet
}
match self.last_notification_time {
None => true, // First measurement
Some(last) => {
// Immediate on decrease
if self.observed_changed && self.last_observed_mtu < self.current_mtu {
return true;
}
// Periodic confirmation
now.duration_since(last) >= self.notification_interval
}
}
}
/// Build a PathMtuNotification from current state.
///
/// Returns the path_mtu value to send. Caller handles encoding.
pub fn build_notification(&mut self, now: Instant) -> Option<u16> {
if self.last_observed_mtu == u16::MAX {
return None;
}
self.last_notification_time = Some(now);
self.observed_changed = false;
Some(self.last_observed_mtu)
}
// --- Source side ---
/// Apply a received PathMtuNotification.
///
/// - Decrease: immediate (take the lower value).
/// - Increase: require 3 consecutive notifications with the same higher
/// value, spanning at least 2 * notification_interval.
///
/// Returns `true` if the effective MTU changed.
pub fn apply_notification(&mut self, reported_mtu: u16, now: Instant) -> bool {
if reported_mtu < self.current_mtu {
// Decrease: immediate
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
if reported_mtu > self.current_mtu {
// Increase: track consecutive notifications
if reported_mtu == self.pending_increase_mtu {
self.consecutive_increase_count += 1;
} else {
// Different value: reset sequence
self.pending_increase_mtu = reported_mtu;
self.consecutive_increase_count = 1;
self.first_increase_time = Some(now);
}
// Accept increase after 3 consecutive spanning 2 * interval
if self.consecutive_increase_count >= 3
&& let Some(first_time) = self.first_increase_time
{
let required = self.notification_interval * 2;
if now.duration_since(first_time) >= required {
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
}
}
// No change (equal or increase not yet confirmed)
false
}
}
impl Default for PathMtuState {
fn default() -> Self {
Self::new()
}
}
impl Debug for MmpPeerState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpPeerState")
.field("mode", &self.mode)
.finish_non_exhaustive()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_mode_default() {
assert_eq!(MmpMode::default(), MmpMode::Full);
}
#[test]
fn test_mode_display() {
assert_eq!(MmpMode::Full.to_string(), "full");
assert_eq!(MmpMode::Lightweight.to_string(), "lightweight");
assert_eq!(MmpMode::Minimal.to_string(), "minimal");
}
#[test]
fn test_mode_serde_roundtrip() {
let yaml = "full";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Full);
let yaml = "lightweight";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Lightweight);
let yaml = "minimal";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Minimal);
}
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
}

View File

@@ -1,385 +0,0 @@
//! MMP report wire format: SenderReport and ReceiverReport.
//!
//! Serialization and deserialization for the two report types exchanged
//! between link-layer peers. Wire format follows the MMP design doc.
use crate::protocol::ProtocolError;
// ============================================================================
// SenderReport (msg_type 0x01, 48-byte body including type byte)
// ============================================================================
/// Link-layer sender report.
///
/// Wire layout (48 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x01
/// [1-3] reserved (zero)
/// [4-11] interval_start_counter: u64 LE
/// [12-19] interval_end_counter: u64 LE
/// [20-23] interval_start_timestamp: u32 LE
/// [24-27] interval_end_timestamp: u32 LE
/// [28-31] interval_bytes_sent: u32 LE
/// [32-39] cumulative_packets_sent: u64 LE
/// [40-47] cumulative_bytes_sent: u64 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SenderReport {
pub interval_start_counter: u64,
pub interval_end_counter: u64,
pub interval_start_timestamp: u32,
pub interval_end_timestamp: u32,
pub interval_bytes_sent: u32,
pub cumulative_packets_sent: u64,
pub cumulative_bytes_sent: u64,
}
/// ReceiverReport (msg_type 0x02, 68-byte body including type byte)
///
/// Wire layout (68 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x02
/// [1-3] reserved (zero)
/// [4-11] highest_counter: u64 LE
/// [12-19] cumulative_packets_recv: u64 LE
/// [20-27] cumulative_bytes_recv: u64 LE
/// [28-31] timestamp_echo: u32 LE
/// [32-33] dwell_time: u16 LE
/// [34-35] max_burst_loss: u16 LE
/// [36-37] mean_burst_loss: u16 LE (u8.8 fixed-point)
/// [38-39] reserved: u16 LE
/// [40-43] jitter: u32 LE (microseconds)
/// [44-47] ecn_ce_count: u32 LE
/// [48-51] owd_trend: i32 LE (µs/s)
/// [52-55] burst_loss_count: u32 LE
/// [56-59] cumulative_reorder_count: u32 LE
/// [60-63] interval_packets_recv: u32 LE
/// [64-67] interval_bytes_recv: u32 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReceiverReport {
pub highest_counter: u64,
pub cumulative_packets_recv: u64,
pub cumulative_bytes_recv: u64,
pub timestamp_echo: u32,
pub dwell_time: u16,
pub max_burst_loss: u16,
pub mean_burst_loss: u16,
pub jitter: u32,
pub ecn_ce_count: u32,
pub owd_trend: i32,
pub burst_loss_count: u32,
pub cumulative_reorder_count: u32,
pub interval_packets_recv: u32,
pub interval_bytes_recv: u32,
}
// Encode/decode will be implemented in Step 2.
impl SenderReport {
/// Encode to wire format (48 bytes: msg_type + 3 reserved + 44 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(48);
buf.push(0x01); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.interval_start_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_end_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_start_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_end_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_sent.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 47 {
return Err(ProtocolError::MessageTooShort {
expected: 47,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
interval_start_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
interval_end_counter: u64::from_le_bytes(p[8..16].try_into().unwrap()),
interval_start_timestamp: u32::from_le_bytes(p[16..20].try_into().unwrap()),
interval_end_timestamp: u32::from_le_bytes(p[20..24].try_into().unwrap()),
interval_bytes_sent: u32::from_le_bytes(p[24..28].try_into().unwrap()),
cumulative_packets_sent: u64::from_le_bytes(p[28..36].try_into().unwrap()),
cumulative_bytes_sent: u64::from_le_bytes(p[36..44].try_into().unwrap()),
})
}
}
impl ReceiverReport {
/// Encode to wire format (68 bytes: msg_type + 3 reserved + 64 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(68);
buf.push(0x02); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.highest_counter.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_recv.to_le_bytes());
buf.extend_from_slice(&self.timestamp_echo.to_le_bytes());
buf.extend_from_slice(&self.dwell_time.to_le_bytes());
buf.extend_from_slice(&self.max_burst_loss.to_le_bytes());
buf.extend_from_slice(&self.mean_burst_loss.to_le_bytes());
buf.extend_from_slice(&[0u8; 2]); // reserved
buf.extend_from_slice(&self.jitter.to_le_bytes());
buf.extend_from_slice(&self.ecn_ce_count.to_le_bytes());
buf.extend_from_slice(&self.owd_trend.to_le_bytes());
buf.extend_from_slice(&self.burst_loss_count.to_le_bytes());
buf.extend_from_slice(&self.cumulative_reorder_count.to_le_bytes());
buf.extend_from_slice(&self.interval_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_recv.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 67 {
return Err(ProtocolError::MessageTooShort {
expected: 67,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
highest_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
cumulative_packets_recv: u64::from_le_bytes(p[8..16].try_into().unwrap()),
cumulative_bytes_recv: u64::from_le_bytes(p[16..24].try_into().unwrap()),
timestamp_echo: u32::from_le_bytes(p[24..28].try_into().unwrap()),
dwell_time: u16::from_le_bytes(p[28..30].try_into().unwrap()),
max_burst_loss: u16::from_le_bytes(p[30..32].try_into().unwrap()),
mean_burst_loss: u16::from_le_bytes(p[32..34].try_into().unwrap()),
// skip 2 reserved bytes at p[34..36]
jitter: u32::from_le_bytes(p[36..40].try_into().unwrap()),
ecn_ce_count: u32::from_le_bytes(p[40..44].try_into().unwrap()),
owd_trend: i32::from_le_bytes(p[44..48].try_into().unwrap()),
burst_loss_count: u32::from_le_bytes(p[48..52].try_into().unwrap()),
cumulative_reorder_count: u32::from_le_bytes(p[52..56].try_into().unwrap()),
interval_packets_recv: u32::from_le_bytes(p[56..60].try_into().unwrap()),
interval_bytes_recv: u32::from_le_bytes(p[60..64].try_into().unwrap()),
})
}
}
// ============================================================================
// Conversions between link-layer and session-layer report types
// ============================================================================
use crate::protocol::{SessionReceiverReport, SessionSenderReport};
impl From<&SenderReport> for SessionSenderReport {
fn from(r: &SenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&SessionSenderReport> for SenderReport {
fn from(r: &SessionSenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&ReceiverReport> for SessionReceiverReport {
fn from(r: &ReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
impl From<&SessionReceiverReport> for ReceiverReport {
fn from(r: &SessionReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
fn sample_sender_report() -> SenderReport {
SenderReport {
interval_start_counter: 100,
interval_end_counter: 200,
interval_start_timestamp: 5000,
interval_end_timestamp: 6000,
interval_bytes_sent: 50_000,
cumulative_packets_sent: 10_000,
cumulative_bytes_sent: 5_000_000,
}
}
fn sample_receiver_report() -> ReceiverReport {
ReceiverReport {
highest_counter: 195,
cumulative_packets_recv: 9_500,
cumulative_bytes_recv: 4_750_000,
timestamp_echo: 5900,
dwell_time: 5,
max_burst_loss: 3,
mean_burst_loss: 384, // 1.5 in u8.8
jitter: 1200,
ecn_ce_count: 0,
owd_trend: -50,
burst_loss_count: 2,
cumulative_reorder_count: 10,
interval_packets_recv: 95,
interval_bytes_recv: 47_500,
}
}
#[test]
fn test_sender_report_encode_size() {
let sr = sample_sender_report();
let encoded = sr.encode();
assert_eq!(encoded.len(), 48);
assert_eq!(encoded[0], 0x01); // msg_type
}
#[test]
fn test_sender_report_roundtrip() {
let sr = sample_sender_report();
let encoded = sr.encode();
// decode expects payload after msg_type
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_sender_report_too_short() {
let result = SenderReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_receiver_report_encode_size() {
let rr = sample_receiver_report();
let encoded = rr.encode();
assert_eq!(encoded.len(), 68);
assert_eq!(encoded[0], 0x02); // msg_type
}
#[test]
fn test_receiver_report_roundtrip() {
let rr = sample_receiver_report();
let encoded = rr.encode();
// decode expects payload after msg_type
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_too_short() {
let result = ReceiverReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_sender_report_zero_values() {
let sr = SenderReport {
interval_start_counter: 0,
interval_end_counter: 0,
interval_start_timestamp: 0,
interval_end_timestamp: 0,
interval_bytes_sent: 0,
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
};
let encoded = sr.encode();
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_receiver_report_max_values() {
let rr = ReceiverReport {
highest_counter: u64::MAX,
cumulative_packets_recv: u64::MAX,
cumulative_bytes_recv: u64::MAX,
timestamp_echo: u32::MAX,
dwell_time: u16::MAX,
max_burst_loss: u16::MAX,
mean_burst_loss: u16::MAX,
jitter: u32::MAX,
ecn_ce_count: u32::MAX,
owd_trend: i32::MAX,
burst_loss_count: u32::MAX,
cumulative_reorder_count: u32::MAX,
interval_packets_recv: u32::MAX,
interval_bytes_recv: u32::MAX,
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_negative_owd_trend() {
let rr = ReceiverReport {
owd_trend: -12345,
..sample_receiver_report()
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(decoded.owd_trend, -12345);
}
}

View File

@@ -1,418 +0,0 @@
//! MMP sender state machine.
//!
//! Tracks what this node has sent to a specific peer and produces
//! SenderReport messages on demand. One `SenderState` per active peer.
use std::time::{Duration, Instant};
use crate::mmp::report::SenderReport;
use crate::mmp::{
COLD_START_SAMPLES, DEFAULT_COLD_START_INTERVAL_MS, MAX_REPORT_INTERVAL_MS,
MIN_REPORT_INTERVAL_MS,
};
/// Per-peer sender-side MMP state.
///
/// Records cumulative and interval counters for every frame transmitted
/// to this peer. Produces `SenderReport` snapshots on demand.
pub struct SenderState {
// --- Cumulative (lifetime) ---
cumulative_packets_sent: u64,
cumulative_bytes_sent: u64,
// --- Current interval ---
interval_start_counter: u64,
interval_start_timestamp: u32,
interval_bytes_sent: u32,
/// Counter of the most recently sent frame.
last_counter: u64,
/// Timestamp of the most recently sent frame.
last_timestamp: u32,
/// Whether any frames have been sent in the current interval.
interval_has_data: bool,
// --- Report timing ---
last_report_time: Option<Instant>,
report_interval: Duration,
// --- Send failure backoff ---
/// Consecutive send failure count for backoff calculation.
consecutive_send_failures: u32,
// --- Cold-start tracking ---
/// Number of SRTT-based interval updates received.
srtt_sample_count: u32,
}
impl SenderState {
pub fn new() -> Self {
Self::new_with_cold_start(DEFAULT_COLD_START_INTERVAL_MS)
}
/// Create with a custom cold-start interval (ms).
///
/// Used by session-layer MMP which needs a longer initial interval
/// since reports consume bandwidth on every transit link.
pub fn new_with_cold_start(cold_start_ms: u64) -> Self {
Self {
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
interval_start_counter: 0,
interval_start_timestamp: 0,
interval_bytes_sent: 0,
last_counter: 0,
last_timestamp: 0,
interval_has_data: false,
last_report_time: None,
report_interval: Duration::from_millis(cold_start_ms),
consecutive_send_failures: 0,
srtt_sample_count: 0,
}
}
/// Record a frame sent to this peer.
///
/// Called on the TX path for every encrypted link message.
/// `counter` is the AEAD nonce/counter, `timestamp` is the inner header
/// session-relative timestamp (ms), `bytes` is the wire payload size.
pub fn record_sent(&mut self, counter: u64, timestamp: u32, bytes: usize) {
if !self.interval_has_data {
self.interval_start_counter = counter;
self.interval_start_timestamp = timestamp;
self.interval_has_data = true;
}
self.last_counter = counter;
self.last_timestamp = timestamp;
self.interval_bytes_sent = self.interval_bytes_sent.saturating_add(bytes as u32);
self.cumulative_packets_sent += 1;
self.cumulative_bytes_sent += bytes as u64;
}
/// Build a SenderReport from current state and reset the interval.
///
/// Returns `None` if no frames have been sent since the last report.
pub fn build_report(&mut self, now: Instant) -> Option<SenderReport> {
if !self.interval_has_data {
return None;
}
let report = SenderReport {
interval_start_counter: self.interval_start_counter,
interval_end_counter: self.last_counter,
interval_start_timestamp: self.interval_start_timestamp,
interval_end_timestamp: self.last_timestamp,
interval_bytes_sent: self.interval_bytes_sent,
cumulative_packets_sent: self.cumulative_packets_sent,
cumulative_bytes_sent: self.cumulative_bytes_sent,
};
// Reset interval
self.interval_has_data = false;
self.interval_bytes_sent = 0;
self.last_report_time = Some(now);
Some(report)
}
/// Check if it's time to send a report.
///
/// When consecutive send failures have occurred, the effective interval
/// is multiplied by an exponential backoff factor (2^failures, capped at 32×).
pub fn should_send_report(&self, now: Instant) -> bool {
if !self.interval_has_data {
return false;
}
match self.last_report_time {
None => true, // Never sent a report — send immediately
Some(last) => {
let effective = self
.report_interval
.mul_f64(self.send_failure_backoff_multiplier());
now.duration_since(last) >= effective
}
}
}
/// Record a send failure. Returns the new consecutive failure count.
pub fn record_send_failure(&mut self) -> u32 {
self.consecutive_send_failures += 1;
self.consecutive_send_failures
}
/// Record a successful send. Returns the previous failure count (for summary logging).
pub fn record_send_success(&mut self) -> u32 {
let prev = self.consecutive_send_failures;
self.consecutive_send_failures = 0;
prev
}
/// Get the backoff multiplier based on consecutive failures.
///
/// Returns 1.0 for no failures, 2.0 for 1 failure, 4.0 for 2, ...
/// capped at 32.0 (5 failures).
pub fn send_failure_backoff_multiplier(&self) -> f64 {
if self.consecutive_send_failures == 0 {
1.0
} else {
2.0_f64.powi(self.consecutive_send_failures.min(5) as i32)
}
}
/// Update the report interval based on SRTT (link-layer defaults).
///
/// Sender reports at 2× SRTT clamped to [floor, MAX]. During cold-start
/// (first `COLD_START_SAMPLES` updates), the floor is the cold-start
/// interval (200ms) for fast SRTT convergence. After that, it rises to
/// `MIN_REPORT_INTERVAL_MS` (1000ms) for steady-state efficiency.
pub fn update_report_interval_from_srtt(&mut self, srtt_us: i64) {
self.srtt_sample_count = self.srtt_sample_count.saturating_add(1);
let floor = if self.srtt_sample_count <= COLD_START_SAMPLES {
DEFAULT_COLD_START_INTERVAL_MS
} else {
MIN_REPORT_INTERVAL_MS
};
self.update_report_interval_with_bounds(srtt_us, floor, MAX_REPORT_INTERVAL_MS);
}
/// Update the report interval based on SRTT with custom bounds.
///
/// Used by session-layer MMP which needs higher clamp values since
/// each report consumes bandwidth on every transit link.
pub fn update_report_interval_with_bounds(&mut self, srtt_us: i64, min_ms: u64, max_ms: u64) {
if srtt_us <= 0 {
return;
}
let interval_us = (srtt_us * 2) as u64;
let interval_ms = (interval_us / 1000).clamp(min_ms, max_ms);
self.report_interval = Duration::from_millis(interval_ms);
}
// --- Accessors ---
pub fn cumulative_packets_sent(&self) -> u64 {
self.cumulative_packets_sent
}
pub fn cumulative_bytes_sent(&self) -> u64 {
self.cumulative_bytes_sent
}
pub fn report_interval(&self) -> Duration {
self.report_interval
}
pub fn consecutive_send_failures(&self) -> u32 {
self.consecutive_send_failures
}
}
impl Default for SenderState {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_new_sender_state() {
let s = SenderState::new();
assert_eq!(s.cumulative_packets_sent(), 0);
assert_eq!(s.cumulative_bytes_sent(), 0);
}
#[test]
fn test_record_sent() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
s.record_sent(2, 200, 600);
assert_eq!(s.cumulative_packets_sent(), 2);
assert_eq!(s.cumulative_bytes_sent(), 1100);
}
#[test]
fn test_build_report_empty() {
let mut s = SenderState::new();
assert!(s.build_report(Instant::now()).is_none());
}
#[test]
fn test_build_report() {
let mut s = SenderState::new();
s.record_sent(10, 1000, 500);
s.record_sent(11, 1100, 600);
s.record_sent(12, 1200, 400);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 10);
assert_eq!(report.interval_end_counter, 12);
assert_eq!(report.interval_start_timestamp, 1000);
assert_eq!(report.interval_end_timestamp, 1200);
assert_eq!(report.interval_bytes_sent, 1500);
assert_eq!(report.cumulative_packets_sent, 3);
assert_eq!(report.cumulative_bytes_sent, 1500);
}
#[test]
fn test_build_report_resets_interval() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
let _ = s.build_report(Instant::now());
// Second report with no new data returns None
assert!(s.build_report(Instant::now()).is_none());
// New data starts a fresh interval
s.record_sent(2, 200, 300);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 2);
assert_eq!(report.interval_bytes_sent, 300);
// Cumulative continues
assert_eq!(report.cumulative_packets_sent, 2);
assert_eq!(report.cumulative_bytes_sent, 800);
}
#[test]
fn test_should_send_report_no_data() {
let s = SenderState::new();
assert!(!s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_first_time() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
assert!(s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_respects_interval() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
s.record_sent(2, 200, 500);
// Immediately after report — should not send
assert!(!s.should_send_report(t0));
// After interval elapses
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(s.should_send_report(t1));
}
#[test]
fn test_update_report_interval_cold_start() {
let mut s = SenderState::new();
// During cold-start, floor is 200ms (DEFAULT_COLD_START_INTERVAL_MS)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to cold-start floor 200ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(s.report_interval(), Duration::from_millis(200));
// 500ms RTT → 1000ms sender interval (above cold-start floor)
s.update_report_interval_from_srtt(500_000);
assert_eq!(s.report_interval(), Duration::from_millis(1000));
}
#[test]
fn test_update_report_interval_after_cold_start() {
let mut s = SenderState::new();
// Burn through cold-start samples (COLD_START_SAMPLES = 5)
for _ in 0..COLD_START_SAMPLES {
s.update_report_interval_from_srtt(500_000);
}
// 6th sample: now in steady state, floor is MIN_REPORT_INTERVAL_MS (1000ms)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to 1000ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MIN_REPORT_INTERVAL_MS)
);
// 3s RTT → 6s, clamped to max 5s
s.update_report_interval_from_srtt(3_000_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MAX_REPORT_INTERVAL_MS)
);
}
#[test]
fn test_backoff_multiplier_progression() {
let mut s = SenderState::new();
// No failures → multiplier 1.0
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
assert_eq!(s.consecutive_send_failures(), 0);
// Progressive failures: 2^1, 2^2, 2^3, 2^4, 2^5
let expected = [2.0, 4.0, 8.0, 16.0, 32.0];
for (i, &exp) in expected.iter().enumerate() {
let count = s.record_send_failure();
assert_eq!(count, (i + 1) as u32);
assert_eq!(s.send_failure_backoff_multiplier(), exp);
}
// Beyond 5 failures: stays capped at 32.0
s.record_send_failure(); // 6th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
s.record_send_failure(); // 7th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
}
#[test]
fn test_backoff_reset_on_success() {
let mut s = SenderState::new();
// Accumulate failures
s.record_send_failure();
s.record_send_failure();
s.record_send_failure();
assert_eq!(s.consecutive_send_failures(), 3);
assert_eq!(s.send_failure_backoff_multiplier(), 8.0);
// Success resets and returns previous count
let prev = s.record_send_success();
assert_eq!(prev, 3);
assert_eq!(s.consecutive_send_failures(), 0);
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
}
#[test]
fn test_backoff_success_with_no_prior_failures() {
let mut s = SenderState::new();
// Success with no failures returns 0
let prev = s.record_send_success();
assert_eq!(prev, 0);
assert_eq!(s.consecutive_send_failures(), 0);
}
#[test]
fn test_should_send_report_respects_backoff() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
// Record a failure: multiplier becomes 2.0
s.record_send_failure();
s.record_sent(2, 200, 500);
// At 1× interval: should NOT send (backoff requires 2×)
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(!s.should_send_report(t1));
// At 2× interval: should send
let t2 = t0 + s.report_interval() * 2 + Duration::from_millis(1);
assert!(s.should_send_report(t2));
}
}

View File

@@ -4,12 +4,12 @@
//! including debounced propagation to peers.
use crate::NodeAddr;
use crate::bloom::BloomFilter;
use crate::protocol::FilterAnnounce;
use crate::proto::bloom::BloomFilter;
use crate::proto::bloom::FilterAnnounce;
use super::reject::BloomReject;
use super::{Node, NodeError};
use std::collections::HashMap;
use std::collections::BTreeMap;
use tracing::{debug, warn};
impl Node {
@@ -17,8 +17,8 @@ impl Node {
///
/// Returns a map of (peer_node_addr -> filter) for peers that
/// have sent us a FilterAnnounce.
pub(super) fn peer_inbound_filters(&self) -> HashMap<NodeAddr, BloomFilter> {
let mut filters = HashMap::new();
pub(super) fn peer_inbound_filters(&self) -> BTreeMap<NodeAddr, BloomFilter> {
let mut filters = BTreeMap::new();
for (addr, peer) in &self.peers {
if self.is_tree_peer(addr)
&& let Some(filter) = peer.inbound_filter()
@@ -87,7 +87,7 @@ impl Node {
// operator to see one clear message, not spam.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let out_fill = sent_filter.fill_ratio();
let out_fpr = out_fill.powi(sent_filter.hash_count() as i32);
let out_fpr = sent_filter.fpr();
if out_fpr > max_fpr {
let now = std::time::Instant::now();
let should_warn = self
@@ -213,7 +213,7 @@ impl Node {
// to wipe a victim's contribution to aggregation.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let fill = announce.filter.fill_ratio();
let fpr = fill.powi(announce.filter.hash_count() as i32);
let fpr = announce.filter.fpr();
if fpr > max_fpr {
self.metrics()
.bloom

View File

@@ -142,20 +142,24 @@ impl Node {
(peer_sa, local, recv_buf, send_buf, tx)
};
// Open the connected socket on the kernel side.
let socket = std::sync::Arc::new(
crate::transport::udp::connected_peer::ConnectedPeerSocket::open(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("ConnectedPeerSocket::open: {e}"))?,
);
// Open the connected socket on the kernel side, then adopt the
// fd into the owning handle.
let owned = crate::transport::udp::open_connected_fd(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("open_connected_fd: {e}"))?;
let socket = std::sync::Arc::new(crate::peer::connected_udp::ConnectedPeerSocket::from_fd(
owned,
peer_socket_addr,
local_addr,
));
// Spawn the drain thread. It feeds `packet_tx` exactly like
// the wildcard listen socket — rx_loop dispatches identically.
let drain = crate::transport::udp::peer_drain::PeerRecvDrain::spawn(
let drain = crate::peer::connected_udp::PeerRecvDrain::spawn(
socket.clone(),
transport_id,
peer_socket_addr,

View File

@@ -73,7 +73,7 @@ impl Node {
/// entries — other removal paths (link-dead, decrypt failure, peer
/// restart) all schedule reconnect.
pub(in crate::node) fn handle_disconnect(&mut self, from: &NodeAddr, payload: &[u8]) {
let disconnect = match crate::protocol::Disconnect::decode(payload) {
let disconnect = match crate::proto::fmp::Disconnect::decode(payload) {
Ok(msg) => msg,
Err(e) => {
debug!(from = %self.peer_display_name(from), error = %e, "Malformed disconnect message");
@@ -93,7 +93,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
/// Remove an active peer and clean up all associated state.
@@ -187,6 +187,15 @@ impl Node {
// Remove link and address mapping
self.remove_link(&link_id);
// Bound `peer_machines`: drop this peer's machine
// entry, keyed by the `link_id` derived above BEFORE the `peers` removal.
// This cleans up the OLD peer's machine on an inbound restart and prevents
// unbounded growth on the establish success path. NEUTRAL: nothing on the
// live path reads `peer_machines` except the establish executor, which only
// ever touches the in-flight establish's (distinct) `link_id`; no reader
// depends on a stale entry, so removal changes no behavior — it only bounds
// the map.
self.peer_machines.remove(&link_id);
if let Some(transport_id) = transport_id {
self.cleanup_bootstrap_transport_if_unused(transport_id);
}

View File

@@ -1,10 +1,11 @@
//! Encrypted frame handling (hot path).
use crate::node::Node;
use crate::node::wire::{EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header};
use crate::noise::NoiseError;
use crate::proto::fmp::wire::{
EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header,
};
use crate::transport::ReceivedPacket;
use std::time::Instant;
use tracing::{debug, trace, warn};
/// Force-remove a peer after this many consecutive decryption failures.
@@ -171,7 +172,7 @@ impl Node {
#[cfg(unix)]
{
let cache_key = (packet.transport_id, header.receiver_idx.as_u32());
if let Some(workers) = self.decrypt_workers.as_ref().cloned()
if let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned()
&& self.decrypt_registered_sessions.contains(&cache_key)
{
let job = crate::node::decrypt_worker::DecryptJob {
@@ -259,7 +260,7 @@ impl Node {
};
// MMP per-frame processing and statistics
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let ce_flag = header.flags & FLAG_CE != 0;
let sp_flag = header.flags & FLAG_SP != 0;
@@ -270,9 +271,9 @@ impl Node {
timestamp,
packet.data.len(),
ce_flag,
now,
now_ms,
);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now_ms);
}
peer.set_current_addr(packet.transport_id, packet.remote_addr.clone());
peer.link_stats_mut()
@@ -355,7 +356,7 @@ impl Node {
} else {
return;
};
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let mut address_changed = false;
if let Some(peer) = self.peers.get_mut(node_addr) {
peer.reset_decrypt_failures();
@@ -365,8 +366,8 @@ impl Node {
peer.touch(packet_timestamp_ms);
if let Some(mmp) = peer.mmp_mut() {
mmp.receiver
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now);
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now_ms);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now_ms);
}
}
// Address rotation invalidates the per-peer connect()-ed UDP
@@ -450,7 +451,7 @@ impl Node {
/// black-hole the session.
#[cfg(unix)]
pub(in crate::node) fn register_decrypt_worker_session(&mut self, node_addr: &crate::NodeAddr) {
let Some(workers) = self.decrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned() else {
return;
};
let (cache_key, state) = {
@@ -491,7 +492,7 @@ impl Node {
&mut self,
cache_key: (crate::transport::TransportId, u32),
) {
if let Some(workers) = self.decrypt_workers.as_ref() {
if let Some(workers) = self.supervisor.decrypt_workers.as_ref() {
workers.unregister_session(cache_key);
}
self.decrypt_registered_sessions.remove(&cache_key);
@@ -533,7 +534,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
}
}

View File

@@ -7,15 +7,14 @@
use crate::NodeAddr;
use crate::node::reject::ForwardingReject;
use crate::node::session_wire::{
use crate::node::{Node, NodeError, NodeRoutingView};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED, FSP_PHASE_MSG1, FSP_PHASE_MSG2,
FspCommonPrefix, parse_encrypted_coords,
};
use crate::node::{Node, NodeError};
use crate::protocol::{
CoordsRequired, MtuExceeded, PathBroken, SessionAck, SessionDatagram, SessionDatagramRef,
SessionSetup,
};
use crate::proto::fsp::{SessionAck, SessionSetup};
use crate::proto::link::{SessionDatagram, SessionDatagramRef};
use crate::proto::routing::{DropReason, NextHop, RouteAction, RouteOutcome};
use std::time::{Duration, Instant};
use tracing::{debug, warn};
@@ -43,123 +42,166 @@ impl Node {
}
};
// TTL enforcement: decrement for forwarding and drop only if the
// received datagram was already exhausted.
if datagram_ref.ttl == 0 {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
"SessionDatagram TTL exhausted, dropping"
);
return;
}
let forwarded_ttl = datagram_ref.ttl - 1;
let my_addr = *self.node_addr();
// Coordinate cache warming from plaintext session-layer headers
self.try_warm_coord_cache_ref(&datagram_ref);
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
if datagram_ref.dest_addr == *self.node_addr() {
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
return;
// Coordinate cache warming from plaintext session-layer headers. Gated
// on a non-exhausted TTL so a datagram the core will drop as
// TTL-exhausted does not warm the cache, matching the pre-refactor
// ordering (warming ran only after the TTL early-return).
if datagram_ref.ttl != 0 {
self.try_warm_coord_cache_ref(&datagram_ref);
}
let mut datagram = datagram_ref.into_owned();
datagram.ttl = forwarded_ttl;
// Pre-resolve the next hop only for genuine transit packets (TTL > 0
// and not locally destined) so `find_next_hop`'s coord-cache LRU-touch
// side effect keeps the same scope it had inline. Warming above has
// already run, so the resolution observes freshly cached coords.
let next_hop = if datagram_ref.ttl != 0 && datagram_ref.dest_addr != my_addr {
self.resolve_next_hop(&datagram_ref.dest_addr)
} else {
None
};
// Find next hop toward destination
let next_hop_addr = match self.find_next_hop(&datagram.dest_addr) {
Some(peer) => *peer.node_addr(),
None => {
// Read local congestion once and reuse it for both the CE decision
// (via the view) and the congestion metric/log below, keeping
// `detect_congestion` the single source of truth.
let congested = next_hop
.as_ref()
.map(|nh| self.detect_congestion(&nh.addr))
.unwrap_or(false);
// Borrow the routing tables disjointly from `&mut self.routing` for
// the pure decision, then release both before driving the outcome.
let outcome = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested,
};
self.routing
.route(&datagram_ref, &my_addr, incoming_ce, next_hop, &view)
};
match outcome {
RouteOutcome::Drop {
reason: DropReason::TtlExhausted,
} => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
"SessionDatagram TTL exhausted, dropping"
);
}
RouteOutcome::DeliverLocal => {
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
}
RouteOutcome::NoRoute => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::NoRoute, payload.len());
let original = datagram_ref.into_owned();
debug!(
src = %self.peer_display_name(&datagram.src_addr),
dest = %self.peer_display_name(&datagram.dest_addr),
src = %self.peer_display_name(&original.src_addr),
dest = %self.peer_display_name(&original.dest_addr),
bytes = payload.len(),
"Dropping transit SessionDatagram: no route to destination"
);
self.send_routing_error(&datagram).await;
return;
self.send_routing_error(&original).await;
}
};
RouteOutcome::Forward {
next_hop,
bytes,
outgoing_ce,
} => {
let dest = datagram_ref.dest_addr;
// Apply path_mtu min() from the outgoing link's transport MTU
if let Some(peer) = self.peers.get(&next_hop_addr)
// ECN CE relay: congestion was detected locally above; emit the
// metric and rate-limited log at the transit chokepoint.
if congested {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop, "Congestion detected, CE flag set on forwarded packet");
}
}
match self
.send_encrypted_link_message_with_ce(&next_hop, &bytes, outgoing_ce)
.await
{
Err(NodeError::MtuExceeded { mtu, .. }) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(dest, datagram_ref.src_addr, mtu)
.await;
}
Err(e) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop,
dest = %dest,
error = %e,
"Failed to forward SessionDatagram"
);
}
Ok(()) => {
self.metrics().forwarding.record_forwarded(bytes.len());
// Classify this transit forward by route class (partition
// of forwarded_packets). Done here, at the data-plane
// chokepoint, so the error-signal routing callers of
// find_next_hop are excluded.
let class = self.classify_forward(&dest, &next_hop);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
}
}
}
}
/// Resolve the next hop toward `dest` into its address plus the outgoing
/// link's transport MTU. Returns `None` when there is no route.
///
/// The MTU defaults to `u16::MAX` (a no-op min-fold) when the peer's
/// transport is not resolvable, matching the pre-refactor inline behavior
/// where the MTU `if let` chain simply did not fire.
fn resolve_next_hop(&mut self, dest: &NodeAddr) -> Option<NextHop> {
let addr = *self.find_next_hop(dest)?.node_addr();
let link_mtu = if let Some(peer) = self.peers.get(&addr)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
if let Some(addr) = peer.current_addr() {
datagram.path_mtu = datagram.path_mtu.min(transport.link_mtu(addr));
} else {
datagram.path_mtu = datagram.path_mtu.min(transport.mtu());
}
}
// ECN CE relay: propagate incoming CE and detect local congestion
let local_congestion = self.detect_congestion(&next_hop_addr);
let outgoing_ce = incoming_ce || local_congestion;
if local_congestion {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop_addr, "Congestion detected, CE flag set on forwarded packet");
}
}
// Forward: re-encode (includes 0x00 type byte) and send
let encoded = datagram.encode();
if let Err(e) = self
.send_encrypted_link_message_with_ce(&next_hop_addr, &encoded, outgoing_ce)
.await
{
match e {
NodeError::MtuExceeded { mtu, .. } => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(&datagram, mtu).await;
}
_ => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop_addr,
dest = %datagram.dest_addr,
error = %e,
"Failed to forward SessionDatagram"
);
}
match peer.current_addr() {
Some(link_addr) => transport.link_mtu(link_addr),
None => transport.mtu(),
}
} else {
self.metrics().forwarding.record_forwarded(encoded.len());
// Classify this transit forward by route class (partition of
// forwarded_packets). Done here, at the data-plane chokepoint, so
// the error-signal routing callers of find_next_hop are excluded.
let class = self.classify_forward(&datagram.dest_addr, &next_hop_addr);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
u16::MAX
};
Some(NextHop { addr, link_mtu })
}
/// Attempt to warm the coordinate cache from session-layer payload headers.
@@ -260,35 +302,41 @@ impl Node {
/// If we can't route the error back to the source either, drop silently.
/// No cascading errors.
async fn send_routing_error(&mut self, original: &SessionDatagram) {
// Rate limit: one error signal per destination per 100ms
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
let my_addr = *self.node_addr();
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let default_ttl = self.config().node.session.default_ttl;
let error_payload =
if let Some(coords) = self.coord_cache().get(&original.dest_addr, now_ms) {
let coords = coords.clone();
PathBroken::new(original.dest_addr, my_addr)
.with_last_coords(coords)
.encode()
} else {
CoordsRequired::new(original.dest_addr, my_addr).encode()
// Pure decision: rate-limit gate + PathBroken/CoordsRequired choice +
// error-PDU encode. Borrow the routing tables disjointly from
// `&mut self.routing`, then release them before the reverse-hop lookup.
let action = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested: false,
};
self.routing.synth_routing_error(
&original.dest_addr,
&original.src_addr,
&my_addr,
&view,
now_ms,
default_ttl,
)
};
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
@@ -300,9 +348,8 @@ impl Node {
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -324,37 +371,50 @@ impl Node {
/// Called when `send_encrypted_link_message()` fails with
/// `NodeError::MtuExceeded` during forwarding. The signal tells the
/// source the bottleneck MTU so it can immediately reduce its path MTU.
async fn send_mtu_exceeded_error(&mut self, original: &SessionDatagram, bottleneck_mtu: u16) {
// Rate limit: reuse routing_error_rate_limiter keyed on dest_addr
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
///
/// `dest` is the failed datagram's destination (rate-limit key); `toward`
/// is its source, where the signal is routed back.
async fn send_mtu_exceeded_error(
&mut self,
dest: NodeAddr,
toward: NodeAddr,
bottleneck_mtu: u16,
) {
let my_addr = *self.node_addr();
let now_ms = Self::now_ms();
let default_ttl = self.config().node.session.default_ttl;
let error_payload = MtuExceeded::new(original.dest_addr, my_addr, bottleneck_mtu).encode();
// Pure decision: rate-limit gate + MtuExceeded PDU + encode.
let action = self.routing.synth_mtu_exceeded(
&dest,
&toward,
&my_addr,
bottleneck_mtu,
now_ms,
default_ttl,
);
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
src = %original.src_addr,
dest = %original.dest_addr,
src = %toward,
dest = %dest,
"Cannot route MtuExceeded signal back to source, dropping"
);
return;
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -364,8 +424,8 @@ impl Node {
);
} else {
debug!(
original_dest = %original.dest_addr,
error_dest = %original.src_addr,
original_dest = %dest,
error_dest = %toward,
bottleneck_mtu,
"Sent MtuExceeded error signal"
);

18
src/node/dataplane/mod.rs Normal file
View File

@@ -0,0 +1,18 @@
//! Data plane: the RX `select!` loop and the per-packet forwarding path.
//!
//! Holds the whole hot path in one home: the `select!` run loop
//! (`rx_loop`), transit/local datagram forwarding (`forwarding`), the
//! link-message router (`dispatch`), the RX decrypt path including responder
//! K-bit cutover and address-roam writes (`encrypted`), and the per-peer
//! connected-UDP fast-path socket activation (`connected_udp`). Each module
//! contributes `impl Node` methods driven by the run loop.
#[cfg(unix)]
pub(crate) mod connected_udp;
mod dispatch;
mod encrypted;
mod forwarding;
mod peer_actions;
mod rx_loop;
pub(in crate::node) use peer_actions::PeerActionCtx;

View File

@@ -0,0 +1,468 @@
//! Executor for the per-peer control machine's [`PeerAction`]s.
//!
//! The per-peer FSM in [`crate::peer::machine`] is a sans-IO reducer: it decides
//! *what* must happen and returns a `Vec<PeerAction>`; this module is the *doing*
//! half — the thin driver that maps each action onto the exact shell call it
//! stands for (`build_msg2` + `transport.send`, `promote_connection`,
//! `remove_active_peer`, `index_allocator.free`, `note_link_dead`, …).
//!
//! ## Shadow-only skeleton
//!
//! The machine home (`Node.peer_machines`), the
//! executor, and the disjoint-borrow advance helper. It is **unwired** — the live
//! `handle_msg1`/`handle_msg2` path does not drive it yet, so every method here is
//! `#[allow(dead_code)]`. The inbound cutover (`handle_msg1` → `step(InboundMsg1)`)
//! and the outbound cutover (`handle_msg2` / dial) are wired later.
//!
//! Arms not yet exercised are inert stubs (outbound dial, rekey/crypto installs,
//! link-control frames, timers, and the connected-UDP plane are inert stubs
//! realized as those planes are wired).
//! `RegisterDecryptSession` is a deliberate no-op — see its arm for the note.
use crate::node::Node;
use crate::node::reject::{HandshakeReject, RejectReason};
use crate::peer::machine::{PeerAction, PeerEvent};
use crate::proto::fmp::PromotionResult;
use crate::proto::fmp::wire::build_msg2;
use crate::transport::{LinkId, TransportAddr, TransportId};
use crate::utils::index::SessionIndex;
use crate::{NodeAddr, PeerIdentity};
use std::collections::VecDeque;
use tracing::{debug, trace, warn};
/// Ambient shell facts a [`PeerAction`] executor needs that the machine's
/// runtime-agnostic action payloads deliberately omit (verified identity,
/// transport target, the msg2 framing indices, the promotion timestamp).
///
/// Unlike a machine event/action payload this is **executor-side**, so it may
/// hold real values resolved from the wire context (cf. `handle_msg1`'s
/// `wire`/`packet` locals and `promote_connection`'s ambient args). It is
/// built fresh per driven step by the caller at cutover time.
#[allow(dead_code)]
pub(in crate::node) struct PeerActionCtx {
/// The authenticated peer identity: `PromoteToActive` /
/// `InvalidateSendState` resolve their `NodeAddr` from this.
pub(in crate::node) verified_identity: PeerIdentity,
/// The transport the exchange is happening over (msg2 send target, decrypt
/// cache-key transport half).
pub(in crate::node) transport_id: TransportId,
/// The peer's wire address (msg2 send target).
pub(in crate::node) remote_addr: TransportAddr,
/// Our session index for this exchange (msg2 framing sender_idx).
pub(in crate::node) our_index: Option<SessionIndex>,
/// The peer's session index for this exchange (msg2 framing
/// receiver_idx).
pub(in crate::node) their_index: Option<SessionIndex>,
/// The wire timestamp driving this step (promotion ts / loss-report clock).
pub(in crate::node) now_ms: u64,
/// Establish direction for this exchange. Discriminates the
/// `PromoteToActive` failure cleanup: the pre-refactor inbound
/// (`handle_msg1`) and outbound (`handle_msg2`) promote-Err arms were NOT
/// byte-identical, so the executor must reproduce each. `false` = inbound
/// (drop link + reverse map + free index), `true` = outbound (record the
/// reject only; leave the dead link/`addr_to_link` for the stale-connection
/// reaper, matching old `handle_msg2`).
pub(in crate::node) is_outbound: bool,
}
impl Node {
/// Advance the machine for `link` by one event and execute the resulting
/// actions.
///
/// The borrow structure the whole seam turns on: the machine
/// needs `&mut IndexAllocator` as a synchronous capability *while it is
/// itself borrowed mutably out of `peer_machines`*. `peer_machines` and
/// `index_allocator` are **distinct `Node` fields**, so the collect below is
/// a disjoint two-field borrow the checker accepts; once the actions are
/// collected both borrows drop and the executor runs against `&mut self`.
#[allow(dead_code)]
pub(in crate::node) async fn advance_peer_machine(
&mut self,
link: LinkId,
event: PeerEvent,
now: u64,
ambient: &PeerActionCtx,
) {
let actions = match self.peer_machines.get_mut(&link) {
// Disjoint field borrow: `self.peer_machines` (the map entry) and
// `self.index_allocator` (the capability) are separate fields.
Some(machine) => machine.step(event, now, &mut self.index_allocator),
None => return,
};
self.execute_peer_actions(link, ambient, actions).await;
}
/// Map each [`PeerAction`] onto its shell call.
///
/// `PromoteToActive` feeds its [`PromotionResult`](crate::proto::fmp::PromotionResult)
/// back into the machine and appends the follow-up actions to the same
/// worklist — a queue rather than self-recursion so the async executor stays a
/// single flat future (no boxing) and the emitted order is preserved (the
/// establish sequences always end in `PromoteToActive`, so its follow-ups run
/// after any siblings).
#[allow(dead_code)]
pub(in crate::node) async fn execute_peer_actions(
&mut self,
link: LinkId,
ambient: &PeerActionCtx,
actions: Vec<PeerAction>,
) {
let mut queue: VecDeque<PeerAction> = actions.into();
while let Some(action) = queue.pop_front() {
match action {
PeerAction::OpenTransport { .. } => {
// Outbound dial (`initiate_connection`,
// `lifecycle/mod.rs:470`). Outbound establish is not cut over
// yet; inert in the shadow-only skeleton.
}
PeerAction::SendHandshake { bytes } => {
// The machine payload is the UNFRAMED Noise msg2 payload;
// frame it with our/their index (mirrors `handshake.rs:472`'s
// `build_msg2(our_index, their_index, &payload)`) before the
// wire send. A fresh-outbound msg1 (empty payload → build msg1
// from indices) is framed differently and is wired later.
if let (Some(sender_idx), Some(receiver_idx)) =
(ambient.our_index, ambient.their_index)
{
let frame = build_msg2(sender_idx, receiver_idx, &bytes);
// Surface the send Result. A missing transport skips
// the send and continues (mirrors `handle_msg1`'s
// `if let Some(transport)` guard); a send *error* runs the
// pre-refactor msg2-send-failure cleanup (`handle_msg1`
// L494-503) and ABORTS the remaining queue so the queued
// `PromoteToActive` never runs.
let send_err = match self.transports.get(&ambient.transport_id) {
Some(transport) => {
transport.send(&ambient.remote_addr, &frame).await.err()
}
None => None,
};
if let Some(e) = send_err {
// Restored pre-refactor msg2-send-failure warn!
// (`handle_msg1` L665): the send error text is surfaced
// at the executor point where the failure is now handled.
warn!(link_id = %link, error = %e, "Failed to send msg2");
self.connections.remove(&link);
self.links.remove(&link);
self.addr_to_link
.remove(&(ambient.transport_id, ambient.remote_addr.clone()));
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.peer_machines.remove(&link);
self.stats_mut()
.record_reject(RejectReason::Handshake(HandshakeReject::BadState));
return;
}
}
}
PeerAction::SendRekey { .. } => {
// Rekey msg2 framing (`build_msg2(our_new_index, …)`,
// `handshake.rs:365`) + send. Rekey fold is not yet wired.
}
PeerAction::SendLinkMessage { .. } => {
// Encrypt + send a link-control frame (heartbeat / filter
// / tree / disconnect). Data-plane-owned; not yet wired.
}
PeerAction::PromoteToActive { link: promote_link } => {
// Ambient supplies the verified identity + promotion ts
// that `promote_connection` needs (resolved from the wire ctx).
match self.promote_connection(
promote_link,
ambient.verified_identity,
ambient.now_ms,
) {
Ok(result) => {
// The decrypt-worker registration relocated
// OUT of `promote_connection` into THIS single executor
// arm — the one live caller of `promote_connection` (both
// the inbound `handle_msg1` and outbound `handle_msg2`
// net-new establish paths reach it here). Register iff the
// promotion actually created or replaced a peer
// (`Promoted | CrossConnectionWon`), NEVER on
// `CrossConnectionLost`. Run synchronously right after
// `promote_connection` returns, before feeding
// `PromotionResolved` and before any await — the exact
// synchronous point (and Promoted/Won gating) of the
// pre-refactor in-`promote_connection` call. No-op when
// the worker pool isn't spawned (`register_...` early-
// returns), so the direct `promote_connection` test
// callers (which bypass this executor) are unaffected.
#[cfg(unix)]
match result {
PromotionResult::Promoted(node_addr)
| PromotionResult::CrossConnectionWon { node_addr, .. } => {
self.register_decrypt_worker_session(&node_addr);
}
PromotionResult::CrossConnectionLost { .. } => {}
}
// Feed the outcome back into the machine and fold the
// follow-up actions (RegisterDecryptSession — now a
// redundant no-op, see its arm — and the cross-conn index
// frees) into the worklist. Disjoint field borrow again.
let follow = match self.peer_machines.get_mut(&promote_link) {
Some(machine) => machine.step(
PeerEvent::PromotionResolved { result },
ambient.now_ms,
&mut self.index_allocator,
),
None => Vec::new(),
};
queue.extend(follow);
// Defensive cross-connection loser-link surgery.
// LINK-ONLY: close the losing transport connection, drop
// its link, and re-point `addr_to_link`, reproducing the
// pre-refactor inline `handle_msg2`/`handle_msg1` per-arm
// order EXACTLY. The index-plane frees/unregisters are
// owned by the machine's `PromotionResolved{Won/Lost}`
// follow-up (queued just above), so NOTHING here touches
// an index — no double-free.
//
// UNREACHABLE on every current driven path: the inbound
// and outbound net-new establish arms only route to the
// machine when no promoted peer exists for the node_addr
// (and `RestartThenPromote` removes the old peer first),
// so `promote_connection` always returns `Promoted`. The
// `debug_assert!(false, ..)` catches any future path that
// drives a cross-connection through the executor without
// the matching send-state handling.
match result {
PromotionResult::CrossConnectionWon { loser_link_id, .. } => {
debug_assert!(
false,
"executor CrossConnectionWon is unreachable on \
driven net-new establish paths"
);
// Close the losing transport connection (no-op for
// connectionless) via the LOSER link's own
// transport/addr, then drop the losing link.
if let Some(loser_link) = self.links.get(&loser_link_id) {
let loser_tid = loser_link.transport_id();
let loser_addr = loser_link.remote_addr().clone();
if let Some(transport) = self.transports.get(&loser_tid) {
transport.close_connection(&loser_addr).await;
}
}
self.remove_link(&loser_link_id);
// Point `addr_to_link` at the winning (current)
// link.
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
promote_link,
);
}
PromotionResult::CrossConnectionLost { winner_link_id } => {
debug_assert!(
false,
"executor CrossConnectionLost is unreachable on \
driven net-new establish paths"
);
// Close this (losing) connection, drop its link,
// and restore `addr_to_link` to the winner.
if let Some(transport) =
self.transports.get(&ambient.transport_id)
{
transport.close_connection(&ambient.remote_addr).await;
}
self.remove_link(&promote_link);
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
winner_link_id,
);
}
PromotionResult::Promoted(_) => {}
}
}
Err(e) => {
// Promotion failed. `promote_connection` already
// removed `connections[link]` and (on error) handled its
// own index internally. The pre-refactor inbound and
// outbound promote-Err arms were NOT byte-identical, so
// discriminate on `ambient.is_outbound`. The queue is
// drained (PromoteToActive is the last establish action),
// so no explicit abort.
if ambient.is_outbound {
// OLD outbound (`handle_msg2` promote-Err): warn +
// record_reject ONLY. NO `remove_link`, NO
// `index_allocator.free`, NO `addr_to_link` removal —
// the dead link/addr_to_link/pending_outbound were
// left for the 30s stale-connection reaper
// (`promote_connection` already handled
// `connections[link]`/its index on error). Restored
// pre-refactor outbound warn! ("Failed to promote
// connection").
//
// The transient outbound machine was inserted BEFORE
// execute; it is additive state that
// did not exist pre-refactor, so removing the just-
// inserted machine on failure is neutral vs old and
// prevents a leak.
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote connection"
);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
self.peer_machines.remove(&promote_link);
} else {
// OLD inbound (`handle_msg1` L587-591): drop the link
// + reverse map, free our index, discard the machine,
// and record the reject. Restored pre-refactor inbound
// promote-failure warn! (`handle_msg1` L757).
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote inbound connection"
);
self.remove_link(&promote_link);
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.peer_machines.remove(&promote_link);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
}
}
}
}
PeerAction::SwapSendState { .. } => {
// Initiator cutover. Reproduces the `ConnAction::Cutover`
// body in `handlers/rekey.rs:53-88` EXACTLY. `addr` is resolved
// from the ambient verified identity (as `InvalidateSendState`
// does). The decrypt re-register folds HERE, gated on
// `did_cutover` — the generic `RegisterDecryptSession` arm stays a
// no-op so a promote never double-registers. Shadow-only until the
// cadence fold routes here.
let node_addr = *ambient.verified_identity.node_addr();
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
// Pin the target to the pre-refactor module: this
// cutover log relocated from handlers/rekey.rs into
// the executor, but operators (and the test harness)
// filter it under fips::node::handlers::rekey. Keeping
// the target preserves the observable log contract.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
PeerAction::CompleteDrain { peer: node_addr } => {
// Initiator drain completion. Reproduces the
// `ConnAction::Drain` body in `handlers/rekey.rs:90-111` EXACTLY.
// Extract the real previous index + transport_id under the peer
// borrow, drop the borrow, then run the cache_key cleanup (which
// takes &mut self for unregister_decrypt_worker_session).
// Shadow-only until the cadence fold routes here.
let drained = self.peers.get_mut(&node_addr).and_then(|peer| {
peer.complete_drain().map(|idx| (idx, peer.transport_id()))
});
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
// Pin to the pre-refactor module (see the cutover log
// above) so the relocated drain log stays visible under
// the operator's fips::node::handlers::rekey filter.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
PeerAction::InvalidateSendState => {
// The FULL teardown. `remove_active_peer`
// (`dispatch.rs:107`) frees the four index slots
// (current/rekey/pending/previous), drops `peers_by_index`,
// unregisters the decrypt worker, removes the FSP `sessions`
// entry and `pending_tun_packets`. The machine emits NO
// `FreeIndex` for those slots, so there is no double-free.
self.remove_active_peer(ambient.verified_identity.node_addr());
}
PeerAction::RegisterDecryptSession { index } => {
let _ = index;
// No-op by design. The decrypt-worker
// registration relocated into the `PromoteToActive` Ok arm above, gated on
// the returned `PromotionResult`, so it runs once per live
// promote (Promoted/Won) at the pre-refactor synchronous point.
// This machine-emitted action is now redundant with that arm;
// kept as an inert no-op (rather than removing the emission) so
// the machine's action sequence and its unit tests stay
// unchanged. The keyed-by-NodeAddr register does not need the
// machine's `index` payload.
}
PeerAction::UnregisterDecryptSession { index } => {
// Executor supplies `transport_id` from ambient; keyed by
// (tid, index) like `remove_active_peer` / the rekey drain path.
#[cfg(unix)]
self.unregister_decrypt_worker_session((ambient.transport_id, index.as_u32()));
#[cfg(not(unix))]
let _ = index;
}
PeerAction::FreeIndex { index } => {
let _ = self.index_allocator.free(index);
}
PeerAction::ActivateConnectedUdp | PeerAction::TeardownConnectedUdp => {
// Connected-UDP plane ownership (`connected_udp.rs`).
}
PeerAction::SetTimer { .. } | PeerAction::CancelTimer { .. } => {
// Timers become actions on the existing quantized tick.
// INERT — the legacy tick timers still run, so driving
// these would double-schedule.
}
PeerAction::ReportLost { peer } => {
// The single loss token → the reconciler reflex (`driver.rs:48`).
self.report_peer_lost(peer, ambient.now_ms);
}
}
}
}
/// `ReportLost` → `note_link_dead` (kept as a named seam so the ambient clock
/// source is explicit and the reconciler-computed backoff can be threaded later).
#[allow(dead_code)]
fn report_peer_lost(&mut self, peer: NodeAddr, now_ms: u64) {
self.note_link_dead(peer, now_ms);
}
}

View File

@@ -1,10 +1,10 @@
//! RX event loop and packet dispatch.
use crate::control::{ControlSocket, commands};
use crate::node::wire::{
use crate::node::{Node, NodeError};
use crate::proto::fmp::wire::{
COMMON_PREFIX_SIZE, CommonPrefix, FMP_VERSION, PHASE_ESTABLISHED, PHASE_MSG1, PHASE_MSG2,
};
use crate::node::{Node, NodeError};
use crate::transport::ReceivedPacket;
use std::time::Duration;
use tracing::{debug, info, warn};
@@ -42,12 +42,42 @@ impl Node {
/// This method takes ownership of the packet_rx channel and runs
/// until the channel is closed (typically when stop() is called).
pub async fn run_rx_loop(&mut self) -> Result<(), NodeError> {
// No shutdown observer → today's infinite loop, byte-identical. All
// existing callers/tests use this; `pending()` never fires, so the
// shutdown/deadline arms below stay permanently disabled.
self.run_rx_loop_with_shutdown(std::future::pending()).await
}
/// The rx event loop, which serves until `shutdown` fires and then drains
/// **in place** before returning.
///
/// The channel receivers are moved into this frame's locals and live across
/// both serve and drain, so — unlike a `select!`-cancelled loop — they are
/// never destructively dropped mid-flight; they are released only on clean
/// exit, after which teardown does not need them.
///
/// - While serving (`drain_deadline == None`) the loop is behaviorally
/// identical to before: the shutdown arm, the deadline arm, and the
/// peers-empty early-exit are all guarded off, so the hot per-packet path
/// and the `biased` order of the real arms are unchanged.
/// - When `shutdown` fires, the loop calls [`Node::enter_drain`] once
/// (broadcast Disconnect, gate the reconciler off) and arms the bounded
/// deadline, then keeps servicing inbound/tick/peer-removal until all
/// peers clear or the deadline elapses, then returns. The caller
/// ([`Node::finish_shutdown`]) closes the window and tears down.
pub async fn run_rx_loop_with_shutdown(
&mut self,
shutdown: impl std::future::Future<Output = ()>,
) -> Result<(), NodeError> {
tokio::pin!(shutdown);
// `None` = serving; `Some(deadline)` = draining (bounded window).
let mut drain_deadline: Option<tokio::time::Instant> = None;
let mut packet_rx = self.packet_rx.take().ok_or(NodeError::NotStarted)?;
// Take the TUN outbound receiver, or create a dummy channel that never
// produces messages (when TUN is disabled). Holding the sender prevents
// the channel from closing.
let (mut tun_outbound_rx, _tun_guard) = match self.tun_outbound_rx.take() {
let (mut tun_outbound_rx, _tun_guard) = match self.supervisor.tun_outbound_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -57,7 +87,19 @@ impl Node {
// Take the DNS identity receiver, or create a dummy channel (when DNS
// is disabled). Same pattern as TUN outbound.
let (mut dns_identity_rx, _dns_guard) = match self.dns_identity_rx.take() {
let (mut dns_identity_rx, _dns_guard) = match self.supervisor.dns_identity_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
(rx, Some(tx))
}
};
// Take the runtime child-liveness receiver, or a dummy channel (when the
// node was seeded straight into Running without a start()). Holding the
// dummy sender in the guard keeps the channel open. Same pattern as TUN
// outbound / DNS identity.
let (mut child_exit_rx, _child_exit_guard) = match self.child_exit_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -122,6 +164,13 @@ impl Node {
crate::perf_profile::maybe_spawn_reporter();
loop {
// Bounded drain mode: break as soon as all peers have cleared. In
// normal mode (`None`) this short-circuits before touching
// `self.peers`, so the loop is byte-identical.
if drain_deadline.is_some() && self.peers.is_empty() {
info!("Drain complete: all peers cleared, ending drain loop");
break;
}
tokio::select! {
biased;
// Decrypt-worker fallback drains FIRST. Under sustained
@@ -214,6 +263,27 @@ impl Node {
}
}
}
// Runtime child-liveness. Placed AFTER `packet_rx` so the hot
// inbound path keeps its `biased` priority. A directly-observable
// child (TUN threads, DNS/mDNS/Nostr) exited on its own; feed the
// FSM, which republishes health (Degraded here — a Running node
// always has ≥1 transport up). `on_child_exited` only ever emits
// `PublishState`; other variants are ignored defensively.
maybe_child = child_exit_rx.recv() => {
if let Some(child) = maybe_child {
let actions = self
.supervisor
.fsm
.step(crate::node::lifecycle::supervisor::Event::ChildExited { child });
for action in actions {
if let crate::node::lifecycle::supervisor::Action::PublishState(ns) =
action
{
self.supervisor.state = ns;
}
}
}
}
Some(ipv6_packet) = tun_outbound_rx.recv() => {
self.handle_tun_outbound(ipv6_packet).await;
let mut drained = 0;
@@ -257,13 +327,13 @@ impl Node {
// is polled separately from `reload_peer_acl` because the
// ACL's embedded alias reloader and this snapshot are
// distinct resources; the `path_mtu_lookup` cache and the
// `nostr_discovery` subsystem are deliberately excluded
// `nostr_rendezvous` subsystem are deliberately excluded
// from `Reloadable` since neither reloads from a backing
// file (see `node::reloadable`).
self.reload_host_map().await;
self.poll_pending_connects().await;
self.poll_nostr_discovery().await;
self.poll_lan_discovery().await;
self.poll_nostr_rendezvous().await;
self.poll_lan_rendezvous().await;
self.resend_pending_handshakes(now_ms).await;
self.resend_pending_rekeys(now_ms).await;
self.resend_pending_session_handshakes(now_ms).await;
@@ -285,6 +355,27 @@ impl Node {
#[cfg(any(target_os = "linux", target_os = "macos"))]
self.activate_connected_udp_sessions().await;
}
// Shutdown signal → enter the bounded drain in place, ONCE.
// Gated on `is_none()` so it only fires while serving; after
// entering drain the arm is disabled (the completed signal is
// never polled again) and the deadline arm below bounds the
// window. Placed after the real arms so their `biased` priority
// is unchanged, and inert while serving with `pending()`.
_ = &mut shutdown, if drain_deadline.is_none() => {
self.enter_drain().await;
drain_deadline =
Some(tokio::time::Instant::now() + self.config().node.drain_timeout());
}
// Bounded drain deadline (drain mode only). Placed LAST so the
// `biased` priority of the normal arms is unchanged, and gated
// on `is_some()` so in normal mode the branch is disabled — the
// future is created but never polled and never fires.
_ = tokio::time::sleep_until(
drain_deadline.unwrap_or_else(tokio::time::Instant::now)
), if drain_deadline.is_some() => {
info!("Drain deadline elapsed, ending drain loop");
break;
}
}
}
@@ -319,12 +410,16 @@ impl Node {
// though no msg1/msg2 exchange can ever succeed. Bump the
// discovery-layer cooldown to the long protocol-mismatch
// window and emit a single WARN per fresh observation.
if self.bootstrap_transports.contains(&packet.transport_id)
if self
.supervisor
.nostr_rendezvous
.is_bootstrap_transport(&packet.transport_id)
&& let Some(npub) = self
.bootstrap_transport_npubs
.get(&packet.transport_id)
.supervisor
.nostr_rendezvous
.bootstrap_transport_npub(&packet.transport_id)
.cloned()
&& let Some(handle) = self.nostr_discovery_handle()
&& let Some(handle) = self.nostr_rendezvous_handle()
{
let now_ms = Self::now_ms();
let cooldown_secs = handle.protocol_mismatch_cooldown_secs();

View File

@@ -561,7 +561,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound2);
let counter: u64 = 7;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
// Build a wire packet `[16-byte header][4-byte inner ts][1 byte link msg]`
// with capacity for the trailing AEAD tag. Header bytes
// double as AAD and as the on-wire prefix.
@@ -569,7 +569,7 @@ mod tests {
// Header: fill the flags byte (the second byte) with both
// FLAG_CE and FLAG_SP set; the rest is uninterpreted by the
// worker (it just AADs the whole 16 bytes).
let flags_byte = crate::node::wire::FLAG_CE | crate::node::wire::FLAG_SP;
let flags_byte = crate::proto::fmp::wire::FLAG_CE | crate::proto::fmp::wire::FLAG_SP;
let mut header = [0u8; HDR];
header[1] = flags_byte;
wire.extend_from_slice(&header);
@@ -626,11 +626,11 @@ mod tests {
"fmp_flags must round-trip from DecryptJob to DecryptFallback"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_CE != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_CE != 0,
"FLAG_CE bit lost on worker path"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_SP != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_SP != 0,
"FLAG_SP bit lost on worker path"
);
}
@@ -724,7 +724,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound);
let counter: u64 = 11;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
let header = [0u8; HDR];
let mut wire = Vec::with_capacity(HDR + 4 + 1 + 16);
wire.extend_from_slice(&header);

View File

@@ -1,376 +0,0 @@
//! Discovery protocol rate limiting and backoff.
//!
//! Two complementary mechanisms:
//!
//! - **`DiscoveryBackoff`** (originator-side, optional): Exponential
//! suppression of fresh lookups after the per-attempt sequence in
//! `node.discovery.attempt_timeouts_secs` has been exhausted.
//! **Disabled by default** (base/cap = 0); the per-attempt sequence
//! is the only retry pacing in the standard configuration. Reset on
//! topology changes (parent change, new peer, first RTT, reconnection).
//!
//! - **`DiscoveryForwardRateLimiter`** (transit-side): Per-target minimum
//! interval for forwarded requests. Defense-in-depth against misbehaving
//! nodes generating fresh request_ids at high rate.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
// ============================================================================
// Originator-side: Discovery Backoff
// ============================================================================
/// Default base backoff after first lookup failure. `0` = disabled.
const DEFAULT_BACKOFF_BASE_SECS: u64 = 0;
/// Default maximum backoff cap. `0` = disabled.
const DEFAULT_BACKOFF_MAX_SECS: u64 = 0;
/// Backoff multiplier per consecutive failure.
const BACKOFF_MULTIPLIER: u64 = 2;
/// Exponential backoff for failed discovery lookups.
///
/// Tracks targets whose lookups have timed out and suppresses
/// re-initiation with increasing delays. Cleared on topology changes.
pub struct DiscoveryBackoff {
/// Maps target → (suppress_until, consecutive_failures).
entries: HashMap<NodeAddr, BackoffEntry>,
/// Base backoff duration (first failure).
base: Duration,
/// Maximum backoff cap.
max: Duration,
}
struct BackoffEntry {
/// Don't re-initiate until this instant.
suppress_until: Instant,
/// Consecutive failures (drives exponential backoff).
failures: u32,
}
impl DiscoveryBackoff {
/// Create with default parameters (disabled — base/cap = 0).
pub fn new() -> Self {
Self::with_params(DEFAULT_BACKOFF_BASE_SECS, DEFAULT_BACKOFF_MAX_SECS)
}
/// Create with custom base and max backoff in seconds.
pub fn with_params(base_secs: u64, max_secs: u64) -> Self {
Self {
entries: HashMap::new(),
base: Duration::from_secs(base_secs),
max: Duration::from_secs(max_secs),
}
}
/// Check if a lookup for this target is suppressed.
///
/// Returns true if the target is in backoff and should not be
/// looked up yet.
pub fn is_suppressed(&self, target: &NodeAddr) -> bool {
if let Some(entry) = self.entries.get(target) {
Instant::now() < entry.suppress_until
} else {
false
}
}
/// Record a lookup failure (timeout) for a target.
///
/// Increments the failure count and sets the next suppression
/// window using exponential backoff.
pub fn record_failure(&mut self, target: &NodeAddr) {
let now = Instant::now();
let failures = self.entries.get(target).map_or(0, |e| e.failures) + 1;
let backoff_secs = self
.base
.as_secs()
.saturating_mul(BACKOFF_MULTIPLIER.saturating_pow(failures.saturating_sub(1)));
let backoff = Duration::from_secs(backoff_secs.min(self.max.as_secs()));
self.entries.insert(
*target,
BackoffEntry {
suppress_until: now + backoff,
failures,
},
);
}
/// Record a successful lookup — remove backoff for this target.
pub fn record_success(&mut self, target: &NodeAddr) {
self.entries.remove(target);
}
/// Clear all backoff entries.
///
/// Called on topology changes that might make previously-unreachable
/// targets reachable (parent change, new peer, first RTT, reconnection).
pub fn reset_all(&mut self) {
self.entries.clear();
}
/// Whether any entries exist.
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// Current number of entries.
pub fn entry_count(&self) -> usize {
self.entries.len()
}
/// Get the failure count for a target (for logging).
pub fn failure_count(&self, target: &NodeAddr) -> u32 {
self.entries.get(target).map_or(0, |e| e.failures)
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.entries.len()
}
}
impl Default for DiscoveryBackoff {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Transit-side: Discovery Forward Rate Limiter
// ============================================================================
/// Default minimum interval between forwarded lookups for the same target.
const DEFAULT_FORWARD_MIN_INTERVAL: Duration = Duration::from_secs(2);
/// Maximum age of entries before cleanup.
const FORWARD_MAX_AGE: Duration = Duration::from_secs(60);
/// Rate limiter for forwarded discovery requests.
///
/// Tracks the last time a LookupRequest was forwarded for each target
/// and enforces a minimum interval to prevent floods from misbehaving
/// nodes generating fresh request_ids.
pub struct DiscoveryForwardRateLimiter {
last_forwarded: HashMap<NodeAddr, Instant>,
min_interval: Duration,
max_age: Duration,
}
impl DiscoveryForwardRateLimiter {
/// Create with default parameters (2s interval).
pub fn new() -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval: DEFAULT_FORWARD_MIN_INTERVAL,
max_age: FORWARD_MAX_AGE,
}
}
/// Create with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval,
max_age: FORWARD_MAX_AGE,
}
}
/// Check if we should forward a lookup for this target.
///
/// Returns true if enough time has passed since the last forward
/// for this target. Updates internal state when returning true.
pub fn should_forward(&mut self, target: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_forwarded.get(target)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_forwarded.insert(*target, now);
self.cleanup(now);
true
}
/// Replace the minimum interval (e.g., set to zero to disable).
#[cfg(test)]
pub fn set_interval(&mut self, interval: Duration) {
self.min_interval = interval;
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_forwarded
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_forwarded.len()
}
}
impl Default for DiscoveryForwardRateLimiter {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
// --- DiscoveryBackoff tests ---
#[test]
fn test_backoff_not_suppressed_initially() {
let backoff = DiscoveryBackoff::new();
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_suppressed_after_failure() {
// Backoff is opt-in; exercise the suppression path with explicit params.
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
// Different target not affected
assert!(!backoff.is_suppressed(&addr(2)));
}
#[test]
fn test_backoff_cleared_on_success() {
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
backoff.record_success(&addr(1));
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_reset_all() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
backoff.record_failure(&addr(2));
assert_eq!(backoff.len(), 2);
backoff.reset_all();
assert_eq!(backoff.len(), 0);
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_exponential() {
let mut backoff = DiscoveryBackoff::with_params(1, 300);
// First failure: 1s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 1);
// Second failure: 2s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 2);
// Third failure: 4s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 3);
}
#[test]
fn test_backoff_expires() {
let mut backoff = DiscoveryBackoff::with_params(0, 0);
backoff.record_failure(&addr(1));
// With 0s backoff, should not be suppressed
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_capped() {
let mut backoff = DiscoveryBackoff::with_params(1, 10);
// Record many failures
for _ in 0..20 {
backoff.record_failure(&addr(1));
}
// Backoff should be capped at max (10s), not overflow
let entry = backoff.entries.get(&addr(1)).unwrap();
let remaining = entry.suppress_until.duration_since(Instant::now());
assert!(remaining <= Duration::from_secs(11));
}
// --- DiscoveryForwardRateLimiter tests ---
#[test]
fn test_forward_first_allowed() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_rapid_rate_limited() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_different_targets_independent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(2)));
}
#[test]
fn test_forward_allowed_after_interval() {
let mut limiter = DiscoveryForwardRateLimiter::with_interval(Duration::from_millis(100));
assert!(limiter.should_forward(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_cleanup_removes_old() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(61);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_forward_cleanup_preserves_recent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
}

View File

@@ -50,9 +50,9 @@
// warnings rather than gate every function individually.
#![cfg_attr(not(unix), allow(dead_code))]
use crate::node::session_wire::FSP_HEADER_SIZE;
use crate::node::wire::ESTABLISHED_HEADER_SIZE;
use crate::transport::udp::socket::AsyncUdpSocket;
use crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
use crate::proto::fsp::wire::FSP_HEADER_SIZE;
use crate::transport::udp::io::AsyncUdpSocket;
#[cfg(not(target_os = "macos"))]
use crossbeam_channel::{Receiver, SendError, Sender, TrySendError, bounded};
use ring::aead::{Aad, LessSafeKey, Nonce};
@@ -132,8 +132,7 @@ pub(crate) struct FmpSendJob {
/// the job completes and the worker drops it, only the peer's
/// strong ref remains.
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
pub connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
/// Bulk endpoint data may be dropped when the kernel reports UDP
/// send-queue exhaustion. Control/rekey frames keep retrying so
/// congestion cannot strand the session.
@@ -716,8 +715,7 @@ fn mac_now_ms() -> u64 {
struct MacSequencedSendFlow {
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
next_seq: std::sync::atomic::AtomicU64,
last_used_ms: std::sync::atomic::AtomicU64,
@@ -754,9 +752,7 @@ impl MacSequencedSendFlow {
fn spawn(
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket: Option<
std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>,
>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
now_ms: u64,
) -> Arc<Self> {
@@ -1024,8 +1020,7 @@ fn flush_batch_sync(
struct EncryptedGroup {
socket: AsyncUdpSocket,
#[cfg(any(target_os = "linux", target_os = "macos"))]
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
wire_packets: Vec<Vec<u8>>,
drop_on_backpressure: bool,
@@ -1710,7 +1705,7 @@ fn send_batch_gso(
}
/// Direct `sendmmsg(2)` wrapper for the sync worker. The
/// `transport::udp::socket` module's existing `send_batch` is
/// `transport::udp::io` module's existing `send_batch` is
/// pub(crate) on `UdpRawSocket`, but we don't have a handle to the
/// raw socket from here — we just have the FD. Re-implementing
/// inline is ~15 lines and avoids tunnelling the inner socket
@@ -1780,7 +1775,7 @@ fn send_batch_raw(
#[cfg(all(test, unix))]
mod unix_tests {
use super::*;
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -1904,10 +1899,12 @@ mod unix_tests {
#[test]
fn pipelined_send_wire_layout_roundtrips_canonical_decoders() {
use crate::NodeAddr;
use crate::node::session_wire::build_fsp_header;
use crate::node::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::noise::TAG_SIZE;
use crate::protocol::{LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef};
use crate::proto::fmp::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::proto::fsp::wire::build_fsp_header;
use crate::proto::link::{
LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef,
};
use crate::utils::index::SessionIndex;
let rt = tokio::runtime::Builder::new_current_thread()
@@ -2218,7 +2215,7 @@ mod tests {
/// AsRawFd impl.
#[test]
fn flush_batch_routes_each_target_separately() {
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -2264,7 +2261,7 @@ mod tests {
const B_WIRE: usize = 16 + B_PLAINTEXT + 16; // 96
fn make_job(
socket: crate::transport::udp::socket::AsyncUdpSocket,
socket: crate::transport::udp::io::AsyncUdpSocket,
cipher: &LessSafeKey,
counter: u64,
dest: SocketAddr,

View File

@@ -1,753 +0,0 @@
//! LookupRequest/LookupResponse discovery protocol handlers.
//!
//! Handles coordinate discovery via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::reject::DiscoveryReject;
use crate::node::{Node, RecentRequest};
use crate::protocol::{LookupRequest, LookupResponse};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
const MAX_RECENT_DISCOVERY_REQUESTS: usize = 4096;
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().discovery.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
self.purge_expired_requests(now_ms);
// Dedup: drop if we've already seen this request_id.
// Also serves as loop protection — tree routing is loop-free,
// but request_id dedup catches edge cases during tree restructuring.
if self.recent_requests.contains_key(&request.request_id) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
return;
}
if self.recent_requests.len() >= MAX_RECENT_DISCOVERY_REQUESTS {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = self.recent_requests.len(),
max_recent_requests = MAX_RECENT_DISCOVERY_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
return;
}
// Record for reverse-path forwarding and dedup
self.recent_requests
.insert(request.request_id, RecentRequest::new(*from, now_ms));
// Are we the target?
if request.target == *self.node_addr() {
self.metrics().discovery.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
return;
}
// Forward if TTL permits
if request.can_forward() {
// Transit-side rate limit: collapse rapid-fire lookups for the
// same target from misbehaving nodes generating fresh request_ids.
if !self
.discovery_forward_limiter
.should_forward(&request.target)
{
self.metrics().discovery.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
return;
}
self.metrics().discovery.req_forwarded.inc();
self.forward_lookup_request(request).await;
} else {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().discovery.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
if let Some(recent) = self.recent_requests.get_mut(&response.request_id) {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
if recent.response_forwarded {
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
return;
}
recent.response_forwarded = true;
// Transit node: reverse-path forward
let from_peer = recent.from_peer;
self.metrics().discovery.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
} else {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data =
LookupResponse::proof_bytes(response.request_id, &target, &response.target_coords);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().discovery.resp_accepted.inc();
// Clear backoff on success — target is reachable
self.discovery_backoff.record_success(&target);
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
self.coord_cache
.insert_with_path_mtu(target, response.target_coords, now_ms, path_mtu);
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen the
// clamp. A reactive MtuExceeded or PathMtuNotification
// tighten takes precedence over a looser discovery
// estimate (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
// Clean up pending lookup tracking
self.pending_lookups.remove(&target);
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin via reverse path.
let next_hop_addr = if let Some(recent) = self.recent_requests.get(&request.request_id) {
recent.from_peer
} else {
// Fallback: try greedy tree routing toward origin
match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
}
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
if !request.forward() {
return;
}
// Collect tree peers whose bloom filter contains the target
let forward_to: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
// Fallback: if no tree peer matches, try non-tree bloom-matching peers
let (forward_to, used_fallback) = if forward_to.is_empty() {
let fallback: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| !self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
if fallback.is_empty() {
self.metrics().discovery.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
return;
}
(fallback, true)
} else {
(forward_to, false)
};
if used_fallback {
self.metrics().discovery.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest"
);
}
let encoded = request.encode();
for peer_addr in forward_to {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().discovery.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request = LookupRequest::generate(*target, origin, origin_coords, ttl, 0);
// Send only to tree peers whose bloom filter contains the target
let peer_addrs: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect();
let peer_count = peer_addrs.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
if peer_count == 0 {
return 0;
}
let encoded = request.encode();
for peer_addr in peer_addrs {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.discovery.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Dedup: any pending lookup means we are already trying.
if self.pending_lookups.contains_key(dest) {
self.metrics().discovery.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
return;
}
// Optional post-failure suppression. Defaults are 0/0 (inert);
// operators can opt in by setting `node.discovery.backoff_*_secs`.
if self.discovery_backoff.is_suppressed(dest) {
self.metrics().discovery.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = self.discovery_backoff.failure_count(dest),
"Discovery lookup suppressed by backoff"
);
return;
}
// Bloom filter pre-check: if no peer's filter contains the target,
// it's not in the mesh — skip the lookup and record as failure.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
if !reachable {
self.metrics().discovery.req_bloom_miss.inc();
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
return;
}
self.pending_lookups
.insert(*dest, PendingLookup::new(now_ms));
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
self.pending_lookups.remove(dest);
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.discovery.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let timeouts = self.config().node.discovery.attempt_timeouts_secs.clone();
let max_attempts = timeouts.len() as u8;
// Collect targets needing action
let mut to_retry: Vec<NodeAddr> = Vec::new();
let mut to_timeout: Vec<NodeAddr> = Vec::new();
for (&target, entry) in &self.pending_lookups {
let attempt_idx = (entry.attempt as usize).saturating_sub(1);
let attempt_timeout_ms = timeouts.get(attempt_idx).copied().unwrap_or(0) * 1000;
if now_ms.saturating_sub(entry.last_sent_ms) >= attempt_timeout_ms {
if entry.attempt >= max_attempts {
to_timeout.push(target);
} else {
to_retry.push(target);
}
}
}
// Process retries
for target in to_retry {
if let Some(entry) = self.pending_lookups.get_mut(&target) {
entry.attempt += 1;
entry.last_sent_ms = now_ms;
let attempt = entry.attempt;
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
}
// Process timeouts
for addr in to_timeout {
self.metrics().discovery.resp_timed_out.inc();
self.pending_lookups.remove(&addr);
// Record failure for optional backoff
self.discovery_backoff.record_failure(&addr);
let failures = self.discovery_backoff.failure_count(&addr);
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_discovery_backoff(&mut self) {
if !self.discovery_backoff.is_empty() {
debug!(
entries = self.discovery_backoff.entry_count(),
"Resetting discovery backoff on topology change"
);
self.discovery_backoff.reset_all();
}
}
/// Remove expired entries from the recent_requests cache.
fn purge_expired_requests(&mut self, current_time_ms: u64) {
let expiry_ms = self.config().node.discovery.recent_expiry_secs * 1000;
self.recent_requests
.retain(|_, entry| !entry.is_expired(current_time_ms, expiry_ms));
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}
/// Tracks a pending discovery lookup with retry state.
pub struct PendingLookup {
/// When the lookup was first initiated.
pub initiated_ms: u64,
/// When the last attempt was sent.
pub last_sent_ms: u64,
/// Current attempt number (1 = initial, 2 = first retry, ...).
pub attempt: u8,
}
impl PendingLookup {
pub fn new(now_ms: u64) -> Self {
Self {
initiated_ms: now_ms,
last_sent_ms: now_ms,
attempt: 1,
}
}
}

File diff suppressed because it is too large Load Diff

739
src/node/handlers/lookup.rs Normal file
View File

@@ -0,0 +1,739 @@
//! LookupRequest/LookupResponse mesh lookup protocol handlers.
//!
//! Handles coordinate lookup via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::Node;
use crate::node::reject::DiscoveryReject;
use crate::proto::lookup::{
LookupAction, LookupRequest, LookupResponse, MAX_RECENT_LOOKUP_REQUESTS,
};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
/// Shell adapter exposing the live routing tables to the sans-IO discovery
/// core's `RoutingView` read seam. Lives in `node` so it can read `Node`'s
/// private `peers` map and call the crate-private tree/bloom predicates.
///
/// Holding `&Node` whole is fine for the forward path because it does not
/// also need `&mut self.lookup` concurrently. A later commit whose core
/// step needs `&mut discovery` while reading routing state should narrow this
/// to borrow only `peers` + `tree_state` instead of the whole node.
struct NodeRoutingView<'a> {
node: &'a Node,
}
impl crate::proto::lookup::RoutingView for NodeRoutingView<'_> {
fn is_tree_peer(&self, addr: &NodeAddr) -> bool {
self.node.is_tree_peer(addr)
}
fn peers_reaching(&self, target: &NodeAddr) -> Vec<NodeAddr> {
self.node
.peers
.iter()
.filter(|(_, peer)| peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect()
}
}
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().lookup.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
let recent_expiry_ms = self.config().node.lookup.recent_expiry_secs * 1000;
let my_addr = *self.node_addr();
use crate::proto::lookup::RequestOutcome;
match crate::proto::lookup::classify_request(
&mut self.lookup,
&request,
from,
&my_addr,
now_ms,
recent_expiry_ms,
MAX_RECENT_LOOKUP_REQUESTS,
) {
RequestOutcome::Duplicate => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
}
RequestOutcome::DedupCacheFull { len } => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = len,
max_recent_requests = MAX_RECENT_LOOKUP_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
}
RequestOutcome::RespondAsTarget => {
self.metrics().lookup.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
}
RequestOutcome::Forward => {
self.metrics().lookup.req_forwarded.inc();
self.forward_lookup_request(request).await;
}
RequestOutcome::ForwardRateLimited => {
self.metrics().lookup.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
}
RequestOutcome::TtlExhausted => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().lookup.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
match crate::proto::lookup::classify_response(&mut self.lookup, response.request_id) {
crate::proto::lookup::ResponseRoute::AlreadyForwarded => {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
}
crate::proto::lookup::ResponseRoute::Transit { from_peer } => {
// Transit node: reverse-path forward
self.metrics().lookup.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
}
crate::proto::lookup::ResponseRoute::Originator => {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data = LookupResponse::proof_bytes(
response.request_id,
&target,
&response.target_coords,
);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().lookup.resp_accepted.inc();
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
// Apply the accept-side effects: the core clears the success
// state (backoff + pending lookup) and returns the
// cross-subsystem effects for us to drive.
let actions = crate::proto::lookup::on_response_accepted(
&mut self.lookup,
&target,
response.target_coords,
now_ms,
path_mtu,
);
self.drive_response_actions(actions).await;
}
}
}
/// Drive the cross-subsystem effects returned by the discovery core's
/// accept-side planning. Each arm reproduces the original inline effect
/// exactly (same metrics/logs/writes, same order).
async fn drive_response_actions(&mut self, actions: Vec<LookupAction>) {
for action in actions {
match action {
LookupAction::CacheCoords {
target,
coords,
now_ms,
path_mtu,
} => {
self.coord_cache
.insert_with_path_mtu(target, coords, now_ms, path_mtu);
}
LookupAction::WritePathMtu { target, path_mtu } => {
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen
// the clamp. A reactive MtuExceeded or
// PathMtuNotification tighten takes precedence
// over a looser discovery estimate
// (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
}
LookupAction::ResetWarmupIfEstablished { target } => {
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
}
LookupAction::RetryQueuedPackets { target } => {
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
LookupAction::SendLink { peer, bytes } => {
if let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await {
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send discovery link message"
);
}
}
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin. The reverse-path decision (the peer the request
// arrived from, recorded in recent_requests) is the sans-IO core's; the
// greedy tree-route fallback is a &mut coord-cache op kept in the shell.
use crate::proto::lookup::ResponseRouteDecision;
let next_hop_addr = match crate::proto::lookup::plan_response_route(
&self.lookup,
request.request_id,
) {
ResponseRouteDecision::ReversePath(peer) => peer,
ResponseRouteDecision::NeedsTreeRoute => match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
},
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
// Plan the forward with the sans-IO decision core. The core owns the
// TTL decrement, tree/fallback peer selection, and single-encode
// fan-out; the shell keeps all metrics/logging and drives the sends.
let outcome = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_forward(&mut request, &rv)
};
match outcome {
crate::proto::lookup::ForwardOutcome::TtlExhausted => {}
crate::proto::lookup::ForwardOutcome::NoPeers => {
self.metrics().lookup.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
}
crate::proto::lookup::ForwardOutcome::Forward {
actions,
used_fallback,
} => {
let peer_count = actions.len();
if used_fallback {
self.metrics().lookup.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest"
);
}
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().lookup.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request_id = {
use rand::RngExt;
rand::rng().random()
};
let request = LookupRequest::new(request_id, *target, origin, origin_coords, ttl, 0);
// Tree-peer bloom-match selection + single encode live in the sans-IO
// core. The core keeps the tree-only (no non-tree fallback) behavior;
// the shell drives the sends and keeps all metrics/logging.
let actions = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_initiate(&request, &rv)
};
let peer_count = actions.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.lookup.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Bloom filter pre-check (view read) BEFORE the core call: if no peer's
// filter contains the target, it's not in the mesh. Reading `self.peers`
// here keeps the `&mut self.lookup` borrow in `initiate_gate` from
// overlapping the immutable peer-table read.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
use crate::proto::lookup::InitiateDecision;
match crate::proto::lookup::initiate_gate(&mut self.lookup, dest, now_ms, reachable) {
InitiateDecision::Deduplicated => {
self.metrics().lookup.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
}
InitiateDecision::Suppressed { failures } => {
self.metrics().lookup.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = failures,
"Discovery lookup suppressed by backoff"
);
}
InitiateDecision::BloomMiss => {
self.metrics().lookup.req_bloom_miss.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
}
InitiateDecision::Proceed => {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
crate::proto::lookup::initiate_failed(&mut self.lookup, dest, now_ms);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.lookup.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let attempt_timeouts = self.config().node.lookup.attempt_timeouts_secs.clone();
let outcome =
crate::proto::lookup::poll_pending(&mut self.lookup, now_ms, &attempt_timeouts);
for (target, attempt) in outcome.retries {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
for (addr, failures) in outcome.timeouts {
self.metrics().lookup.resp_timed_out.inc();
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_lookup_backoff(&mut self) {
let cleared = self.lookup.reset_backoff();
if cleared > 0 {
debug!(
entries = cleared,
"Resetting discovery backoff on topology change"
);
}
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}

View File

@@ -5,20 +5,63 @@
//! and teardown metric logs.
use crate::NodeAddr;
use crate::mmp::MmpMode;
use crate::mmp::MmpSessionState;
use crate::mmp::report::{ReceiverReport, SenderReport};
use crate::node::Node;
use crate::node::dataplane::PeerActionCtx;
use crate::node::reject::{MmpReject, RejectReason, TreeReject};
use crate::protocol::{
LinkMessageType, PathMtuNotification, SessionMessageType, SessionReceiverReport,
SessionSenderReport,
use crate::node::tree::sign_declaration;
use crate::peer::machine::PeerEvent;
use crate::proto::link::LinkMessageType;
use crate::proto::mmp::{
LinkReportKind, LinkReportSnapshot, MmpAction, PeerLivenessSnapshot, ReceiverReport, RrLog,
SenderReport,
};
use crate::proto::stp::ParentEval;
use crate::transport::{TransportAddr, TransportId};
use std::time::{Duration, Instant};
use tracing::{debug, info, trace, warn};
/// Emit the operator `trace!` point for a processed ReceiverReport outcome.
///
/// These log points used to live inside `MmpMetrics::process_receiver_report`;
/// the sans-IO migration returns the outcome as an [`RrLog`] and re-emits it
/// here, shell-side, preserving the original field set, content, and (relative
/// to the surrounding handler logs) ordering. The original traces carried no
/// peer identifier, so none is added here.
pub(super) fn log_rr_outcome(rr: &ReceiverReport, our_timestamp_ms: u32, log: RrLog) {
match log {
RrLog::Stale {
prev_highest,
prev_packets,
prev_bytes,
} => trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = prev_highest,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = prev_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = prev_bytes,
"Ignoring stale MMP ReceiverReport"
),
RrLog::RttSample { rtt_ms, srtt_ms } => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
rtt_ms = rtt_ms,
srtt_ms = srtt_ms,
"RTT sample from timestamp echo"
),
RrLog::InvalidRtt => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
"Ignoring invalid MMP RTT sample"
),
RrLog::None => {}
}
}
/// Format bytes/sec as human-readable throughput.
fn format_throughput(bps: f64) -> String {
pub(in crate::node) fn format_throughput(bps: f64) -> String {
if bps == 0.0 {
"n/a".to_string()
} else if bps >= 1_000_000.0 {
@@ -113,10 +156,12 @@ impl Node {
// Process the report: computes RTT from timestamp echo, updates
// loss rate, goodput rate, jitter trend, and ETX.
let now = Instant::now();
let first_rtt = mmp
.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let now_ms = crate::time::mono_ms();
let (first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now_ms);
// Re-emit the operator trace the core used to log mid-decision.
log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -144,25 +189,43 @@ impl Node {
// Trigger re-evaluation so the node doesn't wait for the next
// periodic tick or TreeAnnounce.
if first_rtt {
let peer_costs: std::collections::HashMap<crate::NodeAddr, f64> = self
let peer_costs: std::collections::BTreeMap<crate::NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, p)| p.has_srtt())
.map(|(a, p)| (*a, p.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
// Wall-clock seconds for the escaping declaration timestamp;
// monotonic ms for the flap-dampening / hold-down timers.
let now_secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mono_now_ms = crate::time::mono_ms();
// Compute the flap-dampening / hold-down veto at the edge; a mandatory
// switch bypasses it, a discretionary one is taken only if not suppressed.
let switch_suppressed = self.tree_state.is_switch_suppressed(mono_now_ms);
let new_parent = match self
.tree_state
.evaluate_parent(&peer_costs, &std::collections::BTreeSet::new())
{
ParentEval::Mandatory(p) => Some(p),
ParentEval::Discretionary(p) if !switch_suppressed => Some(p),
ParentEval::Discretionary(_) | ParentEval::None => None,
};
if let Some(new_parent) = new_parent {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
let flap_dampened =
self.tree_state
.set_parent(new_parent, new_seq, now_secs, mono_now_ms);
self.tree_state.recompute_coords();
// Clone identity once: sign_declaration borrows &mut tree_state while
// the identity() accessor borrows all of &self, so an owned copy avoids
// the split-borrow conflict on this infrequent parent-switch path.
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after first-RTT parent eval");
self.metrics()
.tree
@@ -172,7 +235,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
@@ -190,10 +253,12 @@ impl Node {
let all_peers: Vec<crate::NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root() && self.tree_state.should_be_root() {
self.tree_state.become_root();
self.tree_state.become_root(now_secs);
// Clone identity once (see the parent-switch branch above for why).
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign self-root declaration after first-RTT");
self.metrics()
.tree
@@ -203,7 +268,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
@@ -221,65 +286,91 @@ impl Node {
///
/// Called from the tick handler. Also emits periodic operator logs.
pub(in crate::node) async fn check_mmp_reports(&mut self) {
let now = Instant::now();
let now_ms = crate::time::mono_ms();
// Collect peers that need reports (can't borrow self mutably while iterating)
let mut sender_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut receiver_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
// Build one report-gating snapshot per peer, resolving every timing read
// shell-side into a `bool`. `send_sr`/`send_rr` are `true` on the master
// (IK) line — there is no profile negotiation here; the forward-merge to
// `-next` wires them to `peer.send_sr()`/`peer.send_rr()` (plan spot c).
// The snapshots own only `NodeAddr`/`MmpMode`/`bool`, so the
// peer-iteration borrow is released before the pure decision runs and the
// driving loop mutates the reporting state.
let snapshots: Vec<LinkReportSnapshot> = self
.peers
.iter()
.filter_map(|(node_addr, peer)| {
let mmp = peer.mmp()?;
Some(LinkReportSnapshot {
peer: *node_addr,
mode: mmp.mode(),
send_sr: true,
send_rr: true,
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
for (node_addr, peer) in self.peers.iter_mut() {
// Compute display name before taking mutable MMP borrow
let peer_name = self
.peer_aliases
.get(node_addr)
.cloned()
.unwrap_or_else(|| peer.identity().short_npub());
let actions = self.mmp.plan_link_reports(&snapshots);
let Some(mmp) = peer.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
sender_reports.push((*node_addr, sr.encode()));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
receiver_reports.push((*node_addr, rr.encode()));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports
for (node_addr, encoded) in sender_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send SenderReport");
}
}
for (node_addr, encoded) in receiver_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send ReceiverReport");
// Drive the planned actions in their phase-grouped order (all logs, then
// all SenderReports, then all ReceiverReports). Logs run first because the
// operator log reads cumulative_packets_sent, which each report send
// advances (send_encrypted_link_message -> sender.record_sent); the
// pre-refactor handler logged during its collect pass, before any send.
// `build_report` (which advances the interval state) is called only on a
// SendLinkReport action, exactly as the pre-refactor gate did.
for action in actions {
match action {
MmpAction::SendLinkReport { peer, kind } => {
let encoded = self
.peers
.get_mut(&peer)
.and_then(|p| p.mmp_mut())
.and_then(|mmp| match kind {
LinkReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| sr.encode())
}
LinkReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| rr.encode())
}
});
if let Some(encoded) = encoded
&& let Err(e) = self.send_encrypted_link_message(&peer, &encoded).await
{
let label = match kind {
LinkReportKind::Sender => "Failed to send SenderReport",
LinkReportKind::Receiver => "Failed to send ReceiverReport",
};
debug!(peer = %self.peer_display_name(&peer), error = %e, "{}", label);
}
}
MmpAction::LogLink { peer } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub) — not `peer_display_name`,
// which also consults the host map.
let peer_name = self.peer_aliases.get(&peer).cloned().unwrap_or_else(|| {
self.peers
.get(&peer)
.map(|p| p.identity().short_npub())
.unwrap_or_default()
});
if let Some(mmp) = self.peers.get_mut(&peer).and_then(|p| p.mmp_mut()) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Emit periodic MMP metrics for a peer.
fn log_mmp_metrics(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
fn log_mmp_metrics(peer_name: &str, mmp: &crate::proto::mmp::MmpPeerState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
@@ -307,7 +398,10 @@ impl Node {
}
/// Emit a teardown log summarizing lifetime MMP metrics for a removed peer.
pub(in crate::node) fn log_mmp_teardown(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
pub(in crate::node) fn log_mmp_teardown(
peer_name: &str,
mmp: &crate::proto::mmp::MmpPeerState,
) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
@@ -332,205 +426,6 @@ impl Node {
);
}
// === Session-layer MMP ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now = Instant::now();
// Collect reports to send: (dest_addr, msg_type, encoded_body)
let mut reports: Vec<(NodeAddr, u8, Vec<u8>)> = Vec::new();
for (dest_addr, entry) in self.sessions.iter_mut() {
// Compute display name before taking mutable MMP borrow
let session_name = self
.peer_aliases
.get(dest_addr)
.cloned()
.unwrap_or_else(|| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
});
let Some(mmp) = entry.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
let session_sr: SessionSenderReport = SessionSenderReport::from(&sr);
reports.push((
*dest_addr,
SessionMessageType::SenderReport.to_byte(),
session_sr.encode(),
));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
let session_rr: SessionReceiverReport = SessionReceiverReport::from(&rr);
reports.push((
*dest_addr,
SessionMessageType::ReceiverReport.to_byte(),
session_rr.encode(),
));
}
// PathMtu notifications (all modes)
if mmp.path_mtu.should_send_notification(now)
&& let Some(mtu_value) = mmp.path_mtu.build_notification(now)
{
let notif = PathMtuNotification::new(mtu_value);
reports.push((
*dest_addr,
SessionMessageType::PathMtuNotification.to_byte(),
notif.encode(),
));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports via session-layer encryption.
// Track per-destination success/failure for backoff and log suppression.
let mut send_results: Vec<(NodeAddr, bool)> = Vec::new();
for (dest_addr, msg_type, body) in reports {
match self.send_session_msg(&dest_addr, msg_type, &body).await {
Ok(()) => {
send_results.push((dest_addr, true));
}
Err(e) => {
// Peek at current failure count for log suppression
let failures = self
.sessions
.get(&dest_addr)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push((dest_addr, false));
}
}
}
// Update backoff state from send results.
// Deduplicate: a destination counts as success if ANY report succeeded,
// failure only if ALL reports for that destination failed.
let mut dest_success: std::collections::HashMap<NodeAddr, bool> =
std::collections::HashMap::new();
for (dest, ok) in &send_results {
let entry = dest_success.entry(*dest).or_insert(false);
if *ok {
*entry = true;
}
}
for (dest_addr, success) in dest_success {
if let Some(entry) = self.sessions.get_mut(&dest_addr)
&& let Some(mmp) = entry.mmp_mut()
{
if success {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
} else {
mmp.sender.record_send_failure();
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Send heartbeats and remove dead peers.
///
/// Called from the tick handler. Sends a 1-byte heartbeat to each peer
@@ -538,83 +433,159 @@ impl Node {
/// hasn't sent us a frame within the link dead timeout.
pub(in crate::node) async fn check_link_heartbeats(&mut self) {
let now = Instant::now();
// Monotonic ms for the MMP receiver's injected-`u64` liveness clock; the
// Instant `now` is still used for the shell-owned heartbeat timing and
// the session-start fallback (both `ActivePeer` Instants).
let now_ms = crate::time::mono_ms();
let heartbeat_interval = Duration::from_secs(self.config().node.heartbeat_interval_secs);
let dead_timeout = Duration::from_secs(self.config().node.link_dead_timeout_secs);
let dead_timeout_ms = dead_timeout.as_millis() as u64;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
let heartbeat_msg = [LinkMessageType::Heartbeat.to_byte()];
// Collect heartbeats to send and dead peers to remove
let mut heartbeats: Vec<NodeAddr> = Vec::new();
let mut dead_peers: Vec<NodeAddr> = Vec::new();
// Build one liveness snapshot per peer, resolving every clock read and
// the rekey-suppression predicate shell-side. The snapshots own only
// `NodeAddr`/`bool`, so the peer-iteration borrow is released before the
// pure decision runs and the driving loop mutates the registry.
let snapshots: Vec<PeerLivenessSnapshot> = self
.peers
.iter()
.map(|(node_addr, peer)| {
// Check liveness via the MMP receiver's last-received monotonic
// ms. Fall back to session_start (an `ActivePeer` Instant) for
// peers that never sent data, keeping that branch in Instant
// space so no monotonic-ms epoch conversion is needed.
let time_dead = if let Some(mmp) = peer.mmp() {
match mmp.receiver.last_recv_ms() {
Some(last_ms) => now_ms.saturating_sub(last_ms) >= dead_timeout_ms,
None => now.duration_since(peer.session_start()) >= dead_timeout,
}
} else {
false
};
for (node_addr, peer) in self.peers.iter() {
// Check liveness via MMP receiver last_recv_time.
// Fall back to session_start for peers that never sent data.
let time_dead = if let Some(mmp) = peer.mmp() {
let reference_time = mmp
.receiver
.last_recv_time()
.unwrap_or(peer.session_start());
now.duration_since(reference_time) >= dead_timeout
} else {
false
};
// Suppress teardown while an FMP rekey is genuinely in flight
// with budget left: a rekey-handshake link is not silent. The
// msg1 resend cap guarantees this terminates (abandon on
// exhaustion or cutover on completion clears
// `rekey_in_progress`), so a truly dead link is reaped on the
// next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Suppress teardown while an FMP rekey is genuinely in flight with
// budget left: a rekey-handshake link is not silent. The msg1
// resend cap guarantees this terminates (abandon on exhaustion or
// cutover on completion clears `rekey_in_progress`), so a truly
// dead link is reaped on the next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Check if heartbeat is due.
let heartbeat_due = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
let is_dead = time_dead && !rekey_active;
if is_dead {
dead_peers.push(*node_addr);
continue;
}
PeerLivenessSnapshot {
peer: *node_addr,
time_dead,
rekey_active,
heartbeat_due,
}
})
.collect();
// Check if heartbeat is due
let needs_heartbeat = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
if needs_heartbeat {
heartbeats.push(*node_addr);
}
}
let actions = self.mmp.plan_heartbeats(&snapshots);
// Remove dead peers and schedule auto-reconnect
// Wall-clock basis for reconnect scheduling, sourced once (as before).
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
for addr in &dead_peers {
debug!(
peer = %self.peer_display_name(addr),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.remove_active_peer(addr);
self.schedule_reconnect(*addr, now_ms);
}
// Send heartbeats (skip peers we just removed)
for addr in heartbeats {
if dead_peers.contains(&addr) {
continue;
}
if let Some(peer) = self.peers.get_mut(&addr) {
peer.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&addr, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&addr), error = %e, "Failed to send heartbeat");
// Drive the planned actions: all reaps first (each removed +
// reconnect-scheduled), then all heartbeats (a just-reaped peer is never
// heartbeated — the core never emits both for the same peer).
for action in actions {
match action {
MmpAction::ReapPeer { peer } => {
// Log SHELL-SIDE before routing so the reap keeps the
// `fips::node::handlers::mmp` tracing target (no relocation into
// the executor, no target pin needed).
debug!(
peer = %self.peer_display_name(&peer),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.route_link_dead(peer, now_ms).await;
}
MmpAction::Heartbeat { peer } => {
if let Some(p) = self.peers.get_mut(&peer) {
p.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&peer, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&peer), error = %e, "Failed to send heartbeat");
}
}
MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Route a link-dead liveness reap through the peer machine + executor.
/// Mirrors [`route_rekey_cadence`](Node::route_rekey_cadence): the
/// shell already decided (the tick sweep's `plan_heartbeats` batch emitted
/// this `ReapPeer` in phase order), so the machine only CONSUMES the decision
/// via [`PeerEvent::LinkDeadSuspected`]. The resulting executor arms
/// (`InvalidateSendState` → `remove_active_peer`, `ReportLost` →
/// `note_link_dead`) reproduce the pre-refactor inline reap body exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was already a
/// no-op, so we return; if the machine is absent (which should be impossible)
/// we fall back to the byte-identical inline body under a `debug_assert`.
///
/// `now_ms` is the sweep's hoisted wall-clock ms (the same value the old reap
/// fed `note_link_dead`); it flows to the executor `ReportLost` arm via
/// `ambient.now_ms`.
async fn route_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(false, "peer machine present for every established peer");
self.remove_active_peer(&node_addr);
self.note_link_dead(node_addr, now_ms);
return;
}
let ambient = self.link_dead_ctx(&node_addr, now_ms);
self.advance_peer_machine(link, PeerEvent::LinkDeadSuspected, Self::now_ms(), &ambient)
.await;
}
/// Ambient shell facts for the routed liveness reap. Mirrors
/// [`rekey_cadence_ctx`](Node::rekey_cadence_ctx). The executor reads only
/// `verified_identity` (`InvalidateSendState` → `remove_active_peer` resolves
/// its `NodeAddr` from it, so it must equal `node_addr`) and `now_ms`
/// (`ReportLost` → `note_link_dead`, the wall-clock reconnect basis). The
/// transport/index/direction fields are unused by these two arms and are
/// populated best-effort for coherence. `now_ms` is threaded in (rather than
/// re-read) so the value fed to `note_link_dead` is byte-identical to the old
/// reap's hoisted wall-clock for every peer in the sweep.
fn link_dead_ctx(&self, node_addr: &NodeAddr, now_ms: u64) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms,
is_outbound: false,
}
}
}

View File

@@ -1,14 +1,8 @@
//! RX event loop and message handlers.
//! Message handlers: per-message-type behavior on `impl Node`.
#[cfg(unix)]
pub(crate) mod connected_udp;
pub(crate) mod discovery;
mod dispatch;
mod encrypted;
mod forwarding;
mod handshake;
pub(crate) mod lookup;
mod mmp;
mod rekey;
mod rx_loop;
pub(in crate::node) mod session;
mod timeout;

View File

@@ -7,28 +7,28 @@
use crate::NodeAddr;
use crate::node::Node;
use crate::node::wire::build_msg1;
use crate::node::dataplane::PeerActionCtx;
use crate::noise::HandshakeState;
use crate::protocol::{SessionDatagram, SessionSetup};
use crate::peer::machine::PeerEvent;
use crate::proto::fmp::wire::build_msg1;
use crate::proto::fmp::{ConnAction, LifecycleView, PeerSnapshot, RekeyCfg, RekeyResendSnapshot};
use crate::proto::fsp::{
FspAction, RekeyMsg3ResendSnapshot, SessionSetup, SessionSnapshot, cutover_timer_elapsed,
};
use crate::proto::link::SessionDatagram;
use crate::transport::{TransportAddr, TransportId};
use tracing::{debug, trace, warn};
/// Keep previous session alive for this long after cutover.
///
/// FMP-scoped copy for `check_rekey`; the FSP session-rekey timing bounds live
/// in `crate::proto::fsp::limits`.
const DRAIN_WINDOW_SECS: u64 = 10;
/// Suppress local rekey initiation for this long after receiving
/// a peer's rekey msg1.
/// a peer's rekey msg1. FMP-scoped copy for `check_rekey`.
const REKEY_DAMPENING_SECS: u64 = 30;
/// Liveness bound on how long the FSP rekey initiator holds the
/// `current` + `pending` state before cutting over to the new epoch.
///
/// This is NOT safety-critical: overlapping-epoch trial-decrypt covers
/// any skew between the two endpoints' cutovers. The timer only bounds
/// how long the initiator advertises the old K-bit. An opportunistic
/// early cutover also fires if the initiator authenticates a peer frame
/// against its own `pending` session (the responder cut over first).
const FSP_CUTOVER_DELAY_MS: u64 = 2000;
impl Node {
/// Periodic rekey check. Called from the tick loop.
///
@@ -41,121 +41,225 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
// Collect peers that need action (to avoid borrow conflicts)
let mut peers_to_cutover: Vec<NodeAddr> = Vec::new();
let mut peers_to_drain: Vec<NodeAddr> = Vec::new();
let mut peers_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, peer) in &self.peers {
if !peer.has_session() || !peer.is_healthy() {
continue;
}
// 1. Initiator-side cutover: we completed a rekey and have
// a pending session ready. Cut over on the next tick.
if peer.pending_new_session().is_some() && !peer.rekey_in_progress() {
peers_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if peer.is_draining() && peer.drain_expired(DRAIN_WINDOW_SECS) {
peers_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if peer.rekey_in_progress() {
continue;
}
if peer.is_rekey_dampened(REKEY_DAMPENING_SECS) {
continue;
}
let elapsed = peer.session_established_at().elapsed().as_secs();
let counter = peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0);
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(peer.rekey_jitter_secs());
if elapsed >= effective_after_secs || counter >= rekey_after_messages {
peers_to_rekey.push(*node_addr);
// The shell snapshots each healthy peer's rekey ages/flags (every clock
// read resolved here); the core decides cutover/drain/trigger with no
// clock, phase-grouped to preserve the pre-refactor execution order.
// The batch `poll_rekey` + snapshots STAY SHELL-SIDE and BYTE-UNCHANGED:
// the cross-peer phase-grouping (all Cutover → all Drain →
// all InitiateRekey) governs the shared `index_allocator` free-then-alloc
// SEQUENCE that appears on the wire. The machine must NOT re-poll; it
// CONSUMES each decided `ConnAction` in the same order the batch returned.
let snapshots = self.rekey_peers();
for action in self.fmp.poll_rekey(snapshots, &cfg) {
match action {
// Initiator cutover: route the decided action through the peer
// machine + executor. The executor's `SwapSendState` arm
// reproduces the pre-refactor cutover body EXACTLY.
ConnAction::Cutover { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Cutover { peer: node_addr })
.await;
}
// Drain completion: route through the machine + executor. The
// executor's `CompleteDrain` arm reads the REAL previous index
// from `complete_drain()` and frees it at the same point the old
// inline body did (index-order preserving).
ConnAction::Drain { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Drain { peer: node_addr })
.await;
}
// Initiate a new rekey: STAYS INLINE (the Noise msg1 build +
// index allocation are a shell-side leaf, byte-unchanged). Feed
// the machine a `RekeyInitiated` observation afterward so its
// control state stays coherent for the next tick's Cutover/Drain.
ConnAction::InitiateRekey { peer: node_addr } => {
self.initiate_rekey(&node_addr).await;
self.observe_rekey_initiated(&node_addr);
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in peers_to_cutover {
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
/// Route a cadence-decided `Cutover`/`Drain` `ConnAction` through the peer
/// machine + executor. The shell already decided (batch `poll_rekey`);
/// the machine consumes via [`PeerEvent::RekeyConsume`] WITHOUT re-polling,
/// preserving the phase order. The `SwapSendState`/`CompleteDrain` executor
/// arms reproduce the pre-refactor inline effect bodies exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was a no-op, so we do
/// nothing; if the machine is absent (which should be impossible) we fall
/// back to the byte-identical inline body under a `debug_assert`.
async fn route_rekey_cadence(&mut self, node_addr: NodeAddr, action: ConnAction) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
match action {
ConnAction::Cutover { peer } => self.cutover_peer_inline(&peer),
ConnAction::Drain { peer } => self.drain_peer_inline(&peer),
_ => {}
}
return;
}
let ambient = self.rekey_cadence_ctx(&node_addr);
self.advance_peer_machine(
link,
PeerEvent::RekeyConsume { action },
Self::now_ms(),
&ambient,
)
.await;
}
/// Feed the machine the `RekeyInitiated` observation after the inline
/// `initiate_rekey`. The obs emits no action, so there is no executor
/// pass — a bare `step` keeps the machine's control state coherent.
fn observe_rekey_initiated(&mut self, node_addr: &NodeAddr) {
let link = match self.peers.get(node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if let Some(machine) = self.peer_machines.get_mut(&link) {
let acts = machine.step(
PeerEvent::RekeyInitiated,
Self::now_ms(),
&mut self.index_allocator,
);
debug_assert!(acts.is_empty(), "RekeyInitiated is a pure observation");
} else {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
}
}
/// Ambient shell facts for the routed cadence Cutover/Drain step. Only
/// `verified_identity` is read by the `SwapSendState`/`CompleteDrain`
/// executor arms — `SwapSendState` resolves its `NodeAddr` from it (so it must
/// equal `node_addr`), and `CompleteDrain` carries its peer in the action
/// payload. The transport/index/direction fields are unused by these two arms
/// (they matter only to `PromoteToActive`, never emitted on this path) and are
/// populated best-effort for coherence.
fn rekey_cadence_ctx(&self, node_addr: &NodeAddr) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms: Self::now_ms(),
is_outbound: false,
}
}
/// Pre-refactor initiator cutover body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Cutover` arm and to the executor's `SwapSendState` arm.
fn cutover_peer_inline(&mut self, node_addr: &NodeAddr) {
let did_cutover = if let Some(peer) = self.peers.get_mut(node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index during msg2
// handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
} else {
false
};
// Re-register the new session with the decrypt worker — the cache_key
// (transport_id, our_index) just changed, so the old worker entry is
// stale and every packet on the new session would miss the lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
// Execute drain completion
for node_addr in peers_to_drain {
// Extract the old index and transport_id under the peer
// borrow, then drop the borrow so the cache_key cleanup
// below can take &mut self for unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(&node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
/// Pre-refactor drain-completion body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Drain` arm and to the executor's `CompleteDrain` arm.
fn drain_peer_inline(&mut self, node_addr: &NodeAddr) {
// Extract the old index and transport_id under the peer borrow, then drop
// the borrow so the cache_key cleanup below can take &mut self for
// unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in peers_to_rekey {
self.initiate_rekey(&node_addr).await;
}
/// Snapshot every healthy peer with a session for the rekey decision,
/// pre-computing its monotonic ages and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`PeerSnapshot`]).
///
/// Lives here, beside the drain/dampening constants and the FSP analog, so
/// the forward-merge onto `next` reconciles rekey timing in one place.
pub(in crate::node) fn rekey_peer_snapshots(&self) -> Vec<PeerSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.has_session() && peer.is_healthy())
.map(|(node_addr, peer)| PeerSnapshot {
addr: *node_addr,
has_pending: peer.pending_new_session().is_some(),
rekey_in_progress: peer.rekey_in_progress(),
is_draining: peer.is_draining(),
drain_expired: peer.drain_expired(DRAIN_WINDOW_SECS),
is_dampened: peer.is_rekey_dampened(REKEY_DAMPENING_SECS),
elapsed_secs: peer.session_established_at().elapsed().as_secs(),
counter: peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0),
jitter_secs: peer.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an outbound rekey to a peer.
@@ -260,60 +364,74 @@ impl Node {
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
// Collect peers needing action
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
// The shell snapshots each in-flight rekey (resend-due predicate
// resolved here); the core classifies abandon-vs-resend and computes
// the backoff, abandons first.
let candidates = self.rekey_resend_candidates(now_ms);
for action in
self.fmp
.poll_rekey_resends(candidates, now_ms, interval_ms, backoff, max_resends)
{
match action {
// Abandon rekey cycles that exhausted their retransmission budget.
ConnAction::AbandonRekey { peer: node_addr } => {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
ConnAction::ResendRekeyMsg1 {
peer: node_addr,
bytes,
next_resend_at_ms,
} => {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
for (node_addr, peer) in &self.peers {
if !peer.rekey_in_progress() || peer.rekey_msg1().is_none() {
continue;
}
if peer.rekey_msg1_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
if peer.needs_msg1_resend(now_ms) {
to_resend.push((*node_addr, peer.rekey_msg1().unwrap().to_vec()));
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
peer.record_rekey_msg1_resend(next_resend_at_ms);
let count = peer.rekey_msg1_resend_count();
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
for (node_addr, msg1_bytes) in to_resend {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &msg1_bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
let count = peer.rekey_msg1_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
peer.record_rekey_msg1_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
/// Snapshot every peer with a rekey handshake in flight (and a stored
/// msg1) for the retransmission decision, pre-evaluating the resend-due
/// predicate against `now_ms` so the core reads no clock.
pub(in crate::node) fn rekey_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.rekey_in_progress() && peer.rekey_msg1().is_some())
.map(|(node_addr, peer)| RekeyResendSnapshot {
peer: *node_addr,
resend_count: peer.rekey_msg1_resend_count(),
needs_resend: peer.needs_msg1_resend(now_ms),
msg1: peer.rekey_msg1().unwrap().to_vec(),
})
.collect()
}
/// Retransmit FSP rekey msg3 until the responder is confirmed on the
@@ -352,66 +470,77 @@ impl Node {
let ttl = self.config().node.session.default_ttl;
let my_addr = *self.node_addr();
// Collect rekey initiators whose msg3 retransmission is due.
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
// Only the rekey initiator retains a msg3 payload.
let payload = match entry.rekey_msg3_payload() {
Some(p) => p,
None => continue,
};
if entry.rekey_msg3_next_resend_ms() == 0 || now_ms < entry.rekey_msg3_next_resend_ms()
{
continue;
}
if entry.rekey_msg3_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
to_resend.push((*node_addr, payload.to_vec()));
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
}
// Retransmit msg3 for cycles still within budget.
for (node_addr, payload) in to_resend {
let mut datagram = SessionDatagram::new(my_addr, node_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
// The shell snapshots each session retaining a msg3 payload (resend-due
// predicate resolved here); the core classifies abandon-vs-resend,
// abandons first.
let candidates = self.rekey_msg3_resend_snapshots(now_ms);
for action in self.fsp.poll_rekey_msg3_resends(candidates, max_resends) {
match action {
FspAction::AbandonRekey { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"FSP rekey msg3 retransmission failed"
peer = %self.peer_display_name(&addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
false
}
};
FspAction::ResendSessionMsg3 { addr } => {
let payload = match self
.sessions
.get(&addr)
.and_then(|e| e.rekey_msg3_payload())
{
Some(p) => p.to_vec(),
None => continue,
};
let mut datagram = SessionDatagram::new(my_addr, addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
debug!(
peer = %self.peer_display_name(&addr),
error = %e,
"FSP rekey msg3 retransmission failed"
);
false
}
};
if sent && let Some(entry) = self.sessions.get_mut(&node_addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent FSP rekey msg3"
);
if sent && let Some(entry) = self.sessions.get_mut(&addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next =
now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&addr),
resend = count,
"Resent FSP rekey msg3"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
/// Snapshot every session retaining a rekey-msg3 payload for the
/// retransmission decision, pre-evaluating the resend-due predicate against
/// `now_ms` so the core reads no clock.
fn rekey_msg3_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyMsg3ResendSnapshot> {
self.sessions
.iter()
.filter(|(_, entry)| entry.rekey_msg3_payload().is_some())
.map(|(node_addr, entry)| RekeyMsg3ResendSnapshot {
addr: *node_addr,
resend_count: entry.rekey_msg3_resend_count(),
resend_due: entry.rekey_msg3_next_resend_ms() != 0
&& now_ms >= entry.rekey_msg3_next_resend_ms(),
})
.collect()
}
/// Periodic session (FSP) rekey check. Called from the tick loop.
///
/// For each established session:
@@ -429,100 +558,71 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = crate::proto::fsp::RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
let now_ms = Self::now_ms();
let drain_ms = DRAIN_WINDOW_SECS * 1000;
let dampening_ms = REKEY_DAMPENING_SECS * 1000;
let mut sessions_to_cutover: Vec<NodeAddr> = Vec::new();
let mut sessions_to_drain: Vec<NodeAddr> = Vec::new();
let mut sessions_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
if !entry.is_established() {
continue;
}
// 1. Initiator-side cutover (option A): completed rekey,
// pending session ready, liveness timer elapsed. This is
// an unconditional timer, NOT gated on responder progress —
// overlapping-epoch trial-decrypt covers the cutover skew,
// so flipping the K-bit here is always safe. An
// opportunistic early cutover also happens in
// `handle_encrypted_session_msg` if the initiator
// authenticates a peer frame against its own `pending`.
if entry.pending_new_session().is_some()
&& !entry.has_rekey_in_progress()
&& entry.is_rekey_initiator()
&& now_ms.saturating_sub(entry.rekey_completed_ms()) >= FSP_CUTOVER_DELAY_MS
{
sessions_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if entry.is_draining() && entry.drain_expired(now_ms, drain_ms) {
sessions_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if entry.has_rekey_in_progress() {
continue;
}
if entry.pending_new_session().is_some() {
continue; // Pending session present, awaiting cutover
}
if entry.rekey_msg3_payload().is_some() {
// Initiator already cut over on its liveness timer but is
// still retransmitting msg3 to a responder not yet
// confirmed on the new epoch. Don't start another rekey
// until the current cycle's msg3 is delivered or abandoned.
continue;
}
if entry.is_rekey_dampened(now_ms, dampening_ms) {
continue;
}
let elapsed_secs = now_ms.saturating_sub(entry.session_start_ms()) / 1000;
let counter = entry.send_counter();
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(entry.rekey_jitter_secs());
if elapsed_secs >= effective_after_secs || counter >= rekey_after_messages {
sessions_to_rekey.push(*node_addr);
// The shell snapshots each established session's rekey ages/flags
// (every clock read resolved here); the core decides
// cutover/drain/trigger with no clock, phase-grouped to preserve the
// pre-refactor execution order.
let snapshots = self.session_rekey_snapshots(now_ms);
for action in self.fsp.poll_rekey(snapshots, &cfg) {
match action {
FspAction::CutOver { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
FspAction::CompleteDrain { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&addr),
"FSP drain complete, previous session erased"
);
}
}
FspAction::InitiateRekey { addr } => {
self.initiate_session_rekey(&addr).await;
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in sessions_to_cutover {
if let Some(entry) = self.sessions.get_mut(&node_addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
// Execute drain completion
for node_addr in sessions_to_drain {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&node_addr),
"FSP drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in sessions_to_rekey {
self.initiate_session_rekey(&node_addr).await;
}
/// Snapshot every established session for the FSP rekey decision,
/// pre-computing its monotonic age and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`SessionSnapshot`]).
fn session_rekey_snapshots(&self, now_ms: u64) -> Vec<SessionSnapshot> {
let drain_ms = crate::proto::fsp::limits::DRAIN_WINDOW_SECS * 1000;
let dampening_ms = crate::proto::fsp::limits::REKEY_DAMPENING_SECS * 1000;
self.sessions
.iter()
.filter(|(_, entry)| entry.is_established())
.map(|(node_addr, entry)| SessionSnapshot {
addr: *node_addr,
has_pending: entry.pending_new_session().is_some(),
rekey_in_progress: entry.has_rekey_in_progress(),
is_rekey_initiator: entry.is_rekey_initiator(),
cutover_timer_elapsed: cutover_timer_elapsed(now_ms, entry.rekey_completed_ms()),
is_draining: entry.is_draining(),
drain_expired: entry.drain_expired(now_ms, drain_ms),
has_rekey_msg3_payload: entry.rekey_msg3_payload().is_some(),
is_dampened: entry.is_rekey_dampened(now_ms, dampening_ms),
elapsed_secs: now_ms.saturating_sub(entry.session_start_ms()) / 1000,
counter: entry.send_counter(),
jitter_secs: entry.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an FSP session rekey.

View File

@@ -6,34 +6,39 @@
//! encrypted data, and error signals (CoordsRequired, PathBroken).
use crate::NodeAddr;
use crate::mmp::report::ReceiverReport;
use crate::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::node::handlers::mmp::format_throughput;
use crate::node::reject::{RejectReason, SessionReject};
use crate::node::session::{EndToEndState, EpochSlot, SessionEntry};
use crate::node::session_wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
#[cfg(unix)]
use crate::node::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::node::{Node, NodeError};
use crate::noise::{
HandshakeState, XK_HANDSHAKE_MSG1_SIZE, XK_HANDSHAKE_MSG2_SIZE, XK_HANDSHAKE_MSG3_SIZE,
};
#[cfg(unix)]
use crate::protocol::LinkMessageType;
#[cfg(unix)]
use crate::protocol::SESSION_DATAGRAM_HEADER_SIZE;
use crate::protocol::{
CoordsRequired, FspInnerFlags, MtuExceeded, PathBroken, PathMtuNotification, SessionAck,
SessionDatagram, SessionMessageType, SessionMsg3, SessionReceiverReport, SessionSenderReport,
SessionSetup,
use crate::proto::fmp::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::protocol::{coords_wire_size, encode_coords};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
use crate::proto::fsp::{
DecryptSlot, EpochReaction, FspAction, FspInnerFlags, SessionAck, SessionMessageType,
SessionMsg3, SessionSetup, mark_ipv6_ecn_ce,
};
#[cfg(unix)]
use crate::proto::link::LinkMessageType;
#[cfg(unix)]
use crate::proto::link::SESSION_DATAGRAM_HEADER_SIZE;
use crate::proto::link::SessionDatagram;
use crate::proto::mmp::{
BackoffUpdate, MmpAction, MmpSessionState, PathMtuNotification, ReceiverReport, SendResult,
SessionReceiverReport, SessionReportKind, SessionReportSnapshot, SessionSenderReport,
};
use crate::proto::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::proto::routing::{CoordsRequired, MtuExceeded, PathBroken, RoutingSignalType};
use crate::proto::stp::{coords_wire_size, encode_coords};
#[cfg(unix)]
use crate::transport::TransportHandle;
use crate::upper::icmp::FIPS_OVERHEAD;
@@ -51,8 +56,8 @@ struct PipelinedSend<'a> {
timestamp: u32,
fsp_flags: u8,
inner_plaintext: &'a [u8],
my_coords: Option<&'a crate::tree::TreeCoordinate>,
dest_coords: Option<&'a crate::tree::TreeCoordinate>,
my_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
dest_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
}
impl Node {
@@ -104,14 +109,14 @@ impl Node {
}
let error_type = inner[0];
let error_body = &inner[1..];
match SessionMessageType::from_byte(error_type) {
Some(SessionMessageType::CoordsRequired) => {
match RoutingSignalType::from_byte(error_type) {
Some(RoutingSignalType::CoordsRequired) => {
self.handle_coords_required(error_body).await;
}
Some(SessionMessageType::PathBroken) => {
Some(RoutingSignalType::PathBroken) => {
self.handle_path_broken(error_body).await;
}
Some(SessionMessageType::MtuExceeded) => {
Some(RoutingSignalType::MtuExceeded) => {
self.handle_mtu_exceeded(error_body).await;
}
_ => {
@@ -166,11 +171,14 @@ impl Node {
match parse_encrypted_coords(coord_data) {
Ok((src_coords, dest_coords, bytes_consumed)) => {
let now_ms = Self::now_ms();
if let Some(coords) = src_coords {
self.coord_cache.insert(*src_addr, coords, now_ms);
}
if let Some(coords) = dest_coords {
self.coord_cache.insert(*self.node_addr(), coords, now_ms);
let my_addr = *self.node_addr();
for action in
self.fsp
.plan_cache_coords(*src_addr, my_addr, src_coords, dest_coords)
{
if let FspAction::CacheCoords { addr, coords } = action {
self.coord_cache.insert(addr, coords, now_ms);
}
}
ciphertext_offset += bytes_consumed;
}
@@ -248,44 +256,55 @@ impl Node {
}
};
// React to the epoch the frame decrypted against.
match slot {
EpochSlot::Pending => {
// A frame that authenticates against `pending` is itself
// the cutover signal — proof the peer derived the new
// session and moved to it. Promote now: current
// previous, pending → current, flip the K-bit. The
// header K-bit is no longer the gating event; the
// authenticated decrypt is.
// React to the epoch the frame decrypted against. The shell opened
// the frame; the core classifies the post-decrypt reaction over the
// plain-data slot + session flags, and the shell applies the
// `SessionEntry` mutation.
let decrypt_slot = match slot {
EpochSlot::Current => DecryptSlot::Current,
EpochSlot::Pending => DecryptSlot::Pending,
EpochSlot::Previous => DecryptSlot::Previous,
};
match self.fsp.classify_epoch(
decrypt_slot,
entry.rekey_msg3_payload().is_some(),
entry.pending_new_session().is_some(),
) {
EpochReaction::PromoteConfirming => {
// A frame that authenticates against `pending` is itself the
// cutover signal — proof the peer derived the new session and
// moved to it. The peer received msg3, so confirm it on the new
// epoch (stop retransmitting) before `handle_peer_kbit_flip`
// consumes the pending session, then promote.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
// The peer derived the new session, so it received msg3:
// confirm it on the new epoch and stop retransmitting.
// `handle_peer_kbit_flip` consumes the pending session,
// so confirm first.
if entry.rekey_msg3_payload().is_some() {
entry.confirm_peer_new_epoch();
}
entry.confirm_peer_new_epoch();
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Current => {
// If we still retain a msg3 retransmission payload but no
// longer hold a `pending` session, we are the rekey
// initiator that already cut over on its own timer:
// `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached the new
// epoch. Stop retransmitting msg3.
if entry.rekey_msg3_payload().is_some() && entry.pending_new_session().is_none() {
entry.confirm_peer_new_epoch();
}
EpochReaction::Promote => {
// Promote now: current → previous, pending → current, flip the
// K-bit. The header K-bit is only a hint; the authenticated
// decrypt is the gating event.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Previous => {
// The peer is still on the old epoch. `fsp_trial_decrypt`
// already refreshed the drain deadline so the `previous`
// slot is not retired while the peer keeps using it —
// no further state change here, just deliver.
EpochReaction::ConfirmResponder => {
// We are the rekey initiator that already cut over on its own
// timer: `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached it. Stop
// retransmitting msg3.
entry.confirm_peer_new_epoch();
}
EpochReaction::None => {
// Steady-state `current`, or an old-epoch `previous` straggler:
// `fsp_trial_decrypt` already refreshed the drain deadline so
// the `previous` slot is not retired while the peer keeps using
// it — no further state change, just deliver.
}
}
@@ -305,16 +324,16 @@ impl Node {
if let Some(entry) = self.sessions.get_mut(src_addr)
&& let Some(mmp) = entry.mmp_mut()
{
let now = std::time::Instant::now();
let now_ms = crate::time::mono_ms();
mmp.receiver
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now);
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now_ms);
// Spin bit: advance state machine for correct TX reflection.
// RTT samples not fed into SRTT — timestamp-echo provides
// accurate RTT; spin bit includes variable inter-frame delays.
let inner_flags = FspInnerFlags::from_byte(inner_flags_byte);
let _spin_rtt = mmp
.spin_bit
.rx_observe(inner_flags.spin_bit, header.counter, now);
.rx_observe(inner_flags.spin_bit, header.counter, now_ms);
}
// Feed path_mtu from datagram envelope to MMP path MTU tracking.
@@ -355,7 +374,7 @@ impl Node {
mark_ipv6_ecn_ce(&mut packet);
self.metrics().congestion.ce_received.inc();
}
if let Some(tun_tx) = &self.tun_tx {
if let Some(tun_tx) = &self.supervisor.tun_tx {
if let Err(e) = tun_tx.send(packet) {
debug!(error = %e, "Failed to deliver decompressed IPv6 packet to TUN");
}
@@ -444,7 +463,7 @@ impl Node {
if let Some(existing) = self.sessions.get(src_addr) {
if existing.is_initiating() {
// Simultaneous initiation: smaller NodeAddr wins as initiator
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(self.identity().node_addr(), src_addr) {
// We win — drop their setup, they'll process ours
debug!(
src = %self.peer_display_name(src_addr),
@@ -482,7 +501,10 @@ impl Node {
// simultaneously. Apply tie-breaker — smaller NodeAddr
// wins as initiator (same as initial session setup).
if rekey_in_progress {
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(
self.identity().node_addr(),
src_addr,
) {
// We win as initiator — drop their msg1.
debug!(
src = %self.peer_display_name(src_addr),
@@ -919,6 +941,221 @@ impl Node {
// === Session-layer MMP report handlers ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now_ms = crate::time::mono_ms();
// Build one report-gating snapshot per session, resolving every timing
// read shell-side into a `bool`. The snapshots own only
// `NodeAddr`/`MmpMode`/`bool`, so the session-iteration borrow is released
// before the pure decision runs and the driving loop mutates the
// reporting state / performs the sends.
let snapshots: Vec<SessionReportSnapshot> = self
.sessions
.iter()
.filter_map(|(dest_addr, entry)| {
let mmp = entry.mmp()?;
Some(SessionReportSnapshot {
dest: *dest_addr,
mode: mmp.mode(),
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
mtu_due: mmp.path_mtu.should_send_notification(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
let actions = self.mmp.plan_session_reports(&snapshots);
// Drive the planned actions in phase-grouped order (all logs, then the
// sends in per-session SR/RR/MTU order). Logs run first because the
// session operator log reads cumulative_packets_sent, which each send
// advances (send_session_msg -> sender.record_sent); the pre-refactor
// handler logged during its collect pass, before any send. Each build
// (`build_report`/`build_notification`, which advance interval/
// notification state) runs only on its SendSessionReport action, exactly
// as the pre-refactor collect pass did. Per-destination success/failure
// is collected for the backoff dedup + failure-log suppression.
let mut send_results: Vec<SendResult> = Vec::new();
for action in actions {
match action {
MmpAction::LogSession { dest } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub from the session's remote key).
let session_name = self.peer_aliases.get(&dest).cloned().unwrap_or_else(|| {
self.sessions
.get(&dest)
.map(|entry| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
})
.unwrap_or_default()
});
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::SendSessionReport { dest, kind } => {
let built = self
.sessions
.get_mut(&dest)
.and_then(|entry| entry.mmp_mut())
.and_then(|mmp| match kind {
SessionReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| {
(
SessionMessageType::SenderReport.to_byte(),
SessionSenderReport::from(&sr).encode(),
)
})
}
SessionReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| {
(
SessionMessageType::ReceiverReport.to_byte(),
SessionReceiverReport::from(&rr).encode(),
)
})
}
SessionReportKind::PathMtu => {
mmp.path_mtu.build_notification(now_ms).map(|mtu_value| {
(
SessionMessageType::PathMtuNotification.to_byte(),
PathMtuNotification::new(mtu_value).encode(),
)
})
}
});
let Some((msg_type, body)) = built else {
continue;
};
match self.send_session_msg(&dest, msg_type, &body).await {
Ok(()) => send_results.push(SendResult { dest, ok: true }),
Err(e) => {
// Peek at current failure count for log suppression
// (unchanged by the backoff apply, which runs later).
let failures = self
.sessions
.get(&dest)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push(SendResult { dest, ok: false });
}
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. } => {}
}
}
// Deduplicate send results per destination (any-ok -> success, all-fail
// -> failure) and apply the backoff state transition for each dest.
for update in self.mmp.plan_backoff(&send_results) {
match update {
BackoffUpdate::Success { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
}
}
BackoffUpdate::Failure { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
mmp.sender.record_send_failure();
}
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Handle an incoming session-layer SenderReport (msg_type 0x11).
///
/// Informational only — the peer is telling us about what they sent.
@@ -974,9 +1211,11 @@ impl Node {
return;
};
let now = std::time::Instant::now();
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let (_first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, crate::time::mono_ms());
// Re-emit the operator trace the core used to log mid-decision.
super::mmp::log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning (session-layer bounds)
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -1042,8 +1281,9 @@ impl Node {
};
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
let changed = mmp.path_mtu.apply_notification(notif.path_mtu, now);
let changed = mmp
.path_mtu
.apply_notification(notif.path_mtu, crate::time::mono_ms());
let new_mtu = mmp.path_mtu.current_mtu();
if !changed {
@@ -1065,28 +1305,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(src_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= new_mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, new_mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
existing,
existing = prior.unwrap_or(new_mtu),
"PathMtuNotification: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, new_mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?other,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?prior,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1125,7 +1371,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1141,12 +1387,19 @@ impl Node {
// Only trigger discovery if we have the target's identity cached —
// otherwise we can't verify the LookupResponse proof.
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_coords_required_lookup(msg.dest_addr, has_cached_identity);
if actions.is_empty() {
debug!(dest = %msg.dest_addr,
"Skipping discovery after CoordsRequired: no cached identity for target");
}
for action in actions {
if let FspAction::InitiateLookup { dest } = action {
self.maybe_initiate_lookup(&dest).await;
}
}
// Reset coords warmup counter so the next N packets also include
// COORDS_PRESENT, re-warming transit caches along the path.
@@ -1186,7 +1439,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1200,16 +1453,26 @@ impl Node {
"PathBroken response rate-limited, skipping standalone CoordsWarmup");
}
// Invalidate stale cached coordinates
self.coord_cache.remove(&msg.dest_addr);
// Trigger re-discovery to get fresh coordinates, but only if we have
// the target's identity cached — otherwise we can't verify the
// LookupResponse proof. This avoids a race when the XK responder
// receives PathBroken before msg3 completes (identity unknown).
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
// Invalidate stale cached coordinates, then (only if the target's
// identity is cached — else the LookupResponse proof cannot be verified,
// e.g. when the XK responder receives PathBroken before msg3 completes)
// trigger re-discovery. The core emits invalidate-then-lookup in order.
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_path_broken(msg.dest_addr, has_cached_identity);
for action in actions {
match action {
FspAction::InvalidateCoords { addr } => {
self.coord_cache.remove(&addr);
}
FspAction::InitiateLookup { dest } => {
self.maybe_initiate_lookup(&dest).await;
}
_ => {}
}
}
if !has_cached_identity {
debug!(dest = %msg.dest_addr,
"Skipping discovery after PathBroken: no cached identity for target");
}
@@ -1256,8 +1519,10 @@ impl Node {
&& let Some(mmp) = entry.mmp_mut()
{
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
if mmp.path_mtu.apply_notification(msg.mtu, now) {
if mmp
.path_mtu
.apply_notification(msg.mtu, crate::time::mono_ms())
{
let new_mtu = mmp.path_mtu.current_mtu();
info!(
dest = %peer_name,
@@ -1277,28 +1542,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(&msg.dest_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= msg.mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, msg.mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
existing,
existing = prior.unwrap_or(msg.mtu),
"Reactive MtuExceeded: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, msg.mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?other,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?prior,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1561,7 +1832,7 @@ impl Node {
send: PipelinedSend<'_>,
) -> Result<bool, NodeError> {
let dest_addr = send.dest_addr;
let Some(workers) = self.encrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.encrypt_workers.as_ref().cloned() else {
return Ok(false);
};
@@ -2065,7 +2336,10 @@ impl Node {
/// Returns our own coordinates as a fallback (the SessionSetup will
/// carry src_coords for return path routing; empty dest_coords
/// would fail wire encoding since TreeCoordinate requires ≥1 entry).
pub(in crate::node) fn get_dest_coords(&self, dest: &NodeAddr) -> crate::tree::TreeCoordinate {
pub(in crate::node) fn get_dest_coords(
&self,
dest: &NodeAddr,
) -> crate::proto::stp::TreeCoordinate {
let now_ms = Self::now_ms();
if let Some(coords) = self.coord_cache.get(dest, now_ms) {
return coords.clone();
@@ -2174,7 +2448,7 @@ impl Node {
let our_ipv6 = FipsAddress::from_node_addr(self.node_addr()).to_ipv6();
if let Some(response) =
build_dest_unreachable(original_packet, DestUnreachableCode::NoRoute, our_ipv6)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
let _ = tun_tx.send(response);
}
@@ -2209,7 +2483,7 @@ impl Node {
// causes a PMTUD blackhole when both src and ICMP-src are local.
let dest_addr = Ipv6Addr::from(<[u8; 16]>::try_from(&original_packet[24..40]).unwrap());
if let Some(response) = build_packet_too_big(original_packet, mtu, dest_addr)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
debug!(
original_src = %src_addr,
@@ -2234,10 +2508,7 @@ impl Node {
let per_dest = self.config().node.session.pending_packets_per_dest;
let queue = self.pending_tun_packets.entry(dest_addr).or_default();
if queue.len() >= per_dest {
queue.pop_front(); // Drop oldest
}
queue.push_back(packet);
crate::proto::fsp::push_bounded_pending(queue, packet, per_dest);
}
/// Flush pending packets for a destination whose session just reached Established.
@@ -2288,31 +2559,3 @@ impl Node {
}
}
}
/// Mark ECN-CE in an IPv6 packet's Traffic Class field.
///
/// IPv6 Traffic Class occupies bits across bytes 0 and 1:
/// byte[0] bits[3:0] = TC[7:4]
/// byte[1] bits[7:4] = TC[3:0]
/// ECN is TC[1:0]. Only marks CE (0b11) if the packet is ECN-capable
/// (ECT(0) or ECT(1)). Packets with ECN=0b00 (Not-ECT) are never marked
/// per RFC 3168.
///
/// No checksum update needed: IPv6 has no header checksum, and the Traffic
/// Class field is not part of the TCP/UDP pseudo-header.
pub(in crate::node) fn mark_ipv6_ecn_ce(packet: &mut [u8]) {
if packet.len() < 2 {
return;
}
// Extract 8-bit Traffic Class from IPv6 header bytes 0-1
let tc = ((packet[0] & 0x0F) << 4) | (packet[1] >> 4);
let ecn = tc & 0x03;
// Only mark CE on ECN-capable packets (ECT(0)=0b10 or ECT(1)=0b01)
if ecn == 0 {
return;
}
// Set both ECN bits to 1 (CE = 0b11)
let new_tc = tc | 0x03;
packet[0] = (packet[0] & 0xF0) | (new_tc >> 4);
packet[1] = (new_tc << 4) | (packet[1] & 0x0F);
}

View File

@@ -3,14 +3,69 @@
use crate::node::Node;
use crate::peer::HandshakeState;
use crate::proto::fmp::{
ConnAction, ConnSnapshot, LifecycleView, PeerSnapshot, RekeyResendSnapshot,
};
use crate::transport::LinkId;
use tracing::{debug, info};
impl LifecycleView for Node {
fn stale_connections(&self, now_ms: u64, timeout_ms: u64) -> Vec<ConnSnapshot> {
self.connections
.iter()
.filter(|(_, conn)| conn.is_timed_out(now_ms, timeout_ms) || conn.is_failed())
.map(|(link_id, conn)| ConnSnapshot {
link: *link_id,
is_outbound: conn.is_outbound(),
retry_addr: conn.expected_identity().map(|id| *id.node_addr()),
resend_count: 0,
msg1: Vec::new(),
})
.collect()
}
fn resend_candidates(&self, now_ms: u64, max_resends: u32) -> Vec<ConnSnapshot> {
self.connections
.iter()
.filter(|(_, conn)| {
conn.is_outbound()
&& conn.handshake_state() == HandshakeState::SentMsg1
&& conn.resend_count() < max_resends
&& conn.next_resend_at_ms() > 0
&& now_ms >= conn.next_resend_at_ms()
})
.filter_map(|(link_id, conn)| {
conn.handshake_msg1().map(|msg1| ConnSnapshot {
link: *link_id,
is_outbound: true,
retry_addr: None,
resend_count: conn.resend_count(),
msg1: msg1.to_vec(),
})
})
.collect()
}
fn rekey_peers(&self) -> Vec<PeerSnapshot> {
// The snapshot builder lives in `rekey` beside its drain/dampening
// constants; the read-seam unifies here.
self.rekey_peer_snapshots()
}
fn rekey_resend_candidates(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.rekey_resend_snapshots(now_ms)
}
}
impl Node {
/// Check for timed-out handshake connections and clean them up.
///
/// Called periodically by the RX event loop. Removes connections that have
/// been idle longer than the configured handshake timeout or are in Failed state.
///
/// The stale/failed predicate and every registry mutation stay shell-side;
/// the retry-then-teardown choreography is the pure
/// [`Fmp::poll_timeouts`](crate::proto::fmp::Fmp::poll_timeouts) decision.
pub(in crate::node) fn check_timeouts(&mut self) {
if self.connections.is_empty() {
return;
@@ -19,41 +74,34 @@ impl Node {
let now_ms = Self::now_ms();
let timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
let stale: Vec<LinkId> = self
.connections
.iter()
.filter(|(_, conn)| conn.is_timed_out(now_ms, timeout_ms) || conn.is_failed())
.map(|(link_id, _)| *link_id)
.collect();
for link_id in stale {
// Log and schedule retry before cleanup (need connection state)
if let Some(conn) = self.connections.get(&link_id) {
let direction = conn.direction();
let idle_ms = conn.idle_time(now_ms);
if conn.is_failed() {
debug!(
link_id = %link_id,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link_id,
direction = %direction,
idle_secs = idle_ms / 1000,
"Stale handshake connection timed out"
);
}
// Schedule retry for failed outbound auto-connect peers
if conn.is_outbound()
&& let Some(identity) = conn.expected_identity()
{
self.schedule_retry(*identity.node_addr(), now_ms);
let stale = self.stale_connections(now_ms, timeout_ms);
for action in self.fmp.poll_timeouts(stale) {
match action {
ConnAction::ScheduleRetry { peer } => self.note_handshake_timeout(peer, now_ms),
ConnAction::Teardown { link } => {
// Log before cleanup (needs live connection state).
if let Some(conn) = self.connections.get(&link) {
let direction = conn.direction();
if conn.is_failed() {
debug!(
link_id = %link,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link,
direction = %direction,
idle_secs = conn.idle_time(now_ms) / 1000,
"Stale handshake connection timed out"
);
}
}
self.cleanup_stale_connection(link, now_ms);
}
#[allow(unreachable_patterns)]
_ => {}
}
self.cleanup_stale_connection(link_id, now_ms);
}
}
@@ -97,26 +145,24 @@ impl Node {
let interval_ms = self.config().node.rate_limit.handshake_resend_interval_ms;
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
// Collect resend candidates: outbound, in SentMsg1, with stored msg1,
// under max resends, and past the scheduled time.
let candidates: Vec<(LinkId, Vec<u8>)> = self
.connections
.iter()
.filter(|(_, conn)| {
conn.is_outbound()
&& conn.handshake_state() == HandshakeState::SentMsg1
&& conn.resend_count() < max_resends
&& conn.next_resend_at_ms() > 0
&& now_ms >= conn.next_resend_at_ms()
})
.filter_map(|(link_id, conn)| {
conn.handshake_msg1().map(|msg1| (*link_id, msg1.to_vec()))
})
.collect();
// The shell resolves the resend-candidate predicate and copies the
// opaque msg1 bytes; the core computes the backoff schedule.
let candidates = self.resend_candidates(now_ms, max_resends);
for action in self
.fmp
.poll_resends(candidates, now_ms, interval_ms, backoff)
{
let ConnAction::ResendMsg1 {
link,
bytes,
next_resend_at_ms,
} = action
else {
continue;
};
for (link_id, msg1_bytes) in candidates {
// Get transport and address info from the connection
let (transport_id, remote_addr) = match self.connections.get(&link_id) {
let (transport_id, remote_addr) = match self.connections.get(&link) {
Some(conn) => match (conn.transport_id(), conn.source_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
@@ -126,11 +172,11 @@ impl Node {
// Send the stored msg1
let sent = if let Some(transport) = self.transports.get(&transport_id) {
match transport.send(&remote_addr, &msg1_bytes).await {
match transport.send(&remote_addr, &bytes).await {
Ok(_) => true,
Err(e) => {
debug!(
link_id = %link_id,
link_id = %link,
error = %e,
"Handshake msg1 resend failed"
);
@@ -141,13 +187,11 @@ impl Node {
false
};
if sent && let Some(conn) = self.connections.get_mut(&link_id) {
let count = conn.resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
conn.record_resend(next);
if sent && let Some(conn) = self.connections.get_mut(&link) {
conn.record_resend(next_resend_at_ms);
debug!(
link_id = %link_id,
resend = count,
link_id = %link,
resend = conn.resend_count(),
"Resent handshake msg1"
);
}
@@ -204,7 +248,7 @@ impl Node {
.collect();
for (dest_addr, payload) in candidates {
use crate::protocol::SessionDatagram;
use crate::proto::link::SessionDatagram;
let mut datagram = SessionDatagram::new(my_addr, dest_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -15,8 +15,8 @@ use std::sync::atomic::{AtomicU64, Ordering};
use crate::node::reject::{BloomReject, DiscoveryReject, ForwardingReject, TreeReject};
use crate::node::stats::{
BloomStatsSnapshot, CongestionStatsSnapshot, DiscoveryStatsSnapshot, ErrorSignalStatsSnapshot,
ForwardingStatsSnapshot, TreeStatsSnapshot,
BloomStatsSnapshot, CongestionStatsSnapshot, ErrorSignalStatsSnapshot, ForwardingStatsSnapshot,
LookupStatsSnapshot, TreeStatsSnapshot,
};
/// An atomic counter.
@@ -90,43 +90,10 @@ pub struct ForwardingMetrics {
}
/// Route class of a transit-forwarded packet, classified from tree
/// coordinates at the forwarding decision point. The six variants
/// partition `forwarded_packets` exactly.
///
/// Two variants are up-and-over forwards (destination not in the chosen
/// peer's subtree); they differ in whether they depend on a child
/// advertising cross-link reach *upward* to its parent:
/// - `TreeDownCross`: the chosen peer is our tree descendant, but the
/// destination is *not* in that child's subtree. The forward only fired
/// because the child advertised cross-link reach upward to us, beyond its
/// own subtree. If children advertised only their subtree upward, this
/// forward would route up instead, so its count measures how much
/// forwarding depends on the upward cross-link advertisement — the
/// dive-to-tree-child cut-through.
/// - `CrosslinkAscend`: the chosen peer is lateral (neither ancestor nor
/// descendant) and the destination is not in its subtree. This is a node
/// using its *own* cross-link, learned via the peer's split-horizon
/// advertisement to its neighbors, so it does not depend on any upward
/// advertisement. Tracked alongside `TreeDownCross` as the lateral
/// up-and-over contrast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteClass {
/// Chosen peer is our ancestor (tree-up).
TreeUp,
/// Chosen peer is our descendant and dest is in its subtree (canonical
/// tree-down).
TreeDown,
/// Chosen peer is our descendant but dest is *not* in its subtree: the
/// dive-to-tree-child cut-through enabled by upward cross-link
/// advertisement.
TreeDownCross,
/// Chosen peer is lateral and dest is in its subtree (subtree entry).
CrosslinkDescend,
/// Chosen peer is lateral and dest is not in its subtree (up-and-over).
CrosslinkAscend,
/// Chosen peer is the destination itself (degenerate direct hop).
DirectPeer,
}
/// coordinates at the forwarding decision point. Defined by the sans-IO
/// routing core and re-exported here for the forwarding-metrics counters
/// ([`ForwardingMetrics::record_route_class`]).
pub(crate) use crate::proto::routing::RouteClass;
impl ForwardingMetrics {
/// Record a received packet of `bytes` payload (packets and bytes).
@@ -235,7 +202,7 @@ impl ForwardingMetrics {
/// Discovery metric counters.
#[derive(Default)]
pub struct DiscoveryMetrics {
pub struct LookupMetrics {
pub req_received: Padded<Counter>,
pub req_decode_error: Counter,
pub req_duplicate: Counter,
@@ -260,7 +227,7 @@ pub struct DiscoveryMetrics {
pub resp_timed_out: Counter,
}
impl DiscoveryMetrics {
impl LookupMetrics {
/// Mirror of `DiscoveryStats::record_reject`: route a typed discovery
/// rejection to its counter.
#[inline]
@@ -278,8 +245,8 @@ impl DiscoveryMetrics {
}
/// Sample every counter into a serializable snapshot.
pub fn snapshot(&self) -> DiscoveryStatsSnapshot {
DiscoveryStatsSnapshot {
pub fn snapshot(&self) -> LookupStatsSnapshot {
LookupStatsSnapshot {
req_received: self.req_received.get(),
req_decode_error: self.req_decode_error.get(),
req_duplicate: self.req_duplicate.get(),
@@ -460,7 +427,7 @@ impl ErrorMetrics {
#[derive(Default)]
pub struct MetricsRegistry {
pub forwarding: ForwardingMetrics,
pub discovery: DiscoveryMetrics,
pub lookup: LookupMetrics,
pub tree: TreeMetrics,
pub bloom: BloomMetrics,
pub congestion: CongestionMetrics,
@@ -488,7 +455,7 @@ mod tests {
#[test]
fn discovery_record_reject_routes_to_field() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::RespNoRoute);
@@ -499,7 +466,7 @@ mod tests {
#[test]
fn discovery_direct_counters_increment() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.req_received.inc();
m.req_forwarded.inc();
m.req_forwarded.inc();
@@ -532,9 +499,9 @@ mod tests {
fn registry_subcounters_are_independent() {
let r = MetricsRegistry::new();
r.forwarding.record_received(10);
r.discovery.req_received.inc();
r.lookup.req_received.inc();
assert_eq!(r.forwarding.received_packets.get(), 1);
assert_eq!(r.forwarding.received_bytes.get(), 10);
assert_eq!(r.discovery.req_received.get(), 1);
assert_eq!(r.lookup.req_received.get(), 1);
}
}

File diff suppressed because it is too large Load Diff

148
src/node/peering/driver.rs Normal file
View File

@@ -0,0 +1,148 @@
//! Thin async driver for the peering reconciler.
//!
//! These `impl Node` methods are the I/O edge of the sans-IO
//! [`super::reconcile::PeeringReconciler`]: they snapshot the live dataplane
//! maps into the reconciler's plain-data inputs, invoke the pure core, and
//! perform the dial / advert-refetch I/O each [`PeeringAction`] names. They also
//! host the two gate-guarded reflex wrappers every peer-loss call site routes
//! through, so drain suppression and the connected-guard live in one place.
//!
//! The `Policy` / `Observed` / `Budget` builders these methods consume live in
//! [`crate::node::lifecycle`] next to the surviving budget helpers and limit
//! constants they wrap.
use crate::identity::NodeAddr;
use crate::node::{Node, NodeError};
use tracing::warn;
use super::reconcile::{DiscoveryPools, Gate, PeeringAction};
impl Node {
/// Reflex: an outbound handshake timed out (replaces the old
/// `Node::schedule_retry` call sites).
///
/// Replicates `schedule_retry`'s connected-guard — the pure core cannot
/// observe the peers map, so the driver drops the event when the peer is
/// already connected — then feeds the gate-guarded reconciler reflex with
/// the gate derived from the live published state.
pub(in crate::node) fn note_handshake_timeout(&mut self, node_addr: NodeAddr, now_ms: u64) {
if self.peers.contains_key(&node_addr) {
return;
}
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_handshake_timeout(node_addr, now_ms, &policy, gate);
}
/// Reflex: a link went dead / a peer was lost (replaces the old
/// `Node::schedule_reconnect` call sites).
///
/// No connected-guard — the peer is already gone by the time a link-dead /
/// disconnect event fires (`schedule_reconnect` had none). The gate is
/// derived from the live published state so a drain self-suppresses the
/// reconnect.
pub(in crate::node) fn note_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_link_dead(node_addr, now_ms, &policy, gate);
}
/// Process pending retries whose time has arrived (replaces the old
/// `Node::process_pending_retries` body).
///
/// The pure retry-dial phase owns the decision — drop expired entries, refuse
/// to grow when admission binds, dial the first `retry_per_tick` due entries
/// (bumping their `retry_after_ms` past the handshake window). This driver
/// performs the advert-refetch + dial I/O each emitted `Connect` names, and
/// on an immediate dial error feeds the `on_handshake_timeout` reflex so the
/// optimistic re-fire suppression is overwritten by proper backoff. During a
/// drain the gate is `Suspended`, so the reconcile clears the schedule and
/// emits nothing.
pub(in crate::node) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.peering.reconciler.retry_pending.is_empty() {
return;
}
// Retry-dial cadence slot: empty config floor and empty discovery
// pools, so only the retry-dial phase acts.
let policy = self.build_peering_policy(Vec::new());
let observed = self.observe_peering();
let budget = self.build_peering_budget();
let gate = Gate::from_state(self.supervisor.state);
let actions = self.peering.reconciler.reconcile(
&policy,
&observed,
&budget,
&DiscoveryPools::default(),
now_ms,
gate,
);
for action in actions {
let PeeringAction::Connect(candidate) = action else {
continue;
};
let Some(identity) = candidate.identity else {
continue;
};
let node_addr = *identity.node_addr();
let Some(peer_config) = self
.peering
.reconciler
.retry_pending
.get(&node_addr)
.map(|state| state.peer_config.clone())
else {
continue;
};
// Refresh the peer's overlay advert before retrying. The cache is
// read-only on hit, so a retry without a refetch dials the same
// cached endpoint — and the most common reason a peer landed in the
// retry schedule is that endpoint just stopped working (NAT rebind,
// port change, peer restart). Cheap (one Filter fetch, bounded by
// the retry backoff cadence).
if let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc() {
let _ = bootstrap
.refetch_advert_for_stale_check(&peer_config.npub)
.await;
}
match self.initiate_peer_connection(&peer_config).await {
// The core already pushed `retry_after_ms` past the handshake
// window; a successful promotion clears the entry, a later
// timeout re-fires the reflex with proper backoff.
Ok(()) => {}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale; force a re-fetch so the next tick picks up
// fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt: overwrite the
// optimistic re-fire suppression with backoff.
self.note_handshake_timeout(node_addr, now_ms);
}
}
}
}
}

14
src/node/peering/mod.rs Normal file
View File

@@ -0,0 +1,14 @@
//! Peering homeostasis — the desired-state controller for the node's peer set.
//!
//! This module is the home for the peering-reconciler concept: config defines a
//! desired peer set; the reconciler converges the observed set toward it
//! (auto-connect floor, overlay pool, transport-neighbor growth) under the
//! `node.limits` ceiling. Startup and steady-state are the same loop.
//!
//! The cross-attempt retry schedule (`retry.rs`) lives here because a fresh
//! connection is created per re-dial, so the escalating backoff count must
//! persist in the reconciler, not per-connection.
pub(in crate::node) mod driver;
pub(in crate::node) mod reconcile;
pub(in crate::node) mod retry;

File diff suppressed because it is too large Load Diff

49
src/node/peering/retry.rs Normal file
View File

@@ -0,0 +1,49 @@
//! Cross-attempt retry state for auto-connect peers.
//!
//! [`RetryState`] is the durable per-peer schedule entry the peering reconciler
//! owns (it lives in [`crate::node::peering::reconcile::PeeringReconciler`], not
//! on a per-connection object, because a fresh connection is created per re-dial
//! so the escalating backoff count must persist across attempts). The decision
//! logic that reads and mutates it — the retry-dial phase and the
//! `on_handshake_timeout` / `on_link_dead` reflexes — lives in the sans-IO
//! reconciler core; the driver wrappers that feed it (retry-dial I/O, the
//! gate-guarded reflex call sites) live in [`super::driver`].
use crate::config::PeerConfig;
/// Per-tick cap on retry-dial connection attempts (ceiling).
pub(in crate::node) const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
}

View File

@@ -222,7 +222,7 @@ pub enum MmpReject {
/// Forwarding-path rejection reasons.
///
/// Each variant corresponds to a silent-rejection path in
/// `src/node/handlers/forwarding.rs::handle_session_datagram`. Matching
/// `src/node/dataplane/forwarding.rs::handle_session_datagram`. Matching
/// `ForwardingStats` counters already track packets and bytes for each
/// outcome; `record_reject` mirrors the packet-count side of the bump
/// for parity with the other rejection clusters.

View File

@@ -41,7 +41,7 @@
//! file. There is nothing to poll. (Its read side could adopt the same
//! lock-free `ArcSwap` shape in the future, but that is an optimization, not
//! a reload.)
//! - `nostr_discovery` is an async spawned subsystem, not a snapshot of disk
//! - `nostr_rendezvous` is an async spawned subsystem, not a snapshot of disk
//! state.
//!
//! Both [`HostMapReloadable`] and the peer ACL reloader currently stat

View File

@@ -1,417 +0,0 @@
//! Connection retry logic for auto-connect peers.
//!
//! When an outbound handshake fails (timeout or send error), the node can
//! automatically retry with exponential backoff. Retry state lives on Node
//! (not PeerConnection) because each retry creates a fresh connection.
use super::{Node, NodeError};
use crate::PeerIdentity;
use crate::config::PeerConfig;
use crate::identity::NodeAddr;
use tracing::{debug, info, warn};
// MAX_BACKOFF_MS is now derived from config: node.retry.max_backoff_secs * 1000
const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
/// Calculate the backoff delay in milliseconds for the current retry count.
///
/// Uses exponential backoff: `base_interval_ms * 2^retry_count`,
/// capped at `MAX_BACKOFF_MS`.
pub fn backoff_ms(&self, base_interval_ms: u64, max_backoff_ms: u64) -> u64 {
let multiplier = 1u64.checked_shl(self.retry_count).unwrap_or(u64::MAX);
base_interval_ms
.saturating_mul(multiplier)
.min(max_backoff_ms)
}
}
impl Node {
/// Schedule a retry for a failed outbound connection, if applicable.
///
/// Only schedules if the peer is an auto-connect peer and max retries
/// have not been exhausted (unless `reconnect` is true, which retries
/// indefinitely). Does nothing if the peer is already connected or has
/// a connection in progress.
pub(super) fn schedule_retry(&mut self, node_addr: NodeAddr, now_ms: u64) {
let retry_cfg = &self.config().node.retry;
let max_retries = retry_cfg.max_retries;
if max_retries == 0 {
return;
}
// Don't retry if peer is already connected
if self.peers.contains_key(&node_addr) {
return;
}
let base_interval_ms = retry_cfg.base_interval_secs * 1000;
let max_backoff_ms = retry_cfg.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
// Already tracking — increment
state.retry_count += 1;
if !state.reconnect && state.retry_count > max_retries {
info!(
peer = %peer_name,
attempts = state.retry_count,
"Max retries exhausted, giving up on peer"
);
self.retry_pending.remove(&node_addr);
return;
}
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
reconnect = state.reconnect,
delay_secs = delay / 1000,
"Scheduling connection retry"
);
} else {
// First failure — find the matching PeerConfig
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
if let Some(pc) = peer_config {
let mut state = RetryState::new(pc);
state.retry_count = 1;
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %self.peer_display_name(&node_addr),
delay_secs = delay / 1000,
"First connection attempt failed, scheduling retry"
);
self.retry_pending.insert(node_addr, state);
}
// If not found in auto_connect_peers, no retry (one-shot connection)
}
}
/// Schedule auto-reconnect for a peer removed by MMP dead timeout.
///
/// Looks up the peer in auto-connect config and checks `auto_reconnect`.
/// If enabled, feeds the peer into the retry system with unlimited retries.
///
/// If a retry entry already exists (e.g. from a previous failed handshake
/// attempt during an earlier reconnect cycle), the existing retry count is
/// preserved and incremented rather than reset to zero. This ensures
/// exponential backoff accumulates across repeated link-dead events instead
/// of resetting to the base interval on every peer removal.
pub(super) fn schedule_reconnect(&mut self, node_addr: NodeAddr, now_ms: u64) {
// Find peer in auto-connect config
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
let Some(pc) = peer_config else {
return; // Not an auto-connect peer, no reconnect
};
if !pc.auto_reconnect {
debug!(
peer = %self.peer_display_name(&node_addr),
"Auto-reconnect disabled for peer, skipping"
);
return;
}
let base_interval_ms = self.config().node.retry.base_interval_secs * 1000;
let max_backoff_ms = self.config().node.retry.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
// If we already have accumulated backoff from previous failed attempts,
// preserve and bump it rather than resetting to zero. This prevents the
// exponential backoff from being discarded on each link-dead cycle.
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.reconnect = true;
state.retry_count += 1;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal (backoff preserved)"
);
return;
}
let mut state = RetryState::new(pc);
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal"
);
self.retry_pending.insert(node_addr, state);
}
/// Process pending retries whose time has arrived.
///
/// For each due retry, initiates a fresh connection attempt. The retry
/// entry stays in `retry_pending` until the connection succeeds (cleared
/// in `promote_connection`) or max retries are exhausted (cleared in
/// `schedule_retry`).
pub(super) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.retry_pending.is_empty() {
return;
}
let expired: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter_map(|(addr, state)| {
state
.expires_at_ms
.filter(|expires_at_ms| now_ms >= *expires_at_ms)
.map(|_| *addr)
})
.collect();
for node_addr in expired {
self.retry_pending.remove(&node_addr);
info!(
peer = %self.peer_display_name(&node_addr),
"Retry window expired, dropping pending retry state"
);
}
if self.retry_pending.is_empty() {
return;
}
if !self.outbound_admission_check() {
debug!(
peers = self.peers.len(),
max_peers = self.max_peers(),
retry_pending = self.retry_pending.len(),
"Suppressing auto-reconnect retries: at capacity"
);
return;
}
// Collect retries that are due
let due: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter(|(_, state)| now_ms >= state.retry_after_ms)
.map(|(addr, _)| *addr)
.collect();
let deferred = due.len().saturating_sub(MAX_RETRY_CONNECTIONS_PER_TICK);
if deferred > 0 {
debug!(
due = due.len(),
processing = MAX_RETRY_CONNECTIONS_PER_TICK,
deferred,
"Retry processing budget exhausted; deferring remaining peers"
);
}
for node_addr in due.into_iter().take(MAX_RETRY_CONNECTIONS_PER_TICK) {
// Peer may have connected inbound while we waited
if self.peers.contains_key(&node_addr) {
self.retry_pending.remove(&node_addr);
continue;
}
let state = match self.retry_pending.get(&node_addr) {
Some(s) => s,
None => continue,
};
debug!(
peer = %self.peer_display_name(&node_addr),
retry = state.retry_count,
"Attempting connection retry"
);
let peer_config = state.peer_config.clone();
// Refresh the peer's overlay advert before retrying. The cache is
// read-only on hit (see fetch_advert), so every retry without a
// refetch dials the same cached endpoint — and the most common
// reason a peer ended up in retry_pending is that the cached
// endpoint just stopped working (NAT rebind, port change, peer
// restart on a different port). Without this refresh the retry
// loop dials the same dead address forever.
//
// refetch_advert_for_stale_check uses the relay's advert as
// ground truth: replaces the cache if there's a newer one,
// evicts if the relay has nothing, otherwise leaves it. Cheap
// (one Filter fetch with 2s timeout) and bounded by the retry
// backoff cadence.
if let Some(bootstrap) = self.nostr_discovery.clone() {
let _ = bootstrap
.refetch_advert_for_stale_check(&peer_config.npub)
.await;
}
match self.initiate_peer_connection(&peer_config).await {
Ok(()) => {
// Push retry_after_ms past the handshake timeout window so
// we don't re-fire on the next tick. If the handshake
// succeeds, promote_connection() clears retry_pending. If
// it times out, check_timeouts() calls schedule_retry()
// which bumps the counter and applies proper backoff.
let hs_timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.retry_after_ms = now_ms + hs_timeout_ms;
}
debug!(
peer = %self.peer_display_name(&node_addr),
"Retry connection initiated, suppressing re-fire for {}s",
self.config().node.rate_limit.handshake_timeout_secs,
);
}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale (peer rebound NAT, switched relay, etc.).
// The advert cache is read-only inside fetch_advert, so
// every retry returns the same dead address until the
// entry expires. Force a re-fetch so the next retry tick
// picks up fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.nostr_discovery.clone()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt — schedule next retry
// (reconnect flag is preserved on existing retry_pending entry)
self.schedule_retry(node_addr, now_ms);
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::PeerConfig;
const TEST_MAX_BACKOFF_MS: u64 = 300_000;
#[test]
fn test_backoff_exponential() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
// base = 5000ms
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 5000); // 5s * 2^0
let state = RetryState {
retry_count: 1,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 10_000); // 5s * 2^1
let state = RetryState {
retry_count: 2,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 20_000); // 5s * 2^2
let state = RetryState {
retry_count: 3,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 40_000); // 5s * 2^3
let state = RetryState {
retry_count: 4,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 80_000); // 5s * 2^4
}
#[test]
fn test_backoff_cap() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 20, // 2^20 * 5000 would be huge
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(
state.backoff_ms(5000, TEST_MAX_BACKOFF_MS),
TEST_MAX_BACKOFF_MS
);
}
#[test]
fn test_backoff_zero_base() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 3,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(state.backoff_ms(0, TEST_MAX_BACKOFF_MS), 0);
}
}

View File

@@ -1,161 +0,0 @@
//! Routing error signal rate limiting.
//!
//! Prevents routing error floods (CoordsRequired / PathBroken) by
//! rate-limiting error signals per destination address at transit nodes.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
/// Rate limiter for routing error signals (CoordsRequired / PathBroken).
///
/// Tracks the last time a routing error was sent for each destination
/// address and enforces a minimum interval to prevent floods.
pub struct RoutingErrorRateLimiter {
/// Maps destination NodeAddr to the last time we sent an error about it.
last_sent: HashMap<NodeAddr, Instant>,
/// Minimum interval between error signals for the same destination.
min_interval: Duration,
/// Maximum age of entries before cleanup.
max_age: Duration,
}
impl RoutingErrorRateLimiter {
/// Create a new rate limiter.
///
/// Default: max 10 errors/sec per destination (100ms interval).
pub fn new() -> Self {
Self {
last_sent: HashMap::new(),
min_interval: Duration::from_millis(100),
max_age: Duration::from_secs(10),
}
}
/// Create a rate limiter with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_sent: HashMap::new(),
min_interval,
max_age: Duration::from_secs(10),
}
}
/// Check if we should send a routing error for this destination.
///
/// Returns true if enough time has passed since the last error for
/// this destination, or if this is the first error. Updates internal
/// state when returning true.
pub fn should_send(&mut self, dest_addr: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_sent.get(dest_addr)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_sent.insert(*dest_addr, now);
self.cleanup(now);
true
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_sent
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_sent.len()
}
}
impl Default for RoutingErrorRateLimiter {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
#[test]
fn test_first_send_allowed() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_rapid_sends_rate_limited() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
}
#[test]
fn test_different_destinations_independent() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(2)));
}
#[test]
fn test_send_allowed_after_interval() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_cleanup_removes_old_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(11);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_cleanup_preserves_recent_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
#[test]
fn test_with_interval_custom_rate() {
let mut limiter = RoutingErrorRateLimiter::with_interval(Duration::from_millis(500));
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
// Still rate-limited after 200ms (would pass with default 100ms)
thread::sleep(Duration::from_millis(200));
assert!(!limiter.should_send(&addr(1)));
// Allowed after 500ms total
thread::sleep(Duration::from_millis(350));
assert!(limiter.should_send(&addr(1)));
}
}

View File

@@ -5,13 +5,11 @@
//! (SessionSetup/SessionAck/SessionMsg3) carried inside SessionDatagram
//! envelopes through the mesh.
use std::time::Instant;
use crate::NodeAddr;
use crate::config::SessionMmpConfig;
use crate::mmp::MmpSessionState;
use crate::node::REKEY_JITTER_SECS;
use crate::noise::{HandshakeState, NoiseSession};
use crate::proto::mmp::MmpSessionState;
use rand::RngExt;
use secp256k1::PublicKey;
@@ -338,7 +336,12 @@ impl SessionEntry {
/// Initialize session-layer MMP state (called on Established transition).
pub(crate) fn init_mmp(&mut self, config: &SessionMmpConfig) {
self.mmp = Some(MmpSessionState::new(config, self.is_initiator));
self.mmp = Some(MmpSessionState::new(
config.mode,
config.log_interval_secs,
config.owd_window_size,
self.is_initiator,
));
}
// === Traffic Counters ===
@@ -665,9 +668,9 @@ impl SessionEntry {
self.rekey_jitter_secs = draw_rekey_jitter();
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
let now_ms = crate::time::mono_ms();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
mmp.reset_for_rekey(now_ms);
}
true
}
@@ -798,8 +801,8 @@ impl SessionEntry {
#[cfg(test)]
mod overlapping_epoch_tests {
use super::*;
use crate::node::session_wire::{FSP_FLAG_K, build_fsp_header};
use crate::noise::HandshakeState;
use crate::proto::fsp::wire::{FSP_FLAG_K, build_fsp_header};
use secp256k1::{Keypair, Secp256k1, SecretKey};
/// Deterministic keypair from a single seed byte.
@@ -1262,4 +1265,155 @@ mod overlapping_epoch_tests {
"window must expire on the plain wall-clock timer when peer is off the old epoch"
);
}
// ========================================================================
// Rekey-policy characterization (pins `check_session_rekey`'s decision
// boundaries before the `Fsp::poll_rekey` hoist — these thresholds have no
// other test module).
// ========================================================================
/// The initiator liveness-cutover delay used by `check_session_rekey`
/// (`FSP_CUTOVER_DELAY_MS`). Mirrored here as the characterization anchor.
const CUTOVER_DELAY_MS: u64 = 2000;
/// Build an established entry that has completed a rekey as initiator and
/// holds a pending session awaiting the K-bit cutover.
fn entry_pending_cutover(rekey_completed_ms: u64) -> SessionEntry {
let (_cur_send, cur_recv) = xk_pair(1, 2);
let (_new_send, new_recv) = xk_pair(3, 4);
let mut entry = entry_with_current(cur_recv);
// Mark ourselves the rekey initiator, then land the completed session
// as pending (clears rekey_state, so has_rekey_in_progress() == false).
entry.set_rekey_state(HandshakeState::new_xk_responder(keypair(7)), true);
entry.set_pending_session(new_recv);
entry.set_rekey_completed_ms(rekey_completed_ms);
entry
}
// The initiator-side cutover predicate: pending session present, no rekey
// in progress, we are the initiator, and the liveness timer has elapsed.
#[test]
fn rekey_cutover_predicate_boundary() {
let completed = 1_000u64;
let entry = entry_pending_cutover(completed);
assert!(entry.pending_new_session().is_some());
assert!(!entry.has_rekey_in_progress());
assert!(entry.is_rekey_initiator());
// Not yet eligible one ms before the delay elapses.
let just_before = completed + CUTOVER_DELAY_MS - 1;
assert!(
just_before.saturating_sub(entry.rekey_completed_ms()) < CUTOVER_DELAY_MS,
"cutover must not fire before the liveness delay"
);
// Eligible exactly at the delay.
let at = completed + CUTOVER_DELAY_MS;
assert!(
at.saturating_sub(entry.rekey_completed_ms()) >= CUTOVER_DELAY_MS,
"cutover fires once the liveness delay has elapsed"
);
}
// Rekey-trigger threshold: elapsed time (with symmetric jitter applied)
// OR the send counter crossing its configured bound.
#[test]
fn rekey_trigger_threshold_arithmetic() {
let after_secs = 100u64;
let after_messages = 1_000u64;
// Jitter is always within [-REKEY_JITTER_SECS, +REKEY_JITTER_SECS].
let (_s, recv) = xk_pair(1, 2);
let entry = entry_with_current(recv);
let jitter = entry.rekey_jitter_secs();
assert!(
jitter.abs() <= REKEY_JITTER_SECS,
"jitter within configured bound"
);
// Effective time threshold applies the symmetric jitter.
let effective_after = after_secs.saturating_add_signed(jitter);
// Reproduce the policy's OR predicate directly.
let triggers =
|elapsed: u64, counter: u64| elapsed >= effective_after || counter >= after_messages;
// Time arm: fires at/after the effective threshold, not below it.
assert!(!triggers(effective_after - 1, 0), "below time threshold");
assert!(triggers(effective_after, 0), "at time threshold");
// Counter arm: fires independently of elapsed time.
assert!(!triggers(0, after_messages - 1), "below counter threshold");
assert!(triggers(0, after_messages), "at counter threshold");
}
// Dampening boundary: within `dampening_ms` of the peer's rekey msg1, local
// initiation is suppressed; at/after the window it is not.
#[test]
fn rekey_dampening_boundary() {
let (_s, recv) = xk_pair(1, 2);
let mut entry = entry_with_current(recv);
const DAMP_MS: u64 = 30_000;
// No peer rekey recorded → never dampened.
assert!(!entry.is_rekey_dampened(50_000, DAMP_MS));
entry.record_peer_rekey(10_000);
assert!(
entry.is_rekey_dampened(10_000 + DAMP_MS - 1, DAMP_MS),
"dampened within the window"
);
assert!(
!entry.is_rekey_dampened(10_000 + DAMP_MS, DAMP_MS),
"not dampened once the window has elapsed"
);
}
// Epoch-reaction: a frame authenticating against `pending` while a msg3
// retransmission is retained confirms the peer on the new epoch (clears the
// msg3 payload) and then promotes.
#[test]
fn epoch_reaction_pending_confirms_then_promotes() {
let (mut p_send, p_recv) = xk_pair(3, 4);
let (_cur_send, cur_recv) = xk_pair(1, 2);
let mut entry = entry_with_current(cur_recv);
let k_before = entry.current_k_bit();
entry.set_pending_session(p_recv);
entry.set_rekey_msg3_payload(vec![0xAB; 8], 5_000);
assert!(entry.rekey_msg3_payload().is_some());
let (ct, counter, hdr) = seal(&mut p_send, b"new-epoch", !k_before);
let (_pt, slot) = entry
.fsp_trial_decrypt(&ct, counter, &hdr, !k_before, 2_000)
.expect("pending frame decrypts");
assert_eq!(slot, EpochSlot::Pending);
// Reaction order: confirm (while pending still held) then promote.
entry.confirm_peer_new_epoch();
assert!(entry.rekey_msg3_payload().is_none());
entry.handle_peer_kbit_flip(2_000);
assert!(entry.pending_new_session().is_none());
assert_ne!(entry.current_k_bit(), k_before);
}
// Epoch-reaction: as the initiator that already cut over on its own timer
// (msg3 retained, no pending), a frame authenticating against `current`
// confirms the responder reached the new epoch.
#[test]
fn epoch_reaction_current_confirms_responder() {
let (mut cur_send, cur_recv) = xk_pair(1, 2);
let mut entry = entry_with_current(cur_recv);
entry.set_rekey_msg3_payload(vec![0xCD; 8], 5_000);
assert!(entry.pending_new_session().is_none());
assert!(entry.rekey_msg3_payload().is_some());
let (ct, counter, hdr) = seal(&mut cur_send, b"steady", false);
let (_pt, slot) = entry
.fsp_trial_decrypt(&ct, counter, &hdr, false, 2_000)
.expect("current frame decrypts");
assert_eq!(slot, EpochSlot::Current);
// The Current-with-retained-msg3-and-no-pending arm confirms.
entry.confirm_peer_new_epoch();
assert!(entry.rekey_msg3_payload().is_none());
}
}

View File

@@ -1,618 +0,0 @@
//! FSP Wire Format Parsing and Serialization
//!
//! Defines the FIPS session-layer wire format (FSP) for packet dispatch.
//! All FSP messages begin with a 4-byte common prefix followed by phase-specific
//! fields. Encrypted messages use a 12-byte cleartext header as AAD for AEAD,
//! and a 6-byte encrypted inner header containing timestamps and message type.
//!
//! ## Common Prefix (4 bytes)
//!
//! ```text
//! [ver+phase:1][flags:1][payload_len:2 LE]
//! ```
//!
//! ## DataPacket Port Multiplexing
//!
//! DataPacket (msg_type 0x10) payloads inside the AEAD envelope carry a 4-byte
//! port header for service dispatch:
//!
//! ```text
//! [src_port:2 LE][dst_port:2 LE][service payload...]
//! ```
//!
//! Port 256 (0x100) = IPv6 shim with header compression.
//!
//! ## Message Classes
//!
//! | Phase | U Flag | Type | Description |
//! |-------|--------|------------------|-----------------------------------|
//! | 0x0 | 0 | Encrypted | Post-handshake encrypted data |
//! | 0x0 | 1 | Plaintext error | CoordsRequired, PathBroken |
//! | 0x1 | - | Handshake msg1 | SessionSetup (Noise XK msg1) |
//! | 0x2 | - | Handshake msg2 | SessionAck (Noise XK msg2) |
//! | 0x3 | - | Handshake msg3 | SessionMsg3 (Noise XK msg3) |
use crate::protocol::{ProtocolError, decode_optional_coords};
use crate::tree::TreeCoordinate;
// ============================================================================
// Constants
// ============================================================================
/// FSP protocol version (4 high bits of byte 0).
pub const FSP_VERSION: u8 = 0;
/// Phase value for established (encrypted or plaintext error) messages.
pub const FSP_PHASE_ESTABLISHED: u8 = 0x0;
/// Phase value for SessionSetup (Noise IK message 1).
pub const FSP_PHASE_MSG1: u8 = 0x1;
/// Phase value for SessionAck (Noise handshake message 2).
pub const FSP_PHASE_MSG2: u8 = 0x2;
/// Phase value for XK message 3 (initiator's encrypted static).
pub const FSP_PHASE_MSG3: u8 = 0x3;
/// Size of the common packet prefix (all FSP message types).
pub const FSP_COMMON_PREFIX_SIZE: usize = 4;
/// Size of the full encrypted message header (prefix + counter).
pub const FSP_HEADER_SIZE: usize = 12;
/// Size of the encrypted inner header (timestamp + msg_type + inner_flags).
pub const FSP_INNER_HEADER_SIZE: usize = 6;
/// AEAD authentication tag size (ChaCha20-Poly1305).
const TAG_SIZE: usize = 16;
/// Minimum size for an encrypted FSP message: header + tag (no plaintext).
pub const FSP_ENCRYPTED_MIN_SIZE: usize = FSP_HEADER_SIZE + TAG_SIZE; // 28 bytes
// FSP DataPacket port header constants.
/// Size of the FSP DataPacket port header (src_port + dst_port).
pub const FSP_PORT_HEADER_SIZE: usize = 4;
/// FSP port: IPv6 shim service.
pub const FSP_PORT_IPV6_SHIM: u16 = 256;
// Cleartext flag bit constants (byte 1 of common prefix, phase 0x0 only).
/// Coords Present — source and destination coordinates follow the header.
pub const FSP_FLAG_CP: u8 = 0x01;
/// Key Epoch — selects active key during rekeying.
#[allow(dead_code)]
pub const FSP_FLAG_K: u8 = 0x02;
/// Unencrypted — payload is plaintext (error signals).
pub const FSP_FLAG_U: u8 = 0x04;
// Inner flag bit constants (byte 5 of decrypted inner header).
/// Spin bit for end-to-end RTT measurement (inside AEAD).
#[allow(dead_code)]
pub const FSP_INNER_FLAG_SP: u8 = 0x01;
// ============================================================================
// Common Prefix
// ============================================================================
/// Parsed FSP common packet prefix (first 4 bytes of every FSP message).
///
/// Wire format:
/// ```text
/// [ver(4bits)+phase(4bits)][flags:1][payload_len:2 LE]
/// ```
#[derive(Clone, Debug)]
pub struct FspCommonPrefix {
/// Protocol version (high nibble of byte 0).
#[cfg_attr(not(test), allow(dead_code))]
pub version: u8,
/// Session lifecycle phase (low nibble of byte 0).
pub phase: u8,
/// Per-message signal flags.
pub flags: u8,
/// Length of payload following the phase-specific header.
#[cfg_attr(not(test), allow(dead_code))]
pub payload_len: u16,
}
impl FspCommonPrefix {
/// Parse a common prefix from the first 4 bytes of FSP message data.
pub fn parse(data: &[u8]) -> Option<Self> {
if data.len() < FSP_COMMON_PREFIX_SIZE {
return None;
}
let version = data[0] >> 4;
let phase = data[0] & 0x0F;
let flags = data[1];
let payload_len = u16::from_le_bytes([data[2], data[3]]);
Some(Self {
version,
phase,
flags,
payload_len,
})
}
/// Check if the Unencrypted flag is set.
pub fn is_unencrypted(&self) -> bool {
self.flags & FSP_FLAG_U != 0
}
/// Check if the Coords Present flag is set.
pub fn has_coords(&self) -> bool {
self.flags & FSP_FLAG_CP != 0
}
/// Encode the ver+phase byte.
fn ver_phase_byte(version: u8, phase: u8) -> u8 {
(version << 4) | (phase & 0x0F)
}
}
// ============================================================================
// Encrypted Message Header
// ============================================================================
/// Parsed FSP encrypted message header (phase 0x0, U flag clear).
///
/// Wire format (12 bytes):
/// ```text
/// [ver+phase:1][flags:1][payload_len:2 LE][counter:8 LE]
/// ```
///
/// The full 12-byte header is used as AAD for the AEAD construction.
/// No receiver_idx — unlike FMP, FSP is end-to-end (dispatched by src_addr
/// from the SessionDatagram envelope, not by index).
#[derive(Clone, Debug)]
pub struct FspEncryptedHeader {
/// Per-message flags (CP, K).
pub flags: u8,
/// Length of encrypted payload (excluding AEAD tag).
#[cfg_attr(not(test), allow(dead_code))]
pub payload_len: u16,
/// Monotonic counter used as AEAD nonce.
pub counter: u64,
/// Raw 12-byte header for use as AEAD AAD.
pub header_bytes: [u8; FSP_HEADER_SIZE],
}
impl FspEncryptedHeader {
/// Parse an encrypted message header from FSP message data.
///
/// Returns None if the data is too short or has wrong version/phase,
/// or if the U flag is set (plaintext messages use a different path).
pub fn parse(data: &[u8]) -> Option<Self> {
if data.len() < FSP_ENCRYPTED_MIN_SIZE {
return None;
}
let version = data[0] >> 4;
let phase = data[0] & 0x0F;
if version != FSP_VERSION || phase != FSP_PHASE_ESTABLISHED {
return None;
}
let flags = data[1];
// U flag means plaintext — not an encrypted message
if flags & FSP_FLAG_U != 0 {
return None;
}
let payload_len = u16::from_le_bytes([data[2], data[3]]);
let counter = u64::from_le_bytes([
data[4], data[5], data[6], data[7], data[8], data[9], data[10], data[11],
]);
let mut header_bytes = [0u8; FSP_HEADER_SIZE];
header_bytes.copy_from_slice(&data[..FSP_HEADER_SIZE]);
Some(Self {
flags,
payload_len,
counter,
header_bytes,
})
}
/// Check if the Coords Present flag is set.
pub fn has_coords(&self) -> bool {
self.flags & FSP_FLAG_CP != 0
}
/// Offset where ciphertext (or coords if CP) begins in the original data.
#[cfg_attr(not(test), allow(dead_code))]
pub fn data_offset(&self) -> usize {
FSP_HEADER_SIZE
}
}
// ============================================================================
// Serialization Helpers
// ============================================================================
/// Build the 12-byte cleartext header for an encrypted FSP message.
///
/// Returns the header bytes for use as AEAD AAD.
pub fn build_fsp_header(counter: u64, flags: u8, payload_len: u16) -> [u8; FSP_HEADER_SIZE] {
let mut header = [0u8; FSP_HEADER_SIZE];
header[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, FSP_PHASE_ESTABLISHED);
header[1] = flags;
header[2..4].copy_from_slice(&payload_len.to_le_bytes());
header[4..12].copy_from_slice(&counter.to_le_bytes());
header
}
/// Assemble a wire-format encrypted FSP message.
///
/// Format: `[header:12][ciphertext+tag]`
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_encrypted(header: &[u8; FSP_HEADER_SIZE], ciphertext: &[u8]) -> Vec<u8> {
let mut packet = Vec::with_capacity(FSP_HEADER_SIZE + ciphertext.len());
packet.extend_from_slice(header);
packet.extend_from_slice(ciphertext);
packet
}
/// Build a 4-byte common prefix for a handshake message.
///
/// `phase` should be `FSP_PHASE_MSG1`, `FSP_PHASE_MSG2`, or `FSP_PHASE_MSG3`.
/// Flags are zero during handshake.
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_handshake_prefix(phase: u8, payload_len: u16) -> [u8; FSP_COMMON_PREFIX_SIZE] {
let mut prefix = [0u8; FSP_COMMON_PREFIX_SIZE];
prefix[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, phase);
prefix[1] = 0x00; // flags must be zero during handshake
prefix[2..4].copy_from_slice(&payload_len.to_le_bytes());
prefix
}
/// Build a 4-byte common prefix for a plaintext error signal.
///
/// Sets phase 0x0 and U flag.
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_error_prefix(payload_len: u16) -> [u8; FSP_COMMON_PREFIX_SIZE] {
let mut prefix = [0u8; FSP_COMMON_PREFIX_SIZE];
prefix[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, FSP_PHASE_ESTABLISHED);
prefix[1] = FSP_FLAG_U;
prefix[2..4].copy_from_slice(&payload_len.to_le_bytes());
prefix
}
// ============================================================================
// Inner Header Helpers
// ============================================================================
/// Prepend the 6-byte FSP inner header to a message payload.
///
/// Inner header: `[timestamp:4 LE][msg_type:1][inner_flags:1]`
///
/// The caller provides the message-type-specific payload (e.g., application
/// data for msg_type 0x10, report fields for SenderReport). This function
/// prepends the inner header.
pub fn fsp_prepend_inner_header(
timestamp_ms: u32,
msg_type: u8,
inner_flags: u8,
payload: &[u8],
) -> Vec<u8> {
let mut buf = Vec::with_capacity(FSP_INNER_HEADER_SIZE + payload.len());
buf.extend_from_slice(&timestamp_ms.to_le_bytes());
buf.push(msg_type);
buf.push(inner_flags);
buf.extend_from_slice(payload);
buf
}
/// Strip the 6-byte FSP inner header from a decrypted payload.
///
/// Returns `(timestamp, msg_type, inner_flags, &rest)` or None if too short.
pub fn fsp_strip_inner_header(plaintext: &[u8]) -> Option<(u32, u8, u8, &[u8])> {
if plaintext.len() < FSP_INNER_HEADER_SIZE {
return None;
}
let timestamp = u32::from_le_bytes([plaintext[0], plaintext[1], plaintext[2], plaintext[3]]);
let msg_type = plaintext[4];
let inner_flags = plaintext[5];
Some((
timestamp,
msg_type,
inner_flags,
&plaintext[FSP_INNER_HEADER_SIZE..],
))
}
// ============================================================================
// Coordinate Parsing (for transit nodes and receive path)
// ============================================================================
/// Parse source and destination coordinates from the cleartext section
/// of an encrypted FSP message when the CP flag is set.
///
/// Coordinates appear between the 12-byte header and the ciphertext:
/// `[src_coords_count:2 LE][src_coords:16×n][dest_coords_count:2 LE][dest_coords:16×m]`
///
/// Returns `(src_coords, dest_coords, bytes_consumed)`.
pub fn parse_encrypted_coords(
data: &[u8],
) -> Result<(Option<TreeCoordinate>, Option<TreeCoordinate>, usize), ProtocolError> {
let (src_coords, src_consumed) = decode_optional_coords(data)?;
let (dest_coords, dest_consumed) = decode_optional_coords(&data[src_consumed..])?;
Ok((src_coords, dest_coords, src_consumed + dest_consumed))
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
// ===== Size Constant Tests =====
#[test]
fn test_wire_sizes() {
assert_eq!(FSP_COMMON_PREFIX_SIZE, 4);
assert_eq!(FSP_HEADER_SIZE, 12);
assert_eq!(FSP_INNER_HEADER_SIZE, 6);
assert_eq!(FSP_ENCRYPTED_MIN_SIZE, 28); // 12 + 16
}
// ===== Common Prefix Tests =====
#[test]
fn test_common_prefix_parse_established() {
let data = [0x00, 0x01, 0x40, 0x00]; // ver=0, phase=0, flags=CP, payload_len=64
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.version, 0);
assert_eq!(prefix.phase, FSP_PHASE_ESTABLISHED);
assert_eq!(prefix.flags, FSP_FLAG_CP);
assert_eq!(prefix.payload_len, 64);
assert!(prefix.has_coords());
assert!(!prefix.is_unencrypted());
}
#[test]
fn test_common_prefix_parse_handshake() {
let data = [0x01, 0x00, 0x50, 0x00]; // ver=0, phase=1, flags=0, payload_len=80
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.version, 0);
assert_eq!(prefix.phase, FSP_PHASE_MSG1);
assert_eq!(prefix.flags, 0);
assert_eq!(prefix.payload_len, 80);
}
#[test]
fn test_common_prefix_parse_error_signal() {
let data = [0x00, FSP_FLAG_U, 0x22, 0x00]; // ver=0, phase=0, U flag, payload_len=34
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.phase, FSP_PHASE_ESTABLISHED);
assert!(prefix.is_unencrypted());
assert_eq!(prefix.payload_len, 34);
}
#[test]
fn test_common_prefix_too_short() {
assert!(FspCommonPrefix::parse(&[0, 0, 0]).is_none());
}
// ===== Encrypted Header Tests =====
#[test]
fn test_encrypted_header_parse() {
let counter = 42u64;
let flags = FSP_FLAG_CP;
let payload_len = 100u16;
let header = build_fsp_header(counter, flags, payload_len);
// Build a minimal packet: header + 16 bytes of fake ciphertext (tag)
let mut packet = Vec::from(header);
packet.extend_from_slice(&[0xaa; TAG_SIZE]);
let parsed = FspEncryptedHeader::parse(&packet).unwrap();
assert_eq!(parsed.counter, 42);
assert_eq!(parsed.flags, FSP_FLAG_CP);
assert_eq!(parsed.payload_len, 100);
assert!(parsed.has_coords());
assert_eq!(parsed.header_bytes, header);
assert_eq!(parsed.data_offset(), FSP_HEADER_SIZE);
}
#[test]
fn test_encrypted_header_too_short() {
let packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE - 1];
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_wrong_phase() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[0] = 0x01; // phase 1 (msg1), not established
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_wrong_version() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[0] = 0x10; // version 1, phase 0
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_u_flag_rejected() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[1] = FSP_FLAG_U; // U flag set → not encrypted
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
// ===== Build Header Tests =====
#[test]
fn test_build_fsp_header() {
let header = build_fsp_header(1000, FSP_FLAG_CP, 200);
assert_eq!(header[0], 0x00); // ver=0, phase=0
assert_eq!(header[1], FSP_FLAG_CP);
assert_eq!(u16::from_le_bytes([header[2], header[3]]), 200);
assert_eq!(
u64::from_le_bytes([
header[4], header[5], header[6], header[7], header[8], header[9], header[10],
header[11],
]),
1000
);
}
#[test]
fn test_build_fsp_encrypted() {
let header = build_fsp_header(0, 0, 10);
let ciphertext = vec![0xCC; 26]; // 10 payload + 16 tag
let packet = build_fsp_encrypted(&header, &ciphertext);
assert_eq!(packet.len(), FSP_HEADER_SIZE + 26);
assert_eq!(&packet[..FSP_HEADER_SIZE], &header);
assert_eq!(&packet[FSP_HEADER_SIZE..], &ciphertext[..]);
}
// ===== Handshake Prefix Tests =====
#[test]
fn test_build_fsp_handshake_prefix_msg1() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG1, 100);
assert_eq!(prefix[0], 0x01); // ver=0, phase=1
assert_eq!(prefix[1], 0x00); // flags zero
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 100);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert_eq!(parsed.phase, FSP_PHASE_MSG1);
}
#[test]
fn test_build_fsp_handshake_prefix_msg2() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG2, 50);
assert_eq!(prefix[0], 0x02); // ver=0, phase=2
assert_eq!(prefix[1], 0x00);
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 50);
}
#[test]
fn test_build_fsp_handshake_prefix_msg3() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG3, 73);
assert_eq!(prefix[0], 0x03); // ver=0, phase=3
assert_eq!(prefix[1], 0x00); // flags zero
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 73);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert_eq!(parsed.phase, FSP_PHASE_MSG3);
}
// ===== Error Prefix Tests =====
#[test]
fn test_build_fsp_error_prefix() {
let prefix = build_fsp_error_prefix(34);
assert_eq!(prefix[0], 0x00); // ver=0, phase=0
assert_eq!(prefix[1], FSP_FLAG_U);
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 34);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert!(parsed.is_unencrypted());
assert_eq!(parsed.phase, FSP_PHASE_ESTABLISHED);
}
// ===== Inner Header Tests =====
#[test]
fn test_inner_header_prepend_strip() {
let timestamp: u32 = 12345;
let msg_type: u8 = 0x10;
let inner_flags: u8 = 0x01; // SP bit
let payload = vec![0xAA, 0xBB, 0xCC];
let with_header = fsp_prepend_inner_header(timestamp, msg_type, inner_flags, &payload);
assert_eq!(with_header.len(), FSP_INNER_HEADER_SIZE + 3);
let (ts, mt, flags, rest) = fsp_strip_inner_header(&with_header).unwrap();
assert_eq!(ts, 12345);
assert_eq!(mt, 0x10);
assert_eq!(flags, 0x01);
assert_eq!(rest, &payload[..]);
}
#[test]
fn test_inner_header_empty_payload() {
let with_header = fsp_prepend_inner_header(0, 0x13, 0, &[]);
assert_eq!(with_header.len(), FSP_INNER_HEADER_SIZE);
let (ts, mt, flags, rest) = fsp_strip_inner_header(&with_header).unwrap();
assert_eq!(ts, 0);
assert_eq!(mt, 0x13);
assert_eq!(flags, 0);
assert!(rest.is_empty());
}
#[test]
fn test_inner_header_too_short() {
assert!(fsp_strip_inner_header(&[0, 0, 0, 0, 0]).is_none()); // needs 6 bytes
assert!(fsp_strip_inner_header(&[]).is_none());
}
// ===== Flag Constants Tests =====
#[test]
fn test_flag_bits_distinct() {
// Cleartext flags don't overlap
assert_eq!(FSP_FLAG_CP & FSP_FLAG_K, 0);
assert_eq!(FSP_FLAG_CP & FSP_FLAG_U, 0);
assert_eq!(FSP_FLAG_K & FSP_FLAG_U, 0);
}
#[test]
fn test_header_roundtrip() {
let counter = 0xDEADBEEF_12345678u64;
let flags = FSP_FLAG_CP | FSP_FLAG_K;
let payload_len = 1234u16;
let header = build_fsp_header(counter, flags, payload_len);
let ciphertext = vec![0xFF; payload_len as usize + TAG_SIZE];
let packet = build_fsp_encrypted(&header, &ciphertext);
let parsed = FspEncryptedHeader::parse(&packet).unwrap();
assert_eq!(parsed.counter, counter);
assert_eq!(parsed.flags, flags);
assert_eq!(parsed.payload_len, payload_len);
assert!(parsed.has_coords());
assert_eq!(parsed.header_bytes, header);
}
#[test]
fn test_all_message_types_through_prefix() {
// Encrypted (phase 0, no U)
let prefix = FspCommonPrefix::parse(&[0x00, 0x00, 0x10, 0x00]).unwrap();
assert_eq!(prefix.phase, 0);
assert!(!prefix.is_unencrypted());
// Error signal (phase 0, U set)
let prefix = FspCommonPrefix::parse(&[0x00, FSP_FLAG_U, 0x22, 0x00]).unwrap();
assert_eq!(prefix.phase, 0);
assert!(prefix.is_unencrypted());
// SessionSetup (phase 1)
let prefix = FspCommonPrefix::parse(&[0x01, 0x00, 0x50, 0x00]).unwrap();
assert_eq!(prefix.phase, 1);
// SessionAck (phase 2)
let prefix = FspCommonPrefix::parse(&[0x02, 0x00, 0x21, 0x00]).unwrap();
assert_eq!(prefix.phase, 2);
// SessionMsg3 (phase 3)
let prefix = FspCommonPrefix::parse(&[0x03, 0x00, 0x49, 0x00]).unwrap();
assert_eq!(prefix.phase, 3);
}
}

View File

@@ -238,7 +238,7 @@ pub struct ForwardingStatsSnapshot {
}
#[derive(Clone, Debug, Default, Serialize)]
pub struct DiscoveryStatsSnapshot {
pub struct LookupStatsSnapshot {
pub req_received: u64,
pub req_decode_error: u64,
pub req_duplicate: u64,

View File

@@ -2,7 +2,7 @@ use super::*;
use crate::ReceivedPacket;
use crate::node::acl::PeerAclReloader;
use crate::node::reloadable::HostMapReloadable;
use crate::node::wire::{build_msg1, build_msg2};
use crate::proto::fmp::wire::{build_msg1, build_msg2};
use crate::upper::hosts::HostMap;
use crate::utils::index::SessionIndex;
use std::path::PathBuf;

View File

@@ -455,9 +455,9 @@ async fn test_bloom_filter_split_horizon() {
/// counted once, not the double-count fingerprint.
#[test]
fn compute_mesh_size_counts_each_peer_filter_once() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::tree::ParentDeclaration;
use crate::proto::bloom::BloomFilter;
use crate::proto::stp::ParentDeclaration;
let mut node = make_node();
let my_addr = *node.tree_state().my_node_addr();
@@ -498,8 +498,8 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
// Seed parent ancestry first so recompute_coords can extend it and
// flip is_root() to false; child ancestry is for completeness.
let parent_ancestry = crate::tree::TreeCoordinate::root_with_meta(parent_addr, 1, 1);
let child_ancestry = crate::tree::TreeCoordinate::root_with_meta(child_addr, 1, 1);
let parent_ancestry = crate::proto::stp::TreeCoordinate::root_with_meta(parent_addr, 1, 1);
let child_ancestry = crate::proto::stp::TreeCoordinate::root_with_meta(child_addr, 1, 1);
// Inject the stale-cache scenario: peer_declaration(P) still names
// US (M) as P's parent (the pre-switch advert that the cache hasn't
// refreshed yet). Q is a legitimate child also naming M as parent.
@@ -513,7 +513,7 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
.update_peer(child_decl, child_ancestry);
// Switch our parent to P and recompute coords so root flips off self.
node.tree_state_mut().set_parent(parent_addr, 2, 1);
node.tree_state_mut().set_parent(parent_addr, 2, 1, 1);
node.tree_state_mut().recompute_coords();
assert!(
!node.tree_state().is_root(),
@@ -550,8 +550,8 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
/// `estimated_mesh_size` carries.
#[test]
fn compute_mesh_size_unions_overlapping_filters() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
let mut node = make_node();
@@ -632,8 +632,8 @@ fn compute_mesh_size_unions_overlapping_filters() {
/// removes the parent, and asserts the estimate does not collapse.
#[test]
fn compute_mesh_size_stable_across_parent_drop_with_cross_link() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
let mut node = make_node();

View File

@@ -7,9 +7,9 @@
//! bloom.rs.
use super::*;
use crate::bloom::{BloomFilter, DEFAULT_FILTER_SIZE_BITS, DEFAULT_HASH_COUNT};
use crate::peer::ActivePeer;
use crate::protocol::FilterAnnounce;
use crate::proto::bloom::FilterAnnounce;
use crate::proto::bloom::{BloomFilter, DEFAULT_FILTER_SIZE_BITS, DEFAULT_HASH_COUNT};
/// Inject a synthetic active peer into the node with a known NodeAddr.
/// Returns the peer's NodeAddr.

View File

@@ -3,7 +3,7 @@
use super::*;
use crate::EstablishedTraversal;
use crate::config::{TransportInstances, UdpConfig};
use crate::node::wire::{PHASE_MSG1, PHASE_MSG2};
use crate::proto::fmp::wire::{PHASE_MSG1, PHASE_MSG2};
use crate::transport::udp::UdpTransport;
use crate::utils::index::IndexAllocator;
use std::collections::HashMap;
@@ -24,17 +24,17 @@ async fn test_adopted_udp_traversal_completes_handshake() {
let (packet_tx_a, packet_rx_a) = packet_channel(64);
let (packet_tx_b, packet_rx_b) = packet_channel(64);
node_a.packet_tx = Some(packet_tx_a.clone());
node_a.supervisor.packet_tx = Some(packet_tx_a.clone());
node_a.packet_rx = Some(packet_rx_a);
node_a.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
let mut transport_b = UdpTransport::new(transport_id_b, None, udp_config, packet_tx_b.clone());
transport_b.start_async().await.unwrap();
let addr_b = transport_b.local_addr().unwrap();
node_b.packet_tx = Some(packet_tx_b.clone());
node_b.supervisor.packet_tx = Some(packet_tx_b.clone());
node_b.packet_rx = Some(packet_rx_b);
node_b.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
node_b
.transports
.insert(transport_id_b, TransportHandle::Udp(transport_b));
@@ -91,9 +91,9 @@ async fn test_adopted_udp_traversal_completes_handshake() {
async fn test_failed_adopted_traversal_cleans_up_transport() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
node.index_allocator = IndexAllocator::with_max_attempts(0);
let peer = make_node();
@@ -121,9 +121,9 @@ async fn test_failed_adopted_traversal_cleans_up_transport() {
async fn test_adopted_traversal_skips_already_connected_peer() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let transport_id = TransportId::new(1);
let link_id = LinkId::new(1);
@@ -180,17 +180,17 @@ async fn test_third_peer_can_handshake_via_adopted_transport_socket() {
let (packet_tx_b, packet_rx_b) = packet_channel(64);
let (packet_tx_c, packet_rx_c) = packet_channel(64);
node_a.packet_tx = Some(packet_tx_a.clone());
node_a.supervisor.packet_tx = Some(packet_tx_a.clone());
node_a.packet_rx = Some(packet_rx_a);
node_a.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
node_b.packet_tx = Some(packet_tx_b.clone());
node_b.supervisor.packet_tx = Some(packet_tx_b.clone());
node_b.packet_rx = Some(packet_rx_b);
node_b.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
node_c.packet_tx = Some(packet_tx_c.clone());
node_c.supervisor.packet_tx = Some(packet_tx_c.clone());
node_c.packet_rx = Some(packet_rx_c);
node_c.state = NodeState::Running;
node_c.supervisor.state = NodeState::Running;
let mut transport_a = UdpTransport::new(transport_id_a, None, udp_config.clone(), packet_tx_a);
transport_a.start_async().await.unwrap();
@@ -300,9 +300,9 @@ async fn test_adopted_udp_inherits_mtu_from_single_primary_config() {
let mut node = make_node_with(config);
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let peer = make_node();
let adopted_socket = std::net::UdpSocket::bind("127.0.0.1:0").unwrap();
@@ -350,9 +350,9 @@ async fn test_adopted_udp_inherits_mtu_from_named_primary_config() {
let mut node = make_node_with(config);
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let peer = make_node();
let adopted_socket = std::net::UdpSocket::bind("127.0.0.1:0").unwrap();

View File

@@ -1,6 +1,6 @@
//! Tests for the consecutive-decrypt-failure threshold force-removal path.
//!
//! Covers `Node::handle_decrypt_failure` (in `node/handlers/encrypted.rs`),
//! Covers `Node::handle_decrypt_failure` (in `node/dataplane/encrypted.rs`),
//! which increments `ActivePeer::increment_decrypt_failures` on each AEAD
//! verification failure and force-removes the peer once
//! `DECRYPT_FAILURE_THRESHOLD` consecutive failures are observed. The
@@ -18,7 +18,7 @@ use super::*;
/// the full `peers_by_index` cleanup path (not just the bare `peers` table).
#[test]
fn test_decrypt_failure_threshold_removes_peer() {
// Threshold constant in node/handlers/encrypted.rs (kept in sync with
// Threshold constant in node/dataplane/encrypted.rs (kept in sync with
// production code; see DECRYPT_FAILURE_THRESHOLD).
const THRESHOLD: u32 = 20;

View File

@@ -6,7 +6,7 @@
use super::spanning_tree::*;
use super::*;
use crate::protocol::{Disconnect, DisconnectReason};
use crate::proto::fmp::{Disconnect, DisconnectReason};
/// 3-node chain: middle node disconnects one peer.
///
@@ -295,7 +295,7 @@ async fn test_disconnect_clears_session() {
);
// Node 0 sends Disconnect to node 1.
let disconnect = crate::protocol::Disconnect::new(DisconnectReason::Shutdown);
let disconnect = crate::proto::fmp::Disconnect::new(DisconnectReason::Shutdown);
nodes[0]
.node
.send_encrypted_link_message(&node1_addr, &disconnect.encode())

View File

@@ -5,9 +5,8 @@
//! response routing.
use super::*;
use crate::node::RecentRequest;
use crate::protocol::{LookupRequest, LookupResponse};
use crate::tree::TreeCoordinate;
use crate::proto::lookup::{LookupRequest, LookupResponse, RecentRequest};
use crate::proto::stp::TreeCoordinate;
use spanning_tree::{
cleanup_nodes, generate_random_edges, lock_large_network_test, process_available_packets,
run_tree_test, verify_tree_convergence,
@@ -23,7 +22,7 @@ async fn test_request_decode_error() {
let from = make_node_addr(0xAA);
// Too-short payload: should log error and return without panic
node.handle_lookup_request(&from, &[0x00; 5]).await;
assert!(node.recent_requests.is_empty());
assert!(node.lookup.recent_requests.is_empty());
}
#[tokio::test]
@@ -39,11 +38,11 @@ async fn test_request_dedup() {
// First request: accepted
node.handle_lookup_request(&from, payload).await;
assert_eq!(node.recent_requests.len(), 1);
assert_eq!(node.lookup.recent_requests.len(), 1);
// Duplicate request: dropped
node.handle_lookup_request(&from, payload).await;
assert_eq!(node.recent_requests.len(), 1);
assert_eq!(node.lookup.recent_requests.len(), 1);
}
#[tokio::test]
@@ -61,7 +60,7 @@ async fn test_request_target_is_self() {
// Should succeed without panic (response send will fail silently
// since we have no peers to route toward origin)
node.handle_lookup_request(&from, payload).await;
assert!(node.recent_requests.contains_key(&777));
assert!(node.lookup.recent_requests.contains_key(&777));
}
#[tokio::test]
@@ -77,7 +76,7 @@ async fn test_request_ttl_zero_not_forwarded() {
node.handle_lookup_request(&from, payload).await;
// Request recorded, but not forwarded (TTL=0, and no peers anyway)
assert!(node.recent_requests.contains_key(&666));
assert!(node.lookup.recent_requests.contains_key(&666));
}
// ============================================================================
@@ -115,7 +114,7 @@ async fn test_response_originator_caches_route() {
let payload = &response.encode()[1..]; // skip msg_type
// No entry in recent_requests for 555 → we're the originator
assert!(!node.recent_requests.contains_key(&555));
assert!(!node.lookup.recent_requests.contains_key(&555));
node.handle_lookup_response(&from, payload).await;
@@ -149,7 +148,8 @@ async fn test_response_transit_needs_recent_request() {
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
node.recent_requests
node.lookup
.recent_requests
.insert(444, RecentRequest::new(make_node_addr(0xDD), now_ms));
// Handle response — should try to reverse-path forward to 0xDD
@@ -319,14 +319,16 @@ async fn test_recent_request_expiry() {
.as_millis() as u64;
// Insert an old request (11 seconds ago)
node.recent_requests
node.lookup
.recent_requests
.insert(123, RecentRequest::new(make_node_addr(1), now_ms - 11_000));
// Insert a recent request
node.recent_requests
node.lookup
.recent_requests
.insert(456, RecentRequest::new(make_node_addr(2), now_ms));
assert_eq!(node.recent_requests.len(), 2);
assert_eq!(node.lookup.recent_requests.len(), 2);
// Trigger purge via a new lookup request
let target = make_node_addr(0xBB);
@@ -338,9 +340,9 @@ async fn test_recent_request_expiry() {
.await;
// Old entry (123) should be purged, recent entry (456) and new entry (789) kept
assert!(!node.recent_requests.contains_key(&123));
assert!(node.recent_requests.contains_key(&456));
assert!(node.recent_requests.contains_key(&789));
assert!(!node.lookup.recent_requests.contains_key(&123));
assert!(node.lookup.recent_requests.contains_key(&456));
assert!(node.lookup.recent_requests.contains_key(&789));
}
// ============================================================================
@@ -379,7 +381,7 @@ async fn test_request_forwarding_two_node() {
// Node1 should have recorded the request
assert!(
nodes[1].node.recent_requests.contains_key(&42),
nodes[1].node.lookup.recent_requests.contains_key(&42),
"Node 1 should have recorded the forwarded request"
);
@@ -447,13 +449,13 @@ async fn test_request_three_node_chain() {
// Node1 should have been a transit node (has the request_id in recent_requests)
assert!(
!nodes[1].node.recent_requests.is_empty(),
!nodes[1].node.lookup.recent_requests.is_empty(),
"Node 1 should have recorded the forwarded request"
);
// Node2 should have received the request (it's the target)
assert!(
!nodes[2].node.recent_requests.is_empty(),
!nodes[2].node.lookup.recent_requests.is_empty(),
"Node 2 should have received the request"
);
@@ -502,7 +504,7 @@ async fn test_request_dedup_convergent_paths() {
// Node2 (the target) must have received the request
assert!(
nodes[2].node.recent_requests.contains_key(&300),
nodes[2].node.lookup.recent_requests.contains_key(&300),
"Node 2 (target) should have received the request"
);
@@ -958,28 +960,28 @@ async fn test_originator_lookup_response_keeps_tighter_path_mtu_lookup() {
/// Pin the iterate-filter-queue contract of `run_open_discovery_sweep`.
///
/// Builds a `Node` with `nostr.policy = Open` and an empty peer list,
/// then injects three cached adverts into a test `NostrDiscovery` and
/// then injects three cached adverts into a test `NostrRendezvous` and
/// asserts the sweep:
/// - queues a retry for an eligible (unknown, not-self) advert,
/// - skips the advert whose author is our own node identity, and
/// - skips the advert whose author is an already-connected peer.
///
/// Uses `NostrDiscovery::new_for_test()` and `insert_advert_for_test()`
/// Uses `NostrRendezvous::new_for_test()` and `insert_advert_for_test()`
/// (both `#[cfg(test)]`-gated test escape hatches in
/// `src/discovery/nostr/runtime.rs`) to populate the cache without
/// requiring live relay subscriptions.
#[tokio::test]
async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
use crate::config::NostrDiscoveryPolicy;
use crate::discovery::nostr::{NostrDiscovery, OverlayEndpointAdvert, OverlayTransportKind};
use crate::config::NostrRendezvousPolicy;
use crate::nostr::{NostrRendezvous, OverlayEndpointAdvert, OverlayTransportKind};
use crate::peer::ActivePeer;
use crate::transport::LinkId;
use std::sync::Arc;
// Build node with open-discovery enabled.
let mut config = crate::Config::new();
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.policy = NostrDiscoveryPolicy::Open;
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.policy = NostrRendezvousPolicy::Open;
let mut node = crate::Node::new(config).unwrap();
// Identity of an already-connected peer; insert into node.peers
@@ -1002,8 +1004,8 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
let self_npub = crate::encode_npub(&node.identity().pubkey());
let self_node_addr = *node.identity().node_addr();
// Build a NostrDiscovery test instance and inject the three adverts.
let bootstrap = Arc::new(NostrDiscovery::new_for_test());
// Build a NostrRendezvous test instance and inject the three adverts.
let bootstrap = Arc::new(NostrRendezvous::new_for_test());
let endpoint = OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: "203.0.113.7:2121".to_string(),
@@ -1014,36 +1016,57 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
.unwrap_or(0);
for npub in [&eligible_npub, &connected_npub, &self_npub] {
let advert =
NostrDiscovery::cached_advert_for_test(npub.clone(), endpoint.clone(), now_secs);
NostrRendezvous::cached_advert_for_test(npub.clone(), endpoint.clone(), now_secs);
bootstrap.insert_advert_for_test(npub.clone(), advert).await;
}
// The sweep now runs through the gate-checked reconciler overlay layer,
// which is inert unless the node is Running/Degraded. In production the
// sweep fires only from the rx_loop tick (which spins after `start()`
// returns Running), so drive the node into `Running` to reflect that.
node.supervisor.state = crate::node::NodeState::Running;
// Run the sweep.
node.run_open_discovery_sweep(&bootstrap, Some(3_600), "test")
.await;
node.run_open_discovery_sweep(&bootstrap, Some(3_600)).await;
// Eligible peer was queued.
assert!(
node.retry_pending.contains_key(&eligible_node_addr),
node.peering
.reconciler
.retry_pending
.contains_key(&eligible_node_addr),
"eligible advert should be queued for retry"
);
let queued = node.retry_pending.get(&eligible_node_addr).unwrap();
let queued = node
.peering
.reconciler
.retry_pending
.get(&eligible_node_addr)
.unwrap();
assert_eq!(queued.peer_config.npub, eligible_npub);
// Connected-peer skip filter held.
assert!(
!node.retry_pending.contains_key(&connected_node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&connected_node_addr),
"advert for already-connected peer must not be queued"
);
// Self skip filter held.
assert!(
!node.retry_pending.contains_key(&self_node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&self_node_addr),
"advert authored by own node must not be queued"
);
// Exactly one queued entry from the three injected adverts.
assert_eq!(node.retry_pending.len(), 1);
assert_eq!(node.peering.reconciler.retry_pending.len(), 1);
}
// ============================================================================
@@ -1053,17 +1076,17 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
/// Pin the per-attempt timeout sequence in `check_pending_lookups`.
///
/// Drives the state machine deterministically through the default
/// `node.discovery.attempt_timeouts_secs = [1, 2, 4, 8]` sequence.
/// `node.lookup.attempt_timeouts_secs = [1, 2, 4, 8]` sequence.
/// Asserts:
/// 1. **Sequence timing** — retries fire at the cumulative deadlines
/// (t=1100ms, 3100ms, 7100ms) and unreachable at t=15100ms.
/// 2. **Fresh `initiate_lookup` per attempt** — `req_initiated` counter
/// increments by exactly one on each retry. The actual `request_id`
/// is generated by `LookupRequest::generate(...)` via `rand::random()`
/// inside `initiate_lookup` and is not stored on the originator
/// side, so per-attempt freshness is verified indirectly: each
/// `req_initiated` increment corresponds to one fresh
/// `LookupRequest::generate` call.
/// is drawn via `rand::rng().random()` at the shell inside
/// `initiate_lookup` and passed to `LookupRequest::new(...)`; it is
/// not stored on the originator side, so per-attempt freshness is
/// verified indirectly: each `req_initiated` increment corresponds
/// to one fresh `initiate_lookup` call.
/// 3. **Final-timeout state transitions** — `pending_lookups` entry is
/// removed, `discovery.resp_timed_out` counter ticks, queued packet
/// is drained, and an ICMPv6 Destination Unreachable frame is
@@ -1075,9 +1098,9 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
/// that `initiate_lookup` ran fresh on each attempt.
#[tokio::test]
async fn test_check_pending_lookups_default_sequence_unreachable() {
use crate::bloom::BloomFilter;
use crate::node::handlers::discovery::PendingLookup;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
use crate::proto::lookup::PendingLookup;
use crate::transport::LinkId;
use std::sync::mpsc;
@@ -1086,14 +1109,14 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Default attempt_timeouts_secs is [1, 2, 4, 8]. Confirm so the test
// cannot silently drift if the default changes.
assert_eq!(
node.config().node.discovery.attempt_timeouts_secs,
node.config().node.lookup.attempt_timeouts_secs,
vec![1, 2, 4, 8],
"test pins the [1,2,4,8] default; update the test if the default changes"
);
// Inject a TUN sender so `send_icmpv6_dest_unreachable` is observable.
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
node.tun_tx = Some(tun_tx);
node.supervisor.tun_tx = Some(tun_tx);
// Build a target identity (the unreachable destination).
let target_identity = Identity::generate();
@@ -1119,7 +1142,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// as its parent. `is_tree_peer` checks both directions — the child
// direction (peer.parent_id == self.node_addr) is what we exercise.
let our_addr = *node.node_addr();
let peer_decl = crate::tree::ParentDeclaration::new(peer_addr, our_addr, 1, 0);
let peer_decl = crate::proto::stp::ParentDeclaration::new(peer_addr, our_addr, 1, 0);
let peer_coords = TreeCoordinate::from_addrs(vec![peer_addr, our_addr]).unwrap();
node.tree_state_mut().update_peer(peer_decl, peer_coords);
assert!(node.is_tree_peer(&peer_addr), "peer must be a tree peer");
@@ -1145,16 +1168,18 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Inject a PendingLookup directly: attempt=1, last_sent_ms=0. This
// mirrors the post-condition of a successful `maybe_initiate_lookup`
// at t=0 without depending on wall-clock-derived `Self::now_ms()`.
node.pending_lookups
node.lookup
.pending_lookups
.insert(target_addr, PendingLookup::new(0));
let baseline_initiated = node.metrics().discovery.req_initiated.get();
let baseline_timed_out = node.metrics().discovery.resp_timed_out.get();
let baseline_initiated = node.metrics().lookup.req_initiated.get();
let baseline_timed_out = node.metrics().lookup.resp_timed_out.get();
// --- t = 1100ms: first retry deadline (1*1000) ---
node.check_pending_lookups(1100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1162,7 +1187,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 1100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 1,
"retry #1 must invoke initiate_lookup exactly once"
);
@@ -1171,6 +1196,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
node.check_pending_lookups(3100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1178,7 +1204,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 3100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 2,
"retry #2 must invoke initiate_lookup exactly once more"
);
@@ -1187,6 +1213,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
node.check_pending_lookups(7100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1194,7 +1221,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 7100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"retry #3 must invoke initiate_lookup exactly once more"
);
@@ -1202,16 +1229,16 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// --- Just-before-final: at t=15099ms the 8s window is not yet reached ---
node.check_pending_lookups(15_099).await;
assert!(
node.pending_lookups.contains_key(&target_addr),
node.lookup.pending_lookups.contains_key(&target_addr),
"8s window not yet expired: pending_lookup must persist"
);
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"no new attempt before final deadline"
);
assert_eq!(
node.metrics().discovery.resp_timed_out.get(),
node.metrics().lookup.resp_timed_out.get(),
baseline_timed_out,
"no timeout before final deadline"
);
@@ -1225,18 +1252,18 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Pending lookup is dropped.
assert!(
!node.pending_lookups.contains_key(&target_addr),
!node.lookup.pending_lookups.contains_key(&target_addr),
"final timeout must remove the pending_lookups entry"
);
// resp_timed_out counter ticked.
assert_eq!(
node.metrics().discovery.resp_timed_out.get(),
node.metrics().lookup.resp_timed_out.get(),
baseline_timed_out + 1,
"final timeout must increment discovery.resp_timed_out"
);
// No additional initiate_lookup on the timeout step.
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"the final-timeout step must NOT call initiate_lookup"
);

View File

@@ -0,0 +1,998 @@
//! Characterization tests for the inbound-handshake (`handle_msg1`) establish
//! branches.
//!
//! These lock in the *current* observable behavior of the undertested inbound
//! classification paths by driving a REAL framed msg1 into `handle_msg1`
//! (constructing a genuine Noise IK msg1 and delivering it as a
//! `ReceivedPacket`), rather than poking rekey state directly the way the
//! `arm_rekey` helper does. They are an oracle for a later behavior-neutral
//! refactor: assertions capture what happens today, surprising or not.
//!
//! Coverage map (branch → test):
//! * epoch-restart → `chartest_msg1_epoch_restart_replaces_active_peer`
//! * duplicate (pre-crypto) → `chartest_msg1_duplicate_pending_resends_stored_msg2`
//! * duplicate (post-crypto) → `chartest_msg1_duplicate_active_same_epoch_resends_stored_msg2`
//! * cross-connection precedence→ `chartest_msg1_inbound_promote_defers_pending_outbound_to_same_identity`
//! * max-peers cap (bypass) → `chartest_msg1_at_cap_with_pending_outbound_bypasses_early_gate`
//! * tie-break (winner+loser) → `chartest_cross_connection_tiebreak_winner_and_loser`
//! * rekey-responder → `chartest_msg1_rekey_responder_stores_pending_session`
//! * rekey dual-init (we win) → `chartest_msg1_rekey_dual_init_we_win_drops_their_msg1`
//! * rekey dual-init (we lose) → `chartest_msg1_rekey_dual_init_we_lose_becomes_responder`
//!
//! The three rekey branches sit behind the hardcoded
//! `existing_session_age_secs >= 30` guard in `handle_msg1`, resolved from
//! `ActivePeer::session_established_at()` (a monotonic `std::time::Instant`
//! with no natural test seam — the field is private and `tokio::time` cannot
//! advance a std `Instant`). They are unblocked by the sole `#[cfg(test)]`
//! production seam `ActivePeer::test_backdate_session_established(age)`, which
//! only shifts that private timestamp — it changes no decision logic and no
//! threshold, and is compiled out of release builds.
use super::*;
use crate::config::UdpConfig;
use crate::noise::HandshakeState;
use crate::peer::ActivePeer;
use crate::transport::udp::UdpTransport;
use crate::transport::{TransportHandle, packet_channel};
use tokio::time::timeout;
/// Build a genuine wire-format Noise IK msg1 addressed to `node`, carrying a
/// chosen startup `epoch` and `sender_index`, from `sender`'s identity. Returns
/// the opaque wire bytes ready to place in a `ReceivedPacket`.
fn craft_msg1_wire(
node: &Node,
sender: &Identity,
epoch: [u8; 8],
sender_index: SessionIndex,
ts: u64,
) -> Vec<u8> {
use crate::proto::fmp::wire::build_msg1;
let peer_b_identity = PeerIdentity::from_pubkey_full(node.identity().pubkey_full());
let link_id = LinkId::new(0x0BAD_C0DE);
let mut conn = PeerConnection::outbound(link_id, peer_b_identity, ts);
let noise_msg1 = conn
.start_handshake(sender.keypair(), epoch, ts)
.expect("start_handshake produces noise msg1");
build_msg1(sender_index, &noise_msg1)
}
/// Register a real UDP transport on `node` and return an independent socket
/// (plus its addr) that plays the peer: the node's msg2 responses are sent to
/// this addr, so a test can observe wire-level output.
async fn register_udp_with_peer_socket(
node: &mut Node,
transport_id: TransportId,
) -> (tokio::net::UdpSocket, TransportAddr) {
let peer_sock = tokio::net::UdpSocket::bind("127.0.0.1:0")
.await
.expect("bind peer socket");
let peer_addr = TransportAddr::from_string(&peer_sock.local_addr().unwrap().to_string());
let cfg = UdpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
mtu: Some(1280),
..Default::default()
};
let (tx, _rx) = packet_channel(64);
let mut transport = UdpTransport::new(transport_id, None, cfg, tx);
transport.start_async().await.unwrap();
node.transports
.insert(transport_id, TransportHandle::Udp(transport));
(peer_sock, peer_addr)
}
/// Local re-impl of the `unit.rs` dummy-peer injector (that one is private to
/// its module). Fills the peer table with distinct identities so cap tests can
/// reach saturation.
fn inject_dummy_peers(node: &mut Node, count: usize) {
for i in 0..count {
let identity = make_peer_identity();
let addr = *identity.node_addr();
let peer = ActivePeer::new(identity, LinkId::new((i + 1) as u64), 0);
node.peers.insert(addr, peer);
}
}
/// Epoch-restart: an inbound msg1 from an already-active peer that carries a
/// DIFFERENT startup epoch is treated as a peer restart. The stale peer is torn
/// down and the fresh handshake is promoted in its place.
///
/// Oracle: after the push, the identity is still present but is a NEW peer —
/// it now holds a live Noise session (the stale one had none), its stored
/// remote epoch has advanced to the restart value, and it occupies a fresh link
/// and session index. `schedule_reconnect` is a no-op under a bare test config
/// (no auto-connect peer configured), so `retry_pending` stays empty.
#[tokio::test]
async fn chartest_msg1_epoch_restart_replaces_active_peer() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
let sender = Identity::generate();
let sender_pid = PeerIdentity::from_pubkey_full(sender.pubkey_full());
let sender_addr = *sender_pid.node_addr();
let old_epoch = [1u8; 8];
let new_epoch = [2u8; 8];
let old_link = LinkId::new(4242);
// Pre-existing active peer at the OLD epoch, sessionless. `current_addr`
// makes `should_admit_msg1` recognize the source as an established peer.
let mut old_peer = ActivePeer::new(sender_pid, old_link, 1000);
old_peer.set_remote_epoch(Some(old_epoch));
old_peer.set_current_addr(transport_id, peer_addr.clone());
node.peers.insert(sender_addr, old_peer);
assert!(!node.get_peer(&sender_addr).unwrap().has_session());
assert_eq!(
node.get_peer(&sender_addr).unwrap().remote_epoch(),
Some(old_epoch)
);
// Real msg1 carrying the NEW (restart) epoch.
let data = craft_msg1_wire(&node, &sender, new_epoch, SessionIndex::new(0x77), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let peer = node
.get_peer(&sender_addr)
.expect("restarted peer must remain present (replaced, not dropped)");
assert!(
peer.has_session(),
"restart promotes a fresh handshake, so the new peer holds a session"
);
assert_eq!(
peer.remote_epoch(),
Some(new_epoch),
"stored remote epoch advances to the restart value"
);
assert_ne!(
peer.link_id(),
old_link,
"restart replaces the link with the freshly allocated one"
);
let our_index = peer.our_index().expect("promoted peer has our_index");
assert!(
node.peers_by_index
.contains_key(&(transport_id, our_index.as_u32())),
"fresh session index registered in peers_by_index"
);
assert_eq!(node.peer_count(), 1, "old peer removed, new peer added");
assert!(
node.peering.reconciler.retry_pending.is_empty(),
"schedule_reconnect is a no-op with no auto-connect config"
);
// A msg2 response was emitted to the restarting peer.
let mut buf = [0u8; 2048];
let got = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf)).await;
assert!(
got.is_ok(),
"restart path must emit a msg2 response to the peer"
);
}
/// Duplicate msg1, pre-crypto short-circuit: a second msg1 from an address that
/// already has a genuinely-pending (not yet promoted) inbound link resends the
/// stored msg2 without paying the crypto cost or touching registry state.
///
/// Oracle: the exact stored msg2 bytes are resent verbatim, nothing is
/// promoted, the pending connection is left intact, and the msg1 rate limiter
/// rebalances (start then complete) to baseline.
#[tokio::test]
async fn chartest_msg1_duplicate_pending_resends_stored_msg2() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
// A pending inbound connection with a stored msg2, keyed in addr_to_link,
// NOT promoted to an active peer.
let link_id = node.allocate_link_id();
let mut conn =
PeerConnection::inbound_with_transport(link_id, transport_id, peer_addr.clone(), 1000);
let stored_msg2 = vec![0xC1, 0xC2, 0xC3, 0xC4, 0xC5];
conn.set_handshake_msg2(stored_msg2.clone());
let link = Link::connectionless(
link_id,
transport_id,
peer_addr.clone(),
LinkDirection::Inbound,
Duration::from_millis(100),
);
node.links.insert(link_id, link);
node.addr_to_link
.insert((transport_id, peer_addr.clone()), link_id);
node.connections.insert(link_id, conn);
assert_eq!(node.peer_count(), 0);
let before_pending = node.msg1_rate_limiter.pending_count();
// Duplicate msg1 from the same address. Content is irrelevant past a valid
// header — the pre-crypto branch fires before any decrypt.
let sender = Identity::generate();
let data = craft_msg1_wire(&node, &sender, [9u8; 8], SessionIndex::new(5), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let mut buf = [0u8; 2048];
let (n, _) = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf))
.await
.expect("stored msg2 must be resent")
.expect("recv_from");
assert_eq!(
&buf[..n],
&stored_msg2[..],
"the exact stored msg2 is resent for a duplicate msg1"
);
assert_eq!(node.peer_count(), 0, "duplicate msg1 promotes nothing");
assert!(
node.connections.contains_key(&link_id),
"pending connection is left intact"
);
assert_eq!(
node.msg1_rate_limiter.pending_count(),
before_pending,
"rate limiter rebalances to baseline"
);
}
/// Duplicate msg1, post-crypto same-epoch path: an inbound msg1 from an active
/// peer at the SAME epoch, on a session too young (< 30s) to be a rekey, is a
/// duplicate. The peer's stored msg2 is resent.
///
/// Oracle: the active peer's stored msg2 is resent, no new peer or session
/// index is allocated, and the existing peer is untouched.
#[tokio::test]
async fn chartest_msg1_duplicate_active_same_epoch_resends_stored_msg2() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
let sender = Identity::generate();
let sender_pid = PeerIdentity::from_pubkey_full(sender.pubkey_full());
let sender_addr = *sender_pid.node_addr();
let epoch = [7u8; 8];
let stored_msg2 = vec![0xD0, 0xD1, 0xD2, 0xD3];
let link_id = LinkId::new(555);
let mut peer = ActivePeer::new(sender_pid, link_id, 1000);
peer.set_remote_epoch(Some(epoch));
peer.set_current_addr(transport_id, peer_addr.clone());
peer.set_handshake_msg2(stored_msg2.clone());
node.peers.insert(sender_addr, peer);
// Session age is ~0 (< 30s) → the rekey gate is false → a same-epoch msg1
// classifies as a duplicate, not a rekey initiation.
assert!(!node.get_peer(&sender_addr).unwrap().has_session());
let data = craft_msg1_wire(&node, &sender, epoch, SessionIndex::new(0x33), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let mut buf = [0u8; 2048];
let (n, _) = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf))
.await
.expect("stored msg2 must be resent")
.expect("recv_from");
assert_eq!(&buf[..n], &stored_msg2[..]);
assert_eq!(node.peer_count(), 1);
assert_eq!(
node.get_peer(&sender_addr).unwrap().link_id(),
link_id,
"existing peer untouched by a duplicate msg1"
);
assert!(
node.peers_by_index.is_empty(),
"no new session index allocated on the duplicate path"
);
}
/// Cross-connection precedence: an inbound establish that promotes a peer while
/// a concurrent PENDING OUTBOUND connection to the SAME identity exists must NOT
/// tear that outbound down — it is deferred (kept alive so its later msg2 can
/// update `their_index` on the promoted peer).
///
/// Oracle: the inbound msg1 promotes the peer (with a live session), and both
/// the pending outbound connection and its `pending_outbound` index entry are
/// preserved.
#[tokio::test]
async fn chartest_msg1_inbound_promote_defers_pending_outbound_to_same_identity() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (_peer_sock, inbound_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
let sender = Identity::generate();
let sender_pid = PeerIdentity::from_pubkey_full(sender.pubkey_full());
let sender_addr = *sender_pid.node_addr();
// A concurrent pending OUTBOUND connection to the same identity, at a
// different source address.
let out_link = node.allocate_link_id();
let out_addr = TransportAddr::from_string("10.0.0.9:2121");
let mut out_conn = PeerConnection::outbound(out_link, sender_pid, 1000);
let our_keypair = node.identity().keypair();
let _ = out_conn
.start_handshake(our_keypair, node.startup_epoch(), 1000)
.unwrap();
let out_index = node.index_allocator.allocate().unwrap();
out_conn.set_our_index(out_index);
out_conn.set_transport_id(transport_id);
out_conn.set_source_addr(out_addr.clone());
let out_l = Link::connectionless(
out_link,
transport_id,
out_addr.clone(),
LinkDirection::Outbound,
Duration::from_millis(100),
);
node.links.insert(out_link, out_l);
node.addr_to_link
.insert((transport_id, out_addr.clone()), out_link);
node.connections.insert(out_link, out_conn);
node.pending_outbound
.insert((transport_id, out_index.as_u32()), out_link);
assert_eq!(node.peer_count(), 0);
// Inbound msg1 from the same identity, different source addr.
let data = craft_msg1_wire(&node, &sender, [3u8; 8], SessionIndex::new(0x22), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: inbound_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let peer = node
.get_peer(&sender_addr)
.expect("inbound establish must promote the peer");
assert!(peer.has_session());
assert_eq!(node.peer_count(), 1);
assert!(
node.connections.contains_key(&out_link),
"pending outbound to the same identity must be preserved (deferred cleanup)"
);
assert!(
node.pending_outbound
.contains_key(&(transport_id, out_index.as_u32())),
"the outbound pending_outbound entry is preserved for msg2 index-learning"
);
}
/// Max-peers cap, pending-outbound bypass: at saturation, a msg1 from a NEW
/// identity that already has a pending outbound to it is NOT silent-dropped by
/// the early cap gate — it proceeds far enough to emit a msg2, then the late gate
/// inside `promote_connection` rejects it (peer table is full).
///
/// Oracle discriminator vs. the plain new-peer silent-drop: a msg2 IS observed
/// on the wire (the early gate was bypassed), yet the peer is NOT promoted (the
/// late gate rejects). This locks in the asymmetry between the two cap gates.
#[tokio::test]
async fn chartest_msg1_at_cap_with_pending_outbound_bypasses_early_gate() {
let mut node = make_node_with_max_peers(2);
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
inject_dummy_peers(&mut node, 2);
assert_eq!(node.peer_count(), 2, "precondition: at cap");
let sender = Identity::generate();
let sender_pid = PeerIdentity::from_pubkey_full(sender.pubkey_full());
let sender_addr = *sender_pid.node_addr();
// A pending outbound to the (new) sender identity — this sets
// `has_pending_outbound_to_peer`, which turns off the early silent-drop.
let out_link = node.allocate_link_id();
let out_addr = TransportAddr::from_string("10.0.0.9:2121");
let mut out_conn = PeerConnection::outbound(out_link, sender_pid, 1000);
let our_keypair = node.identity().keypair();
let _ = out_conn
.start_handshake(our_keypair, node.startup_epoch(), 1000)
.unwrap();
let out_index = node.index_allocator.allocate().unwrap();
out_conn.set_our_index(out_index);
out_conn.set_transport_id(transport_id);
out_conn.set_source_addr(out_addr.clone());
node.connections.insert(out_link, out_conn);
node.pending_outbound
.insert((transport_id, out_index.as_u32()), out_link);
let data = craft_msg1_wire(&node, &sender, [4u8; 8], SessionIndex::new(0x44), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
// Late gate rejects: still at cap, sender not promoted.
assert_eq!(
node.peer_count(),
2,
"late cap gate rejects the new identity"
);
assert!(
!node.peers.contains_key(&sender_addr),
"new identity is not adopted at capacity"
);
// But the early gate was bypassed: a msg2 WAS put on the wire before the
// late-gate rejection (the discriminator against the plain silent-drop).
let mut buf = [0u8; 2048];
let got = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf)).await;
assert!(
got.is_ok() && got.unwrap().is_ok(),
"pending-outbound identity bypasses the early silent-drop, so a msg2 \
is emitted before the late cap gate rejects"
);
}
/// Cross-connection tie-break, winner AND loser in one deterministic run: both
/// nodes initiate to each other (simultaneous cross-connection). The rule is
/// "the smaller node_addr's OUTBOUND wins" (`cross_connection_winner`). After
/// both sides exchange msg1 (promote inbound) and msg2 (resolve), the winner has
/// swapped to its outbound session index while the loser keeps the inbound index
/// it assigned during its own msg1 handling.
///
/// Oracle: the smaller-addr node's peer.our_index equals the OUTBOUND index it
/// allocated at setup; the larger-addr node's peer.our_index equals the INBOUND
/// index it assigned while promoting the peer's msg1.
#[tokio::test]
async fn chartest_cross_connection_tiebreak_winner_and_loser() {
use crate::proto::fmp::wire::build_msg1;
let mut node_a = make_node();
let mut node_b = make_node();
let transport_id_a = TransportId::new(1);
let transport_id_b = TransportId::new(1);
let udp_config = UdpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
mtu: Some(1280),
..Default::default()
};
let (packet_tx_a, mut packet_rx_a) = packet_channel(64);
let (packet_tx_b, mut packet_rx_b) = packet_channel(64);
let mut transport_a = UdpTransport::new(transport_id_a, None, udp_config.clone(), packet_tx_a);
let mut transport_b = UdpTransport::new(transport_id_b, None, udp_config, packet_tx_b);
transport_a.start_async().await.unwrap();
transport_b.start_async().await.unwrap();
let addr_a = transport_a.local_addr().unwrap();
let addr_b = transport_b.local_addr().unwrap();
let remote_addr_b = TransportAddr::from_string(&addr_b.to_string());
let remote_addr_a = TransportAddr::from_string(&addr_a.to_string());
node_a
.transports
.insert(transport_id_a, TransportHandle::Udp(transport_a));
node_b
.transports
.insert(transport_id_b, TransportHandle::Udp(transport_b));
let peer_b_identity = PeerIdentity::from_pubkey_full(node_b.identity().pubkey_full());
let peer_a_identity = PeerIdentity::from_pubkey_full(node_a.identity().pubkey_full());
let node_a_addr = *node_a.node_addr();
let node_b_addr = *node_b.node_addr();
// A initiates to B.
let link_a_out = node_a.allocate_link_id();
let mut conn_a = PeerConnection::outbound(link_a_out, peer_b_identity, 1000);
let out_index_a = node_a.index_allocator.allocate().unwrap();
let noise_msg1_a = conn_a
.start_handshake(node_a.identity().keypair(), node_a.startup_epoch(), 1000)
.unwrap();
conn_a.set_our_index(out_index_a);
conn_a.set_transport_id(transport_id_a);
conn_a.set_source_addr(remote_addr_b.clone());
let wire_msg1_a = build_msg1(out_index_a, &noise_msg1_a);
node_a.links.insert(
link_a_out,
Link::connectionless(
link_a_out,
transport_id_a,
remote_addr_b.clone(),
LinkDirection::Outbound,
Duration::from_millis(100),
),
);
node_a
.addr_to_link
.insert((transport_id_a, remote_addr_b.clone()), link_a_out);
node_a.connections.insert(link_a_out, conn_a);
node_a
.pending_outbound
.insert((transport_id_a, out_index_a.as_u32()), link_a_out);
// B initiates to A.
let link_b_out = node_b.allocate_link_id();
let mut conn_b = PeerConnection::outbound(link_b_out, peer_a_identity, 1000);
let out_index_b = node_b.index_allocator.allocate().unwrap();
let noise_msg1_b = conn_b
.start_handshake(node_b.identity().keypair(), node_b.startup_epoch(), 1000)
.unwrap();
conn_b.set_our_index(out_index_b);
conn_b.set_transport_id(transport_id_b);
conn_b.set_source_addr(remote_addr_a.clone());
let wire_msg1_b = build_msg1(out_index_b, &noise_msg1_b);
node_b.links.insert(
link_b_out,
Link::connectionless(
link_b_out,
transport_id_b,
remote_addr_a.clone(),
LinkDirection::Outbound,
Duration::from_millis(100),
),
);
node_b
.addr_to_link
.insert((transport_id_b, remote_addr_a.clone()), link_b_out);
node_b.connections.insert(link_b_out, conn_b);
node_b
.pending_outbound
.insert((transport_id_b, out_index_b.as_u32()), link_b_out);
// Both put msg1 on the wire.
node_a
.transports
.get(&transport_id_a)
.unwrap()
.send(&remote_addr_b, &wire_msg1_a)
.await
.unwrap();
node_b
.transports
.get(&transport_id_b)
.unwrap()
.send(&remote_addr_a, &wire_msg1_b)
.await
.unwrap();
// Each processes the other's msg1 (promotes inbound, assigns an inbound index).
let pkt_at_b = timeout(Duration::from_secs(1), packet_rx_b.recv())
.await
.unwrap()
.unwrap();
node_b.handle_msg1(pkt_at_b).await;
let pkt_at_a = timeout(Duration::from_secs(1), packet_rx_a.recv())
.await
.unwrap()
.unwrap();
node_a.handle_msg1(pkt_at_a).await;
// Inbound indices assigned during promotion (before resolution).
let inbound_index_a = node_a.get_peer(&node_b_addr).unwrap().our_index().unwrap();
let inbound_index_b = node_b.get_peer(&node_a_addr).unwrap().our_index().unwrap();
// Each processes the other's msg2 (cross-connection resolution).
let msg2_at_a = timeout(Duration::from_secs(1), packet_rx_a.recv())
.await
.unwrap()
.unwrap();
node_a.handle_msg2(msg2_at_a).await;
let msg2_at_b = timeout(Duration::from_secs(1), packet_rx_b.recv())
.await
.unwrap()
.unwrap();
node_b.handle_msg2(msg2_at_b).await;
// Rule: smaller node_addr's OUTBOUND wins → that node swaps to its outbound
// index; the larger node keeps the inbound index from its own msg1 handling.
let a_is_winner = node_a_addr < node_b_addr;
let final_our_index_a = node_a.get_peer(&node_b_addr).unwrap().our_index().unwrap();
let final_our_index_b = node_b.get_peer(&node_a_addr).unwrap().our_index().unwrap();
if a_is_winner {
assert_eq!(
final_our_index_a, out_index_a,
"winner (smaller addr) swaps to its outbound session index"
);
assert_eq!(
final_our_index_b, inbound_index_b,
"loser (larger addr) keeps the inbound index from its msg1 handling"
);
} else {
assert_eq!(
final_our_index_b, out_index_b,
"winner (smaller addr) swaps to its outbound session index"
);
assert_eq!(
final_our_index_a, inbound_index_a,
"loser (larger addr) keeps the inbound index from its msg1 handling"
);
}
// Both remain single, sendable peers after resolution.
assert_eq!(node_a.peer_count(), 1);
assert_eq!(node_b.peer_count(), 1);
assert!(node_a.get_peer(&node_b_addr).unwrap().can_send());
assert!(node_b.get_peer(&node_a_addr).unwrap().can_send());
for (_, t) in node_a.transports.iter_mut() {
t.stop().await.ok();
}
for (_, t) in node_b.transports.iter_mut() {
t.stop().await.ok();
}
}
// ===========================================================================
// Rekey establish branches (unblocked by the `#[cfg(test)]`
// `ActivePeer::test_backdate_session_established` seam that lets a test age a
// real session past the hardcoded 30s rekey gate in `handle_msg1`).
// ===========================================================================
/// Drive a real inbound msg1 through `handle_msg1` so `node` promotes an active
/// peer for `sender` at startup `epoch`, draining the msg2 the promotion emits.
/// Returns the sender's NodeAddr.
async fn establish_active_peer_via_msg1(
node: &mut Node,
sender: &Identity,
epoch: [u8; 8],
transport_id: TransportId,
peer_addr: &TransportAddr,
peer_sock: &tokio::net::UdpSocket,
ts: u64,
) -> NodeAddr {
let sender_addr = *PeerIdentity::from_pubkey_full(sender.pubkey_full()).node_addr();
let data = craft_msg1_wire(node, sender, epoch, SessionIndex::new(0x01), ts);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: ts,
};
node.handle_msg1(packet).await;
// Promotion emits a msg2 AND an initial TreeAnnounce; drain every queued
// datagram so a later recv observes only the rekey response (or its
// absence), never a leftover from establishment.
let mut buf = [0u8; 2048];
while timeout(Duration::from_millis(150), peer_sock.recv_from(&mut buf))
.await
.is_ok()
{}
sender_addr
}
/// Draw a fresh sender identity whose NodeAddr is greater-than (`want_greater`)
/// or less-than the node's own NodeAddr, so the dual-init tie-break outcome is
/// deterministic. The comparison invariant is enforced, so the test outcome is
/// deterministic even though the identity draw is random.
fn sender_with_addr_relation(node: &Node, want_greater: bool) -> Identity {
let node_addr = *node.node_addr();
loop {
let s = Identity::generate();
let a = *PeerIdentity::from_pubkey_full(s.pubkey_full()).node_addr();
if a != node_addr && (a > node_addr) == want_greater {
return s;
}
}
}
/// Arm a local in-flight (initiator) rekey on `node`'s peer for `sender`, with a
/// real allocated index registered in `peers_by_index`/`pending_outbound` (as a
/// genuine in-flight rekey would be). Returns the armed rekey index.
fn arm_local_rekey(
node: &mut Node,
sender: &Identity,
sender_addr: &NodeAddr,
transport_id: TransportId,
) -> SessionIndex {
let rekey_index = node.index_allocator.allocate().unwrap();
node.peers_by_index
.insert((transport_id, rekey_index.as_u32()), *sender_addr);
node.pending_outbound
.insert((transport_id, rekey_index.as_u32()), LinkId::new(0xF00D));
let local = Identity::generate();
let hs = HandshakeState::new_initiator(local.keypair(), sender.pubkey_full());
node.get_peer_mut(sender_addr)
.unwrap()
.set_rekey_state(hs, rekey_index, vec![0xAB; 64], 0);
rekey_index
}
/// Rekey-responder: a genuine rekey msg1 (a fresh IK handshake at the SAME
/// epoch) arriving for an active peer whose session is past the 30s gate is
/// processed as a rekey. The responder extracts the new session and holds it as
/// PENDING (awaiting K-bit cutover) without disturbing the live session.
///
/// Oracle: the new session lands in `pending_new_session` with a freshly
/// allocated `pending_our_index` and `pending_their_index` == the rekey msg1
/// sender index; the current session/index stay live and registered; the new
/// index is additionally registered in `peers_by_index`; and a rekey msg2 is
/// emitted. The peer is neither replaced nor left `rekey_in_progress`.
#[tokio::test]
async fn chartest_msg1_rekey_responder_stores_pending_session() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
let sender = Identity::generate();
let epoch = [5u8; 8];
let sender_addr = establish_active_peer_via_msg1(
&mut node,
&sender,
epoch,
transport_id,
&peer_addr,
&peer_sock,
1000,
)
.await;
let old_index = {
let p = node.get_peer(&sender_addr).expect("peer established");
assert!(p.has_session());
assert!(p.is_healthy());
assert_eq!(p.remote_epoch(), Some(epoch));
assert!(!p.rekey_in_progress());
assert!(p.pending_new_session().is_none());
p.our_index().unwrap()
};
assert!(
node.peers_by_index
.contains_key(&(transport_id, old_index.as_u32()))
);
let index_count_before = node.index_allocator.count();
// Age the live session past the 30s rekey gate (test-only seam).
node.get_peer_mut(&sender_addr)
.unwrap()
.test_backdate_session_established(Duration::from_secs(31));
// A genuine rekey msg1 (fresh IK handshake, SAME epoch) arrives.
let rekey_sender_index = SessionIndex::new(0xBEEF);
let data = craft_msg1_wire(&node, &sender, epoch, rekey_sender_index, 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let p = node
.get_peer(&sender_addr)
.expect("peer still present (not replaced)");
assert_eq!(
node.peer_count(),
1,
"rekey neither adds nor replaces the peer"
);
assert!(
p.pending_new_session().is_some(),
"new session held as pending"
);
let new_index = p.pending_our_index().expect("pending our_index allocated");
assert_eq!(
p.pending_their_index(),
Some(rekey_sender_index),
"pending their_index = the rekey msg1 sender index"
);
assert!(
!p.rekey_in_progress(),
"set_pending_session clears rekey_in_progress"
);
assert_eq!(
p.our_index(),
Some(old_index),
"current session index stays live until cutover"
);
assert!(p.has_session(), "current session remains live");
assert!(
node.peers_by_index
.contains_key(&(transport_id, old_index.as_u32())),
"current index still registered"
);
assert!(
node.peers_by_index
.contains_key(&(transport_id, new_index.as_u32())),
"new pending index registered"
);
assert_eq!(
node.index_allocator.count(),
index_count_before + 1,
"exactly one extra index allocated for the pending session"
);
let mut buf = [0u8; 2048];
let got = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf)).await;
assert!(
got.is_ok() && got.unwrap().is_ok(),
"rekey responder emits a rekey msg2"
);
}
/// Rekey dual-init, WE WIN: with a local rekey in flight, a simultaneous rekey
/// msg1 arrives from a peer whose NodeAddr is larger than ours. The tie-break
/// ("smaller NodeAddr wins as initiator") makes us the winner, so we DROP their
/// msg1 and keep driving our own rekey.
///
/// Oracle: our in-flight rekey is untouched (`rekey_in_progress` stays true, our
/// rekey index retained and still registered), no responder session is stored,
/// no responder index is allocated, and no rekey msg2 is emitted.
#[tokio::test]
async fn chartest_msg1_rekey_dual_init_we_win_drops_their_msg1() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
// We win when our node_addr < peer's → pick a sender greater than us.
let sender = sender_with_addr_relation(&node, true);
let epoch = [6u8; 8];
let sender_addr = establish_active_peer_via_msg1(
&mut node,
&sender,
epoch,
transport_id,
&peer_addr,
&peer_sock,
1000,
)
.await;
assert!(
*node.node_addr() < sender_addr,
"precondition: node wins the tie-break"
);
node.get_peer_mut(&sender_addr)
.unwrap()
.test_backdate_session_established(Duration::from_secs(31));
let rekey_index = arm_local_rekey(&mut node, &sender, &sender_addr, transport_id);
assert!(node.get_peer(&sender_addr).unwrap().rekey_in_progress());
let index_count_before = node.index_allocator.count();
// Their simultaneous rekey msg1 arrives.
let data = craft_msg1_wire(&node, &sender, epoch, SessionIndex::new(0xAAAA), 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let p = node.get_peer(&sender_addr).expect("peer present");
assert!(
p.rekey_in_progress(),
"our rekey survives; the winner does not abandon it"
);
assert!(
p.pending_new_session().is_none(),
"no responder session stored on the winner path"
);
assert_eq!(
p.rekey_our_index(),
Some(rekey_index),
"our rekey index retained"
);
assert!(
node.peers_by_index
.contains_key(&(transport_id, rekey_index.as_u32())),
"our rekey index still registered in peers_by_index"
);
assert!(
node.pending_outbound
.contains_key(&(transport_id, rekey_index.as_u32())),
"our rekey pending_outbound entry retained"
);
assert_eq!(
node.index_allocator.count(),
index_count_before,
"no responder index allocated on the winner path"
);
let mut buf = [0u8; 2048];
let got = timeout(Duration::from_millis(300), peer_sock.recv_from(&mut buf)).await;
assert!(
got.is_err(),
"winner emits no msg2 in response to the dropped rekey msg1"
);
}
/// Rekey dual-init, WE LOSE: with a local rekey in flight, a simultaneous rekey
/// msg1 arrives from a peer whose NodeAddr is smaller than ours. The tie-break
/// makes us the loser, so we ABANDON our own rekey and respond as the rekey
/// responder.
///
/// Oracle: our in-flight rekey is abandoned (`rekey_in_progress` cleared, the
/// abandoned index freed and unregistered from `pending_outbound`), the new
/// session is stored as pending with `pending_their_index` == the rekey msg1
/// sender index, the pending index is registered, and a rekey msg2 is emitted.
#[tokio::test]
async fn chartest_msg1_rekey_dual_init_we_lose_becomes_responder() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let (peer_sock, peer_addr) = register_udp_with_peer_socket(&mut node, transport_id).await;
// We lose when our node_addr > peer's → pick a sender smaller than us.
let sender = sender_with_addr_relation(&node, false);
let epoch = [6u8; 8];
let sender_addr = establish_active_peer_via_msg1(
&mut node,
&sender,
epoch,
transport_id,
&peer_addr,
&peer_sock,
1000,
)
.await;
assert!(
*node.node_addr() > sender_addr,
"precondition: node loses the tie-break"
);
node.get_peer_mut(&sender_addr)
.unwrap()
.test_backdate_session_established(Duration::from_secs(31));
let rekey_index = arm_local_rekey(&mut node, &sender, &sender_addr, transport_id);
assert!(node.get_peer(&sender_addr).unwrap().rekey_in_progress());
// Their simultaneous rekey msg1 arrives.
let rekey_sender_index = SessionIndex::new(0xCCCC);
let data = craft_msg1_wire(&node, &sender, epoch, rekey_sender_index, 2000);
let packet = ReceivedPacket {
transport_id,
remote_addr: peer_addr.clone(),
data,
timestamp_ms: 2000,
};
node.handle_msg1(packet).await;
let p = node.get_peer(&sender_addr).expect("peer present");
assert!(
!p.rekey_in_progress(),
"we abandoned our rekey and became responder"
);
assert!(
p.pending_new_session().is_some(),
"responder stores the new session as pending"
);
assert_eq!(
p.pending_their_index(),
Some(rekey_sender_index),
"pending their_index = the rekey msg1 sender index"
);
let new_index = p
.pending_our_index()
.expect("responder allocated a pending index");
assert!(
!node
.pending_outbound
.contains_key(&(transport_id, rekey_index.as_u32())),
"abandoned rekey pending_outbound entry removed"
);
assert!(
node.peers_by_index
.contains_key(&(transport_id, new_index.as_u32())),
"new pending index registered"
);
let mut buf = [0u8; 2048];
let got = timeout(Duration::from_millis(500), peer_sock.recv_from(&mut buf)).await;
assert!(
got.is_ok() && got.unwrap().is_ok(),
"loser (now responder) emits a rekey msg2"
);
}

View File

@@ -83,7 +83,7 @@ async fn make_test_node_ethernet(interface: &str) -> TestNode {
let config = EthernetConfig {
interface: interface.to_string(),
discovery: Some(false),
listen: Some(false),
announce: Some(false),
accept_connections: Some(true),
..Default::default()

View File

@@ -5,9 +5,11 @@
//! multi-hop forwarding through live node topologies.
use super::*;
use crate::node::session_wire::{FSP_FLAG_CP, build_fsp_header};
use crate::protocol::{SessionAck, SessionDatagram, SessionSetup, encode_coords};
use crate::tree::TreeCoordinate;
use crate::proto::fsp::wire::{FSP_FLAG_CP, build_fsp_header};
use crate::proto::fsp::{SessionAck, SessionSetup};
use crate::proto::link::SessionDatagram;
use crate::proto::stp::TreeCoordinate;
use crate::proto::stp::encode_coords;
use spanning_tree::{
TestNode, cleanup_nodes, process_available_packets, run_tree_test, verify_tree_convergence,
};
@@ -591,7 +593,7 @@ async fn test_forwarding_with_cache_warming_enables_routing() {
// ============================================================================
use crate::node::TransportDropState;
use crate::node::handlers::session::mark_ipv6_ecn_ce;
use crate::proto::fsp::mark_ipv6_ecn_ce;
use crate::transport::TransportId;
/// Build a minimal IPv6 header (40 bytes) with specified ECN bits.

View File

@@ -5,7 +5,7 @@ use super::*;
#[tokio::test]
async fn test_two_node_handshake_udp() {
use crate::config::UdpConfig;
use crate::node::wire::{
use crate::proto::fmp::wire::{
build_encrypted, build_established_header, build_msg1, prepend_inner_header,
};
use crate::transport::udp::UdpTransport;
@@ -243,7 +243,7 @@ async fn test_two_node_handshake_udp() {
#[tokio::test]
async fn test_run_rx_loop_handshake() {
use crate::config::UdpConfig;
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
use crate::transport::udp::UdpTransport;
use tokio::time::Duration;
@@ -285,8 +285,8 @@ async fn test_run_rx_loop_handshake() {
node_b.packet_rx = Some(packet_rx_b);
// Set node state to Running (transports need to be operational)
node_a.state = NodeState::Running;
node_b.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
// === Phase 1: Node A initiates handshake to Node B ===
@@ -434,7 +434,7 @@ async fn test_run_rx_loop_handshake() {
#[tokio::test]
async fn test_cross_connection_both_initiate() {
use crate::config::UdpConfig;
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
use crate::transport::udp::UdpTransport;
use tokio::time::{Duration, timeout};
@@ -788,7 +788,7 @@ async fn test_failed_connection_cleanup() {
/// Test that msg1 bytes are stored on connection for resend.
#[tokio::test]
async fn test_msg1_stored_for_resend() {
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
let mut node = make_node();
let transport_id = TransportId::new(1);
@@ -846,7 +846,7 @@ async fn test_resend_scheduling() {
conn.set_source_addr(remote_addr.clone());
// Store msg1 with first resend at now + 1000ms
let wire_msg1 = crate::node::wire::build_msg1(our_index, &noise_msg1);
let wire_msg1 = crate::proto::fmp::wire::build_msg1(our_index, &noise_msg1);
conn.set_handshake_msg1(wire_msg1, now_ms + 1000);
let link = Link::connectionless(
@@ -924,7 +924,7 @@ fn test_resend_count_tracking() {
/// Test that duplicate msg2 is silently dropped when pending_outbound is already cleared.
#[tokio::test]
async fn test_duplicate_msg2_dropped() {
use crate::node::wire::build_msg2;
use crate::proto::fmp::wire::build_msg2;
use crate::transport::ReceivedPacket;
let mut node = make_node();
@@ -1072,7 +1072,7 @@ async fn test_should_admit_msg1_admits_rekey_when_udp_accept_off() {
///
/// The carve-out predicate must also consult peer state by source
/// address: `current_addr()` is updated from inbound encrypted-frame
/// source addrs (`handlers/encrypted.rs`), so an established peer can
/// source addrs (`dataplane/encrypted.rs`), so an established peer can
/// be matched even when the addr_to_link key is hostname-form and the
/// incoming addr is numeric.
#[tokio::test]

View File

@@ -0,0 +1,439 @@
//! Characterization tests for the three under-tested MMP tick handlers.
//!
//! These lock in the *current* observable behavior of the MMP fan-out and
//! first-RTT paths so a later behavior-neutral sans-IO extraction has an
//! equality oracle. The `check_link_heartbeats` handler already has a good
//! oracle (`heartbeat.rs` + `tcp.rs`) and is not re-covered here; this file
//! targets the three paths with no direct handler tests:
//!
//! * `check_mmp_reports` — link-layer mode/flag fan-out gating
//! * `check_session_mmp_reports` — session mode + PathMtu gating + backoff dedup
//! * `handle_receiver_report` — the first-RTT tree re-evaluation branch
//!
//! Assertions capture what the code does today, surprising or not.
//!
//! Report-generation is probed through the reused `proto/mmp/` primitives
//! (`should_send_report` / `should_send_notification`): after a handler tick,
//! a *consumed* interval reads as "not due" (the report was built) while an
//! *ungated* interval still reads as "due" (the report was suppressed by the
//! mode/flag gate). This survives the later refactor because those primitives
//! stay in `proto/mmp/` unchanged.
//!
//! Two `#[cfg(test)]` production seams are used, both on `ActivePeer`:
//! * `test_init_mmp(mode)` — attach link MMP with a chosen mode to a
//! bare (sessionless) peer, so mode gating is exercisable.
//! * `test_backdate_session_start` — age `session_elapsed_ms()` so a crafted
//! ReceiverReport yields a positive RTT sample (first-RTT trigger).
//!
//! Neither changes any decision logic or threshold.
use super::*;
use crate::config::SessionMmpConfig;
use crate::node::session::{EndToEndState, SessionEntry};
use crate::noise::HandshakeState;
use crate::peer::ActivePeer;
use crate::proto::mmp::{MmpMode, ReceiverReport};
use crate::proto::stp::{ParentDeclaration, TreeCoordinate};
// ===========================================================================
// Helpers
// ===========================================================================
/// Insert a bare (sessionless) peer carrying link-layer MMP state in `mode`.
/// Returns the peer's NodeAddr.
fn insert_link_peer(node: &mut Node, mode: MmpMode) -> NodeAddr {
let identity = make_peer_identity();
let addr = *identity.node_addr();
let mut peer = ActivePeer::new(identity, LinkId::new(1), 0);
peer.test_init_mmp(mode);
node.peers.insert(addr, peer);
addr
}
/// Arm both sender and receiver link-MMP intervals so a report would be built.
fn arm_link_mmp(node: &mut Node, addr: &NodeAddr) {
let mmp = node.get_peer_mut(addr).unwrap().mmp_mut().unwrap();
mmp.sender.record_sent(1, 100, 500);
mmp.receiver
.record_recv(1, 100, 500, false, crate::time::mono_ms());
}
/// Complete an in-memory Noise IK handshake, returning the initiator session.
fn make_noise_session(
our_identity: &crate::Identity,
remote_identity: &crate::Identity,
) -> crate::noise::NoiseSession {
let mut initiator =
HandshakeState::new_initiator(our_identity.keypair(), remote_identity.pubkey_full());
let mut responder = HandshakeState::new_responder(remote_identity.keypair());
let mut init_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut init_epoch);
initiator.set_local_epoch(init_epoch);
let mut resp_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut resp_epoch);
responder.set_local_epoch(resp_epoch);
let msg1 = initiator.write_message_1().unwrap();
responder.read_message_1(&msg1).unwrap();
let msg2 = responder.write_message_2().unwrap();
initiator.read_message_2(&msg2).unwrap();
initiator.into_session().unwrap()
}
/// Insert an Established session carrying session-layer MMP state in `mode`.
/// Returns the destination NodeAddr.
fn insert_session(node: &mut Node, mode: MmpMode) -> NodeAddr {
let remote = crate::Identity::generate();
let remote_addr = *remote.node_addr();
let session = make_noise_session(node.identity(), &remote);
let mut entry = SessionEntry::new(
remote_addr,
remote.pubkey_full(),
EndToEndState::Established(session),
1000,
true,
);
let cfg = SessionMmpConfig {
mode,
..SessionMmpConfig::default()
};
entry.init_mmp(&cfg);
node.sessions.insert(remote_addr, entry);
remote_addr
}
/// Arm both sender and receiver session-MMP intervals.
fn arm_session_mmp(node: &mut Node, addr: &NodeAddr) {
let mmp = node.sessions.get_mut(addr).unwrap().mmp_mut().unwrap();
mmp.sender.record_sent(1, 100, 500);
mmp.receiver
.record_recv(1, 100, 500, false, crate::time::mono_ms());
}
// ===========================================================================
// check_mmp_reports — link-layer mode fan-out
// ===========================================================================
/// Full mode: both a SenderReport and a ReceiverReport are generated (both
/// intervals consumed).
#[tokio::test]
async fn mmp_full_mode_builds_sender_and_receiver_reports() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Full);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
!mmp.sender.should_send_report(now),
"Full mode consumes the sender interval (SenderReport built)"
);
assert!(
!mmp.receiver.should_send_report(now),
"Full mode consumes the receiver interval (ReceiverReport built)"
);
}
/// Lightweight mode: only a ReceiverReport is generated; the sender interval
/// is left intact (no SenderReport in Lightweight).
#[tokio::test]
async fn mmp_lightweight_mode_builds_receiver_report_only() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Lightweight);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Lightweight mode suppresses the SenderReport (sender interval intact)"
);
assert!(
!mmp.receiver.should_send_report(now),
"Lightweight mode still builds the ReceiverReport (receiver interval consumed)"
);
}
/// Minimal mode: neither report is generated; both intervals stay intact.
#[tokio::test]
async fn mmp_minimal_mode_builds_nothing() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Minimal);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Minimal mode suppresses the SenderReport"
);
assert!(
mmp.receiver.should_send_report(now),
"Minimal mode suppresses the ReceiverReport"
);
}
/// Periodic operator logging fires once per interval: a fresh peer is due for
/// a log, and after one tick the log is marked (not due again within the
/// interval).
#[tokio::test]
async fn mmp_should_log_marks_logged_once_per_interval() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Full);
assert!(
node.get_peer(&addr)
.unwrap()
.mmp()
.unwrap()
.should_log(crate::time::mono_ms()),
"a freshly created peer is due for its first operator log"
);
node.check_mmp_reports().await;
assert!(
!node
.get_peer(&addr)
.unwrap()
.mmp()
.unwrap()
.should_log(crate::time::mono_ms()),
"after one tick the log is marked and not due again within the interval"
);
}
// ===========================================================================
// check_session_mmp_reports — session mode + PathMtu gating + backoff dedup
// ===========================================================================
/// Full mode session: both SenderReport and ReceiverReport are generated
/// (both intervals consumed) even though the send has no route and fails.
#[tokio::test]
async fn session_full_mode_builds_sender_and_receiver_reports() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Full);
arm_session_mmp(&mut node, &addr);
node.check_session_mmp_reports().await;
let mmp = node.get_session(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
!mmp.sender.should_send_report(now),
"Full session consumes the sender interval"
);
assert!(
!mmp.receiver.should_send_report(now),
"Full session consumes the receiver interval"
);
}
/// PathMtu notifications gate on all modes: in Minimal mode neither report is
/// built, yet a PathMtuNotification is still generated when an MTU has been
/// observed.
#[tokio::test]
async fn session_minimal_mode_still_sends_path_mtu() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Minimal);
arm_session_mmp(&mut node, &addr);
// Observe an MTU so a notification becomes due (all modes).
node.sessions
.get_mut(&addr)
.unwrap()
.mmp_mut()
.unwrap()
.path_mtu
.observe_incoming_mtu(1200);
let now_before = crate::time::mono_ms();
assert!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.path_mtu
.should_send_notification(now_before),
"precondition: a PathMtuNotification is due after observing an MTU"
);
node.check_session_mmp_reports().await;
let mmp = node.get_session(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Minimal mode suppresses the session SenderReport"
);
assert!(
mmp.receiver.should_send_report(now),
"Minimal mode suppresses the session ReceiverReport"
);
assert!(
!mmp.path_mtu.should_send_notification(now),
"PathMtuNotification is generated in Minimal mode (gate is mode-independent)"
);
}
/// Backoff dedup, all-fail side: a Full-mode session generates two reports
/// (SR + RR) to one destination; with no route both sends fail. The
/// per-destination dedup collapses the two failures into exactly ONE
/// `record_send_failure` (consecutive count advances by 1, not 2).
#[tokio::test]
async fn session_backoff_all_reports_fail_records_single_failure() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Full);
arm_session_mmp(&mut node, &addr);
assert_eq!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.sender
.consecutive_send_failures(),
0,
"precondition: no prior send failures"
);
node.check_session_mmp_reports().await;
assert_eq!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.sender
.consecutive_send_failures(),
1,
"two failed reports to one dest dedup to a single record_send_failure"
);
}
// ===========================================================================
// handle_receiver_report — first-RTT tree re-evaluation branch
// ===========================================================================
/// Build a peer (NodeAddr strictly smaller than the node's own) that carries
/// link MMP but no RTT yet, and register it in the tree as a self-root with
/// that smaller address. This makes it a mandatory parent-switch target once
/// it becomes eligible. Returns the peer's NodeAddr.
fn setup_smaller_root_peer(node: &mut Node) -> NodeAddr {
let my_addr = *node.node_addr();
let (identity, addr) = loop {
let id = make_peer_identity();
let a = *id.node_addr();
if a < my_addr {
break (id, a);
}
};
let mut peer = ActivePeer::new(identity, LinkId::new(1), 0);
peer.test_init_mmp(MmpMode::Full);
// Age the session so a crafted ReceiverReport yields a positive RTT.
peer.test_backdate_session_start(std::time::Duration::from_secs(10));
node.peers.insert(addr, peer);
// Register the peer as a self-root in the tree at its (smaller) address.
node.tree_state_mut().update_peer(
ParentDeclaration::self_root(addr, 1, 0),
TreeCoordinate::root(addr),
);
addr
}
/// Craft a ReceiverReport whose timestamp echo yields a valid first RTT
/// sample. `highest`/`pkts`/`bytes` advance the cumulative counters so a
/// second report is not dropped as stale/duplicate.
fn craft_rr_payload(highest: u64, pkts: u64, bytes: u64) -> Vec<u8> {
let rr = ReceiverReport {
highest_counter: highest,
cumulative_packets_recv: pkts,
cumulative_bytes_recv: bytes,
timestamp_echo: 1000,
dwell_time: 0,
max_burst_loss: 0,
mean_burst_loss: 0,
jitter: 0,
ecn_ce_count: 0,
owd_trend: 0,
burst_loss_count: 0,
cumulative_reorder_count: 0,
interval_packets_recv: pkts as u32,
interval_bytes_recv: bytes as u32,
};
// handle_receiver_report receives the body with the msg_type byte stripped.
rr.encode()[1..].to_vec()
}
/// A first RTT sample flips the peer eligible for parent selection AND fires
/// the shell-resident tree branch: the node (initially self-root) adopts the
/// smaller-addressed peer as its new root.
#[tokio::test]
async fn first_rtt_flips_peer_eligible_and_triggers_tree_reeval() {
let mut node = make_node();
let addr = setup_smaller_root_peer(&mut node);
assert!(
node.tree_state().is_root(),
"precondition: node starts as its own root"
);
assert!(
!node.get_peer(&addr).unwrap().has_srtt(),
"precondition: peer has no RTT measurement yet"
);
let switches_before = node.metrics().tree.parent_switches.get();
node.handle_receiver_report(&addr, &craft_rr_payload(10, 5, 500))
.await;
assert!(
node.get_peer(&addr).unwrap().has_srtt(),
"first RTT sample makes the peer eligible for parent selection"
);
assert!(
!node.tree_state().is_root(),
"the first-RTT tree branch fired: node adopted a parent"
);
assert_eq!(
node.tree_state().root(),
&addr,
"node switched its root to the smaller-addressed peer"
);
assert!(
node.metrics().tree.parent_switches.get() > switches_before,
"the parent-switch was recorded in the tree metrics"
);
}
/// Regression guard: a *second* ReceiverReport (RTT already initialized, so
/// `first_rtt` is false) does NOT re-enter the tree branch — no further parent
/// switch is recorded.
#[tokio::test]
async fn non_first_receiver_report_does_not_retrigger_tree() {
let mut node = make_node();
let addr = setup_smaller_root_peer(&mut node);
// First report: fires the branch (established by the test above).
node.handle_receiver_report(&addr, &craft_rr_payload(10, 5, 500))
.await;
let switches_after_first = node.metrics().tree.parent_switches.get();
assert!(node.get_peer(&addr).unwrap().has_srtt());
// Second report with advanced counters: first_rtt is now false.
node.handle_receiver_report(&addr, &craft_rr_payload(20, 10, 1000))
.await;
assert_eq!(
node.metrics().tree.parent_switches.get(),
switches_after_first,
"a non-first ReceiverReport does not re-enter the first-RTT tree branch"
);
}

View File

@@ -13,11 +13,13 @@ mod bootstrap;
mod decrypt_failure;
mod disconnect;
mod discovery;
mod establish_chartests;
#[cfg(target_os = "linux")]
mod ethernet;
mod forwarding;
mod handshake;
mod heartbeat;
mod mmp_chartests;
mod routing;
mod session;
mod spanning_tree;
@@ -28,6 +30,25 @@ pub(super) fn make_node() -> Node {
make_node_with(Config::new())
}
/// A test node that reaches `Full` health on `start()`.
///
/// A default [`make_node`] configures no transports, so its `start()` now
/// resolves to `NodeState::Failed` (zero transports up) and
/// returns `NoOperationalTransports`. Lifecycle-state tests that need a running
/// node build one with a single loopback UDP transport (ephemeral port) as the
/// sole configured child — DNS disabled — so bring-up has exactly one
/// configured child and it comes up (`Full`). Mirrors the udp config in
/// `test_node_start_does_not_wait_for_nostr_relay_startup`.
pub(super) fn make_healthy_node() -> Node {
let mut config = Config::new();
config.transports.udp = crate::config::TransportInstances::Single(crate::config::UdpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
..Default::default()
});
config.dns.enabled = false;
make_node_with(config)
}
/// Build a test node from an explicit `Config`. Immutable state lives solely in
/// the shared `NodeContext`, built once at construction — there is no
/// post-construction field to poke, so set limits/config on the `Config` here.

View File

@@ -4,8 +4,8 @@
//! filter priority, greedy tree routing, and tie-breaking.
use super::*;
use crate::bloom::BloomFilter;
use crate::tree::{ParentDeclaration, TreeCoordinate};
use crate::proto::bloom::BloomFilter;
use crate::proto::stp::{ParentDeclaration, TreeCoordinate};
use spanning_tree::{
TestNode, cleanup_nodes, drain_all_packets, generate_random_edges, initiate_handshake,
lock_large_network_test, make_test_node, run_tree_test, verify_tree_convergence,
@@ -809,7 +809,7 @@ async fn test_routing_reachability_100_nodes() {
/// Node 0 should no longer be able to route to node 3.
#[tokio::test]
async fn test_routing_stops_after_peer_removal() {
use crate::protocol::{Disconnect, DisconnectReason};
use crate::proto::fmp::{Disconnect, DisconnectReason};
let edges = vec![(0, 1), (1, 2), (2, 3)];
let mut nodes = run_tree_test(4, &edges, false).await;
@@ -826,7 +826,7 @@ async fn test_routing_stops_after_peer_removal() {
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let all_coords: Vec<(NodeAddr, crate::tree::TreeCoordinate)> = nodes
let all_coords: Vec<(NodeAddr, crate::proto::stp::TreeCoordinate)> = nodes
.iter()
.map(|tn| {
(
@@ -1007,7 +1007,7 @@ async fn test_routing_source_only_coords_100_nodes() {
.unwrap_or(0);
// Collect all coords for injection
let all_coords: Vec<(NodeAddr, crate::tree::TreeCoordinate)> = nodes
let all_coords: Vec<(NodeAddr, crate::proto::stp::TreeCoordinate)> = nodes
.iter()
.map(|tn| {
(
@@ -1361,7 +1361,7 @@ fn test_parent_loss_reparent_invalidates_coord_cache() {
TreeCoordinate::from_addrs(vec![alt, root]).unwrap(),
);
// Adopt `parent`; our coords become [my_addr, parent, root], root = `root`.
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().set_parent(parent, 1, 1000, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());
assert_eq!(node.tree_state().root(), &root);
@@ -1416,7 +1416,7 @@ fn test_parent_loss_selfroot_invalidates_coord_cache() {
ParentDeclaration::new(parent, old_root, 1, 1000),
TreeCoordinate::from_addrs(vec![parent, old_root]).unwrap(),
);
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().set_parent(parent, 1, 1000, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());

View File

@@ -6,7 +6,8 @@ use crate::node::tests::spanning_tree::{
TestNode, cleanup_nodes, generate_random_edges, lock_large_network_test,
process_available_packets, run_tree_test, run_tree_test_with_mtus, verify_tree_convergence,
};
use crate::protocol::{SessionAck, SessionDatagram};
use crate::proto::fsp::SessionAck;
use crate::proto::link::SessionDatagram;
/// Populate all nodes' coordinate caches with each other's coords.
///
@@ -18,7 +19,7 @@ fn populate_all_coord_caches(nodes: &mut [TestNode]) {
.unwrap()
.as_millis() as u64;
let all_coords: Vec<(NodeAddr, crate::tree::TreeCoordinate)> = nodes
let all_coords: Vec<(NodeAddr, crate::proto::stp::TreeCoordinate)> = nodes
.iter()
.map(|tn| {
(
@@ -656,7 +657,7 @@ async fn test_session_100_nodes() {
let mut tun_receivers: Vec<mpsc::Receiver<Vec<u8>>> = Vec::with_capacity(NUM_NODES);
for tn in nodes.iter_mut() {
let (tx, rx) = mpsc::channel();
tn.node.tun_tx = Some(tx);
tn.node.supervisor.tun_tx = Some(tx);
tun_receivers.push(rx);
}
@@ -1001,7 +1002,7 @@ fn build_ipv6_packet(
#[test]
fn test_identity_cache_populated_on_promote() {
use crate::peer::PromotionResult;
use crate::proto::fmp::PromotionResult;
let mut node = make_node();
let transport_id = TransportId::new(1);
@@ -1072,7 +1073,7 @@ async fn test_tun_outbound_established_session() {
// Install TUN receiver on Node 1
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[1].node.tun_tx = Some(tun_tx);
nodes[1].node.supervisor.tun_tx = Some(tun_tx);
// Build and inject an IPv6 packet
let test_payload = b"data-plane-test-12345";
@@ -1116,7 +1117,7 @@ async fn test_tun_outbound_triggers_session_initiation() {
// Install TUN receiver on Node 1
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[1].node.tun_tx = Some(tun_tx);
nodes[1].node.supervisor.tun_tx = Some(tun_tx);
// Build and inject an IPv6 packet (identity cache populated at peer promotion)
let test_payload = b"trigger-session-test";
@@ -1169,7 +1170,7 @@ async fn test_tun_outbound_unknown_destination() {
// Install TUN receiver on Node 0 (for ICMPv6 response)
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[0].node.tun_tx = Some(tun_tx);
nodes[0].node.supervisor.tun_tx = Some(tun_tx);
let src_fips = crate::FipsAddress::from_node_addr(nodes[0].node.node_addr());
@@ -1221,7 +1222,7 @@ async fn test_tun_outbound_3node_forwarded() {
// Install TUN receiver on Node 2
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[2].node.tun_tx = Some(tun_tx);
nodes[2].node.supervisor.tun_tx = Some(tun_tx);
// Build and inject an IPv6 packet (triggers session initiation to Node 2)
let test_payload = b"forwarded-data-plane";
@@ -1266,7 +1267,7 @@ async fn test_tun_outbound_pending_queue_flush() {
// Install TUN receiver on Node 1
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[1].node.tun_tx = Some(tun_tx);
nodes[1].node.supervisor.tun_tx = Some(tun_tx);
// Send 5 packets before any session exists
let mut packets = Vec::new();
@@ -1911,13 +1912,13 @@ async fn test_tun_outbound_path_mtu_generates_ptb() {
let entry = nodes[0].node.get_session_mut(&node1_addr).unwrap();
let mmp = entry.mmp_mut().unwrap();
mmp.path_mtu
.apply_notification(reduced_mtu, std::time::Instant::now());
.apply_notification(reduced_mtu, crate::time::mono_ms());
assert_eq!(mmp.path_mtu.current_mtu(), reduced_mtu);
}
// Install TUN receiver on source node to capture ICMPv6 PTB
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
nodes[0].node.tun_tx = Some(tun_tx);
nodes[0].node.supervisor.tun_tx = Some(tun_tx);
// Build an IPv6 packet that fits local MTU but exceeds path MTU
let reduced_ipv6_mtu = crate::upper::icmp::effective_ipv6_mtu(reduced_mtu) as usize;
@@ -1974,7 +1975,7 @@ async fn test_tun_outbound_path_mtu_generates_ptb() {
// Verify a packet that fits within path MTU passes through (no PTB)
let (tun_tx2, tun_rx2) = std::sync::mpsc::channel();
nodes[0].node.tun_tx = Some(tun_tx2);
nodes[0].node.supervisor.tun_tx = Some(tun_tx2);
let fitting_payload = vec![0u8; reduced_ipv6_mtu - 41]; // fits within path MTU
let fitting_packet = build_ipv6_packet(&src_fips, &dst_fips, &fitting_payload);
assert!(fitting_packet.len() <= reduced_ipv6_mtu);
@@ -2113,7 +2114,7 @@ async fn test_multihop_pmtud_heterogeneous_mtu() {
// should check PathMtuState and generate ICMPv6 PTB on TUN instead
// of forwarding.
let (tun_tx2, tun_rx2) = std::sync::mpsc::channel();
nodes[0].node.tun_tx = Some(tun_tx2);
nodes[0].node.supervisor.tun_tx = Some(tun_tx2);
nodes[0].node.handle_tun_outbound(ipv6_packet.clone()).await;
@@ -2157,7 +2158,7 @@ async fn test_multihop_pmtud_heterogeneous_mtu() {
// Verify a fitting packet still passes through without PTB
let (tun_tx3, tun_rx3) = std::sync::mpsc::channel();
nodes[0].node.tun_tx = Some(tun_tx3);
nodes[0].node.supervisor.tun_tx = Some(tun_tx3);
let fitting_payload = vec![0xCDu8; 600 - 40]; // 600-byte IPv6 packet, well within 694
let fitting_packet = build_ipv6_packet(&src_fips, &dst_fips, &fitting_payload);

View File

@@ -5,9 +5,10 @@
//! reused by bloom filter tests.
use super::*;
use crate::protocol::TreeAnnounce;
use crate::node::tree::sign_declaration;
use crate::proto::stp::TreeAnnounce;
use crate::proto::stp::{CoordEntry, ParentDeclaration, TreeCoordinate};
use crate::transport::loopback::{LoopbackRegistry, LoopbackTransport, new_registry};
use crate::tree::{CoordEntry, ParentDeclaration, TreeCoordinate};
static LARGE_NETWORK_TEST_LOCK: std::sync::LazyLock<tokio::sync::Mutex<()>> =
std::sync::LazyLock::new(|| tokio::sync::Mutex::new(()));
@@ -104,7 +105,7 @@ pub(super) async fn make_test_node_with_mtu(mtu: u16) -> TestNode {
/// Sends msg1 over UDP. The drain loop will handle msg1 processing,
/// msg2 response, and subsequent TreeAnnounce exchange.
pub(super) async fn initiate_handshake(nodes: &mut [TestNode], i: usize, j: usize) {
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
// Extract responder info before mutably borrowing initiator
let responder_addr = nodes[j].addr.clone();
@@ -266,7 +267,7 @@ pub(super) fn print_tree_snapshot(label: &str, nodes: &[TestNode]) {
///
/// Returns the number of packets processed.
pub(super) async fn process_available_packets(nodes: &mut [TestNode]) -> usize {
use crate::node::wire::{
use crate::proto::fmp::wire::{
COMMON_PREFIX_SIZE, CommonPrefix, FMP_VERSION, PHASE_ESTABLISHED, PHASE_MSG1, PHASE_MSG2,
};
@@ -891,7 +892,7 @@ async fn test_rejects_tree_announce_with_inconsistent_root() {
// sequence/timestamp so the announce would be acceptable on freshness
// grounds if its ancestry semantics were valid.
let mut declaration = ParentDeclaration::new(a_addr, fake_parent, 99, 12345);
declaration.sign(nodes[0].node.identity()).unwrap();
sign_declaration(&mut declaration, nodes[0].node.identity()).unwrap();
let announce = TreeAnnounce::new(
declaration,
@@ -970,18 +971,15 @@ async fn test_tree_announce_repushed_on_root_disagreement() {
// it had never processed the root's attaching announce. The child's
// advertised root is now itself, disagreeing with the root's view, and its
// own periodic re-evaluation cannot recover it (single peer).
nodes[child_idx].node.tree_state_mut().become_root();
nodes[child_idx].node.tree_state_mut().become_root(1000);
nodes[child_idx]
.node
.tree_state_mut()
.remove_peer(&root_addr);
{
let identity = nodes[child_idx].node.identity().clone();
nodes[child_idx]
.node
.tree_state_mut()
.sign_declaration(&identity)
.unwrap();
let decl_mut = nodes[child_idx].node.tree_state_mut().my_declaration_mut();
sign_declaration(decl_mut, &identity).unwrap();
}
assert!(nodes[child_idx].node.tree_state().is_root());
@@ -1029,3 +1027,369 @@ async fn test_tree_announce_repushed_on_root_disagreement() {
cleanup_nodes(&mut nodes).await;
}
// ===== Direct handler characterization tests =====
//
// These drive `handle_tree_announce` directly to pin the individual
// classification and validation arms that the aggregate convergence suite
// only exercises indirectly: the four validation rejects (addr-mismatch,
// sig-fail, stale, unknown-peer) and the self-root / loop-drop /
// same-parent ancestry-update transitions. `run_tree_test(2, ..)` supplies
// two handshaked peers (so the sender's pubkey is known); the transition
// tests then force the receiver's local tree state into the precise shape
// each arm requires — the same `tree_state_mut()` seam
// `test_tree_announce_repushed_on_root_disagreement` uses above.
//
// `make_node_addr(0)` is the all-zero address, strictly smaller than every
// randomly-generated real node address, so it is used as a synthetic global
// root that keeps forced ancestries valid (advertised root = path minimum).
/// A TreeAnnounce whose declared `node_addr` does not match the sending peer
/// must be rejected (addr-mismatch) before any state mutation. The addr-match
/// gate precedes signature verification, so a harvested-but-unrelated
/// signature is enough to reach it.
#[tokio::test]
async fn test_tree_announce_rejects_addr_mismatch() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let a_addr = *nodes[0].node.node_addr();
let sig = nodes[0].node.identity().sign(&[0u8; 48]).to_byte_array();
let bogus = make_node_addr(200);
let declaration = ParentDeclaration::with_signature(bogus, bogus, 5, 2000, sig);
let announce = TreeAnnounce::new(
declaration,
TreeCoordinate::from_addrs(vec![bogus]).unwrap(),
);
let encoded = announce.encode().unwrap();
let mismatch_before = nodes[1].node.metrics().tree.addr_mismatch.get();
let accepted_before = nodes[1].node.metrics().tree.accepted.get();
let root_before = *nodes[1].node.tree_state().root();
// Sender is the known peer a_addr, but the declaration claims `bogus`.
nodes[1]
.node
.handle_tree_announce(&a_addr, &encoded[1..])
.await;
assert_eq!(
nodes[1].node.metrics().tree.addr_mismatch.get(),
mismatch_before + 1
);
assert_eq!(nodes[1].node.metrics().tree.accepted.get(), accepted_before);
assert_eq!(*nodes[1].node.tree_state().root(), root_before);
cleanup_nodes(&mut nodes).await;
}
/// A TreeAnnounce whose declared node_addr matches the sender but whose
/// signature does not verify under the sender's pubkey must be rejected
/// (sig-fail) without mutating tree state.
#[tokio::test]
async fn test_tree_announce_rejects_bad_signature() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let a_addr = *nodes[0].node.node_addr();
// A valid A-signature, but over a *different* declaration, so verifying it
// against the forged declaration's signing bytes fails.
let mut signed_other = ParentDeclaration::new(a_addr, a_addr, 99, 88);
sign_declaration(&mut signed_other, nodes[0].node.identity()).unwrap();
let sig = *signed_other.signature().unwrap();
let forged = ParentDeclaration::with_signature(a_addr, a_addr, 5, 2000, sig);
let announce = TreeAnnounce::new(forged, TreeCoordinate::from_addrs(vec![a_addr]).unwrap());
let encoded = announce.encode().unwrap();
let sig_failed_before = nodes[1].node.metrics().tree.sig_failed.get();
let accepted_before = nodes[1].node.metrics().tree.accepted.get();
nodes[1]
.node
.handle_tree_announce(&a_addr, &encoded[1..])
.await;
assert_eq!(
nodes[1].node.metrics().tree.sig_failed.get(),
sig_failed_before + 1
);
assert_eq!(nodes[1].node.metrics().tree.accepted.get(), accepted_before);
cleanup_nodes(&mut nodes).await;
}
/// Replaying a peer's already-known declaration verbatim (same sequence) is
/// not fresher, so `update_peer` reports no change and the announce is counted
/// stale and ignored rather than accepted.
#[tokio::test]
async fn test_tree_announce_stale_ignored() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let a_addr = *nodes[0].node.node_addr();
let stored_decl = nodes[1]
.node
.tree_state()
.peer_declaration(&a_addr)
.expect("node 1 should hold A's declaration after convergence")
.clone();
let stored_coords = nodes[1]
.node
.tree_state()
.peer_coords(&a_addr)
.expect("node 1 should hold A's coordinates after convergence")
.clone();
let announce = TreeAnnounce::new(stored_decl, stored_coords);
let encoded = announce.encode().unwrap();
let stale_before = nodes[1].node.metrics().tree.stale.get();
let accepted_before = nodes[1].node.metrics().tree.accepted.get();
nodes[1]
.node
.handle_tree_announce(&a_addr, &encoded[1..])
.await;
assert_eq!(nodes[1].node.metrics().tree.stale.get(), stale_before + 1);
assert_eq!(nodes[1].node.metrics().tree.accepted.get(), accepted_before);
cleanup_nodes(&mut nodes).await;
}
/// A TreeAnnounce from a node that is not a known peer must be rejected
/// (unknown-peer) at the pubkey-lookup gate.
#[tokio::test]
async fn test_tree_announce_rejects_unknown_peer() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let unknown = make_node_addr(201);
let sig = nodes[0].node.identity().sign(&[0u8; 48]).to_byte_array();
let declaration = ParentDeclaration::with_signature(unknown, unknown, 1, 1000, sig);
let announce = TreeAnnounce::new(
declaration,
TreeCoordinate::from_addrs(vec![unknown]).unwrap(),
);
let encoded = announce.encode().unwrap();
let unknown_before = nodes[1].node.metrics().tree.unknown_peer.get();
nodes[1]
.node
.handle_tree_announce(&unknown, &encoded[1..])
.await;
assert_eq!(
nodes[1].node.metrics().tree.unknown_peer.get(),
unknown_before + 1
);
cleanup_nodes(&mut nodes).await;
}
/// A non-root node whose only visible root is larger than its own address must
/// self-promote to root. P (the smaller-addr node) is forced into a child of
/// its larger peer L rooted at a synthetic smaller root; when L then announces
/// itself as its own (larger) root, P's smallest visible root becomes L, so P
/// promotes itself.
#[tokio::test]
async fn test_tree_announce_self_root_promotion() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
// P must be the smaller-addr node (the one that should win root); L larger.
let (p_idx, l_idx) = if nodes[0].node.node_addr() < nodes[1].node.node_addr() {
(0, 1)
} else {
(1, 0)
};
let p_addr = *nodes[p_idx].node.node_addr();
let l_addr = *nodes[l_idx].node.node_addr();
let fake_root = make_node_addr(0);
// Force P into a non-root child of L rooted at fake_root: coords
// [P, L, fake_root]. P > fake_root, so recompute keeps it attached.
{
let ts = nodes[p_idx].node.tree_state_mut();
ts.remove_peer(&l_addr);
ts.update_peer(
ParentDeclaration::new(l_addr, fake_root, 1, 1000),
TreeCoordinate::from_addrs(vec![l_addr, fake_root]).unwrap(),
);
ts.set_parent(l_addr, 1, 1000, 1000);
ts.recompute_coords();
}
{
let identity = nodes[p_idx].node.identity().clone();
let decl_mut = nodes[p_idx].node.tree_state_mut().my_declaration_mut();
sign_declaration(decl_mut, &identity).unwrap();
}
assert!(!nodes[p_idx].node.tree_state().is_root());
assert_eq!(*nodes[p_idx].node.tree_state().root(), fake_root);
// L announces a fresh self-root (root = L > P).
let mut decl = ParentDeclaration::self_root(l_addr, 5, 2000);
sign_declaration(&mut decl, nodes[l_idx].node.identity()).unwrap();
let announce = TreeAnnounce::new(decl, TreeCoordinate::from_addrs(vec![l_addr]).unwrap());
let encoded = announce.encode().unwrap();
let switched_before = nodes[p_idx].node.metrics().tree.parent_switches.get();
nodes[p_idx]
.node
.handle_tree_announce(&l_addr, &encoded[1..])
.await;
assert!(
nodes[p_idx].node.tree_state().is_root(),
"P should self-promote to root when its only visible root is larger"
);
assert_eq!(*nodes[p_idx].node.tree_state().root(), p_addr);
assert_eq!(
nodes[p_idx].node.metrics().tree.parent_switches.get(),
switched_before + 1
);
cleanup_nodes(&mut nodes).await;
}
/// When our current parent's freshly announced ancestry comes to contain us, a
/// loop has formed and the parent must be dropped. P is forced into a child of
/// Q rooted at a synthetic root; Q then announces an ancestry [Q, P, root] that
/// runs back through P, so P detects the loop and (having no alternative) falls
/// back to self-root.
#[tokio::test]
async fn test_tree_announce_loop_detection_drops_parent() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let (p_idx, q_idx) = (0, 1);
let p_addr = *nodes[p_idx].node.node_addr();
let q_addr = *nodes[q_idx].node.node_addr();
let root = make_node_addr(0);
// Force P into a child of Q rooted at `root`: coords [P, Q, root].
{
let ts = nodes[p_idx].node.tree_state_mut();
ts.remove_peer(&q_addr);
ts.update_peer(
ParentDeclaration::new(q_addr, root, 1, 1000),
TreeCoordinate::from_addrs(vec![q_addr, root]).unwrap(),
);
ts.set_parent(q_addr, 1, 1000, 1000);
ts.recompute_coords();
}
{
let identity = nodes[p_idx].node.identity().clone();
let decl_mut = nodes[p_idx].node.tree_state_mut().my_declaration_mut();
sign_declaration(decl_mut, &identity).unwrap();
}
assert!(!nodes[p_idx].node.tree_state().is_root());
assert_eq!(
nodes[p_idx].node.tree_state().my_declaration().parent_id(),
&q_addr
);
// Q announces an ancestry that now runs through P (declaring P as its own
// parent): [Q, P, root]. Adopting it would form a loop.
let mut decl = ParentDeclaration::new(q_addr, p_addr, 5, 2000);
sign_declaration(&mut decl, nodes[q_idx].node.identity()).unwrap();
let announce = TreeAnnounce::new(
decl,
TreeCoordinate::from_addrs(vec![q_addr, p_addr, root]).unwrap(),
);
let encoded = announce.encode().unwrap();
let loop_before = nodes[p_idx].node.metrics().tree.loop_detected.get();
nodes[p_idx]
.node
.handle_tree_announce(&q_addr, &encoded[1..])
.await;
assert_eq!(
nodes[p_idx].node.metrics().tree.loop_detected.get(),
loop_before + 1
);
// No alternative parent remains, so P falls back to self-root.
assert!(nodes[p_idx].node.tree_state().is_root());
assert_eq!(*nodes[p_idx].node.tree_state().root(), p_addr);
cleanup_nodes(&mut nodes).await;
}
/// When our parent keeps the same root and depth but swaps a mid-chain
/// ancestor, we keep the parent yet must recompute our coordinates and
/// re-announce (the `old_addrs != new_addrs` gate). P is forced into
/// [P, Q, mid, root]; Q re-announces [Q, new_mid, root], leaving root and depth
/// unchanged while replacing the interior ancestor.
#[tokio::test]
async fn test_tree_announce_same_parent_ancestry_update() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let (p_idx, q_idx) = (0, 1);
let q_addr = *nodes[q_idx].node.node_addr();
let root = make_node_addr(0);
let mid = make_node_addr(1);
let new_mid = make_node_addr(2);
// Force P into a child of Q rooted at `root` via `mid`: [P, Q, mid, root].
{
let ts = nodes[p_idx].node.tree_state_mut();
ts.remove_peer(&q_addr);
ts.update_peer(
ParentDeclaration::new(q_addr, mid, 1, 1000),
TreeCoordinate::from_addrs(vec![q_addr, mid, root]).unwrap(),
);
ts.set_parent(q_addr, 1, 1000, 1000);
ts.recompute_coords();
}
{
let identity = nodes[p_idx].node.identity().clone();
let decl_mut = nodes[p_idx].node.tree_state_mut().my_declaration_mut();
sign_declaration(decl_mut, &identity).unwrap();
}
let depth_before = nodes[p_idx].node.tree_state().my_coords().depth();
assert_eq!(
nodes[p_idx].node.tree_state().my_declaration().parent_id(),
&q_addr
);
// Q keeps root and depth but swaps its mid-chain ancestor mid -> new_mid.
let mut decl = ParentDeclaration::new(q_addr, new_mid, 5, 2000);
sign_declaration(&mut decl, nodes[q_idx].node.identity()).unwrap();
let announce = TreeAnnounce::new(
decl,
TreeCoordinate::from_addrs(vec![q_addr, new_mid, root]).unwrap(),
);
let encoded = announce.encode().unwrap();
let ancestry_before = nodes[p_idx].node.metrics().tree.ancestry_changed.get();
nodes[p_idx]
.node
.handle_tree_announce(&q_addr, &encoded[1..])
.await;
assert_eq!(
nodes[p_idx].node.metrics().tree.ancestry_changed.get(),
ancestry_before + 1
);
// Same parent, same depth, but the recomputed path now runs through new_mid.
assert_eq!(
nodes[p_idx].node.tree_state().my_declaration().parent_id(),
&q_addr
);
assert_eq!(
nodes[p_idx].node.tree_state().my_coords().depth(),
depth_before
);
let path: Vec<NodeAddr> = nodes[p_idx]
.node
.tree_state()
.my_coords()
.node_addrs()
.copied()
.collect();
assert!(
path.contains(&new_mid),
"recomputed path should include the swapped-in ancestor"
);
assert!(
!path.contains(&mid),
"old mid-chain ancestor should be gone from the recomputed path"
);
cleanup_nodes(&mut nodes).await;
}

View File

@@ -1,6 +1,6 @@
use super::*;
use crate::discovery::nostr::{BootstrapEvent, NostrDiscovery};
use crate::peer::PromotionResult;
use crate::nostr::{BootstrapEvent, NostrRendezvous};
use crate::proto::fmp::PromotionResult;
use crate::transport::udp::UdpTransport;
use crate::transport::{TransportHandle, packet_channel};
use std::sync::Arc;
@@ -31,7 +31,7 @@ fn test_node_with_identity() {
fn test_node_with_identity_validates_config() {
let identity = Identity::generate();
let mut config = Config::new();
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
config.peers = vec![crate::config::PeerConfig {
npub: "npub1peer".to_string(),
via_nostr: true,
@@ -56,7 +56,7 @@ async fn test_nat_bootstrap_failure_falls_back_to_direct_udp_address() {
let peer_identity = Identity::generate();
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx.clone());
node.supervisor.packet_tx = Some(packet_tx.clone());
node.packet_rx = Some(packet_rx);
let transport_id = TransportId::new(1);
@@ -102,7 +102,7 @@ async fn test_try_peer_addresses_races_all_concrete_udp_candidates() {
let peer_identity = Identity::generate();
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx.clone());
node.supervisor.packet_tx = Some(packet_tx.clone());
node.packet_rx = Some(packet_rx);
let transport_id = TransportId::new(1);
@@ -151,13 +151,16 @@ async fn test_try_peer_addresses_races_all_concrete_udp_candidates() {
#[tokio::test]
async fn test_node_state_transitions() {
let mut node = make_node();
// A transport-less node now resolves to `Failed` on start, so exercise
// state transitions with a genuinely healthy node.
let mut node = make_healthy_node();
assert!(!node.is_running());
assert!(node.state().can_start());
node.start().await.unwrap();
assert!(node.is_running());
assert_eq!(node.state(), NodeState::Running);
assert!(!node.state().can_start());
node.stop().await.unwrap();
@@ -165,15 +168,67 @@ async fn test_node_state_transitions() {
assert_eq!(node.state(), NodeState::Stopped);
}
#[tokio::test]
async fn test_transportless_start_fails_and_publishes_failed() {
// The intended behavioral change: a node with zero
// transports up cannot serve, so `start()` returns `NoOperationalTransports`
// and leaves the published state at `Failed` (not operational, not
// restartable in-process).
let mut node = make_node();
assert!(node.state().can_start());
let result = node.start().await;
assert!(matches!(result, Err(NodeError::NoOperationalTransports)));
assert_eq!(node.state(), NodeState::Failed);
assert!(!node.is_running());
assert!(!node.state().is_operational());
assert!(!node.state().can_start());
}
#[tokio::test]
async fn test_drain_publishes_draining_state() {
let mut node = make_healthy_node();
node.start().await.unwrap();
assert_eq!(node.state(), NodeState::Running);
assert!(node.state().is_operational());
// Enter the bounded drain in place: publishes the operator-visible
// `Draining` state (not operational) without tearing down.
node.enter_drain().await;
assert_eq!(node.state(), NodeState::Draining);
assert!(!node.state().is_operational());
// `Draining` is neither startable nor externally stoppable; the daemon
// drain finishes via the supervisor's `DrainDeadlineElapsed`, not `stop()`.
assert!(!node.state().can_start());
assert!(!node.state().can_stop());
// Finishing shutdown from `Draining` tears down to `Stopped`.
node.finish_shutdown().await;
assert_eq!(node.state(), NodeState::Stopped);
}
#[tokio::test]
async fn test_immediate_stop_never_publishes_draining() {
let mut node = make_healthy_node();
node.start().await.unwrap();
assert_eq!(node.state(), NodeState::Running);
// The immediate stop() path (used by tests and the stop-now path)
// transitions Running → Stopping → Stopped and never enters `Draining`.
node.stop().await.unwrap();
assert_eq!(node.state(), NodeState::Stopped);
assert_ne!(node.state(), NodeState::Draining);
}
#[tokio::test]
async fn test_node_start_does_not_wait_for_nostr_relay_startup() {
let mut config = Config::new();
config.node.control.enabled = false;
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.advertise = true;
config.node.discovery.nostr.policy = crate::config::NostrDiscoveryPolicy::Open;
config.node.discovery.nostr.advert_relays = vec!["wss://127.0.0.1:9".to_string()];
config.node.discovery.nostr.dm_relays = vec!["wss://127.0.0.1:9".to_string()];
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.advertise = true;
config.node.rendezvous.nostr.policy = crate::config::NostrRendezvousPolicy::Open;
config.node.rendezvous.nostr.advert_relays = vec!["wss://127.0.0.1:9".to_string()];
config.node.rendezvous.nostr.dm_relays = vec!["wss://127.0.0.1:9".to_string()];
config.transports.udp = crate::config::TransportInstances::Single(crate::config::UdpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
advertise_on_nostr: Some(true),
@@ -189,14 +244,14 @@ async fn test_node_start_does_not_wait_for_nostr_relay_startup() {
.unwrap();
assert!(node.is_running());
assert!(node.nostr_discovery_handle().is_some());
assert!(node.nostr_rendezvous_handle().is_some());
node.stop().await.unwrap();
}
#[tokio::test]
async fn test_node_double_start() {
let mut node = make_node();
let mut node = make_healthy_node();
node.start().await.unwrap();
let result = node.start().await;
@@ -543,7 +598,7 @@ async fn test_node_rx_loop_requires_start() {
#[tokio::test]
async fn test_node_rx_loop_takes_channel() {
let mut node = make_node();
let mut node = make_healthy_node();
node.start().await.unwrap();
// packet_rx should be available after start
@@ -715,12 +770,17 @@ fn test_schedule_retry_creates_entry() {
let mut node = Node::new(config).unwrap();
assert!(node.retry_pending.is_empty());
assert!(node.peering.reconciler.retry_pending.is_empty());
node.schedule_retry(peer_node_addr, 1000);
node.note_handshake_timeout(peer_node_addr, 1000);
assert_eq!(node.retry_pending.len(), 1);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert_eq!(node.peering.reconciler.retry_pending.len(), 1);
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap();
assert_eq!(state.retry_count, 1);
assert!(
state.reconnect,
@@ -748,15 +808,25 @@ fn test_schedule_retry_increments() {
let mut node = Node::new(config).unwrap();
// First failure
node.schedule_retry(peer_node_addr, 1000);
node.note_handshake_timeout(peer_node_addr, 1000);
assert_eq!(
node.retry_pending.get(&peer_node_addr).unwrap().retry_count,
node.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap()
.retry_count,
1
);
// Second failure
node.schedule_retry(peer_node_addr, 11_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
node.note_handshake_timeout(peer_node_addr, 11_000);
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap();
assert_eq!(state.retry_count, 2);
// backoff_ms(5000) with retry_count=2 = 5000 * 4 = 20000
assert_eq!(state.retry_after_ms, 11_000 + 20_000);
@@ -774,9 +844,9 @@ async fn test_process_pending_retries_is_budgeted_per_tick() {
let npub = identity.npub();
let peer_identity = PeerIdentity::from_npub(&npub).unwrap();
let node_addr = *peer_identity.node_addr();
node.retry_pending.insert(
node.peering.reconciler.retry_pending.insert(
node_addr,
crate::node::retry::RetryState {
crate::node::peering::retry::RetryState {
peer_config: crate::config::PeerConfig::new(npub, "udp", "10.0.0.2:2121"),
retry_count: 0,
retry_after_ms: 0,
@@ -787,12 +857,18 @@ async fn test_process_pending_retries_is_budgeted_per_tick() {
addrs.push(node_addr);
}
// The retry-dial tick runs only under a `Reconciling` gate (it fires from the
// rx loop, which spins only after start() reaches Running). Put the node in
// Running so the retry-dial budget — the property under test — is exercised.
node.supervisor.state = crate::node::NodeState::Running;
node.process_pending_retries(1).await;
let processed = addrs
.iter()
.filter(|addr| {
node.retry_pending
node.peering
.reconciler
.retry_pending
.get(addr)
.is_some_and(|state| state.retry_count > 0)
})
@@ -801,7 +877,7 @@ async fn test_process_pending_retries_is_budgeted_per_tick() {
assert_eq!(processed, 16);
assert_eq!(deferred, 4);
assert_eq!(node.retry_pending.len(), 20);
assert_eq!(node.peering.reconciler.retry_pending.len(), 20);
}
/// Test that auto-connect peers retry indefinitely (never exhaust).
@@ -822,20 +898,38 @@ fn test_schedule_retry_auto_connect_never_exhausts() {
let mut node = Node::new(config).unwrap();
// All attempts should keep the entry alive despite max_retries=2
node.schedule_retry(peer_node_addr, 1000);
assert!(node.retry_pending.contains_key(&peer_node_addr));
node.note_handshake_timeout(peer_node_addr, 1000);
assert!(
node.peering
.reconciler
.retry_pending
.contains_key(&peer_node_addr)
);
node.schedule_retry(peer_node_addr, 2000);
assert!(node.retry_pending.contains_key(&peer_node_addr));
node.note_handshake_timeout(peer_node_addr, 2000);
assert!(
node.peering
.reconciler
.retry_pending
.contains_key(&peer_node_addr)
);
// Attempt 3 would have exhausted before, but now retries indefinitely
node.schedule_retry(peer_node_addr, 3000);
node.note_handshake_timeout(peer_node_addr, 3000);
assert!(
node.retry_pending.contains_key(&peer_node_addr),
node.peering
.reconciler
.retry_pending
.contains_key(&peer_node_addr),
"Auto-connect peers should never exhaust retries"
);
assert_eq!(
node.retry_pending.get(&peer_node_addr).unwrap().retry_count,
node.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap()
.retry_count,
3
);
}
@@ -857,9 +951,9 @@ fn test_schedule_retry_disabled() {
let mut node = Node::new(config).unwrap();
node.schedule_retry(peer_node_addr, 1000);
node.note_handshake_timeout(peer_node_addr, 1000);
assert!(
node.retry_pending.is_empty(),
node.peering.reconciler.retry_pending.is_empty(),
"No retry should be scheduled when max_retries=0"
);
}
@@ -873,9 +967,9 @@ fn test_schedule_retry_ignores_non_autoconnect() {
// No peers configured at all
let mut node = make_node();
node.schedule_retry(peer_node_addr, 1000);
node.note_handshake_timeout(peer_node_addr, 1000);
assert!(
node.retry_pending.is_empty(),
node.peering.reconciler.retry_pending.is_empty(),
"No retry for unconfigured peer"
);
}
@@ -895,9 +989,9 @@ fn test_schedule_retry_skips_connected_peer() {
assert_eq!(node.peer_count(), 1);
// Scheduling a retry for an already-connected peer should be a no-op
node.schedule_retry(node_addr, 3000);
node.note_handshake_timeout(node_addr, 3000);
assert!(
node.retry_pending.is_empty(),
node.peering.reconciler.retry_pending.is_empty(),
"No retry for already-connected peer"
);
}
@@ -1050,7 +1144,7 @@ async fn update_peers_races_new_alternative_without_dropping_active_peer() {
config.peers = vec![old_peer.clone()];
let mut node = make_node_with(config);
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx.clone());
node.supervisor.packet_tx = Some(packet_tx.clone());
node.packet_rx = Some(packet_rx);
let transport_id = TransportId::new(1);
@@ -1125,28 +1219,30 @@ async fn test_nostr_traversal_failure_skips_connected_peer() {
node.promote_connection(link_id, peer_identity, 2000)
.unwrap();
let bootstrap = Arc::new(NostrDiscovery::new_for_test());
let bootstrap = Arc::new(NostrRendezvous::new_for_test());
bootstrap.push_event_for_test(BootstrapEvent::Failed {
peer_config: crate::config::PeerConfig::new(peer_identity.npub(), "udp", "127.0.0.1:9"),
reason: "stale traversal failure".to_string(),
});
node.nostr_discovery = Some(bootstrap.clone());
node.supervisor
.nostr_rendezvous
.set_engine(bootstrap.clone());
node.poll_nostr_discovery().await;
node.poll_nostr_rendezvous().await;
assert!(
bootstrap.failure_state_snapshot().is_empty(),
"stale failures for connected peers must not affect traversal cooldown"
);
assert!(
node.retry_pending.is_empty(),
node.peering.reconciler.retry_pending.is_empty(),
"stale failures for connected peers must not enqueue reconnect attempts"
);
}
#[tokio::test]
async fn test_nostr_traversal_established_skips_connected_peer() {
use crate::discovery::EstablishedTraversal;
use crate::nostr::EstablishedTraversal;
use std::net::UdpSocket;
let mut node = make_node();
@@ -1159,7 +1255,7 @@ async fn test_nostr_traversal_established_skips_connected_peer() {
let link_count = node.link_count();
let connection_count = node.connection_count();
let bootstrap = Arc::new(NostrDiscovery::new_for_test());
let bootstrap = Arc::new(NostrRendezvous::new_for_test());
let socket = UdpSocket::bind("127.0.0.1:0").expect("bind local UDP socket");
let remote_addr = "127.0.0.1:9999".parse().expect("parse remote addr");
bootstrap.push_event_for_test(BootstrapEvent::Established {
@@ -1170,9 +1266,11 @@ async fn test_nostr_traversal_established_skips_connected_peer() {
socket,
),
});
node.nostr_discovery = Some(bootstrap.clone());
node.supervisor
.nostr_rendezvous
.set_engine(bootstrap.clone());
node.poll_nostr_discovery().await;
node.poll_nostr_rendezvous().await;
assert_eq!(
node.link_count(),
@@ -1185,7 +1283,7 @@ async fn test_nostr_traversal_established_skips_connected_peer() {
"stale established handoff must not start a new handshake"
);
assert!(
node.retry_pending.is_empty(),
node.peering.reconciler.retry_pending.is_empty(),
"stale established handoff must not enqueue a reconnect"
);
}
@@ -1197,7 +1295,7 @@ async fn test_process_pending_retries_drops_expired_entries() {
let peer_npub = peer_identity.npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
let mut state = super::super::retry::RetryState::new(crate::config::PeerConfig::new(
let mut state = super::super::peering::retry::RetryState::new(crate::config::PeerConfig::new(
peer_npub,
"udp",
"127.0.0.1:9",
@@ -1205,12 +1303,22 @@ async fn test_process_pending_retries_drops_expired_entries() {
state.retry_after_ms = 0;
state.expires_at_ms = Some(1_000);
state.reconnect = true;
node.retry_pending.insert(peer_node_addr, state);
node.peering
.reconciler
.retry_pending
.insert(peer_node_addr, state);
// Retry-dial runs only under a `Reconciling` gate; put the node in Running so
// the expired-entry drop (the property under test) is reached.
node.supervisor.state = crate::node::NodeState::Running;
node.process_pending_retries(1_000).await;
assert!(
!node.retry_pending.contains_key(&peer_node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&peer_node_addr),
"expired retry entries should be dropped before retry processing"
);
}
@@ -1238,19 +1346,29 @@ fn test_schedule_reconnect_preserves_backoff() {
let mut node = Node::new(config).unwrap();
// Simulate two stale handshake timeouts incrementing the retry count.
node.schedule_retry(peer_node_addr, 1_000); // count=1, delay=10s
node.schedule_retry(peer_node_addr, 11_000); // count=2, delay=20s
node.note_handshake_timeout(peer_node_addr, 1_000); // count=1, delay=10s
node.note_handshake_timeout(peer_node_addr, 11_000); // count=2, delay=20s
{
let state = node.retry_pending.get(&peer_node_addr).unwrap();
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap();
assert_eq!(state.retry_count, 2, "Two failures should yield count=2");
}
// Now simulate a link-dead removal triggering schedule_reconnect.
// The existing retry entry (count=2) should be preserved and bumped to 3,
// NOT reset to 0 as it was before the fix.
node.schedule_reconnect(peer_node_addr, 31_000);
node.note_link_dead(peer_node_addr, 31_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap();
assert!(state.reconnect, "Entry should be marked as reconnect");
assert_eq!(
state.retry_count, 3,
@@ -1260,7 +1378,7 @@ fn test_schedule_reconnect_preserves_backoff() {
// With count=3, backoff should be 5s * 2^3 = 40s.
let base_ms = node.config().node.retry.base_interval_secs * 1000;
let max_ms = node.config().node.retry.max_backoff_secs * 1000;
let expected_delay = state.backoff_ms(base_ms, max_ms);
let expected_delay = crate::proto::fmp::backoff_ms(state.retry_count, base_ms, max_ms);
assert_eq!(
state.retry_after_ms,
31_000 + expected_delay,
@@ -1285,9 +1403,14 @@ fn test_schedule_reconnect_fresh_state() {
let mut node = Node::new(config).unwrap();
// No prior retry entry — first reconnect should use base delay.
node.schedule_reconnect(peer_node_addr, 1_000);
node.note_link_dead(peer_node_addr, 1_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.unwrap();
assert!(state.reconnect, "Entry should be marked as reconnect");
assert_eq!(
state.retry_count, 0,
@@ -1296,7 +1419,7 @@ fn test_schedule_reconnect_fresh_state() {
// Base delay: 5s * 2^0 = 5s
let base_ms = node.config().node.retry.base_interval_secs * 1000;
let max_ms = node.config().node.retry.max_backoff_secs * 1000;
let expected_delay = state.backoff_ms(base_ms, max_ms);
let expected_delay = crate::proto::fmp::backoff_ms(state.retry_count, base_ms, max_ms);
assert_eq!(state.retry_after_ms, 1_000 + expected_delay);
}
@@ -1308,7 +1431,7 @@ fn test_schedule_reconnect_fresh_state() {
/// decrypt failure, peer restart) all schedule reconnect.
#[test]
fn test_disconnect_schedules_reconnect() {
use crate::protocol::{Disconnect, DisconnectReason};
use crate::proto::fmp::{Disconnect, DisconnectReason};
let peer_identity = Identity::generate();
let peer_npub = peer_identity.npub();
@@ -1327,6 +1450,8 @@ fn test_disconnect_schedules_reconnect() {
node.handle_disconnect(&peer_node_addr, &payload);
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.expect("handle_disconnect should schedule reconnect for auto-connect peer");
@@ -1348,17 +1473,21 @@ fn test_promote_clears_retry_pending() {
let node_addr = *identity.node_addr();
// Simulate a retry entry existing for this peer
node.retry_pending.insert(
node.peering.reconciler.retry_pending.insert(
node_addr,
super::super::retry::RetryState::new(crate::config::PeerConfig::default()),
super::super::peering::retry::RetryState::new(crate::config::PeerConfig::default()),
);
assert_eq!(node.retry_pending.len(), 1);
assert_eq!(node.peering.reconciler.retry_pending.len(), 1);
node.add_connection(conn).unwrap();
node.promote_connection(link_id, identity, 2000).unwrap();
assert!(
!node.retry_pending.contains_key(&node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&node_addr),
"retry_pending should be cleared on successful promotion"
);
}
@@ -1384,12 +1513,15 @@ async fn test_initiate_peer_connections_schedules_retry_on_no_transport() {
));
let mut node = Node::new(config).unwrap();
assert!(node.retry_pending.is_empty());
assert!(node.peering.reconciler.retry_pending.is_empty());
node.initiate_peer_connections().await;
assert!(
node.retry_pending.contains_key(&peer_node_addr),
node.peering
.reconciler
.retry_pending
.contains_key(&peer_node_addr),
"startup peer-init failure must enqueue a retry so the peer can recover \
without a daemon restart"
);
@@ -1424,7 +1556,7 @@ async fn test_transport_mtu_returns_min_across_operational() {
// iteration order. This is the core ISSUE-2026-0011 regression test.
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let udp1 = make_udp_transport_with_mtu(1, 1497).await;
@@ -1461,7 +1593,7 @@ async fn test_transport_mtu_min_with_single_operational() {
// operational.
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let udp = make_udp_transport_with_mtu(1, 1452).await;
@@ -1484,7 +1616,7 @@ async fn test_transport_mtu_min_with_single_operational() {
async fn test_seed_path_mtu_inserts_when_empty() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let udp = make_udp_transport_with_mtu(1, 1452).await;
@@ -1517,7 +1649,7 @@ async fn test_seed_path_mtu_inserts_when_empty() {
async fn test_seed_path_mtu_keeps_tighter_existing_value() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let udp = make_udp_transport_with_mtu(1, 1452).await;
@@ -1557,7 +1689,7 @@ async fn test_seed_path_mtu_keeps_tighter_existing_value() {
async fn test_seed_path_mtu_tightens_looser_existing_value() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let udp = make_udp_transport_with_mtu(1, 1280).await;
@@ -1661,18 +1793,24 @@ async fn process_pending_retries_gated_at_capacity() {
let peer_identity = Identity::generate();
let peer_npub = peer_identity.npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
let mut state = super::super::retry::RetryState::new(crate::config::PeerConfig::new(
let mut state = super::super::peering::retry::RetryState::new(crate::config::PeerConfig::new(
peer_npub,
"udp",
"127.0.0.1:9",
));
state.retry_after_ms = 0;
state.reconnect = true;
node.retry_pending.insert(peer_node_addr, state);
node.peering
.reconciler
.retry_pending
.insert(peer_node_addr, state);
let before_peers = node.peer_count();
let before_connections = node.connection_count();
// Running gate so the admission short-circuit (not the startup gate) is what
// suppresses the dial — the fingerprint this test asserts on.
node.supervisor.state = crate::node::NodeState::Running;
node.process_pending_retries(1_000).await;
// At capacity: gate short-circuits before due-list collection. The
@@ -1682,6 +1820,8 @@ async fn process_pending_retries_gated_at_capacity() {
// (which fails without a registered transport), and the failure
// handler would call `schedule_retry`, bumping `retry_count` to 1.
let state = node
.peering
.reconciler
.retry_pending
.get(&peer_node_addr)
.expect("retry entry must be preserved when suppressed at capacity");
@@ -1708,14 +1848,14 @@ async fn process_pending_retries_gated_at_capacity() {
}
#[tokio::test]
async fn poll_nostr_discovery_established_gated_at_capacity() {
use crate::discovery::EstablishedTraversal;
async fn poll_nostr_rendezvous_established_gated_at_capacity() {
use crate::nostr::EstablishedTraversal;
use std::net::UdpSocket;
let mut node = make_node_with_max_peers(2);
inject_dummy_peers(&mut node, 2);
let bootstrap = Arc::new(NostrDiscovery::new_for_test());
let bootstrap = Arc::new(NostrRendezvous::new_for_test());
let socket = UdpSocket::bind("127.0.0.1:0").expect("bind local UDP socket");
let remote_addr = "127.0.0.1:9999".parse().expect("parse remote addr");
let peer_identity = Identity::generate();
@@ -1727,13 +1867,15 @@ async fn poll_nostr_discovery_established_gated_at_capacity() {
socket,
),
});
node.nostr_discovery = Some(bootstrap.clone());
node.supervisor
.nostr_rendezvous
.set_engine(bootstrap.clone());
let before_peers = node.peer_count();
let before_links = node.link_count();
let before_connections = node.connection_count();
node.poll_nostr_discovery().await;
node.poll_nostr_rendezvous().await;
assert_eq!(
node.peer_count(),
@@ -1753,11 +1895,11 @@ async fn poll_nostr_discovery_established_gated_at_capacity() {
}
#[test]
fn nostr_discovery_outbound_admission_atomic_roundtrip() {
fn nostr_rendezvous_outbound_admission_atomic_roundtrip() {
// Verifies the runtime-side plumbing for the two NAT-traversal gate
// points: the setter mutates the atomic and the (super-visible)
// reader observes the value the Node-side wiring would publish.
let bootstrap = NostrDiscovery::new_for_test();
let bootstrap = NostrRendezvous::new_for_test();
assert!(
bootstrap.outbound_admission_allowed(),
"default must allow (start unsaturated)"
@@ -1788,7 +1930,7 @@ async fn craft_and_send_msg1(
addr_b: std::net::SocketAddr,
timestamp_ms: u64,
) -> NodeAddr {
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
use crate::utils::index::SessionIndex;
let peer_b_identity = PeerIdentity::from_pubkey_full(node_b.identity().pubkey_full());

View File

@@ -3,15 +3,63 @@
//! Handles building, sending, and receiving TreeAnnounce messages,
//! including periodic root refresh and rate-limited propagation.
use std::collections::HashMap;
use std::collections::{BTreeMap, BTreeSet};
use crate::NodeAddr;
use crate::protocol::TreeAnnounce;
use secp256k1::XOnlyPublicKey;
use secp256k1::schnorr::Signature;
use crate::proto::stp::{ParentDeclaration, Stp, TreeAnnounce, TreeDecision, TreeError};
use crate::{Identity, NodeAddr};
use super::reject::TreeReject;
use super::{Node, NodeError};
use tracing::{debug, info, trace, warn};
/// Sign a node's own tree declaration, writing the 64-byte signature back into
/// it. The key-crypto boundary (§6): `proto::stp` owns the declaration data and
/// the pure `signing_bytes()` serialization; the shell owns the `secp256k1`
/// sign (`Identity::sign` hashes with SHA-256 internally). Mirrors discovery's
/// shell-side proof signing.
pub(super) fn sign_declaration(
decl: &mut ParentDeclaration,
identity: &Identity,
) -> Result<(), TreeError> {
if identity.node_addr() != decl.node_addr() {
return Err(TreeError::InvalidSignature(*decl.node_addr()));
}
let signature = identity.sign(&decl.signing_bytes());
decl.set_signature(signature.to_byte_array());
Ok(())
}
/// Verify a peer's tree declaration signature against their pubkey. The shell
/// side of the key-crypto boundary (§6): runs the `sha2` hash + `secp256k1`
/// schnorr verification over the in-core declaration's `signing_bytes()`.
pub(super) fn verify_declaration(
decl: &ParentDeclaration,
pubkey: &XOnlyPublicKey,
) -> Result<(), TreeError> {
let sig_bytes = decl
.signature()
.ok_or(TreeError::InvalidSignature(*decl.node_addr()))?;
let signature = Signature::from_slice(sig_bytes)
.map_err(|_| TreeError::InvalidSignature(*decl.node_addr()))?;
let secp = secp256k1::Secp256k1::verification_only();
let hash = signing_hash(decl);
secp.verify_schnorr(&signature, &hash, pubkey)
.map_err(|_| TreeError::InvalidSignature(*decl.node_addr()))
}
/// Compute the SHA-256 hash of a declaration's signing bytes (shell side, §6).
fn signing_hash(decl: &ParentDeclaration) -> [u8; 32] {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(decl.signing_bytes());
hasher.finalize().into()
}
impl Node {
/// Build a TreeAnnounce from our current tree state.
fn build_tree_announce(&self) -> Result<TreeAnnounce, NodeError> {
@@ -163,7 +211,7 @@ impl Node {
return;
}
if let Err(e) = announce.declaration.verify(&pubkey) {
if let Err(e) = verify_declaration(&announce.declaration, &pubkey) {
self.metrics().tree.sig_failed.inc();
warn!(
from = %self.peer_display_name(from),
@@ -245,7 +293,7 @@ impl Node {
// receive path and is naturally bounded by the per-peer 500 ms
// tree-announce rate limiter, so it does not storm during normal
// convergence (it stops as soon as the peer adopts our root).
if *announce.ancestry.root_id() > *self.tree_state.root()
if Stp::should_echo(announce.ancestry.root_id(), self.tree_state.root())
&& let Err(e) = self.send_tree_announce_to_peer(from).await
{
debug!(
@@ -267,107 +315,137 @@ impl Node {
// Re-evaluate parent selection with current link costs.
// Exclude peers without MMP RTT data — they are not yet eligible
// as parent candidates (prevents oscillation from optimistic defaults).
let peer_costs: HashMap<NodeAddr, f64> = self
let peer_costs: BTreeMap<NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// No peers are excluded from parent candidacy on this branch; the
// non-full/leaf skip is a next-only shell refinement.
let skip: BTreeSet<NodeAddr> = BTreeSet::new();
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
// recompute_coords may demote to self_root if the new path would be
// invalid; sign AFTER recompute so the signature covers the final
// declaration.
self.tree_state.recompute_coords();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
warn!(error = %e, "Failed to sign declaration after parent switch");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
// Monotonic ms for the flap-dampening / hold-down timers (distinct from
// the wall-clock `now_ms` above used for the peer's tree position). Read
// once and threaded into classify + the state mutators.
let mono_now_ms = crate::time::mono_ms();
// Compute the flap-dampening / hold-down veto at the edge; the classify core
// is clock-free and consumes only this pre-computed verdict.
let switch_suppressed = self.tree_state.is_switch_suppressed(mono_now_ms);
match Stp::classify_announce(
&self.tree_state,
*from,
&peer_costs,
&skip,
switch_suppressed,
) {
TreeDecision::Switch {
new_parent,
new_seq,
} => {
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
let flap_dampened =
self.tree_state
.set_parent(new_parent, new_seq, timestamp, mono_now_ms);
// recompute_coords may demote to self_root if the new path would be
// invalid; sign AFTER recompute so the signature covers the final
// declaration.
self.tree_state.recompute_coords();
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after parent switch");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
new_seq = new_seq,
new_root = %self.tree_state.root(),
depth = self.tree_state.my_coords().depth(),
"Parent switched, invalidated downstream coord cache entries, announcing to all peers"
);
if flap_dampened {
self.metrics().tree.flap_dampened.inc();
warn!("Flap dampening engaged: excessive parent switches detected");
}
self.send_tree_announce_to_all().await;
// Tree structure changed — trigger bloom filter exchange with all peers
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
new_seq = new_seq,
new_root = %self.tree_state.root(),
depth = self.tree_state.my_coords().depth(),
"Parent switched, invalidated downstream coord cache entries, announcing to all peers"
);
if flap_dampened {
self.metrics().tree.flap_dampened.inc();
warn!("Flap dampening engaged: excessive parent switches detected");
TreeDecision::SelfRoot => {
// Self is the smallest visible NodeAddr — promote to root rather
// than continuing to advertise a stale ancestry rooted elsewhere.
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state.become_root(timestamp);
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign self-root declaration");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
"Self-promoted to root: smallest visible NodeAddr"
);
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
}
self.send_tree_announce_to_all().await;
// Tree structure changed — trigger bloom filter exchange with all peers
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root() && self.tree_state.should_be_root() {
// Self is the smallest visible NodeAddr — promote to root rather
// than continuing to advertise a stale ancestry rooted elsewhere.
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state.become_root();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
warn!(error = %e, "Failed to sign self-root declaration");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
"Self-promoted to root: smallest visible NodeAddr"
);
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root()
&& *self.tree_state.my_declaration().parent_id() == *from
{
// Check for loop: if parent's ancestry now contains us, drop parent
if let Some(parent_coords) = self.tree_state.peer_coords(from)
&& parent_coords.contains(self.identity().node_addr())
{
TreeDecision::LoopDrop => {
self.metrics().tree.loop_detected.inc();
warn!(
parent = %self.peer_display_name(from),
"Parent ancestry contains us — loop detected, dropping parent"
);
let peer_costs: HashMap<NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
if self.tree_state.handle_parent_lost(&peer_costs) {
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
if self
.tree_state
.handle_parent_lost(&peer_costs, timestamp, mono_now_ms)
{
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after loop detection");
self.metrics()
.tree
@@ -380,77 +458,85 @@ impl Node {
.invalidate_via_node(our_identity.node_addr());
self.coord_cache
.invalidate_other_roots(self.tree_state.root());
self.reset_discovery_backoff();
self.reset_lookup_backoff();
self.send_tree_announce_to_all().await;
}
return;
}
TreeDecision::AncestryUpdate { parent, new_seq } => {
// Our parent's ancestry changed but we're keeping the same parent.
// Recompute our own coordinates (which derive from parent's ancestry)
// and re-announce so downstream nodes stay current.
//
// Compare the full address path (not just root + depth) so that a
// mid-chain ancestor swap also triggers re-announce. A reroute that
// replaces an interior ancestor without changing the root or the
// path length leaves both `root` and `depth` unchanged but still
// alters our coords; downstream peers must learn the new path or
// they will route into a phantom intermediate that no longer
// exists on our parent's tree.
let old_root = *self.tree_state.root();
let old_depth = self.tree_state.my_coords().depth();
let old_addrs: Vec<NodeAddr> =
self.tree_state.my_coords().node_addrs().copied().collect();
// Our parent's ancestry changed but we're keeping the same parent.
// Recompute our own coordinates (which derive from parent's ancestry)
// and re-announce so downstream nodes stay current.
//
// Compare the full address path (not just root + depth) so that a
// mid-chain ancestor swap also triggers re-announce. A reroute that
// replaces an interior ancestor without changing the root or the
// path length leaves both `root` and `depth` unchanged but still
// alters our coords; downstream peers must learn the new path or
// they will route into a phantom intermediate that no longer
// exists on our parent's tree.
let old_root = *self.tree_state.root();
let old_depth = self.tree_state.my_coords().depth();
let old_addrs: Vec<NodeAddr> =
self.tree_state.my_coords().node_addrs().copied().collect();
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state
.set_parent(parent, new_seq, timestamp, mono_now_ms);
self.tree_state.recompute_coords();
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after parent update");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_lookup_backoff();
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state.set_parent(*from, new_seq, timestamp);
self.tree_state.recompute_coords();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
warn!(error = %e, "Failed to sign declaration after parent update");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
let new_addrs: Vec<NodeAddr> =
self.tree_state.my_coords().node_addrs().copied().collect();
if old_addrs != new_addrs {
self.metrics().tree.ancestry_changed.inc();
info!(
parent = %self.peer_display_name(from),
old_root = %old_root,
new_root = %self.tree_state.root(),
old_depth = old_depth,
new_depth = self.tree_state.my_coords().depth(),
"Parent ancestry changed, re-announcing"
);
self.send_tree_announce_to_all().await;
// Bloom contents do not depend on path structure, only on
// identity sets. Our parent_id is unchanged in this branch,
// so our tree-peer set is unchanged and our outgoing filter
// content is unchanged. Use mark_changed_peers, which
// checks for actual content delta against last_sent_filters,
// instead of mark_all_updates_needed, which marks
// unconditionally regardless of whether content changed.
let peer_addrs: Vec<NodeAddr> = self.peers.keys().copied().collect();
let peer_filters = self.peer_inbound_filters();
self.bloom_state
.mark_changed_peers(from, &peer_addrs, &peer_filters);
}
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
let new_addrs: Vec<NodeAddr> =
self.tree_state.my_coords().node_addrs().copied().collect();
if old_addrs != new_addrs {
self.metrics().tree.ancestry_changed.inc();
info!(
parent = %self.peer_display_name(from),
old_root = %old_root,
new_root = %self.tree_state.root(),
old_depth = old_depth,
new_depth = self.tree_state.my_coords().depth(),
"Parent ancestry changed, re-announcing"
);
self.send_tree_announce_to_all().await;
// Bloom contents do not depend on path structure, only on
// identity sets. Our parent_id is unchanged in this branch,
// so our tree-peer set is unchanged and our outgoing filter
// content is unchanged. Use mark_changed_peers, which
// checks for actual content delta against last_sent_filters,
// instead of mark_all_updates_needed, which marks
// unconditionally regardless of whether content changed.
let peer_addrs: Vec<NodeAddr> = self.peers.keys().copied().collect();
let peer_filters = self.peer_inbound_filters();
self.bloom_state
.mark_changed_peers(from, &peer_addrs, &peer_filters);
TreeDecision::NoChange => {}
// classify_announce never yields PeriodicRebroadcast (the periodic
// path's no-change tail) nor ParentLost (the removal drive's outcome).
TreeDecision::PeriodicRebroadcast | TreeDecision::ParentLost => {
unreachable!("classify_announce yields neither PeriodicRebroadcast nor ParentLost")
}
}
}
@@ -491,97 +577,135 @@ impl Node {
self.last_parent_reeval = Some(now);
let peer_costs: HashMap<NodeAddr, f64> = self
let peer_costs: BTreeMap<NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
// No peers are excluded from parent candidacy on this branch; the
// non-full/leaf skip is a next-only shell refinement.
let skip: BTreeSet<NodeAddr> = BTreeSet::new();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Monotonic ms for the flap-dampening / hold-down timers, read once and
// threaded into classify + the state mutators.
let mono_now_ms = crate::time::mono_ms();
// Compute the flap-dampening / hold-down veto at the edge; the classify core
// is clock-free and consumes only this pre-computed verdict.
let switch_suppressed = self.tree_state.is_switch_suppressed(mono_now_ms);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
self.tree_state.recompute_coords();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
warn!(error = %e, "Failed to sign declaration after periodic parent re-eval");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
match Stp::classify_periodic(&self.tree_state, &peer_costs, &skip, switch_suppressed) {
TreeDecision::Switch {
new_parent,
new_seq,
} => {
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
let flap_dampened =
self.tree_state
.set_parent(new_parent, new_seq, timestamp, mono_now_ms);
self.tree_state.recompute_coords();
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after periodic parent re-eval");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
new_seq = new_seq,
new_root = %self.tree_state.root(),
depth = self.tree_state.my_coords().depth(),
trigger = "periodic",
"Parent switched via periodic cost re-evaluation"
);
if flap_dampened {
self.metrics().tree.flap_dampened.inc();
warn!("Flap dampening engaged: excessive parent switches detected");
}
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
}
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
new_seq = new_seq,
new_root = %self.tree_state.root(),
depth = self.tree_state.my_coords().depth(),
trigger = "periodic",
"Parent switched via periodic cost re-evaluation"
);
if flap_dampened {
self.metrics().tree.flap_dampened.inc();
warn!("Flap dampening engaged: excessive parent switches detected");
TreeDecision::SelfRoot => {
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state.become_root(timestamp);
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign self-root declaration in periodic reeval");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
}
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
trigger = "periodic",
"Self-promoted to root in periodic reeval: smallest visible NodeAddr"
);
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
}
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root() && self.tree_state.should_be_root() {
// Clone identity up front to avoid a split borrow against the
// &mut self.tree_state / &mut self.coord_cache calls below (cold path).
let our_identity = self.identity().clone();
self.tree_state.become_root();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
warn!(error = %e, "Failed to sign self-root declaration in periodic reeval");
self.metrics()
.tree
.record_reject(TreeReject::OutboundSignFailed);
return;
TreeDecision::PeriodicRebroadcast => {
// Periodic re-broadcast on no-change: makes TreeAnnounce
// distribution eventually-consistent. Receivers coalesce
// by sequence via ParentDeclaration::is_fresher_than and
// short-circuit at the `if !updated` gate in
// handle_tree_announce; the per-peer 500 ms rate-limiter
// never blocks at this 60 s cadence. Closes the cross-init
// in-flight loss recovery gap where the swap window can
// strand one side's announce on a session-index the other
// side cannot decrypt.
trace!(
seq = self.tree_state.my_declaration().sequence(),
root = %self.tree_state.root(),
"Periodic TreeAnnounce re-broadcast (no state change)"
);
self.send_tree_announce_to_all().await;
}
// classify_periodic never yields these: a periodic tick has no
// announcing peer, so the same-parent loop-drop / ancestry-update
// arms cannot arise, the no-change tail is PeriodicRebroadcast, and
// ParentLost is the removal drive's outcome.
TreeDecision::LoopDrop
| TreeDecision::AncestryUpdate { .. }
| TreeDecision::ParentLost
| TreeDecision::NoChange => {
unreachable!(
"classify_periodic yields only Switch / SelfRoot / PeriodicRebroadcast"
)
}
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
trigger = "periodic",
"Self-promoted to root in periodic reeval: smallest visible NodeAddr"
);
self.send_tree_announce_to_all().await;
let all_peers: Vec<NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else {
// Periodic re-broadcast on no-change: makes TreeAnnounce
// distribution eventually-consistent. Receivers coalesce
// by sequence via ParentDeclaration::is_fresher_than and
// short-circuit at the `if !updated` gate in
// handle_tree_announce; the per-peer 500 ms rate-limiter
// never blocks at this 60 s cadence. Closes the cross-init
// in-flight loss recovery gap where the swap window can
// strand one side's announce on a session-index the other
// side cannot decrypt.
trace!(
seq = self.tree_state.my_declaration().sequence(),
root = %self.tree_state.root(),
"Periodic TreeAnnounce re-broadcast (no state change)"
);
self.send_tree_announce_to_all().await;
}
}
@@ -597,20 +721,50 @@ impl Node {
self.tree_state.remove_peer(node_addr);
if was_parent {
self.metrics().tree.parent_losses.inc();
let peer_costs: HashMap<NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
let changed = self.tree_state.handle_parent_lost(&peer_costs);
if changed {
if !was_parent {
return false;
}
// The removed peer was our parent. `parent_losses` counts the loss
// itself (independent of whether we recover), so it is stamped here —
// before the recovery mutation — exactly as before.
self.metrics().tree.parent_losses.inc();
let peer_costs: BTreeMap<NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
// Wall-clock seconds stamped onto the new declaration; monotonic ms for
// the parent re-evaluation's flap timers.
let now_secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mono_now_ms = crate::time::mono_ms();
// Removal is not a pure classify: `handle_parent_lost` is a &mut mutator
// whose returned `changed` bool IS the decision. Drive it and map the
// outcome onto the TreeDecision vocabulary.
let decision = if self
.tree_state
.handle_parent_lost(&peer_costs, now_secs, mono_now_ms)
{
TreeDecision::ParentLost
} else {
TreeDecision::NoChange
};
match decision {
TreeDecision::ParentLost => {
// Re-sign the new declaration. Clone identity to avoid a split
// borrow against the &mut self.tree_state receiver (cold path).
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after parent loss");
self.metrics()
.tree
@@ -631,10 +785,11 @@ impl Node {
is_root = self.tree_state.is_root(),
"Tree state updated after parent loss"
);
true
}
changed
} else {
false
TreeDecision::NoChange => false,
// The removal drive constructs only ParentLost / NoChange above.
_ => unreachable!("removal drive yields only ParentLost / NoChange"),
}
}
}

505
src/nostr/advert.rs Normal file
View File

@@ -0,0 +1,505 @@
//! Synchronous decision core for the Nostr overlay-advert lifecycle.
//!
//! `AdvertMachine` owns the advert-related state that previously lived
//! directly on `NostrRendezvous` — the peer advert cache, the local
//! advert we publish, and the id of our most recently published advert
//! event — and hosts the *decision* logic for publishing, caching,
//! fetching, and pruning adverts.
//!
//! Following the sans-IO shape used by `failure_state`, every method here
//! is synchronous, performs no network I/O and no `.await`, holds its
//! state behind `std::sync::Mutex`, and takes the current time as an
//! explicit `now_ms: u64` input rather than reading a clock. The async
//! driver on `NostrRendezvous` reads the clock at the call site, invokes
//! these methods, and performs the actual relay I/O (`send_event_to`,
//! `fetch_events_from`, gift-wrap crypto), event signing, NIP-09 deletes,
//! and `Notify` wakeups described by the returned decisions.
use std::collections::HashMap;
use std::sync::Mutex;
use nostr::prelude::{Event, EventId};
use super::runtime::{NostrRendezvous, endpoint_advert_is_publicly_usable};
use super::types::{
ADVERT_IDENTIFIER, ADVERT_VERSION, BootstrapError, CachedOverlayAdvert, OverlayAdvert,
OverlayEndpointAdvert,
};
/// What the async driver should do to satisfy a publish request. Returned
/// by [`AdvertMachine::plan_publish`]; the machine performs the pure
/// decision (which advert body, or a delete, or nothing) and the driver
/// executes the corresponding relay I/O.
#[derive(Debug, Clone)]
pub(super) enum PublishPlan {
/// Nothing to publish (advertising disabled with no prior event, no
/// local advert yet, or the advert has no publicly usable endpoints).
Nothing,
/// Advertising is disabled but a prior advert event exists; the driver
/// should emit a NIP-09 delete for `EventId` then call
/// [`AdvertMachine::clear_event_id`].
Delete(EventId),
/// Publish this fully-prepared advert body. The driver builds the
/// tags/expiration, signs, sends, then records the new event id via
/// [`AdvertMachine::set_event_id`].
Publish(OverlayAdvert),
}
pub(super) struct AdvertMachine {
/// Our own npub. Used to avoid logging/attributing self-authored
/// adverts as peer discoveries.
npub: String,
/// Whether this node advertises at all (`config.advertise`).
advertise: bool,
/// Grace-extended max age for a cached advert, in ms
/// (`advert_ttl_secs * 1000 * stale-grace-multiplier`).
advert_max_age_ms: u64,
/// Size cap for the peer advert cache.
cache_max_entries: usize,
/// Peer advert cache keyed by author npub.
cache: Mutex<HashMap<String, CachedOverlayAdvert>>,
/// The advert body we currently want to publish, if any.
local_advert: Mutex<Option<OverlayAdvert>>,
/// Id of our most recently published advert event (for NIP-09 delete
/// on withdrawal).
current_event_id: Mutex<Option<EventId>>,
}
impl AdvertMachine {
pub(super) fn new(
npub: String,
advertise: bool,
advert_max_age_ms: u64,
cache_max_entries: usize,
) -> Self {
Self {
npub,
advertise,
advert_max_age_ms,
cache_max_entries,
cache: Mutex::new(HashMap::new()),
local_advert: Mutex::new(None),
current_event_id: Mutex::new(None),
}
}
// --- validity (time-injected) --------------------------------------
/// Compute the validity horizon of an advert event, or `None` if it is
/// already stale. Thin time-injected wrapper over the pure
/// `compute_advert_valid_until_ms`.
pub(super) fn event_valid_until_ms(&self, event: &Event, now_ms: u64) -> Option<u64> {
NostrRendezvous::compute_advert_valid_until_ms(event, self.advert_max_age_ms, now_ms)
}
// --- cache: prune / observe / fetch --------------------------------
/// TTL + size-cap eviction. Drops entries past their validity horizon,
/// then evicts the oldest (by `valid_until_ms`) beyond the cap.
///
/// Returns `Some((evicted, retained))` when a size-cap eviction
/// occurred so the driver can log it; `None` otherwise.
pub(super) fn prune(&self, now_ms: u64) -> Option<(usize, usize)> {
let mut cache = self.lock_cache();
cache.retain(|_, entry| entry.valid_until_ms > now_ms);
if cache.len() <= self.cache_max_entries {
return None;
}
let mut oldest = cache
.iter()
.map(|(npub, entry)| (npub.clone(), entry.valid_until_ms))
.collect::<Vec<_>>();
oldest.sort_by_key(|(_, ts)| *ts);
let overflow = cache.len().saturating_sub(self.cache_max_entries);
for (npub, _) in oldest.into_iter().take(overflow) {
cache.remove(&npub);
}
Some((overflow, cache.len()))
}
/// Observe an advert event received on the notify loop. Replaces the
/// cached entry iff its `created_at` is newer-or-equal to the cached
/// one (or none is cached).
///
/// Returns `true` when the caller should log a "peer cached" line —
/// i.e. the entry was (re)cached *and* it is not our own advert.
pub(super) fn observe_advert(
&self,
author_npub: &str,
advert: OverlayAdvert,
created_at: u64,
valid_until_ms: u64,
) -> bool {
let mut cache = self.lock_cache();
let should_replace = cache
.get(author_npub)
.map(|existing| existing.created_at <= created_at)
.unwrap_or(true);
if !should_replace {
return false;
}
let is_peer = author_npub != self.npub;
cache.insert(
author_npub.to_string(),
CachedOverlayAdvert {
author_npub: author_npub.to_string(),
advert,
created_at,
valid_until_ms,
},
);
is_peer
}
/// Cache-hit lookup for the fetch path: return the cached advert body
/// if present, `None` if the driver must fetch from relays.
pub(super) fn cached_advert(&self, peer_npub: &str) -> Option<OverlayAdvert> {
self.lock_cache()
.get(peer_npub)
.map(|cached| cached.advert.clone())
}
/// The `created_at` of a cached advert, if any. Used by the stale-check
/// refetch path to decide whether a relay result is newer.
pub(super) fn cached_created_at(&self, peer_npub: &str) -> Option<u64> {
self.lock_cache()
.get(peer_npub)
.map(|cached| cached.created_at)
}
/// Insert a freshly-fetched advert into the cache (fetch-miss path and
/// stale-check refresh).
pub(super) fn insert_fetched(&self, peer_npub: &str, cached: CachedOverlayAdvert) {
self.lock_cache().insert(peer_npub.to_string(), cached);
}
/// Remove a peer's cached advert (stale-check eviction).
pub(super) fn remove(&self, peer_npub: &str) {
self.lock_cache().remove(peer_npub);
}
/// Validity-filtered snapshot of cacheable peers for open discovery:
/// entries authored by someone other than us and still valid at
/// `now_ms`.
pub(super) fn open_discovery_candidates(
&self,
max: usize,
now_ms: u64,
) -> Vec<(String, Vec<OverlayEndpointAdvert>, u64)> {
let cache = self.lock_cache();
cache
.values()
.filter(|entry| entry.author_npub != self.npub)
.filter(|entry| entry.valid_until_ms > now_ms)
.map(|entry| {
(
entry.author_npub.clone(),
entry.advert.endpoints.clone(),
entry.created_at,
)
})
.take(max)
.collect()
}
// --- local advert / publish ----------------------------------------
/// Set the local advert we want to publish. Returns `true` when the
/// value changed (so the driver should request a republish).
pub(super) fn set_local_advert(&self, advert: Option<OverlayAdvert>) -> bool {
let mut slot = self.lock_local();
if *slot == advert {
false
} else {
*slot = advert;
true
}
}
/// Build the publish decision: which advert body to publish, a delete
/// to emit, or nothing. Pure logic — the driver performs the relay I/O
/// and event signing.
pub(super) fn plan_publish(&self) -> Result<PublishPlan, BootstrapError> {
let previous_event_id = *self.lock_event_id();
if !self.advertise {
return Ok(match previous_event_id {
Some(event_id) => PublishPlan::Delete(event_id),
None => PublishPlan::Nothing,
});
}
let mut advert = match self.lock_local().clone() {
Some(advert) => advert,
// Transient absence (e.g., a single tick during startup where
// build_overlay_advert briefly returns None). Don't proactively
// emit a NIP-09 delete: the next publish supersedes the old
// event via parameterized-replaceable semantics, and the NIP-40
// expiration tag bounds the worst case if we never re-publish.
None => return Ok(PublishPlan::Nothing),
};
advert.identifier = ADVERT_IDENTIFIER.to_string();
advert.version = ADVERT_VERSION;
advert.endpoints.retain(endpoint_advert_is_publicly_usable);
// Defensive: build_overlay_advert returns None on empty endpoints,
// so this is only reachable from non-lifecycle callers.
if advert.endpoints.is_empty() {
return Ok(PublishPlan::Nothing);
}
if advert.has_udp_nat_endpoint() {
if advert
.signal_relays
.as_ref()
.is_none_or(|relays| relays.is_empty())
{
return Err(BootstrapError::InvalidAdvert(
"udp:nat endpoint requires non-empty signalRelays".to_string(),
));
}
if advert
.stun_servers
.as_ref()
.is_none_or(|servers| servers.is_empty())
{
return Err(BootstrapError::InvalidAdvert(
"udp:nat endpoint requires non-empty stunServers".to_string(),
));
}
} else {
advert.signal_relays = None;
advert.stun_servers = None;
}
Ok(PublishPlan::Publish(advert))
}
// --- current advert event id (NIP-09 delete-on-withdraw) -----------
/// Record the id of a just-published advert event.
pub(super) fn set_event_id(&self, event_id: EventId) {
*self.lock_event_id() = Some(event_id);
}
/// Clear the recorded advert event id (after emitting a delete).
pub(super) fn clear_event_id(&self) {
*self.lock_event_id() = None;
}
/// Take and clear the recorded advert event id (shutdown path).
pub(super) fn take_event_id(&self) -> Option<EventId> {
self.lock_event_id().take()
}
// --- lock helpers ---------------------------------------------------
fn lock_cache(&self) -> std::sync::MutexGuard<'_, HashMap<String, CachedOverlayAdvert>> {
self.cache
.lock()
.expect("advert-machine cache mutex poisoned")
}
fn lock_local(&self) -> std::sync::MutexGuard<'_, Option<OverlayAdvert>> {
self.local_advert
.lock()
.expect("advert-machine local-advert mutex poisoned")
}
fn lock_event_id(&self) -> std::sync::MutexGuard<'_, Option<EventId>> {
self.current_event_id
.lock()
.expect("advert-machine event-id mutex poisoned")
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::nostr::types::OverlayTransportKind;
fn ep(addr: &str) -> OverlayEndpointAdvert {
OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: addr.to_string(),
}
}
fn advert(endpoints: Vec<OverlayEndpointAdvert>) -> OverlayAdvert {
OverlayAdvert {
identifier: ADVERT_IDENTIFIER.to_string(),
version: ADVERT_VERSION,
endpoints,
signal_relays: None,
stun_servers: None,
}
}
fn cached(author: &str, created_at: u64, valid_until_ms: u64) -> CachedOverlayAdvert {
CachedOverlayAdvert {
author_npub: author.to_string(),
advert: advert(vec![ep("1.2.3.4:9000")]),
created_at,
valid_until_ms,
}
}
fn machine() -> AdvertMachine {
// npub=self, advertise=true, max-age huge, cap=3
AdvertMachine::new("npub1self".to_string(), true, 10_000_000, 3)
}
#[test]
fn observe_advert_replaces_only_when_newer_and_flags_peer() {
let m = machine();
// Fresh peer advert -> cached, log flagged (peer).
assert!(m.observe_advert("npub1peer", advert(vec![ep("1.2.3.4:9000")]), 100, 5000));
// Older created_at -> not replaced, no log.
assert!(!m.observe_advert("npub1peer", advert(vec![ep("1.2.3.4:9001")]), 50, 5000));
assert_eq!(m.cached_created_at("npub1peer"), Some(100));
// Newer created_at -> replaced, log flagged.
assert!(m.observe_advert("npub1peer", advert(vec![ep("1.2.3.4:9002")]), 200, 5000));
assert_eq!(m.cached_created_at("npub1peer"), Some(200));
}
#[test]
fn observe_advert_self_author_caches_but_does_not_flag_log() {
let m = machine();
// Own advert is still cached (should_replace true) but must NOT be
// flagged as a peer-cached log line.
assert!(!m.observe_advert("npub1self", advert(vec![ep("1.2.3.4:9000")]), 100, 5000));
assert_eq!(m.cached_created_at("npub1self"), Some(100));
}
#[test]
fn prune_drops_expired_and_reports_no_eviction_under_cap() {
let m = machine();
m.insert_fetched("npub1a", cached("npub1a", 1, 1000));
m.insert_fetched("npub1b", cached("npub1b", 1, 3000));
// now=2000 -> npub1a expired, npub1b retained, under cap -> None.
assert_eq!(m.prune(2000), None);
assert_eq!(m.cached_created_at("npub1a"), None);
assert!(m.cached_created_at("npub1b").is_some());
}
#[test]
fn prune_size_cap_evicts_oldest_by_valid_until() {
let m = machine(); // cap = 3
// Four still-valid entries; oldest valid_until must be evicted.
m.insert_fetched("npub1a", cached("npub1a", 1, 1000));
m.insert_fetched("npub1b", cached("npub1b", 1, 2000));
m.insert_fetched("npub1c", cached("npub1c", 1, 3000));
m.insert_fetched("npub1d", cached("npub1d", 1, 4000));
let evicted = m.prune(500);
assert_eq!(evicted, Some((1, 3)));
// Oldest validity (npub1a) evicted; newest kept.
assert_eq!(m.cached_created_at("npub1a"), None);
assert!(m.cached_created_at("npub1d").is_some());
}
#[test]
fn open_discovery_candidates_filters_self_and_expired() {
let m = machine();
m.insert_fetched("npub1self", cached("npub1self", 1, 9000));
m.insert_fetched("npub1peer", cached("npub1peer", 1, 9000));
m.insert_fetched("npub1stale", cached("npub1stale", 1, 1000));
let out = m.open_discovery_candidates(10, 2000);
assert_eq!(out.len(), 1, "only the valid non-self peer survives");
assert_eq!(out[0].0, "npub1peer");
}
#[test]
fn open_discovery_candidates_respects_max() {
let m = machine();
for i in 0..5 {
let npub = format!("npub1p{i}");
m.insert_fetched(&npub, cached(&npub, 1, 9000));
}
assert_eq!(m.open_discovery_candidates(2, 1000).len(), 2);
}
#[test]
fn set_local_advert_detects_change() {
let m = machine();
let a = advert(vec![ep("1.2.3.4:9000")]);
assert!(m.set_local_advert(Some(a.clone())), "first set is a change");
assert!(
!m.set_local_advert(Some(a.clone())),
"identical set is no change"
);
assert!(m.set_local_advert(None), "clearing is a change");
}
#[test]
fn plan_publish_strips_relays_for_non_nat_advert() {
let m = machine();
let mut a = advert(vec![ep("1.2.3.4:9000")]);
a.signal_relays = Some(vec!["wss://relay".to_string()]);
a.stun_servers = Some(vec!["stun:host:3478".to_string()]);
m.set_local_advert(Some(a));
match m.plan_publish().expect("plan ok") {
PublishPlan::Publish(out) => {
assert!(
out.signal_relays.is_none(),
"non-nat advert strips signalRelays"
);
assert!(
out.stun_servers.is_none(),
"non-nat advert strips stunServers"
);
assert_eq!(out.identifier, ADVERT_IDENTIFIER);
assert_eq!(out.version, ADVERT_VERSION);
}
other => panic!("expected Publish, got {other:?}"),
}
}
#[test]
fn plan_publish_keeps_relays_for_nat_advert() {
let m = machine();
let mut a = advert(vec![ep("nat")]);
a.signal_relays = Some(vec!["wss://relay".to_string()]);
a.stun_servers = Some(vec!["stun:host:3478".to_string()]);
m.set_local_advert(Some(a));
match m.plan_publish().expect("plan ok") {
PublishPlan::Publish(out) => {
assert!(out.has_udp_nat_endpoint());
assert_eq!(out.signal_relays.as_deref().map(<[_]>::len), Some(1));
assert_eq!(out.stun_servers.as_deref().map(<[_]>::len), Some(1));
}
other => panic!("expected Publish, got {other:?}"),
}
}
#[test]
fn plan_publish_nat_without_relays_errors() {
let m = machine();
m.set_local_advert(Some(advert(vec![ep("nat")])));
assert!(matches!(
m.plan_publish(),
Err(BootstrapError::InvalidAdvert(_))
));
}
#[test]
fn plan_publish_nothing_when_no_local_advert() {
let m = machine();
assert!(matches!(m.plan_publish(), Ok(PublishPlan::Nothing)));
}
#[test]
fn plan_publish_nothing_when_disabled_without_prior_event() {
// advertise=false, no prior event id -> Nothing.
let m = AdvertMachine::new("npub1self".to_string(), false, 10_000_000, 3);
m.set_local_advert(Some(advert(vec![ep("1.2.3.4:9000")])));
assert!(matches!(m.plan_publish(), Ok(PublishPlan::Nothing)));
}
#[test]
fn event_id_set_clear_take_roundtrip() {
let m = machine();
assert_eq!(m.take_event_id(), None);
m.clear_event_id();
assert_eq!(m.take_event_id(), None);
}
}

399
src/nostr/driver.rs Normal file
View File

@@ -0,0 +1,399 @@
//! Node-side driver state for the Nostr overlay peer-rendezvous subsystem.
//!
//! [`RendezvousDriver`] consolidates the rendezvous-subsystem state that
//! previously lived as loose fields on the `Node` struct: the engine handle,
//! its startup timestamp, the one-shot startup-sweep latch, and the
//! per-peer bootstrap-transport bookkeeping adopted from NAT-traversal
//! handoffs. Keeping it in the `nostr` module gives the subsystem a single
//! home while leaving the transport/connection-table mutations that consume
//! this state on `Node`.
use std::collections::{HashMap, HashSet};
use std::net::SocketAddr;
use std::sync::Arc;
use tracing::{debug, info, warn};
use crate::config::{NostrRendezvousConfig, NostrRendezvousPolicy, PeerAddress, PeerConfig};
use crate::transport::TransportId;
use super::{
ADVERT_IDENTIFIER, ADVERT_VERSION, BootstrapError, NostrRendezvous, OverlayAdvert,
OverlayEndpointAdvert, OverlayTransportKind,
};
/// Snapshot of a single operational transport's advertisable endpoint
/// inputs, captured on `Node` at advert-build time so the driver can
/// assemble the overlay advert without borrowing the transport table.
/// Only transports whose type matched a configured listener are included;
/// the `advertise` gate is carried verbatim so the driver reproduces the
/// original per-transport branch logic exactly.
pub enum AdvertTransportSnapshot {
Udp {
advertise: bool,
is_public: bool,
external_addr: Option<SocketAddr>,
local_addr: Option<SocketAddr>,
transport_key: u32,
},
Tcp {
advertise: bool,
external_addr: Option<SocketAddr>,
local_addr: Option<SocketAddr>,
},
Tor {
advertise: bool,
onion_addr: Option<String>,
advertised_port: u16,
},
}
/// Node-side rendezvous-subsystem state and bootstrap-transport bookkeeping.
#[derive(Default)]
pub struct RendezvousDriver {
/// Optional Nostr/STUN overlay discovery coordinator for `udp:nat` peers.
engine: Option<Arc<NostrRendezvous>>,
/// Wall-clock ms when Nostr discovery successfully started, used to
/// schedule the one-shot startup advert sweep after a settle delay.
/// `None` until discovery comes up; remains `None` if discovery is
/// disabled or failed to start.
started_at_ms: Option<u64>,
/// Whether the one-shot startup advert sweep has run. Set to true
/// after the first sweep fires (under `policy: open`); thereafter
/// only the per-tick `queue_open_discovery_retries` continues.
startup_sweep_done: bool,
/// Per-peer UDP transports adopted from NAT traversal handoff.
bootstrap_transports: HashSet<TransportId>,
/// Originating peer npub (bech32) for each adopted bootstrap
/// transport, captured at `adopt_established_traversal` time.
/// Populated alongside `bootstrap_transports`; cleared in
/// `cleanup_bootstrap_transport_if_unused`. Used by the rx loop to
/// route fatal-protocol-mismatch observations back to the
/// Nostr-discovery `failure_state` for long cooldown application.
bootstrap_transport_npubs: HashMap<TransportId, String>,
}
impl RendezvousDriver {
/// Borrow the engine handle if discovery is running.
pub fn engine(&self) -> Option<&NostrRendezvous> {
self.engine.as_deref()
}
/// Clone the engine `Arc` handle if discovery is running.
pub fn engine_arc(&self) -> Option<Arc<NostrRendezvous>> {
self.engine.clone()
}
/// Install the engine handle once discovery starts.
pub fn set_engine(&mut self, engine: Arc<NostrRendezvous>) {
self.engine = Some(engine);
}
/// Take the engine handle for shutdown, clearing it.
pub fn take_engine(&mut self) -> Option<Arc<NostrRendezvous>> {
self.engine.take()
}
/// Record the wall-clock ms at which discovery successfully started.
pub fn set_started_at_ms(&mut self, now_ms: u64) {
self.started_at_ms = Some(now_ms);
}
/// Wall-clock ms when discovery started, if it has.
pub fn started_at_ms(&self) -> Option<u64> {
self.started_at_ms
}
/// Whether the one-shot startup sweep has already run.
pub fn startup_sweep_done(&self) -> bool {
self.startup_sweep_done
}
/// Latch the one-shot startup sweep as done.
pub fn set_startup_sweep_done(&mut self) {
self.startup_sweep_done = true;
}
/// Whether `transport_id` is an adopted bootstrap transport.
pub fn is_bootstrap_transport(&self, transport_id: &TransportId) -> bool {
self.bootstrap_transports.contains(transport_id)
}
/// Originating peer npub for an adopted bootstrap transport, if any.
pub fn bootstrap_transport_npub(&self, transport_id: &TransportId) -> Option<&String> {
self.bootstrap_transport_npubs.get(transport_id)
}
/// Register an adopted bootstrap transport and its originating npub.
pub fn insert_bootstrap_transport(&mut self, transport_id: TransportId, npub: String) {
self.bootstrap_transports.insert(transport_id);
self.bootstrap_transport_npubs.insert(transport_id, npub);
}
/// Drop an adopted bootstrap transport from both bookkeeping maps.
pub fn remove_bootstrap_transport(&mut self, transport_id: &TransportId) {
self.bootstrap_transports.remove(transport_id);
self.bootstrap_transport_npubs.remove(transport_id);
}
/// Convert an advertised overlay endpoint into a `PeerAddress` candidate.
/// Pure mapping; `seen_at_ms` is supplied by the caller.
pub fn overlay_endpoint_to_peer_address(
endpoint: &OverlayEndpointAdvert,
priority: u8,
seen_at_ms: u64,
) -> Option<PeerAddress> {
let transport = match endpoint.transport {
OverlayTransportKind::Udp => "udp",
OverlayTransportKind::Tcp => "tcp",
OverlayTransportKind::Tor => "tor",
};
Some(
PeerAddress::with_priority(transport, endpoint.addr.clone(), priority)
.with_seen_at_ms(seen_at_ms),
)
}
/// Kick off a Nostr-mediated UDP NAT-traversal attempt for `peer_config`.
/// Returns whether an attempt was started (false if discovery is down).
pub async fn request_nostr_bootstrap(&self, peer_config: &PeerConfig) -> bool {
let Some(bootstrap) = self.engine_arc() else {
debug!(npub = %peer_config.npub, "No Nostr overlay runtime for udp:nat address");
return false;
};
bootstrap.request_connect(peer_config.clone()).await;
info!(npub = %peer_config.npub, "Started Nostr UDP NAT traversal attempt");
true
}
/// Resolve additional overlay `PeerAddress` candidates for a `via_nostr`
/// configured peer by fetching its published advert endpoints. `existing`
/// is the already-known static address list (used for priority and dedup);
/// `now_ms` stamps the returned candidates' `seen_at`.
pub async fn nostr_peer_fallback_addresses(
&self,
peer_config: &PeerConfig,
existing: &[PeerAddress],
nostr_cfg: &NostrRendezvousConfig,
now_ms: u64,
) -> Vec<PeerAddress> {
if !nostr_cfg.enabled
|| !peer_config.via_nostr
|| nostr_cfg.policy == NostrRendezvousPolicy::Disabled
{
return Vec::new();
}
let Some(bootstrap) = self.engine_arc() else {
return Vec::new();
};
let endpoints = match bootstrap.advert_endpoints_for_peer(&peer_config.npub).await {
Ok(endpoints) => endpoints,
Err(err) => {
debug!(
npub = %peer_config.npub,
error = %err,
"Failed to resolve Nostr advert endpoints for configured peer"
);
return Vec::new();
}
};
let mut fallback = Vec::new();
let mut next_priority = existing
.iter()
.map(|addr| addr.priority)
.max()
.unwrap_or(100)
.saturating_add(1);
let seen_at_ms = now_ms;
for endpoint in endpoints {
let Some(candidate) =
Self::overlay_endpoint_to_peer_address(&endpoint, next_priority, seen_at_ms)
else {
continue;
};
if existing
.iter()
.any(|addr| addr.transport == candidate.transport && addr.addr == candidate.addr)
|| fallback.iter().any(|addr: &PeerAddress| {
addr.transport == candidate.transport && addr.addr == candidate.addr
})
{
continue;
}
fallback.push(candidate);
next_priority = next_priority.saturating_add(1);
}
fallback
}
/// Publish (or withdraw) the local overlay advert built from `snapshot`.
/// `bootstrap` is passed explicitly because the startup path refreshes the
/// advert before the engine handle is installed on the driver.
pub async fn refresh_overlay_advert(
&self,
bootstrap: &Arc<NostrRendezvous>,
snapshot: Vec<AdvertTransportSnapshot>,
nostr_cfg: &NostrRendezvousConfig,
) -> Result<(), BootstrapError> {
let advert = self
.build_overlay_advert(bootstrap, snapshot, nostr_cfg)
.await;
bootstrap.update_local_advert(advert).await
}
/// Assemble the local `OverlayAdvert` from the per-transport `snapshot`.
/// The STUN `learn_public_udp_addr` await for wildcard-bound public UDP
/// sockets is reached through the `bootstrap` handle.
async fn build_overlay_advert(
&self,
bootstrap: &Arc<NostrRendezvous>,
snapshot: Vec<AdvertTransportSnapshot>,
nostr_cfg: &NostrRendezvousConfig,
) -> Option<OverlayAdvert> {
if !nostr_cfg.enabled {
return None;
}
let mut endpoints = Vec::new();
let mut has_udp_nat = false;
for entry in snapshot {
match entry {
AdvertTransportSnapshot::Udp {
advertise,
is_public,
external_addr,
local_addr,
transport_key,
} => {
if !advertise {
continue;
}
if is_public {
// Precedence:
// 1. operator-supplied `external_addr` (skips STUN)
// 2. non-wildcard `local_addr` (operator bound to
// a specific public IP directly)
// 3. STUN auto-discovery against ephemeral socket
// 4. loud warn + omit endpoint
if let Some(explicit) = external_addr {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: explicit.to_string(),
});
} else {
match local_addr {
Some(addr) if !addr.ip().is_unspecified() => {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: addr.to_string(),
});
}
Some(addr) => {
let key = transport_key;
let port = addr.port();
if let Some(public) =
bootstrap.learn_public_udp_addr(key, port).await
{
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: public.to_string(),
});
} else {
warn!(
transport_id = key,
bind_addr = %addr,
"advert: udp public=true bound to wildcard but \
STUN observation failed; advertising no UDP \
endpoint. Either set transports.udp.external_addr, \
bind to a specific public IP, or ensure \
node.rendezvous.nostr.stun_servers is reachable"
);
}
}
None => {}
}
}
} else {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: "nat".to_string(),
});
has_udp_nat = true;
}
}
AdvertTransportSnapshot::Tcp {
advertise,
external_addr,
local_addr,
} => {
if !advertise {
continue;
}
// Precedence:
// 1. operator-supplied `external_addr` (only path that
// works on cloud-NAT setups where the public IP is
// not on a host interface).
// 2. non-wildcard `local_addr` (operator bound to a
// specific public IP directly).
// 3. loud warn + omit endpoint (no TCP STUN equivalent).
if let Some(explicit) = external_addr {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Tcp,
addr: explicit.to_string(),
});
} else {
match local_addr {
Some(addr) if !addr.ip().is_unspecified() => {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Tcp,
addr: addr.to_string(),
});
}
Some(addr) => {
warn!(
bind_addr = %addr,
"advert: tcp advertise_on_nostr=true bound to wildcard \
and no transports.tcp.external_addr set; advertising no \
TCP endpoint. Either set external_addr to the public \
IP (recommended for cloud 1:1-NAT setups) or bind \
explicitly to the public IP"
);
}
None => {}
}
}
}
AdvertTransportSnapshot::Tor {
advertise,
onion_addr,
advertised_port,
} => {
if !advertise {
continue;
}
if let Some(addr) = onion_addr {
endpoints.push(OverlayEndpointAdvert {
transport: OverlayTransportKind::Tor,
addr: format!("{}:{}", addr, advertised_port),
});
}
}
}
}
if endpoints.is_empty() {
return None;
}
Some(OverlayAdvert {
identifier: ADVERT_IDENTIFIER.to_string(),
version: ADVERT_VERSION,
endpoints,
signal_relays: has_udp_nat.then(|| nostr_cfg.dm_relays.clone()),
stun_servers: has_udp_nat.then(|| nostr_cfg.stun_servers.clone()),
})
}
}

View File

@@ -6,9 +6,6 @@
//! it hands the live socket and selected remote endpoint to FIPS so the
//! existing Noise/FMP transport path can take over.
pub mod lan;
pub mod nostr;
use crate::config::UdpConfig;
use crate::{NodeAddr, TransportId};
use std::net::{SocketAddr, UdpSocket};
@@ -16,9 +13,9 @@ use std::net::{SocketAddr, UdpSocket};
/// Punch-probe magic ("NPTC", network byte order). First byte `0x4E`
/// collides with FMP's prefix-version high-nibble check, so the UDP
/// transport silently filters packets carrying this magic to keep
/// post-adoption handshake logs clean. Defined at the top-level
/// `discovery` module so the UDP filter and the nostr submodule's
/// punch sender share the same constant.
/// post-adoption handshake logs clean. Defined in the nostr rendezvous
/// home so the UDP filter and the nostr submodule's punch sender share
/// the same constant.
pub const PUNCH_MAGIC: u32 = 0x4E505443;
/// Punch-probe-ack magic ("NPTA", network byte order). Same filter as

View File

@@ -1,14 +1,20 @@
mod advert;
mod driver;
mod failure_state;
mod handoff;
mod runtime;
mod signal;
mod stun;
mod traversal;
mod traversal_machine;
mod types;
#[cfg(test)]
mod tests;
pub use runtime::NostrDiscovery;
pub use driver::{AdvertTransportSnapshot, RendezvousDriver};
pub use handoff::{BootstrapHandoffResult, EstablishedTraversal, is_punch_packet};
pub use runtime::NostrRendezvous;
pub use types::{
ADVERT_IDENTIFIER, ADVERT_KIND, ADVERT_VERSION, BootstrapError, BootstrapEvent,
CachedOverlayAdvert, NostrFailureDecision, NostrPeerFailureView, NostrRefetchOutcome,

View File

@@ -16,7 +16,9 @@ use tokio::sync::{Mutex, Notify, RwLock, Semaphore, broadcast, mpsc, oneshot};
use tokio::task::JoinHandle;
use tracing::{debug, info, trace, warn};
use super::advert::{AdvertMachine, PublishPlan};
use super::failure_state::FailureState;
use super::handoff::EstablishedTraversal;
use super::signal::{
FreshnessOutcome, SignalEnvelope, build_signal_event, create_traversal_answer,
create_traversal_offer, estimate_clock_skew, unwrap_signal_event, validate_offer_freshness,
@@ -24,15 +26,15 @@ use super::signal::{
};
use super::stun::observe_traversal_addresses;
use super::traversal::{nonce, now_ms, planned_remote_endpoints, run_punch_attempt};
use super::traversal_machine::{OfferDisposition, SeenDecision, TraversalMachine};
use super::types::{
ADVERT_IDENTIFIER, ADVERT_KIND, ADVERT_VERSION, BootstrapError, BootstrapEvent,
CachedOverlayAdvert, NostrFailureDecision, NostrPeerFailureView, NostrRefetchOutcome,
OverlayAdvert, OverlayEndpointAdvert, PROTOCOL_VERSION, PunchHint, SIGNAL_KIND,
TraversalAnswer, TraversalOffer,
};
use crate::config::{NostrDiscoveryConfig, PeerConfig};
use crate::discovery::EstablishedTraversal;
use crate::{NodeAddr, PeerIdentity};
use crate::PeerIdentity;
use crate::config::{NostrRendezvousConfig, PeerConfig};
const ADVERT_CACHE_STALE_GRACE_MULTIPLIER: u64 = 2;
@@ -51,42 +53,6 @@ fn short_id(id: &str) -> String {
}
}
/// Decide whether an incoming-offer responder session should be suppressed
/// in favour of our own already-running outbound initiator session.
///
/// Two peers that each have the other as `auto_connect` simultaneously run an
/// initiator traversal *and* a responder traversal for the same peer, binding a
/// separate UDP socket per session. Each node then emits two
/// `BootstrapEvent::Established` events and `adopt_established_traversal` keeps
/// only the first on a non-deterministic race; when the two nodes' independent
/// races resolve to mismatched sessions, each side's Noise msg1 lands on a peer
/// port the peer already stopped draining and both handshakes stall (root cause
/// of ISSUE-2026-0031).
///
/// To collapse the four-socket dance to a single, guaranteed-matching socket
/// pair, both nodes deterministically keep the session **initiated by the
/// smaller `NodeAddr`** — reusing the project's existing NodeAddr tie-breaker
/// convention (`cross_connection_winner`, the rekey dual-init resolution, and
/// the dual-cross-init adopt path in `lifecycle.rs`).
///
/// This is evaluated on the responder path, where the session being handled is
/// *peer-initiated*. It returns `true` (suppress this responder session) only
/// when genuine duplication exists — i.e. we also have an in-flight outbound
/// initiator for this same peer (`have_active_initiator`) — and our own
/// initiator session is the preferred one (`our_addr < peer_addr`). When there
/// is no co-active initiator (the asymmetric / one-sided `auto_connect` case,
/// where only one session exists at all) it never suppresses, so connectivity
/// is preserved. The `our_addr == peer_addr` case (self / loopback) and any
/// caller that cannot derive a peer `NodeAddr` likewise fall through to "do not
/// suppress".
pub(super) fn suppress_responder_for_own_initiator(
our_addr: &NodeAddr,
peer_addr: &NodeAddr,
have_active_initiator: bool,
) -> bool {
have_active_initiator && our_addr < peer_addr
}
fn endpoint_summary(endpoints: &[OverlayEndpointAdvert]) -> String {
endpoints
.iter()
@@ -117,7 +83,7 @@ fn is_unroutable_direct_advert_ip(ip: std::net::IpAddr) -> bool {
}
}
fn endpoint_advert_is_publicly_usable(endpoint: &OverlayEndpointAdvert) -> bool {
pub(super) fn endpoint_advert_is_publicly_usable(endpoint: &OverlayEndpointAdvert) -> bool {
let addr = endpoint.addr.trim();
if addr.is_empty() {
return false;
@@ -157,7 +123,7 @@ fn endpoint_advert_is_publicly_usable(endpoint: &OverlayEndpointAdvert) -> bool
}
/// Cached STUN-derived public address for an advert-eligible UDP transport
/// bound to a wildcard. Lives on `NostrDiscovery` so the freshness window
/// bound to a wildcard. Lives on `NostrRendezvous` so the freshness window
/// survives advert refresh cycles.
struct CachedPublicUdpAddr {
/// Most recent STUN observation. `None` means the last attempt failed
@@ -177,18 +143,15 @@ const PUBLIC_UDP_ADDR_FAILURE_TTL: Duration = Duration::from_secs(60);
const RELAY_STARTUP_OP_TIMEOUT: Duration = Duration::from_secs(5);
const ADVERT_PUBLISH_TIMEOUT: Duration = Duration::from_secs(10);
pub struct NostrDiscovery {
pub struct NostrRendezvous {
client: Client,
keys: nostr::Keys,
pubkey: PublicKey,
npub: String,
config: NostrDiscoveryConfig,
advert_cache: RwLock<HashMap<String, CachedOverlayAdvert>>,
local_advert: RwLock<Option<OverlayAdvert>>,
current_advert_event_id: RwLock<Option<EventId>>,
config: NostrRendezvousConfig,
advert: AdvertMachine,
traversal: TraversalMachine,
pending_answers: Mutex<HashMap<String, oneshot::Sender<SignalEnvelope<TraversalAnswer>>>>,
active_initiators: Mutex<HashSet<String>>,
seen_sessions: Mutex<HashMap<String, u64>>,
offer_slots: Arc<Semaphore>,
event_tx: mpsc::UnboundedSender<BootstrapEvent>,
event_rx: Mutex<mpsc::UnboundedReceiver<BootstrapEvent>>,
@@ -212,10 +175,29 @@ pub struct NostrDiscovery {
outbound_admission: AtomicBool,
}
impl NostrDiscovery {
impl NostrRendezvous {
/// Whether the primary Nostr connection task has exited (runtime liveness).
///
/// "Nostr exited" is defined as the primary `connect_task` having finished.
/// It is `Some` for the engine's whole running life (installed in `start`);
/// `shutdown` takes it, leaving `None` — a taken handle means the engine has
/// been shut down, which counts as finished, so a `None` inner maps to
/// `true` (this lets the liveness poll monitor terminate after a stop rather
/// than spinning forever). `connect_task` is a `tokio::sync::Mutex`, so this
/// sync accessor uses the non-blocking `try_lock`: a momentarily-contended
/// lock (only start/stop hold it, briefly) reports "not finished", the safe
/// direction — the 2s liveness poll re-checks next tick and never spuriously
/// degrades a healthy node.
pub fn is_finished(&self) -> bool {
self.connect_task
.try_lock()
.map(|g| g.as_ref().is_none_or(|h| h.is_finished()))
.unwrap_or(false)
}
pub async fn start(
identity: &crate::Identity,
config: NostrDiscoveryConfig,
config: NostrRendezvousConfig,
) -> Result<Arc<Self>, BootstrapError> {
if !config.enabled {
return Err(BootstrapError::Disabled);
@@ -249,18 +231,25 @@ impl NostrDiscovery {
config.failure_state_max_entries,
);
let advert = AdvertMachine::new(
npub.clone(),
config.advertise,
config.advert_ttl_secs * 1000 * ADVERT_CACHE_STALE_GRACE_MULTIPLIER,
config.advert_cache_max_entries,
);
let traversal = TraversalMachine::new(
config.replay_window_secs * 1000,
config.seen_sessions_max_entries,
);
let runtime = Arc::new(Self {
client,
keys,
pubkey,
npub,
config,
advert_cache: RwLock::new(HashMap::new()),
local_advert: RwLock::new(None),
current_advert_event_id: RwLock::new(None),
advert,
traversal,
pending_answers: Mutex::new(HashMap::new()),
active_initiators: Mutex::new(HashSet::new()),
seen_sessions: Mutex::new(HashMap::new()),
offer_slots,
event_tx,
event_rx: Mutex::new(event_rx),
@@ -309,11 +298,8 @@ impl NostrDiscovery {
pub async fn request_connect(self: &Arc<Self>, peer_config: PeerConfig) {
let peer_npub = peer_config.npub.clone();
{
let mut active = self.active_initiators.lock().await;
if !active.insert(peer_npub.clone()) {
return;
}
if !self.traversal.begin_initiator(&peer_npub) {
return;
}
let runtime = Arc::clone(self);
@@ -326,7 +312,7 @@ impl NostrDiscovery {
},
};
let _ = runtime.event_tx.send(event);
runtime.active_initiators.lock().await.remove(&peer_npub);
runtime.traversal.end_initiator(&peer_npub);
});
}
@@ -507,12 +493,7 @@ impl NostrDiscovery {
if self.config.advert_relays.is_empty() {
return NostrRefetchOutcome::Skipped;
}
let cached_created_at = self
.advert_cache
.read()
.await
.get(peer_npub)
.map(|c| c.created_at);
let cached_created_at = self.advert.cached_created_at(peer_npub);
let events = match self
.client
@@ -541,7 +522,7 @@ impl NostrDiscovery {
let Some((relay_created_at, ev)) = newest else {
// Absent on relays. Evict any stale cache entry.
self.advert_cache.write().await.remove(peer_npub);
self.advert.remove(peer_npub);
self.failure_state.reset_streak_after_refresh(peer_npub);
return NostrRefetchOutcome::Evicted;
};
@@ -561,10 +542,7 @@ impl NostrDiscovery {
created_at: relay_created_at,
valid_until_ms,
};
self.advert_cache
.write()
.await
.insert(peer_npub.to_string(), updated);
self.advert.insert_fetched(peer_npub, updated);
self.failure_state.reset_streak_after_refresh(peer_npub);
NostrRefetchOutcome::Refreshed
}
@@ -584,19 +562,9 @@ impl NostrDiscovery {
self: &Arc<Self>,
advert: Option<OverlayAdvert>,
) -> Result<(), BootstrapError> {
let changed = {
let mut slot = self.local_advert.write().await;
if *slot == advert {
false
} else {
*slot = advert;
true
}
};
if !changed {
return Ok(());
if self.advert.set_local_advert(advert) {
self.request_publish_advert();
}
self.request_publish_advert();
Ok(())
}
@@ -617,22 +585,8 @@ impl NostrDiscovery {
&self,
max: usize,
) -> Vec<(String, Vec<OverlayEndpointAdvert>, u64)> {
self.prune_advert_cache().await;
let now = now_ms();
let cache = self.advert_cache.read().await;
cache
.values()
.filter(|entry| entry.author_npub != self.npub)
.filter(|entry| entry.valid_until_ms > now)
.map(|entry| {
(
entry.author_npub.clone(),
entry.advert.endpoints.clone(),
entry.created_at,
)
})
.take(max)
.collect()
self.prune_advert_cache();
self.advert.open_discovery_candidates(max, now_ms())
}
pub async fn shutdown(&self) -> Result<(), BootstrapError> {
@@ -655,7 +609,7 @@ impl NostrDiscovery {
// permanent shutdown. An explicit retraction races with the next
// daemon's republish on strict relays (e.g. Damus rate-limits the
// burst, leaving the advert deleted and never restored).
let _ = self.current_advert_event_id.write().await.take();
let _ = self.advert.take_event_id();
if let Some(handle) = self.notify_task.lock().await.take() {
handle.abort();
@@ -701,32 +655,23 @@ impl NostrDiscovery {
&& let Ok(advert) =
Self::parse_overlay_advert_event(&event, &self.config.app)
{
let mut cache = self.advert_cache.write().await;
let should_replace = cache
.get(&author_npub)
.map(|existing| existing.created_at <= event.created_at.as_secs())
.unwrap_or(true);
if should_replace && author_npub != self.npub {
let endpoints = endpoint_summary(&advert.endpoints);
let created_at = event.created_at.as_secs();
if self.advert.observe_advert(
&author_npub,
advert,
created_at,
valid_until_ms,
) {
debug!(
peer = %short_npub(&author_npub),
endpoints = %endpoint_summary(&advert.endpoints),
endpoints = %endpoints,
event = %short_id(&event.id.to_string()),
"advert: peer cached"
);
}
if should_replace {
cache.insert(
author_npub.clone(),
CachedOverlayAdvert {
author_npub,
advert,
created_at: event.created_at.as_secs(),
valid_until_ms,
},
);
}
}
self.prune_advert_cache().await;
self.prune_advert_cache();
continue;
}
@@ -949,59 +894,17 @@ impl NostrDiscovery {
}
async fn publish_advert(&self) -> Result<(), BootstrapError> {
let previous_event_id = self.current_advert_event_id.read().await.to_owned();
if !self.config.advertise {
if let Some(event_id) = previous_event_id {
let advert = match self.advert.plan_publish()? {
PublishPlan::Nothing => return Ok(()),
PublishPlan::Delete(event_id) => {
self.publish_delete(&self.config.advert_relays, [event_id])
.await?;
*self.current_advert_event_id.write().await = None;
self.advert.clear_event_id();
return Ok(());
}
return Ok(());
}
let mut advert = match self.local_advert.read().await.clone() {
Some(advert) => advert,
// Transient absence (e.g., a single tick during startup where
// build_overlay_advert briefly returns None). Don't proactively
// emit a NIP-09 delete: the next publish supersedes the old
// event via parameterized-replaceable semantics, and the NIP-40
// expiration tag bounds the worst case if we never re-publish.
None => return Ok(()),
PublishPlan::Publish(advert) => advert,
};
advert.identifier = ADVERT_IDENTIFIER.to_string();
advert.version = ADVERT_VERSION;
advert.endpoints.retain(endpoint_advert_is_publicly_usable);
// Defensive: build_overlay_advert returns None on empty endpoints,
// so this is only reachable from non-lifecycle callers.
if advert.endpoints.is_empty() {
return Ok(());
}
if advert.has_udp_nat_endpoint() {
if advert
.signal_relays
.as_ref()
.is_none_or(|relays| relays.is_empty())
{
return Err(BootstrapError::InvalidAdvert(
"udp:nat endpoint requires non-empty signalRelays".to_string(),
));
}
if advert
.stun_servers
.as_ref()
.is_none_or(|servers| servers.is_empty())
{
return Err(BootstrapError::InvalidAdvert(
"udp:nat endpoint requires non-empty stunServers".to_string(),
));
}
} else {
advert.signal_relays = None;
advert.stun_servers = None;
}
let expires_at = now_ms() + self.config.advert_ttl_secs * 1000;
let tags = vec![
Tag::identifier(ADVERT_IDENTIFIER.to_string()),
@@ -1031,7 +934,7 @@ impl NostrDiscovery {
// NIP-09 delete here is redundant and races with the replacement
// publish, which strict relays (e.g. Damus) honor by removing the
// new advert too.
*self.current_advert_event_id.write().await = Some(event.id);
self.advert.set_event_id(event.id);
Ok(())
}
@@ -1270,17 +1173,17 @@ impl NostrDiscovery {
// single matching socket pair survives on both sides. Asymmetric /
// one-sided `auto_connect` (no co-active initiator) is never suppressed,
// preserving connectivity. See `suppress_responder_for_own_initiator`.
if self.active_initiators.lock().await.contains(&sender_npub) {
match (
PeerIdentity::from_npub(&self.npub),
PeerIdentity::from_npub(&sender_npub),
) {
(Ok(ours), Ok(theirs)) => {
if suppress_responder_for_own_initiator(
ours.node_addr(),
theirs.node_addr(),
true,
) {
match (
PeerIdentity::from_npub(&self.npub),
PeerIdentity::from_npub(&sender_npub),
) {
(Ok(ours), Ok(theirs)) => {
match self.traversal.classify_incoming_offer(
&sender_npub,
ours.node_addr(),
theirs.node_addr(),
) {
OfferDisposition::Suppress => {
debug!(
peer = %peer_short,
session = %short_id(&offer.session_id),
@@ -1288,20 +1191,38 @@ impl NostrDiscovery {
);
return Ok(());
}
OfferDisposition::Proceed => {}
}
_ => {
// Could not derive a NodeAddr for one side; fall through and
// answer rather than risk suppressing the only session.
trace!(
peer = %peer_short,
"traversal: could not derive NodeAddr for dedup, answering offer"
}
_ => {
// Could not derive a NodeAddr for one side; fall through and
// answer rather than risk suppressing the only session.
trace!(
peer = %peer_short,
"traversal: could not derive NodeAddr for dedup, answering offer"
);
}
}
match self
.traversal
.note_session_seen(&offer.session_id, now_ms())
{
SeenDecision::Replay => {
return Err(BootstrapError::Replay(offer.session_id.clone()));
}
SeenDecision::Fresh { evicted } => {
if let Some((evicted, retained)) = evicted {
debug!(
evicted = evicted,
retained = retained,
cap = self.config.seen_sessions_max_entries,
"seen-sessions cache overflow; evicted oldest entries"
);
}
}
}
self.mark_session_seen(&offer.session_id).await?;
let base_socket = std::net::UdpSocket::bind(("0.0.0.0", 0))?;
base_socket.set_nonblocking(true)?;
let (reflexive_address, local_addresses, stun_server) = observe_traversal_addresses(
@@ -1396,15 +1317,15 @@ impl NostrDiscovery {
peer_npub: &str,
target_pubkey: PublicKey,
) -> Result<OverlayAdvert, BootstrapError> {
self.prune_advert_cache().await;
if let Some(cached) = self.advert_cache.read().await.get(peer_npub).cloned() {
self.prune_advert_cache();
if let Some(advert) = self.advert.cached_advert(peer_npub) {
debug!(
peer = %short_npub(peer_npub),
source = "cache",
endpoints = %endpoint_summary(&cached.advert.endpoints),
endpoints = %endpoint_summary(&advert.endpoints),
"advert: resolved"
);
return Ok(cached.advert);
return Ok(advert);
}
let events = self
@@ -1453,11 +1374,8 @@ impl NostrDiscovery {
endpoints = %endpoint_summary(&cached.advert.endpoints),
"advert: resolved"
);
self.advert_cache
.write()
.await
.insert(peer_npub.to_string(), cached.clone());
self.prune_advert_cache().await;
self.advert.insert_fetched(peer_npub, cached.clone());
self.prune_advert_cache();
Ok(cached.advert)
}
@@ -1603,39 +1521,19 @@ impl NostrDiscovery {
Ok(advert)
}
async fn prune_advert_cache(&self) {
let now = now_ms();
let mut cache = self.advert_cache.write().await;
cache.retain(|_, entry| entry.valid_until_ms > now);
if cache.len() <= self.config.advert_cache_max_entries {
return;
fn prune_advert_cache(&self) {
if let Some((evicted, retained)) = self.advert.prune(now_ms()) {
debug!(
evicted,
retained,
cap = self.config.advert_cache_max_entries,
"advert cache overflow; evicted oldest entries"
);
}
let mut oldest = cache
.iter()
.map(|(npub, entry)| (npub.clone(), entry.valid_until_ms))
.collect::<Vec<_>>();
oldest.sort_by_key(|(_, ts)| *ts);
let overflow = cache
.len()
.saturating_sub(self.config.advert_cache_max_entries);
for (npub, _) in oldest.into_iter().take(overflow) {
cache.remove(&npub);
}
debug!(
evicted = overflow,
retained = cache.len(),
cap = self.config.advert_cache_max_entries,
"advert cache overflow; evicted oldest entries"
);
}
fn advert_max_age_ms(&self) -> u64 {
self.config.advert_ttl_secs * 1000 * ADVERT_CACHE_STALE_GRACE_MULTIPLIER
}
fn event_valid_until_ms(&self, event: &Event) -> Option<u64> {
Self::compute_advert_valid_until_ms(event, self.advert_max_age_ms(), now_ms())
self.advert.event_valid_until_ms(event, now_ms())
}
pub(super) fn compute_advert_valid_until_ms(
@@ -1699,42 +1597,11 @@ impl NostrDiscovery {
.map_err(|e| BootstrapError::Nostr(e.to_string()))?;
Ok(())
}
async fn mark_session_seen(&self, session_id: &str) -> Result<(), BootstrapError> {
let now = now_ms();
let expiry = now + self.config.replay_window_secs * 1000;
let mut seen = self.seen_sessions.lock().await;
seen.retain(|_, expires_at| *expires_at > now);
if seen.contains_key(session_id) {
return Err(BootstrapError::Replay(session_id.to_string()));
}
seen.insert(session_id.to_string(), expiry);
if seen.len() > self.config.seen_sessions_max_entries {
let mut oldest = seen
.iter()
.map(|(session, expires_at)| (session.clone(), *expires_at))
.collect::<Vec<_>>();
oldest.sort_by_key(|(_, expires_at)| *expires_at);
let overflow = seen
.len()
.saturating_sub(self.config.seen_sessions_max_entries);
for (session, _) in oldest.into_iter().take(overflow) {
seen.remove(&session);
}
debug!(
evicted = overflow,
retained = seen.len(),
cap = self.config.seen_sessions_max_entries,
"seen-sessions cache overflow; evicted oldest entries"
);
}
Ok(())
}
}
#[cfg(test)]
impl NostrDiscovery {
/// Build a minimal `NostrDiscovery` for unit tests. No relay client is
impl NostrRendezvous {
/// Build a minimal `NostrRendezvous` for unit tests. No relay client is
/// connected and no background tasks are spawned; only the in-memory
/// `advert_cache` and `npub` are usable. Intended for cache-injection
/// tests of consumers (e.g. `Node::run_open_discovery_sweep`).
@@ -1746,7 +1613,7 @@ impl NostrDiscovery {
.signer(keys.clone())
.opts(ClientOptions::new().autoconnect(false))
.build();
let config = NostrDiscoveryConfig::default();
let config = NostrRendezvousConfig::default();
let offer_slots = Arc::new(Semaphore::new(config.max_concurrent_incoming_offers));
let (event_tx, event_rx) = mpsc::unbounded_channel();
let failure_state = FailureState::new(
@@ -1755,18 +1622,25 @@ impl NostrDiscovery {
config.warn_log_interval_secs,
config.failure_state_max_entries,
);
let advert = AdvertMachine::new(
npub.clone(),
config.advertise,
config.advert_ttl_secs * 1000 * ADVERT_CACHE_STALE_GRACE_MULTIPLIER,
config.advert_cache_max_entries,
);
let traversal = TraversalMachine::new(
config.replay_window_secs * 1000,
config.seen_sessions_max_entries,
);
Self {
client,
keys,
pubkey,
npub,
config,
advert_cache: RwLock::new(HashMap::new()),
local_advert: RwLock::new(None),
current_advert_event_id: RwLock::new(None),
advert,
traversal,
pending_answers: Mutex::new(HashMap::new()),
active_initiators: Mutex::new(HashSet::new()),
seen_sessions: Mutex::new(HashMap::new()),
offer_slots,
event_tx,
event_rx: Mutex::new(event_rx),
@@ -1806,8 +1680,7 @@ impl NostrDiscovery {
/// Insert a cached advert directly into the in-memory cache. Used by
/// unit tests to set up consumer-side state without needing live relays.
pub(crate) async fn insert_advert_for_test(&self, npub: String, advert: CachedOverlayAdvert) {
let mut cache = self.advert_cache.write().await;
cache.insert(npub, advert);
self.advert.insert_fetched(&npub, advert);
}
/// Queue a bootstrap event directly for lifecycle tests without live relays

View File

@@ -1,6 +1,6 @@
use nostr::prelude::{EventBuilder, Kind, Tag, Timestamp};
use super::runtime::{NostrDiscovery, suppress_responder_for_own_initiator};
use super::runtime::NostrRendezvous;
use super::signal::{
FreshnessOutcome, build_signal_event, create_traversal_answer, create_traversal_offer,
estimate_clock_skew, validate_offer_freshness, validate_traversal_answer_for_offer,
@@ -10,6 +10,7 @@ use super::traversal::{
PunchStrategy, build_punch_packet, parse_punch_packet, plan_punch_targets,
planned_remote_endpoints, session_hash,
};
use super::traversal_machine::suppress_responder_for_own_initiator;
use super::{
ADVERT_IDENTIFIER, ADVERT_KIND, ADVERT_VERSION, OverlayAdvert, OverlayEndpointAdvert,
OverlayTransportKind, PunchHint, PunchPacketKind, TraversalAddress,
@@ -104,7 +105,7 @@ fn rejects_invalid_overlay_adverts() {
signal_relays: None,
stun_servers: None,
};
assert!(NostrDiscovery::validate_overlay_advert(missing_nat_metadata).is_err());
assert!(NostrRendezvous::validate_overlay_advert(missing_nat_metadata).is_err());
let wrong_identifier = OverlayAdvert {
identifier: "not-fips-overlay".to_string(),
@@ -116,7 +117,7 @@ fn rejects_invalid_overlay_adverts() {
signal_relays: None,
stun_servers: None,
};
assert!(NostrDiscovery::validate_overlay_advert(wrong_identifier).is_err());
assert!(NostrRendezvous::validate_overlay_advert(wrong_identifier).is_err());
}
#[test]
@@ -142,7 +143,7 @@ fn validate_overlay_advert_filters_unroutable_direct_endpoints() {
stun_servers: None,
};
let validated = NostrDiscovery::validate_overlay_advert(advert).unwrap();
let validated = NostrRendezvous::validate_overlay_advert(advert).unwrap();
assert_eq!(validated.endpoints.len(), 1);
assert_eq!(validated.endpoints[0].addr, "8.8.8.8:443");
}
@@ -166,7 +167,7 @@ fn validate_overlay_advert_rejects_only_unroutable_direct_endpoints() {
stun_servers: None,
};
let err = NostrDiscovery::validate_overlay_advert(advert).unwrap_err();
let err = NostrRendezvous::validate_overlay_advert(advert).unwrap_err();
assert!(err.to_string().contains("missing publicly routable"));
}
@@ -175,7 +176,7 @@ fn advert_freshness_rejects_expired_events() {
let now_secs = Timestamp::now().as_secs();
let event = signed_overlay_advert_event(now_secs, Some(now_secs.saturating_sub(1)));
let valid_until =
NostrDiscovery::compute_advert_valid_until_ms(&event, 600_000, now_secs * 1000);
NostrRendezvous::compute_advert_valid_until_ms(&event, 600_000, now_secs * 1000);
assert!(valid_until.is_none());
}
@@ -185,7 +186,7 @@ fn advert_freshness_rejects_stale_created_at_without_expiration() {
let stale_created = now_secs.saturating_sub(10_000);
let event = signed_overlay_advert_event(stale_created, None);
let valid_until =
NostrDiscovery::compute_advert_valid_until_ms(&event, 600_000, now_secs * 1000);
NostrRendezvous::compute_advert_valid_until_ms(&event, 600_000, now_secs * 1000);
assert!(valid_until.is_none());
}
@@ -194,7 +195,7 @@ fn advert_freshness_uses_earliest_expiration_bound() {
let now_secs = Timestamp::now().as_secs();
let event = signed_overlay_advert_event(now_secs.saturating_sub(10), Some(now_secs + 30));
let valid_until =
NostrDiscovery::compute_advert_valid_until_ms(&event, 3_600_000, now_secs * 1000)
NostrRendezvous::compute_advert_valid_until_ms(&event, 3_600_000, now_secs * 1000)
.expect("event should be fresh");
assert_eq!(valid_until, (now_secs + 30) * 1000);
}

Some files were not shown because too many files have changed in this diff Show More