diff --git a/docs/design/fips-configuration.md b/docs/design/fips-configuration.md index 0025f8b..b9792c3 100644 --- a/docs/design/fips-configuration.md +++ b/docs/design/fips-configuration.md @@ -68,6 +68,8 @@ handle infrastructure concerns only. | `node.leaf_only` | bool | `false` | Leaf-only mode: node does not forward traffic or participate in routing | | `node.tick_interval_secs` | u64 | `1` | Periodic maintenance tick interval (retry checks, timeout cleanup, tree refresh) | | `node.base_rtt_ms` | u64 | `100` | Initial RTT estimate for new links before measurements converge | +| `node.heartbeat_interval_secs` | u64 | `10` | Heartbeat send interval per peer for liveness detection | +| `node.link_dead_timeout_secs` | u64 | `30` | No-traffic timeout before a peer is declared dead and removed | ### Resource Limits (`node.limits.*`) @@ -89,6 +91,9 @@ Handshake rate limiting protects against DoS on the Noise IK handshake path. | `node.rate_limit.handshake_burst` | u32 | `100` | Token bucket burst capacity | | `node.rate_limit.handshake_rate` | f64 | `10.0` | Tokens per second refill rate | | `node.rate_limit.handshake_timeout_secs` | u64 | `30` | Stale handshake cleanup timeout | +| `node.rate_limit.handshake_resend_interval_ms` | u64 | `1000` | Initial handshake message resend interval | +| `node.rate_limit.handshake_resend_backoff` | f64 | `2.0` | Resend backoff multiplier (1s, 2s, 4s, 8s, 16s with defaults) | +| `node.rate_limit.handshake_max_resends` | u32 | `5` | Max resends per handshake attempt | ### Retry / Backoff (`node.retry.*`) @@ -100,6 +105,10 @@ Connection retry with exponential backoff. | `node.retry.base_interval_secs` | u64 | `5` | Base backoff interval | | `node.retry.max_backoff_secs` | u64 | `300` | Cap on exponential backoff (5 minutes) | +Auto-reconnect (triggered by MMP link-dead removal) uses the same backoff +parameters but bypasses `max_retries`, retrying indefinitely. See +`peers[].auto_reconnect` below. + ### Cache Parameters (`node.cache.*`) Controls caching of tree coordinates and identity mappings. @@ -235,6 +244,7 @@ Static peer list. Each entry defines a peer to connect to. | `peers[].addresses[].addr` | string | *(required)* | Transport address (e.g., `"10.0.0.2:4000"`) | | `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) | | `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` | +| `peers[].auto_reconnect` | bool | `true` | Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries) | ## Minimal Example @@ -296,6 +306,8 @@ node: leaf_only: false tick_interval_secs: 1 base_rtt_ms: 100 + heartbeat_interval_secs: 10 + link_dead_timeout_secs: 30 limits: max_connections: 256 max_peers: 128 @@ -305,6 +317,9 @@ node: handshake_burst: 100 handshake_rate: 10.0 handshake_timeout_secs: 30 + handshake_resend_interval_ms: 1000 + handshake_resend_backoff: 2.0 + handshake_max_resends: 5 retry: max_retries: 5 base_interval_secs: 5 @@ -360,5 +375,13 @@ transports: recv_buf_size: 2097152 # 2 MB (kernel doubles to 4 MB actual) send_buf_size: 2097152 # 2 MB -peers: [] +peers: # static peer list + # - npub: "npub1..." + # alias: "node-b" + # addresses: + # - transport: udp + # addr: "10.0.0.2:4000" + # priority: 100 + # connect_policy: auto_connect + # auto_reconnect: true # reconnect after link-dead removal ``` diff --git a/docs/design/fips-link-layer.md b/docs/design/fips-link-layer.md index 9e5a250..bdcbf45 100644 --- a/docs/design/fips-link-layer.md +++ b/docs/design/fips-link-layer.md @@ -129,6 +129,50 @@ handshake completes successfully, it replaces the old session. This handles legitimate reconnection (network change, process restart, NAT rebinding) without disrupting ongoing traffic until the new session is confirmed. +### Auto-Reconnect + +When MMP's liveness detection removes a peer (dead timeout exceeded), FLP +automatically re-initiates the connection if the peer is configured for it. +The auto-reconnect path: + +1. `check_link_heartbeats()` detects the dead peer and calls + `remove_active_peer()`, tearing down the link and triggering tree/bloom + reconvergence +2. `schedule_reconnect()` checks whether the peer is in the auto-connect list + with `auto_reconnect: true` (the default) +3. If eligible, the peer is fed into the retry system with unlimited retries + and exponential backoff (same base interval and max backoff as startup + retries, configured via `node.retry.*`) +4. On each retry tick, a fresh Noise IK handshake is initiated toward the + peer's configured transport addresses + +Auto-reconnect only applies to peers in the static peer list with +`connect_policy: auto_connect`. Inbound-only peers (those not in the local +config) are not reconnected — the owning node (the one with the outbound +config) is responsible for re-establishing the link. + +### Handshake Message Retry + +Both link-layer (Noise IK msg1/msg2) and session-layer (SessionSetup/ +SessionAck) handshakes use message-level retry with exponential backoff +within the handshake timeout window. This handles packet loss on the +underlying transport without waiting for the full handshake timeout to +expire. + +Configuration (under `node.rate_limit.*`): + +- `handshake_resend_interval_ms` (default 1000): initial resend interval +- `handshake_resend_backoff` (default 2.0): backoff multiplier per resend +- `handshake_max_resends` (default 5): max resends per handshake attempt + +With defaults, resends occur at 1s, 2s, 4s, 8s, 16s — all within the 30s +handshake timeout. Under 19% per-attempt loss (10% bidirectional), the +probability of all 6 attempts (initial + 5 resends) failing is ~0.005%. + +Session-layer resends wrap the stored payload in a fresh SessionDatagram so +routing adapts to topology changes between resends. Responder idempotency: +duplicate msg1/SessionSetup triggers resend of the stored msg2/SessionAck. + ## Link Encryption All traffic between authenticated peers is encrypted. Every packet on a link @@ -284,21 +328,32 @@ stopped. ## Liveness Detection -FLP detects link liveness through timeout-based mechanisms. There are no -dedicated keepalive or ping/pong messages. Liveness is inferred from: +FLP detects link liveness through a combination of explicit heartbeats and +traffic observation. -- **Data traffic**: Any successfully decrypted packet confirms the peer is - alive -- **Gossip messages**: TreeAnnounce and FilterAnnounce messages sent - periodically as part of normal mesh operation serve as implicit heartbeats +### Heartbeat -When no traffic is received for the configured timeout period, FLP marks the -link as stale. If the stale state persists, the link is torn down. +A Heartbeat message (0x51) is sent to each active peer every +`node.heartbeat_interval_secs` (default 10s). The heartbeat is a minimal +encrypted frame with no payload beyond the standard inner header (timestamp + +message type). Any successfully decrypted frame — data, gossip, MMP report, +or heartbeat — resets the peer's last-receive timestamp tracked by the MMP +receiver. -The two-state liveness model: -- **Connected → Stale**: No traffic received for threshold duration -- **Stale → Connected**: Valid traffic received (peer is alive again) -- **Stale → Disconnected**: Timeout exceeded, link torn down +### Dead Timeout + +When no traffic (of any kind) is received from a peer for +`node.link_dead_timeout_secs` (default 30s), the peer is declared dead and +removed via `remove_active_peer()`. This triggers the full teardown cascade: +spanning tree parent reselection (if the dead peer was the parent), +TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute. + +If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect] +(#auto-reconnect)), reconnection is scheduled immediately after removal. + +The heartbeat is independent of MMP — it is needed because idle links in +Lightweight MMP mode have no guaranteed periodic traffic (gossip is +event-driven, and MMP reports require at least one side running Full mode). ## Link Message Types @@ -314,6 +369,7 @@ FLP defines eight message types carried inside encrypted frames: | 0x01 | SenderReport | MMP sender-side metrics report | | 0x02 | ReceiverReport | MMP receiver-side metrics report | | 0x50 | Disconnect | Orderly link teardown with reason code | +| 0x51 | Heartbeat | Link liveness probe | Additionally, handshake messages (phase 0x1 msg1, phase 0x2 msg2) are sent unencrypted before the link session is established. @@ -444,8 +500,10 @@ an attacker sends invalid packets to elicit responses. | Roaming (address-follows-crypto) | **Implemented** | | Rate limiting (token bucket) | **Implemented** | | Disconnect with reason codes | **Implemented** | -| Liveness detection (timeout-based) | **Implemented** | +| Heartbeat liveness detection | **Implemented** | | Reconnection handling | **Implemented** | +| Auto-reconnect after link-dead removal | **Implemented** | +| Handshake message retry (link + session layer) | **Implemented** | | Common prefix framing | **Implemented** | | AAD binding on encrypted frames | **Implemented** | | Inner header timestamps | **Implemented** | diff --git a/docs/design/fips-wire-formats.md b/docs/design/fips-wire-formats.md index 27de02e..2f65197 100644 --- a/docs/design/fips-wire-formats.md +++ b/docs/design/fips-wire-formats.md @@ -135,7 +135,7 @@ Minimum frame: 37 bytes (empty body) | 0x30 | LookupRequest | Coordinate discovery request | | 0x31 | LookupResponse | Coordinate discovery response | | 0x50 | Disconnect | Orderly link teardown | -| 0x51 | Keepalive | Keepalive probe (reserved) | +| 0x51 | Heartbeat | Link liveness probe | ### Noise IK Message 1 (phase 0x1) @@ -393,7 +393,7 @@ Orderly link teardown with reason code. | 0x04 | ResourceExhaustion | Memory or connection limit | | 0x05 | SecurityViolation | Authentication or policy violation | | 0x06 | ConfigurationChange | Peer removed from configuration | -| 0x07 | Timeout | Keepalive or stale detection timeout | +| 0x07 | Timeout | Heartbeat liveness timeout | | 0xFF | Other | Unspecified reason | ### SenderReport (0x01) diff --git a/testing/chaos/README.md b/testing/chaos/README.md index c77d320..2a25bb6 100644 --- a/testing/chaos/README.md +++ b/testing/chaos/README.md @@ -118,6 +118,15 @@ logging: When `ensure_connected` is true (default), the generator retries up to 50 times to produce a connected graph. +### Directed Outbound Configs + +The config generator assigns each edge to exactly one node for outbound +connection using a BFS spanning tree rooted at the lowest node ID. Tree +edges are assigned parent-to-child; non-tree edges are assigned from the +lower node ID to the higher. This eliminates the dual-connect race +condition where both sides initiate simultaneously, and creates a clear +"owning side" for each link — relevant for auto-reconnect testing. + ## Output Results written to `sim-results/` (configurable via @@ -125,6 +134,7 @@ Results written to `sim-results/` (configurable via - `analysis.txt` -- Summary: panics, errors, sessions, metrics - `metadata.txt` -- Seed, node count, edges, adjacency list +- `runner.log` -- Orchestration events (topology, netem, churn, traffic) with timestamps - `fips-node-nXX.log` -- Per-node log output Exit code 0 on success, 2 if panics detected.