Update design docs for heartbeat, auto-reconnect, handshake retry, and sim improvements

fips-link-layer.md:
- Rewrite Liveness Detection: explicit Heartbeat (0x51) with 10s interval
  and 30s dead timeout replaces vague gossip-as-heartbeat description
- Add Auto-Reconnect section: MMP dead timeout triggers retry with
  unlimited backoff for auto_reconnect peers
- Add Handshake Message Retry section: link + session layer resend with
  exponential backoff within timeout window
- Add Heartbeat to Link Message Types table
- Update Implementation Status with three new implemented features

fips-configuration.md:
- Add handshake_resend_interval_ms, handshake_resend_backoff,
  handshake_max_resends to rate_limit table
- Add heartbeat_interval_secs, link_dead_timeout_secs to general table
- Add peers[].auto_reconnect to peers table
- Note auto-reconnect bypasses max_retries in retry section
- Update complete reference YAML with all new parameters

fips-wire-formats.md:
- Rename 0x51 from reserved Keepalive to implemented Heartbeat
- Update Disconnect reason 0x07 to Heartbeat liveness timeout

testing/chaos/README.md:
- Add runner.log to output files
- Add Directed Outbound Configs subsection
This commit is contained in:
Johnathan Corgan
2026-02-21 13:10:02 +00:00
parent 78a73e1749
commit cfb087a95d
4 changed files with 107 additions and 16 deletions

View File

@@ -68,6 +68,8 @@ handle infrastructure concerns only.
| `node.leaf_only` | bool | `false` | Leaf-only mode: node does not forward traffic or participate in routing |
| `node.tick_interval_secs` | u64 | `1` | Periodic maintenance tick interval (retry checks, timeout cleanup, tree refresh) |
| `node.base_rtt_ms` | u64 | `100` | Initial RTT estimate for new links before measurements converge |
| `node.heartbeat_interval_secs` | u64 | `10` | Heartbeat send interval per peer for liveness detection |
| `node.link_dead_timeout_secs` | u64 | `30` | No-traffic timeout before a peer is declared dead and removed |
### Resource Limits (`node.limits.*`)
@@ -89,6 +91,9 @@ Handshake rate limiting protects against DoS on the Noise IK handshake path.
| `node.rate_limit.handshake_burst` | u32 | `100` | Token bucket burst capacity |
| `node.rate_limit.handshake_rate` | f64 | `10.0` | Tokens per second refill rate |
| `node.rate_limit.handshake_timeout_secs` | u64 | `30` | Stale handshake cleanup timeout |
| `node.rate_limit.handshake_resend_interval_ms` | u64 | `1000` | Initial handshake message resend interval |
| `node.rate_limit.handshake_resend_backoff` | f64 | `2.0` | Resend backoff multiplier (1s, 2s, 4s, 8s, 16s with defaults) |
| `node.rate_limit.handshake_max_resends` | u32 | `5` | Max resends per handshake attempt |
### Retry / Backoff (`node.retry.*`)
@@ -100,6 +105,10 @@ Connection retry with exponential backoff.
| `node.retry.base_interval_secs` | u64 | `5` | Base backoff interval |
| `node.retry.max_backoff_secs` | u64 | `300` | Cap on exponential backoff (5 minutes) |
Auto-reconnect (triggered by MMP link-dead removal) uses the same backoff
parameters but bypasses `max_retries`, retrying indefinitely. See
`peers[].auto_reconnect` below.
### Cache Parameters (`node.cache.*`)
Controls caching of tree coordinates and identity mappings.
@@ -235,6 +244,7 @@ Static peer list. Each entry defines a peer to connect to.
| `peers[].addresses[].addr` | string | *(required)* | Transport address (e.g., `"10.0.0.2:4000"`) |
| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) |
| `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` |
| `peers[].auto_reconnect` | bool | `true` | Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries) |
## Minimal Example
@@ -296,6 +306,8 @@ node:
leaf_only: false
tick_interval_secs: 1
base_rtt_ms: 100
heartbeat_interval_secs: 10
link_dead_timeout_secs: 30
limits:
max_connections: 256
max_peers: 128
@@ -305,6 +317,9 @@ node:
handshake_burst: 100
handshake_rate: 10.0
handshake_timeout_secs: 30
handshake_resend_interval_ms: 1000
handshake_resend_backoff: 2.0
handshake_max_resends: 5
retry:
max_retries: 5
base_interval_secs: 5
@@ -360,5 +375,13 @@ transports:
recv_buf_size: 2097152 # 2 MB (kernel doubles to 4 MB actual)
send_buf_size: 2097152 # 2 MB
peers: []
peers: # static peer list
# - npub: "npub1..."
# alias: "node-b"
# addresses:
# - transport: udp
# addr: "10.0.0.2:4000"
# priority: 100
# connect_policy: auto_connect
# auto_reconnect: true # reconnect after link-dead removal
```

View File

@@ -129,6 +129,50 @@ handshake completes successfully, it replaces the old session. This handles
legitimate reconnection (network change, process restart, NAT rebinding)
without disrupting ongoing traffic until the new session is confirmed.
### Auto-Reconnect
When MMP's liveness detection removes a peer (dead timeout exceeded), FLP
automatically re-initiates the connection if the peer is configured for it.
The auto-reconnect path:
1. `check_link_heartbeats()` detects the dead peer and calls
`remove_active_peer()`, tearing down the link and triggering tree/bloom
reconvergence
2. `schedule_reconnect()` checks whether the peer is in the auto-connect list
with `auto_reconnect: true` (the default)
3. If eligible, the peer is fed into the retry system with unlimited retries
and exponential backoff (same base interval and max backoff as startup
retries, configured via `node.retry.*`)
4. On each retry tick, a fresh Noise IK handshake is initiated toward the
peer's configured transport addresses
Auto-reconnect only applies to peers in the static peer list with
`connect_policy: auto_connect`. Inbound-only peers (those not in the local
config) are not reconnected — the owning node (the one with the outbound
config) is responsible for re-establishing the link.
### Handshake Message Retry
Both link-layer (Noise IK msg1/msg2) and session-layer (SessionSetup/
SessionAck) handshakes use message-level retry with exponential backoff
within the handshake timeout window. This handles packet loss on the
underlying transport without waiting for the full handshake timeout to
expire.
Configuration (under `node.rate_limit.*`):
- `handshake_resend_interval_ms` (default 1000): initial resend interval
- `handshake_resend_backoff` (default 2.0): backoff multiplier per resend
- `handshake_max_resends` (default 5): max resends per handshake attempt
With defaults, resends occur at 1s, 2s, 4s, 8s, 16s — all within the 30s
handshake timeout. Under 19% per-attempt loss (10% bidirectional), the
probability of all 6 attempts (initial + 5 resends) failing is ~0.005%.
Session-layer resends wrap the stored payload in a fresh SessionDatagram so
routing adapts to topology changes between resends. Responder idempotency:
duplicate msg1/SessionSetup triggers resend of the stored msg2/SessionAck.
## Link Encryption
All traffic between authenticated peers is encrypted. Every packet on a link
@@ -284,21 +328,32 @@ stopped.
## Liveness Detection
FLP detects link liveness through timeout-based mechanisms. There are no
dedicated keepalive or ping/pong messages. Liveness is inferred from:
FLP detects link liveness through a combination of explicit heartbeats and
traffic observation.
- **Data traffic**: Any successfully decrypted packet confirms the peer is
alive
- **Gossip messages**: TreeAnnounce and FilterAnnounce messages sent
periodically as part of normal mesh operation serve as implicit heartbeats
### Heartbeat
When no traffic is received for the configured timeout period, FLP marks the
link as stale. If the stale state persists, the link is torn down.
A Heartbeat message (0x51) is sent to each active peer every
`node.heartbeat_interval_secs` (default 10s). The heartbeat is a minimal
encrypted frame with no payload beyond the standard inner header (timestamp +
message type). Any successfully decrypted frame — data, gossip, MMP report,
or heartbeat — resets the peer's last-receive timestamp tracked by the MMP
receiver.
The two-state liveness model:
- **Connected → Stale**: No traffic received for threshold duration
- **Stale → Connected**: Valid traffic received (peer is alive again)
- **Stale → Disconnected**: Timeout exceeded, link torn down
### Dead Timeout
When no traffic (of any kind) is received from a peer for
`node.link_dead_timeout_secs` (default 30s), the peer is declared dead and
removed via `remove_active_peer()`. This triggers the full teardown cascade:
spanning tree parent reselection (if the dead peer was the parent),
TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.
If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect]
(#auto-reconnect)), reconnection is scheduled immediately after removal.
The heartbeat is independent of MMP — it is needed because idle links in
Lightweight MMP mode have no guaranteed periodic traffic (gossip is
event-driven, and MMP reports require at least one side running Full mode).
## Link Message Types
@@ -314,6 +369,7 @@ FLP defines eight message types carried inside encrypted frames:
| 0x01 | SenderReport | MMP sender-side metrics report |
| 0x02 | ReceiverReport | MMP receiver-side metrics report |
| 0x50 | Disconnect | Orderly link teardown with reason code |
| 0x51 | Heartbeat | Link liveness probe |
Additionally, handshake messages (phase 0x1 msg1, phase 0x2 msg2) are sent
unencrypted before the link session is established.
@@ -444,8 +500,10 @@ an attacker sends invalid packets to elicit responses.
| Roaming (address-follows-crypto) | **Implemented** |
| Rate limiting (token bucket) | **Implemented** |
| Disconnect with reason codes | **Implemented** |
| Liveness detection (timeout-based) | **Implemented** |
| Heartbeat liveness detection | **Implemented** |
| Reconnection handling | **Implemented** |
| Auto-reconnect after link-dead removal | **Implemented** |
| Handshake message retry (link + session layer) | **Implemented** |
| Common prefix framing | **Implemented** |
| AAD binding on encrypted frames | **Implemented** |
| Inner header timestamps | **Implemented** |

View File

@@ -135,7 +135,7 @@ Minimum frame: 37 bytes (empty body)
| 0x30 | LookupRequest | Coordinate discovery request |
| 0x31 | LookupResponse | Coordinate discovery response |
| 0x50 | Disconnect | Orderly link teardown |
| 0x51 | Keepalive | Keepalive probe (reserved) |
| 0x51 | Heartbeat | Link liveness probe |
### Noise IK Message 1 (phase 0x1)
@@ -393,7 +393,7 @@ Orderly link teardown with reason code.
| 0x04 | ResourceExhaustion | Memory or connection limit |
| 0x05 | SecurityViolation | Authentication or policy violation |
| 0x06 | ConfigurationChange | Peer removed from configuration |
| 0x07 | Timeout | Keepalive or stale detection timeout |
| 0x07 | Timeout | Heartbeat liveness timeout |
| 0xFF | Other | Unspecified reason |
### SenderReport (0x01)

View File

@@ -118,6 +118,15 @@ logging:
When `ensure_connected` is true (default), the generator retries up to
50 times to produce a connected graph.
### Directed Outbound Configs
The config generator assigns each edge to exactly one node for outbound
connection using a BFS spanning tree rooted at the lowest node ID. Tree
edges are assigned parent-to-child; non-tree edges are assigned from the
lower node ID to the higher. This eliminates the dual-connect race
condition where both sides initiate simultaneously, and creates a clear
"owning side" for each link — relevant for auto-reconnect testing.
## Output
Results written to `sim-results/` (configurable via
@@ -125,6 +134,7 @@ Results written to `sim-results/` (configurable via
- `analysis.txt` -- Summary: panics, errors, sessions, metrics
- `metadata.txt` -- Seed, node count, edges, adjacency list
- `runner.log` -- Orchestration events (topology, netem, churn, traffic) with timestamps
- `fips-node-nXX.log` -- Per-node log output
Exit code 0 on success, 2 if panics detected.