mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-22 07:48:26 +00:00
Update design docs for heartbeat, auto-reconnect, handshake retry, and sim improvements
fips-link-layer.md: - Rewrite Liveness Detection: explicit Heartbeat (0x51) with 10s interval and 30s dead timeout replaces vague gossip-as-heartbeat description - Add Auto-Reconnect section: MMP dead timeout triggers retry with unlimited backoff for auto_reconnect peers - Add Handshake Message Retry section: link + session layer resend with exponential backoff within timeout window - Add Heartbeat to Link Message Types table - Update Implementation Status with three new implemented features fips-configuration.md: - Add handshake_resend_interval_ms, handshake_resend_backoff, handshake_max_resends to rate_limit table - Add heartbeat_interval_secs, link_dead_timeout_secs to general table - Add peers[].auto_reconnect to peers table - Note auto-reconnect bypasses max_retries in retry section - Update complete reference YAML with all new parameters fips-wire-formats.md: - Rename 0x51 from reserved Keepalive to implemented Heartbeat - Update Disconnect reason 0x07 to Heartbeat liveness timeout testing/chaos/README.md: - Add runner.log to output files - Add Directed Outbound Configs subsection
This commit is contained in:
@@ -68,6 +68,8 @@ handle infrastructure concerns only.
|
||||
| `node.leaf_only` | bool | `false` | Leaf-only mode: node does not forward traffic or participate in routing |
|
||||
| `node.tick_interval_secs` | u64 | `1` | Periodic maintenance tick interval (retry checks, timeout cleanup, tree refresh) |
|
||||
| `node.base_rtt_ms` | u64 | `100` | Initial RTT estimate for new links before measurements converge |
|
||||
| `node.heartbeat_interval_secs` | u64 | `10` | Heartbeat send interval per peer for liveness detection |
|
||||
| `node.link_dead_timeout_secs` | u64 | `30` | No-traffic timeout before a peer is declared dead and removed |
|
||||
|
||||
### Resource Limits (`node.limits.*`)
|
||||
|
||||
@@ -89,6 +91,9 @@ Handshake rate limiting protects against DoS on the Noise IK handshake path.
|
||||
| `node.rate_limit.handshake_burst` | u32 | `100` | Token bucket burst capacity |
|
||||
| `node.rate_limit.handshake_rate` | f64 | `10.0` | Tokens per second refill rate |
|
||||
| `node.rate_limit.handshake_timeout_secs` | u64 | `30` | Stale handshake cleanup timeout |
|
||||
| `node.rate_limit.handshake_resend_interval_ms` | u64 | `1000` | Initial handshake message resend interval |
|
||||
| `node.rate_limit.handshake_resend_backoff` | f64 | `2.0` | Resend backoff multiplier (1s, 2s, 4s, 8s, 16s with defaults) |
|
||||
| `node.rate_limit.handshake_max_resends` | u32 | `5` | Max resends per handshake attempt |
|
||||
|
||||
### Retry / Backoff (`node.retry.*`)
|
||||
|
||||
@@ -100,6 +105,10 @@ Connection retry with exponential backoff.
|
||||
| `node.retry.base_interval_secs` | u64 | `5` | Base backoff interval |
|
||||
| `node.retry.max_backoff_secs` | u64 | `300` | Cap on exponential backoff (5 minutes) |
|
||||
|
||||
Auto-reconnect (triggered by MMP link-dead removal) uses the same backoff
|
||||
parameters but bypasses `max_retries`, retrying indefinitely. See
|
||||
`peers[].auto_reconnect` below.
|
||||
|
||||
### Cache Parameters (`node.cache.*`)
|
||||
|
||||
Controls caching of tree coordinates and identity mappings.
|
||||
@@ -235,6 +244,7 @@ Static peer list. Each entry defines a peer to connect to.
|
||||
| `peers[].addresses[].addr` | string | *(required)* | Transport address (e.g., `"10.0.0.2:4000"`) |
|
||||
| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) |
|
||||
| `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` |
|
||||
| `peers[].auto_reconnect` | bool | `true` | Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries) |
|
||||
|
||||
## Minimal Example
|
||||
|
||||
@@ -296,6 +306,8 @@ node:
|
||||
leaf_only: false
|
||||
tick_interval_secs: 1
|
||||
base_rtt_ms: 100
|
||||
heartbeat_interval_secs: 10
|
||||
link_dead_timeout_secs: 30
|
||||
limits:
|
||||
max_connections: 256
|
||||
max_peers: 128
|
||||
@@ -305,6 +317,9 @@ node:
|
||||
handshake_burst: 100
|
||||
handshake_rate: 10.0
|
||||
handshake_timeout_secs: 30
|
||||
handshake_resend_interval_ms: 1000
|
||||
handshake_resend_backoff: 2.0
|
||||
handshake_max_resends: 5
|
||||
retry:
|
||||
max_retries: 5
|
||||
base_interval_secs: 5
|
||||
@@ -360,5 +375,13 @@ transports:
|
||||
recv_buf_size: 2097152 # 2 MB (kernel doubles to 4 MB actual)
|
||||
send_buf_size: 2097152 # 2 MB
|
||||
|
||||
peers: []
|
||||
peers: # static peer list
|
||||
# - npub: "npub1..."
|
||||
# alias: "node-b"
|
||||
# addresses:
|
||||
# - transport: udp
|
||||
# addr: "10.0.0.2:4000"
|
||||
# priority: 100
|
||||
# connect_policy: auto_connect
|
||||
# auto_reconnect: true # reconnect after link-dead removal
|
||||
```
|
||||
|
||||
@@ -129,6 +129,50 @@ handshake completes successfully, it replaces the old session. This handles
|
||||
legitimate reconnection (network change, process restart, NAT rebinding)
|
||||
without disrupting ongoing traffic until the new session is confirmed.
|
||||
|
||||
### Auto-Reconnect
|
||||
|
||||
When MMP's liveness detection removes a peer (dead timeout exceeded), FLP
|
||||
automatically re-initiates the connection if the peer is configured for it.
|
||||
The auto-reconnect path:
|
||||
|
||||
1. `check_link_heartbeats()` detects the dead peer and calls
|
||||
`remove_active_peer()`, tearing down the link and triggering tree/bloom
|
||||
reconvergence
|
||||
2. `schedule_reconnect()` checks whether the peer is in the auto-connect list
|
||||
with `auto_reconnect: true` (the default)
|
||||
3. If eligible, the peer is fed into the retry system with unlimited retries
|
||||
and exponential backoff (same base interval and max backoff as startup
|
||||
retries, configured via `node.retry.*`)
|
||||
4. On each retry tick, a fresh Noise IK handshake is initiated toward the
|
||||
peer's configured transport addresses
|
||||
|
||||
Auto-reconnect only applies to peers in the static peer list with
|
||||
`connect_policy: auto_connect`. Inbound-only peers (those not in the local
|
||||
config) are not reconnected — the owning node (the one with the outbound
|
||||
config) is responsible for re-establishing the link.
|
||||
|
||||
### Handshake Message Retry
|
||||
|
||||
Both link-layer (Noise IK msg1/msg2) and session-layer (SessionSetup/
|
||||
SessionAck) handshakes use message-level retry with exponential backoff
|
||||
within the handshake timeout window. This handles packet loss on the
|
||||
underlying transport without waiting for the full handshake timeout to
|
||||
expire.
|
||||
|
||||
Configuration (under `node.rate_limit.*`):
|
||||
|
||||
- `handshake_resend_interval_ms` (default 1000): initial resend interval
|
||||
- `handshake_resend_backoff` (default 2.0): backoff multiplier per resend
|
||||
- `handshake_max_resends` (default 5): max resends per handshake attempt
|
||||
|
||||
With defaults, resends occur at 1s, 2s, 4s, 8s, 16s — all within the 30s
|
||||
handshake timeout. Under 19% per-attempt loss (10% bidirectional), the
|
||||
probability of all 6 attempts (initial + 5 resends) failing is ~0.005%.
|
||||
|
||||
Session-layer resends wrap the stored payload in a fresh SessionDatagram so
|
||||
routing adapts to topology changes between resends. Responder idempotency:
|
||||
duplicate msg1/SessionSetup triggers resend of the stored msg2/SessionAck.
|
||||
|
||||
## Link Encryption
|
||||
|
||||
All traffic between authenticated peers is encrypted. Every packet on a link
|
||||
@@ -284,21 +328,32 @@ stopped.
|
||||
|
||||
## Liveness Detection
|
||||
|
||||
FLP detects link liveness through timeout-based mechanisms. There are no
|
||||
dedicated keepalive or ping/pong messages. Liveness is inferred from:
|
||||
FLP detects link liveness through a combination of explicit heartbeats and
|
||||
traffic observation.
|
||||
|
||||
- **Data traffic**: Any successfully decrypted packet confirms the peer is
|
||||
alive
|
||||
- **Gossip messages**: TreeAnnounce and FilterAnnounce messages sent
|
||||
periodically as part of normal mesh operation serve as implicit heartbeats
|
||||
### Heartbeat
|
||||
|
||||
When no traffic is received for the configured timeout period, FLP marks the
|
||||
link as stale. If the stale state persists, the link is torn down.
|
||||
A Heartbeat message (0x51) is sent to each active peer every
|
||||
`node.heartbeat_interval_secs` (default 10s). The heartbeat is a minimal
|
||||
encrypted frame with no payload beyond the standard inner header (timestamp +
|
||||
message type). Any successfully decrypted frame — data, gossip, MMP report,
|
||||
or heartbeat — resets the peer's last-receive timestamp tracked by the MMP
|
||||
receiver.
|
||||
|
||||
The two-state liveness model:
|
||||
- **Connected → Stale**: No traffic received for threshold duration
|
||||
- **Stale → Connected**: Valid traffic received (peer is alive again)
|
||||
- **Stale → Disconnected**: Timeout exceeded, link torn down
|
||||
### Dead Timeout
|
||||
|
||||
When no traffic (of any kind) is received from a peer for
|
||||
`node.link_dead_timeout_secs` (default 30s), the peer is declared dead and
|
||||
removed via `remove_active_peer()`. This triggers the full teardown cascade:
|
||||
spanning tree parent reselection (if the dead peer was the parent),
|
||||
TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.
|
||||
|
||||
If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect]
|
||||
(#auto-reconnect)), reconnection is scheduled immediately after removal.
|
||||
|
||||
The heartbeat is independent of MMP — it is needed because idle links in
|
||||
Lightweight MMP mode have no guaranteed periodic traffic (gossip is
|
||||
event-driven, and MMP reports require at least one side running Full mode).
|
||||
|
||||
## Link Message Types
|
||||
|
||||
@@ -314,6 +369,7 @@ FLP defines eight message types carried inside encrypted frames:
|
||||
| 0x01 | SenderReport | MMP sender-side metrics report |
|
||||
| 0x02 | ReceiverReport | MMP receiver-side metrics report |
|
||||
| 0x50 | Disconnect | Orderly link teardown with reason code |
|
||||
| 0x51 | Heartbeat | Link liveness probe |
|
||||
|
||||
Additionally, handshake messages (phase 0x1 msg1, phase 0x2 msg2) are sent
|
||||
unencrypted before the link session is established.
|
||||
@@ -444,8 +500,10 @@ an attacker sends invalid packets to elicit responses.
|
||||
| Roaming (address-follows-crypto) | **Implemented** |
|
||||
| Rate limiting (token bucket) | **Implemented** |
|
||||
| Disconnect with reason codes | **Implemented** |
|
||||
| Liveness detection (timeout-based) | **Implemented** |
|
||||
| Heartbeat liveness detection | **Implemented** |
|
||||
| Reconnection handling | **Implemented** |
|
||||
| Auto-reconnect after link-dead removal | **Implemented** |
|
||||
| Handshake message retry (link + session layer) | **Implemented** |
|
||||
| Common prefix framing | **Implemented** |
|
||||
| AAD binding on encrypted frames | **Implemented** |
|
||||
| Inner header timestamps | **Implemented** |
|
||||
|
||||
@@ -135,7 +135,7 @@ Minimum frame: 37 bytes (empty body)
|
||||
| 0x30 | LookupRequest | Coordinate discovery request |
|
||||
| 0x31 | LookupResponse | Coordinate discovery response |
|
||||
| 0x50 | Disconnect | Orderly link teardown |
|
||||
| 0x51 | Keepalive | Keepalive probe (reserved) |
|
||||
| 0x51 | Heartbeat | Link liveness probe |
|
||||
|
||||
### Noise IK Message 1 (phase 0x1)
|
||||
|
||||
@@ -393,7 +393,7 @@ Orderly link teardown with reason code.
|
||||
| 0x04 | ResourceExhaustion | Memory or connection limit |
|
||||
| 0x05 | SecurityViolation | Authentication or policy violation |
|
||||
| 0x06 | ConfigurationChange | Peer removed from configuration |
|
||||
| 0x07 | Timeout | Keepalive or stale detection timeout |
|
||||
| 0x07 | Timeout | Heartbeat liveness timeout |
|
||||
| 0xFF | Other | Unspecified reason |
|
||||
|
||||
### SenderReport (0x01)
|
||||
|
||||
@@ -118,6 +118,15 @@ logging:
|
||||
When `ensure_connected` is true (default), the generator retries up to
|
||||
50 times to produce a connected graph.
|
||||
|
||||
### Directed Outbound Configs
|
||||
|
||||
The config generator assigns each edge to exactly one node for outbound
|
||||
connection using a BFS spanning tree rooted at the lowest node ID. Tree
|
||||
edges are assigned parent-to-child; non-tree edges are assigned from the
|
||||
lower node ID to the higher. This eliminates the dual-connect race
|
||||
condition where both sides initiate simultaneously, and creates a clear
|
||||
"owning side" for each link — relevant for auto-reconnect testing.
|
||||
|
||||
## Output
|
||||
|
||||
Results written to `sim-results/` (configurable via
|
||||
@@ -125,6 +134,7 @@ Results written to `sim-results/` (configurable via
|
||||
|
||||
- `analysis.txt` -- Summary: panics, errors, sessions, metrics
|
||||
- `metadata.txt` -- Seed, node count, edges, adjacency list
|
||||
- `runner.log` -- Orchestration events (topology, netem, churn, traffic) with timestamps
|
||||
- `fips-node-nXX.log` -- Per-node log output
|
||||
|
||||
Exit code 0 on success, 2 if panics detected.
|
||||
|
||||
Reference in New Issue
Block a user