Files
fips/docs/design/fips-mesh-operation.md
Johnathan Corgan a16370e78d Update changelog, version, and design docs for v0.2.0
Bump version to 0.2.0 and finalize changelog with discovery rework,
Tor transport, connect/disconnect commands, reproducible builds, and
12 bug fixes.

Update design documentation for discovery protocol rework:
- fips-wire-formats.md: remove visited bloom filter from LookupRequest,
  update size calculations
- fips-mesh-operation.md: replace flooding description with bloom-guided
  tree routing, add retry/backoff/rate-limiting subsections
- fips-configuration.md: add 5 new discovery config parameters, update
  control socket description for connect/disconnect commands
2026-03-22 20:26:09 +00:00

33 KiB
Raw Blame History

FIPS Mesh Operation

This document describes how the FIPS mesh operates at the link layer — how spanning tree, bloom filters, routing decisions, discovery, and error recovery work together as a coherent system. It treats spanning tree and bloom filters as black boxes (what they provide to routing) and focuses on how the pieces interact.

For spanning tree algorithms and data structures, see fips-spanning-tree.md. For bloom filter parameters and mathematics, see fips-bloom-filters.md.

Overview

FIPS mesh operation is entirely distributed. Each node makes forwarding decisions using only local information: its direct peers, their spanning tree positions, and their bloom filters. There are no routing tables pushed from above, no link-state floods, and no distance-vector exchanges.

Two complementary mechanisms provide the information each node needs:

  • Spanning tree gives every node a coordinate in the network — its ancestry path from itself to the root. These coordinates enable distance calculations between any two nodes without global topology knowledge.
  • Bloom filters summarize which destinations are reachable through each peer. Because they propagate along tree edges, they encode directional reachability — which subtree contains a given destination.

Together, they enable a routing decision process that is local, efficient, and self-healing.

Spanning Tree Formation and Maintenance

What the Spanning Tree Provides

The spanning tree gives each node a coordinate: its ancestry path from itself to the root, expressed as a sequence of node_addrs. These coordinates enable:

  • Distance calculation: The tree distance between two nodes is the number of hops from each to their lowest common ancestor (LCA). This provides a routing metric without any node knowing the full topology.
  • Greedy routing: At each hop, forward to the peer that minimizes tree distance to the destination. The strictly-decreasing distance invariant guarantees loop-free forwarding.

How the Tree Forms

Nodes self-organize into a spanning tree through distributed parent selection:

  1. Root discovery: The node with the smallest node_addr becomes the root. No election protocol — this is a consequence of each node independently preferring lower-addressed roots.
  2. Parent selection: Each node selects a single parent from among its direct peers based on which offers the lowest effective depth (tree depth weighted by local link cost).
  3. Coordinate computation: Once a node has a parent, its coordinate is computed from its ancestry path.

How the Tree Maintains Itself

Nodes exchange TreeAnnounce messages with their direct peers (not forwarded — peer-to-peer only). Each TreeAnnounce carries the sender's current ancestry chain and a sequence number.

Changes cascade through the tree:

  • A node that changes its parent recomputes its coordinates and announces to all peers
  • Each receiving peer evaluates whether the change affects its own parent selection
  • Only nodes that actually change their coordinates (root or depth changed) propagate further

TreeAnnounce propagation is rate-limited at 500ms minimum interval per peer. A tree of depth D reconverges in roughly D×0.5s to D×1.0s.

The initial tree forms based on hop count alone — all links default to a cost of 1.0 before measurements are available. As the Metrics Measurement Protocol (MMP) accumulates bidirectional delivery ratios and round-trip time estimates, each node computes a per-link cost:

link_cost = ETX × (1.0 + SRTT_ms / 100.0)

ETX (Expected Transmission Count) captures loss — a perfect link has ETX = 1.0, while 10% loss in each direction yields ETX ≈ 1.23. The SRTT term weights latency so that a low-loss but high-latency link (e.g., a satellite hop) costs more than a low-loss, low-latency link.

Parent selection uses effective depth rather than raw hop count:

effective_depth = peer.depth + link_cost_to_peer

This allows a node to trade a shorter but lossy path for a longer but higher-quality one. A node two hops from the root over clean links (effective depth ≈ 3.0) is preferred over a node one hop away over a degraded link (effective depth ≈ 4.5).

Parent reselection is triggered by three paths:

  1. TreeAnnounce: When a peer announces a new tree position, the node re-evaluates using current link costs
  2. Periodic re-evaluation: Every 60s (configurable), the node re-evaluates its parent choice using the latest MMP metrics, catching gradual link degradation that doesn't trigger TreeAnnounce
  3. Parent loss: When the current parent is removed, the node immediately selects the best alternative

To prevent oscillation from metric noise, parent switches are subject to hysteresis: a candidate must offer an effective depth at least 20% better than the current parent to trigger a switch. A hold-down period (default 30s) suppresses non-mandatory re-evaluation after a switch, allowing MMP metrics to stabilize on the new link before reconsidering.

Flap Dampening

Unstable links that repeatedly connect and disconnect can cause cascading tree reconvergence. The spanning tree uses flap dampening with hysteresis and hold-down periods to suppress rapid parent oscillation. Links that flap above a configurable threshold are temporarily penalized, preventing them from being selected as parent until the link stabilizes.

Each node sends a dedicated Heartbeat message (0x51, 1 byte, no payload) to every peer at a fixed interval (default 10s). Any authenticated encrypted frame — heartbeat, MMP report, TreeAnnounce, data packet — resets the peer's liveness timer. On an idle link with no application data or topology changes, the heartbeat is the only traffic that keeps the link alive.

Peers that are silent for a configurable dead timeout (default 30s) are considered dead and removed from the peer table. With the default 10s heartbeat interval, a peer must miss three consecutive heartbeats before removal. This triggers tree reconvergence and bloom filter recomputation for the affected subtree.

Partition Handling

If the network partitions, each segment independently rediscovers its own root (the smallest node_addr in the segment) and reconverges. When segments rejoin, nodes discover the globally-smallest root through TreeAnnounce exchange and reconverge to a single tree.

See fips-spanning-tree.md for algorithm details and spanning-tree-dynamics.md for convergence walkthroughs.

Bloom Filter Gossip and Propagation

What Bloom Filters Provide

Each node maintains a bloom filter per peer, answering: "can peer P possibly reach destination D?" The answer is either "no" (definitive) or "maybe" (probabilistic — false positives are possible).

Because filters propagate along tree edges with split-horizon exclusion, they encode directional reachability: a bloom hit on a tree peer reliably indicates which subtree contains the destination. When multiple peers match, tree coordinate distance ranks them.

How Filters Propagate

Nodes exchange FilterAnnounce messages with all direct peers. Each FilterAnnounce replaces the previous filter for that peer — there is no incremental update.

Filter computation uses tree-only merge with split-horizon exclusion: the outbound filter for peer Q is computed by merging the local node's own identity, its leaf-only dependents (if any), and the inbound filters from tree peers (parent and children) except Q. Filters from non-tree mesh peers are stored locally for routing queries but are not merged into outgoing filters. This prevents saturation where mesh shortcuts cause filters to converge toward the full network.

The restriction creates directional asymmetry: upward filters (child → parent) contain the child's subtree, while downward filters (parent → child) contain the complement. Together they cover the entire network.

Filters propagate transitively through tree edges. At steady state, every reachable destination appears in at least one tree peer's filter.

Update Triggers

Filter updates are event-driven, not periodic:

  • Peer connects or disconnects
  • A peer's incoming filter changes (triggers recomputation for other peers)
  • Tree relationship changes (new parent, new child, parent switch)
  • Local state changes (new identity, leaf-only dependent changes)

Updates are rate-limited at 500ms to prevent storms during topology changes.

Scale Properties

At moderate network sizes, bloom filters are highly accurate. At larger scales (~1M nodes), hub nodes with many peers may see elevated false positive rates (715% for nodes with 20+ peers). False positives may cause a packet to be forwarded toward the wrong subtree, but the self-distance check at each hop prevents loops and the packet falls through to greedy tree routing.

See fips-bloom-filters.md for filter parameters, FPR calculations, and size class folding.

Routing Decision Process

At each hop, FMP makes a local forwarding decision using the find_next_hop() priority chain. This is the core routing algorithm.

Priority Chain

  1. Local delivery — The destination node_addr matches the local node. Deliver to FSP above.

  2. Direct peer — The destination is an authenticated neighbor. Forward directly. No coordinates or bloom filters needed.

  3. Bloom-guided routing — One or more peers' bloom filters contain the destination. Select the best peer by composite key: (link_cost, tree_distance, node_addr). This requires the destination's tree coordinates to be in the local coordinate cache.

  4. Greedy tree routing — Fallback when bloom filters haven't converged for this destination. Forward to the peer that minimizes tree distance. Also requires destination coordinates.

  5. No route — Destination unreachable. Generate an error signal (CoordsRequired or PathBroken) back to the source.

The Coordinate Requirement

All multi-hop routing (steps 34) requires the destination's tree coordinates to be in the local coordinate cache. Without coordinates, find_next_hop() returns None immediately — bloom filters are never even consulted.

This creates two simultaneous convergence requirements for multi-hop routing:

  1. Bloom convergence: Filters must propagate so peers advertise reachability
  2. Coordinate availability: Destination coordinates must be cached at every transit node on the path

Both must be satisfied simultaneously. Bloom convergence without coordinates causes a coordinate cache miss. Coordinates without bloom convergence falls through to greedy tree routing (functional but suboptimal).

Candidate Ranking

When bloom filters identify multiple candidate peers, they are ranked by a composite key:

  1. link_cost — Per-link quality metric derived from ETX (Expected Transmission Count), computed from bidirectional delivery ratios in MMP metrics. In practice this is an uncommon tie-breaker: most forwarding decisions are resolved by tree distance alone, and link_cost only differentiates candidates when multiple peers offer the same tree distance to the destination.
  2. tree_distance — Coordinate-based distance to destination through this peer
  3. node_addr — Deterministic tie-breaker

A peer with a bloom filter hit but no entry in the peer ancestry table (missing TreeAnnounce) defaults to maximum distance and is effectively invisible to routing.

Loop Prevention

The routing decision enforces strict progress: a packet is only forwarded to a peer that is strictly closer (by tree distance) to the destination than the current node. This self-distance check prevents routing loops even with stale coordinates, because each transit node evaluates using its own freshly-computed coordinates.

If no peer is closer than the current node (a local minimum in the tree distance metric), find_next_hop() returns None and the caller generates a PathBroken error.

Coordinate Caching

The coordinate cache maps NodeAddr → TreeCoordinate and is the critical data structure for multi-hop routing. Without it, forwarding decisions cannot be made.

Unified Cache

The coordinate cache is a single unified cache. All sources — SessionSetup transit, CP-flagged data packets, LookupResponse — write to the same cache.

Population Sources

Source When What
SessionSetup transit Session establishment Both src and dest coordinates
SessionAck transit Session establishment Both src and dest coordinates
CP-flagged data packet Warmup or recovery Both src and dest coordinates (cleartext)
LookupResponse Discovery Target's coordinates

Eviction

  • TTL-based: Entries expire after 300s (configurable)
  • Refresh on use: Active routing refreshes the TTL, keeping hot entries alive
  • LRU: When full, least recently used entries are evicted first
  • Flush on parent change: When the local node's tree parent changes, the entire cache is flushed. Parent changes mean the node's own coordinates have changed, making relative distance calculations with cached coordinates potentially invalid. Flushing is preferred over stale routing: the cost of re-discovery is lower than routing packets to dead ends.

Cache and Session Timer Ordering

Timer values are ordered so that idle sessions tear down before transit caches expire:

Timer Default Purpose
Session idle 90s Session teardown
Coordinate cache TTL 300s Coordinate expiration

When traffic stops, the session tears down at 90s. When traffic resumes, a fresh SessionSetup re-warms transit caches (still within their 300s TTL).

Discovery Protocol

Discovery resolves a destination's tree coordinates so that multi-hop routing can proceed. Requests are forwarded using bloom-guided tree routing — only to tree peers (parent + children) whose bloom filter contains the target — producing single-path forwarding through the spanning tree.

When Discovery Is Needed

  • First contact with a destination (no cached coordinates)
  • After receiving CoordsRequired (transit node lost coordinates)
  • After receiving PathBroken (coordinates may be stale)

LookupRequest

The source creates a LookupRequest containing:

  • request_id: Unique identifier for deduplication
  • target: The node_addr being sought
  • origin: The requester's node_addr
  • origin_coords: The requester's current tree coordinates (so the response can route back)
  • TTL: Bounds the forwarding radius

Bloom-Guided Tree Routing

Rather than flooding to all peers, the request is forwarded only to tree peers (parent + children) whose bloom filter contains the target. Because bloom filters propagate along tree edges with split-horizon exclusion, typically only one tree peer matches — producing a single directed path through the spanning tree toward the target's subtree. This reduces discovery traffic by roughly 90% compared to flooding.

If no tree peer's bloom filter matches the target, the request falls back to non-tree peers whose bloom filter contains the target. This recovers from dead ends caused by stale bloom filters, tree restructuring, or transit node failures. If no peer at all has a bloom match, the request is dropped at that node.

Loop prevention: The spanning tree is inherently loop-free, so tree-only forwarding cannot loop. The request_id dedup cache (default 10s window) provides defense-in-depth, catching edge cases during tree restructuring where a request might arrive via both tree and fallback paths.

Retry Logic

Single-path forwarding is more fragile than flooding — if any transit node on the path has a stale bloom filter or loses a link, the request fails. To compensate, the originator retries:

  • T=0: Initial lookup sent
  • T=5s: Retry if no response (configurable via retry_interval_secs)
  • T=10s: Timeout, fail (configurable via timeout_secs)

The default max_attempts is 2 (initial + one retry). Each retry generates a fresh request_id and re-evaluates bloom filter matches, so it can take a different path if the tree has restructured.

Originator Backoff

After a lookup times out or no peer's bloom filter contains the target, the originator enters exponential backoff before re-attempting discovery for the same target:

  • Base delay: 30s (configurable via backoff_base_secs)
  • Multiplier: 2x per consecutive failure
  • Cap: 300s (configurable via backoff_max_secs)

Backoff is reset on topology changes that might make previously unreachable targets reachable: parent switch, new peer connection, first RTT measurement from MMP, or peer reconnection.

Bloom Filter Pre-Check

Before initiating a lookup, the originator checks whether any peer's bloom filter contains the target. If no peer advertises reachability, the lookup is skipped entirely and recorded as a failure for backoff purposes. This avoids wasting network resources when the target is not in the mesh.

Transit-Side Rate Limiting

Transit nodes enforce a per-target minimum interval (default 2s, configurable via forward_min_interval_secs) for forwarded lookups. This is defense-in-depth against misbehaving nodes that generate fresh request_ids at high rate to bypass dedup. The rate limiter collapses rapid-fire lookups for the same target regardless of request_id.

LookupResponse

When the request reaches the target (or a node that has the target as a direct peer), a LookupResponse is created containing:

  • request_id: Echoed from the request
  • target: The target's node_addr
  • target_coords: The target's current tree coordinates
  • path_mtu: Minimum MTU along the response path (transit-annotated, initialized to u16::MAX by the target)
  • proof: Signature covering (request_id || target || target_coords) — authenticates that the response is genuine and the target holds the claimed tree position

The response routes back to the requester using reverse-path routing as the primary mechanism: each transit node looks up the request_id in its recent_requests table to find the peer that forwarded the original request, and sends the response back through that peer. This ensures the response follows the same path as the request. Greedy tree routing toward the origin_coords is used only as a fallback if the reverse-path entry has expired.

Response-forwarded flag: Each recent_requests entry tracks whether a response has already been forwarded for that request_id. If a second response arrives (e.g., from convergent request paths that reached the target via different routes), the transit node drops it. This prevents response routing loops where multiple responses for the same request circulate through the network.

Proof verification: The source verifies the Schnorr proof upon receipt, confirming that the target actually signed the response. The proof covers (request_id || target || target_coords) — coordinates are included because verification at the source confirms the target holds the claimed position. The path_mtu field is excluded from the proof because it is a transit annotation modified at each hop.

Discovery Outcome

On receiving a verified LookupResponse, the source caches the target's coordinates and clears any backoff state for that target. Subsequent routing to that destination can proceed via the normal find_next_hop() priority chain.

If discovery times out (no response after all retry attempts), queued packets receive ICMPv6 Destination Unreachable and the target enters backoff.

SessionSetup Self-Bootstrapping

SessionSetup is the mechanism that warms transit node coordinate caches along a path, enabling subsequent data packets to route efficiently.

How It Works

SessionSetup carries plaintext coordinates (outside the Noise handshake payload, visible to transit nodes):

  • src_coords: Source's current tree coordinates
  • dest_coords: Destination's tree coordinates (learned from discovery)

As the SessionSetup transits each intermediate node:

  1. The transit node extracts both coordinate sets
  2. Caches src_addr → src_coords and dest_addr → dest_coords in its coordinate cache
  3. Forwards the message using the cached destination coordinates

SessionAck returns along the reverse path, carrying both the responder's and initiator's coordinates and warming caches in the other direction. This ensures return-path transit nodes can route even when the reverse path diverges from the forward path (e.g., after tree reconvergence).

Result

After the handshake completes, the entire forward and reverse paths have cached coordinates for both endpoints. Subsequent data packets use minimal headers (no coordinates) and route efficiently through the warmed caches.

Hybrid Coordinate Warmup (CP + CoordsWarmup)

The CP flag in the FSP common prefix and the standalone CoordsWarmup message (0x14) together provide a hybrid cache-warming mechanism that complements SessionSetup. See fips-session-layer.md for the full warmup strategy.

Transit nodes parse the CP flag from the FSP header and extract source and destination coordinates from the cleartext section between the header and ciphertext — no decryption needed. This is the same caching operation performed for SessionSetup coordinates. CoordsWarmup messages use the same CP-flag format and are handled identically by transit nodes via the existing try_warm_coord_cache() path.

Error Recovery

When routing fails, transit nodes signal the source endpoint so it can take corrective action.

CoordsRequired

Trigger: A transit node receives a SessionDatagram but has no cached coordinates for the destination. It cannot make a forwarding decision.

Transit node action:

  1. Create a new SessionDatagram addressed back to the original source, carrying a CoordsRequired payload identifying the unreachable destination
  2. Route the error via find_next_hop(src_addr)
  3. If the source is also unreachable, drop silently (no cascading errors)

Source recovery:

  1. Immediately send a standalone CoordsWarmup (0x14) message to re-warm transit caches along the path (rate-limited: at most one per destination per configurable interval, default 2s)
  2. Reset CP warmup counter — subsequent data packets piggyback coordinates when possible, or trigger additional CoordsWarmup messages when piggybacking would exceed the transport MTU
  3. Initiate discovery (bloom-guided LookupRequest) for the destination
  4. When discovery completes, warmup counter resets again (covers timing gap)

The crypto session remains active throughout — only routing state is refreshed.

PathBroken

Trigger: A transit node has cached coordinates for the destination but no peer is closer to the destination than itself (a local minimum in the tree distance metric). The cached coordinates may be stale.

Transit node action: Same as CoordsRequired — generate error back to source.

Source recovery:

  1. Immediately send a standalone CoordsWarmup (0x14) message (rate-limited, same per-destination interval as CoordsRequired response)
  2. Remove stale coordinates from cache
  3. Initiate discovery for the destination
  4. Reset CP warmup counter

MtuExceeded

Trigger: A transit node receives a SessionDatagram but the total packet size exceeds the next-hop link MTU. The packet cannot be forwarded without fragmentation, which FIPS does not perform at the mesh layer.

Transit node action:

  1. Create a new SessionDatagram addressed back to the original source, carrying an MtuExceeded payload identifying the destination, the reporting router, and the bottleneck MTU
  2. Route the error via find_next_hop(src_addr)
  3. Drop the original oversized packet

Source recovery: FSP uses the reported bottleneck MTU to adjust its session-layer path MTU estimate (immediate decrease). The source can then reduce payload sizes to fit within the discovered path MTU. MtuExceeded is the reactive complement to the proactive path_mtu field in SessionDatagram and LookupResponse — the proactive field tracks the minimum MTU along the forward path, while MtuExceeded signals when an actual packet exceeds the limit.

Error Signal Rate Limiting

All three error types are rate-limited at transit nodes: maximum one error per destination per 100ms. This prevents storms during topology changes when many packets to the same destination hit the same routing failure simultaneously.

At the source side, CoordsWarmup responses to CoordsRequired/PathBroken are independently rate-limited: at most one standalone CoordsWarmup per destination per coords_response_interval_ms (default 2000ms, configurable). This prevents amplification where a burst of error signals would generate a corresponding burst of warmup messages.

Error signals (CoordsRequired, PathBroken, MtuExceeded) are handled asynchronously outside the packet receive path, allowing the RX loop to continue processing without blocking on discovery or session repair.

Error Routing Limitation

Error signals route back to the source using find_next_hop(src_addr). For steady-state data packets (after the CP warmup window), the transit node may lack cached coordinates for the source. If so, the error is silently dropped.

This blind spot is partially addressed by CP warmup: transit nodes receive source coordinates during the warmup phase. But after warmup expires and transit caches for the source expire, errors may be lost. The session idle timeout (90s) limits the window — if traffic stops long enough for transit caches to fully expire, the session tears down and re-establishment re-warms the path.

Cold Start → Warm Cache → Steady State

Cold Start

A new node or a node reaching a new destination goes through the following sequence:

  1. DNS resolution (IPv6 adapter only): Resolve npub.fips → populate identity cache with NodeAddr + PublicKey
  2. Session initiation attempt: Fails because no coordinates are cached for the destination
  3. Discovery: LookupRequest routes through the spanning tree via bloom-guided forwarding; LookupResponse returns the destination's coordinates
  4. Session establishment: SessionSetup carries coordinates, warming transit caches along the path
  5. Warmup: First N data packets include CP flag, reinforcing transit caches

The first packet to a new destination always triggers this sequence. The packet is queued (bounded) until the session is established.

Warm Cache

After session establishment and warmup:

  • Transit nodes have cached coordinates for both endpoints
  • Bloom filters have converged for the destination
  • Data packets use minimal headers (no coordinates)
  • Routing decisions are fast: bloom candidate selection + distance ranking

Steady State

In steady state, the mesh is mostly self-maintaining:

  • TreeAnnounce gossip keeps the spanning tree current
  • FilterAnnounce gossip keeps bloom filters current
  • Coordinate caches are refreshed by active routing traffic
  • Occasional cache misses trigger CP warmup or discovery, but these are rare when traffic is flowing

Cache Expiry and Recovery

When traffic to a destination stops:

  1. Session idles out (90s) — session torn down
  2. Coordinate caches expire (300s) — transit nodes forget coordinates
  3. Bloom filters remain — they have no TTL, so tree-propagated reachability information persists

When traffic resumes:

  1. Identity cache: usually still populated (LRU, no TTL)
  2. Session: new establishment required (full handshake)
  3. Coordinates: discovery may be needed if cache has expired
  4. SessionSetup re-warms transit caches on the new path

Leaf-Only Operation (under development)

Leaf-only operation is an optimization for resource-constrained nodes (sensors, battery-powered devices). The core infrastructure exists (config flag, node constructor, bloom filter support) but is not yet enabled in normal operation.

Concept

A leaf-only node connects to a single upstream peer that handles all routing on its behalf:

  • No bloom filter storage or processing: The upstream peer includes the leaf's identity in its own outbound bloom filters
  • No spanning tree participation: The leaf does not offer itself as a potential parent to other nodes
  • Simplified routing: All traffic tunnels through the upstream peer
  • Minimal resource usage: Suitable for ESP32-class devices (~500KB RAM)

Upstream Peer Responsibilities

The upstream peer:

  • Includes the leaf's identity in its outbound bloom filters
  • Forwards all traffic addressed to the leaf
  • Handles discovery responses on behalf of the leaf
  • Maintains the link session with the leaf

What the Leaf Retains

Even as a leaf-only node, it still:

  • Maintains its own Noise IK link session with the upstream peer (FMP layer)
  • Can establish end-to-end FSP sessions with arbitrary destinations
  • Has its own identity (npub, node_addr)

The optimization is purely at the routing/mesh layer — the leaf delegates routing decisions but retains its own end-to-end encryption and identity.

Packet Type Summary

Message Typical Size When Forwarded?
TreeAnnounce Variable (depth-dependent) Topology changes No (peer-to-peer)
FilterAnnounce ~1 KB Topology changes No (peer-to-peer)
LookupRequest ~300 bytes First contact, recovery Yes (bloom-guided tree)
LookupResponse ~400 bytes Response to discovery Yes (greedy routed)
SessionDatagram + SessionSetup ~232402 bytes Session establishment Yes (routed)
SessionDatagram + SessionAck ~170 bytes Session confirmation Yes (routed)
SessionDatagram + Data (minimal) 77 bytes + IPv6 payload Bulk IPv6 traffic (compressed) Yes (routed)
SessionDatagram + Data (with CP) 77 + coords + IPv6 payload Warmup/recovery (compressed) Yes (routed)
SessionDatagram + CoordsRequired 70 bytes Cache miss error Yes (routed)
SessionDatagram + PathBroken 70+ bytes Dead-end error Yes (routed)
Disconnect 2 bytes Link teardown No (peer-to-peer)

See fips-wire-formats.md for byte-level layouts.

Privacy Considerations

Source and destination node_addrs are visible to every transit node (required for forwarding decisions and error signal routing). FIPS prioritizes low-latency greedy routing with explicit error signaling over metadata privacy.

The node_addr is SHA-256(pubkey) truncated to 128 bits — a one-way hash. Transit nodes learn which node_addr pairs are communicating but cannot determine the actual Nostr identities (npubs) of the endpoints. An observer can verify "does this node_addr belong to pubkey X?" but cannot enumerate communicating identities from traffic alone.

Onion routing was considered and rejected because it requires the sender to know the full path upfront (incompatible with self-organizing routing) and prevents per-hop error feedback (incompatible with CoordsRequired/PathBroken recovery).

Implementation Status

Feature Status
Spanning tree formation Implemented
TreeAnnounce gossip Implemented
Bloom filter computation (split-horizon) Implemented
FilterAnnounce gossip Implemented
find_next_hop() priority chain Implemented
Coordinate cache (unified, TTL + refresh) Implemented
Flush coord cache on parent change Implemented
LookupRequest/LookupResponse discovery Implemented
SessionSetup self-bootstrapping Implemented
Hybrid coordinate warmup (CP + CoordsWarmup) Implemented
CoordsRequired recovery Implemented
PathBroken recovery Implemented
MtuExceeded recovery Implemented
LookupResponse proof verification Implemented
Discovery reverse-path routing Implemented
Error signal rate limiting Implemented
Flap dampening (hysteresis + hold-down) Implemented
Link liveness (dead timeout) Implemented
Discovery request deduplication Implemented
Discovery bloom-guided tree routing Implemented
Discovery retry logic Implemented
Discovery originator backoff Implemented
Discovery transit-side rate limiting Implemented
Discovery response-forwarded dedup Implemented
Leaf-only operation Under development
Link cost in parent selection (ETX) Implemented
Link cost in candidate ranking Implemented

References