mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Corrections to the user, reference and design documentation and to source comments where they disagree with the code. No behaviour or wire change; the tests added here pin the behaviour the corrected text now states. Correct the OpenWrt tutorial on restricting port forwards. The OpenWrt gateway tutorial said the FIPS mesh firewall is the layer that limits which peers reach a port forward, pointed at the mesh firewall how-to, and told readers to add a fips-firewall drop-in for the listen port. None of that applies: the OpenWrt package ships neither the mesh firewall baseline nor fips-firewall.service, and on Linux the baseline hooks the input chain only and never sees a forward to a LAN host. A reader following the text would leave the forward open to every mesh peer that can reach the AP. The tutorial now says that the package's fips zone accepts port forwards from every mesh peer by default, and shows the fw4 rules that restrict one: an accept for the allowed peer followed by a REJECT for the rest, matching the target's port, with the note that an accept alone restricts nothing because other peers fall through to the packaged accept. The summary bullet and the troubleshooting row point at fw4 instead of the mesh firewall, and Linux readers are sent to the forward-hook recipe in deploy-gateway.md. Describe conntrack-less pinning and BLE evictions correctly. The gateway troubleshooting guide said that with no readable conntrack source a mapping is reclaimed on its TTL and grace period alone. It left out what the pool does on top of that: no mapping counts as having carried traffic, so a re-query does not extend a name (only a used mapping is refreshed, and the answer TTL is capped at the time left since creation), and at the ceiling a new name can replace any mapping. The guide now quotes the daemon's whole warning, states both effects, and says when mappings count traffic again: those created after conntrack reads have succeeded for three ticks in a row. The doc comment on the BLE pool's eviction counter said it counts a non-static peer being displaced. There is no static-peer notion in the pool; it records the eviction of an unverified link past its verify grace by a newcomer, which is the only kind the pool makes. Bring the IPv6 adapter docs and comments in line with the code. Several statements about the IPv6 adapter disagreed with the code: - FIPS_IPV6_OVERHEAD was described as 77 bytes on top of the 106-byte FIPS_OVERHEAD; 77 is the total (106 + 4 + 7 - 40). - The effective-MTU table called a 1472-byte transport the standard deployment; the UDP default is 1280. - All ICMPv6 errors were said to be rate-limited by a per-source token bucket. Only Packet Too Big is limited, by a per-source minimum interval of 100 ms; Destination Unreachable is not limited. - The TUN writer was said to clamp MSS on inbound SYN-ACKs only, and clamped SYN-ACKs were listed as one of its packet sources. The clamp acts on any SYN-flagged segment, in both the reader and the writer. The design docs, the MTU how-to, the writer's doc comments and its trace message now say so. - The design doc said the reader needs no "is this for me?" check. It has one: a packet to the node's own address is written back to the TUN, which macOS utun needs. - The app-owned TUN docs (design doc and the enable_app_owned_tun rustdoc) said FIPS neither filters off-mesh destinations nor answers them with ICMPv6 there, and that the embedder clamps only outbound SYNs. On that path an off-mesh destination is looked up and answered with Destination Unreachable on app_inbound_rx (an oversize packet with Packet Too Big), and no MSS is clamped in either direction. - The transport_mtu fallback comment said "UDP first, then Ethernet"; it takes a configured UDP instance's MTU, else 1280. A new test pins the app-owned behaviour the rustdoc now states: an off-mesh unicast destination is answered with Destination Unreachable from the node's own address on app_inbound_rx, and a multicast destination gets no answer. State what the debug inject command and the outbound path do. The debug_commands documentation (the config field's doc comment, the configuration reference and the Inject parameters doc) said inject writes only into one of the calling client's own flows, and so grants the client nothing it does not already have. inject looks the flow up by id in the node-wide flow table, so it can write into any flow on the node, including one another client opened or a listener accepted; the comment on the command itself and the native API reference already said so. All three now say the same, and the claim that it grants nothing new is gone. The handle_native_outbound doc said an error is reported to the client rather than as an ICMPv6 message. Nothing is reported: an oversize datagram is counted in drop_oversize and dropped, a send or session-initiation failure is logged at debug, and a datagram past the pending-queue bounds is dropped. The doc now lists those. Stop claiming a withdrawn advert is deleted with NIP-09. The Nostr events reference, the discovery design doc and the generic port-advertisement doc said the node withdraws its advert with a NIP-09 kind 5 deletion when advertising is turned off or the responder retires. The node never sends one. The delete plan is produced only when advertising is off and an advert event id is recorded, but an id is recorded only after a successful publish, which needs advertising on, and the advertise flag is fixed when the advert machine is built. A transient missing advert and shutdown both deliberately skip the delete, because on relays that apply deletions strictly it can also remove the replacement the next start publishes. The docs now say that no deletion is sent, and that an advert the node stops publishing stays on relays until its NIP-40 expiration, advert_ttl_secs after publication, unless a newer advert replaces it. The generic description keeps deletion as something a responder may do, and says FIPS does not. The doc comments on the publish plan's delete variant, plan_publish and clear_event_id now say that the delete path is never reached by a running node, and why. Correct the traversal signaling description. The Nostr events reference and the design docs disagreed with the signaling code in four places: - inReplyTo was described as the offer's event id. The responder sets it to the offer's nonce, the initiator matches answers by nonce, and the validator refuses any other value as a session mismatch. An implementation written from the docs would have every answer dropped. - The answer's PunchHint was said to tell both sides when to probe. Only the responder probes on it; the initiator does not read the hint and starts probing from its own punch_* settings when the answer arrives. Fixed in the reference, the port-advertisement doc and the discovery design doc. - All three event kinds were said to be signed with the node's identity key. The kind 21059 wrap is signed by a one-time key; only the seal inside it is signed with the identity key. - The FIPS advert example did not say that FIPS emits only the d, protocol, version and expiration tags and carries its signaling relays and STUN servers in the content as signalRelays and stunServers, present only with a udp:nat endpoint. Two tests pin the behaviour the docs now state: an answer whose inReplyTo is the offer's event id is refused as a session mismatch, and a signal's wrap is signed by a key other than the sender's while unwrapping recovers the sender's identity from the seal. Label the second msg2 DH as a repeat of es, not se. The IK msg2 code, the noise module doc, the wire-formats reference, the mesh-layer design doc and the msg2 diagram all called the second DH in msg2 `se`. Both sides actually pair the responder's static key with the initiator's ephemeral key, which is msg1's `es` pairing, so the step mixes the msg1 `es` value a second time and nothing computes Noise IK's `se`. Relabel it as `es` again everywhere, say in the code that it is not Noise's `se`, that it is the FMP v1 wire format and cannot change without breaking v1 peers, and that it leaves FMP v1 open to key compromise impersonation (FRC-1201 Section 9.3). Rename the two local bindings to `es2`. No behaviour or wire change. Correct the path MTU notification doc and the fips-mtu design doc. The PathMtuNotification doc said it is sent by a node that learns a path MTU from transit router feedback or ICMP Packet Too Big. It is sent by the session destination, echoing the forward-path minimum it saw in the envelope's path_mtu; a received MtuExceeded only updates local state. fips-mtu.md said TCP reports a per-connection MTU from TCP_MAXSEG. TCP reports the configured mtu and checks sends against it. It also said the TCP clamp mirror is tighter-only and never loosened. Entries are released on PathBroken (at most once a second per destination), peer removal and session timeout, discovery-learned entries expire with the coordinate cache TTL, and a link seed from a different transport replaces the stored value. Describe that lifecycle, and add a short section on the defences against forged signals: the 256-byte floors, the admission gate, corroboration against what this node has sent, and the rate-limited release. Correct transport comments and docs on TCP MTU, mDNS, BLE, Ethernet, Tor and UDP. Several transport comments and docs described behaviour the code does not have: - TCP: the config doc, configuration reference and transport-layer doc said the per-connection MTU comes from TCP_MAXSEG. TCP reports the configured mtu to FMP and checks sends against it. - mDNS: the module and supervisor comments said identity is proven by a Noise XX handshake. FMP here is Noise IK. Drop the note in the Nostr discovery doc that pointed at those comments. - BLE: the pubkey-exchange prefix doc said it is distinguished from FMP packets of version 1 or more. FMP is version 0, so the framer cannot tell them apart, as the caution below it already says. - Ethernet: the config doc said the runtime requires Linux. It is built for Linux (AF_PACKET) and macOS (BPF). - Tor: the config and transport docs, and the onion-service how-to, said control_port mode runs an inbound onion service. It only adds control-port monitoring to SOCKS5 outbound; directory mode is the only inbound mode. - UDP: the security reference said a UDP instance has no inbound exposure until bind_addr is set. A declared UDP instance listens on 0.0.0.0:2121 by default unless outbound_only is set. Correct the spanning tree docs on signatures, sequence, sizes, cache invalidation and flap dampening. - The TreeAnnounce signature was described as covering the entire message. It covers SHA-256 of the 48-byte declaration (node_addr, parent, sequence, timestamp); version and ancestry are not signed. - The sequence was said to increment on parent change. It also increments on becoming root and on an ancestry change under the same parent. - The TreeAnnounce size table added msg_type twice in the link-overhead column. The measured sizes are 168/264/328/488 bytes; note the 36-byte link overhead under the table. - Several design docs and a rate-limit comment said a parent change flushes the coordinate cache. The node drops only cached coordinates that route through it (parent switch or ancestry change) or sit under another root (becoming root), and both on parent loss or loop detection. - Flap dampening was described as engaging after more than flap_threshold switches in a sliding window. It engages when the count reaches flap_threshold, within a fixed window opened by the first switch. Correct the bloom filter frame size, triggers and propagation bound. - The wire-formats reference gave the v1 FilterAnnounce with link overhead as 1,072 bytes. It is 1,071 (1,035 payload plus 16 header, 4 timestamp and 16 tag), as the bloom design doc already says. - The bloom design doc listed local state changes (new identity, leaf-only dependent changes) as a send trigger. The identity does not change at runtime and nothing outside the tests adds a leaf-only dependent, so drop it. - The bloom and spanning tree docs bounded propagation at about 500 ms per hop. Announces leave only on the node's maintenance tick, so the bound is about one node.tick_interval_secs per hop (1 s by default); the 500 ms debounce is a floor below that. Correct forwarding docs on ranking, loop freedom and the coordinate cache. - The mesh-operation doc called link_cost an uncommon tie-breaker resolved by tree distance. Candidate ranking compares link cost first; tree distance and node_addr break ties. - Several docs and comments said the strict-progress check prevents routing loops even with stale coordinates. Each hop judges progress on its own cached coordinates for the destination, so the check does not guarantee loop freedom when hops hold different coordinates. Say that, and that the TTL bounds any loop. - The session-layer doc said FMP can fall back to bloom-only routing without cached coordinates. find_next_hop returns no next hop on a cache miss and the caller signals CoordsRequired. - The insert_coord_hint doc said a foreign-root destination can never be selected by the bloom step. A peer advertising coordinates under that root can be; the refusal stands for the reflector reason given in the same paragraph. - The find_next_hop and try_warm_coord_cache_ref doc blocks sat above the wrong functions, so rustdoc gave those two no docs and gave insert_coord_hint and warm_coord the wrong opening line. Move each block onto its function and fix the find_next_hop ranking wording. Correct FSP restart handling, error datagram sizes and XK labels. - The session-layer doc said an epoch mismatch on a later handshake marks a peer restart and triggers re-establishment. FSP exchanges the epoch but never reads it. A SessionSetup naming an established peer only arms a handshake beside the running session, and new keys are adopted at msg3 only when the authenticated static key matches. Describe that, and rename the status row accordingly. - Wire-format sizes: SessionDatagram + CoordsRequired is 74 bytes, not 70; SessionDatagram + PathBroken is 76 + 16n, not "70+"; PathBroken alone is 40 + 16n, not 35 + 16n; CoordsRequired with link overhead is 110, not 111; and the CoordsWarmup example is 238, not 239. State in the summary table which rows include link overhead. - Three FSP wire comments called the session handshake Noise IK. It is Noise XK, as the module's own phase table says. Correct the rationale for dropping our own looped-back request. The comment on the own-request check in classify_request, its test and the handle_lookup_response doc said that without the check the answer to our own lookup would be reverse-path forwarded to the peer that looped the request, and that recent_requests decides originator status. That stopped being true when classify_response began deciding originator status from the pending lookup before it reads the transit record. Say what the check still does: it stops the copy being forwarded a second time and keeps our own id out of the transit record, so a response after the lookup completes has nothing to be relayed by. Extend the test to pin that: once the lookup is accepted, a later response for the same id is unsolicited.
655 lines
30 KiB
Markdown
655 lines
30 KiB
Markdown
# FIPS Mesh Operation
|
|
|
|
This document describes how the FIPS mesh operates at the link layer — how
|
|
spanning tree, bloom filters, routing decisions, discovery, and error recovery
|
|
work together as a coherent system. It treats spanning tree and bloom filters
|
|
as black boxes (what they provide to routing) and focuses on how the pieces
|
|
interact.
|
|
|
|
For spanning tree algorithms and data structures, see
|
|
[fips-spanning-tree.md](fips-spanning-tree.md). For bloom filter parameters
|
|
and mathematics, see [fips-bloom-filters.md](fips-bloom-filters.md).
|
|
|
|
## Overview
|
|
|
|
FIPS mesh operation is entirely distributed. Each node makes forwarding
|
|
decisions using only local information: its direct peers, their spanning tree
|
|
positions, and their bloom filters. There are no routing tables pushed from
|
|
above, no link-state floods, and no distance-vector exchanges.
|
|
|
|
Two complementary mechanisms provide the information each node needs:
|
|
|
|
- **Spanning tree** gives every node a coordinate in the network — its
|
|
ancestry path from itself to the root. These coordinates enable distance
|
|
calculations between any two nodes without global topology knowledge.
|
|
- **Bloom filters** summarize which destinations are reachable through each
|
|
peer. Because they propagate along tree edges, they encode directional
|
|
reachability — which subtree contains a given destination.
|
|
|
|
Together, they enable a routing decision process that is local, efficient,
|
|
and self-healing.
|
|
|
|
## Spanning Tree Formation and Maintenance
|
|
|
|
For routing purposes, the spanning tree provides each node with a
|
|
coordinate (its ancestry path from itself to the root) plus a way to
|
|
compute distance between any two nodes (hops to their lowest common
|
|
ancestor). Each hop forwards only to a peer strictly closer to the
|
|
destination by its own cached view of the destination's coordinates.
|
|
That does not guarantee loop freedom when hops hold different
|
|
coordinates; the TTL bounds any loop.
|
|
|
|
The tree forms through distributed parent selection — root is the
|
|
smallest node_addr (no election), and each node picks the peer with
|
|
the lowest `effective_depth = depth + link_cost`. Cost-aware parent
|
|
selection lets the tree trade hop count for link quality once MMP has
|
|
accumulated SRTT and ETX metrics. Hysteresis (20% improvement
|
|
required to switch) and hold-down (suppress non-mandatory
|
|
re-evaluation after a switch) keep the tree stable under metric
|
|
noise. Partitions self-resolve — each segment converges to its own
|
|
root and reconverges to the smallest reachable root when segments
|
|
rejoin.
|
|
|
|
Liveness is detected via FMP heartbeats; dead-peer removal triggers
|
|
tree reconvergence and bloom filter recomputation for the affected
|
|
subtree. The heartbeat and dead-timeout mechanism lives at the link
|
|
layer; see [fips-mesh-layer.md](fips-mesh-layer.md#liveness-detection).
|
|
|
|
For the parent-selection algorithm, hold-down/hysteresis details, and
|
|
the convergence walkthroughs, see
|
|
[fips-spanning-tree.md](fips-spanning-tree.md) and
|
|
[spanning-tree-dynamics.md](spanning-tree-dynamics.md).
|
|
|
|
## Bloom Filter Gossip and Propagation
|
|
|
|
For routing purposes, each node maintains a bloom filter per peer
|
|
that answers "can peer P possibly reach destination D?" — either "no"
|
|
(definitive) or "maybe" (probabilistic). Because filters propagate
|
|
along tree edges with split-horizon exclusion, a bloom hit on a tree
|
|
peer reliably indicates which subtree contains the destination, and
|
|
tree-coordinate distance ranks competing matches.
|
|
|
|
FilterAnnounce updates are event-driven (peer changes, tree
|
|
restructuring) and rate-limited to prevent storms. False positives at
|
|
large scale never cause loops — the self-distance check at each hop
|
|
guarantees forward progress, and
|
|
mismatched bloom matches fall through to greedy tree routing.
|
|
|
|
For the filter computation, split-horizon merge rules, FPR analysis,
|
|
size classes, and folding, see
|
|
[fips-bloom-filters.md](fips-bloom-filters.md).
|
|
|
|
## Routing Decision Process
|
|
|
|
At each hop, FMP makes a local forwarding decision using the `find_next_hop()`
|
|
priority chain. This is the core routing algorithm.
|
|
|
|
### Priority Chain
|
|
|
|
1. **Local delivery** — The destination node_addr matches the local node.
|
|
Deliver to FSP above.
|
|
|
|
2. **Direct peer** — The destination is an authenticated neighbor. Forward
|
|
directly. No coordinates or bloom filters needed.
|
|
|
|
3. **Coordinate cache check** — Multi-hop forwarding requires the
|
|
destination's tree coordinates to be in the local cache. On miss,
|
|
`find_next_hop()` returns None immediately — bloom filters are never
|
|
consulted — and the source receives a CoordsRequired error signal.
|
|
|
|
4. **Bloom-guided routing** — One or more peers' bloom filters contain the
|
|
destination. Select the best peer by composite key:
|
|
`(link_cost, tree_distance, node_addr)`.
|
|
|
|
5. **Greedy tree routing** — Fall-through when bloom yields no candidate.
|
|
Forward to the peer that minimizes tree distance. If the tree has no
|
|
next hop closer to the destination, the source receives a PathBroken
|
|
error signal.
|
|
|
|
### Convergence Requirements
|
|
|
|
Multi-hop routing depends on two propagation processes that must run
|
|
to convergence simultaneously:
|
|
|
|
1. **Bloom convergence**: Filters must propagate so peers advertise
|
|
reachability
|
|
2. **Coordinate availability**: Destination coordinates must be cached at
|
|
every transit node on the path
|
|
|
|
Bloom convergence without coordinates trips step 3 (coord-cache miss →
|
|
CoordsRequired). Coordinates without bloom convergence falls through to
|
|
greedy tree routing — functional but suboptimal.
|
|
|
|
### Candidate Ranking
|
|
|
|
When bloom filters identify multiple candidate peers, they are ranked by a
|
|
composite key:
|
|
|
|
1. **link_cost** — Per-link quality metric (RTT-weighted ETX, Expected
|
|
Transmission Count), computed from MMP metrics. It is compared first:
|
|
among the peers strictly closer to the destination than this node, the
|
|
lowest link cost wins.
|
|
2. **tree_distance** — Coordinate-based distance to destination through this
|
|
peer; breaks ties in link cost
|
|
3. **node_addr** — Deterministic tie-breaker
|
|
|
|
A peer with a bloom filter hit but no entry in the peer ancestry table
|
|
(missing TreeAnnounce) defaults to maximum distance and is effectively
|
|
invisible to routing.
|
|
|
|
### Routing Decision Flowchart
|
|
|
|

|
|
|
|
### Loop Prevention
|
|
|
|
The routing decision enforces strict progress: a packet is only forwarded
|
|
to a peer that is strictly closer (by tree distance) to the destination than
|
|
the current node. Each hop judges this on its own cached coordinates for the
|
|
destination, so the self-distance check does not guarantee loop freedom when
|
|
hops hold different coordinates for it; the TTL bounds any loop.
|
|
|
|
If no peer is closer than the current node (a local minimum in the tree
|
|
distance metric), `find_next_hop()` returns None and the caller generates a
|
|
PathBroken error.
|
|
|
|
## Coordinate Caching
|
|
|
|
The coordinate cache maps `NodeAddr → TreeCoordinate` and is the
|
|
critical data structure for multi-hop routing. The session layer owns
|
|
this cache (its eviction policy, TTL/refresh semantics, selective
|
|
invalidation on parent change, and timer ordering with session idle timeout); see
|
|
[fips-session-layer.md](fips-session-layer.md#coordinate-cache) for
|
|
the canonical treatment.
|
|
|
|
## Discovery Protocol
|
|
|
|
Discovery resolves a destination's tree coordinates so that multi-hop routing
|
|
can proceed. Requests are forwarded using **bloom-guided tree routing** —
|
|
only to tree peers (parent + children) whose bloom filter contains the
|
|
target — producing single-path forwarding through the spanning tree.
|
|
|
|
### When Discovery Is Needed
|
|
|
|
- First contact with a destination (no cached coordinates)
|
|
- After receiving CoordsRequired (transit node lost coordinates)
|
|
- After receiving PathBroken (coordinates may be stale)
|
|
- After a send on an established session finds no next hop (this node lost
|
|
the coordinates itself, e.g. a link under the path changed while the
|
|
session survived)
|
|
|
|
### LookupRequest
|
|
|
|
The source creates a LookupRequest containing:
|
|
|
|
- **request_id**: Unique identifier for deduplication
|
|
- **target**: The node_addr being sought
|
|
- **origin**: The requester's node_addr
|
|
- **origin_coords**: The requester's current tree coordinates (so the
|
|
response can route back)
|
|
- **TTL**: Bounds the forwarding radius
|
|
|
|
### Bloom-Guided Tree Routing
|
|
|
|
Rather than flooding to all peers, the request is forwarded only to **tree
|
|
peers** (parent + children) whose bloom filter contains the target. Because
|
|
bloom filters propagate along tree edges with split-horizon exclusion,
|
|
typically only one tree peer matches — producing a single directed path
|
|
through the spanning tree toward the target's subtree. This reduces
|
|
discovery traffic by roughly 90% compared to flooding.
|
|
|
|
If no tree peer's bloom filter matches the target, the request falls back
|
|
to **non-tree peers** whose bloom filter contains the target. This recovers
|
|
from dead ends caused by stale bloom filters, tree restructuring, or transit
|
|
node failures. If no peer at all has a bloom match, the request is dropped
|
|
at that node.
|
|
|
|
**Loop prevention**: The spanning tree is inherently loop-free, so tree-only
|
|
forwarding cannot loop. The `request_id` dedup cache (default 10s window)
|
|
provides defense-in-depth, catching edge cases during tree restructuring
|
|
where a request might arrive via both tree and fallback paths.
|
|
|
|
### Retry Logic
|
|
|
|
Single-path forwarding is more fragile than flooding — if any transit node
|
|
on the path has a stale bloom filter or loses a link, the request fails.
|
|
To compensate, each discovery is a sequence of attempts with growing
|
|
per-attempt timeouts. The default sequence is `[1s, 2s, 4s, 8s]`
|
|
(configurable via `node.lookup.attempt_timeouts_secs`); the destination
|
|
is declared unreachable only after the full sequence is exhausted (15s
|
|
total at default).
|
|
|
|
When the current attempt's deadline elapses without a `LookupResponse`,
|
|
the originator sends another `LookupRequest` with a **fresh `request_id`**
|
|
and the next entry in the sequence as its deadline. Fresh `request_id`s
|
|
let each attempt take a different forwarding path as the bloom and tree
|
|
state evolve, which is particularly useful during cold-start convergence.
|
|
|
|
### Originator Backoff (optional, off by default)
|
|
|
|
After the per-attempt sequence is exhausted, the originator can additionally
|
|
suppress further fresh lookups for the same target with exponential
|
|
post-failure backoff. This is **disabled by default** (`backoff_base_secs:
|
|
0`); the per-attempt sequence is the only retry pacing in the standard
|
|
configuration. Operators may opt in via `node.lookup.backoff_base_secs`
|
|
and `node.lookup.backoff_max_secs` if their deployment has chatty apps
|
|
generating repeated lookups for genuinely unreachable destinations. When
|
|
enabled, backoff is **reset on topology changes** that might make
|
|
previously unreachable targets reachable: parent switch, new peer
|
|
connection, first RTT measurement from MMP, or peer reconnection.
|
|
|
|
While backoff is disabled (a zero base or a zero cap), no failure is
|
|
recorded at all. When it is enabled, the failure table is bounded: an entry
|
|
is forgotten once its window and one further cap interval have passed, so a
|
|
target that fails again soon after its window still escalates, and the
|
|
table holds at most 4096 targets. A new failure at that bound evicts the
|
|
entry whose window ends first, which can only let that target be retried
|
|
sooner.
|
|
|
|
### Bloom Filter Pre-Check
|
|
|
|
Before initiating a lookup, the originator checks whether *any* peer's
|
|
bloom filter contains the target. If no peer advertises reachability, the
|
|
lookup is skipped entirely and, when backoff is enabled, recorded as a
|
|
failure for backoff purposes. This avoids wasting network resources when
|
|
the target is not in the mesh. TUN traffic for such a target is answered with
|
|
ICMPv6 Destination Unreachable (no route), except while the node has no
|
|
peers and for 30 seconds after it gains one, when it is held while routing
|
|
information arrives.
|
|
|
|
### Transit-Side Rate Limiting
|
|
|
|
Transit nodes enforce a minimum interval per link peer and target (default
|
|
2s, configurable via `forward_min_interval_secs`) for forwarded lookups. This
|
|
is defense-in-depth against misbehaving nodes that generate fresh
|
|
`request_id`s at high rate to bypass dedup. The rate limiter collapses
|
|
rapid-fire lookups for the same target from the same link peer regardless of
|
|
`request_id`. It is keyed on the link peer rather than the request's origin,
|
|
which is unsigned and chosen by the sender, so one neighbour repeating a
|
|
target does not suppress other neighbours' requests for it. Only a request
|
|
that some peer can carry is recorded. The limiter holds at most 4096 entries;
|
|
at that bound the peer holding the most entries loses its oldest, and the new
|
|
entry is always recorded.
|
|
|
|
### LookupResponse
|
|
|
|
When the request reaches the target (or a node that has the target as a
|
|
direct peer), a LookupResponse is created containing:
|
|
|
|
- **request_id**: Echoed from the request
|
|
- **target**: The target's node_addr
|
|
- **target_coords**: The target's current tree coordinates
|
|
- **path_mtu**: Minimum MTU along the response path (transit-annotated,
|
|
initialized to `u16::MAX` by the target)
|
|
- **proof**: Signature covering `(request_id || target || target_coords)` —
|
|
authenticates that the response is genuine and the target holds the
|
|
claimed tree position
|
|
|
|
The response routes back to the requester using **reverse-path routing** as
|
|
the primary mechanism: each transit node looks up the `request_id` in its
|
|
`recent_requests` table to find the peer that forwarded the original request,
|
|
and sends the response back through that peer. This ensures the response
|
|
follows the same path as the request. Greedy tree routing toward the
|
|
`origin_coords` is used only as a fallback if the reverse-path entry has
|
|
expired.
|
|
|
|
**Originator check**: on the response path, a node tests its own outstanding
|
|
lookups before consulting `recent_requests`. A request is flooded to every
|
|
bloom-matching tree peer, and a bloom false positive can send a copy out into
|
|
the wider network and back to the originator, which would otherwise file its
|
|
own `request_id` as an ordinary transit entry and relay its own answer away.
|
|
The originator arm therefore wins: a response naming a target with a lookup
|
|
outstanding, carrying a `request_id` that lookup issued, is accepted here
|
|
whatever the dedup cache holds. A returning copy of the request is dropped
|
|
rather than recorded, so while the check recognises the id, it never enters the
|
|
transit cache. The drop is counted as `req_own_loopback` rather than as a
|
|
duplicate, because the cause differs: the node's own fan-out returning, not a
|
|
request id it has already recorded arriving again. Neither counter identifies
|
|
the peer that delivered the copy. The request path runs the two tests the other
|
|
way round, the dedup test first, and the order is immaterial there: an id the
|
|
originator check recognises is never in `recent_requests`, because the
|
|
originator records the ids it issues only in its pending lookups and a request
|
|
is recorded only after it has passed the check. Separately, the check
|
|
recognises an id only while its lookup is outstanding and the id is among the
|
|
last `MAX_RECORDED_IDS` (eight) that the lookup's retry ladder issued for that
|
|
target. A copy that returns after the lookup completed or timed out, or a copy
|
|
of an older attempt on a longer ladder, is recorded and forwarded as transit,
|
|
and a later copy of it is dropped as a duplicate.
|
|
|
|
**Relay eligibility**: Each `recent_requests` entry records the request's
|
|
target and the link peers the transit forwarded it to, fixed when it was
|
|
forwarded. A response is relayed only if it names that target and arrives from
|
|
one of those peers. A transit cannot verify the proof, so a response from any
|
|
other peer, including the peer the request came from or one whose filter
|
|
claimed the target only after the forward, cannot be told from a forgery and
|
|
is dropped.
|
|
|
|
**Relayed-response memory**: Each entry also remembers the first eight bytes
|
|
of the proof of the last eight responses relayed for it. A response whose
|
|
digest is remembered is dropped, which stops response routing loops: a copy
|
|
circulating on a reverse-path cycle is relayed once at each node. A response
|
|
with a new digest is relayed however many have gone before, so a forgery
|
|
that arrives first does not stop the genuine response. The cost is that an
|
|
eligible peer can have each distinct forgery it sends relayed to the origin,
|
|
which verifies and discards each one.
|
|
|
|
**Proof verification**: The source verifies the Schnorr proof upon receipt,
|
|
confirming that the target actually signed the response. The proof covers
|
|
`(request_id || target || target_coords)` — coordinates are included because
|
|
verification at the source confirms the target holds the claimed position.
|
|
The `path_mtu` field is excluded from the proof because it is a transit
|
|
annotation modified at each hop.
|
|
|
|
### Coordinate Discovery Sequence
|
|
|
|

|
|
|
|
### Discovery Outcome
|
|
|
|
On receiving a verified LookupResponse, the source caches the target's
|
|
coordinates and clears any backoff state for that target. Subsequent routing
|
|
to that destination can proceed via the normal `find_next_hop()` priority
|
|
chain.
|
|
|
|
If discovery times out (no response after all retry attempts), queued
|
|
packets receive ICMPv6 Destination Unreachable and, when backoff is
|
|
enabled, the target enters backoff.
|
|
|
|
## Coordinate Cache Warming
|
|
|
|
SessionSetup carries plaintext source and destination coordinates,
|
|
which transit nodes cache as the message travels — warming the
|
|
forward path. SessionAck carries them back along the reverse path,
|
|
warming return-path caches. Steady-state data packets piggyback
|
|
coordinates via the FSP CP flag during the warmup window, falling
|
|
back to standalone CoordsWarmup messages when piggybacking would
|
|
exceed the transport MTU. See
|
|
[fips-session-layer.md](fips-session-layer.md#hybrid-coordinate-warmup-strategy)
|
|
for the canonical hybrid-warmup design (SessionSetup
|
|
self-bootstrapping plus CP-flag piggyback plus standalone
|
|
CoordsWarmup).
|
|
|
|
## Error Recovery
|
|
|
|
When routing fails, transit nodes signal the source endpoint so it can take
|
|
corrective action.
|
|
|
|
### CoordsRequired
|
|
|
|
**Trigger**: A transit node receives a SessionDatagram but has no cached
|
|
coordinates for the destination. It cannot make a forwarding decision.
|
|
|
|
**Transit node action**:
|
|
|
|
1. Create a new SessionDatagram addressed back to the original source,
|
|
carrying a CoordsRequired payload identifying the unreachable destination
|
|
2. Route the error via `find_next_hop(src_addr)`
|
|
3. If the source is also unreachable, drop silently (no cascading errors)
|
|
|
|
**Source recovery**:
|
|
|
|
1. Immediately send a standalone CoordsWarmup (0x14) message to re-warm
|
|
transit caches along the path (rate-limited: at most one per destination
|
|
per configurable interval, default 2s)
|
|
2. Reset CP warmup counter — subsequent data packets piggyback coordinates
|
|
when possible, or trigger additional CoordsWarmup messages when
|
|
piggybacking would exceed the transport MTU
|
|
3. Initiate discovery (bloom-guided LookupRequest) for the destination
|
|
4. When discovery completes, warmup counter resets again (covers timing gap)
|
|
|
|
The crypto session remains active throughout — only routing state is
|
|
refreshed.
|
|
|
|
### PathBroken
|
|
|
|
**Trigger**: A transit node has cached coordinates for the destination but
|
|
no peer is closer to the destination than itself (a local minimum in the
|
|
tree distance metric). The cached coordinates may be stale.
|
|
|
|
**Transit node action**: Same as CoordsRequired — generate error back to
|
|
source.
|
|
|
|
**Source recovery**:
|
|
|
|
1. Immediately send a standalone CoordsWarmup (0x14) message (rate-limited,
|
|
same per-destination interval as CoordsRequired response)
|
|
2. Handle the cached coordinates by where they came from. Unverified
|
|
coordinates (a hint copied off a passing packet, or a lookup result
|
|
whose verification has aged out) are removed. Coordinates a lookup
|
|
verified are kept while discovery re-validates them, because the signal
|
|
is unauthenticated and removing them would let the next forged warm
|
|
replace them. They are demoted to an unverified hint, keeping their value,
|
|
only when PathBroken signals naming the destination arrive over two
|
|
different links within 15 seconds. The vote is the authenticated link
|
|
peer, not the reporter the signal names, which the sender chooses. A node
|
|
whose signals all arrive over one link never demotes this way; its
|
|
verified coordinates last until discovery replaces them or their
|
|
verification ages out after 300 seconds.
|
|
3. Initiate discovery for the destination. If its identity is not cached,
|
|
cache it first from the session's key, so the response can be verified
|
|
4. Reset CP warmup counter
|
|
|
|
The source also counts, without refusing anything, a PathBroken that
|
|
arrives over a link other than its forward link to the destination, and one
|
|
whose reporter is not closer to the destination than the source is. Both
|
|
have a non-zero healthy floor.
|
|
|
|
### MtuExceeded
|
|
|
|
**Trigger**: A transit node receives a SessionDatagram but the total
|
|
packet size exceeds the next-hop link MTU. The packet cannot be forwarded
|
|
without fragmentation, which FIPS does not perform at the mesh layer.
|
|
|
|
**Transit node action**:
|
|
|
|
1. Create a new SessionDatagram addressed back to the original source,
|
|
carrying an MtuExceeded payload identifying the destination, the
|
|
reporting router, and the bottleneck MTU
|
|
2. Route the error via `find_next_hop(src_addr)`
|
|
3. Drop the original oversized packet
|
|
|
|
**Source recovery**: FSP uses the reported bottleneck MTU to adjust its
|
|
session-layer path MTU estimate (immediate decrease). The source can then
|
|
reduce payload sizes to fit within the discovered path MTU. MtuExceeded is
|
|
the reactive complement to the proactive `path_mtu` field in
|
|
SessionDatagram and LookupResponse — the proactive field tracks the
|
|
minimum MTU along the forward path, while MtuExceeded signals when an
|
|
actual packet exceeds the limit.
|
|
|
|
### Error Signal Rate Limiting
|
|
|
|
All three error types are rate-limited at transit nodes: maximum one error per
|
|
destination per 100ms. This prevents storms during topology changes when many
|
|
packets to the same destination hit the same routing failure simultaneously.
|
|
|
|
At the source side, CoordsWarmup responses to CoordsRequired/PathBroken are
|
|
independently rate-limited: at most one standalone CoordsWarmup per destination
|
|
per `coords_response_interval_ms` (default 2000ms, configurable). This
|
|
prevents amplification where a burst of error signals would generate a
|
|
corresponding burst of warmup messages.
|
|
|
|
Error signals (CoordsRequired, PathBroken, MtuExceeded) are handled
|
|
asynchronously outside the packet receive path, allowing the RX loop to
|
|
continue processing without blocking on discovery or session repair.
|
|
|
|
### Error Routing Limitation
|
|
|
|
Error signals route back to the source using `find_next_hop(src_addr)`. For
|
|
steady-state data packets (after the CP warmup window), the
|
|
transit node may lack cached coordinates for the source. If so, the error is
|
|
silently dropped.
|
|
|
|
This blind spot is partially addressed by CP warmup: transit
|
|
nodes receive source coordinates during the warmup phase. But after warmup
|
|
expires and transit caches for the source expire, errors may be lost. The
|
|
session idle timeout (90s) limits the window — if traffic stops long enough
|
|
for transit caches to fully expire, the session tears down and re-establishment
|
|
re-warms the path.
|
|
|
|
## Cold Start → Warm Cache → Steady State
|
|
|
|
### Cold Start
|
|
|
|
A new node or a node reaching a new destination goes through the following
|
|
sequence:
|
|
|
|
1. **DNS resolution** (IPv6 adapter only): Resolve `npub.fips` → populate
|
|
identity cache with NodeAddr + PublicKey
|
|
2. **Session initiation attempt**: Fails because no coordinates are cached
|
|
for the destination
|
|
3. **Discovery**: LookupRequest routes through the spanning tree via
|
|
bloom-guided forwarding; LookupResponse returns the destination's
|
|
coordinates
|
|
4. **Session establishment**: SessionSetup carries coordinates, warming
|
|
transit caches along the path
|
|
5. **Warmup**: First N data packets include CP flag, reinforcing transit
|
|
caches
|
|
|
|
The first packet to a new destination always triggers this sequence. The
|
|
packet is queued (bounded) until the session is established.
|
|
|
|
### Warm Cache
|
|
|
|
After session establishment and warmup:
|
|
|
|
- Transit nodes have cached coordinates for both endpoints
|
|
- Bloom filters have converged for the destination
|
|
- Data packets use minimal headers (no coordinates)
|
|
- Routing decisions are fast: bloom candidate selection + distance ranking
|
|
|
|
### Steady State
|
|
|
|
In steady state, the mesh is mostly self-maintaining:
|
|
|
|
- TreeAnnounce gossip keeps the spanning tree current
|
|
- FilterAnnounce gossip keeps bloom filters current
|
|
- Coordinate caches are refreshed by active routing traffic
|
|
- Occasional cache misses trigger CP warmup or discovery, but these
|
|
are rare when traffic is flowing
|
|
|
|
### Cache Expiry and Recovery
|
|
|
|
When traffic to a destination stops:
|
|
|
|
1. **Session idles out** (90s) — session torn down
|
|
2. **Coordinate caches expire** (300s) — transit nodes forget coordinates
|
|
3. **Bloom filters remain** — they have no TTL, so tree-propagated
|
|
reachability information persists
|
|
|
|
When traffic resumes:
|
|
|
|
1. Identity cache: usually still populated (LRU, no TTL)
|
|
2. Session: new establishment required (full handshake)
|
|
3. Coordinates: discovery may be needed if cache has expired
|
|
4. SessionSetup re-warms transit caches on the new path
|
|
|
|
## Leaf-Only Operation *(under development)*
|
|
|
|
Leaf-only operation is an optimization for resource-constrained nodes
|
|
(sensors, battery-powered devices). The core infrastructure exists (config
|
|
flag, node constructor, bloom filter support) but is not yet enabled in
|
|
normal operation.
|
|
|
|
### Concept
|
|
|
|
A leaf-only node connects to a single upstream peer that handles all routing
|
|
on its behalf:
|
|
|
|
- **No bloom filter storage or processing**: The upstream peer includes the
|
|
leaf's identity in its own outbound bloom filters
|
|
- **No spanning tree participation**: The leaf does not offer itself as a
|
|
potential parent to other nodes
|
|
- **Simplified routing**: All traffic tunnels through the upstream peer
|
|
- **Minimal resource usage**: Suitable for ESP32-class devices (~500KB RAM)
|
|
|
|
### Upstream Peer Responsibilities
|
|
|
|
The upstream peer:
|
|
|
|
- Includes the leaf's identity in its outbound bloom filters
|
|
- Forwards all traffic addressed to the leaf
|
|
- Handles discovery responses on behalf of the leaf
|
|
- Maintains the link session with the leaf
|
|
|
|
### What the Leaf Retains
|
|
|
|
Even as a leaf-only node, it still:
|
|
|
|
- Maintains its own Noise IK link session with the upstream peer (FMP layer)
|
|
- Can establish end-to-end FSP sessions with arbitrary destinations
|
|
- Has its own identity (npub, node_addr)
|
|
|
|
The optimization is purely at the routing/mesh layer — the leaf delegates
|
|
routing decisions but retains its own end-to-end encryption and identity.
|
|
|
|
## Packet Type Summary
|
|
|
|
For typical sizes, forwarding category, and the byte-level layouts
|
|
of each FMP and FSP message type, see
|
|
[../reference/wire-formats.md](../reference/wire-formats.md). The
|
|
canonical Packet Type Summary table lives there.
|
|
|
|
## Privacy Considerations
|
|
|
|
Source and destination node_addrs are visible to every transit node (required
|
|
for forwarding decisions and error signal routing). FIPS prioritizes
|
|
low-latency greedy routing with explicit error signaling over metadata
|
|
privacy.
|
|
|
|
The node_addr is `SHA-256(pubkey)` truncated to 128 bits — a one-way hash.
|
|
Transit nodes learn which node_addr pairs are communicating but cannot
|
|
determine the actual Nostr identities (npubs) of the endpoints. An observer
|
|
can verify "does this node_addr belong to pubkey X?" but cannot enumerate
|
|
communicating identities from traffic alone.
|
|
|
|
Onion routing was considered and rejected because it requires the sender to
|
|
know the full path upfront (incompatible with self-organizing routing) and
|
|
prevents per-hop error feedback (incompatible with CoordsRequired/PathBroken
|
|
recovery).
|
|
|
|
## Implementation Status
|
|
|
|
| Feature | Status |
|
|
| ------- | ------ |
|
|
| Spanning tree formation | **Implemented** |
|
|
| TreeAnnounce gossip | **Implemented** |
|
|
| Bloom filter computation (split-horizon) | **Implemented** |
|
|
| FilterAnnounce gossip | **Implemented** |
|
|
| find_next_hop() priority chain | **Implemented** |
|
|
| Coordinate cache (unified, TTL + refresh) | **Implemented** |
|
|
| Selective coord cache invalidation on parent change | **Implemented** |
|
|
| LookupRequest/LookupResponse discovery | **Implemented** |
|
|
| SessionSetup self-bootstrapping | **Implemented** |
|
|
| Hybrid coordinate warmup (CP + CoordsWarmup) | **Implemented** |
|
|
| CoordsRequired recovery | **Implemented** |
|
|
| PathBroken recovery | **Implemented** |
|
|
| MtuExceeded recovery | **Implemented** |
|
|
| LookupResponse proof verification | **Implemented** |
|
|
| Discovery reverse-path routing | **Implemented** |
|
|
| Error signal rate limiting | **Implemented** |
|
|
| Flap dampening (hysteresis + hold-down) | **Implemented** |
|
|
| Link liveness (dead timeout) | **Implemented** |
|
|
| Discovery request deduplication | **Implemented** |
|
|
| Discovery bloom-guided tree routing | **Implemented** |
|
|
| Discovery retry logic | **Implemented** |
|
|
| Discovery originator backoff | **Implemented** |
|
|
| Discovery transit-side rate limiting | **Implemented** |
|
|
| Discovery response relay eligibility and memory | **Implemented** |
|
|
| Leaf-only operation | Under development |
|
|
| Link cost in parent selection (ETX) | **Implemented** |
|
|
| Link cost in candidate ranking | **Implemented** |
|
|
|
|
## References
|
|
|
|
- [fips-concepts.md](fips-concepts.md) — Protocol overview
|
|
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
|
|
identity model
|
|
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP specification
|
|
- [fips-spanning-tree.md](fips-spanning-tree.md) — Tree algorithms and data
|
|
structures
|
|
- [fips-bloom-filters.md](fips-bloom-filters.md) — Filter parameters and math
|
|
- [../reference/wire-formats.md](../reference/wire-formats.md) — Wire
|
|
format reference
|
|
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence
|
|
walkthroughs
|