mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 07:37:36 +00:00
Corrections to the user, reference and design documentation and to source comments where they disagree with the code. No behaviour or wire change; the tests added here pin the behaviour the corrected text now states. Correct the OpenWrt tutorial on restricting port forwards. The OpenWrt gateway tutorial said the FIPS mesh firewall is the layer that limits which peers reach a port forward, pointed at the mesh firewall how-to, and told readers to add a fips-firewall drop-in for the listen port. None of that applies: the OpenWrt package ships neither the mesh firewall baseline nor fips-firewall.service, and on Linux the baseline hooks the input chain only and never sees a forward to a LAN host. A reader following the text would leave the forward open to every mesh peer that can reach the AP. The tutorial now says that the package's fips zone accepts port forwards from every mesh peer by default, and shows the fw4 rules that restrict one: an accept for the allowed peer followed by a REJECT for the rest, matching the target's port, with the note that an accept alone restricts nothing because other peers fall through to the packaged accept. The summary bullet and the troubleshooting row point at fw4 instead of the mesh firewall, and Linux readers are sent to the forward-hook recipe in deploy-gateway.md. Describe conntrack-less pinning and BLE evictions correctly. The gateway troubleshooting guide said that with no readable conntrack source a mapping is reclaimed on its TTL and grace period alone. It left out what the pool does on top of that: no mapping counts as having carried traffic, so a re-query does not extend a name (only a used mapping is refreshed, and the answer TTL is capped at the time left since creation), and at the ceiling a new name can replace any mapping. The guide now quotes the daemon's whole warning, states both effects, and says when mappings count traffic again: those created after conntrack reads have succeeded for three ticks in a row. The doc comment on the BLE pool's eviction counter said it counts a non-static peer being displaced. There is no static-peer notion in the pool; it records the eviction of an unverified link past its verify grace by a newcomer, which is the only kind the pool makes. Bring the IPv6 adapter docs and comments in line with the code. Several statements about the IPv6 adapter disagreed with the code: - FIPS_IPV6_OVERHEAD was described as 77 bytes on top of the 106-byte FIPS_OVERHEAD; 77 is the total (106 + 4 + 7 - 40). - The effective-MTU table called a 1472-byte transport the standard deployment; the UDP default is 1280. - All ICMPv6 errors were said to be rate-limited by a per-source token bucket. Only Packet Too Big is limited, by a per-source minimum interval of 100 ms; Destination Unreachable is not limited. - The TUN writer was said to clamp MSS on inbound SYN-ACKs only, and clamped SYN-ACKs were listed as one of its packet sources. The clamp acts on any SYN-flagged segment, in both the reader and the writer. The design docs, the MTU how-to, the writer's doc comments and its trace message now say so. - The design doc said the reader needs no "is this for me?" check. It has one: a packet to the node's own address is written back to the TUN, which macOS utun needs. - The app-owned TUN docs (design doc and the enable_app_owned_tun rustdoc) said FIPS neither filters off-mesh destinations nor answers them with ICMPv6 there, and that the embedder clamps only outbound SYNs. On that path an off-mesh destination is looked up and answered with Destination Unreachable on app_inbound_rx (an oversize packet with Packet Too Big), and no MSS is clamped in either direction. - The transport_mtu fallback comment said "UDP first, then Ethernet"; it takes a configured UDP instance's MTU, else 1280. A new test pins the app-owned behaviour the rustdoc now states: an off-mesh unicast destination is answered with Destination Unreachable from the node's own address on app_inbound_rx, and a multicast destination gets no answer. State what the debug inject command and the outbound path do. The debug_commands documentation (the config field's doc comment, the configuration reference and the Inject parameters doc) said inject writes only into one of the calling client's own flows, and so grants the client nothing it does not already have. inject looks the flow up by id in the node-wide flow table, so it can write into any flow on the node, including one another client opened or a listener accepted; the comment on the command itself and the native API reference already said so. All three now say the same, and the claim that it grants nothing new is gone. The handle_native_outbound doc said an error is reported to the client rather than as an ICMPv6 message. Nothing is reported: an oversize datagram is counted in drop_oversize and dropped, a send or session-initiation failure is logged at debug, and a datagram past the pending-queue bounds is dropped. The doc now lists those. Stop claiming a withdrawn advert is deleted with NIP-09. The Nostr events reference, the discovery design doc and the generic port-advertisement doc said the node withdraws its advert with a NIP-09 kind 5 deletion when advertising is turned off or the responder retires. The node never sends one. The delete plan is produced only when advertising is off and an advert event id is recorded, but an id is recorded only after a successful publish, which needs advertising on, and the advertise flag is fixed when the advert machine is built. A transient missing advert and shutdown both deliberately skip the delete, because on relays that apply deletions strictly it can also remove the replacement the next start publishes. The docs now say that no deletion is sent, and that an advert the node stops publishing stays on relays until its NIP-40 expiration, advert_ttl_secs after publication, unless a newer advert replaces it. The generic description keeps deletion as something a responder may do, and says FIPS does not. The doc comments on the publish plan's delete variant, plan_publish and clear_event_id now say that the delete path is never reached by a running node, and why. Correct the traversal signaling description. The Nostr events reference and the design docs disagreed with the signaling code in four places: - inReplyTo was described as the offer's event id. The responder sets it to the offer's nonce, the initiator matches answers by nonce, and the validator refuses any other value as a session mismatch. An implementation written from the docs would have every answer dropped. - The answer's PunchHint was said to tell both sides when to probe. Only the responder probes on it; the initiator does not read the hint and starts probing from its own punch_* settings when the answer arrives. Fixed in the reference, the port-advertisement doc and the discovery design doc. - All three event kinds were said to be signed with the node's identity key. The kind 21059 wrap is signed by a one-time key; only the seal inside it is signed with the identity key. - The FIPS advert example did not say that FIPS emits only the d, protocol, version and expiration tags and carries its signaling relays and STUN servers in the content as signalRelays and stunServers, present only with a udp:nat endpoint. Two tests pin the behaviour the docs now state: an answer whose inReplyTo is the offer's event id is refused as a session mismatch, and a signal's wrap is signed by a key other than the sender's while unwrapping recovers the sender's identity from the seal. Label the second msg2 DH as a repeat of es, not se. The IK msg2 code, the noise module doc, the wire-formats reference, the mesh-layer design doc and the msg2 diagram all called the second DH in msg2 `se`. Both sides actually pair the responder's static key with the initiator's ephemeral key, which is msg1's `es` pairing, so the step mixes the msg1 `es` value a second time and nothing computes Noise IK's `se`. Relabel it as `es` again everywhere, say in the code that it is not Noise's `se`, that it is the FMP v1 wire format and cannot change without breaking v1 peers, and that it leaves FMP v1 open to key compromise impersonation (FRC-1201 Section 9.3). Rename the two local bindings to `es2`. No behaviour or wire change. Correct the path MTU notification doc and the fips-mtu design doc. The PathMtuNotification doc said it is sent by a node that learns a path MTU from transit router feedback or ICMP Packet Too Big. It is sent by the session destination, echoing the forward-path minimum it saw in the envelope's path_mtu; a received MtuExceeded only updates local state. fips-mtu.md said TCP reports a per-connection MTU from TCP_MAXSEG. TCP reports the configured mtu and checks sends against it. It also said the TCP clamp mirror is tighter-only and never loosened. Entries are released on PathBroken (at most once a second per destination), peer removal and session timeout, discovery-learned entries expire with the coordinate cache TTL, and a link seed from a different transport replaces the stored value. Describe that lifecycle, and add a short section on the defences against forged signals: the 256-byte floors, the admission gate, corroboration against what this node has sent, and the rate-limited release. Correct transport comments and docs on TCP MTU, mDNS, BLE, Ethernet, Tor and UDP. Several transport comments and docs described behaviour the code does not have: - TCP: the config doc, configuration reference and transport-layer doc said the per-connection MTU comes from TCP_MAXSEG. TCP reports the configured mtu to FMP and checks sends against it. - mDNS: the module and supervisor comments said identity is proven by a Noise XX handshake. FMP here is Noise IK. Drop the note in the Nostr discovery doc that pointed at those comments. - BLE: the pubkey-exchange prefix doc said it is distinguished from FMP packets of version 1 or more. FMP is version 0, so the framer cannot tell them apart, as the caution below it already says. - Ethernet: the config doc said the runtime requires Linux. It is built for Linux (AF_PACKET) and macOS (BPF). - Tor: the config and transport docs, and the onion-service how-to, said control_port mode runs an inbound onion service. It only adds control-port monitoring to SOCKS5 outbound; directory mode is the only inbound mode. - UDP: the security reference said a UDP instance has no inbound exposure until bind_addr is set. A declared UDP instance listens on 0.0.0.0:2121 by default unless outbound_only is set. Correct the spanning tree docs on signatures, sequence, sizes, cache invalidation and flap dampening. - The TreeAnnounce signature was described as covering the entire message. It covers SHA-256 of the 48-byte declaration (node_addr, parent, sequence, timestamp); version and ancestry are not signed. - The sequence was said to increment on parent change. It also increments on becoming root and on an ancestry change under the same parent. - The TreeAnnounce size table added msg_type twice in the link-overhead column. The measured sizes are 168/264/328/488 bytes; note the 36-byte link overhead under the table. - Several design docs and a rate-limit comment said a parent change flushes the coordinate cache. The node drops only cached coordinates that route through it (parent switch or ancestry change) or sit under another root (becoming root), and both on parent loss or loop detection. - Flap dampening was described as engaging after more than flap_threshold switches in a sliding window. It engages when the count reaches flap_threshold, within a fixed window opened by the first switch. Correct the bloom filter frame size, triggers and propagation bound. - The wire-formats reference gave the v1 FilterAnnounce with link overhead as 1,072 bytes. It is 1,071 (1,035 payload plus 16 header, 4 timestamp and 16 tag), as the bloom design doc already says. - The bloom design doc listed local state changes (new identity, leaf-only dependent changes) as a send trigger. The identity does not change at runtime and nothing outside the tests adds a leaf-only dependent, so drop it. - The bloom and spanning tree docs bounded propagation at about 500 ms per hop. Announces leave only on the node's maintenance tick, so the bound is about one node.tick_interval_secs per hop (1 s by default); the 500 ms debounce is a floor below that. Correct forwarding docs on ranking, loop freedom and the coordinate cache. - The mesh-operation doc called link_cost an uncommon tie-breaker resolved by tree distance. Candidate ranking compares link cost first; tree distance and node_addr break ties. - Several docs and comments said the strict-progress check prevents routing loops even with stale coordinates. Each hop judges progress on its own cached coordinates for the destination, so the check does not guarantee loop freedom when hops hold different coordinates. Say that, and that the TTL bounds any loop. - The session-layer doc said FMP can fall back to bloom-only routing without cached coordinates. find_next_hop returns no next hop on a cache miss and the caller signals CoordsRequired. - The insert_coord_hint doc said a foreign-root destination can never be selected by the bloom step. A peer advertising coordinates under that root can be; the refusal stands for the reflector reason given in the same paragraph. - The find_next_hop and try_warm_coord_cache_ref doc blocks sat above the wrong functions, so rustdoc gave those two no docs and gave insert_coord_hint and warm_coord the wrong opening line. Move each block onto its function and fix the find_next_hop ranking wording. Correct FSP restart handling, error datagram sizes and XK labels. - The session-layer doc said an epoch mismatch on a later handshake marks a peer restart and triggers re-establishment. FSP exchanges the epoch but never reads it. A SessionSetup naming an established peer only arms a handshake beside the running session, and new keys are adopted at msg3 only when the authenticated static key matches. Describe that, and rename the status row accordingly. - Wire-format sizes: SessionDatagram + CoordsRequired is 74 bytes, not 70; SessionDatagram + PathBroken is 76 + 16n, not "70+"; PathBroken alone is 40 + 16n, not 35 + 16n; CoordsRequired with link overhead is 110, not 111; and the CoordsWarmup example is 238, not 239. State in the summary table which rows include link overhead. - Three FSP wire comments called the session handshake Noise IK. It is Noise XK, as the module's own phase table says. Correct the rationale for dropping our own looped-back request. The comment on the own-request check in classify_request, its test and the handle_lookup_response doc said that without the check the answer to our own lookup would be reverse-path forwarded to the peer that looped the request, and that recent_requests decides originator status. That stopped being true when classify_response began deciding originator status from the pending lookup before it reads the transit record. Say what the check still does: it stops the copy being forwarded a second time and keeps our own id out of the transit record, so a response after the lookup completes has nothing to be relayed by. Extend the test to pin that: once the lookup is accepted, a later response for the same id is unsolicited.
642 lines
34 KiB
Markdown
642 lines
34 KiB
Markdown
# FIPS Discovery: Nostr-Mediated and LAN/mDNS
|
||
|
||
FIPS nodes have two discovery mechanisms beyond the static `peers[]`
|
||
list. The bulk of this document describes **Nostr-mediated discovery**,
|
||
which works across the internet using public Nostr relays as a
|
||
signaling channel and can punch through UDP NAT. A second, much
|
||
simpler mechanism — **LAN/mDNS discovery** — finds peers on the same
|
||
local link with no relay, STUN, or NAT traversal at all; it is
|
||
described in its own section near the end. The two are independent: a
|
||
node can enable either, both, or neither.
|
||
|
||
Nostr-mediated discovery lets FIPS nodes find each other, and if
|
||
necessary, punch through UDP NAT, using public Nostr relays as the
|
||
signaling channel. A node publishes its reachable transport endpoints to
|
||
a small set of relays under its own Nostr identity (which is also its
|
||
FIPS identity), and peers resolve those endpoints at dial time by npub.
|
||
For peers behind UDP NAT, the same relay channel carries an encrypted
|
||
offer/answer exchange, and STUN supplies the reflexive address used for
|
||
a coordinated hole-punch.
|
||
|
||
Nostr discovery is unconditionally compiled into the `fips` binary on
|
||
every supported platform and ships in every published release artifact
|
||
(`.deb`, AUR, systemd tarball, OpenWrt `.ipk` and `.apk`, FreeBSD
|
||
`.pkg`, macOS `.pkg`, Windows `.zip`). It is runtime-opt-in: the YAML
|
||
configuration defaults to disabled (`node.rendezvous.nostr.enabled:
|
||
false`), so the discovery runtime stays dormant — and opens no relay
|
||
connections — until an operator flips the flag. Default relay and
|
||
STUN-server lists ship in the config; both are optional overrides. When
|
||
disabled, nodes behave exactly as before: only the static `peers[]`
|
||
addresses are used.
|
||
|
||
## Role
|
||
|
||
The feature adds three capabilities on top of FIPS's static peer model:
|
||
|
||
- **Advertising.** A node publishes the transport endpoints it wants
|
||
peers to use (direct UDP, direct TCP, a Tor onion, or the special
|
||
`udp:nat` rendezvous token) as a signed Nostr event. The advert is
|
||
anchored to the node's FIPS identity key — a peer that knows the npub
|
||
knows the advert is authentic.
|
||
- **Lookup.** When dialing a configured peer marked `via_nostr`, or any
|
||
peer in `policy: open` mode, the node fetches that peer's advert from
|
||
the configured relays and appends the advertised endpoints to its
|
||
dial list. Static addresses are always tried first.
|
||
- **UDP NAT hole-punch.** When both sides of a connection have UDP NAT
|
||
endpoints, the advert carries enough information to run a STUN-based
|
||
offer/answer exchange over encrypted ([NIP-59](https://github.com/nostr-protocol/nips/blob/master/59.md))
|
||
Nostr events. Each side observes its reflexive address via STUN,
|
||
exchanges candidate pairs through the relay, and both sides send UDP
|
||
probes at a shared punch time. On the first successful probe, the
|
||
punch socket is handed to FMP and becomes a normal UDP transport.
|
||
|
||
## When to use it
|
||
|
||
- **You run a public node** and want peers who know your npub to reach
|
||
you without you distributing an address list out-of-band.
|
||
- **You want to reach a peer behind UDP NAT** without deploying a relay
|
||
or running Tor on both sides. The peer advertises `udp:nat` and you
|
||
dial by npub.
|
||
- **You want zero-touch peer discovery** within a known application
|
||
namespace (`policy: open`), subject to an admission budget.
|
||
- **You want to advertise a Tor onion** so peers don't need to know the
|
||
`.onion` address out-of-band.
|
||
|
||
Skip the feature when every peer is already reachable through a stable
|
||
static address (a LAN mesh, a pre-configured test bed, or a deployment
|
||
where operators distribute `peers[]` blocks directly). The feature adds
|
||
relay dependencies, STUN round-trips for NAT cases, and a small ambient
|
||
background of relay traffic; none of that is useful when you already
|
||
know where peers are.
|
||
|
||
## Scenarios and configuration
|
||
|
||
For end-to-end operator recipes — each of the five activation scenarios
|
||
(advertise a directly-reachable UDP node, advertise a Tor onion node,
|
||
look up a configured peer by npub without advertising, NAT hole-punch
|
||
between two configured peers, and open discovery within an `app`
|
||
namespace) — see
|
||
[../how-to/enable-nostr-discovery.md](../how-to/enable-nostr-discovery.md).
|
||
The full configuration knob tables, per-transport keys, and startup
|
||
validation rules live in
|
||
[../reference/configuration.md](../reference/configuration.md) under
|
||
`node.rendezvous.nostr.*`. The Kind 37195 advert event format is in
|
||
[../reference/nostr-events.md](../reference/nostr-events.md). The rest
|
||
of this document covers the design of the discovery runtime itself.
|
||
|
||
## Under the covers
|
||
|
||
The rest of this document describes how the feature works inside the
|
||
node. For the generic protocol shape (event tags, NIP usage, on-the-
|
||
wire offer/answer schema, failure-suppression machinery), see
|
||
[port-advertisement-and-nat-traversal.md](port-advertisement-and-nat-traversal.md).
|
||
|
||
### Overview
|
||
|
||
The discovery runtime is a background task group started during node
|
||
initialization when `nostr.enabled` is true. It maintains a single
|
||
`nostr-sdk` client connected to the union of `advert_relays` and
|
||
`dm_relays`, and runs four loops: advert publication, advert
|
||
subscription (for open discovery and cache warming), DM subscription
|
||
(for incoming offers and answers), and a periodic advert-cache prune.
|
||
Discovery has no CLI surface; all operations are driven by the
|
||
configuration and by connection attempts made by the rest of the node.
|
||
|
||
```text
|
||
+-----------------------+
|
||
| Discovery runtime |
|
||
+-----------------------+
|
||
| | |
|
||
advert publish | | DM sub (offers, answers)
|
||
| |
|
||
v v
|
||
+-------------------------+
|
||
| Nostr relay pool | (advert_relays ∪ dm_relays)
|
||
+-------------------------+
|
||
^ ^
|
||
advert fetch/cache | | encrypted signaling
|
||
| |
|
||
+----------------+ | | +--------------------+
|
||
| connect_peer |--+ +->| offer / answer |
|
||
| (node side) | | handler |
|
||
+----------------+ +--------------------+
|
||
| |
|
||
v v
|
||
+---------+ +--------------+
|
||
| STUN |<-- same socket --->| UDP punch |
|
||
+---------+ +--------------+
|
||
|
|
||
v
|
||
adopt_established_traversal()
|
||
|
|
||
v
|
||
FMP IK handshake
|
||
on adopted socket
|
||
```
|
||
|
||
### Phase 1 — Advertisement
|
||
|
||
Adverts are published as Nostr kind `37195` parameterized replaceable
|
||
events (FIPS-specific, in the application-defined replaceable range
|
||
`30000–39999`; the digits visually spell `FIPS` — 7=F, 1=I, 9=P, 5=S).
|
||
The `d` tag is hardcoded to the wire-format identifier
|
||
`fips-overlay-v1` (or `fips-overlay-v1-next` on the `next` branch),
|
||
so each node has a single, in-place-updatable advert under its
|
||
identity. The configurable `app` value populates a separate
|
||
`protocol` tag, which scopes adverts within a relay set without
|
||
splitting them across multiple `d`-tag streams. The event is signed
|
||
with the node's FIPS identity key; there is no separate Nostr key. A
|
||
NIP-40 `expiration` tag is set to now + `advert_ttl_secs`, and a
|
||
`version` tag carries the protocol version. The advert content is a
|
||
JSON document shaped as `OverlayAdvert` (see
|
||
[../reference/nostr-events.md](../reference/nostr-events.md) for the
|
||
schema).
|
||
|
||
Publication happens on startup, again whenever the set of advertised
|
||
endpoints changes (for example, when a Tor onion hostname first
|
||
becomes available), and on a refresh timer every `advert_refresh_secs`.
|
||
The node sends no NIP-09 deletion for its advert: one it stops
|
||
publishing stays on relays until its NIP-40 expiration, unless a newer
|
||
advert replaces it first (see
|
||
[../reference/nostr-events.md](../reference/nostr-events.md#replacement-and-deletion)).
|
||
Advert publication is
|
||
fan-out: the same event is sent to every relay in `advert_relays` with
|
||
no explicit failover — relay redundancy is implicit.
|
||
|
||
For a UDP or TCP transport with `public: true`, the address advertised
|
||
follows a fixed precedence: an operator-supplied `external_addr` wins;
|
||
otherwise a non-wildcard bound `local_addr` is used directly;
|
||
otherwise — only for UDP — the runtime asks `stun_servers` for the
|
||
reflexive address of the bound socket and advertises that. TCP has no
|
||
STUN equivalent, so wildcard-bound TCP without `external_addr`
|
||
produces a loud WARN and the endpoint is omitted from the advert.
|
||
|
||
### Phase 2 — Lookup
|
||
|
||
When the node decides to dial a peer that is eligible for Nostr
|
||
resolution (a `via_nostr` peer, or any peer under `policy: open`), it
|
||
issues a Nostr REQ filtered by `author = peer_pubkey`, `kind = 37195`,
|
||
`#d = fips-overlay-v1`. The fetch is time-bounded (~2 s) and runs
|
||
against all configured `advert_relays` in parallel. The first valid
|
||
advert wins; adverts whose `protocol` tag does not match the local
|
||
`app` value are rejected at validation.
|
||
|
||
Results are kept in an in-memory cache keyed by author npub. Cache
|
||
entries carry the advert's expiration time, and a periodic prune drops
|
||
expired entries. When the cache holds `advert_cache_max_entries` valid
|
||
adverts, an advert from a new author that is neither a configured peer
|
||
nor linked to this node is refused until an entry expires; adverts
|
||
already cached are not evicted for it. Configured and linked authors are
|
||
always cached, taking the slot of the unprotected entry cached most
|
||
recently. A parallel long-lived subscription on the advert relays
|
||
populates the cache passively, so open-discovery candidates do not
|
||
require per-dial fetches. Each open-discovery sweep considers the
|
||
configured authors not yet linked and a fresh random draw of up to 64
|
||
other cached authors that are neither linked nor already queued.
|
||
|
||
On cache hit, advert endpoints are appended to the peer's static
|
||
address list with lower priority; the static list is tried first.
|
||
|
||
### Phase 3 — Offer/Answer signaling
|
||
|
||
For any endpoint shaped as `udp:nat`, dialing triggers an
|
||
offer/answer exchange before the first packet is sent. Signaling events
|
||
are Nostr kind `21059` (ephemeral, not stored by conforming relays),
|
||
gift-wrapped per [NIP-59](https://github.com/nostr-protocol/nips/blob/master/59.md)
|
||
and encrypted with [NIP-44](https://github.com/nostr-protocol/nips/blob/master/44.md),
|
||
so only the intended recipient can decrypt the payload.
|
||
|
||
The initiator performs STUN first (see Phase 4), then builds a
|
||
`TraversalOffer` containing:
|
||
|
||
- A unique `sessionId` and a random `nonce` (used to correlate the
|
||
answer).
|
||
- Its reflexive address (if STUN succeeded).
|
||
- Its list of local (private) addresses for same-LAN paths.
|
||
- The STUN server it used, for informational reporting only.
|
||
- An `expiresAt` equal to now + `signal_ttl_secs`.
|
||
|
||
The offer is sealed to the recipient's npub and published to the merge
|
||
of the peer's NIP-17 inbox relay list (kind 10050), the signal relays
|
||
its advert names, and our own `dm_relays`, kept to relays in our own
|
||
relay pool. Each side publishes its own inbox relay list on startup
|
||
and daily so dialers can discover it.
|
||
|
||
On the receiving side, admission is a pair of bounds taken together: a
|
||
per-sender allowance of `max_concurrent_offers_per_npub`, keyed on the
|
||
npub that signed the gift wrap, nested inside a global
|
||
`max_concurrent_incoming_offers`. A sender over its own allowance is
|
||
refused at debug, since by definition it is sending faster than the node
|
||
wants and a record per rejection would turn the spam into log volume; the
|
||
global bound being reached is the operator-visible warn, because that one
|
||
says the node is genuinely saturated. Together they keep one identity
|
||
from holding the whole pool. They do not make the pool inexhaustible:
|
||
nostr identities are free to generate, so an attacker running
|
||
`ceil(max_concurrent_incoming_offers / max_concurrent_offers_per_npub)`
|
||
throwaway npubs still saturates it at the same total offer rate. Raising
|
||
the attacker's cost beyond keypairs would mean pricing the offer itself.
|
||
A `sessionId` replay cache (bounded by `seen_sessions_max_entries`, with
|
||
entries valid for `replay_window_secs`) rejects duplicates.
|
||
|
||
The responder runs its own STUN query and replies with a
|
||
`TraversalAnswer` carrying its reflexive and local addresses plus a
|
||
`PunchHint { startAtMs, intervalMs, durationMs }` giving its own
|
||
probing schedule, which the responder probes on. The initiator
|
||
does not read the hint: it starts its own probing when the answer
|
||
arrives, on its local `punch_*` settings. If the responder has no
|
||
usable addresses at all, it replies with `accepted: false` and a
|
||
`reason` string.
|
||
|
||
### Phase 4 — UDP hole-punch
|
||
|
||
Each side runs STUN (parsing XOR-MAPPED-ADDRESS from the response, all
|
||
other attributes ignored) on the *same* UDP socket it will later use
|
||
for punching and for the adopted FMP transport. This is critical: NAT
|
||
state is per-socket, so the punch has to reuse the socket that taught
|
||
the NAT about this binding.
|
||
|
||
Given its own reflexive + local addresses and the peer's, each side
|
||
builds a candidate-pair plan that tries, in priority order:
|
||
|
||
1. **Reflexive ↔ reflexive.** The classic STUN path. Tried first because
|
||
it is the only candidate that's reliable across arbitrary network
|
||
topologies — host candidates from one peer that happen to be
|
||
reachable from the other (via a corporate VPN, a Tailscale subnet
|
||
route, or overlapping private address space) will succeed at the
|
||
socket layer in the punch but fail in the FMP handshake when the
|
||
return path doesn't match.
|
||
2. **LAN ↔ LAN.** When one of this node's interface prefixes holds
|
||
both a local candidate of ours and the peer's private IPv4
|
||
candidate, the two are on one link and likely reachable directly.
|
||
Only fires when both peers shared local host candidates (which
|
||
requires `share_local_candidates` to be enabled — off by default).
|
||
More generally, a peer's private IPv4 candidate, its reflexive
|
||
address included, is punched only when one of our interface
|
||
prefixes holds both it and one of our own addresses (a shared local
|
||
candidate, or our reflexive address when that is private); other
|
||
private and unique-local candidates are refused.
|
||
3. **Mixed.** Reflexive on one side, local on the other — catches
|
||
hairpin and one-side-public scenarios.
|
||
|
||
Each side then sends 24-byte probe packets on the candidate pair(s)
|
||
on its own schedule. The responder follows the hint it sent: it starts
|
||
at `startAtMs` and probes every `intervalMs` for up to `durationMs`.
|
||
The initiator starts `punch_start_delay_ms` after it has the answer
|
||
and uses its own `punch_interval_ms` and `punch_duration_ms`. A
|
||
probe carries a 4-byte magic (`NPTC`), a 4-byte sequence, and the
|
||
first 16 bytes of `SHA256(sessionId)`; both sides can compute the same
|
||
session hash independently from the public `sessionId`, so no shared
|
||
secret is needed on the punch path itself. On receiving a valid probe,
|
||
a side replies with an `NPTA` ack. The first valid probe or ack seen
|
||
from the far side records the working remote address and completes the
|
||
attempt.
|
||
|
||
On timeout (`attempt_timeout_secs` as overall bound,
|
||
`punch_duration_ms` as probe window), the attempt reports failure up
|
||
to the discovery runtime's `BootstrapEvent::Failed` channel.
|
||
|
||
Neither side publishes a NIP-9 deletion request for the offer or
|
||
answer, on success or failure. A relay honouring NIP-59 deletes a gift
|
||
wrap only at the request of its p-tagged recipient, so such a request
|
||
would have to be signed by the node's routing key and would name the
|
||
traversal's events, tying that key to them on every relay it reached.
|
||
The wraps are ephemeral kinds carrying a NIP-40 expiration tag, so
|
||
relays that store them at all keep them until they expire.
|
||
|
||
### Phase 5 — Adoption
|
||
|
||
On success, the discovery runtime emits `BootstrapEvent::Established`
|
||
carrying the session id, the punch socket, and the learned remote
|
||
address. `adopt_established_traversal()` in the node lifecycle takes
|
||
the socket, registers it with the UDP transport layer as a new
|
||
transport instance, and calls `initiate_connection()` with the peer's
|
||
FIPS identity as the expected remote. FMP's Noise IK handshake runs on
|
||
the same socket — there is no "promote link" step between punch and
|
||
handshake; the punch socket *is* the FMP socket.
|
||
|
||
From that moment on, the connection is a normal FMP link and is
|
||
subject to the usual liveness (MMP heartbeats), rekey, and removal
|
||
behavior. A link-dead event does not re-enter the discovery runtime
|
||
automatically; reconnection relies on `auto_reconnect` and the same
|
||
dial path that triggered the original punch.
|
||
|
||
### Auto-connect semantics
|
||
|
||
Discovery does not itself initiate connections. It only supplies
|
||
addresses. Dial attempts originate from the existing peer-connection
|
||
machinery:
|
||
|
||
- **Configured peers** (`peers[]` with `connect_policy: auto_connect`)
|
||
are dialed on startup and on retry. When `via_nostr` is set, advert
|
||
endpoints are appended to the dial list with lower priority than
|
||
static entries.
|
||
- **Open discovery peers** are assembled from the advert cache, fenced
|
||
by the peer ACL, and enqueued into a bounded retry queue sized by
|
||
`open_discovery_max_pending`. There is no event-driven
|
||
"connect on every advert" — a peer re-enters the queue only when its
|
||
prior attempt has drained.
|
||
- **Manual dials** (`fipsctl connect`) can target any configured peer
|
||
and use the same dial path, including Nostr resolution if configured.
|
||
|
||
### Rate limits and safeguards
|
||
|
||
| Mechanism | Default | What it prevents | Behavior at limit |
|
||
| --- | --- | --- | --- |
|
||
| Offer semaphore (`max_concurrent_incoming_offers`) | 16 | CPU and memory exhaustion from offer spam on DM relays. | Warn log, offer dropped. |
|
||
| Per-npub offer allowance (`max_concurrent_offers_per_npub`) | 4 | One sender identity holding every offer slot and denying traversal onboarding to everyone else. Does not prevent the same denial from several throwaway npubs. | Debug log, offer dropped. |
|
||
| Advert cache (`advert_cache_max_entries`) | 2048 | Memory growth from ambient advert traffic under `policy: open`. | New unconfigured authors refused until an entry expires; configured and linked authors kept. |
|
||
| Seen-sessions (`seen_sessions_max_entries`) | 2048 | Replay of stale `sessionId` values. | Oldest entry evicted. |
|
||
| Signal TTL (`signal_ttl_secs`) | 120 s | Indefinite in-flight offers on relays. | Expired offers rejected at validation. |
|
||
| Open discovery queue (`open_discovery_max_pending`) | 64 | Unbounded retry queue under ambient advert load. | New candidates skipped until the queue drains. |
|
||
| Punch window (`punch_duration_ms`) | 10 s | Endless probe traffic after one side has given up. | Attempt declared failed; sockets discarded. |
|
||
| Failure-streak threshold (`failure_streak_threshold`) | 5 | Repeated traversal attempts against a peer that keeps failing. | Peer enters extended cooldown. |
|
||
| Extended cooldown (`extended_cooldown_secs`) | 1800 s | Tight retry loops after a failure streak. | Per-peer suppression for the cooldown window. |
|
||
| WARN log throttle (`warn_log_interval_secs`) | 300 s | Log floods from a peer that fails on every attempt. | One WARN per peer per interval; the rest demote to debug. |
|
||
| Failure-state cap (`failure_state_max_entries`) | 4096 | Memory growth from per-peer failure tracking. | LRU eviction. |
|
||
|
||
The load-shedding mechanisms (`max_concurrent_incoming_offers` and the
|
||
failure-streak / extended-cooldown pair) are deliberately conservative
|
||
so that a misbehaving relay cannot flood the node with offers and a
|
||
chronically unreachable peer cannot keep the traversal pipeline
|
||
saturated. The remaining rows are capacity bounds.
|
||
|
||
Adverts also undergo a stale-advert sweep: cached entries whose
|
||
`expiresAt` has passed are evicted on the periodic prune tick. Inbound
|
||
signaling tolerates ±60 s of clock skew between sender and receiver,
|
||
and the runtime maintains an NTP-style skew estimate per remote so
|
||
that consistently-skewed relays don't trip the freshness check.
|
||
|
||
### Relay model
|
||
|
||
All configured relays (advert + DM) are opened on a single
|
||
`nostr-sdk::Client` at startup. Publication is fan-out: the same event
|
||
is sent to every relay in the target list, with no explicit retry or
|
||
relay selection. Redundancy is implicit — a downed relay simply means
|
||
its copy of the advert or signal is unavailable, while other relays
|
||
still serve the same data.
|
||
|
||
For signaling specifically, the node sends to the merge of the
|
||
recipient's NIP-17 inbox relays (the recipient publishes its list as a
|
||
kind 10050 event to its own DM relays on startup and daily), the
|
||
signal relays the recipient's advert names, and the local `dm_relays`
|
||
list, kept to relays already in the node's own pool. The inbox list
|
||
costs one extra NIP-17 fetch per offer.
|
||
|
||
There is no per-relay rate limiting or health check. The relay model
|
||
assumes that an operator chooses relays they trust to be best-effort
|
||
available and that outright misbehavior is handled at the offer
|
||
semaphore and replay-cache layers downstream.
|
||
|
||
## Security and threat model
|
||
|
||
- **Relay operators can observe metadata.** They see which npubs
|
||
publish adverts, to whom offers are sent, and the timing of that
|
||
traffic. The *contents* of offer and answer events are
|
||
NIP-59/NIP-44 sealed — only the intended recipient decrypts them.
|
||
Adverts are public by design.
|
||
- **STUN servers see the node's public IP and port.** Only the STUN
|
||
servers listed in the node's own `stun_servers` are ever contacted
|
||
for reflexive discovery. Peer-advertised STUN values are
|
||
informational; a malicious peer cannot steer this node to a
|
||
chosen STUN target. See the doc comment on
|
||
`node.rendezvous.nostr.stun_servers`.
|
||
- **The FIPS identity key signs adverts.** Compromise of
|
||
`fips.key` is compromise of the node's Nostr identity — an attacker
|
||
can publish adverts on behalf of the node. The recovery path is
|
||
the same as for any identity compromise: rotate the key and
|
||
re-advertise. There is no separate Nostr keypair to rotate
|
||
independently.
|
||
- **Tor advertising leaks timing via clearnet relays.** When a
|
||
Tor-only node advertises its onion address, the advert itself is
|
||
published on clearnet WebSocket relays. Operators who want full
|
||
unlinkability between the advertising identity and the node's
|
||
IP must route relay traffic through Tor as well — for example by
|
||
running `fips` inside a network namespace with a Tor SOCKS
|
||
proxy as its only egress, or by pointing `advert_relays` and
|
||
`dm_relays` at onion relay endpoints.
|
||
- **Open discovery accepts anyone publishing on the same `app`.**
|
||
Admission control is the peer ACL, not the discovery layer. Verify
|
||
the ACL before enabling `policy: open`, and consider using a
|
||
non-default `app` value to scope visibility.
|
||
- **Nothing about discovery bypasses FMP.** A successful punch yields
|
||
a UDP socket with a claimed remote identity. That identity is not
|
||
trusted until FMP's Noise IK handshake completes. A peer whose
|
||
advert says "I am npub X at 1.2.3.4:5678" but whose FMP handshake
|
||
presents a different static key is rejected at the mesh layer.
|
||
|
||
## LAN/mDNS discovery
|
||
|
||
LAN discovery is a separate, link-local discovery mechanism that finds
|
||
peers on the same broadcast domain using mDNS / DNS-SD
|
||
([RFC 6762](https://www.rfc-editor.org/rfc/rfc6762) /
|
||
[RFC 6763](https://www.rfc-editor.org/rfc/rfc6763)). Unlike
|
||
Nostr-mediated discovery, it contacts no relay, runs no STUN
|
||
observation, and performs no NAT traversal: an endpoint learned from a
|
||
LAN advert is by construction routable from the consumer's own link.
|
||
The result is sub-second peer pairing on the same LAN.
|
||
|
||
It is unrelated to the "LAN candidate" terminology used in the
|
||
NAT-traversal sections above (which refers to a host's own
|
||
locally-bound address offered as a hole-punch candidate). LAN/mDNS
|
||
discovery is a distinct subsystem under `src/mdns/`.
|
||
|
||
### Role
|
||
|
||
LAN discovery adds two capabilities, both confined to the local link:
|
||
|
||
- **Advertising.** The node publishes a `_fips._udp.local.` DNS-SD
|
||
service advert carrying its `npub`, its protocol version, and (if
|
||
configured) a discovery scope. The advert is multicast on the local
|
||
link only; it does not leave the broadcast domain unless the
|
||
operator's network bridges mDNS.
|
||
- **Browsing.** The node concurrently browses for the same service
|
||
type, learns the endpoints of other FIPS nodes on the link, and
|
||
initiates a normal FMP link to each newly-seen peer. It dials an
|
||
advertised address only when it lies on the link its address record
|
||
arrived on (inside a prefix of the interface that received the
|
||
record, or IPv6 link-local with a scope); an address outside it, for
|
||
example one carried in by an mDNS reflector between subnets or one
|
||
in the prefix of the node's VPN or container interface, is ignored.
|
||
The advert's service host must also be the advertised node's own
|
||
`fips-` host name, since the mDNS library looks a host's addresses
|
||
up on every interface and does not report which interface the
|
||
advert arrived on; a sender on one link can therefore still name a
|
||
FIPS node on another of the node's links, at any port and under
|
||
npubs it mints to share that node's host name, but no other host.
|
||
New LAN dials are paced: they share the per-tick discovery budget,
|
||
need room under `max_peers`, and at most 8 handshakes to
|
||
unconfigured LAN peers are in flight at once, at most 2 of them to
|
||
any one address. Adverts waiting for a dial are held for up to ten
|
||
minutes in a set of at most 64, with configured peers dialled
|
||
first.
|
||
|
||
The mDNS service type is `_fips._udp.local.`
|
||
(`src/mdns/mod.rs:45`). Per RFC 6763 the `_udp` label denotes
|
||
the IP transport used for the advert, not the FIPS upper protocol —
|
||
both UDP and TCP FIPS endpoints announce under the same service type
|
||
because the link-layer handshake travels over UDP either way. (In
|
||
practice LAN discovery dials only over a UDP transport; see the
|
||
handshake subsection.)
|
||
|
||
### When to use it
|
||
|
||
- **You run several FIPS nodes on one LAN** (a lab bench, an office
|
||
segment, a home network) and want them to find each other without
|
||
hand-maintaining `peers[]` blocks or standing up Nostr discovery.
|
||
- **You want the lowest-latency pairing path.** Same-link pairing
|
||
completes in well under a second with no relay round-trip.
|
||
|
||
Skip it when nodes are not on a shared broadcast domain (mDNS does not
|
||
cross routed boundaries), or when you do not want the node to multicast
|
||
its identity on the local link. LAN discovery is **opt-in and disabled
|
||
by default**, so doing nothing leaves it off.
|
||
|
||
### How it works
|
||
|
||
The LAN discovery runtime (`src/mdns/mod.rs`) is started
|
||
during node initialization when `node.rendezvous.lan.enabled` is true.
|
||
It is independent of Nostr discovery and runs even when Nostr is
|
||
disabled (`src/node/lifecycle/supervisor.rs:432-437`). Startup requires an
|
||
operational UDP transport: the node advertises the port of its
|
||
lowest-`TransportId` operational, non-bootstrap UDP transport, chosen
|
||
deterministically so the advertised port is stable across restarts
|
||
(`src/node/lifecycle/mod.rs:1598-1609`). If no such port exists, the
|
||
runtime returns `NoAdvertisedPort` and LAN discovery does not start
|
||
(`src/mdns/mod.rs:165-167`).
|
||
|
||
The runtime does two things concurrently:
|
||
|
||
1. **Responder.** It registers a DNS-SD service with instance name
|
||
`fips-<first-16-chars-of-npub>` and a TXT record carrying the keys
|
||
below. `mdns-sd`'s address auto-detection appends every non-loopback
|
||
interface address, with `127.0.0.1` seeded so same-host peers and
|
||
integration tests can still resolve the advert
|
||
(`src/mdns/mod.rs:179-212`).
|
||
2. **Browser.** A background pump receives `ServiceResolved` events for
|
||
the same service type. For each resolved advert it extracts the
|
||
`npub` and `scope` TXT values, drops adverts that echo the node's own
|
||
npub, drops cross-scope adverts (see scope filtering), drops records
|
||
without an `npub`, and surfaces one `LanDiscoveredPeer` per routable
|
||
interface address (`src/mdns/mod.rs:230-297`). IPv6
|
||
unicast link-local addresses without an interface scope id are
|
||
skipped, since they cannot be dialed unambiguously
|
||
(`src/mdns/mod.rs:357-370`).
|
||
|
||
The TXT record carries three keys (`src/mdns/mod.rs:48-55`):
|
||
|
||
| TXT key | Contents |
|
||
| --- | --- |
|
||
| `npub` | bech32-encoded npub of the advertising node |
|
||
| `scope` | the node's discovery scope, if one is configured (omitted otherwise) |
|
||
| `v` | FIPS protocol version (the same `PROTOCOL_VERSION` used by the Nostr advert) |
|
||
|
||
Once per node tick, the node drains browser events and acts on them in
|
||
`poll_lan_rendezvous()` (`src/node/lifecycle/mod.rs:1131`, called from
|
||
`src/node/dataplane/rx_loop.rs:444`). For each discovered peer it finds
|
||
a UDP transport whose family matches the peer address, parses the
|
||
`npub` into a `PeerIdentity`, skips peers it is already connected to or
|
||
currently connecting to, and otherwise initiates a connection.
|
||
|
||
### Handshake: Noise IK
|
||
|
||
LAN-discovered peers are dialed through the standard FMP outbound link
|
||
path. `poll_lan_rendezvous()` calls `initiate_connection()`
|
||
(`src/node/lifecycle/mod.rs:448`), which, for connectionless transports
|
||
such as UDP, allocates a link and **starts the Noise IK handshake**
|
||
(documented at `src/node/lifecycle/mod.rs:438-442`). This is the same
|
||
link-layer handshake used by every other FMP connection — IK at the
|
||
link layer per the FIPS architecture — not a different pattern for LAN
|
||
peers.
|
||
|
||
The mDNS advert is **unauthenticated**: anyone on the link can
|
||
multicast a TXT claiming any `npub`. Identity is proven end-to-end by
|
||
the Noise IK handshake against the observed endpoint. A spoofed advert
|
||
carrying another node's npub fails the handshake — the impostor does
|
||
not hold the matching static key — and the half-open link is dropped.
|
||
The mDNS advert is therefore a routing hint, never an identity
|
||
assertion, exactly as a Nostr advert is treated (a successful contact
|
||
is not trusted until FMP's Noise IK handshake completes).
|
||
|
||
### Scope filtering
|
||
|
||
When a discovery scope is configured, the advert carries it in the
|
||
`scope` TXT entry and the browser surfaces only peers whose advert
|
||
carries a matching scope. Nodes on the same physical LAN but configured
|
||
for different mesh networks therefore do not cross-feed each other.
|
||
|
||
The scope is resolved by `lan_rendezvous_scope()`
|
||
(`src/node/lifecycle/mod.rs:1104`): the explicit
|
||
`node.rendezvous.lan.scope`, if non-empty, is used directly. Otherwise
|
||
the node falls back to deriving a scope from the Nostr discovery `app`
|
||
tag (stripping the `fips-overlay-v1:` prefix when present). This lets
|
||
an application keep its public, relay-visible Nostr `app` tag generic
|
||
while still isolating LAN discovery per private network, or share one
|
||
value across both. A node with no scope on either side surfaces all
|
||
adverts it sees on the link.
|
||
|
||
### Configuration
|
||
|
||
LAN discovery is configured under `node.rendezvous.lan.*`
|
||
(`src/config/node.rs:334`, `src/mdns/mod.rs:92-114`):
|
||
|
||
| Key | Type | Default | Meaning |
|
||
| --- | --- | --- | --- |
|
||
| `node.rendezvous.lan.enabled` | bool | `false` | Master switch. LAN discovery is opt-in; default-off avoids an unexpected per-link identity multicast on upgrade. |
|
||
| `node.rendezvous.lan.service_type` | string | `_fips._udp.local.` | DNS-SD service type. Overridable mainly so integration tests can isolate multiple services on one loopback interface. |
|
||
| `node.rendezvous.lan.scope` | string (optional) | unset | Application/network scope carried in the LAN-only `scope` TXT record. Kept deliberately separate from the public Nostr `app` tag. When unset, the scope falls back to the derived Nostr `app` value. |
|
||
|
||
The identity surface published over mDNS (`npub`, version, optional
|
||
scope) is a strict subset of what `nostr.advertise` already publishes
|
||
publicly, so enabling LAN discovery adds no marginal privacy cost
|
||
beyond making the node's presence observable on its own local link.
|
||
|
||
### Relationship to Nostr discovery
|
||
|
||
The two mechanisms are complementary and independent:
|
||
|
||
| | Nostr-mediated | LAN/mDNS |
|
||
| --- | --- | --- |
|
||
| Reach | Internet-wide, via relays | Same broadcast domain only |
|
||
| Signaling channel | Public Nostr relays | mDNS multicast on the local link |
|
||
| NAT traversal | STUN + UDP hole-punch for `udp:nat` peers | None — endpoint is link-routable by construction |
|
||
| Identity carrier | signed kind 37195 advert (authenticated at publish) | unauthenticated mDNS TXT (routing hint only) |
|
||
| Identity proof | FMP Noise IK on the connection | FMP Noise IK on the connection |
|
||
| Default | disabled (`nostr.enabled: false`) | disabled (`lan.enabled: false`) |
|
||
| Scope key | `app` tag (public) | `scope` TXT (link-local), falls back to `app` |
|
||
|
||
Both ultimately converge on the same trust boundary: discovery only
|
||
supplies candidate endpoints, and no peer is trusted until FMP's Noise
|
||
IK handshake confirms the claimed identity. A node may run both at
|
||
once — for example, advertising globally over Nostr while also pairing
|
||
instantly with same-LAN peers — with no interaction between the two
|
||
beyond the shared scope fallback.
|
||
|
||
## See also
|
||
|
||
- [../how-to/enable-nostr-discovery.md](../how-to/enable-nostr-discovery.md)
|
||
— operator activation recipes grouped under three capabilities
|
||
(resolve, advertise, open) across five scenarios.
|
||
- [../tutorials/resolve-peers-via-nostr.md](../tutorials/resolve-peers-via-nostr.md),
|
||
[../tutorials/advertise-your-node.md](../tutorials/advertise-your-node.md),
|
||
and [../tutorials/open-discovery.md](../tutorials/open-discovery.md)
|
||
— hand-held walkthroughs of the three capabilities, in
|
||
pedagogical order.
|
||
- [../reference/configuration.md](../reference/configuration.md) — full
|
||
configuration reference, including all surrounding keys elided from
|
||
the scenarios above.
|
||
- [../reference/nostr-events.md](../reference/nostr-events.md) — Kind
|
||
37195 (overlay advert), Kind 21059 (gift-wrapped traversal
|
||
signaling), Kind 10050 (NIP-17 inbox relay list).
|
||
- [../reference/security.md](../reference/security.md) — consolidated
|
||
security reference, including how the FIPS identity key signs both
|
||
adverts and Noise handshakes.
|
||
- [fips-transport-layer.md](fips-transport-layer.md) — UDP, TCP, and
|
||
Tor transport mechanics; the punch socket is adopted as a normal
|
||
UDP transport after handoff.
|
||
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP Noise IK handshake
|
||
that runs on the adopted socket.
|
||
- [port-advertisement-and-nat-traversal.md](port-advertisement-and-nat-traversal.md)
|
||
— generic protocol reference (event tags, NIP usage, on-the-wire
|
||
offer/answer schema, failure-suppression machinery), with the
|
||
FIPS-specific values called out as worked examples.
|