Johnathan Corgan
51d34ac574
Corrections to the user, reference and design documentation and to source comments where they disagree with the code. No behaviour or wire change; the tests added here pin the behaviour the corrected text now states. Correct the OpenWrt tutorial on restricting port forwards. The OpenWrt gateway tutorial said the FIPS mesh firewall is the layer that limits which peers reach a port forward, pointed at the mesh firewall how-to, and told readers to add a fips-firewall drop-in for the listen port. None of that applies: the OpenWrt package ships neither the mesh firewall baseline nor fips-firewall.service, and on Linux the baseline hooks the input chain only and never sees a forward to a LAN host. A reader following the text would leave the forward open to every mesh peer that can reach the AP. The tutorial now says that the package's fips zone accepts port forwards from every mesh peer by default, and shows the fw4 rules that restrict one: an accept for the allowed peer followed by a REJECT for the rest, matching the target's port, with the note that an accept alone restricts nothing because other peers fall through to the packaged accept. The summary bullet and the troubleshooting row point at fw4 instead of the mesh firewall, and Linux readers are sent to the forward-hook recipe in deploy-gateway.md. Describe conntrack-less pinning and BLE evictions correctly. The gateway troubleshooting guide said that with no readable conntrack source a mapping is reclaimed on its TTL and grace period alone. It left out what the pool does on top of that: no mapping counts as having carried traffic, so a re-query does not extend a name (only a used mapping is refreshed, and the answer TTL is capped at the time left since creation), and at the ceiling a new name can replace any mapping. The guide now quotes the daemon's whole warning, states both effects, and says when mappings count traffic again: those created after conntrack reads have succeeded for three ticks in a row. The doc comment on the BLE pool's eviction counter said it counts a non-static peer being displaced. There is no static-peer notion in the pool; it records the eviction of an unverified link past its verify grace by a newcomer, which is the only kind the pool makes. Bring the IPv6 adapter docs and comments in line with the code. Several statements about the IPv6 adapter disagreed with the code: - FIPS_IPV6_OVERHEAD was described as 77 bytes on top of the 106-byte FIPS_OVERHEAD; 77 is the total (106 + 4 + 7 - 40). - The effective-MTU table called a 1472-byte transport the standard deployment; the UDP default is 1280. - All ICMPv6 errors were said to be rate-limited by a per-source token bucket. Only Packet Too Big is limited, by a per-source minimum interval of 100 ms; Destination Unreachable is not limited. - The TUN writer was said to clamp MSS on inbound SYN-ACKs only, and clamped SYN-ACKs were listed as one of its packet sources. The clamp acts on any SYN-flagged segment, in both the reader and the writer. The design docs, the MTU how-to, the writer's doc comments and its trace message now say so. - The design doc said the reader needs no "is this for me?" check. It has one: a packet to the node's own address is written back to the TUN, which macOS utun needs. - The app-owned TUN docs (design doc and the enable_app_owned_tun rustdoc) said FIPS neither filters off-mesh destinations nor answers them with ICMPv6 there, and that the embedder clamps only outbound SYNs. On that path an off-mesh destination is looked up and answered with Destination Unreachable on app_inbound_rx (an oversize packet with Packet Too Big), and no MSS is clamped in either direction. - The transport_mtu fallback comment said "UDP first, then Ethernet"; it takes a configured UDP instance's MTU, else 1280. A new test pins the app-owned behaviour the rustdoc now states: an off-mesh unicast destination is answered with Destination Unreachable from the node's own address on app_inbound_rx, and a multicast destination gets no answer. State what the debug inject command and the outbound path do. The debug_commands documentation (the config field's doc comment, the configuration reference and the Inject parameters doc) said inject writes only into one of the calling client's own flows, and so grants the client nothing it does not already have. inject looks the flow up by id in the node-wide flow table, so it can write into any flow on the node, including one another client opened or a listener accepted; the comment on the command itself and the native API reference already said so. All three now say the same, and the claim that it grants nothing new is gone. The handle_native_outbound doc said an error is reported to the client rather than as an ICMPv6 message. Nothing is reported: an oversize datagram is counted in drop_oversize and dropped, a send or session-initiation failure is logged at debug, and a datagram past the pending-queue bounds is dropped. The doc now lists those. Stop claiming a withdrawn advert is deleted with NIP-09. The Nostr events reference, the discovery design doc and the generic port-advertisement doc said the node withdraws its advert with a NIP-09 kind 5 deletion when advertising is turned off or the responder retires. The node never sends one. The delete plan is produced only when advertising is off and an advert event id is recorded, but an id is recorded only after a successful publish, which needs advertising on, and the advertise flag is fixed when the advert machine is built. A transient missing advert and shutdown both deliberately skip the delete, because on relays that apply deletions strictly it can also remove the replacement the next start publishes. The docs now say that no deletion is sent, and that an advert the node stops publishing stays on relays until its NIP-40 expiration, advert_ttl_secs after publication, unless a newer advert replaces it. The generic description keeps deletion as something a responder may do, and says FIPS does not. The doc comments on the publish plan's delete variant, plan_publish and clear_event_id now say that the delete path is never reached by a running node, and why. Correct the traversal signaling description. The Nostr events reference and the design docs disagreed with the signaling code in four places: - inReplyTo was described as the offer's event id. The responder sets it to the offer's nonce, the initiator matches answers by nonce, and the validator refuses any other value as a session mismatch. An implementation written from the docs would have every answer dropped. - The answer's PunchHint was said to tell both sides when to probe. Only the responder probes on it; the initiator does not read the hint and starts probing from its own punch_* settings when the answer arrives. Fixed in the reference, the port-advertisement doc and the discovery design doc. - All three event kinds were said to be signed with the node's identity key. The kind 21059 wrap is signed by a one-time key; only the seal inside it is signed with the identity key. - The FIPS advert example did not say that FIPS emits only the d, protocol, version and expiration tags and carries its signaling relays and STUN servers in the content as signalRelays and stunServers, present only with a udp:nat endpoint. Two tests pin the behaviour the docs now state: an answer whose inReplyTo is the offer's event id is refused as a session mismatch, and a signal's wrap is signed by a key other than the sender's while unwrapping recovers the sender's identity from the seal. Label the second msg2 DH as a repeat of es, not se. The IK msg2 code, the noise module doc, the wire-formats reference, the mesh-layer design doc and the msg2 diagram all called the second DH in msg2 `se`. Both sides actually pair the responder's static key with the initiator's ephemeral key, which is msg1's `es` pairing, so the step mixes the msg1 `es` value a second time and nothing computes Noise IK's `se`. Relabel it as `es` again everywhere, say in the code that it is not Noise's `se`, that it is the FMP v1 wire format and cannot change without breaking v1 peers, and that it leaves FMP v1 open to key compromise impersonation (FRC-1201 Section 9.3). Rename the two local bindings to `es2`. No behaviour or wire change. Correct the path MTU notification doc and the fips-mtu design doc. The PathMtuNotification doc said it is sent by a node that learns a path MTU from transit router feedback or ICMP Packet Too Big. It is sent by the session destination, echoing the forward-path minimum it saw in the envelope's path_mtu; a received MtuExceeded only updates local state. fips-mtu.md said TCP reports a per-connection MTU from TCP_MAXSEG. TCP reports the configured mtu and checks sends against it. It also said the TCP clamp mirror is tighter-only and never loosened. Entries are released on PathBroken (at most once a second per destination), peer removal and session timeout, discovery-learned entries expire with the coordinate cache TTL, and a link seed from a different transport replaces the stored value. Describe that lifecycle, and add a short section on the defences against forged signals: the 256-byte floors, the admission gate, corroboration against what this node has sent, and the rate-limited release. Correct transport comments and docs on TCP MTU, mDNS, BLE, Ethernet, Tor and UDP. Several transport comments and docs described behaviour the code does not have: - TCP: the config doc, configuration reference and transport-layer doc said the per-connection MTU comes from TCP_MAXSEG. TCP reports the configured mtu to FMP and checks sends against it. - mDNS: the module and supervisor comments said identity is proven by a Noise XX handshake. FMP here is Noise IK. Drop the note in the Nostr discovery doc that pointed at those comments. - BLE: the pubkey-exchange prefix doc said it is distinguished from FMP packets of version 1 or more. FMP is version 0, so the framer cannot tell them apart, as the caution below it already says. - Ethernet: the config doc said the runtime requires Linux. It is built for Linux (AF_PACKET) and macOS (BPF). - Tor: the config and transport docs, and the onion-service how-to, said control_port mode runs an inbound onion service. It only adds control-port monitoring to SOCKS5 outbound; directory mode is the only inbound mode. - UDP: the security reference said a UDP instance has no inbound exposure until bind_addr is set. A declared UDP instance listens on 0.0.0.0:2121 by default unless outbound_only is set. Correct the spanning tree docs on signatures, sequence, sizes, cache invalidation and flap dampening. - The TreeAnnounce signature was described as covering the entire message. It covers SHA-256 of the 48-byte declaration (node_addr, parent, sequence, timestamp); version and ancestry are not signed. - The sequence was said to increment on parent change. It also increments on becoming root and on an ancestry change under the same parent. - The TreeAnnounce size table added msg_type twice in the link-overhead column. The measured sizes are 168/264/328/488 bytes; note the 36-byte link overhead under the table. - Several design docs and a rate-limit comment said a parent change flushes the coordinate cache. The node drops only cached coordinates that route through it (parent switch or ancestry change) or sit under another root (becoming root), and both on parent loss or loop detection. - Flap dampening was described as engaging after more than flap_threshold switches in a sliding window. It engages when the count reaches flap_threshold, within a fixed window opened by the first switch. Correct the bloom filter frame size, triggers and propagation bound. - The wire-formats reference gave the v1 FilterAnnounce with link overhead as 1,072 bytes. It is 1,071 (1,035 payload plus 16 header, 4 timestamp and 16 tag), as the bloom design doc already says. - The bloom design doc listed local state changes (new identity, leaf-only dependent changes) as a send trigger. The identity does not change at runtime and nothing outside the tests adds a leaf-only dependent, so drop it. - The bloom and spanning tree docs bounded propagation at about 500 ms per hop. Announces leave only on the node's maintenance tick, so the bound is about one node.tick_interval_secs per hop (1 s by default); the 500 ms debounce is a floor below that. Correct forwarding docs on ranking, loop freedom and the coordinate cache. - The mesh-operation doc called link_cost an uncommon tie-breaker resolved by tree distance. Candidate ranking compares link cost first; tree distance and node_addr break ties. - Several docs and comments said the strict-progress check prevents routing loops even with stale coordinates. Each hop judges progress on its own cached coordinates for the destination, so the check does not guarantee loop freedom when hops hold different coordinates. Say that, and that the TTL bounds any loop. - The session-layer doc said FMP can fall back to bloom-only routing without cached coordinates. find_next_hop returns no next hop on a cache miss and the caller signals CoordsRequired. - The insert_coord_hint doc said a foreign-root destination can never be selected by the bloom step. A peer advertising coordinates under that root can be; the refusal stands for the reflector reason given in the same paragraph. - The find_next_hop and try_warm_coord_cache_ref doc blocks sat above the wrong functions, so rustdoc gave those two no docs and gave insert_coord_hint and warm_coord the wrong opening line. Move each block onto its function and fix the find_next_hop ranking wording. Correct FSP restart handling, error datagram sizes and XK labels. - The session-layer doc said an epoch mismatch on a later handshake marks a peer restart and triggers re-establishment. FSP exchanges the epoch but never reads it. A SessionSetup naming an established peer only arms a handshake beside the running session, and new keys are adopted at msg3 only when the authenticated static key matches. Describe that, and rename the status row accordingly. - Wire-format sizes: SessionDatagram + CoordsRequired is 74 bytes, not 70; SessionDatagram + PathBroken is 76 + 16n, not "70+"; PathBroken alone is 40 + 16n, not 35 + 16n; CoordsRequired with link overhead is 110, not 111; and the CoordsWarmup example is 238, not 239. State in the summary table which rows include link overhead. - Three FSP wire comments called the session handshake Noise IK. It is Noise XK, as the module's own phase table says. Correct the rationale for dropping our own looped-back request. The comment on the own-request check in classify_request, its test and the handle_lookup_response doc said that without the check the answer to our own lookup would be reverse-path forwarded to the peer that looped the request, and that recent_requests decides originator status. That stopped being true when classify_response began deciding originator status from the pending lookup before it reads the transit record. Say what the check still does: it stops the copy being forwarded a second time and keeps our own id out of the transit record, so a response after the lookup completes has nothing to be relayed by. Extend the test to pin that: once the lookup is accepted, a later response for the same id is unsolicited.
14 KiB
Enable Nostr-Mediated Discovery and NAT Traversal
Nostr-mediated discovery lets FIPS nodes find each other (and punch
through UDP NAT) using public Nostr relays as the signaling channel.
The feature ships in every stock packaging artifact but is off by
default — it activates when an operator sets
node.rendezvous.nostr.enabled: true. Default relay and STUN-server
lists ship in the config; both are optional overrides. See
../design/fips-nostr-discovery.md
for the design and rationale; see
../reference/configuration.md for the
full knob inventory.
The table was called node.discovery before v0.5.0; that spelling still
parses and logs one deprecation warning naming the move, so an existing
config keeps working (see
../reference/configuration.md).
Nostr discovery provides three independent capabilities. They can be enabled separately; most deployments end up using two or three of them together.
- Resolve a known peer's address by npub. Your daemon consumes adverts from the relays to look up the current network endpoint for a peer you have configured by npub. You don't have to know their IP / port / transport in advance.
- Publish your own endpoint so others can resolve you. Your daemon publishes a signed advert listing the transports it will accept connections on. Has two sub-shapes depending on your network topology: UDP (using NAT traversal if needed) or TCP. Running a Tor onion service is a separate deployment mode, covered in its own section below.
- Discover peers without prior configuration. Your daemon subscribes to all adverts on a chosen application namespace and treats any publisher as a connection candidate. The most permissive posture; useful for ambient mesh participation.
Each capability is covered below as one or more scenarios with the minimal YAML fragment that enables it. Only keys relevant to Nostr discovery are shown; surrounding node, transport, TUN, DNS, and peer configuration follows the usual shape.
All scenarios assume node.identity is set to a persistent key — an
ephemeral identity would invalidate any advert the moment the node
restarts. See persistent-identity.md for
the persistent-key setup.
For hand-held walkthroughs of each capability, see the resolve-peers-via-nostr, advertise-your-node, and open-discovery tutorials.
Capability 1: Resolve a known peer's address by npub
The node does not publish any advert of its own. It only consumes
adverts for peers it has explicitly listed with via_nostr: true.
This is the right shape for a client that wants Nostr-mediated
resolution without becoming a rendezvous target itself.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: false
policy: configured_only
transports:
udp:
bind_addr: "0.0.0.0:2121"
peers:
- npub: "npub1peer..."
alias: "remote-node"
via_nostr: true
connect_policy: auto_connect
What this achieves: dial endpoints for this peer are taken from
the peer's published Nostr advert. configured_only is the
default — it is shown here for clarity.
Note: You can also supply a static address alongside
via_nostr: true(for example, while testing, or as a known-good fallback if the advert is stale). Add anaddressesblock to the peer entry; static addresses are tried first on dial and Nostr-resolved endpoints are appended as additional candidates.
Capability 2: Publish your own endpoint so others can resolve you
This capability has three sub-scenarios depending on the network shape your node sits behind.
Sub-scenario 2a: UDP (using NAT traversal if needed)
The node has a public IP (or a stable port-forward) and binds UDP on
a known port. It publishes udp:host:port to the advert relays. Any
peer that knows this node's npub and has Nostr discovery enabled can
dial it without knowing the address out-of-band.
When UDP is wildcard-bound (0.0.0.0:2121, the default), the daemon
needs help knowing what IP to put in the advert. There are two ways:
STUN auto-discovery (public: true) or an explicit override
(external_addr). Both are first-class options; pick the one that
fits the deployment.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: true # ← STUN auto-discovery
Or, when the public IP is known up front (static residential IP, cloud Elastic IP behind 1:1 NAT, etc.):
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: true # ← required, master switch
external_addr: "203.0.113.45:2121" # ← explicit address
external_addr accepts a bare IP (combined with the bind port) or a
full host:port. public: true is the master switch that gates UDP
advertisement; inside that branch, the daemon picks the advertised
address in precedence order: explicit external_addr (no STUN
observation), a non-wildcard bind_addr, or STUN auto-discovery.
Setting external_addr alongside public: true skips STUN entirely
— there is no logging cross-check. If UDP is bound directly to a
public IP rather than to a wildcard, neither external_addr nor STUN
is needed — but advertise_on_nostr: true and public: true are
still both required for the daemon to publish the endpoint.
What this achieves: the node publishes a single
udp:<public-ip>:2121 endpoint to the three default advert relays
(wss://relay.damus.io, wss://nos.lol, wss://offchain.pub).
What the other side needs: either a static addresses entry for this
peer, or a peer entry with via_nostr: true and an empty (or
omitted) addresses list — the advert-resolved endpoint will be used
at dial time. Static and Nostr-resolved addresses can also be
combined: when both are present, static addresses are tried first and
Nostr-resolved endpoints are appended as fallback.
When the node is behind NAT
If this node doesn't have a stable public UDP endpoint, advertise
udp:nat. The daemon runs the STUN + offer/answer exchange with
the peer and punches through the NAT to establish a direct UDP
link. The peer can either have a public endpoint of its own or
also be behind NAT — both shapes work, as long as at least one
side has a NAT type compatible with hole-punching.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: true
dm_relays: # overrides the default three-relay
- "wss://relay.damus.io" # set with two for demonstration;
- "wss://nos.lol" # omit this block to keep the defaults
stun_servers:
- "stun:stun.l.google.com:19302"
- "stun:stun.cloudflare.com:3478"
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: false
peers:
- npub: "npub1peer..."
alias: "nat-peer"
addresses:
- transport: udp
addr: "nat"
via_nostr: true
connect_policy: auto_connect
What this achieves: the node publishes a udp:nat endpoint plus its
signaling relays in the advert. When either side initiates, an
encrypted offer is sealed to the peer's npub, a matching answer
comes back, and each side starts punching on its own schedule: the
responder at the time its answer names, the initiator on its local
punch_* settings once the answer arrives. On success,
the punch socket is adopted as an FMP UDP transport and Noise IK
proceeds normally.
Validation:
advertise_on_nostr: truewithpublic: falseon UDP requiresdm_relaysandstun_serversto be non-empty. Both ship with non-empty defaults (three relays and three STUN servers respectively), so the default config passes. The node fails startup only if the operator has explicitly emptied either list — audp:natadvert without signaling relays or STUN servers is unreachable by construction.
Hole-punching is best-effort. It works reliably when both sides are
full-cone or port-restricted NATs. Symmetric NAT on either side
typically defeats the punch — the public port a peer sees varies per
remote endpoint, so the address learned via STUN does not match the
mapping the peer actually needs. The punch attempt times out after
punch_duration_ms. udp:nat is the only NAT-traversal mechanism
in FIPS; when it can't succeed, there's no in-protocol substitute.
Being reachable then becomes a deployment-prerequisite question
rather than a transport question — a publicly reachable port (UDP
or TCP — both require the same kind of network resource) published
as a direct advert per Sub-scenario 2a or 2b.
Sub-scenario 2b: TCP
The node has a public IP (or a stable port-forward) and accepts
inbound TCP. It publishes tcp:host:port to the advert relays.
TCP endpoints exist to serve peers whose networks filter outbound UDP (corporate LANs, restrictive guest WiFi). NAT traversal does not apply: the publishing node is publicly reachable on TCP, and the dialing peer's network only needs to permit outbound TCP to the advertised port.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: true
transports:
tcp:
bind_addr: "0.0.0.0:8443"
advertise_on_nostr: true
external_addr: "203.0.113.45:8443"
external_addr is typically required on cloud setups (AWS Elastic
IP, etc.) where binding directly to the public IP returns
EADDRNOTAVAIL. When TCP is bound directly to a public IP, the
override is unnecessary.
What this achieves: the node publishes a tcp:<public-ip>:8443
endpoint to the advert relays. Peers with Nostr discovery enabled
dial by npub without out-of-band address exchange.
Tor onion node
A separate deployment mode for nodes that want anonymity and censorship-resistance properties on the data plane. Functionally this still uses Capability 2 (publishing an endpoint to advert relays) — the difference is that the published endpoint is a Tor hidden service rather than a public IP.
The node runs a Tor onion service in directory mode (Tor-managed
HiddenServiceDir) and advertises the .onion address. Peers dial
via their local Tor SOCKS5 proxy without ever knowing the onion
string out-of-band. For the Tor daemon side of this setup, including the inbound-mode
trade-offs and the torrc directives each requires, see
deploy-tor-onion.md.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: true
transports:
tor:
mode: directory
socks5_addr: "127.0.0.1:9050"
advertised_port: 8443
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
advertise_on_nostr: true
What this achieves: the node publishes a tor:<hash>.onion:8443
endpoint alongside any other advertised transports. The advert itself
is still published over clearnet WebSocket relays — Tor protects the
data plane, not the discovery plane. See the security and threat
model section in
../design/fips-nostr-discovery.md
for the trade-off and how to route relay traffic through Tor as well.
Capability 3: Discover peers without prior configuration
Under policy: open, any node that publishes an advert under the
same app namespace becomes a candidate. Discovered peers are queued
for connection attempts subject to open_discovery_max_pending.
node:
identity:
persistent: true
rendezvous:
nostr:
enabled: true
advertise: true
policy: open
open_discovery_max_pending: 64
app: "my-experiment.v1"
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: true
peers: []
What this achieves: peers are discovered entirely through ambient
advert traffic on the configured relays. Setting a non-default app
value (replacing fips-overlay-v1) scopes the discovery set to
participants who opt into the same experiment and avoids being joined
to unrelated overlays that happen to share the default namespace.
Scope warning: Open discovery is an admission-free mode. Any node that publishes on the same
appname and passes the peer-ACL check becomes a connection candidate. If you rely on peer ACLs for admission control, verify that list is set correctly before enabling this mode. See ../reference/security.md for the peer ACL format.
See also
- ../tutorials/resolve-peers-via-nostr.md — hand-held walkthrough of capability 1
- ../tutorials/advertise-your-node.md
— hand-held walkthrough of capability 2 (publish, plus a
short section on
udp:natNAT traversal) - ../tutorials/open-discovery.md — hand-held walkthrough of capability 3 (open ambient discovery, the additive policy: open mode)
- ../design/fips-nostr-discovery.md — discovery runtime design, security model
- ../reference/configuration.md —
full
node.rendezvous.nostr.*and per-transportadvertise_on_nostr/publictable - ../reference/nostr-events.md — Kind 37195 advert format, Kind 21059 traversal signaling, Kind 10050 inbox relay list
- deploy-tor-onion.md — Tor daemon-side setup for advertising onion endpoints