Breaking wire format change: DataPacket payloads inside the AEAD envelope now carry a 4-byte port header [src_port:2 LE][dst_port:2 LE] before the service payload. The receiver dispatches by destination port. Port multiplexing: - send_session_data() takes src_port/dst_port params, prepends port header - New send_ipv6_packet() compresses IPv6 header and sends on port 256 - Receive path dispatches DataPackets by port: port 256 decompresses IPv6 header from session context and delivers to TUN, unknown ports dropped - Port constants: FSP_PORT_HEADER_SIZE (4 bytes), FSP_PORT_IPV6_SHIM (256) IPv6 header compression: - New ipv6_shim module with compress_ipv6()/decompress_ipv6() pure functions - Strips src/dst addresses (32 bytes) and payload length (2 bytes) from each packet, preserving traffic class, flow label, next header, and hop limit as 6-byte residual fields - Addresses reconstructed from session context on receive side - Net savings: 29 bytes per packet (overhead 106 → 77 bytes) - FIPS_IPV6_OVERHEAD constant (77 bytes), effective_ipv6_mtu() updated - 16 unit tests for round-trip fidelity, field preservation, error cases Documentation: - fips-wire-formats: DataPacket port header, port registry, IPv6 shim format tables, updated encapsulation walkthrough and overhead budget - fips-ipv6-adapter: FIPS_IPV6_OVERHEAD (77 bytes), updated MTU numbers, TUN reader/writer flow with compression steps, impl status - fips-session-layer: port-based service dispatch section, data transfer description, impl status - fips-intro: IPv6 adapter as port 256 service, node architecture updated - fips-mesh-operation: packet size summary with compressed overhead - DataPacket doc updated with port header and dispatch model - session_wire.rs module doc: DataPacket Port Multiplexing section
27 KiB
FIPS Session Protocol (FSP)
The FIPS Session Protocol is the top protocol layer in the FIPS stack. It sits above the FIPS Mesh Protocol (FMP) and below applications (native FIPS API or IPv6 adapter). FSP provides end-to-end authenticated, encrypted datagram delivery between any two FIPS nodes, regardless of how many intermediate hops separate them.
Role
FSP manages end-to-end communication sessions between FIPS nodes identified by their public keys (npubs). Each session provides:
- End-to-end encryption: Payload confidentiality independent of how many intermediate nodes handle the traffic
- Mutual authentication: Both parties prove they control the private key for their claimed identity
- Replay protection: Counter-based nonces with sliding window, tolerant of UDP packet loss and reordering
- Transport independence: Sessions survive transport changes, route changes, and address changes — they are bound to npub identities, not to transport paths
FSP is a datagram session protocol. It provides encrypted datagrams, not reliable streams. There is no FIPS equivalent of TCP; if applications need reliability, ordering, or flow control, they provide it themselves (typically by running TCP over the FIPS IPv6 adapter).
Services Provided to Applications
Applications access the FIPS mesh through two interfaces, both served by FSP:
Native FIPS API
Applications address destinations directly by npub or public key. The FIPS stack resolves the destination's node_addr, establishes or reuses a session, encrypts the payload, and routes through FMP. No DNS involvement.
IPv6 Adapter
Unmodified IPv6 applications use a TUN device with fd00::/8 routing. A local
DNS service maps npub → IPv6 address and primes the identity cache. Packets
arriving at the TUN are translated to FIPS datagrams and routed through FSP.
See fips-ipv6-adapter.md for the IPv6 adaptation layer.
Port-Based Service Dispatch
FSP DataPackets carry a 4-byte port header (source and destination port) inside the AEAD envelope, enabling multiple services to share a single session. The IPv6 adapter runs on port 256; the native FIPS API and future services (gateways, application protocols) register on other ports. Port dispatch is internal to the FSP layer — services see only their payload, not the port header.
What Applications Get
- Authenticated datagram delivery: Each datagram is encrypted and authenticated with session keys bound to both parties' npubs
- Session transparency: Sessions are established on demand and maintained automatically. Applications send packets; FSP handles session setup, encryption, and teardown.
- Endpoint identity: Applications address destinations by npub. The FIPS address is the public key.
What Applications Do Not Get
- Reliability: Datagrams may be lost, duplicated, or delivered out of order. FSP provides no retransmission or ordering.
- Path MTU discovery: FSP does not signal MTU to applications. The IPv6 adapter handles MTU enforcement via ICMP Packet Too Big and TCP MSS clamping.
- Congestion control: FSP does not throttle traffic. Applications running TCP over IPv6 get TCP's congestion control; native API applications must manage their own sending rate.
Services Required from FMP
FSP treats FMP as a black box providing three services. FSP knows nothing about transports, transport addresses, links, peers, spanning trees, coordinates, bloom filters, hop counts, or network topology.
SessionDatagram Forwarding
FMP accepts a SessionDatagram (source node_addr, destination node_addr, TTL, path MTU, payload) and delivers it best-effort toward the destination. Delivery may traverse multiple hops, each with independent link encryption.
Error Signaling
FMP signals routing failures asynchronously:
- CoordsRequired: A transit node lacks the destination's tree coordinates. FSP responds by sending a standalone CoordsWarmup (0x14) message (rate-limited), re-initiating discovery, and resetting the coordinate warmup counter.
- PathBroken: Greedy routing reached a dead end. FSP responds by sending a standalone CoordsWarmup (rate-limited), re-discovering the destination's current coordinates, and resetting the warmup counter.
- MtuExceeded: A transit node cannot forward a SessionDatagram because
the packet exceeds the next-hop link MTU. FSP uses the reported bottleneck
MTU to adjust its session-layer path MTU estimate. MtuExceeded is the
reactive complement to the proactive
path_mtufield in SessionDatagram.
All three signals are generated by transit nodes (not the destination) and travel back to the source inside a new SessionDatagram. They are plaintext (not end-to-end encrypted) because transit nodes have no session with the source.
Local Delivery
When a SessionDatagram arrives with a destination node_addr matching the local node, FMP delivers it to FSP for session-layer processing.
Session Lifecycle
Session Establishment
Sessions are established on demand when the first datagram needs to be sent to a destination with no existing session.
FSP uses Noise XK for session key agreement (Noise Protocol Framework;
Perrin 2018). The initiator knows the destination's npub (required for
XK's pre-message s token); the responder learns the initiator's
identity from msg3 (not msg1, unlike IK at the link layer). This provides stronger initiator identity hiding
— the initiator's static key is encrypted under the established shared
secret rather than under only the responder's static key.
The handshake is a three-message flow carried in SessionSetup, SessionAck, and SessionMsg3:
- Initiator sends SessionSetup containing Noise XK msg1 (ephemeral key only) and both parties' tree coordinates
- Responder processes msg1, sends SessionAck containing Noise XK msg2 (ephemeral key + encrypted epoch) and both parties' tree coordinates. The responder transitions to AwaitingMsg3 state.
- Initiator processes msg2, sends SessionMsg3 containing the encrypted static key and encrypted epoch. Both parties derive identical symmetric session keys and the session is established.
Each side's epoch (an 8-byte random value generated at startup) is exchanged encrypted in msg2 and msg3. On subsequent handshakes, an epoch mismatch indicates the peer has restarted, triggering session re-establishment — similar to IKEv2's INITIAL_CONTACT notification for peer restart detection (RFC 7296 §2.4).
Packets that trigger session establishment are queued (with bounded buffer) and transmitted after the session is established.
Self-Bootstrapping
SessionSetup is self-bootstrapping for routing. It carries the source's and destination's tree coordinates in the clear (not inside the Noise payload). As the message transits intermediate nodes, each node caches these coordinates, warming the path for subsequent data packets that carry only addresses (no coordinates).
SessionAck carries both the responder's and initiator's coordinates back along the reverse path, warming caches in the other direction. This ensures return-path transit nodes can route even when the reverse path diverges from the forward path (e.g., after tree reconvergence).
Simultaneous Initiation
When both nodes attempt to establish a session simultaneously ("crossing hellos"), a deterministic tie-breaker resolves the conflict, mirroring IKEv2's simultaneous initiation resolution (RFC 7296 §2.8):
- If
local_node_addr < remote_node_addr: Continue as initiator, ignore incoming setup - If
local_node_addr > remote_node_addr: Abort own initiation, switch to responder role
This lowest-address-wins approach ensures exactly one handshake completes.
Data Transfer
Once established, sessions carry encrypted data using the FSP pipeline. Each encrypted message includes:
- A 12-byte cleartext header (used as AEAD AAD) containing the counter and flags (including the CP flag for coordinate cache warming)
- Optional cleartext coordinates when the CP flag is set
- An AEAD-encrypted payload containing a 6-byte inner header (session-relative timestamp, message type, inner flags) followed by DataPacket content: a 4-byte port header (src_port, dst_port) and the service payload. The receiver dispatches by destination port to the registered service handler.
Session Idle Timeout
Sessions that see no traffic for a configurable duration (default 90s) are torn down. When traffic resumes, a new session is established automatically.
Only DataPacket (message type 0x10) send/receive and session establishment
reset last_activity. MMP traffic — SenderReport, ReceiverReport, and
PathMtuNotification — does not reset the idle timer. This means a session
carrying only MMP reports and no application data will still tear down after
node.session.idle_timeout_secs. MMP reports continue flowing until the
session is torn down, providing final measurement data for the teardown log.
The idle timeout is deliberately shorter than the coordinate cache TTL (300s). This ordering ensures that when traffic stops and the session tears down, the transit node coordinate caches are still warm when a new session is established. The fresh SessionSetup re-warms the caches, maintaining routing continuity.
Session Independence from Transport
Sessions exist above the routing layer and are bound to npub identities, not transport addresses or routing paths — following WireGuard's approach (Donenfeld 2017) of binding cryptographic sessions to identity keys rather than network addresses. A session survives:
- Transport failover (UDP → Ethernet → back to UDP)
- Route changes (different intermediate hops)
- Transport address changes (IP address or port changes)
- Topology changes (direct peer becomes multi-hop or vice versa)
End-to-End Encryption
Noise XK Pattern
FSP uses Noise XK for session encryption, distinct from the Noise IK
pattern used at the link layer. The full Noise descriptor is
Noise_XK_secp256k1_ChaChaPoly_SHA256.
The XK pattern (pre-message: ← s):
- msg1 (
→ e, es): Initiator sends ephemeral key only. The initiator's static identity is not revealed in this message. - msg2 (
← e, ee): Responder sends ephemeral key and encrypted epoch. - msg3 (
→ s, se): Initiator sends encrypted static key and encrypted epoch. Both parties now share identical session keys.
After the handshake, Noise produces two directional symmetric keys
(send_key, recv_key) used with ChaCha20-Poly1305 for all subsequent data.
The XK pattern requires the initiator to know the responder's static key
in advance (the ← s pre-message), which is satisfied by the discovery
or DNS lookup that precedes session establishment. In exchange, XK
provides stronger initiator identity protection than IK — the initiator's
static key is encrypted under the full shared secret (after three DH
operations) rather than under only the responder's static key.
Cryptographic Primitives
| Component | Choice | Notes |
|---|---|---|
| Curve | secp256k1 | Nostr-native |
| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman |
| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 |
| Hash | SHA-256 | Nostr-native |
| Key derivation | HKDF-SHA256 | Standard Noise KDF |
These choices prioritize compatibility with the Nostr cryptographic stack — secp256k1 + ChaCha20-Poly1305 + SHA-256 aligns with the NIP-44 encrypted messaging standard.
secp256k1 Parity Normalization
Nostr npubs encode x-only public keys (32 bytes, no y-coordinate parity). The Noise XK pre-message mixes the responder's static key as a 33-byte compressed key, and the default secp256k1 ECDH hash includes a parity-dependent version byte.
Both operations are normalized to be parity-independent: the pre-message hash
uses even parity (0x02 prefix), and ECDH hashes only the x-coordinate of the
result point. This ensures handshakes succeed regardless of the responder's
actual key parity.
Privacy Note
Noise XK provides stronger initiator identity protection than IK. In XK, the initiator's static key is encrypted in msg3 under the full shared secret (derived from three DH operations), so an attacker who compromises only the responder's nsec cannot decrypt the initiator's identity from captured handshake messages (they would also need the responder's ephemeral key). This is the primary reason FSP uses XK rather than IK — session-layer traffic traverses untrusted intermediate nodes, making initiator identity protection more valuable than at the link layer.
Data Packet Authentication
FSP uses AEAD authentication only — no per-packet signatures. The Noise handshake binds session keys to both parties' static keys, so only holders of the corresponding nsecs can derive the session keys. This provides implicit authentication for every packet, matching WireGuard and Lightning's approach.
Forward Secrecy
Ephemeral keys in the Noise handshake provide forward secrecy — a standard
property of Noise handshake patterns that include ephemeral key exchange.
Compromise of static keys (nsec) does not reveal past session keys, because
session keys are derived in part from ephemeral-ephemeral DH (ee), and
ephemeral keys are discarded after the handshake.
Replay Protection
FSP uses explicit 8-byte counters on the wire for replay protection, adapted from the DTLS anti-replay mechanism (Rescorla & Modadugu, RFC 6347). Each side maintains a monotonically increasing send counter, included in the 12-byte cleartext header of every encrypted message. The receiver maintains a sliding window (2048-entry bitmap) tracking which counters have been seen.
This design is critical for operation over unreliable transports. Under UDP
packet loss or reordering, implicit nonce counters (where the receiver
increments on each decrypt attempt) would desynchronize permanently — a failed
decrypt() increments the nonce, and the desync grows with each lost packet.
Explicit counters with a sliding bitmap window — the standard DTLS approach —
allow the receiver to decrypt any packet independently, regardless of what
packets were lost or reordered.
The same ReplayWindow and decrypt_with_replay_check() implementation is
used at both the link and session layers.
Hybrid Coordinate Warmup Strategy
Session establishment (SessionSetup/SessionAck) warms transit node coordinate caches along the path. But coordinate caches have a finite TTL (default 300s), and entries may be evicted under memory pressure. When a transit node's cache entry expires, it cannot forward data packets (which carry only addresses, not coordinates) and sends a CoordsRequired error.
FSP uses a hybrid warmup strategy combining proactive piggybacking with reactive standalone messages to keep transit caches populated (Yggdrasil uses a similar approach of embedding coordinates in session traffic to warm transit node caches):
Proactive Warmup Phase
After session establishment, the first N data packets (configurable, default 5) per session attempt to piggyback source and destination coordinates via the CP flag in the FSP common prefix. The coordinates appear in cleartext between the 12-byte header and the ciphertext, allowing transit nodes to cache them without decryption.
If piggybacking coordinates would cause the total packet to exceed the transport MTU, the source sends a standalone CoordsWarmup message (0x14) first, followed by the data packet without the CP flag. This ensures transit caches are warmed even when data packets are near the MTU limit.
CoordsWarmup Message (0x14)
CoordsWarmup is an encrypted FSP message (phase 0x0) with the CP flag set and inner msg_type 0x14. It carries cleartext source and destination coordinates between the FSP header and the AEAD ciphertext, using the same format as CP-flagged data packets. The inner body is empty — the message exists solely to deliver coordinates to transit nodes.
Transit nodes extract coordinates via the existing try_warm_coord_cache()
code path with zero changes to the transit forwarding logic. CoordsWarmup is
indistinguishable from any other CP-flagged message at the transit layer.
On the wire, a CoordsWarmup message consists of the standard 12-byte FSP header (used as AEAD AAD) with ver=0, phase=0, flags=CP, followed by cleartext source and destination coordinates (same encoding as any CP-flagged packet), followed by 22 bytes of AEAD ciphertext (6-byte inner header and 16-byte Poly1305 tag). The total FSP payload is 12 bytes of header, the coordinate data, and 22 bytes of ciphertext.
Steady State
After the warmup count is reached, FSP clears the CP flag and sends minimal data packets (12-byte header + ciphertext). Transit nodes serve from their coordinate caches.
Reactive Re-Warm
When FSP receives a CoordsRequired or PathBroken signal:
- A standalone CoordsWarmup message is sent immediately to re-warm transit
caches, rate-limited at one per destination per configurable interval
(default 2000ms,
node.session.coords_response_interval_ms) - The warmup counter resets — subsequent data packets piggyback coordinates again when possible (or send additional CoordsWarmup messages when the data packet would exceed the MTU)
- A new LookupRequest may be initiated to rediscover the destination's current coordinates (always for PathBroken; optionally for CoordsRequired)
- When the LookupResponse arrives for an established session, the warmup counter resets again (handling the timing gap where warmup packets might fire before transit caches are repopulated by discovery)
The source-side rate limiting prevents amplification: at most one standalone
CoordsWarmup response per destination per coords_response_interval_ms
(default 2s). This is independent of the transit-node rate limiting on error
signal generation (100ms per destination).
Warmup State Machine
Identity Cache
The identity cache maps FIPS address prefix (15 bytes, the fd00::/8 IPv6
address minus the fd prefix) to (NodeAddr, PublicKey). This cache is
needed only when using the IPv6 adapter — the native FIPS API provides the
public key directly.
The mapping is deterministic (derived from the public key via SHA-256) and never becomes stale. The cache uses LRU-only eviction bounded by a configurable size (default 10K entries). There is no TTL — entries are evicted only when the cache is full and space is needed for a new entry.
Cache population mechanisms:
- DNS lookup: The primary path. Resolving
npub1xxx...xxx.fipsderives the IPv6 address and populates the identity cache. - Inbound traffic: Authenticated sessions from other nodes populate the cache with their identity information.
Coordinate Cache
The coordinate cache maps NodeAddr → TreeCoordinate and is the critical
data structure that enables efficient multi-hop routing. Without cached
coordinates for a destination, FMP cannot make forwarding decisions and must
either fall back to bloom-filter-only routing or signal CoordsRequired.
Unified Cache
The coordinate cache is a single unified cache (merged from previously separate coord_cache and route_cache). All coordinate sources — SessionSetup transit, CP-flagged data packets, LookupResponse — write to the same cache.
Eviction Policy
- TTL-based expiration: Entries expire after a configurable duration (default 300s)
- Refresh on use: Active routing through a cache entry resets its TTL, keeping hot entries alive
- LRU eviction: When the cache is full, least recently used entries are evicted first
- Flush on parent change: When the local node's tree parent changes, the entire coordinate cache is flushed. Tree parent changes mean the node's own coordinates have changed, making cached coordinates for other nodes potentially stale for routing purposes.
Timer Ordering
Cache and session timers are ordered so that idle sessions tear down before transit caches expire:
| Timer | Default | Purpose |
|---|---|---|
| Session idle timeout | 90s | Tear down unused sessions |
| Coordinate cache TTL | 300s | Expire stale coordinates |
| DNS TTL | 300s | Expire DNS resolutions |
When traffic stops: the session tears down at 90s. When traffic resumes: DNS re-resolves the identity, a fresh SessionSetup carries coordinates, and transit node caches (still within their 300s TTL) are re-warmed.
Session-Layer MMP
Each established session runs its own Metrics Measurement Protocol instance, providing end-to-end quality metrics independent of the number of hops.
Relationship to Link-Layer MMP
Session-layer MMP uses the same report wire format (SenderReport 0x11, ReceiverReport 0x12) and identical algorithms as link-layer MMP, but with two key differences:
- End-to-end routing: Session reports are encrypted and forwarded through every transit link. A 3-hop session generates report traffic on all 3 links, making bandwidth cost proportional to path length.
- Independent configuration: The
node.session_mmp.*parameters are separate fromnode.mmp.*, allowing operators to run a lighter mode for sessions (e.g., Lightweight) while keeping Full mode on links.
Metrics
The same metrics as link-layer MMP: SRTT, loss rate, jitter, goodput, OWD trend, ETX, and dual EWMA trends. Session MMP additionally tracks observed path MTU.
Report Intervals
Session-layer report intervals are higher than link-layer to account for bandwidth cost: clamped to [500ms, 10s] with a cold-start interval of 1s (vs. link-layer [100ms, 2s] with 500ms cold-start).
Path MTU Tracking
PathMtuNotification (message type 0x13) provides end-to-end path MTU
feedback, adapting RFC 1191 Path MTU Discovery for overlay networks — the
transit-node min() propagation replaces ICMP Packet Too Big:
- The source sets
path_mtuin each SessionDatagram envelope to its outbound link MTU. - Each transit node applies
min(current, transport.link_mtu(addr))before forwarding. - The destination receives the forward-path minimum and sends a PathMtuNotification (2-byte body: u16 LE path_mtu) back to the source.
- The source applies the notification with hysteresis:
- Decrease: immediate (take lower value).
- Increase: requires 3 consecutive higher-value notifications spanning at least 2 × notification interval.
- Notifications are sent on first measurement, on any decrease, and
periodically at
max(10s, 5 × SRTT).
Send Failure Backoff
When a session MMP report cannot be delivered (destination unreachable, no route), the sender applies exponential backoff to the probe interval (a standard distributed systems pattern for transient failure handling):
- Each consecutive failure doubles the interval: 2x, 4x, 8x, 16x, 32x
- Backoff caps at 32x the base interval (5 consecutive failures)
- A successful send resets to the normal SRTT-based interval
- Debug logging is suppressed after 3 consecutive failures; a summary is logged when the destination becomes reachable again
This prevents wasted CPU and log noise when a session's remote endpoint has departed the network but the local session has not yet timed out.
Idle Timeout Interaction
MMP reports (SenderReport, ReceiverReport) and PathMtuNotification do not
reset the session idle timer. Only application data (DataPacket, type 0x10)
resets last_activity. This ensures sessions with no application traffic
tear down after node.session.idle_timeout_secs (default 90s), while MMP
continues providing measurement data up to the teardown moment.
Operator Logging
Session metrics are logged at info level (configurable via
node.session_mmp.log_interval_secs, default 30s):
MMP session metrics session=npub1tdwa...84le rtt=4.3ms loss=0.6% jitter=0.2ms goodput=71.3MB/s mtu=1472 tx_pkts=1234 rx_pkts=5678
Implementation Status
| Feature | Status |
|---|---|
| Session establishment (Noise XK) | Implemented |
| Peer restart detection (epoch exchange) | Implemented |
| MtuExceeded handling | Implemented |
| End-to-end encryption (ChaCha20-Poly1305) | Implemented |
| Explicit counter replay protection | Implemented |
| Hybrid coordinate warmup (CP + CoordsWarmup) | Implemented |
| FSP wire format (prefix, AAD, inner header) | Implemented |
| Session-layer MMP (with send-failure backoff) | Implemented |
| Identity cache (LRU-only) | Implemented |
| Coordinate cache (unified, TTL + refresh) | Implemented |
| Session idle timeout | Implemented |
| CoordsRequired handling | Implemented |
| PathBroken handling | Implemented |
| Simultaneous initiation tie-breaker | Implemented |
| Flush coord cache on parent change | Implemented |
| Rekey | Implemented |
| Port-based service multiplexing | Implemented |
| IPv6 header compression (shim on port 256) | Implemented |
| Path MTU tracking (FMP SessionDatagram field) | Implemented |
| Path MTU notification (end-to-end echo) | Implemented |
References
FIPS Internal Documentation
- fips-intro.md — Protocol overview and architecture
- fips-mesh-layer.md — FMP specification (below FSP)
- fips-ipv6-adapter.md — IPv6 adaptation layer (above FSP)
- fips-mesh-operation.md — Routing, discovery, and error recovery
- fips-wire-formats.md — Wire format reference for all session message types
External References
-
Perrin, T. "The Noise Protocol Framework". Revision 34, 2018. Framework for building crypto protocols using Diffie-Hellman key agreement and AEAD ciphers. FSP uses the XK handshake pattern.
-
Donenfeld, J.A. "WireGuard: Next Generation Kernel Network Tunnel". NDSS 2017. Transport-independent cryptographic sessions bound to identity keys rather than network addresses; AEAD-only authentication model.
-
Rescorla, E., Modadugu, N. RFC 6347: "Datagram Transport Layer Security Version 1.2". 2012. Explicit sequence numbers with sliding bitmap window for replay protection over unreliable transports.
-
Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., Kivinen, T. RFC 7296: "Internet Key Exchange Protocol Version 2 (IKEv2)". 2014. Simultaneous initiation resolution (§2.8) and INITIAL_CONTACT peer restart detection (§2.4).
-
Mogul, J., Deering, S. RFC 1191: "Path MTU Discovery". 1990. End-to-end path MTU discovery; FSP adapts this for overlay networks using transit-node min() propagation.
-
Yggdrasil Network. Coordinate-based overlay routing with session traffic used to warm transit node coordinate caches.