mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-22 07:48:26 +00:00
Add FIPS software architecture document
Create comprehensive architecture document covering: - Core entities: Node, Transport, Link, Peer hierarchy - Transport vs Link distinction (interface vs connection) - State machines for Transport, Link, and Peer lifecycles - Reference transport types: UDP/IP, Ethernet, WiFi, Tor - Spanning tree and Bloom filter state requirements - Self-healing protocol design (no ack/retry needed) - Leaf-only operation for constrained devices - Comprehensive sysctl-style configuration reference Also add architecture review document capturing identified issues to resolve before implementation begins.
This commit is contained in:
Johnathan Corgan
285
docs/design/fips-architecture-review.md
Normal file
285
docs/design/fips-architecture-review.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# FIPS Architecture Document Review
|
||||
|
||||
**Date**: 2025-01-29
|
||||
**Document reviewed**: fips-architecture.md
|
||||
|
||||
This file captures critique and open issues identified during architecture review.
|
||||
Items should be addressed before implementation begins or marked as intentionally
|
||||
deferred.
|
||||
|
||||
---
|
||||
|
||||
## High Priority (Implementation Blockers)
|
||||
|
||||
### 1. Coordinate Ordering Inconsistency
|
||||
|
||||
**Issue**: Architecture says `[self, parent, ..., root]` but spanning-tree-dynamics.md
|
||||
sometimes uses `[root, ..., self]` ordering.
|
||||
|
||||
**Location**: fips-architecture.md line 165, spanning-tree-dynamics.md line 346-349
|
||||
|
||||
**Resolution needed**: Standardize on one ordering across all documents.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 2. Authentication Protocol Not Referenced
|
||||
|
||||
**Issue**: Architecture references "FIPS auth handshake" but doesn't define it or
|
||||
reference where it's defined.
|
||||
|
||||
**Location**: fips-architecture.md line 285
|
||||
|
||||
**Resolution needed**: Add reference to fips-design.md lines 115-205, or include
|
||||
summary in architecture document.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 3. Concurrency Model Unspecified
|
||||
|
||||
**Issue**: Document doesn't specify whether implementation should be:
|
||||
- Single-threaded async
|
||||
- Multi-threaded with message passing
|
||||
- How state machines are driven (polling, callbacks, async/await)
|
||||
- Whether transports run in separate threads/tasks
|
||||
|
||||
**Location**: Entire document
|
||||
|
||||
**Resolution needed**: Add "Concurrency Model" section specifying the expected
|
||||
runtime architecture.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 4. Keepalive Message Format Unspecified
|
||||
|
||||
**Issue**: Configuration specifies `peer.keepalive.interval` but no message type
|
||||
is defined for keepalives. RTT measurement mechanism is unclear.
|
||||
|
||||
**Location**: fips-architecture.md line 762
|
||||
|
||||
**Resolution needed**: Clarify whether Dummy (0x00) message from fips-design.md
|
||||
is used, and how RTT is measured (request/response probing or passive observation).
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 5. Cache Naming Confusion
|
||||
|
||||
**Issue**: Multiple cache references that may or may not be the same thing:
|
||||
- `coord_cache: CoordCache` on Node (line 35)
|
||||
- `coord_cache: HashMap<Ipv6Addr, CachedCoords>` on TreeState (line 167)
|
||||
- `discovery.cache.max_entries` configuration (line 745)
|
||||
- `session.cache.max_entries` configuration (line 751)
|
||||
|
||||
**Resolution needed**: Clarify whether these are the same cache or different caches.
|
||||
If different, explain the distinction.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
## Medium Priority (Design Clarity)
|
||||
|
||||
### 6. Peer vs Link Relationship
|
||||
|
||||
**Issue**: Document states "one-to-one mapping between peers and links" (line 148)
|
||||
but Links can exist before authentication completes, meaning Links exist without
|
||||
Peers temporarily.
|
||||
|
||||
**Resolution needed**: Clarify the relationship lifecycle:
|
||||
- Link created on connection (before auth)
|
||||
- Peer created on successful auth
|
||||
- Peer always references exactly one Link
|
||||
- Link can exist without Peer (during auth or after auth failure)
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 7. Error Handling Strategy Absent
|
||||
|
||||
**Issue**: No systematic error handling definitions:
|
||||
- Signature verification failure handling
|
||||
- Malformed packet handling
|
||||
- Error codes and response formats
|
||||
- Logging/reporting strategy
|
||||
|
||||
**Resolution needed**: Add "Error Handling" section or reference to protocol spec.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 8. Memory Bounds Missing
|
||||
|
||||
**Issue**: Configuration specifies cache sizes but no limits for:
|
||||
- Maximum peers
|
||||
- Maximum transports
|
||||
- Maximum pending operations/queues
|
||||
- Overall memory budget
|
||||
|
||||
**Resolution needed**: Add resource limit configuration or document that these
|
||||
are implementation-defined.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 9. Timer Management at Scale
|
||||
|
||||
**Issue**: Multiple timers per peer (auth timeout, keepalive, reconnect delay,
|
||||
filter debounce). For 100 peers, this could be hundreds of timers.
|
||||
|
||||
**Resolution needed**: Note that timer wheel or hierarchical timing wheel may
|
||||
be needed for efficient implementation at scale.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 10. Initialization/Shutdown Sequences
|
||||
|
||||
**Issue**: No startup sequence defined:
|
||||
- Transport start order
|
||||
- TUN interface initialization relative to peering
|
||||
- Behavior when configured peers are unreachable
|
||||
- When node is "ready" to route
|
||||
|
||||
No shutdown procedure defined:
|
||||
- Should nodes announce departure?
|
||||
- Session termination
|
||||
- TUN interface cleanup
|
||||
|
||||
**Resolution needed**: Add "Initialization" and "Shutdown" sections.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
## Lower Priority (Polish)
|
||||
|
||||
### 11. Configuration Validation Rules
|
||||
|
||||
**Issue**: No validation rules specified:
|
||||
- What if `filter.size` is not a power of 2?
|
||||
- What if `peer.keepalive.timeout` < `peer.keepalive.interval`?
|
||||
- What if `timeout.adaptive.min` > `timeout.adaptive.max`?
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 12. Default Value Rationales
|
||||
|
||||
**Issue**: Several defaults lack rationale:
|
||||
- `filter.scope = 2`: Why 2?
|
||||
- `tree.parent.hold_time = 10s`: Why 10 seconds?
|
||||
- `filter.stale.threshold = 300s`: Based on what analysis?
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 13. Terminology Inconsistency Across Documents
|
||||
|
||||
**Issue**: "Link" and "Transport" meanings differ between documents:
|
||||
- fips-links.md uses "link" to mean underlying transport protocol
|
||||
- fips-architecture.md uses "Link" to mean connection over Transport
|
||||
- fips-design.md has `FipsLink` trait which is really a transport interface
|
||||
|
||||
**Resolution needed**: Standardize terminology or add glossary.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 14. Gateway/Subnet Routing Details Incomplete
|
||||
|
||||
**Issue**: Document mentions `leaf_dependents` but doesn't explain:
|
||||
- How a node becomes a gateway for a subnet
|
||||
- How gateway prefixes are advertised in Bloom filters
|
||||
- Configuration for gateway mode
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
### 15. DiscoveredPeer Hint Field Mismatch
|
||||
|
||||
**Issue**: Transport trait shows `discover() -> Result<Vec<DiscoveredPeer>>` but
|
||||
event shows `DiscoveredPeer { transport_id, addr, hint: Option<PublicKey> }`.
|
||||
The discover() return type doesn't show the hint field.
|
||||
|
||||
**Resolution needed**: Align trait signature with event structure.
|
||||
|
||||
**Status**: OPEN
|
||||
|
||||
---
|
||||
|
||||
## Edge Cases Not Addressed
|
||||
|
||||
### Root Node Failure
|
||||
- How long until new root is elected?
|
||||
- How are in-flight sessions affected?
|
||||
- Any proactive root backup mechanism?
|
||||
|
||||
### Rapid Peer Churn
|
||||
- Debounce (500ms) may not be sufficient
|
||||
- Could lead to Bloom filter oscillation or announcement storms
|
||||
|
||||
### Network Partition Healing
|
||||
- What happens to sessions that existed in both partitions?
|
||||
- How do routers with stale cache entries recover?
|
||||
|
||||
### Transport Startup Timing
|
||||
- If Tor takes 180s but UDP is instant, what's node status during window?
|
||||
- Are Tor-only peers unreachable during Tor bootstrap?
|
||||
|
||||
### MTU Mismatch Across Path
|
||||
- Path includes links with different MTUs (Ethernet 1500 vs LoRa 222)
|
||||
- No path MTU discovery or fragmentation strategy defined
|
||||
|
||||
### Clock Skew
|
||||
- TreeAnnounce timestamps with significant clock skew
|
||||
- 5-minute tolerance in auth (fips-design.md) should be reflected here
|
||||
|
||||
### Resource Exhaustion Attacks
|
||||
- Memory exhaustion via many fake peers
|
||||
- CPU exhaustion via signature verification flooding
|
||||
- Bandwidth exhaustion via Bloom filter spam
|
||||
|
||||
### Conflicting Peer Configurations
|
||||
- Two configured peers with same npub but different addresses
|
||||
- Discovered peer conflicts with configured peer
|
||||
|
||||
### TUN Interface Unavailable
|
||||
- TUN creation fails (permissions, kernel module)
|
||||
- Can node run in "relay-only" mode without TUN?
|
||||
|
||||
---
|
||||
|
||||
## Resolution Tracking
|
||||
|
||||
| # | Issue | Priority | Status | Resolution |
|
||||
|---|-------|----------|--------|------------|
|
||||
| 1 | Coordinate ordering | High | OPEN | |
|
||||
| 2 | Auth protocol reference | High | OPEN | |
|
||||
| 3 | Concurrency model | High | OPEN | |
|
||||
| 4 | Keepalive format | High | OPEN | |
|
||||
| 5 | Cache naming | High | OPEN | |
|
||||
| 6 | Peer/Link relationship | Medium | OPEN | |
|
||||
| 7 | Error handling | Medium | OPEN | |
|
||||
| 8 | Memory bounds | Medium | OPEN | |
|
||||
| 9 | Timer management | Medium | OPEN | |
|
||||
| 10 | Init/shutdown | Medium | OPEN | |
|
||||
| 11 | Config validation | Low | OPEN | |
|
||||
| 12 | Default rationales | Low | OPEN | |
|
||||
| 13 | Terminology | Low | OPEN | |
|
||||
| 14 | Gateway details | Low | OPEN | |
|
||||
| 15 | DiscoveredPeer hint | Low | OPEN | |
|
||||
841
docs/design/fips-architecture.md
Normal file
841
docs/design/fips-architecture.md
Normal file
@@ -0,0 +1,841 @@
|
||||
# FIPS Software Architecture
|
||||
|
||||
**Status**: Design Draft
|
||||
|
||||
This document describes the software architecture for a FIPS node implementation,
|
||||
covering core entities, state machines, transport abstractions, and configuration.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
FIPS is a Layer 3 mesh routing protocol that provides IPv6 connectivity over
|
||||
heterogeneous link types. A FIPS node exposes a TUN interface to local applications,
|
||||
routes packets via a spanning tree topology, and uses Bloom filters for efficient
|
||||
reachability lookup.
|
||||
|
||||
The architecture is event-driven with multiple focused state machines rather than
|
||||
a single monolithic event handler. Control protocols are designed for eventual
|
||||
consistency, tolerating packet loss without requiring acknowledgment/retry machinery.
|
||||
|
||||
---
|
||||
|
||||
## Core Entities
|
||||
|
||||
### Node
|
||||
|
||||
The top-level entity representing a running FIPS instance.
|
||||
|
||||
```
|
||||
Node
|
||||
├── identity: Identity // cryptographic identity (npub/nsec)
|
||||
├── config: Config // loaded configuration
|
||||
├── tun: TunInterface // IPv6 interface to local applications
|
||||
├── tree_state: TreeState // local view of spanning tree
|
||||
├── coord_cache: CoordCache // address → coordinates for routing
|
||||
├── transports: HashMap<TransportId, Transport>
|
||||
├── links: HashMap<LinkId, Link>
|
||||
└── peers: HashMap<NodeId, Peer>
|
||||
```
|
||||
|
||||
### Identity
|
||||
|
||||
Cryptographic identity using Nostr keys (secp256k1).
|
||||
|
||||
```
|
||||
Identity
|
||||
├── npub: PublicKey // public key (bech32: npub1...)
|
||||
├── nsec: SecretKey // secret key (bech32: nsec1...)
|
||||
├── node_id: NodeId // SHA-256(npub), 32 bytes
|
||||
└── address: FipsAddress // IPv6 ULA derived from node_id (fd::/8)
|
||||
```
|
||||
|
||||
`NodeId` is the routing identifier, derived deterministically from `npub`.
|
||||
Transport addresses and FIPS identity are fully decoupled.
|
||||
|
||||
### Transport
|
||||
|
||||
A physical or logical interface over which links can be established.
|
||||
|
||||
```
|
||||
Transport (trait)
|
||||
├── transport_id: TransportId
|
||||
├── transport_type: TransportType
|
||||
├── config: TransportConfig
|
||||
├── state: TransportState
|
||||
├── mtu: u16
|
||||
│
|
||||
├── start() -> Result<()>
|
||||
├── stop() -> Result<()>
|
||||
├── send(addr: &TransportAddr, data: &[u8]) -> Result<()>
|
||||
├── recv() -> Result<(TransportAddr, Vec<u8>)>
|
||||
└── discover() -> Result<Vec<DiscoveredPeer>>
|
||||
```
|
||||
|
||||
**Transport metadata (static per type):**
|
||||
|
||||
```
|
||||
TransportType
|
||||
├── name: &'static str // "udp", "ethernet", "wifi", "tor"
|
||||
├── connection_oriented: bool // requires link establishment?
|
||||
└── reliable: bool // delivery guaranteed by transport?
|
||||
```
|
||||
|
||||
Transports handle framing, fragmentation, and any transport-layer encryption
|
||||
internally. The FIPS routing layer sees only FIPS packets.
|
||||
|
||||
### Link
|
||||
|
||||
A communication channel to a specific remote endpoint over a transport.
|
||||
|
||||
```
|
||||
Link
|
||||
├── link_id: LinkId
|
||||
├── transport_id: TransportId
|
||||
├── remote_addr: TransportAddr // opaque, transport-specific
|
||||
├── direction: Inbound | Outbound
|
||||
├── state: LinkState // trivial for connectionless, real for Tor
|
||||
├── base_rtt: Duration // hint from transport type
|
||||
└── io: ... // connection handles for connection-oriented
|
||||
```
|
||||
|
||||
For connectionless transports (UDP, Ethernet, WiFi), links are lightweight—just
|
||||
`(transport_id, remote_addr)` with implicit "always connected" state.
|
||||
|
||||
For connection-oriented transports (Tor), links track real connection state and
|
||||
hold I/O handles.
|
||||
|
||||
**Link statistics (measured):**
|
||||
|
||||
```
|
||||
LinkStats
|
||||
├── packets_sent: u64
|
||||
├── packets_recv: u64
|
||||
├── bytes_sent: u64
|
||||
├── bytes_recv: u64
|
||||
├── last_recv: Timestamp
|
||||
├── rtt_estimate: Duration // measured from probes
|
||||
├── loss_rate: f32 // observed (meaningful for unreliable)
|
||||
└── throughput_estimate: u64 // bytes/sec observed
|
||||
```
|
||||
|
||||
### Peer
|
||||
|
||||
An authenticated remote FIPS node, reachable via a link.
|
||||
|
||||
```
|
||||
Peer
|
||||
├── node_id: NodeId // routing identity
|
||||
├── npub: PublicKey // cryptographic identity
|
||||
├── link_id: LinkId // which link reaches this peer
|
||||
├── state: PeerState // lifecycle state
|
||||
│
|
||||
│ // Spanning tree
|
||||
├── declaration: ParentDeclaration // their latest
|
||||
├── ancestry: Vec<NodeId> // their path to root
|
||||
│
|
||||
│ // Bloom filter (inbound—what's reachable through them)
|
||||
├── inbound_filter: BloomFilter
|
||||
├── filter_sequence: u64
|
||||
├── filter_ttl: u8
|
||||
├── filter_received_at: Timestamp
|
||||
├── pending_filter_update: bool // we owe them an update
|
||||
│
|
||||
│ // Statistics
|
||||
└── link_stats: LinkStats
|
||||
```
|
||||
|
||||
There is a one-to-one mapping between peers and links. If the same remote node
|
||||
is reachable via multiple transports, that would be multiple Peer entries (though
|
||||
for initial implementation, we assume single transport per peer).
|
||||
|
||||
---
|
||||
|
||||
## Spanning Tree State
|
||||
|
||||
### Per-Node State
|
||||
|
||||
```
|
||||
TreeState
|
||||
├── my_declaration: ParentDeclaration
|
||||
│ ├── node_id: NodeId
|
||||
│ ├── parent_id: NodeId // self if root candidate
|
||||
│ ├── sequence: u64 // monotonic
|
||||
│ └── signature: Signature
|
||||
├── my_coords: Vec<NodeId> // [self, parent, grandparent, ..., root]
|
||||
├── root: NodeId // elected root (smallest reachable node_id)
|
||||
└── coord_cache: HashMap<Ipv6Addr, CachedCoords> // for data packet routing
|
||||
```
|
||||
|
||||
### Per-Peer State
|
||||
|
||||
From each peer, we receive and store:
|
||||
- Their `ParentDeclaration`
|
||||
- Their `ancestry` (path from peer to root)
|
||||
|
||||
This provides their tree coordinates for routing decisions.
|
||||
|
||||
### Bounded State
|
||||
|
||||
Each node's TreeState contains O(P × D) entries, not O(N):
|
||||
- P = direct peer count
|
||||
- D = tree depth
|
||||
|
||||
A node knows only:
|
||||
1. Its own parent declaration
|
||||
2. Direct peers' parent declarations
|
||||
3. Ancestry chains from each peer to root
|
||||
|
||||
Nodes do NOT know about other subtrees—only paths toward root.
|
||||
|
||||
---
|
||||
|
||||
## Bloom Filter State
|
||||
|
||||
### Per-Node State
|
||||
|
||||
```
|
||||
BloomState
|
||||
├── own_node_id: NodeId // always included in outgoing filters
|
||||
├── leaf_dependents: HashSet<NodeId> // leaf-only nodes we speak for
|
||||
├── is_leaf_only: bool // if true, no filter processing
|
||||
└── update_debounce: Duration // rate limit outgoing updates
|
||||
```
|
||||
|
||||
### Per-Peer State
|
||||
|
||||
Stored on Peer:
|
||||
- `inbound_filter`: what they advertise to us (4KB Bloom filter)
|
||||
- `filter_sequence`: freshness/dedup
|
||||
- `filter_ttl`: remaining propagation hops
|
||||
- `filter_received_at`: for staleness detection
|
||||
|
||||
### Computed (On-Demand)
|
||||
|
||||
Outgoing filter to peer Q is computed, not stored:
|
||||
|
||||
```
|
||||
outbound_filter(Q) =
|
||||
own_node_id
|
||||
∪ leaf_dependents
|
||||
∪ { entries from peer[P].inbound_filter for all P ≠ Q where filter_ttl > 0 }
|
||||
```
|
||||
|
||||
TTL is decremented on contributed entries. Recomputation is cheap (4KB filter,
|
||||
7 hashes) so on-demand is preferred over cache invalidation complexity.
|
||||
|
||||
---
|
||||
|
||||
## State Machines
|
||||
|
||||
### Transport Lifecycle
|
||||
|
||||
```
|
||||
Configured ──► Starting ──► Up ──► Down
|
||||
│ │ │
|
||||
v v │
|
||||
Failed ◄────────────┘
|
||||
```
|
||||
|
||||
- `Configured`: in config, not started
|
||||
- `Starting`: initialization in progress (instant for UDP, slow for Tor)
|
||||
- `Up`: ready for links
|
||||
- `Down`: was up, now unavailable
|
||||
- `Failed`: couldn't start
|
||||
|
||||
**Events:**
|
||||
- `Start` (from config policy or API)
|
||||
- `Started` / `StartFailed`
|
||||
- `Shutdown`
|
||||
- `TransportError`
|
||||
|
||||
**Cascading:** Transport down → all links over it disconnect → all peers on
|
||||
those links disconnect.
|
||||
|
||||
### Link Lifecycle
|
||||
|
||||
**Connectionless transports (UDP, Ethernet, WiFi):**
|
||||
|
||||
Links are always implicitly "active"—no state machine needed. Link exists when
|
||||
we have `(transport_id, remote_addr)`.
|
||||
|
||||
**Connection-oriented transports (Tor):**
|
||||
|
||||
```
|
||||
Outbound:
|
||||
(connect requested) ──► Connecting ──► Connected ──► Disconnected
|
||||
│ ▲
|
||||
v │
|
||||
Failed ─────────────────────────┘
|
||||
|
||||
Inbound:
|
||||
(transport accepts) ──► Connected ──► Disconnected
|
||||
```
|
||||
|
||||
- `Connecting`: establishing connection (circuit for Tor)
|
||||
- `Connected`: ready for FIPS traffic
|
||||
- `Disconnected`: was connected, now gone
|
||||
- `Failed`: connection attempt failed
|
||||
|
||||
### Peer Lifecycle
|
||||
|
||||
```
|
||||
Discovered ──► Connecting ──► Authenticating ──► Active ──► Disconnected
|
||||
│ │ │ ▲
|
||||
│ v v │
|
||||
└──────── [timeout/fail] ──────────────────────────────────┘
|
||||
```
|
||||
|
||||
- `Discovered`: known via discovery or config, no link yet
|
||||
- `Connecting`: link establishment in progress (connection-oriented only)
|
||||
- `Authenticating`: FIPS auth handshake in progress
|
||||
- `Active`: fully integrated (has declaration, ancestry, filter)
|
||||
- `Disconnected`: was active, now gone
|
||||
|
||||
**Events:**
|
||||
|
||||
```
|
||||
PeerEvent
|
||||
├── Discovered { link_id, transport_addr, hint: Option<PublicKey> }
|
||||
├── LinkConnected
|
||||
├── LinkFailed { reason }
|
||||
├── AuthChallengeReceived { challenge }
|
||||
├── AuthResponseReceived { response }
|
||||
├── AuthSuccess
|
||||
├── AuthFailed { reason }
|
||||
├── TreeAnnounceReceived { declaration, ancestry }
|
||||
├── FilterAnnounceReceived { filter, sequence, ttl }
|
||||
├── Timeout { kind: TimeoutKind }
|
||||
├── PacketReceived { ... }
|
||||
└── LinkDisconnected { reason }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Reference Transport Types
|
||||
|
||||
### UDP/IP
|
||||
|
||||
```
|
||||
UdpTransport
|
||||
├── bind_addr: SocketAddr // e.g., 0.0.0.0:4000
|
||||
├── socket: UdpSocket
|
||||
└── state: TransportState
|
||||
|
||||
TransportAddr = SocketAddr // IP:port
|
||||
```
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection-oriented | No |
|
||||
| Reliable | No |
|
||||
| MTU | 1280-1472 |
|
||||
| Latency | Low (1-500ms) |
|
||||
| Scope | Internet |
|
||||
| Discovery | DNS-SD, Nostr, static config |
|
||||
| Privileges | None |
|
||||
| NAT | Hole punching possible |
|
||||
|
||||
### Ethernet
|
||||
|
||||
```
|
||||
EthernetTransport
|
||||
├── interface: String // "eth0"
|
||||
├── socket: RawSocket // AF_PACKET
|
||||
├── local_mac: MacAddr
|
||||
├── ethertype: u16 // FIPS ethertype
|
||||
└── state: TransportState
|
||||
|
||||
TransportAddr = MacAddr // 6 bytes
|
||||
```
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection-oriented | No |
|
||||
| Reliable | No |
|
||||
| MTU | 1500 |
|
||||
| Latency | <1ms |
|
||||
| Scope | Local segment |
|
||||
| Discovery | Multicast |
|
||||
| Privileges | CAP_NET_RAW |
|
||||
|
||||
### WiFi
|
||||
|
||||
```
|
||||
WifiTransport
|
||||
├── interface: String // "wlan0"
|
||||
├── socket: RawSocket
|
||||
├── local_mac: MacAddr
|
||||
├── mode: Infrastructure | AdHoc | Direct
|
||||
└── state: TransportState
|
||||
|
||||
TransportAddr = MacAddr // same as Ethernet
|
||||
```
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection-oriented | No |
|
||||
| Reliable | No |
|
||||
| MTU | 1500 |
|
||||
| Latency | 1-10ms |
|
||||
| Scope | Local segment (or Direct group) |
|
||||
| Discovery | Multicast, P2P service discovery |
|
||||
| Privileges | CAP_NET_RAW |
|
||||
|
||||
Infrastructure and Ad-hoc modes behave like Ethernet. WiFi Direct has its own
|
||||
service discovery mechanism.
|
||||
|
||||
### Tor Onion
|
||||
|
||||
```
|
||||
TorTransport
|
||||
├── tor_client: TorClient // arti or external daemon
|
||||
├── onion_service: Option<OnionService>
|
||||
├── local_onion_addr: Option<OnionAddr>
|
||||
└── state: TransportState
|
||||
|
||||
TransportAddr = OnionAddr // "abc...xyz.onion:port"
|
||||
```
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection-oriented | Yes |
|
||||
| Reliable | Yes (stream) |
|
||||
| MTU | Stream (framed) |
|
||||
| Latency | 500ms-5s |
|
||||
| Scope | Internet (anonymous) |
|
||||
| Discovery | Nostr, static config |
|
||||
| Privileges | None |
|
||||
| Transport startup | Slow (30s-2min for Tor bootstrap) |
|
||||
|
||||
Tor links require framing (length-prefix) over the stream. The .onion address
|
||||
is independent of FIPS npub—identity verified via FIPS auth after connecting.
|
||||
|
||||
### Transport Comparison
|
||||
|
||||
| Aspect | UDP | Ethernet | WiFi | Tor |
|
||||
|--------|-----|----------|------|-----|
|
||||
| Connection | No | No | No | Yes |
|
||||
| Link state machine | Trivial | Trivial | Trivial | Real |
|
||||
| Address type | IP:port | MAC | MAC | .onion:port |
|
||||
| Startup time | Instant | Instant | Instant | 30s-2min |
|
||||
| Base RTT hint | 50ms | 1ms | 5ms | 2s |
|
||||
| Framing | Datagram | Datagram | Datagram | Length-prefix |
|
||||
|
||||
---
|
||||
|
||||
## Event-Driven Architecture
|
||||
|
||||
The system uses multiple focused state machines rather than one giant event
|
||||
handler:
|
||||
|
||||
1. **Transport state machines** — one per transport instance
|
||||
2. **Link state machines** — one per link (meaningful for connection-oriented)
|
||||
3. **Peer state machines** — one per peer
|
||||
4. **Spanning tree module** — reacts to peer events, emits announcements
|
||||
5. **Bloom filter module** — reacts to peer/filter events, emits updates
|
||||
|
||||
**Event flow:**
|
||||
|
||||
```
|
||||
Transport
|
||||
│
|
||||
├──► DiscoveredPeer { transport_id, addr, hint }
|
||||
├──► InboundConnection { transport_id, addr, io } (connection-oriented)
|
||||
└──► PacketReceived { transport_id, addr, data }
|
||||
│
|
||||
v
|
||||
Link
|
||||
│
|
||||
├──► LinkConnected { link_id }
|
||||
├──► LinkDisconnected { link_id, reason }
|
||||
└──► FipsPacketReceived { link_id, packet }
|
||||
│
|
||||
v
|
||||
Peer
|
||||
│
|
||||
├──► AuthSuccess { peer_id }
|
||||
├──► TreeAnnounceReceived { peer_id, decl, ancestry }
|
||||
└──► FilterAnnounceReceived { peer_id, filter, seq, ttl }
|
||||
│
|
||||
v
|
||||
TreeState / BloomState
|
||||
```
|
||||
|
||||
Timers drive keepalives, timeouts, and periodic refresh (debounced announcements).
|
||||
|
||||
---
|
||||
|
||||
## Protocol Self-Healing Design
|
||||
|
||||
Control protocols tolerate packet loss without ack/retry machinery:
|
||||
|
||||
### TreeAnnounce
|
||||
|
||||
- Monotonic sequence numbers (receiver keeps highest, ignores stale/dup)
|
||||
- Full state (declaration + ancestry), not deltas
|
||||
- Periodic refresh ensures convergence
|
||||
- Lost announcement? Next one carries same or newer state
|
||||
|
||||
### FilterAnnounce
|
||||
|
||||
- Full filter replacement with sequence number
|
||||
- Periodic refresh
|
||||
- Debounced on rapid changes
|
||||
- Lost announcement? Peer has stale filter until next update
|
||||
|
||||
### LookupRequest/LookupResponse
|
||||
|
||||
- Request-response pattern
|
||||
- Lost request/response → sender times out, retries at application level
|
||||
|
||||
### SessionSetup/SessionAck
|
||||
|
||||
- Same as lookup—sender retries on timeout
|
||||
- Lost setup → first data packet fails → triggers re-establishment
|
||||
|
||||
This gossip-style eventual consistency is simpler and avoids the complexity of
|
||||
per-message reliability over unreliable links.
|
||||
|
||||
---
|
||||
|
||||
## Leaf-Only Operation
|
||||
|
||||
Leaf-only mode enables constrained devices (sensors, battery-powered nodes, mobile
|
||||
devices) to participate in FIPS without the overhead of full mesh routing.
|
||||
|
||||
### Architectural Subset
|
||||
|
||||
A leaf-only node uses a minimal subset of the full architecture:
|
||||
|
||||
```
|
||||
LeafOnlyNode
|
||||
├── identity: Identity // full (npub, nsec, node_id, address)
|
||||
├── config: Config // simplified
|
||||
├── tun: TunInterface // full (provides IPv6 to local apps)
|
||||
├── transport: Transport // one
|
||||
├── link: Link // one
|
||||
└── upstream_peer: Peer // one (simplified)
|
||||
```
|
||||
|
||||
### What's Required
|
||||
|
||||
| Component | Usage |
|
||||
|---------------------|------------------------------------------|
|
||||
| Identity | Full—same cryptographic identity model |
|
||||
| TUN interface | Full—provides IPv6 to local applications |
|
||||
| Transport | One instance (to reach upstream peer) |
|
||||
| Link | One instance (to upstream peer) |
|
||||
| Peer | One instance (upstream), with auth only |
|
||||
| Peer authentication | Full—must prove identity to upstream |
|
||||
| Packet send/receive | Full |
|
||||
|
||||
### What's Not Required
|
||||
|
||||
| Component | Reason |
|
||||
|--------------------|--------------------------------------------------|
|
||||
| TreeState | No tree participation; upstream handles routing |
|
||||
| ParentDeclaration | Doesn't announce position to network |
|
||||
| Bloom filters | Upstream peer handles reachability |
|
||||
| Filter computation | N/A |
|
||||
| CoordCache | Doesn't route for others |
|
||||
| Discovery protocol | Upstream peer handles lookups |
|
||||
| Multiple peers | Single upstream by design |
|
||||
| Session caching | Tunnels everything to upstream |
|
||||
| Transit routing | Never forwards for others |
|
||||
|
||||
### Simplified Peer Structure
|
||||
|
||||
The upstream peer entry for a leaf-only node:
|
||||
|
||||
```
|
||||
UpstreamPeer (leaf-only)
|
||||
├── node_id: NodeId
|
||||
├── npub: PublicKey
|
||||
├── link_id: LinkId
|
||||
├── state: PeerState // auth lifecycle only
|
||||
└── link_stats: LinkStats // for keepalive/timeout
|
||||
|
||||
// NOT present:
|
||||
// - declaration, ancestry (no tree participation)
|
||||
// - inbound_filter, filter_* (no Bloom filters)
|
||||
```
|
||||
|
||||
### Routing Behavior
|
||||
|
||||
```
|
||||
Outbound (local app → network):
|
||||
TUN → upstream peer (unconditionally)
|
||||
|
||||
Inbound (network → local app):
|
||||
upstream peer → TUN (if dest == self)
|
||||
upstream peer → DROP (if dest ≠ self, never transit)
|
||||
```
|
||||
|
||||
The leaf-only node doesn't make routing decisions—it tunnels everything to/from
|
||||
its upstream peer.
|
||||
|
||||
### Upstream Peer Responsibilities
|
||||
|
||||
The upstream peer (a full participant) handles:
|
||||
|
||||
- Including leaf-only node in its Bloom filter
|
||||
- Responding to LookupRequests for the leaf-only node
|
||||
- Forwarding packets to/from the leaf-only node
|
||||
- The leaf-only node appears as an entry in the upstream's `leaf_dependents` set
|
||||
|
||||
### State Machine (Simplified)
|
||||
|
||||
Only peer lifecycle matters:
|
||||
|
||||
```
|
||||
Configured ──► Connecting ──► Authenticating ──► Active ──► Disconnected
|
||||
│ │ │
|
||||
v v │
|
||||
[fail] ─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
No TreeAnnounce or FilterAnnounce processing in the Active state.
|
||||
|
||||
### Configuration (Leaf-Only Subset)
|
||||
|
||||
```
|
||||
# Required
|
||||
node.identity.nsec # or auto-generated
|
||||
node.leaf_only = true
|
||||
node.tun.device
|
||||
node.tun.mtu
|
||||
|
||||
# One transport
|
||||
transport.udp.enabled = true # or ethernet, wifi, tor
|
||||
transport.udp.bind_addr
|
||||
|
||||
# One peer (the upstream)
|
||||
peers[0].npub # required: upstream identity
|
||||
peers[0].addresses[0].type # transport type
|
||||
peers[0].addresses[0].addr # how to reach them
|
||||
peers[0].connect_policy = auto_connect
|
||||
|
||||
# Timeouts
|
||||
peer.auth.timeout
|
||||
peer.keepalive.interval
|
||||
peer.keepalive.timeout
|
||||
peer.reconnect.*
|
||||
```
|
||||
|
||||
Parameters NOT relevant to leaf-only operation:
|
||||
|
||||
```
|
||||
tree.* # no tree participation
|
||||
filter.* # no Bloom filters
|
||||
discovery.* # upstream handles
|
||||
session.* # no session caching
|
||||
transport.*.discovery.* # single configured peer
|
||||
transport.*.auto_connect # single configured peer
|
||||
```
|
||||
|
||||
### Resource Comparison
|
||||
|
||||
| Resource | Full Participant | Leaf-Only |
|
||||
|---------------------|---------------------|-----------|
|
||||
| RAM (Bloom filters) | d × 4KB (d = peers) | 0 |
|
||||
| RAM (coord cache) | 10K-100K entries | 0 |
|
||||
| RAM (tree state) | O(P × D) entries | 0 |
|
||||
| Bandwidth (idle) | < 1 KB/sec | Near zero |
|
||||
| CPU (filter ops) | Moderate | None |
|
||||
| Peers | Multiple | One |
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **IoT sensors**: Send telemetry, receive commands
|
||||
- **Mobile devices**: Battery/bandwidth constraints
|
||||
- **Privacy-conscious**: Don't see others' traffic
|
||||
- **Monitoring nodes**: Observe network, don't route
|
||||
- **Embedded systems**: Limited RAM/CPU
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Peer Configuration
|
||||
|
||||
Peers are configured separately from transports:
|
||||
|
||||
```
|
||||
PeerConfig
|
||||
├── npub: PublicKey // required: who is this
|
||||
├── alias: Option<String> // human-readable label
|
||||
├── addresses: Vec<PeerAddress> // how to reach them
|
||||
└── connect_policy: ConnectPolicy
|
||||
|
||||
PeerAddress
|
||||
├── transport_type: TransportType // "udp", "ethernet", "tor", etc.
|
||||
├── addr: String // transport-specific, parsed by driver
|
||||
└── priority: u8 // preference order
|
||||
|
||||
ConnectPolicy
|
||||
├── AutoConnect // connect on startup
|
||||
├── OnDemand // connect when traffic needs routing
|
||||
└── Manual // wait for explicit API call
|
||||
```
|
||||
|
||||
### Transport Configuration
|
||||
|
||||
```
|
||||
TransportConfig
|
||||
├── transport_type: TransportType
|
||||
├── driver_config: DriverConfig // type-specific
|
||||
├── start_policy: StartPolicy
|
||||
├── discovery_enabled: bool
|
||||
└── auto_connect: bool // auto-connect to discovered peers
|
||||
```
|
||||
|
||||
### Discovery
|
||||
|
||||
Discovery is per-transport:
|
||||
- Transports emit `DiscoveredPeer { addr, hint }` events
|
||||
- Node matches against known peer configs or creates "unknown peer" entries
|
||||
- Policy (`auto_connect`, per-peer `connect_policy`) determines action
|
||||
|
||||
---
|
||||
|
||||
## Configuration Reference (sysctl-style)
|
||||
|
||||
### Node Identity
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `node.identity.nsec` | string | (generated) | Secret key (nsec1... or hex) |
|
||||
|
||||
### TUN Interface
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `node.tun.device` | string | "fips0" | TUN device name |
|
||||
| `node.tun.mtu` | u16 | 1280 | TUN interface MTU |
|
||||
|
||||
### Spanning Tree
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `tree.announce.interval` | duration | 30s | Periodic TreeAnnounce refresh |
|
||||
| `tree.announce.on_change` | bool | true | Immediate announce on parent change |
|
||||
| `tree.parent.hold_time` | duration | 10s | Min time before switching parent |
|
||||
| `tree.parent.hysteresis` | f32 | 0.1 | Cost improvement threshold to switch |
|
||||
|
||||
### Bloom Filters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `filter.size` | u32 | 32768 | Filter size in bits (4KB) |
|
||||
| `filter.hash_count` | u8 | 7 | Number of hash functions |
|
||||
| `filter.scope` | u8 | 2 | TTL for filter propagation (K) |
|
||||
| `filter.refresh.interval` | duration | 60s | Periodic FilterAnnounce refresh |
|
||||
| `filter.update.debounce` | duration | 500ms | Min interval between updates |
|
||||
| `filter.stale.threshold` | duration | 300s | Consider peer's filter stale |
|
||||
|
||||
### Discovery Protocol
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `discovery.lookup.ttl` | u8 | 8 | Initial TTL for LookupRequest |
|
||||
| `discovery.lookup.timeout` | duration | 10s | Timeout waiting for response |
|
||||
| `discovery.lookup.retry_count` | u8 | 3 | Retries before giving up |
|
||||
| `discovery.cache.max_entries` | u32 | 10000 | Route cache size |
|
||||
| `discovery.cache.ttl` | duration | 300s | Cached coordinates expiry |
|
||||
|
||||
### Session Management
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `session.setup.timeout` | duration | 5s | SessionSetup ack timeout |
|
||||
| `session.cache.max_entries` | u32 | 50000 | Coord cache size (per router) |
|
||||
| `session.cache.ttl` | duration | 300s | Cached coordinates expiry |
|
||||
| `session.refresh.interval` | duration | 240s | Proactive session refresh |
|
||||
|
||||
### Peer Defaults
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `peer.auth.timeout` | duration | 10s | Auth handshake timeout |
|
||||
| `peer.auth.timeout_tor` | duration | 60s | Auth timeout for Tor links |
|
||||
| `peer.keepalive.interval` | duration | 30s | Keepalive probe interval |
|
||||
| `peer.keepalive.timeout` | duration | 90s | Declare peer dead after silence |
|
||||
| `peer.reconnect.policy` | enum | backoff | none, immediate, backoff |
|
||||
| `peer.reconnect.delay_initial` | duration | 1s | Initial reconnect delay |
|
||||
| `peer.reconnect.delay_max` | duration | 300s | Maximum reconnect delay |
|
||||
| `peer.reconnect.max_attempts` | u32 | 0 | 0 = unlimited |
|
||||
|
||||
### Transport: UDP
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `transport.udp.enabled` | bool | true | Enable UDP transport |
|
||||
| `transport.udp.bind_addr` | string | "0.0.0.0:4000" | Bind address |
|
||||
| `transport.udp.discovery.enabled` | bool | true | Enable discovery |
|
||||
| `transport.udp.discovery.dns_sd` | bool | false | Use DNS-SD discovery |
|
||||
| `transport.udp.discovery.nostr_relays` | list | [] | Relays for peer discovery |
|
||||
| `transport.udp.auto_connect` | bool | true | Connect to discovered peers |
|
||||
| `transport.udp.base_rtt` | duration | 50ms | RTT hint for timeouts |
|
||||
|
||||
### Transport: Ethernet
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `transport.ethernet.enabled` | bool | false | Enable Ethernet transport |
|
||||
| `transport.ethernet.interface` | string | "eth0" | Interface name |
|
||||
| `transport.ethernet.ethertype` | u16 | 0x88b5 | FIPS EtherType |
|
||||
| `transport.ethernet.discovery.enabled` | bool | true | Enable multicast discovery |
|
||||
| `transport.ethernet.discovery.multicast_addr` | string | "33:33:00:00:ff:05" | Discovery multicast MAC |
|
||||
| `transport.ethernet.auto_connect` | bool | true | Connect to discovered peers |
|
||||
| `transport.ethernet.base_rtt` | duration | 1ms | RTT hint for timeouts |
|
||||
|
||||
### Transport: WiFi
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `transport.wifi.enabled` | bool | false | Enable WiFi transport |
|
||||
| `transport.wifi.interface` | string | "wlan0" | Interface name |
|
||||
| `transport.wifi.mode` | enum | infrastructure | infrastructure, adhoc, direct |
|
||||
| `transport.wifi.discovery.enabled` | bool | true | Enable discovery |
|
||||
| `transport.wifi.auto_connect` | bool | true | Connect to discovered peers |
|
||||
| `transport.wifi.base_rtt` | duration | 5ms | RTT hint for timeouts |
|
||||
|
||||
### Transport: Tor
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `transport.tor.enabled` | bool | false | Enable Tor transport |
|
||||
| `transport.tor.mode` | enum | embedded | embedded, external |
|
||||
| `transport.tor.control_port` | string | "127.0.0.1:9051" | External daemon control |
|
||||
| `transport.tor.onion_service.enabled` | bool | true | Publish onion service |
|
||||
| `transport.tor.onion_service.port` | u16 | 4000 | Onion service port |
|
||||
| `transport.tor.discovery.enabled` | bool | true | Enable discovery |
|
||||
| `transport.tor.discovery.nostr_relays` | list | [] | Relays for peer discovery |
|
||||
| `transport.tor.auto_connect` | bool | false | Connect to discovered peers |
|
||||
| `transport.tor.base_rtt` | duration | 2s | RTT hint for timeouts |
|
||||
| `transport.tor.startup_timeout` | duration | 180s | Tor bootstrap timeout |
|
||||
|
||||
### Adaptive Timeouts
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `timeout.adaptive.enabled` | bool | true | Use measured RTT for timeouts |
|
||||
| `timeout.adaptive.rtt_multiplier` | f32 | 3.0 | Timeout = RTT × multiplier |
|
||||
| `timeout.adaptive.min` | duration | 100ms | Minimum timeout |
|
||||
| `timeout.adaptive.max` | duration | 60s | Maximum timeout |
|
||||
|
||||
### Leaf-Only Mode
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `node.leaf_only` | bool | false | Operate as leaf-only node |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [fips-design.md](fips-design.md) — Overall FIPS protocol design
|
||||
- [fips-links.md](fips-links.md) — Link layer requirements
|
||||
- [fips-routing.md](fips-routing.md) — Routing, Bloom filters, discovery
|
||||
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Tree protocol dynamics
|
||||
Reference in New Issue
Block a user