mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-22 07:48:26 +00:00
Implement FilterAnnounce send/receive, remove TTL/K-hop scoping
Add bloom filter reachability announcement protocol: - FilterAnnounce encode/decode (wire format 0x20, 1035 bytes) - node/bloom.rs: send/receive with debounce, split-horizon loop prevention - Handler wiring: dispatch, tick, peer promotion/removal, cross-connection - Five integration tests: 10-node, star, chain, ring, 100-node convergence Remove TTL/K-hop mechanism from code and design docs after discovering that per-entry TTL scoping is fundamentally incompatible with flat bloom filter merge + regeneration architecture. Each node re-originates filters with fresh TTL, making propagation unbounded regardless of TTL value. Split-horizon remains the primary loop prevention mechanism. Document spanning tree known limitations (v1) in spanning-tree-dynamics.md. 316 tests pass, clean build, zero warnings.
This commit is contained in:
Johnathan Corgan
@@ -156,7 +156,6 @@ Peer
|
||||
│ // Bloom filter (inbound—what's reachable through them)
|
||||
├── inbound_filter: BloomFilter
|
||||
├── filter_sequence: u64
|
||||
├── filter_ttl: u8
|
||||
├── filter_received_at: Timestamp
|
||||
├── pending_filter_update: bool // we owe them an update
|
||||
│
|
||||
@@ -246,9 +245,8 @@ BloomState
|
||||
|
||||
Stored on Peer:
|
||||
|
||||
- `inbound_filter`: what they advertise to us (4KB Bloom filter)
|
||||
- `inbound_filter`: what they advertise to us (1KB Bloom filter)
|
||||
- `filter_sequence`: freshness/dedup
|
||||
- `filter_ttl`: remaining propagation hops
|
||||
- `filter_received_at`: for staleness detection
|
||||
|
||||
### Computed (On-Demand)
|
||||
@@ -259,11 +257,11 @@ Outgoing filter to peer Q is computed, not stored:
|
||||
outbound_filter(Q) =
|
||||
own_node_addr
|
||||
∪ leaf_dependents
|
||||
∪ { entries from peer[P].inbound_filter for all P ≠ Q where filter_ttl > 0 }
|
||||
∪ { entries from peer[P].inbound_filter for all P ≠ Q }
|
||||
```
|
||||
|
||||
TTL is decremented on contributed entries. Recomputation is cheap (4KB filter,
|
||||
7 hashes) so on-demand is preferred over cache invalidation complexity.
|
||||
Recomputation is cheap (1KB filter, 5 hashes) so on-demand is preferred over
|
||||
cache invalidation complexity.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -187,7 +187,6 @@ node's filter indicates which destinations are reachable through it.
|
||||
```text
|
||||
FilterAnnounce {
|
||||
sequence: u64, // For freshness/deduplication
|
||||
ttl: u8, // Remaining propagation hops
|
||||
filter: BloomFilter, // Variable size based on size_class
|
||||
}
|
||||
|
||||
@@ -220,9 +219,6 @@ Receivers can fold larger filters down to their preferred size.
|
||||
**sequence**: Monotonic counter for this node's filter. Allows receivers to
|
||||
detect stale or duplicate announcements.
|
||||
|
||||
**ttl**: Remaining propagation depth. Starts at K (typically 2), decremented
|
||||
each hop. At TTL=0, entries are not propagated further.
|
||||
|
||||
**hash_count**: Number of hash functions used. v1 uses k=5, which is optimal
|
||||
for 800-1,600 entries in a 1 KB filter.
|
||||
|
||||
@@ -244,10 +240,12 @@ A node's outgoing filter to peer Q contains:
|
||||
|
||||
1. This node's own node_addr
|
||||
2. Node_ids of leaf-only dependents (nodes using this node as sole peer)
|
||||
3. Entries from filters received from other peers (not Q) with TTL > 0
|
||||
3. Entries merged from filters received from all other peers (not Q)
|
||||
|
||||
This creates K-hop reachability scope. With K=2, entries propagate ~4 hops
|
||||
before TTL exhaustion.
|
||||
This split-horizon merge (excluding the destination peer's own filter from
|
||||
the computation) prevents a node's entries from being echoed back to it,
|
||||
providing loop prevention. Filters propagate transitively through the
|
||||
network without any hop limit.
|
||||
|
||||
### 3.4 Exchange Rules
|
||||
|
||||
@@ -273,10 +271,10 @@ Rate limiting:
|
||||
|
||||
```text
|
||||
1. Store: peer_filters[P] = received.filter
|
||||
2. If received.ttl > 0:
|
||||
- Include entries in next announcement to other peers
|
||||
- Decrement TTL for propagated entries
|
||||
3. Recompute own outgoing filters if changed
|
||||
2. Recompute outgoing filters for all other peers:
|
||||
- For each peer Q (Q != P):
|
||||
outgoing[Q] = merge(self_filter, peer_filters[all peers except Q])
|
||||
- If outgoing[Q] changed, send FilterAnnounce to Q
|
||||
```
|
||||
|
||||
### 3.5 Filter Expiration
|
||||
@@ -435,7 +433,6 @@ handshake completion.
|
||||
| ROOT_REFRESH_INTERVAL | 30 min | Root regenerates timestamp |
|
||||
| ROOT_TIMEOUT | 60 min | Root declaration considered stale |
|
||||
| TREE_ENTRY_TTL | 5-10 min | Individual entry expiration |
|
||||
| FILTER_TTL_HOPS | 2 | Bloom filter propagation depth |
|
||||
| ANNOUNCE_MIN_INTERVAL | 500 ms | Rate limit for announcements |
|
||||
| LOOKUP_TTL | 8 | Discovery request propagation limit |
|
||||
| LOOKUP_TIMEOUT | 5 sec | Time to wait for response |
|
||||
@@ -575,10 +572,9 @@ Propagates Bloom filter reachability information.
|
||||
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
|
||||
│ │ 0 │ msg_type │ 1 byte │ 0x11 │ │
|
||||
│ │ 1 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │
|
||||
│ │ 9 │ ttl │ 1 byte │ Remaining propagation hops │ │
|
||||
│ │ 10 │ hash_count │ 1 byte │ Number of hash functions (5) │ │
|
||||
│ │ 11 │ size_class │ 1 byte │ Filter size: 512 << class │ │
|
||||
│ │ 12 │ filter_bits │ variable │ 512 << size_class bytes │ │
|
||||
│ │ 9 │ hash_count │ 1 byte │ Number of hash functions (5) │ │
|
||||
│ │ 10 │ size_class │ 1 byte │ Filter size: 512 << class │ │
|
||||
│ │ 11 │ filter_bits │ variable │ 512 << size_class bytes │ │
|
||||
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
|
||||
│ │
|
||||
│ Size classes (powers of 2 for foldability): │
|
||||
@@ -587,8 +583,8 @@ Propagates Bloom filter reachability information.
|
||||
│ 2 = 2,048 bytes (16,384 bits) - Reserved for future │
|
||||
│ 3 = 4,096 bytes (32,768 bits) - Reserved for future │
|
||||
│ │
|
||||
│ v1 total payload: 1 + 8 + 1 + 1 + 1 + 1024 = 1036 bytes │
|
||||
│ With link overhead: 1065 bytes │
|
||||
│ v1 total payload: 1 + 8 + 1 + 1 + 1024 = 1035 bytes │
|
||||
│ With link overhead: 1064 bytes │
|
||||
│ │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ BLOOM FILTER STRUCTURE │
|
||||
@@ -621,12 +617,11 @@ Propagates Bloom filter reachability information.
|
||||
PLAINTEXT BYTES:
|
||||
11 ← msg_type = FilterAnnounce
|
||||
2A 00 00 00 00 00 00 00 ← sequence = 42
|
||||
02 ← ttl = 2 (will propagate 2 more hops)
|
||||
05 ← hash_count = 5
|
||||
01 ← size_class = 1 (1 KB filter)
|
||||
[1024 bytes of filter bits] ← Bloom filter
|
||||
|
||||
Total: 1036 bytes
|
||||
Total: 1035 bytes
|
||||
```
|
||||
|
||||
### A.3 LookupRequest (0x12)
|
||||
|
||||
@@ -11,8 +11,8 @@ For spanning tree dynamics and convergence, see [spanning-tree-dynamics.md](span
|
||||
|
||||
FIPS routing combines three mechanisms:
|
||||
|
||||
1. **Bloom filters**: Fast reachability lookup for nearby destinations (within
|
||||
K-hop scope)
|
||||
1. **Bloom filters**: Fast reachability lookup for destinations reachable
|
||||
through peers
|
||||
2. **Discovery protocol**: Query-based lookup for distant destinations
|
||||
3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree
|
||||
position
|
||||
@@ -34,7 +34,7 @@ coordinates handle the latter.
|
||||
| Scale | Nodes | Bloom Filter Role |
|
||||
|-------|-------|-------------------|
|
||||
| Small private network | 100-1,000 | Covers entire network |
|
||||
| Modest public network | ~1,000,000 | Covers K-hop neighborhood |
|
||||
| Modest public network | ~1,000,000 | Covers transitive peer neighborhood |
|
||||
| Internet-scale | Billions | Out of scope (requires different architecture) |
|
||||
|
||||
The primary design target is networks up to ~1M nodes.
|
||||
@@ -64,11 +64,10 @@ traffic tunnels through that peer.
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Value | Rationale |
|
||||
|-----------|-------|-----------|
|
||||
| Filter size | 1 KB (8,192 bits) | Sized for expected occupancy with margin |
|
||||
| Hash functions | 5 | Optimal for 800-1,600 entries at this size |
|
||||
| Scope (K) | 2 | Effective ~4-hop reach with TTL propagation |
|
||||
| Parameter | Value | Rationale |
|
||||
|----------------|-------------------|--------------------------------------------|
|
||||
| Filter size | 1 KB (8,192 bits) | Sized for expected occupancy with margin |
|
||||
| Hash functions | 5 | Optimal for 800-1,600 entries at this size |
|
||||
|
||||
### Mathematical Foundation
|
||||
|
||||
@@ -99,21 +98,15 @@ For 1% FPR: m ≈ 9.6n bits. For 5% FPR: m ≈ 6.2n bits.
|
||||
|
||||
### Expected Filter Occupancy
|
||||
|
||||
Filter occupancy depends on K-hop scope and node degree, **not** total network
|
||||
size. The TTL mechanism bounds entries regardless of network scale.
|
||||
|
||||
**Nodes within h hops in a tree (branching factor b = d-1):**
|
||||
|
||||
```text
|
||||
nodes_within_h_hops = (b^(h+1) - 1) / (b - 1)
|
||||
```
|
||||
|
||||
For d=8 (b=7), K=2: each peer's 2-hop neighborhood ≈ 57 nodes.
|
||||
Filter occupancy depends on network topology and node degree. In practice,
|
||||
filters reach a natural equilibrium determined by the network's structure —
|
||||
merging peer filters transitively means each filter converges to represent
|
||||
the node's reachable neighborhood.
|
||||
|
||||
**Outgoing filter to peer Q contains:**
|
||||
|
||||
- Self (1 entry)
|
||||
- Entries from (d-1) other peers' filters, with overlap
|
||||
- Entries from (d-1) other peers' filters (excluding Q), with overlap
|
||||
|
||||
**Expected occupancy by node degree:**
|
||||
|
||||
@@ -203,17 +196,14 @@ A node's outgoing filter to peer Q contains:
|
||||
|
||||
1. This node's own Node ID
|
||||
2. Node IDs of leaf-only dependents
|
||||
3. Entries from filters received from other peers (not Q) with TTL > 0
|
||||
3. Entries merged from filters received from all other peers (not Q)
|
||||
|
||||
This creates K-hop reachability scope through TTL-based propagation.
|
||||
|
||||
### K-Hop Scope Emergence
|
||||
|
||||
With TTL starting at K=2:
|
||||
|
||||
- Entries propagate ~2K hops before stopping
|
||||
- Each node's filter contains destinations within ~4-hop effective range
|
||||
- Bounded by O(d^2K) entries regardless of total network size
|
||||
Filters propagate transitively through the network. Each node merges all
|
||||
inbound peer filters (excluding the destination peer) into its outgoing
|
||||
filter — this split-horizon approach prevents a node's own entries from
|
||||
being echoed back to it, providing loop prevention. Propagation is
|
||||
unbounded; filters naturally converge as the Bloom filter's fixed size
|
||||
limits information density.
|
||||
|
||||
### Expiration
|
||||
|
||||
@@ -611,7 +601,7 @@ enum RouteState {
|
||||
|
||||
When nodes join/leave:
|
||||
|
||||
- Bloom filter updates propagate (bounded by K-hop scope)
|
||||
- Bloom filter updates propagate through affected peers
|
||||
- Affected sessions may need re-establishment
|
||||
- Discovery queries for newly-joined nodes
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ The protocol is based on Yggdrasil v0.5's CRDT gossip design.
|
||||
8. [Cost Metrics and Parent Selection](#8-cost-metrics-and-parent-selection)
|
||||
9. [Steady State Behavior](#9-steady-state-behavior)
|
||||
10. [Worked Examples](#10-worked-examples)
|
||||
11. [Known Limitations (v1 Implementation)](#known-limitations-v1-implementation)
|
||||
|
||||
---
|
||||
|
||||
@@ -1059,6 +1060,102 @@ initial exchange).
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations (v1 Implementation)
|
||||
|
||||
The following limitations exist in the current implementation relative to the
|
||||
design described in this document. They are documented here to guide future
|
||||
work.
|
||||
|
||||
### Known Limitation: Root Timeout Not Enforced
|
||||
|
||||
The design specifies a 60-minute root timeout (§1 timing parameters, §6
|
||||
partition detection) after which nodes should treat the root as departed and
|
||||
re-elect. The current implementation does not track root entry timestamps or
|
||||
perform staleness checks.
|
||||
|
||||
**Impact**: If the root node disappears permanently without a graceful
|
||||
disconnect, remaining nodes retain stale root state indefinitely. Nodes that
|
||||
lose their direct parent will re-elect locally (via `handle_parent_lost()`),
|
||||
but nodes with an intact path to a now-departed root will not detect the
|
||||
failure.
|
||||
|
||||
**Required fix**: Track the timestamp of the most recent root declaration in
|
||||
`TreeState`. In `check_tree_state()` (called every 1s from the RX loop),
|
||||
compare against `root_timeout` (default 60 min). On expiration, treat it as
|
||||
root loss — increment sequence number, become own root, and re-announce.
|
||||
|
||||
### Known Limitation: No TTL on Tree Entries
|
||||
|
||||
The design specifies a 5-10 minute TTL on tree entries (§8 timing
|
||||
parameters). Peer entries in `TreeState` are never expired; they persist until
|
||||
explicitly removed by peer disconnection.
|
||||
|
||||
**Impact**: Stale ancestry information from departed nodes remains in
|
||||
`TreeState`, potentially affecting coordinate computation. In practice, this
|
||||
is partially mitigated by parent loss handling, but entries for non-parent
|
||||
peers that depart without a graceful disconnect will linger.
|
||||
|
||||
**Required fix**: Add a `last_seen` timestamp to peer entries in `TreeState`.
|
||||
In `check_tree_state()`, expire entries older than `tree_entry_ttl`. When
|
||||
entries expire, re-evaluate parent selection if the expired entry was the
|
||||
current parent.
|
||||
|
||||
### Known Limitation: No Partition Detection
|
||||
|
||||
The design describes partition detection via gossip staleness (§6) where
|
||||
nodes detect isolation when root announcements stop arriving and the root
|
||||
entry eventually expires, triggering independent partition operation.
|
||||
|
||||
**Impact**: Without root timeout enforcement (see above), partitioned nodes
|
||||
cannot detect that they've lost connectivity to the root. They continue with
|
||||
stale coordinates rather than forming an independent partition with a local
|
||||
root. This affects only the case where the path to root is broken at some
|
||||
intermediate point — direct parent loss is handled correctly.
|
||||
|
||||
**Required fix**: Depends on root timeout implementation. Once root timeout
|
||||
is enforced, partition detection follows naturally: a node whose root entry
|
||||
expires and has no peer with a fresher root declaration is partitioned. It
|
||||
becomes its own root and announces, allowing the partition to converge
|
||||
independently.
|
||||
|
||||
### Known Limitation: Limited Stability Mechanisms
|
||||
|
||||
The implementation includes basic hysteresis (`PARENT_SWITCH_THRESHOLD = 1`
|
||||
depth difference required to switch parents), but the temporal stability
|
||||
mechanisms described in the design are not implemented:
|
||||
|
||||
- No hold timer on parent changes (minimum time before next switch)
|
||||
- No sequence number advancement rate limiting
|
||||
- No announcement suppression during transient topology changes
|
||||
- No minimum stable state duration before re-announcing
|
||||
|
||||
**Impact**: Rapid topology changes (e.g., a flapping link) could cause
|
||||
excessive announcement traffic and repeated coordinate recomputation.
|
||||
Currently mitigated by per-peer rate limiting on TreeAnnounce sends, but
|
||||
the source node is not throttled.
|
||||
|
||||
**Proposed fix**: Add a hold-down timer (e.g., 5-10s) after each parent
|
||||
change during which further parent switches are suppressed unless the current
|
||||
parent is lost entirely. Track announcement rate and suppress if exceeding a
|
||||
threshold.
|
||||
|
||||
### Known Limitation: Integration Test Gaps
|
||||
|
||||
Unit tests for `TreeState` and `TreeCoordinate` are comprehensive, and basic
|
||||
integration tests verify TreeAnnounce exchange and parent ancestry
|
||||
propagation. However, the following failure scenarios lack test coverage:
|
||||
|
||||
- Root node failure and network-wide re-election
|
||||
- Network partition formation and independent operation
|
||||
- Partition healing and root convergence
|
||||
- Stale entry cleanup (depends on TTL implementation)
|
||||
- Parent flapping under rapid topology changes
|
||||
|
||||
These tests are blocked on or related to the limitations above and should be
|
||||
added as each limitation is resolved.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
The gossip-based spanning tree protocol achieves distributed coordination
|
||||
|
||||
Reference in New Issue
Block a user