fips/docs/design/spanning-tree-dynamics.md

FIPS Spanning Tree Protocol Dynamics

A detailed study of the gossip-based spanning tree protocol, focusing on operational behavior under various mesh conditions. This document complements fips-intro.md with step-by-step walkthroughs of protocol dynamics rather than message formats and data structures.

For wire formats, see fips-wire-formats.md (TreeAnnounce section). For spanning tree algorithms and data structures, see fips-spanning-tree.md. For how the spanning tree fits into mesh routing, see fips-mesh-operation.md.

The protocol is based on Yggdrasil v0.5's CRDT gossip design.

Contents

1. Core Concepts
2. Single Node Startup
3. Node Joining an Existing Network
4. Network Convergence
5. Topology Changes and Reconvergence
6. Partition Detection and Handling
7. Link Failure Detection
8. Parent Selection
9. Steady State Behavior
10. Worked Examples
11. Known Limitations (v1 Implementation)

---

1. Core Concepts

The CRDT Approach

The spanning tree is maintained as a distributed soft-state CRDT-Set. Each node makes independent local decisions about parent selection, gossips these decisions to peers, and the system converges to a consistent structure without coordination.

Key properties:

• Consistency: Two peered nodes eventually have identical views of their shared relevant portion of the tree
• Atomicity: Updates to a common ancestor are applied atomically across all peer records in the local routing table
• Convergence: The structure converges in time proportional to tree depth, not network size

What Each Node Knows (Bounded State)

A node's TreeState contains only:

1. Its own parent declaration - who it has selected as parent
2. Direct peer declarations - each peer's parent selection
3. Ancestry of peers - the chain from each peer up to root

This is O(P × D) entries where P is peer count and D is tree depth—not O(N) where N is network size. A node does not know about:

• Other subtrees branching off its ancestors
• Siblings of ancestors
• Nodes in distant parts of the network

This bounded state is sufficient to compute the node's own tree coordinates and distances to any node whose coordinates it learns (via lookup responses).

Example: In a 1000-node network with tree depth 10, a node with 5 peers maintains roughly 50 TreeState entries, not 1000.

Root Election

The root is deterministic: the node with the lexicographically smallest node_addr among all reachable nodes. No explicit election protocol exists—each node independently derives the same answer from its local TreeState.

---

2. Single Node Startup

When a node starts with no peers, it bootstraps as a single-node network.

Step-by-Step: Isolated Startup

Time T0: Node A starts
├── Generates or loads keypair (npub_A, nsec_A)
├── Computes node_addr_A = SHA-256(npub_A)
├── Initializes empty TreeState
├── Sets parent = self (A is its own root)
├── Sets sequence = 1
└── Records timestamp = now

State after T0:
  TreeState_A = { (A, parent=A, seq=1, ts=T0) }
  Root_A = A
  Coordinate_A = [A]

At this point, node A is a fully functional single-node FIPS network. It can:

• Accept incoming peer connections
• Route packets to itself
• Respond to lookups for its own address

What Triggers State Changes

While isolated, A's state only changes on:

1. Peer connection: A new peer triggers gossip exchange (covered in Section 3)

---

3. Node Joining an Existing Network

When a new node connects to an existing network, a sequence of gossip exchanges integrates it into the spanning tree.

Step-by-Step: Node B Joins via Node A

Initial state: Network has nodes A (root), C, D, E. Node B is new.

Existing tree structure:
        A (root, smallest node_addr)
       /|\
      C D E

B's initial state (before connecting):
  TreeState_B = { (B, parent=B, seq=1) }
  Root_B = B

T1: B establishes link to D

Link established B ←→ D

Immediate actions:
├── B sends TreeAnnounce to D:
│   └── Contains: B's declaration (parent=B, seq=1), B's ancestry (just B)
│
└── D sends TreeAnnounce to B:
    └── Contains: D's declaration (parent=A, seq=47), D's ancestry [D, A]

T2: B processes D's announcement

B receives D's TreeAnnounce:
├── Verifies signature on D's parent declaration
├── Verifies signature on A's self-declaration (from ancestry)
├── Merges into TreeState_B:
│   └── TreeState_B = { (B, parent=B, seq=1), (D, parent=A, seq=47), (A, parent=A, seq=203) }
│
├── Evaluates root:
│   └── Compares node_addr_A vs node_addr_B
│   └── If A < B: Root_B = A (A has smaller node_addr)
│
└── Evaluates parent selection:
    └── Only peer is D
    └── D has path to new root A
    └── B selects D as parent

T3: B updates its declaration

B's state change:
├── parent_B = D (was: B)
├── sequence_B = 2 (incremented)
├── timestamp_B = T3
└── Signs new declaration

TreeState_B = { (B, parent=D, seq=2), (D, parent=A, seq=47), (A, parent=A, seq=203) }
Root_B = A
Coordinate_B = [B, D, A]

T4: B announces to D

B sends TreeAnnounce to D:
└── Contains: B's new declaration (parent=D, seq=2), ancestry [B, D, A]

D receives and merges:
├── TreeState_D now includes B's entry
├── D's coordinate unchanged: [D, A]
└── D can now route to B

T5: D updates its bloom filter:

D adds B's node_addr to its bloom filter
D sends FilterAnnounce to parent A

A merges D's bloom filter with its view of D's subtree
A now knows "B is reachable through D" (probabilistically)

This bloom filter update propagates toward root.

Important: D does NOT include B's declaration in TreeAnnounce to A. Tree gossip only includes the sender's ancestry (path to root), not children. Most nodes never learn B's declaration—they learn B is reachable via bloom filters.

Convergence Time

B becomes fully routable when:

1. B has full ancestry (immediate, from D's first announcement)
2. B's bloom filter entry propagates toward root (O(depth) hops)

The propagation time is O(tree depth), not O(network size). In the example:

• B's coordinates are known immediately (B computes from D's ancestry)
• B's reachability propagates via bloom filter: D → A (1 hop to root)
• Any node wanting to reach B does a bloom filter lookup for candidate selection
• Total: 1-2 gossip rounds for B to be locatable

Note: Nodes A, C, E never add B to their TreeState. They can still route to B by using bloom filter lookup for candidate selection to get B's coordinates, then coordinate-based greedy routing.

---

4. Network Convergence

Convergence is the process by which the spanning tree stabilizes into a consistent structure. This does not mean all nodes have the same TreeState— each node only knows its own ancestry and peers. Convergence means:

• All nodes agree on the root identity
• Each node has selected a stable parent
• Peered nodes have consistent views of their shared ancestry

Initial Network Formation

When multiple isolated nodes connect simultaneously, the network must:

1. Elect a single root (determined by smallest node_addr)
2. Form a loop-free tree structure
3. Propagate ancestry information along peer links

Example: Three nodes connect simultaneously

T0: Nodes A, B, C start isolated
    Each is its own root
    node_addr ordering: A < B < C

T1: Links form: A ←→ B, B ←→ C

T2: Gossip round 1
    A sends to B: (A, parent=A)
    B sends to A: (B, parent=B)
    B sends to C: (B, parent=B)
    C sends to B: (C, parent=C)

T3: Processing round 1
    B learns A < B, adopts A as root, selects A as parent
    C learns B exists (but B still claims self as root)

T4: Gossip round 2
    B sends to A: (B, parent=A) — B has re-parented
    B sends to C: (B, parent=A), ancestry includes A

T5: Processing round 2
    C learns A (via B's ancestry), A < C
    C adopts A as root, selects B as parent

T6: Gossip round 3
    C sends to B: (C, parent=B)

T7: Converged state
    Root = A
    Tree: A ← B ← C

Convergence Properties

Consistency guarantee: After gossip quiesces:

• All nodes agree on the identity of the root
• Each node has a stable parent selection
• Peered nodes have identical views of their shared ancestry (the CRDT property)
• Any two nodes can compute accurate distance via their coordinates

Nodes do not have global knowledge—a leaf node knows nothing about distant subtrees. But any node can locate any other node via Bloom-guided candidate selection and then route using tree coordinate distance.

Convergence time: Bounded by tree depth × gossip interval. For a tree of depth D with gossip interval G:

• Worst case: D × G for root information to propagate to deepest leaf
• Typical case: Faster due to parallel gossip on multiple links

No coordination required: Convergence emerges from:

• Deterministic root election (smallest node_addr)
• Deterministic merge rules (highest sequence wins)
• Eventually consistent gossip

Partial Convergence States

During convergence, the network may temporarily have:

• Multiple roots: Different partitions with different root beliefs
• Inconsistent coordinates: Nodes computing distances from stale state
• Routing failures: Coordinate-based greedy routing may fail until coordinates stabilize

These are transient. The protocol guarantees eventual convergence, not instant consistency.

---

5. Topology Changes and Reconvergence

When links are added or removed, the spanning tree must adapt. The CRDT design ensures this happens without coordination.

Link Addition

Adding a link can:

1. Provide a better path to root → parent change
2. Connect previously separate partitions → root change
3. Have no structural effect → just adds routing option

Example: Better path discovered

Before: A ← B ← C ← D (linear chain, A is root)
        D's coordinate: [D, C, B, A], depth 3

New link: A ←→ D established

D receives A's announcement directly:
├── A's ancestry: [A] (depth 0)
├── D evaluates: going through A gives depth 1 vs current depth 3
├── If improvement > stability threshold:
│   └── D re-parents to A
│   └── D's new coordinate: [D, A], depth 1

After: A is root
       ├── B (depth 1)
       │   └── C (depth 2)
       └── D (depth 1)

Link Removal

Removing a link can:

1. Remove parent → must find new parent
2. Partition the network → separate root election
3. Remove non-parent peer → minimal impact

Example: Parent link fails

Before: A ← B ← C, B ← D
        C's parent is B

Link B ←→ C fails:
├── C detects link failure (see Section 7)
├── C's TreeState still contains B's entry (hasn't expired)
├── C has no peers with path to A
├── C becomes its own root temporarily
│
└── If C has other peers:
    └── C may discover path to A through them
    └── C re-parents to best available peer

└── If C is truly isolated:
    └── C remains its own root
    └── C is now a separate single-node network

Reconvergence Dynamics

Stability threshold: To prevent flapping, a node only changes parent when the improvement exceeds a threshold. In v1, this is a depth difference of at least PARENT_SWITCH_THRESHOLD (1 hop). See §8 for details.

Sequence number advancement: Each parent change increments the sequence number. Nodes observing rapid sequence increases can detect instability and may apply damping.

Announcement suppression: A node doesn't immediately announce every transient state. Brief instability may resolve before announcement, reducing gossip noise.

---

6. Partition Detection and Handling

Network partitions create isolated segments that must operate independently.

How Partitions Form

A partition occurs when there's no path between two sets of nodes:

Before:
    A ← B ← C ← D ← E
    (A is root)

Link C ←→ D fails:

After:
    Partition 1: A ← B ← C
    Partition 2: D ← E (or E ← D, depending on node_addrs)

Partition Detection

Nodes detect they're partitioned when:

1. Parent unreachable: Direct link to parent fails
2. Root unreachable: No peer has path to current root

Detection via gossip staleness (not currently implemented — see Known Limitations):

In principle, nodes would also detect partitions through root entry staleness: if no fresh root announcements arrive within a timeout, the root is presumed departed. This requires tracking root entry timestamps and enforcing expiration, which is not yet implemented. Currently, only direct parent loss (case 1) and absence of any peer with a path to root (case 2) trigger partition detection.

Independent Operation

Each partition operates as an independent network:

Partition 1 (nodes A, B, C):
├── Root = A (unchanged, A still reachable)
├── Tree structure unchanged
└── Routing works within partition

Partition 2 (nodes D, E):
├── Previous root A is unreachable
├── D and E exchange announcements
├── New root = min(node_addr_D, node_addr_E)
├── Tree forms between D and E
└── Routing works within partition

Partition Healing

When connectivity is restored:

Link C ←→ D restored:

T1: C and D exchange TreeAnnounce
    C sends: root=A, ancestry [C, B, A]
    D sends: root=D (assuming D < E), ancestry [D]

T2: D processes C's announcement
    D learns about A
    If A < D: D adopts A as new root
    D selects C as parent (path to A)

T3: D announces to E
    E learns about A through D's new ancestry
    E re-evaluates and re-parents if needed

T4: Merged network
    Single root (A)
    All nodes reachable via unified tree structure
    (Each node still only knows its own ancestry, not global topology)

Root Stability Across Partitions

A key design consideration: the root should be stable to minimize reconvergence. If partition 2 elected a "temporary" root with a large node_addr, healing is cheap— that root immediately defers to the global root.

If by chance partition 2's root has a smaller node_addr than partition 1's root, healing causes partition 1 to reconverge to the new global root.

---

7. Link Failure Detection

Detecting failed links is critical for timely reconvergence.

Detection Mechanisms

Traffic-based detection (Yggdrasil v0.5 approach):

On sending data to peer:
    set read_deadline = now + peer_timeout

On receiving data from peer:
    clear read_deadline

On deadline expiration:
    mark link as failed
    remove peer from active peers
    trigger reconvergence if peer was parent

This avoids dedicated keepalive traffic—normal protocol messages serve as implicit heartbeats.

Explicit keepalive (for idle links):

If no traffic sent to peer in keepalive_interval:
    send Dummy message (type 0x00)
    expect acknowledgment within peer_timeout

Failure Response

When a link failure is detected:

link_failed(peer):
    remove peer from active_peers

    if peer == current_parent:
        // Critical: lost path to root
        select_new_parent()
        if no_valid_parent_available:
            become_own_root()
        announce_to_all_peers()
    else:
        // Non-critical: lost a potential route
        // TreeState entries for peer will expire naturally
        // May trigger parent re-evaluation if peer was better path

Timing Considerations

Fast detection vs. stability tradeoff:

• Short timeout: Quick failure detection, but transient issues cause flapping
• Long timeout: Stable under jitter, but slow to respond to real failures

Typical values (see fips-spanning-tree.md for current FIPS-specific parameters):

peer_timeout: 10-30 seconds
keepalive_interval: peer_timeout / 3
gossip_interval: on topology change (no periodic refresh)
tree_entry_ttl: not currently enforced (see Known Limitations)

Asymmetric Failures

Links may fail asymmetrically (A can send to B, but not receive):

A → B: working
B → A: failed

B detects: no responses from A, marks link failed
A doesn't detect: still receiving from B

Resolution:
├── B stops sending to A
├── A eventually times out waiting for B's traffic
├── Both sides converge to "link failed" state

The protocol handles this through bidirectional timeout—both sides must see traffic to consider the link alive.

---

8. Parent Selection

Parent selection determines tree structure and routing efficiency.

v1 Implementation: Depth-Only Selection

The current implementation uses tree depth as the sole selection metric, with a threshold to prevent thrashing between equivalent-depth paths.

Algorithm (TreeState::evaluate_parent() in tree.rs):

evaluate_parent():
    // 1. Find smallest root reachable through any peer
    smallest_root = min(peer.root for peer in peers_with_coords)

    if self == smallest_root and is_root:
        return None  // Already root, no change

    // 2. Among peers reaching smallest_root, find shallowest
    best_peer = min(
        [p for p in peers if p.root == smallest_root],
        key=lambda p: p.depth
    )
    proposed_depth = best_peer.depth + 1

    if best_peer == current_parent:
        return None  // Already using best

    // 3. Always switch if parent is gone or root is changing
    if current_parent not in peers:
        return best_peer  // Path broken
    if current_root != smallest_root:
        return best_peer  // Better root found

    // 4. For same root: require depth improvement ≥ threshold
    current_depth = my_coords.depth()
    if current_depth >= proposed_depth + PARENT_SWITCH_THRESHOLD:
        return best_peer

    return None  // Not enough improvement

Constants:

PARENT_SWITCH_THRESHOLD = 1  // Minimum depth improvement to switch parents

This means a proposed parent must offer a path at least 1 hop shallower than the current parent (under the same root) to trigger a switch. Root changes always trigger a switch regardless of depth.

What this means for tree structure: The v1 algorithm produces minimum-depth trees, which minimizes coordinate path length and hop count for coordinate-based greedy routing. However, it does not account for link quality—a high-latency or lossy link at depth 1 is preferred over a fast link at depth 2.

v2 Planned: Cost Metrics

The following cost-based parent selection is planned but not yet implemented.

Cost components:

• Latency (primary): cost_latency = round_trip_time_ms
• Packet loss (reliability): cost_loss = 1 / (1 - loss_rate) — transforms loss rate into multiplicative cost (10% loss → 1.11, 50% → 2.0)
• Bandwidth (capacity): cost_bandwidth = reference_bandwidth / actual_bandwidth

Combined cost: Weighted combination with application-tunable weights:

effective_cost = w_latency * cost_latency
               + w_loss * cost_loss
               + w_bandwidth * cost_bandwidth

Path cost to root: Recursive — each node advertises its cumulative cost, allowing neighbors to compute total path cost:

path_cost(peer) = link_cost(self, peer) + peer.path_cost_to_root

Stability threshold: Hysteresis with both absolute and relative components:

stability_threshold = base_threshold + current_cost * relative_threshold

Cost measurement: Active probing (periodic RTT measurement), passive observation (inferred from protocol message timing), and exponential smoothing (alpha = 0.1–0.3) to balance responsiveness with stability.

Implementation prerequisites: The cost-based algorithm requires changes to the TreeAnnounce wire format to carry path cost values, and a measurement subsystem for link quality metrics. See Section 10, Example 2 for how cost-based selection would affect tree structure in heterogeneous networks.

---

9. Steady State Behavior

Once converged, what does the network look like and how does it behave?

Characteristics of Steady State

Stable tree structure:

• Single agreed-upon root
• Each node has exactly one parent
• No loops exist
• All nodes reachable from root

Quiescent gossip:

• TreeAnnounce messages sent only on topology changes, not periodically
• No periodic root refresh — the tree is maintained purely by change-driven gossip
• In a stable network, gossip traffic drops to zero
• Bandwidth usage proportional to tree depth, not network size

Consistent coordinates:

• Every node knows its full path to root
• Distance calculations are accurate
• Coordinate-based greedy routing succeeds

Steady State Gossip Pattern

Normal operation (no topology changes):

Root A: No periodic announcements — announces only if topology changes
        └── Root does not refresh its timestamp periodically

Each node: Sends TreeAnnounce only when its own state changes
           └── Parent change, new peer, or peer departure

Typical gossip per node in steady state:
├── No periodic sends — tree gossip is entirely change-driven
├── Announce on parent selection change
├── Announce on peer link up/down
└── Zero gossip traffic when topology is stable

Expected Steady State Properties

Gossip volume:

Per topology change event:
├── Own declaration update: ~100 bytes
├── Delta of changed ancestors: varies
└── Total: O(100 bytes) to O(depth * 100 bytes)

In steady state (no topology changes):
├── Zero gossip traffic — no periodic refreshes
├── Traffic resumes only when links change or nodes join/depart
└── Negligible compared to application traffic

Memory usage:

Per node TreeState:
├── Own entry: ~100 bytes
├── Direct peers: ~100 bytes each
├── Ancestry entries: ~100 bytes each, O(depth) per peer
└── Total: O(peers * depth * 100 bytes)

For node with 5 peers, depth 10:
└── ~5 KB of tree state

CPU usage:

Per gossip message received:
├── Signature verification: O(ancestry_length)
├── TreeState merge: O(ancestry_length)
├── Parent re-evaluation: O(peers)
└── Total: O(peers + depth) per message

In steady state with infrequent updates:
└── Negligible CPU overhead

Monitoring Steady State

Indicators the network has converged:

1. Root stability: Same root over extended period
2. Parent stability: No parent changes in recent interval
3. Sequence number stability: Sequence numbers increment only on topology changes
4. Routing success: Coordinate-based greedy routing doesn't hit local minima

Warning signs of instability:

1. Rapid sequence increments: Node is flapping parents
2. Multiple roots visible: Partitions exist
3. Stale entries: Gossip isn't propagating
4. Frequent path-broken: Tree structure is inconsistent with reality

---

10. Worked Examples

Example 1: Small Office Network

Scenario: Five nodes (A-E) in an office. A is the router with internet, B-E are workstations. All connected via ethernet switch.

Physical topology (full mesh via switch):
    A ──── B
    │╲   ╱│
    │ ╲ ╱ │
    │  ╳  │
    │ ╱ ╲ │
    │╱   ╲│
    D ──── C ──── E

node_addr ordering: A < C < B < E < D

Tree formation:

T0: All nodes start, each is own root

T1: Links established (all pairs discover each other)

T2: Gossip exchange
    Nodes learn about A through peer announcements
    B, C, D select A as parent (direct link)
    E learns about A via peers' ancestry

T3: Converged tree (assuming equal link costs):
        A (root)
       /│\
      B C D
        │
        E

    E selects C as parent (or any direct peer with path to A)

Steady state:

• A is root
• B, C, D are direct children of A
• E is child of C (one hop to A through C)
• No periodic gossip — TreeAnnounce only on topology changes

Link failure scenario:

Link A ←→ C fails:

T1: C detects (no traffic from A, deadline expires)
    C's current TreeState still has A as root (not expired)
    C has peers B, D, E (assuming full connectivity)

T2: C queries peers for path to A
    B and D both have direct path to A
    C selects B or D as new parent (based on cost)

T3: C announces new parent to all peers
    E receives, E's path to root now goes C → B → A (or C → D → A)

T4: Reconverged tree (if C selected B):
        A (root)
       /│
      B D
      │
      C
      │
      E

Example 2: Mesh Network with Constrained Links

Scenario: Rural network with mixed connectivity. Some high-bandwidth internet links, some low-bandwidth radio links.

Physical topology:
    A ═══════ B         (═══ = fiber, 1 Gbps)
    │         ║
    │(radio)  ║(fiber)
    │ 9600bps ║
    │         ║
    C ─────── D ═══════ E
      (DSL)     (fiber)
      1 Mbps

node_addr ordering: B < A < D < E < C

Cost calculation (using bandwidth as primary):

Link costs (normalized to 1 Gbps = 1):
A ═ B: cost = 1
B ═ D: cost = 1
D ═ E: cost = 1
C — D: cost = 1000 (1 Mbps)
A ~ C: cost = 100000 (9600 bps)

Tree formation with costs:

Root = B (smallest node_addr)

Parent selection:
├── A: peers are B (cost 1), C (cost 100000)
│   └── Selects B (much lower cost)
│
├── D: peers are B (cost 1), C (cost 1000), E (cost 1)
│   └── Selects B (direct, cost 1)
│
├── E: peer is D
│   └── Path to B: E → D → B, cost = 1 + 1 = 2
│   └── Selects D
│
└── C: peers are A (cost 100000), D (cost 1000)
    └── Path through A: 100000 + 1 = 100001
    └── Path through D: 1000 + 1 = 1001
    └── Selects D (much lower cost despite higher local cost)

Resulting tree:
        B (root)
       / \
      A   D
          |\
          E C

Note: C chooses D despite A being "closer" in hops, because total path cost through D is lower.

Radio link failure:

If A ~ C radio fails:
└── No tree impact (C's parent is D, not A)
└── C loses a potential backup path, but current tree unchanged

If D — C DSL fails:
├── C loses parent
├── C's only remaining peer is A (radio)
├── C selects A as parent
├── C's path to root: C → A → B (cost 100001)
└── Tree reconverges with C as child of A

Example 3: Network Partition and Healing

Scenario: Two office sites connected by a single WAN link.

Site 1:          WAN link           Site 2:
A ─── B ─────────────────────────── E ─── F
      │                                   │
      C                                   G

node_addr ordering: A < E < B < F < C < G

Normal operation:

Root = A (global smallest)
Tree:
    A
    └── B
        ├── C
        └── E (via WAN)
            └── F
                └── G

Partition (WAN fails):

T1: B ←→ E link fails
    B detects: E unreachable
    E detects: B unreachable

T2: Site 1 state:
    Root = A (still reachable)
    Tree unchanged for A, B, C
    E's entry in B's TreeState expires

T3: Site 2 state:
    E loses path to A
    E evaluates remaining peers: F
    F has no path to A either
    E compares: node_addr_E < node_addr_F
    E becomes new root for Site 2

T4: Site 2 reconverges:
    E (root)
    └── F
        └── G

Network is now two separate trees with roots A and E.

Partition heals:

T5: WAN link restored
    B ←→ E exchange announcements

T6: E receives B's announcement:
    B's ancestry: [A, B]
    E learns: A exists, node_addr_A < node_addr_E
    E adopts A as root
    E selects B as parent

T7: E announces to F:
    E's new ancestry: [E, B, A]
    F learns about A
    F re-parents (E is still valid parent, now with path to A)

T8: F announces to G:
    Similar propagation

T9: Merged network:
    A (root)
    └── B
        ├── C
        └── E
            └── F
                └── G

Convergence time: 4 gossip rounds (depth of Site 2's subtree is 3, plus initial exchange).

---

Known Limitations (v1 Implementation)

The following limitations exist in the current implementation relative to the design described in this document. They are documented here to guide future work.

Known Limitation: Root Timeout Not Enforced

The design specifies a 60-minute root timeout (§6 partition detection) after which nodes should treat the root as departed and re-elect. The current implementation does not track root entry timestamps or perform staleness checks.

Impact: If the root node disappears permanently without a graceful disconnect, remaining nodes retain stale root state indefinitely. Nodes that lose their direct parent will re-elect locally (via handle_parent_lost()), but nodes with an intact path to a now-departed root will not detect the failure.

Required fix: Track the timestamp of the most recent root declaration in TreeState. In check_tree_state() (called every 1s from the RX loop), compare against root_timeout (default 60 min). On expiration, treat it as root loss — increment sequence number, become own root, and re-announce.

Known Limitation: No TTL on Tree Entries

The design specifies a 5-10 minute TTL on tree entries (§8 timing parameters). Peer entries in TreeState are never expired; they persist until explicitly removed by peer disconnection.

Impact: Stale ancestry information from departed nodes remains in TreeState, potentially affecting coordinate computation. In practice, this is partially mitigated by parent loss handling, but entries for non-parent peers that depart without a graceful disconnect will linger.

Required fix: Add a last_seen timestamp to peer entries in TreeState. In check_tree_state(), expire entries older than tree_entry_ttl. When entries expire, re-evaluate parent selection if the expired entry was the current parent.

Known Limitation: No Partition Detection

The design describes partition detection via gossip staleness (§6) where nodes detect isolation when root announcements stop arriving and the root entry eventually expires, triggering independent partition operation.

Impact: Without root timeout enforcement (see above), partitioned nodes cannot detect that they've lost connectivity to the root. They continue with stale coordinates rather than forming an independent partition with a local root. This affects only the case where the path to root is broken at some intermediate point — direct parent loss is handled correctly.

Required fix: Depends on root timeout implementation. Once root timeout is enforced, partition detection follows naturally: a node whose root entry expires and has no peer with a fresher root declaration is partitioned. It becomes its own root and announces, allowing the partition to converge independently.

Known Limitation: Limited Stability Mechanisms

The implementation includes basic hysteresis (PARENT_SWITCH_THRESHOLD = 1 depth difference required to switch parents), but the temporal stability mechanisms described in the design are not implemented:

• No hold timer on parent changes (minimum time before next switch)
• No sequence number advancement rate limiting
• No announcement suppression during transient topology changes
• No minimum stable state duration before re-announcing

Impact: Rapid topology changes (e.g., a flapping link) could cause excessive announcement traffic and repeated coordinate recomputation. Currently mitigated by per-peer rate limiting on TreeAnnounce sends, but the source node is not throttled.

Proposed fix: Add a hold-down timer (e.g., 5-10s) after each parent change during which further parent switches are suppressed unless the current parent is lost entirely. Track announcement rate and suppress if exceeding a threshold.

Known Limitation: Integration Test Gaps

Unit tests for TreeState and TreeCoordinate are comprehensive, and basic integration tests verify TreeAnnounce exchange and parent ancestry propagation. However, the following failure scenarios lack test coverage:

• Root node failure and network-wide re-election
• Network partition formation and independent operation
• Partition healing and root convergence
• Stale entry cleanup (depends on TTL implementation)
• Parent flapping under rapid topology changes

These tests are blocked on or related to the limitations above and should be added as each limitation is resolved.

---

Summary

The gossip-based spanning tree protocol achieves distributed coordination through:

1. Deterministic root election - Smallest node_addr, no negotiation needed
2. Local parent selection - Each node independently chooses best path to root
3. CRDT merge semantics - Conflicts resolved by sequence number, then timestamp
4. Bounded state - O(peers × depth) entries per node, not O(network size)
5. Depth-proportional convergence - Scales with tree height, not node count
6. Traffic-based failure detection - No dedicated keepalive overhead
7. Stability thresholds - Hysteresis prevents flapping on similar-cost paths

Each node maintains only its own ancestry and direct peer information—not global topology. Reachability to arbitrary destinations is provided by Bloom-guided candidate selection (bloom filters propagating up the tree), with coordinate discovery via lookup protocol and coordinate-based greedy routing for forwarding.

The protocol handles partitions gracefully (independent operation), heals automatically when connectivity returns, and adapts to heterogeneous link costs to form efficient tree structures.

---

References

FIPS Internal Documentation

• fips-spanning-tree.md — Spanning tree algorithms and data structures
• fips-mesh-operation.md — How the spanning tree fits into mesh routing
• fips-wire-formats.md — TreeAnnounce wire format

Yggdrasil Documentation

• Yggdrasil v0.5 Release Notes
• Ironwood Routing Library
• The World Tree (Yggdrasil Blog)
• Yggdrasil Implementation Overview

Academic Foundations

Virtual Coordinate Routing

• Rao, A., Ratnasamy, S., Papadimitriou, C., Shenker, S., Stoica, I. "Geographic Routing without Location Information". MobiCom 2003. Established virtual coordinate routing using network topology.

Greedy Embedding Theory

• Kleinberg, R. "Geographic Routing Using Hyperbolic Space". IEEE INFOCOM 2007. Proved every connected graph has a greedy embedding in hyperbolic space; showed spanning trees enable coordinate assignment.

• Cvetkovski, A., Crovella, M. "Hyperbolic Embedding and Routing for Dynamic Graphs". IEEE INFOCOM 2009. Dynamic embedding for nodes joining/leaving; introduced Gravity-Pressure routing for failure recovery.

• Crovella, M. et al. "On the Choice of a Spanning Tree for Greedy Embedding". Networking Science 2013. Analysis of how tree structure affects routing stretch.

• Bläsius, T. et al. "Hyperbolic Embeddings for Near-Optimal Greedy Routing". ACM Journal of Experimental Algorithmics 2020. Achieved 100% success ratio with 6% stretch on Internet graph.

Distributed Systems Primitives

• Shapiro, M., Preguiça, N., Baquero, C., Zawirski, M. "Conflict-free Replicated Data Types". SSS 2011. Formal definition of CRDTs enabling coordination-free consistency.

• Das, A., Gupta, I., Motivala, A. "SWIM: Scalable Weakly-consistent Infection-style Process Group Membership". IPDPS 2002. O(1) failure detection, O(log N) dissemination via gossip.

• Kermarrec, A-M. "Gossiping in Distributed Systems". ACM SIGOPS Operating Systems Review 2007. Framework for gossip-based protocols achieving O(log N) propagation.