fips/docs/tutorials/ground-up-mesh.md

Build a Mesh from the Ground Up

The earlier tutorials in this progression rode existing IP — your daemon reached test-us01 over the public internet through your ISP, your ISP's upstream, and however many hops separate you from the test node. That is the overlay deployment mode of FIPS: useful, but not the new ground.

This tutorial is about the other mode. Two devices, a wire (or a radio link) between them, no IP between them, and FIPS daemons on each end. The two daemons discover each other over the raw link, peer over Noise, and bring up an end-to-end mesh with addressing, naming, and reachability — all from layer 2 up. There is no DHCP, no router, no upstream. The mesh is the network.

This is the deployment mode FIPS was designed for. Overlay mode exists because riding existing IP is a useful convenience; the ground-up mode is what FIPS uniquely enables.

The two modes are not exclusive. A node can carry overlay peers and ground-up peers at the same time — different transports on the same daemon. If you have already worked through join-the-test-mesh, the static peer to test-us01 you configured there can stay in place; the Ethernet peer you add in this tutorial sits alongside it. Traffic flows through whichever path is shortest by mesh metric, and a node on one side can reach a node on the other through your machine acting as a bridge between the two.

What you'll build

   ┌──────────────────────┐    raw Ethernet frames   ┌──────────────────────┐
   │   node A             │  ─────────────────────── │   node B             │
   │   npub1aaa…          │   EtherType 0x2121       │   npub1bbb…          │
   │   fips0  fd97:..:A   │   no IP between them     │   fips0  fd97:..:B   │
   └──────────────────────┘                          └──────────────────────┘
            │                                                 │
            │       a single Ethernet cable                   │
            │       (or both NICs on the same                 │
            │       unmanaged switch — no DHCP,               │
            │       no router, no IP at all)                  │
            └─────────────────────────────────────────────────┘

Two machines, each running fips, joined by a physical Ethernet link. After the worked example:

• The two daemons have discovered each other via L2 beacons on the link, peered over Noise IK, and brought up an FMP link.
• Each fips0 adapter has a routable mesh address; each can ping the other by <npub>.fips.
• Nothing between the two machines speaks IP. The link carries raw FIPS frames at EtherType 0x2121.

The whole exercise should take about twenty minutes if you have the hardware ready.

Why ground-up

Most networking tutorials assume IP is already there: an address arrived from DHCP, a default gateway routes you onward, DNS resolves names. FIPS does not need any of that. Two devices and a way to deliver bytes between them at layer 2 is enough — FIPS supplies the rest:

• Identity: each daemon has an npub (the same kind you saw in the overlay tutorials). Nothing in the ground-up case depends on a network identity from a router; the npub is the identity.
• Addressing: the fips0 adapter takes an fd97:... ULA derived from the npub. No DHCP. No SLAAC. The address is cryptographically tied to the identity.
• Discovery: each daemon broadcasts a small beacon on the link advertising its npub; the other daemon's listener picks it up and dials in over the same link.
• Routing: the FIPS mesh layer builds its own spanning tree across whatever links it has. Add a third node (peered to either A or B) and traffic reaches it transparently.

The point is not that ground-up replaces overlay. It's that overlay is one of two modes the same daemon supports, and ground-up is what unlocks the use cases overlay cannot — ad-hoc local meshes, partitioned networks, situations where no IP infrastructure exists or can be relied on.

Prerequisites

Two devices (call them node A and node B) and a way to join them at layer 2:

• Ethernet (the worked example): a direct cable between two modern NICs (auto-MDI/MDIX handles crossover for you), or both machines on a small unmanaged switch with no DHCP server. USB-Ethernet dongles work; a typical "USB-to-RJ45" adapter is fine on either end. The link does not need to be the machine's primary network interface — a second NIC dedicated to the mesh is the cleanest setup.
• WiFi (a one-line variation, covered later): both machines associated to a common AP that has client (station) isolation off.
• Bluetooth LE (a separate worked example via a how-to, covered later): two BLE-capable Linux hosts within roughly 10 metres line of sight.

On both nodes:

• fips installed and running, per getting-started.
• A persistent identity from persistent-identity. Ephemeral identities work, but on each restart the npub regenerates and you'll have to re-check fipsctl show peers to see the new identity. Persistent makes the lesson stick.
• The daemon running with CAP_NET_RAW (the shipped systemd unit runs as root and gets this for free; running interactively from a user account requires setcap — noted at the relevant step below).

You do not need:

• An IP address on the chosen interface. The Ethernet transport opens a raw socket directly; the kernel does not need to assign an IP to the NIC.
• A default route. The mesh routes itself.
• DNS resolution between the machines via any external service. The local .fips resolver supplies names from the npubs the daemons exchange.

Step 1: Identify the link interface on each node

On each node, list the network interfaces and pick the one that sits on the link between the two machines. If it's a dedicated NIC for the mesh, that NIC has no other purpose; if it's a USB-Ethernet dongle, plug it in first so the kernel names it.

ip link show

Pick out the interface name. Common forms:

• enp3s0, eno1 — built-in NICs under predictable naming.
• eth0 — older or container-style naming.
• enxAABBCCDDEEFF — USB-Ethernet dongles often appear under this MAC-derived form.

Bring the interface up if it isn't:

sudo ip link set dev <interface> up

Confirm:

ip -br link show <interface>

You want UP and LOWER_UP in the flags. The interface does not need an IP address — LOWER_UP indicates the NIC sees carrier (cable plugged into something at the other end), and that is all the Ethernet transport needs.

For the rest of the tutorial we'll write the chosen interface as <eth>. Substitute the actual name on each node when you run the commands. Note that node A and node B may have different interface names — that is normal.

No IP needed. If your chosen interface has an address from a previous DHCP lease, leave it alone or remove it with sudo ip addr flush dev <eth> — the FIPS Ethernet transport uses raw AF_PACKET sockets that bypass the IP stack entirely. The interface needs to be up and LOWER_UP, nothing more.

Step 2: Configure the Ethernet transport on each node

Edit /etc/fips/fips.yaml on both nodes. Under transports:, add an ethernet: block. The key settings are the four discovery flags — both nodes must opt in to all four, and they default to off:

transports:
  ethernet:
    interface: "<eth>"        # the name from Step 1
    announce: true            # broadcast our beacon on the link
    discovery: true           # listen for beacons (default; shown for clarity)
    auto_connect: true        # dial peers we discover
    accept_connections: true  # accept dial-ins from peers we discover

Each flag does one thing:

• announce: true — emit a small beacon every beacon_interval_secs (default 30s) carrying our npub.
• discovery: true — listen for incoming beacons; populate a candidate-peer list keyed by source MAC and observed npub.
• auto_connect: true — when we see a beacon from an npub we have not yet peered with, initiate the outbound Noise handshake.
• accept_connections: true — when a remote npub initiates the handshake on this transport, complete it.

If only one node sets announce, the other won't see it; if only one side sets auto_connect or accept_connections, the roles are asymmetric and the link won't establish unless both are configured. The cleanest pattern for a ground-up tutorial is "all four flags on both ends."

Multiple Ethernet links. If a node has more than one physical interface that participates in the mesh, configure each one as a named instance under ethernet::

transports:
  ethernet:
    lan:
      interface: "eth0"
      announce: true
      discovery: true
      auto_connect: true
      accept_connections: true
    dongle:
      interface: "enx00aabbccddee"
      announce: true
      # ...

Each named instance runs its own socket and discovery state. A single ground-up link only needs the flat form shown first; named instances become useful when the same node bridges multiple physical segments.

Step 3: Grant the daemon permission to open raw sockets

The Ethernet transport opens an AF_PACKET SOCK_DGRAM socket bound to the chosen interface. That requires CAP_NET_RAW.

If you installed FIPS via the Debian package and run via the shipped systemd unit, the daemon runs as root and has CAP_NET_RAW already — there is nothing to do here. Skip to Step 4.

If you are running the daemon interactively as your user (a from-source / development setup), grant the capability once on the binary:

sudo setcap CAP_NET_RAW,CAP_NET_ADMIN+ep "$(which fips)"

CAP_NET_ADMIN is what the daemon needs for the fips0 TUN adapter regardless; CAP_NET_RAW is the ground-up addition. The setcap invocation only needs to be repeated when the binary is replaced.

Step 4: Restart the daemon on each node

sudo systemctl restart fips

Or, if running interactively, restart your fips invocation in whichever way you started it.

Watch the startup logs for the Ethernet transport coming up:

sudo journalctl -u fips -f --since="1 minute ago"

Look for landmarks like:

• A line indicating the Ethernet transport opened the chosen interface and started its receive loop.
• Periodic outbound beacon messages (one per beacon_interval_secs window).
• After the second beacon round on the other node, an inbound beacon parsed and a candidate-peer entry created.
• Once each side dials, a Noise handshake completion log message naming the remote npub.

Beacon interval defaults to 30s, so the first peering can take up to a minute (one beacon window per side, plus handshake). Lower the interval for the tutorial if you want faster feedback:

transports:
  ethernet:
    # ...
    beacon_interval_secs: 10  # minimum allowed

Step 5: Verify the link

On either node:

sudo fipsctl show peers

Expect one entry whose npub matches the other node and whose addresses line shows transport: ethernet. Your existing overlay peers (if any from earlier tutorials) appear alongside it. Each peer has its own row, and the link status columns show whether the Noise session is up.

sudo fipsctl show transports

Confirms that the Ethernet transport is running and shows the beacon counters incrementing. Both beacons_sent and beacons_received should be non-zero if the link is healthy.

Step 6: Reach the other node by name

On node A, ping node B by .fips name. Get node B's npub from its fipsctl show status output (it's the persistent identity you established earlier), then:

ping6 npub1bbb…long-string….fips

Expect ICMPv6 echo replies. The path is:

1. The local .fips resolver translates the npub-form name into an fd97:... mesh address (cryptographically derived from the npub on both ends — the resolver does the computation locally, with no network round trip).
2. The kernel routes the packet via fips0.
3. The FIPS daemon accepts it from the TUN, looks up the mesh route, and hands it to the FMP link to node B.
4. The Ethernet transport on node A frames the FMP packet as a raw EtherType 0x2121 Ethernet frame addressed to node B's MAC, learned from B's beacons.
5. Node B's daemon receives the frame, peels off the Ethernet/FIPS framing, and the packet emerges on node B's fips0.
6. The kernel on node B sees an inbound ICMPv6 echo and replies, and the same path runs in reverse.

If you have a hosts file with shortnames configured (see host-aliases), substitute the shortname for the full npub form.

Step 7: Try a forward composition

If node A also has the test-us01 overlay peer from join-the-test-mesh, node B can reach test-us01 through node A — even though node B has no direct internet path of its own:

On node B:

ping6 npub1qmc3cvfz0yu2hx96nq3gp55zdan2qclealn7xshgr448d3nh6lks7zel98.fips

The packet leaves B's fips0, traverses the Ethernet link to A, gets forwarded by A across the overlay UDP transport to test-us01, and the reply comes back the same way.

This is the composition the chapter intro flagged: the two deployment modes coexist on a single daemon. Node A is participating in the test mesh via the internet and in your local Ethernet mesh. From node B's perspective, the test mesh is reachable. From test-us01's perspective, B is reachable. The mesh handles the rest.

Variations

WiFi (AP mode), same shape as Ethernet

Replace <eth> with the WiFi interface name (typically wlan0 or wlp3s0) on each node. The WiFi NIC is presented as an Ethernet-class interface to the kernel by the mac80211 abstraction; the FIPS Ethernet transport opens the same AF_PACKET socket on it. No FIPS-side configuration change beyond the interface name.

What you do need on the AP side:

• Both nodes associated to the same SSID.
• Client (station) isolation must be OFF on the AP. Most consumer routers ship with it off; many guest networks and "secure" enterprise APs ship with it on. When client isolation is on, the AP refuses to forward station-to-station frames — the broadcast beacons never arrive at the other node, and discovery fails silently. If beacons aren't crossing, this is the first thing to check.

There is no FIPS-specific configuration for WiFi versus Ethernet on the daemon side; the choice is purely the adapter name.

Bluetooth LE (experimental but works)

BLE is a separate transport (transports.ble.*) with its own discovery model — L2CAP advertisements rather than raw L2 broadcasts. The shape of the tutorial is the same (advertise + scan + auto-connect + accept), but the prerequisites are different: BlueZ, bluetoothd, an HCI adapter, and the bluetooth group or capability set.

The full operator recipe is in ../how-to/set-up-bluetooth-peer.md. Mark this transport as experimental: it works in most configurations but the BLE stack has more variability than Ethernet — adapter quirks, BlueZ version differences, and the shorter range all matter.

The BLE transport is Linux-only at present; macOS and Windows builds skip it.

What you've learned

• Ground-up is the new ground. FIPS does not need any IP infrastructure between two devices to mesh them. A wire (or a radio link), CAP_NET_RAW, and a few config flags on each end are sufficient. The mesh supplies its own identity, addressing, discovery, and routing.
• Discovery is a four-flag opt-in. announce, discovery, auto_connect, and accept_connections each control one thing; both ends must agree before a link will form.
• The two modes coexist. Overlay peers and ground-up peers ride the same daemon — same FMP link layer, same FSP session layer, same fips0 adapter. A node can be a bridge between the two without any extra plumbing.
• No IP on the link. The Ethernet transport bypasses the kernel IP stack via AF_PACKET. Whether the interface has an IP address is irrelevant; whether it has carrier is what matters.
• Names work the same way. <npub>.fips resolves locally via the cryptographically-derived ULA. The resolver does not care whether the destination is reached over Ethernet, UDP overlay, or some hop chain combining both.

Troubleshooting

• No beacons received. On either node, sudo fipsctl show transports should show beacons_received incrementing every beacon_interval_secs once the other node is also running. If it stays at zero:
  • Confirm the chosen interface is LOWER_UP (carrier present).
  • Confirm the other node is announcing (its beacons_sent should be non-zero).
  • On WiFi: confirm AP client isolation is off.
  • On a switch: confirm the switch is unmanaged or that EtherType 0x2121 is not being filtered. Most consumer switches forward all EtherTypes; managed switches sometimes don't.
• Beacons received but no peer entry. The handshake is failing. Tail logs (journalctl -u fips) for Noise handshake errors. Common causes: peer ACL active and not including the remote npub (out of scope for this tutorial, but check /etc/fips/peers.allow if you have set one); daemon's clock drift large enough to fail freshness checks (rare).
• Daemon won't start with the Ethernet transport. Likely a permissions error. Check journalctl -u fips for an EPERM or "operation not permitted" message; if running interactively, confirm the binary has CAP_NET_RAW (getcap "$(which fips)").
• Beacons in both directions, peers entries on both sides, but ping6 times out. The handshake completed but the FSP session is not flowing data. Check fipsctl show peers's link status columns — if the FMP link is healthy but FSP is not, the mesh-layer side is fine and the issue is one layer up. The reach-mesh-services § Troubleshooting section covers symptoms at this level.
• AF_PACKET socket bind fails on a kernel-protected interface. Some hardened kernels (grsec, certain containers, certain VMs) restrict raw-socket access even with CAP_NET_RAW. The daemon log will name the failing syscall. The fix is host-side: relax the restriction or pick a different interface.

What's next

You now have the second deployment mode of FIPS in your hands. From here:

• Add a third node. Bring up a third machine on the same Ethernet segment, configure it identically, and watch all three nodes form a mesh. The FIPS spanning tree picks a root and routing converges within a few beacon intervals.
• Mix transports. Add an overlay peer (per join-the-test-mesh) to one of your ground-up nodes; the local mesh now reaches the test mesh through that node, and vice versa.
• Host services. Anything you do on fips0 with overlay peers — bind an HTTP server (per host-a-service), reach a service via the daemon's IPv6 adapter (per reach-mesh-services) — works identically on a ground-up mesh. The data plane is the same.

For more depth on the link-layer machinery:

• ../reference/transports.md § Ethernet — full Ethernet transport reference (counter inventory, per-instance configuration, MTU model).
• ../reference/configuration.md § Ethernet — every configuration key and its default.
• ../how-to/set-up-bluetooth-peer.md — operator recipe for the BLE variant.
• ../design/fips-transport-layer.md — the design doc that describes the per-link MTU model and why each transport is treated as link-layer rather than network-layer.