gateway: add macOS wireguard sidecar example (#51)

Add a documented macOS sidecar setup under examples so gateway traffic can be routed through a local Docker WireGuard sidecar for development and testing. Generate persistent FIPS and WireGuard key material on first run and keep those local runtime artifacts out of version control.
2026-07-22 07:48:26 +00:00 · 2026-04-10 18:36:30 +08:00
parent 0d4ffc61f0
commit 7258469b18
10 changed files with 521 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -282,6 +282,16 @@ sudo journalctl -u fips -f
 See [testing/](testing/) for Docker-based integration test harnesses
 including static topology tests and stochastic chaos simulation.

+## Examples
+
+- [examples/wireguard-sidecar-macos/](examples/wireguard-sidecar-macos/) -
+  Run a local WireGuard sidecar on macOS so `.fips` traffic can reach the mesh
+  through Docker.
+
+The macOS WireGuard sidecar only forwards FIPS IPv6 traffic destined for
+`fd00::/8` from `wg0` to `fips0`. Regular internet traffic does not transit the
+sidecar and continues to use the host network normally.
+
 ## Documentation

 Protocol design documentation is in [docs/design/](docs/design/), organized as
--- a/examples/wireguard-sidecar-macos/.gitignore
+++ b/examples/wireguard-sidecar-macos/.gitignore
@@ -0,0 +1,2 @@
+# Local runtime state generated by the WireGuard sidecar workflow.
+identity/
--- a/examples/wireguard-sidecar-macos/Dockerfile
+++ b/examples/wireguard-sidecar-macos/Dockerfile
@@ -0,0 +1,10 @@
+FROM fips-test:latest
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends vim wireguard-tools && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY entrypoint-gateway.sh /usr/local/bin/entrypoint-gateway.sh
+RUN chmod +x /usr/local/bin/entrypoint-gateway.sh
+
+ENTRYPOINT ["/usr/local/bin/entrypoint-gateway.sh"]
--- a/examples/wireguard-sidecar-macos/README.md
+++ b/examples/wireguard-sidecar-macos/README.md
@@ -0,0 +1,127 @@
+# WireGuard Sidecar For macOS
+
+This example lets macOS reach the FIPS mesh through a local Docker container.
+
+## Files
+
+- `fips-on.sh`: user-facing startup wrapper
+- `fips-host.sh`: small privileged helper for macOS networking
+- `fips-off.sh`: teardown wrapper
+- `entrypoint-gateway.sh`: container startup logic
+- `identity/fips.key` and `identity/fips.pub`: generated persistent gateway identity
+- `fips.yaml`: FIPS node config used inside the container
+
+## Configure Peers
+
+Before first use, replace the placeholder bootstrap peer in `fips.yaml` with a real peer for the mesh you want to join.
+
+## Startup Flow
+
+```text
+macOS user        fips-on.sh         Docker / fips-gateway       fips-host.sh (sudo)
+    |                 |                        |                         |
+    | ./fips-on.sh    |                        |                         |
+    |---------------> |                        |                         |
+    |                 | generate fips.key      |                         |
+    |                 | generate client keys   |                         |
+    |                 |----------------------->| bind-mount identity/    |
+    |                 | start/recreate container                         |
+    |                 |----------------------->| entrypoint starts        |
+    |                 |                        | generate server keys     |
+    |                 |                        | wait for client.pub      |
+    |                 |                        | bring up wg0             |
+    |                 | poll `wg show wg0`    |                         |
+    |                 |<-----------------------|                         |
+    |                 | sudo fips-host.sh on  |                         |
+    |                 |------------------------------------------------->|
+    |                 |                        | write /etc/wireguard     |
+    |                 |                        | wg-quick up fips0        |
+    |                 |                        | write /etc/resolver/fips |
+    |                 |                        | flush DNS                |
+    |                 |<-------------------------------------------------|
+    | ready           |                        |                         |
+```
+
+## What `fips-on.sh` Does
+
+1. Creates `identity/` and `identity/wireguard/` if needed.
+2. Generates `identity/fips.key` and `identity/fips.pub` with `fipsctl keygen --dir /etc/fips` if the gateway does not already have a persistent identity.
+3. Generates the host WireGuard client keypair if missing.
+4. Fixes ownership on the generated client key files so the Docker bind mount stays readable on macOS.
+5. Starts `fips-gateway` with `docker compose up -d --build --force-recreate`.
+6. Waits until the container has created and brought up `wg0`.
+7. Calls the privileged helper to configure host networking:
+   - writes `/etc/wireguard/fips0.conf`
+   - runs `wg-quick up fips0`
+   - writes `/etc/resolver/fips`
+   - flushes macOS DNS caches
+
+## What Happens In The Container
+
+`entrypoint-gateway.sh`:
+
+1. Generates the server WireGuard keypair on first run.
+2. Waits briefly for `identity/wireguard/client.pub` from the host.
+3. Writes the container-side `wg0` config.
+4. Brings up `wg0` on UDP port `51820`.
+5. Starts `fips --config /etc/fips/fips.yaml` using the bind-mounted `identity/fips.key`.
+6. Enables forwarding and NAT66 from `wg0` toward `fips0` for FIPS
+   `fd00::/8` destinations only.
+
+Only FIPS traffic for `fd00::/8` is forwarded through the sidecar. Regular
+internet traffic still uses the macOS host network and does not route through
+`wg0` or `fips0`.
+
+## Gateway FIPS Identity
+
+`fips-on.sh` generates `identity/fips.key` and `identity/fips.pub` on first run by calling:
+
+```bash
+docker run --rm \
+  -v "$PWD/identity:/etc/fips" \
+  fips-test:latest \
+  fipsctl keygen --dir /etc/fips
+```
+
+That key is then bind-mounted into `/etc/fips/fips.key`, so the gateway keeps
+the same FIPS identity even though `fips-on.sh` uses `docker compose up
+--force-recreate`.
+
+Delete `identity/fips.key` if you want the next `./fips-on.sh` run to create a
+fresh gateway identity.
+
+## Why `sudo` Is Still Needed
+
+Only `fips-host.sh` needs root, because macOS requires it for:
+
+- `/etc/wireguard/fips0.conf`
+- `/etc/resolver/fips`
+- `wg-quick up/down`
+- DNS cache flushes
+
+Everything else runs as the normal user.
+
+## Teardown
+
+Run:
+
+```bash
+./fips-off.sh
+```
+
+This:
+
+1. Calls `sudo ./fips-host.sh off` to remove the host WireGuard and DNS config.
+2. Stops the Docker container.
+
+## Generated Files
+
+These are local runtime artifacts under `identity/` and should not be committed:
+
+- `identity/fips.key`
+- `identity/fips.pub`
+- `identity/wireguard/client.key`
+- `identity/wireguard/client.pub`
+- `identity/wireguard/server.key`
+- `identity/wireguard/server.pub`
+- `identity/wireguard/wg0.conf`
--- a/examples/wireguard-sidecar-macos/docker-compose.yml
+++ b/examples/wireguard-sidecar-macos/docker-compose.yml
@@ -0,0 +1,25 @@
+services:
+  fips-gateway:
+    build: .
+    image: fips-gateway:latest
+    container_name: fips-gateway
+    hostname: fips-gateway
+    privileged: true
+    devices:
+      - /dev/net/tun:/dev/net/tun
+    sysctls:
+      - net.ipv6.conf.all.disable_ipv6=0
+      - net.ipv6.conf.all.forwarding=1
+      - net.ipv4.ip_forward=1
+    ports:
+      - "5354:5354/udp"
+      - "5354:5354/tcp"
+      - "2222:22/tcp"
+      - "51820:51820/udp"
+    volumes:
+      - ./fips.yaml:/etc/fips/fips.yaml:ro
+      - ./identity/fips.key:/etc/fips/fips.key
+      - ./identity:/etc/fips/identity
+    environment:
+      - RUST_LOG=info
+    restart: unless-stopped
--- a/examples/wireguard-sidecar-macos/entrypoint-gateway.sh
+++ b/examples/wireguard-sidecar-macos/entrypoint-gateway.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+# Gateway entrypoint: WireGuard tunnel + FIPS daemon.
+set -e
+
+CONFIG="/etc/fips/fips.yaml"
+WG_DIR="/etc/fips/identity/wireguard"
+WG_CONF="$WG_DIR/wg0.conf"
+
+# ── Generate WireGuard keys on first run ───────────────────────
+if [ ! -f "$WG_DIR/server.key" ]; then
+    echo "Generating WireGuard keypair..."
+    mkdir -p "$WG_DIR"
+    wg genkey | tee "$WG_DIR/server.key" | wg pubkey > "$WG_DIR/server.pub"
+    chmod 600 "$WG_DIR/server.key"
+fi
+
+# Wait for client public key (written by fips-on.sh)
+echo "Waiting for WireGuard client key..."
+for i in $(seq 1 60); do
+    if [ -f "$WG_DIR/client.pub" ]; then
+        break
+    fi
+    sleep 0.5
+done
+
+if [ ! -f "$WG_DIR/client.pub" ]; then
+    echo "WARNING: No client public key found, starting without WireGuard"
+else
+    SERVER_KEY=$(cat "$WG_DIR/server.key")
+    CLIENT_PUB=$(cat "$WG_DIR/client.pub")
+
+    # Write WireGuard config
+    cat > "$WG_CONF" <<EOF
+[Interface]
+PrivateKey = $SERVER_KEY
+ListenPort = 51820
+
+[Peer]
+PublicKey = $CLIENT_PUB
+AllowedIPs = 10.99.0.2/32, fc00::/64
+EOF
+
+    # Bring up WireGuard
+    ip link add wg0 type wireguard
+    wg setconf wg0 "$WG_CONF"
+    ip addr add 10.99.0.1/24 dev wg0
+    ip -6 addr add fc00::1/64 dev wg0
+    ip link set wg0 up
+
+    # Enable IPv6 forwarding (fd00::/8 from wg0 → fips0)
+    sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null 2>&1 || true
+    sysctl -w net.ipv4.ip_forward=1 >/dev/null 2>&1 || true
+
+    # NAT66: rewrite source of packets from wg0 to this node's FIPS address
+    # so the FIPS TUN accepts them (it only processes fd00::/8 sources).
+    # Responses are automatically de-NATed back to fc00::2 by conntrack.
+    ip6tables -t nat -A POSTROUTING -o fips0 -s fc00::/64 -j MASQUERADE
+
+    # Also allow WireGuard peer to send to fd00::/8 destinations
+    # (AllowedIPs already permits fc00::2, add fd00::/8 for forwarded traffic)
+
+    echo "WireGuard up: 10.99.0.1/24, fc00::1/64, port 51820 (NAT66 active)"
+fi
+
+# ── Start background services ──────────────────────────────────
+dnsmasq
+/usr/sbin/sshd
+
+# ── Start FIPS ─────────────────────────────────────────────────
+exec fips --config "$CONFIG"
--- a/examples/wireguard-sidecar-macos/fips-host.sh
+++ b/examples/wireguard-sidecar-macos/fips-host.sh
@@ -0,0 +1,107 @@
+#!/bin/bash
+# Privileged macOS host networking helper for fips-on/off.sh.
+set -euo pipefail
+
+WG_CONF="/etc/wireguard/fips0.conf"
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "Error: run with sudo"
+    exit 1
+fi
+
+fix_key_perms() {
+    local real_user="$1"
+    local wg_dir="$2"
+    local file
+
+    for file in "$wg_dir/client.key" "$wg_dir/client.pub"; do
+        [ -e "$file" ] || continue
+        chown "$real_user" "$file"
+        chmod 600 "$file"
+    done
+}
+
+host_on() {
+    local script_dir="$1"
+    local wg_dir="$script_dir/identity/wireguard"
+    local client_key
+    local server_pub
+
+    if [ ! -f "$wg_dir/client.key" ] || [ ! -f "$wg_dir/server.pub" ]; then
+        echo "Error: missing WireGuard keys in $wg_dir"
+        exit 1
+    fi
+
+    client_key="$(cat "$wg_dir/client.key")"
+    server_pub="$(cat "$wg_dir/server.pub")"
+
+    wg-quick down fips0 2>/dev/null || true
+
+    for svc in Wi-Fi Ethernet; do
+        networksetup -setsocksfirewallproxystate "$svc" off 2>/dev/null || true
+    done
+
+    echo "Configuring WireGuard tunnel..."
+    mkdir -p /etc/wireguard
+
+    cat > "$WG_CONF" <<EOF
+[Interface]
+PrivateKey = $client_key
+Address = 10.99.0.2/24, fc00::2/64
+
+[Peer]
+PublicKey = $server_pub
+Endpoint = 127.0.0.1:51820
+AllowedIPs = fd00::/8, fc00::1/128, 10.99.0.1/32
+PersistentKeepalive = 25
+EOF
+
+    chmod 600 "$WG_CONF"
+    wg-quick up fips0
+
+    echo "Configuring macOS DNS resolver for .fips..."
+    mkdir -p /etc/resolver
+    cat > /etc/resolver/fips <<EOF
+nameserver 127.0.0.1
+port 5354
+EOF
+
+    dscacheutil -flushcache 2>/dev/null || true
+    killall -HUP mDNSResponder 2>/dev/null || true
+}
+
+host_off() {
+    echo "Stopping WireGuard tunnel..."
+    wg-quick down fips0 2>/dev/null || true
+    rm -f "$WG_CONF"
+    echo "  fips0 tunnel removed"
+
+    echo "Clearing SOCKS proxy..."
+    for svc in Wi-Fi Ethernet; do
+        networksetup -setsocksfirewallproxystate "$svc" off 2>/dev/null || true
+    done
+
+    echo "Removing .fips DNS resolver..."
+    rm -f /etc/resolver/fips
+    rmdir /etc/resolver 2>/dev/null || true
+
+    echo "Flushing DNS cache..."
+    dscacheutil -flushcache 2>/dev/null || true
+    killall -HUP mDNSResponder 2>/dev/null || true
+}
+
+case "${1:-}" in
+    fix-key-perms)
+        fix_key_perms "$2" "$3"
+        ;;
+    on)
+        host_on "$2"
+        ;;
+    off)
+        host_off
+        ;;
+    *)
+        echo "Usage: $0 {fix-key-perms <user> <wg-dir>|on <script-dir>|off}"
+        exit 1
+        ;;
+esac
--- a/examples/wireguard-sidecar-macos/fips-off.sh
+++ b/examples/wireguard-sidecar-macos/fips-off.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Stop the FIPS gateway and restore macOS network settings.
+#
+# Safe to run multiple times (idempotent).
+#
+# Usage: ./fips-off.sh
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HOST_HELPER="$SCRIPT_DIR/fips-host.sh"
+
+run_as_real_user() {
+    if [ "$(id -u)" -eq 0 ] && [ -n "${SUDO_USER:-}" ]; then
+        sudo -u "$SUDO_USER" "$@"
+    else
+        "$@"
+    fi
+}
+
+run_host_helper() {
+    if [ "$(id -u)" -eq 0 ]; then
+        "$HOST_HELPER" "$@"
+    else
+        sudo "$HOST_HELPER" "$@"
+    fi
+}
+
+run_host_helper off
+
+echo "Stopping FIPS gateway container..."
+run_as_real_user docker compose -f "$SCRIPT_DIR/docker-compose.yml" down 2>/dev/null || true
+echo "  Container stopped"
+
+echo ""
+echo "FIPS gateway is OFF. macOS network restored."
--- a/examples/wireguard-sidecar-macos/fips-on.sh
+++ b/examples/wireguard-sidecar-macos/fips-on.sh
@@ -0,0 +1,99 @@
+#!/bin/bash
+# Start the FIPS gateway and configure macOS to reach the mesh.
+#
+# Sets up:
+#   1. WireGuard tunnel from macOS to the FIPS Docker container
+#   2. macOS DNS resolver for .fips names
+#   3. fd00::/8 routed through the WireGuard tunnel
+#
+# Safe to run multiple times (idempotent).
+#
+# Usage: ./fips-on.sh
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+IDENTITY_DIR="$SCRIPT_DIR/identity"
+WG_DIR="$IDENTITY_DIR/wireguard"
+HOST_HELPER="$SCRIPT_DIR/fips-host.sh"
+REAL_USER="${SUDO_USER:-$USER}"
+
+run_as_real_user() {
+    if [ "$(id -u)" -eq 0 ] && [ -n "${SUDO_USER:-}" ]; then
+        sudo -u "$SUDO_USER" "$@"
+    else
+        "$@"
+    fi
+}
+
+run_host_helper() {
+    if [ "$(id -u)" -eq 0 ]; then
+        "$HOST_HELPER" "$@"
+    else
+        sudo "$HOST_HELPER" "$@"
+    fi
+}
+
+# ── 1. Generate persistent FIPS identity if missing ───────────
+mkdir -p "$IDENTITY_DIR" "$WG_DIR"
+
+if [ ! -f "$IDENTITY_DIR/fips.key" ]; then
+    echo "Generating FIPS identity keypair..."
+    run_as_real_user docker run --rm \
+        -v "$IDENTITY_DIR:/etc/fips" \
+        fips-test:latest \
+        fipsctl keygen --dir /etc/fips
+fi
+
+# ── 2. Generate WireGuard client keys ─────────────────────────
+mkdir -p "$WG_DIR"
+
+if [ ! -f "$WG_DIR/client.key" ]; then
+    echo "Generating WireGuard client keypair..."
+    run_as_real_user sh -c 'umask 077; wg genkey | tee "$1/client.key" | wg pubkey > "$1/client.pub"' sh "$WG_DIR"
+elif [ ! -f "$WG_DIR/client.pub" ]; then
+    echo "Regenerating WireGuard client public key..."
+    run_as_real_user sh -c 'wg pubkey < "$1/client.key" > "$1/client.pub" && chmod 600 "$1/client.pub"' sh "$WG_DIR"
+fi
+
+# Keep bind-mounted client keys readable to Docker Desktop on macOS.
+run_host_helper fix-key-perms "$REAL_USER" "$WG_DIR"
+
+# ── 3. Start Docker container ─────────────────────────────────
+echo "Starting FIPS gateway container..."
+run_as_real_user docker compose -f "$SCRIPT_DIR/docker-compose.yml" up -d --build --force-recreate
+
+echo "Waiting for container..."
+for i in $(seq 1 30); do
+    if run_as_real_user docker exec fips-gateway wg show wg0 >/dev/null 2>&1; then
+        break
+    fi
+    sleep 1
+done
+
+if ! run_as_real_user docker exec fips-gateway wg show wg0 >/dev/null 2>&1; then
+    echo "Error: WireGuard not ready in container after 30s"
+    echo "Check: docker logs fips-gateway"
+    exit 1
+fi
+
+if [ -f "$WG_DIR/server.pub" ]; then
+    echo "  Server pubkey: $(cat "$WG_DIR/server.pub")"
+fi
+
+# ── 4. Configure host networking ──────────────────────────────
+run_host_helper on "$SCRIPT_DIR"
+
+# ── Done ──────────────────────────────────────────────────────
+echo ""
+echo "FIPS gateway is ON."
+echo ""
+echo "  WireGuard:  fips0 tunnel active (fd00::/8 routed)"
+echo "  DNS:        .fips names resolve via localhost:5354"
+echo ""
+echo "Usage:"
+echo "  ping6 -c3 \$(dig +short AAAA your-bootstrap-peer.fips @127.0.0.1 -p 5354)"
+echo ""
+echo "  docker exec fips-gateway fipsctl show status"
+echo "  docker exec fips-gateway fipsctl show peers"
+echo ""
+echo "To stop: ./fips-off.sh"
--- a/examples/wireguard-sidecar-macos/fips.yaml
+++ b/examples/wireguard-sidecar-macos/fips.yaml
@@ -0,0 +1,36 @@
+node:
+  identity:
+    persistent: true
+
+tun:
+  enabled: true
+  name: fips0
+  mtu: 1280
+
+dns:
+  enabled: true
+  bind_addr: "0.0.0.0"
+  port: 5354
+
+transports:
+  udp:
+    bind_addr: "0.0.0.0:2121"
+  tcp:
+    bind_addr: "0.0.0.0:8443"
+
+peers:
+  - npub: "npub1zv58cn7v83mxvttl70w5fwjwuclfmntv9cnmv5wmz2nzz88u5urqvdx96n"
+    alias: "fips.v0l.io"
+    addresses:
+      - transport: tcp
+        addr: "fips.v0l.io:8443"
+      - transport: udp
+        addr: "fips.v0l.io:2121"
+    connect_policy: auto_connect
+
+  - npub: "npub1qmc3cvfz0yu2hx96nq3gp55zdan2qclealn7xshgr448d3nh6lks7zel98"
+    alias: "fips-test-node"
+    addresses:
+      - transport: udp
+        addr: "217.77.8.91:2121"
+    connect_policy: auto_connect