29 Commits

Author SHA1 Message Date
Johnathan Corgan
a16370e78d Update changelog, version, and design docs for v0.2.0
Bump version to 0.2.0 and finalize changelog with discovery rework,
Tor transport, connect/disconnect commands, reproducible builds, and
12 bug fixes.

Update design documentation for discovery protocol rework:
- fips-wire-formats.md: remove visited bloom filter from LookupRequest,
  update size calculations
- fips-mesh-operation.md: replace flooding description with bloom-guided
  tree routing, add retry/backoff/rate-limiting subsections
- fips-configuration.md: add 5 new discovery config parameters, update
  control socket description for connect/disconnect commands
2026-03-22 20:26:09 +00:00
Johnathan Corgan
537eaf0db7 Fix tarball reproducibility: default SOURCE_DATE_EPOCH and sort entries
build-tarball.sh only applied --mtime when SOURCE_DATE_EPOCH was
explicitly set, and did not sort tar entries. This produced different
tarballs across builds due to varying timestamps and filesystem
ordering. Default SOURCE_DATE_EPOCH to the git commit timestamp and
add --sort=name for deterministic output.
2026-03-22 20:26:00 +00:00
Johnathan Corgan
6aff490251 Improve discovery protocol: bloom-guided tree routing with fallback
Replace discovery flooding with bloom-filter-guided tree routing:
lookups sent only to tree peers (parent + children) whose bloom
filter contains the target. If no tree peer matches, fall back to
non-tree peers with bloom matches before dropping the request. This
produces single-path forwarding through the spanning tree (90%
traffic reduction) while recovering from dead ends caused by stale
bloom filters, tree restructuring, or transit node failures.

Remove visited bloom filter from LookupRequest wire format (-257
bytes per request). Tree routing is inherently loop-free; request_id
dedup handles edge cases during tree restructuring. Add response-
forwarded flag to prevent response routing loops from convergent
request paths.

Add originator-side exponential backoff (30s base, 300s cap) after
lookup timeouts and bloom misses. Backoff resets on topology changes
(parent switch, new peer, first RTT, reconnection).

Add transit-side per-target rate limiting (2s minimum interval) for
forwarded lookups as defense-in-depth.

Add discovery retry within the timeout window (default: send at T=0,
retry at T=5s, fail at T=10s) to compensate for single-path fragility.
Lookups with zero eligible tree peers fail immediately.

Improve discovery logging: promote key events to info (initiation,
success, timeout with failure count). Add debug logging for dedup,
pending packet retry, backoff suppression, forward rate limiting,
and backoff reset.

New config: discovery.backoff_base_secs, backoff_max_secs,
forward_min_interval_secs, retry_interval_secs, max_attempts.
New stats: req_backoff_suppressed, req_forward_rate_limited,
req_bloom_miss, req_no_tree_peer, req_fallback_forwarded.
Removed: req_already_visited, visited bloom filter.
2026-03-22 20:02:00 +00:00
Origami74
7a643a9ac3 Add Nostr release publishing to OpenWrt package workflow
Add versioning job with release channel detection, Blossom upload,
NIP-94 event publishing, and relay verification. Replace setup-zig
action with direct download for act runner compatibility. Pin
cargo-zigbuild and zig versions for reproducible builds.
2026-03-21 06:56:46 -07:00
Origami74
c164de8808 Add reproducible build infrastructure
Pin Rust toolchain to 1.94.0 via rust-toolchain.toml for deterministic
compiler output across environments. Set SOURCE_DATE_EPOCH from git
commit timestamp in CI workflows and packaging scripts to normalize
embedded timestamps.

Make packaging archives reproducible: pass --mtime=@$SOURCE_DATE_EPOCH
to tar in .deb, .ipk, and systemd tarball builds. Normalize ownership
to root:root in systemd tarballs. Pin cargo-zigbuild to 0.19.8 for
cross-compilation stability.

Add SHA-256 hash output to CI build and OpenWrt packaging workflows
for binary verification.
2026-03-20 20:36:02 -07:00
Johnathan Corgan
fed6cc6987 Fix chaos sim DNS pipeline, ephemeral npub tracking, and analysis
Three infrastructure bugs fixed:

- DNS resolution never populated the identity cache because Docker
  overwrites /etc/resolv.conf at container start. Fix: bind-mount
  resolv.conf in the chaos compose template, matching the static and
  sidecar test configs.

- Traffic generator used config-time npubs for iperf3 targets, but
  ephemeral nodes generate fresh keypairs at startup. Fix: share the
  peer churn manager's runtime npub cache with the traffic manager.

- Log analysis had no discovery counters and used wrong patterns.
  Fix: add discovery section with correct log message matching.

Also adds timestamped output directories (YYYYMMDD-HHMMSS-scenario),
coord_ttl_secs override in maelstrom.yaml, FIPS_SIM_OUTPUT env var
support, and maelstrom-sparse scenario for sparse topology testing.
2026-03-20 16:14:10 +00:00
Johnathan Corgan
5053cf673d Add connect/disconnect control commands and maelstrom chaos scenario
Add runtime peer management to the FIPS daemon via control socket
commands, and a new chaos simulation scenario that exercises dynamic
topology mutation with ephemeral node identities.

Daemon (connect/disconnect commands):
- Extend control socket Request with optional params field
- Add commands.rs module for mutating command dispatch, separate from
  read-only queries
- Add api_connect() on Node: builds ephemeral PeerConfig (no auto-
  reconnect), pre-seeds identity cache, reuses initiate_peer_connection
- Add api_disconnect() on Node: calls remove_active_peer(), clears
  retry_pending to suppress reconnection
- Route non-show_* commands to async command dispatch in rx_loop

fipsctl CLI:
- Add Connect and Disconnect subcommands accepting npub or hostname
- Resolve hostnames from /etc/fips/hosts before sending to daemon
- Refactor socket I/O into reusable send_request helper

Chaos simulator (maelstrom scenario):
- Add PeerChurnManager: periodically disconnects a random active link
  and connects a random unconnected node pair via control socket
- Add send_command() to control.py using base64-encoded JSON payloads
  to avoid shell quoting issues in docker exec
- Add PeerChurnConfig to scenario with interval and ephemeral_fraction
- Ephemeral identity support: nodes configured without nsec generate
  fresh keypairs on restart; simulator queries show_status for new
  npub and updates its cache via on_node_restart callback
- Add maelstrom.yaml: all chaos dimensions (netem, link flaps, node
  churn, peer topology churn, traffic) with 50% ephemeral identity
2026-03-20 05:44:31 +00:00
Johnathan Corgan
8d51dbd268 Move K8s sidecar from testing/ to examples/
The K8s sidecar is a standalone example, not a test harness — it has
no CI integration unlike testing/sidecar/. Move it to examples/
alongside sidecar-nostr-relay/ where it belongs. Update internal path
references in Dockerfile, build.sh, and README.
2026-03-19 19:36:39 +00:00
Johnathan Corgan
9e63b42bd9 Consolidate Docker test harness infrastructure
Replace 4 near-identical per-harness Docker setups with unified shared
infrastructure. Net result: -463 lines across 55 files, faster CI
(8.5 min vs ~13.5 min), 14 scenarios (down from 21).

Unified Docker image (testing/docker/):
- Single Dockerfile (trixie-slim) with FIPS_TEST_MODE env var for
  mode dispatch: default, chaos, sidecar, tor-socks5, tor-directory
- Single entrypoint.sh with conditional logic per mode
- Replaces 5 Dockerfiles, 3 entrypoints, 4 resolv.conf copies

Shared build and libraries (testing/scripts/, testing/lib/):
- testing/scripts/build.sh: single build script with macOS zigbuild
  support, replaces 4 per-harness copies
- testing/lib/derive_keys.py: shared key derivation module, replaces
  3 copies of derive-keys.py
- testing/lib/log_analysis.py: shared log analysis extracted from
  chaos sim/logs.py, with CLI interface and rekey cutover tracking
- testing/lib/wait-converge.sh: shared convergence wait helpers
  (wait_for_links, wait_for_peers) using fipsctl JSON polling

Scenario consolidation:
- Remove 7 redundant scenarios: tcp-chain (subsumed by tcp-mesh),
  tcp-only (subsumed by tcp-mesh), chaos-10 (replaced by
  churn-mixed --nodes 10), churn-10/churn-20/churn-20-mixed
  (subsumed by parameterized churn-mixed), cost-mixed-7node
  (overlaps mixed-technology)
- Add churn-mixed scenario with --nodes flag for scale testing
- Reduce idle scenario durations: smoke-10 60s→30s,
  ethernet-only 90s→30s, cost-avoidance 120s→45s,
  depth-vs-cost 120s→45s, bottleneck-parent 120s→60s,
  mixed-technology 180s→90s

CI updates:
- ci-local.sh: unified image build, structured chaos suite entries
  with per-scenario flags, --skip-build for sidecar
- ci.yml: shared binary install + image build step, updated scenario
  matrix, chaos_flags support for parameterized scenarios
- Add tcp-mesh and congestion-stress to CI matrix
- Static ping test uses active peer convergence detection instead
  of hardcoded 5s sleep

Chaos infrastructure improvements (from discovery-rework branch):
- Pre-built Docker image instead of per-service build at scale
- --nodes flag in chaos.sh for runtime topology size override
2026-03-19 18:08:36 +00:00
Origami74
c8b7459fbc Add sidecar-nostr-relay example and fix openwrt CI path
Add a complete example running a strfry Nostr relay exclusively over
the FIPS mesh using the sidecar pattern. Includes Docker Compose
stack, network isolation via iptables, .fips DNS resolution, nak CLI,
build script with cross-compilation support, and documentation.

Also fix the package-openwrt.yml workflow path to match the
openwrt → openwrt-ipk directory rename.
2026-03-19 16:23:13 +00:00
Kieran
8c349f524e Add Kubernetes sidecar deployment
Adds a complete Kubernetes sidecar setup under testing/k8s/ that runs
FIPS as a sidecar container, injecting the mesh TUN interface (fips0)
into any co-located app container via shared pod network namespace.

- Multi-stage Dockerfile builds fips and fipsctl from source (debian
  trixie / rust slim-trixie), producing a minimal runtime image with
  iproute2, iptables, dnsmasq, and the two binaries
- entrypoint.sh generates fips.yaml from environment variables, rewrites
  /etc/resolv.conf to route .fips DNS through dnsmasq, applies optional
  iptables isolation, clamps TCP MSS, then starts dnsmasq and execs the
  daemon
- pod.yaml annotated example Pod manifest with Secret, sysctl tuning,
  readiness/liveness probes, and resource limits
- scripts/build.sh convenience wrapper for docker build
- testing/k8s/.dockerignore keeps the build context small
- README.md usage guide covering quick start, isolation modes, env vars,
  multi-peer JSON config, troubleshooting
2026-03-18 15:26:43 +00:00
Johnathan Corgan
0e098cadad Add top-level packaging Makefile and README 2026-03-17 14:57:01 +00:00
Johnathan Corgan
ebabb6e93c Update changelog for post-v0.1.0 fixes and remove no-op clone 2026-03-17 14:45:29 +00:00
Kieran
a5130b357e Fix ICMPv6 Packet Too Big source address for PMTUD
The PTB source was set to the local FIPS address, which Linux ignores
(loopback PTB security check). Change the source to the original
packet's destination (the remote peer) so the kernel sees it as coming
from a remote router and honors the PMTU update.

Add test coverage for the PTB source address invariant in both the
icmp unit tests and the node integration tests.
2026-03-17 02:14:54 +00:00
Johnathan Corgan
6e03adde72 Replace > 0 counter guards with has_prev_rr flag for delta computation
The loss and goodput delta guards used prev_rr_highest_counter > 0 and
prev_rr_cum_bytes > 0, which conflates "never received a report" with
"received a report where the value was 0." Since Noise transport
counters start at 0, this is semantically wrong. Use a dedicated
has_prev_rr boolean flag instead. Fixes GitHub issue #14 (Bug B).
2026-03-16 17:48:55 +00:00
Johnathan Corgan
1020f90828 Fix reverse delivery ratio using per-interval deltas instead of lifetime cumulative
The reverse delivery ratio was computed as cumulative_packets_recv /
highest_counter (lifetime values), while the forward ratio correctly
used per-interval deltas. This made ETX increasingly unresponsive over
session lifetime — early loss haunted the ratio forever, late loss
barely moved it.

Replace set_delivery_ratio_reverse() with update_reverse_delivery()
that tracks previous snapshot state and computes deltas, matching the
forward ratio's approach. Fixes GitHub issue #14 (Bug A).
2026-03-16 17:30:28 +00:00
Johnathan Corgan
0f24333c0d Fix post-rekey jitter spikes from drain-window frames
After rekey cutover, frames encrypted with the old session can still
arrive during the 10s drain window. These carry large sender timestamps
from the old session, producing enormous transit deltas that spike the
EWMA jitter estimator (observed 2,000-7,000ms spikes on UDP links).

Add a time-based grace period (DRAIN_WINDOW_SECS + 5s = 15s) after
rekey that suppresses jitter updates until drain-window frames have
flushed. Baselines still update every frame so calculation resumes
cleanly after grace expires.

Fixes #10
2026-03-16 12:41:41 +00:00
Johnathan Corgan
ef73e54316 Update changelog with post-v0.1.0 bug fixes and additions 2026-03-16 02:39:01 +00:00
Johnathan Corgan
8697897047 Update changelog with post-v0.1.0 bug fixes and additions 2026-03-16 02:33:14 +00:00
Johnathan Corgan
9e11a8e7ba Return NOERROR for A queries on valid .fips names instead of NXDOMAIN
The DNS responder returned NXDOMAIN for non-AAAA queries (e.g. A record)
on valid .fips names. This caused resolvers like nslookup and tinyproxy
to give up without attempting AAAA, since NXDOMAIN means "name does not
exist." Return NOERROR with an empty answer section instead, which tells
the client the name exists but has no records of that type.

Fixes jmcorgan/fips#9.
2026-03-16 02:28:31 +00:00
Kieran
e8ef15acb7 Fix stale session cleanup and identity cache pre-seeding
Cherry-picked from v0l PR #6 (issue #5):

1. remove_active_peer() now removes the end-to-end session from
   self.sessions when evicting a peer. The stale Established entry
   caused initiate_session() to silently return Ok(()) via the
   is_established() guard, preventing session re-establishment after
   link-layer reconnection.

2. initiate_peer_connections() pre-seeds the identity cache from
   configured peer npubs at startup, so TUN packets can be dispatched
   immediately without waiting for handshake completion.

3. schedule_reconnect() preserves accumulated backoff when a retry
   entry already exists, preventing exponential backoff reset on
   repeated link-dead cycles.

Includes regression tests for all three fixes.
2026-03-16 02:06:58 +00:00
Johnathan Corgan
5f7fe989f3 Fix rekey dual-initiation, parent selection, peer reconnect, and control socket
Rekey dual-initiation race (FMP + FSP):
When both sides' rekey timers fire simultaneously on high-latency links
(Tor ~700ms RTT), both msg1s cross in flight before dampening can
suppress the second initiation. Each node acts as both initiator and
responder, with set_pending_session() from the responder path destroying
the initiator's in-progress state. Each side ends up with a
pending_new_session from a different Noise handshake, causing AEAD
verification failure after cutover and link death. Fix: deterministic
tie-breaker in both FMP msg1 handler and FSP SessionSetup handler
(smaller NodeAddr wins as initiator). Also guard against pending session
overwrite from retransmitted msg1s.

Parent selection SRTT gate:
Peers without MMP RTT data are excluded from parent candidacy via
has_srtt() filter, preventing freshly reconnected peers from appearing
artificially attractive. First-RTT triggers immediate parent re-eval.
The gate in evaluate_parent() skips unmeasured candidates when any peer
has cost data, but falls back to default cost 1.0 during cold start
when no peer has MMP data yet.

Auto-reconnect on all peer removal paths:
The excessive-decryption-failure and peer-restart epoch-mismatch removal
paths were missing schedule_reconnect() calls, causing auto-connect peers
to be permanently abandoned after those events.

Control socket accessibility:
Socket and parent directory chowned to root:fips with mode 0770/0750 so
group members can use fipsctl/fipstop without root.

Log level changes:
Default RUST_LOG changed to debug in systemd service files. "Unknown
session index" log reduced from debug to trace.
2026-03-16 00:51:55 +00:00
Johnathan Corgan
324535e76d Make auto-connect peers retry indefinitely on initial connection failure
Previously, static peers configured with AutoConnect gave up after 6
attempts (1 initial + 5 retries). If the remote peer was offline at
startup, the node permanently abandoned the connection. The reconnect
path (after MMP link-dead) already retried indefinitely but only
activated after a peer had been previously connected.

Remove the reconnect parameter from schedule_retry() and always set
reconnect=true when creating retry entries, since only auto-connect
peers reach this code path. The 300s backoff cap prevents resource waste.
The max_retries=0 config still works as an explicit kill switch.
2026-03-15 18:43:11 +00:00
Johnathan Corgan
959134657f Fix FSP rekey cutover race and MMP metric discontinuity
The FSP XK rekey handshake has a race condition where the initiator
can cut over (K-bit flip) and send data encrypted with the new session
before msg3 reaches the responder. The responder has no pending session
yet, so K-bit detection fails and packets are dropped.

Defer FSP initiator cutover by 2 seconds after handshake completion
(FSP_CUTOVER_DELAY_MS) to give msg3 time to traverse the mesh. FMP
(IK, 2 messages) is unaffected since the responder completes during
msg1 processing.

Also fix MMP metric corruption after rekey cutover: the new session
starts with counter 0 but MMP state carries highest_counter and
GapTracker.expected_next from the old session, producing false
reorder counts, jitter spikes, and invalid OWD trends. Add
reset_for_rekey() methods that clear counter-dependent state while
preserving RTT estimates.

Additional fixes:
- Remove stale peers_by_index entry on abandon_rekey error path
- Replace redundant peers_by_index inserts with debug assertions
  verifying the pre-registration invariant
- Tighten rekey integration test to zero tolerance (was 4 failures)
2026-03-15 17:55:35 +00:00
Johnathan Corgan
6c90cf6c02 Implement Tor transport with operator visibility
Add TorTransport in src/transport/tor/ supporting three operating modes:

Outbound (socks5 mode):
- Non-blocking SOCKS5 connect via tokio-socks with per-destination
  circuit isolation (IsolateSOCKSAuth)
- TorAddr enum for .onion and clearnet address types
- Connection pool with per-connection receive tasks, reuses TCP
  stream FMP framing
- connect_async()/connection_state_sync()/promote_connection() follow
  the same non-blocking polling pattern as TCP transport

Inbound (directory mode — recommended for production):
- Tor manages the onion service via HiddenServiceDir in torrc
- FIPS reads .onion address from hostname file at startup
- No control port needed — enables Tor Sandbox 1 (seccomp-bpf)
- Accept loop mirrors TCP pattern with DirectoryServiceConfig

Monitoring (control_port mode and optional in directory mode):
- Async control port client supporting TCP and Unix socket connections
  via Box<dyn AsyncRead/Write> trait objects
- AUTHENTICATE with cookie or password auth
- 8 GETINFO queries: bootstrap, circuits, traffic, liveness, version,
  dormant state, SOCKS listeners
- Background monitoring task polls every 10s, caches TorMonitoringInfo
  in Arc<RwLock> for synchronous query access
- Bootstrap milestone logging (25/50/75/100%), stall warning (>60s),
  network liveness transitions, dormant mode entry
- Directory mode optionally connects to control port when control_addr
  is configured (non-fatal on failure)

Operator visibility:
- show_transports query exposes tor_mode, onion_address, tor_monitoring
  (bootstrap, circuit_established, traffic, liveness, version, dormant)
- fipstop transport detail view: Tor mode, onion address, SOCKS5/control
  errors, connection stats, Tor daemon status section
- fipstop table view: tor(mode) label with truncated onion address hint

Security hardening:
- Per-destination circuit isolation via IsolateSOCKSAuth
- Unix socket default for control port (/run/tor/control)
- Reference torrc with HiddenServiceDir, VanguardsLiteEnabled,
  ConnectionPadding, DoS protections (PoW + intro rate limiting)

Config:
- TorConfig with socks5, control_port, and directory modes
- DirectoryServiceConfig: hostname_file, bind_addr
- control_addr, control_auth, cookie_path, connect_timeout,
  max_inbound_connections

Testing:
- 69 unit + integration tests with mock SOCKS5 and control servers
- Docker tests: socks5-outbound (clearnet via Tor) and directory-mode
  (HiddenServiceDir onion service)

Documentation:
- Transport layer design doc: Tor architecture, directory mode
- Configuration doc: Tor config tables and examples
2026-03-15 16:19:54 +00:00
Johnathan Corgan
c0f30d8fe8 Add DNS hostname support in peer addresses for UDP and TCP transports
Add resolve_socket_addr() with IP fast path and tokio::net::lookup_host()
fallback for DNS hostnames. Peer addresses can now use hostnames like
"peer1.example.com:2121" alongside IP addresses.

UDP transport adds a per-transport DNS cache (60s TTL) to avoid
per-packet resolution. TCP resolves at connect time (one-shot).

Update design docs, config examples, and changelog to reflect hostname
support in transport addressing.
2026-03-15 00:49:54 +00:00
Johnathan Corgan
d873d0e00e Update design docs for non-blocking transport connect and cleanup stale references
- Rewrite TCP "Connect-on-Send" section to document non-blocking connect
  model (ConnectingPool, PendingConnect, poll_pending_connects)
- Add connect() and connection_state() to Transport trait surface
- Expand Connection Lifecycle section with ConnectionState enum
- Remove phantom TCP socks5_proxy field (removed from code, superseded
  by TorConfig)
- Fix "future Tor transport" references (stream reader already shared)
- Replace misleading "tor:" named TCP instance example
- Update fips-intro.md implementation status (TCP and Ethernet are
  implemented, not "under active design")
2026-03-13 16:37:00 +00:00
Origami74
35ff4a5d0a Add logos and update readme with banner 2026-03-13 12:17:50 +00:00
Johnathan Corgan
1bfb58845a Implement non-blocking transport connect for connection-oriented transports
TCP (and future Tor) transports previously established connections
synchronously inside send(), blocking the node's RX event loop during
TCP handshake. This is particularly problematic for Tor where SOCKS5
circuit establishment can take 30-120 seconds.

Add a non-blocking connect path:
- ConnectionState enum in transport layer (None/Connecting/Connected/Failed)
- connect_async() on TcpTransport spawns background TCP connect task
- connection_state_sync() polls task completion, promotes to pool
- TransportHandle gains connect() and connection_state() dispatch methods
- Node tracks PendingConnect entries for connection-oriented transports
- initiate_connection() defers handshake for connection-oriented transports
- start_handshake() extracted as separate method for deferred invocation
- poll_pending_connects() in tick handler polls and completes handshakes
- Failed connects trigger retry via schedule_retry()

Connectionless transports (UDP, Ethernet) are unchanged — connect()
is a no-op and connection_state() always returns Connected.

The existing connect-on-send path in send_async() is preserved as
fallback for reconnection after connection drops.

811 tests pass (6 new), clippy clean.
2026-03-13 03:21:21 +00:00
165 changed files with 10924 additions and 1549 deletions

View File

@@ -18,6 +18,7 @@ permissions:
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
SOURCE_DATE_EPOCH: 0 # overridden per-step after checkout
# ─────────────────────────────────────────────────────────────────────────────
# Job 1 Build matrix
@@ -40,6 +41,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -57,6 +61,9 @@ jobs:
- name: Build
run: cargo build --release
- name: SHA-256 hashes
run: sha256sum target/release/fips target/release/fipsctl target/release/fipstop
# Upload the Linux binary so integration jobs can use it without rebuilding
- name: Upload Linux binary
if: matrix.os == 'ubuntu-latest'
@@ -82,6 +89,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -125,8 +135,8 @@ jobs:
# Runs only when both build and test succeed. Each topology / scenario is a
# separate matrix entry so they run in parallel.
#
# Static topologies → build Docker images, start containers, ping-test
# Chaos scenarios → build sim image, run stochastic simulation
# All harnesses share a single Docker image (fips-test:latest) built once
# in the setup step from testing/docker/.
# ─────────────────────────────────────────────────────────────────────────────
integration:
name: Integration (${{ matrix.suite }})
@@ -153,24 +163,25 @@ jobs:
- suite: chaos-smoke-10
type: chaos
scenario: smoke-10
- suite: chaos-10
- suite: churn-mixed-10
type: chaos
scenario: chaos-10
scenario: churn-mixed
chaos_flags: "--nodes 10 --duration 120"
- suite: ethernet-mesh
type: chaos
scenario: ethernet-mesh
- suite: ethernet-only
type: chaos
scenario: ethernet-only
- suite: tcp-mesh
type: chaos
scenario: tcp-mesh
- suite: bottleneck-parent
type: chaos
scenario: bottleneck-parent
- suite: cost-avoidance
type: chaos
scenario: cost-avoidance
- suite: cost-mixed-7node
type: chaos
scenario: cost-mixed-7node
- suite: cost-reeval
type: chaos
scenario: cost-reeval
@@ -183,6 +194,9 @@ jobs:
- suite: mixed-technology
type: chaos
scenario: mixed-technology
- suite: congestion-stress
type: chaos
scenario: congestion-stress
# ── Sidecar deployment ──────────────────────────────────────────
- suite: sidecar
type: sidecar
@@ -197,24 +211,22 @@ jobs:
name: fips-linux
path: _bin
# ── Static topology ────────────────────────────────────────────────────
- name: Install binary (static)
if: matrix.type == 'static'
# Install binaries to unified docker context and build shared image
- name: Install binaries and build Docker image
run: |
chmod +x _bin/fips _bin/fipsctl
cp _bin/fips testing/static/fips
cp _bin/fipsctl testing/static/fipsctl
[ -f _bin/fipstop ] && chmod +x _bin/fipstop || true
cp _bin/fips testing/docker/fips
cp _bin/fipsctl testing/docker/fipsctl
[ -f _bin/fipstop ] && cp _bin/fipstop testing/docker/fipstop || true
docker build -t fips-test:latest testing/docker
docker build -t fips-test-app:latest -f testing/docker/Dockerfile.app testing/docker
# ── Static topology ────────────────────────────────────────────────────
- name: Generate configs (static)
if: matrix.type == 'static'
run: bash testing/static/scripts/generate-configs.sh ${{ matrix.topology }}
- name: Build Docker images (static)
if: matrix.type == 'static'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile ${{ matrix.topology }} build
- name: Start containers (static)
if: matrix.type == 'static'
run: |
@@ -238,25 +250,12 @@ jobs:
--profile ${{ matrix.topology }} down --volumes --remove-orphans
# ── Rekey integration test ──────────────────────────────────────────────
- name: Install binary (rekey)
if: matrix.type == 'rekey'
run: |
chmod +x _bin/fips _bin/fipsctl
cp _bin/fips testing/static/fips
cp _bin/fipsctl testing/static/fipsctl
- name: Generate and inject configs (rekey)
if: matrix.type == 'rekey'
run: |
bash testing/static/scripts/generate-configs.sh rekey
bash testing/static/scripts/rekey-test.sh inject-config
- name: Build Docker images (rekey)
if: matrix.type == 'rekey'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey build
- name: Start containers (rekey)
if: matrix.type == 'rekey'
run: |
@@ -284,20 +283,9 @@ jobs:
if: matrix.type == 'chaos'
run: pip3 install --quiet pyyaml jinja2
- name: Install binary (chaos)
if: matrix.type == 'chaos'
run: |
chmod +x _bin/fips _bin/fipsctl
cp _bin/fips testing/chaos/fips
cp _bin/fipsctl testing/chaos/fipsctl
- name: Build chaos Docker image
if: matrix.type == 'chaos'
run: docker build -t fips-chaos:latest testing/chaos
- name: Run chaos scenario
if: matrix.type == 'chaos'
run: bash testing/chaos/scripts/chaos.sh ${{ matrix.scenario }}
run: bash testing/chaos/scripts/chaos.sh ${{ matrix.scenario }} ${{ matrix.chaos_flags }}
- name: Upload sim results on failure (chaos)
if: matrix.type == 'chaos' && failure()
@@ -308,17 +296,9 @@ jobs:
retention-days: 7
# ── Sidecar deployment ──────────────────────────────────────────────
- name: Install binary (sidecar)
if: matrix.type == 'sidecar'
run: |
chmod +x _bin/fips _bin/fipsctl _bin/fipstop
cp _bin/fips testing/sidecar/fips
cp _bin/fipsctl testing/sidecar/fipsctl
cp _bin/fipstop testing/sidecar/fipstop
- name: Run sidecar integration test
if: matrix.type == 'sidecar'
run: bash testing/sidecar/scripts/test-sidecar.sh
run: bash testing/sidecar/scripts/test-sidecar.sh --skip-build
- name: Collect logs on failure (sidecar)
if: matrix.type == 'sidecar' && failure()
@@ -327,4 +307,4 @@ jobs:
echo "--- sidecar-${node} logs ---"
docker logs "sidecar-${node}-fips-1" 2>&1 || true
echo ""
done
done

View File

@@ -1,5 +1,4 @@
name: OpenWrt Package
on:
push:
workflow_dispatch:
@@ -11,11 +10,57 @@ on:
env:
CARGO_TERM_COLOR: always
PACKAGE_NAME: "fips"
jobs:
determine-versioning:
runs-on: ubuntu-latest
outputs:
package_version: ${{ steps.version.outputs.package_version }}
release_channel: ${{ steps.channel.outputs.release_channel }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Derive package version
id: version
shell: bash
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
echo "package_version=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
else
BRANCH=$(echo "$GITHUB_REF_NAME" | sed 's|/|-|g')
HEIGHT=$(git rev-list --count HEAD)
HASH=$(git rev-parse --short HEAD)
echo "package_version=${BRANCH}.${HEIGHT}.${HASH}" >> "$GITHUB_OUTPUT"
fi
- name: Determine release channel
id: channel
shell: bash
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
TAG_NAME=${GITHUB_REF#refs/tags/}
if [[ $TAG_NAME =~ ^v[0-9]+\.[0-9]+\.[0-9]+-alpha ]]; then
echo "release_channel=alpha" >> "$GITHUB_OUTPUT"
elif [[ $TAG_NAME =~ ^v[0-9]+\.[0-9]+\.[0-9]+-beta ]]; then
echo "release_channel=beta" >> "$GITHUB_OUTPUT"
elif [[ $TAG_NAME =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "release_channel=stable" >> "$GITHUB_OUTPUT"
else
echo "release_channel=dev" >> "$GITHUB_OUTPUT"
fi
else
echo "release_channel=dev" >> "$GITHUB_OUTPUT"
fi
build:
name: Build .ipk (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: determine-versioning
strategy:
fail-fast: false
@@ -46,19 +91,13 @@ jobs:
with:
fetch-depth: 0
- name: Derive package version
id: version
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
VERSION="${GITHUB_REF_NAME#v}"
else
BRANCH=$(echo "$GITHUB_REF_NAME" | sed 's|/|-|g')
HEIGHT=$(git rev-list --count HEAD)
HASH=$(git rev-parse --short HEAD)
VERSION="${BRANCH}.${HEIGHT}.${HASH}"
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "filename=fips_${VERSION}_${{ matrix.openwrt_arch }}.ipk" >> "$GITHUB_OUTPUT"
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.ipk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Install Rust toolchain (stable)
if: matrix.rust_channel == 'stable'
@@ -73,6 +112,7 @@ jobs:
components: rust-src
- name: Cache Cargo registry + build
if: ${{ env.ACT != 'true' }}
uses: actions/cache@v4
with:
path: |
@@ -84,29 +124,200 @@ jobs:
openwrt-${{ matrix.rust_target }}-
- name: Install cargo-zigbuild
run: cargo install cargo-zigbuild --locked
run: cargo install cargo-zigbuild --version 0.19.8 --locked
- name: Install zig (required by cargo-zigbuild)
uses: goto-bus-stop/setup-zig@v2
run: |
ZIG_VERSION="0.13.0"
ARCH=$(uname -m)
case "$ARCH" in
x86_64|amd64) ZIG_ARCH="x86_64" ;;
aarch64|arm64) ZIG_ARCH="aarch64" ;;
*) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac
curl -fsSL "https://ziglang.org/download/${ZIG_VERSION}/zig-linux-${ZIG_ARCH}-${ZIG_VERSION}.tar.xz" | sudo tar xJ -C /opt
sudo ln -sf /opt/zig-linux-${ZIG_ARCH}-${ZIG_VERSION}/zig /usr/local/bin/zig
zig version
- name: Install llvm-strip
run: sudo apt-get install -y --no-install-recommends llvm
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends llvm
- name: Install nak
shell: bash
run: |
NAK_VERSION="0.16.2"
ARCH=$(uname -m)
case "$ARCH" in
x86_64|amd64) NAK_ARCH="amd64" ;;
aarch64|arm64) NAK_ARCH="arm64" ;;
*) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac
curl -fsSL "https://github.com/fiatjaf/nak/releases/download/v${NAK_VERSION}/nak-v${NAK_VERSION}-linux-${NAK_ARCH}" \
-o /usr/local/bin/nak
chmod +x /usr/local/bin/nak
nak --version
- name: Install jq
run: |
if ! command -v jq &>/dev/null; then
sudo apt-get update && sudo apt-get install -y jq
fi
# Priority: HIVE_CI_NSEC from env (loom job) > repo secret > generate ephemeral
- name: Resolve signing key
id: keys
shell: bash
env:
SECRET_NSEC: ${{ secrets.HIVE_CI_NSEC }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
if [ -n "${HIVE_CI_NSEC:-}" ]; then
echo "Using HIVE_CI_NSEC from loom job environment"
NSEC="$HIVE_CI_NSEC"
elif [ -n "$SECRET_NSEC" ]; then
echo "Using HIVE_CI_NSEC from repository secrets"
NSEC="$SECRET_NSEC"
else
echo "No nsec provided -- generating ephemeral keypair"
NSEC=$(nak key generate)
fi
PUBKEY=$(echo "$NSEC" | nak key public)
echo "::add-mask::$NSEC"
echo "nsec=$NSEC" >> "$GITHUB_OUTPUT"
echo "pubkey=$PUBKEY" >> "$GITHUB_OUTPUT"
echo "Publisher pubkey (hex): $PUBKEY"
- name: Build .ipk
env:
PKG_VERSION: ${{ steps.version.outputs.version }}
PKG_VERSION: ${{ needs.determine-versioning.outputs.package_version }}
LLVM_STRIP: llvm-strip
run: ./packaging/openwrt/build-ipk.sh --arch ${{ matrix.build_arch }}
run: ./packaging/openwrt-ipk/build-ipk.sh --arch ${{ matrix.build_arch }}
- name: Upload artifact
- name: SHA-256 hashes
run: |
echo "==> Binaries:"
sha256sum target/${{ matrix.rust_target }}/release/fips target/${{ matrix.rust_target }}/release/fipsctl target/${{ matrix.rust_target }}/release/fipstop
echo "==> Package:"
sha256sum dist/${{ env.PACKAGE_FILENAME }}
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v4
with:
name: ${{ steps.version.outputs.filename }}
path: dist/${{ steps.version.outputs.filename }}
name: ${{ env.PACKAGE_FILENAME }}
path: dist/${{ env.PACKAGE_FILENAME }}
retention-days: 30
- name: Upload to Blossom
id: blossom_upload
shell: bash
env:
BLOSSOM_SERVER: "https://blossom.primal.net"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null)
echo "Upload response:"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -z "$FILE_HASH" ] || [ "$FILE_HASH" = "null" ]; then
echo "Failed to extract hash from upload response"
exit 1
fi
BLOSSOM_URL="${BLOSSOM_SERVER}/${FILE_HASH}"
echo "url=$BLOSSOM_URL" >> "$GITHUB_OUTPUT"
echo "hash=$FILE_HASH" >> "$GITHUB_OUTPUT"
echo "Uploaded to Blossom: $BLOSSOM_URL"
- name: Publish NIP-94 release event
id: publish
shell: bash
env:
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://relay.primal.net"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
set -e
VERSION="${{ needs.determine-versioning.outputs.package_version }}"
CHANNEL="${{ needs.determine-versioning.outputs.release_channel }}"
nak event --sec "$NSEC" -k 1063 \
-c "FIPS Package: ${{ env.PACKAGE_NAME }} for ${{ matrix.openwrt_arch }}" \
--tag url="${{ steps.blossom_upload.outputs.url }}" \
--tag m="application/octet-stream" \
--tag x="${{ steps.blossom_upload.outputs.hash }}" \
--tag ox="${{ steps.blossom_upload.outputs.hash }}" \
--tag filename="${{ env.PACKAGE_FILENAME }}" \
--tag A="${{ matrix.openwrt_arch }}" \
--tag v="$VERSION" \
--tag n="${{ env.PACKAGE_NAME }}" \
--tag compression="none" \
> event.json 2> event.err
if [ ! -s event.json ]; then
echo "Failed to create event"
cat event.err 2>/dev/null || true
exit 1
fi
echo "=== Event JSON ==="
cat event.json
echo "=================="
EVENT_ID=$(jq -r '.id' event.json)
if [ -z "$EVENT_ID" ] || [ "$EVENT_ID" = "null" ]; then
echo "Failed to extract event ID"
exit 1
fi
# Publish to relays
cat event.json | nak event $RELAYS 2>&1
echo "eventId=$EVENT_ID" >> "$GITHUB_OUTPUT"
echo "Published NIP-94 event: $EVENT_ID"
- name: Verify NIP-94 event on relays
if: ${{ steps.publish.outputs.eventId != '' }}
env:
EVENT_ID: ${{ steps.publish.outputs.eventId }}
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://relay.primal.net"
run: |
echo "Verifying event $EVENT_ID on relays..."
FOUND=0
for relay in $RELAYS; do
echo "Checking $relay..."
RESULT=$(nak req -i "$EVENT_ID" "$relay" 2>/dev/null || echo "")
if [ -n "$RESULT" ]; then
echo "Found on $relay"
FOUND=1
else
echo "Not found on $relay"
fi
done
if [ $FOUND -eq 0 ]; then
echo "Warning: Event not found on any relay yet (may still be propagating)"
else
echo "Event verified on at least one relay"
fi
- name: Build Summary
run: |
echo "Build Summary for ${{ matrix.openwrt_arch }}:"
echo " Package: ${{ env.PACKAGE_FILENAME }}"
echo " Release EventId: ${{ steps.publish.outputs.eventId }}"
echo " Blossom URL: ${{ steps.blossom_upload.outputs.url }}"
release:
name: Publish GitHub Release
name: Publish GitHub Release (github only)
runs-on: ubuntu-latest
needs: build
if: startsWith(github.ref, 'refs/tags/')

10
.gitignore vendored
View File

@@ -20,4 +20,12 @@ vps.env
reference/
dist/
*.ipk
*.ipk
sim-results/
# Python
__pycache__/
*.py[cod]
*.egg-info/
*.egg

View File

@@ -5,7 +5,88 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.2.0] - 2026-03-22
### Added
#### Operator Tooling
- `fipsctl connect` and `disconnect` commands for runtime peer
management via control socket, with hostname resolution from
`/etc/fips/hosts`
#### IPv6 Adapter
- Pre-seed identity cache from configured peer npubs at startup, so TUN packets can be dispatched immediately without waiting for handshake completion ([@v0l](https://github.com/v0l))
#### Mesh Peer Transports
- New Tor transport with SOCKS5 and directory-mode onion service for anonymous inbound and outbound peering
- DNS hostname support in peer addresses for UDP and TCP transports
- Non-blocking transport connect for connection-oriented transports (TCP, Tor)
#### Packaging and Deployment
- Reproducible build infrastructure: Rust toolchain pinning via
`rust-toolchain.toml`, `SOURCE_DATE_EPOCH` in CI and packaging
scripts, deterministic archive timestamps
- Top-level packaging Makefile for unified build across formats
- Kubernetes sidecar deployment example with Nostr relay demo
- Nostr release publishing in OpenWrt package workflow
- SHA-256 hash output in CI build and OpenWrt workflows
#### Testing and CI
- Maelstrom chaos scenario with dynamic topology mutation and
ephemeral node identities via connect/disconnect commands
- Consolidated Docker test harness infrastructure
### Changed
- Discovery protocol: replace flooding with bloom-filter-guided tree
routing. Includes originator retry (T=0/T=5s/T=10s), exponential
backoff after timeouts and bloom misses, and transit-side per-target
rate limiting. Removed 257-byte visited bloom filter from LookupRequest wire format. *This is a breaking change; nodes running versions prior to this release will not be compatible.*
### Fixed
- DNS responder returned NXDOMAIN for A queries on valid `.fips` names,
causing resolvers to give up without trying AAAA. Now returns NOERROR
with empty answers for non-AAAA queries on resolvable names.
(#9, reported by [@alopatindev](https://github.com/alopatindev))
- Stale end-to-end session left in session table after peer removal blocked session re-establishment on reconnect — `remove_active_peer` now cleans up `self.sessions` and `self.pending_tun_packets`. (#5, [@v0l](https://github.com/v0l))
- `schedule_reconnect` reset exponential backoff to zero on each link-dead
cycle instead of preserving accumulated retry count.
(#5, [@v0l](https://github.com/v0l))
- FMP/FSP rekey dual-initiation race on high-latency links (Tor): both
sides' timers fired simultaneously, both msg1s crossed in flight, each
side's responder path destroyed the initiator state. Fixed with
deterministic tie-breaker (smaller NodeAddr wins as initiator).
- Parent selection SRTT gate bypass: `evaluate_parent` used default cost
1.0 for peers filtered out by `has_srtt()`, defeating the MMP eligibility
gate. Now skips unmeasured candidates when any peer has cost data.
- FSP rekey cutover race: initiator cut over before responder received msg3,
causing AEAD failures. Fixed by deferring initiator cutover by 2 seconds.
- MMP metric discontinuity after rekey: receiver state carried stale
counters across rekey, inflating reorder counts and jitter. Fixed via
`reset_for_rekey()`.
- Auto-connect peers exhausted `max_retries` on initial connection failures
and were permanently abandoned. Now retry indefinitely with exponential
backoff capped at 300 seconds.
- Control socket permissions: non-root users couldn't connect. Daemon now
chowns socket and directory to `root:fips` group at bind time.
- Post-rekey jitter spikes: old-session frames arriving via the drain window
produced 2,0007,000ms jitter spikes that corrupted the EWMA estimator.
Added a 15-second grace period after rekey cutover that suppresses jitter
updates until drain-window frames have flushed. (#10)
- ICMPv6 Packet Too Big source was set to the local FIPS address, which
Linux ignores (loopback PTB check). Now uses the original packet's
destination so the kernel honors the PMTU update.
(#16, [@v0l](https://github.com/v0l))
- Reverse delivery ratio used lifetime cumulative counters instead of
per-interval deltas, making ETX unresponsive to recent loss. (#14)
- MMP delta guards used `prev_rr > 0` to detect first report, conflating
it with a legitimate zero counter. Replaced with `has_prev_rr`. (#14)
## [0.1.0] - 2026-03-12

15
Cargo.lock generated
View File

@@ -783,7 +783,7 @@ checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
[[package]]
name = "fips"
version = "0.1.0"
version = "0.2.0"
dependencies = [
"bech32",
"chacha20poly1305",
@@ -807,6 +807,7 @@ dependencies = [
"tempfile",
"thiserror 2.0.18",
"tokio",
"tokio-socks",
"tracing",
"tracing-subscriber",
"tun",
@@ -2422,6 +2423,18 @@ dependencies = [
"syn 2.0.114",
]
[[package]]
name = "tokio-socks"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d4770b8024672c1101b3f6733eab95b18007dbe0847a8afe341fcf79e06043f"
dependencies = [
"either",
"futures-util",
"thiserror 1.0.69",
"tokio",
]
[[package]]
name = "tokio-util"
version = "0.7.18"

View File

@@ -1,6 +1,6 @@
[package]
name = "fips"
version = "0.1.0"
version = "0.2.0"
edition = "2024"
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
license = "MIT"
@@ -36,6 +36,7 @@ tokio = { version = "1", features = ["rt", "macros", "signal", "sync", "net", "t
futures = "0.3"
simple-dns = "0.11.2"
socket2 = { version = "0.6.2", features = ["all"] }
tokio-socks = "0.5"
[package.metadata.deb]
maintainer = "Johnathan Corgan <jcorgan@corganlabs.com>"

View File

@@ -1,5 +1,6 @@
# FIPS: Free Internetworking Peering System
![banner](docs/logos/fips_banner.png)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)
[![Status](https://img.shields.io/badge/status-alpha%20(0.1.0)-yellow.svg)](#status--roadmap)
@@ -241,7 +242,7 @@ testing/ Docker-based integration test harnesses
## Status & Roadmap
FIPS is at **v0.1.0 (alpha)**. The core protocol works end-to-end over
UDP, TCP, and Ethernet but has not been tested beyond small meshes.
UDP, TCP, Ethernet, and Tor but has not been tested beyond small meshes.
### What works today
@@ -254,14 +255,14 @@ UDP, TCP, and Ethernet but has not been tested beyond small meshes.
- Static hostname mapping (`/etc/fips/hosts`) with auto-reload
- Per-link metrics (RTT, loss, jitter, goodput) and mesh size estimation
- ECN congestion signaling (hop-by-hop CE relay, IPv6 CE marking, kernel drop detection)
- UDP, TCP, and Ethernet transports
- UDP, TCP, Ethernet, and Tor transports (SOCKS5 outbound + directory-mode onion service inbound)
- Runtime inspection via `fipsctl` and `fipstop`
- Docker-based integration and chaos testing
### Near-term priorities
- Peer discovery via Nostr relays (bootstrap without static peer lists)
- Additional transports (Bluetooth, Tor)
- Additional transports (Bluetooth)
- Improved routing resilience under churn
- Security audit of cryptographic protocols

View File

@@ -56,9 +56,11 @@ peers: # Static peer list
| `node.control.enabled` | bool | `true` | Enable the Unix domain control socket |
| `node.control.socket_path` | string | *(auto)* | Socket file path. Default: `$XDG_RUNTIME_DIR/fips/control.sock`, then `/run/fips/control.sock` (if root), then `/tmp/fips-control.sock` |
The control socket provides read-only access to node state via the
`fipsctl` command-line tool. See the project
[README](../../README.md#inspect) for the command list.
The control socket provides access to node state and runtime management
via the `fipsctl` command-line tool. In addition to read-only status
queries, `fipsctl connect` and `fipsctl disconnect` enable runtime peer
management. See the project [README](../../README.md#inspect) for the
command list.
All tunable protocol parameters live under `node.*`, organized as sysctl-style
dotted paths. The top-level sections (`tun`, `dns`, `transports`, `peers`)
@@ -144,13 +146,18 @@ Controls caching of tree coordinates and identity mappings.
### Discovery Protocol (`node.discovery.*`)
Controls flood-based node discovery (LookupRequest/LookupResponse).
Controls bloom-guided node discovery (LookupRequest/LookupResponse).
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.discovery.ttl` | u8 | `64` | Hop limit for LookupRequest flood |
| `node.discovery.ttl` | u8 | `64` | Hop limit for LookupRequest forwarding |
| `node.discovery.timeout_secs` | u64 | `10` | Lookup completion timeout |
| `node.discovery.recent_expiry_secs` | u64 | `10` | Dedup cache expiry for recent request IDs |
| `node.discovery.retry_interval_secs` | u64 | `5` | Retry interval within the timeout window; after this interval without a response, resend the lookup |
| `node.discovery.max_attempts` | u8 | `2` | Max attempts per lookup (1 = no retry, 2 = one retry) |
| `node.discovery.backoff_base_secs` | u64 | `30` | Base for exponential backoff after lookup failure; doubles per consecutive failure |
| `node.discovery.backoff_max_secs` | u64 | `300` | Cap on exponential backoff (5 minutes) |
| `node.discovery.forward_min_interval_secs` | u64 | `2` | Transit-side rate limiting: minimum interval between forwarded lookups for the same target |
### Spanning Tree (`node.tree.*`)
@@ -359,7 +366,6 @@ overhead.
| `transports.tcp.recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `transports.tcp.send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `transports.tcp.max_inbound_connections` | usize | `256` | Maximum simultaneous inbound connections |
| `transports.tcp.socks5_proxy` | string | *(none)* | SOCKS5 proxy for outbound connections (implementation deferred) |
**Named instances.** Like other transports, multiple TCP instances can
be configured with named sub-keys:
@@ -369,9 +375,114 @@ transports:
tcp:
public:
bind_addr: "0.0.0.0:443"
tor:
socks5_proxy: "127.0.0.1:9050"
connect_timeout_ms: 30000
internal:
bind_addr: "10.0.0.1:8443"
max_inbound_connections: 64
```
### Tor (`transports.tor.*`)
Tor transport routes FIPS traffic through the Tor network for anonymity.
Requires an external Tor daemon providing a SOCKS5 proxy. Three modes:
`socks5` for outbound-only, `control_port` for outbound + monitoring,
`directory` for outbound + inbound via Tor-managed onion service.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transports.tor.mode` | string | `"socks5"` | Tor access mode: `socks5` (outbound only), `control_port` (outbound + monitoring), or `directory` (outbound + inbound onion service) |
| `transports.tor.socks5_addr` | string | `"127.0.0.1:9050"` | SOCKS5 proxy address (host:port) |
| `transports.tor.connect_timeout_ms` | u64 | `120000` | Connect timeout in milliseconds. Tor circuits take 1060s. |
| `transports.tor.mtu` | u16 | `1400` | Default MTU |
| `transports.tor.control_addr` | string | `"/run/tor/control"` | Tor control port address: Unix socket path or host:port. Used in `control_port` mode; optional in `directory` mode for monitoring. |
| `transports.tor.control_auth` | string | `"cookie"` | Control port authentication: `"cookie"`, `"cookie:/path/to/cookie"`, or `"password:<secret>"`. |
| `transports.tor.cookie_path` | string | `"/var/run/tor/control.authcookie"` | Path to Tor control cookie file. Used when `control_auth` is `"cookie"`. |
| `transports.tor.max_inbound_connections` | usize | `64` | Maximum inbound connections via onion service. |
| `transports.tor.directory_service.hostname_file` | string | `"/var/lib/tor/fips_onion_service/hostname"` | Path to Tor-managed hostname file containing the `.onion` address. |
| `transports.tor.directory_service.bind_addr` | string | `"127.0.0.1:8443"` | Local bind address for the listener that Tor forwards inbound connections to. Must match `HiddenServicePort` target in `torrc`. |
**Named instances.** Like other transports, multiple Tor instances can
be configured with named sub-keys for different SOCKS5 proxy endpoints.
**Directory mode** (recommended for production). Tor manages the onion
service via `HiddenServiceDir` in `torrc`. FIPS reads the `.onion`
address from the hostname file and binds a local TCP listener. This
enables Tor's `Sandbox 1` (seccomp-bpf). If `control_addr` is also
set, the transport connects to the control port for daemon monitoring
(non-fatal on failure).
**Control port mode.** Connects to the Tor daemon's control port for
monitoring only (bootstrap status, circuit health, traffic stats).
No inbound connections. Both `control_addr` and `control_auth` are
required.
### UDP + Tor Bridge Example
A node bridging clearnet (UDP) and anonymous (Tor) portions of the mesh:
```yaml
node:
identity:
persistent: true
tun:
enabled: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
mtu: 1472
tor:
socks5_addr: "127.0.0.1:9050"
peers:
- npub: "npub1abc..."
alias: "clearnet-peer"
addresses:
- transport: udp
addr: "203.0.113.5:2121"
- npub: "npub1def..."
alias: "anonymous-peer"
addresses:
- transport: tor
addr: "abc123...xyz.onion:2121"
```
### Tor Directory Mode Example
A node accepting inbound connections via Tor-managed onion service
(recommended for production — enables Sandbox 1):
```yaml
node:
identity:
persistent: true
tun:
enabled: true
transports:
tor:
mode: "directory"
socks5_addr: "127.0.0.1:9050"
control_addr: "/run/tor/control" # optional, for monitoring
control_auth: "cookie"
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
peers:
- npub: "npub1abc..."
alias: "tor-peer"
addresses:
- transport: tor
addr: "abcdef...xyz.onion:8443"
```
Requires a corresponding `torrc`:
```text
HiddenServiceDir /var/lib/tor/fips
HiddenServicePort 8443 127.0.0.1:8444
```
## Peers (`peers[]`)
@@ -382,8 +493,8 @@ Static peer list. Each entry defines a peer to connect to.
|-----------|------|---------|-------------|
| `peers[].npub` | string | *(required)* | Peer's Nostr public key (npub-encoded) |
| `peers[].alias` | string | *(none)* | Human-readable name for logging |
| `peers[].addresses[].transport` | string | *(required)* | Transport type: `udp`, `tcp`, or `ethernet` |
| `peers[].addresses[].addr` | string | *(required)* | Transport address. UDP/TCP: `"ip:port"`. Ethernet: `"interface/mac"` (e.g., `"eth0/aa:bb:cc:dd:ee:ff"`) |
| `peers[].addresses[].transport` | string | *(required)* | Transport type: `udp`, `tcp`, `ethernet`, or `tor` |
| `peers[].addresses[].addr` | string | *(required)* | Transport address. UDP/TCP: `"host:port"` (IP or DNS hostname). Ethernet: `"interface/mac"` (e.g., `"eth0/aa:bb:cc:dd:ee:ff"`). Tor: `".onion:port"` or `"host:port"` |
| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) |
| `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` |
| `peers[].auto_reconnect` | bool | `true` | Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries) |
@@ -512,6 +623,11 @@ node:
ttl: 64
timeout_secs: 10
recent_expiry_secs: 10
retry_interval_secs: 5
max_attempts: 2
backoff_base_secs: 30
backoff_max_secs: 300
forward_min_interval_secs: 2
tree:
announce_min_interval_ms: 500
parent_hysteresis: 0.2 # cost improvement fraction for parent switch
@@ -590,7 +706,20 @@ transports:
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# max_inbound_connections: 256 # resource protection limit
# socks5_proxy: null # SOCKS5 for outbound (deferred)
# tor: # uncomment to enable Tor transport
# mode: "socks5" # "socks5", "control_port", or "directory"
# socks5_addr: "127.0.0.1:9050" # SOCKS5 proxy address
# connect_timeout_ms: 120000 # connect timeout (120s for Tor circuits)
# mtu: 1400 # default MTU
# # monitoring (control_port mode, or optional in directory mode):
# # control_addr: "/run/tor/control" # Unix socket or host:port
# # control_auth: "cookie" # "cookie" or "password:<secret>"
# # cookie_path: "/var/run/tor/control.authcookie"
# # directory mode (inbound via Tor-managed onion service):
# # directory_service:
# # hostname_file: "/var/lib/tor/fips/hostname"
# # bind_addr: "127.0.0.1:8444"
# # max_inbound_connections: 64
peers: # static peer list
# - npub: "npub1..."

View File

@@ -216,7 +216,7 @@ address prepends `fd` to the first 15 bytes of the node_addr, providing a
ULA overlay address for unmodified IP applications via the TUN interface.
Below the FIPS identity layer, each transport uses its own native addressing
— IP:port tuples, MAC addresses, .onion identifiers. These **link
— IP:port or hostname:port addresses, MAC addresses, .onion identifiers. These **link
addresses** are opaque to everything above FMP and discarded once link
authentication completes.
@@ -524,9 +524,9 @@ forwarding, a publicly addressed peer, or relay through other mesh nodes.
UDP hole punching and relay-assisted NAT traversal are potential future
mechanisms but are not part of the current design.
> **Implementation status**: UDP/IP is implemented. Ethernet and Bluetooth
> transports are under active design and development. All others are future
> directions.
> **Implementation status**: UDP/IP, TCP/IP, Ethernet, and Tor
> (SOCKS5 outbound + directory-mode inbound via onion service)
> transports are implemented. All others are future directions.
See [fips-transport-layer.md](fips-transport-layer.md) for the full transport
layer specification.

View File

@@ -15,7 +15,7 @@ updates, coordinate discovery, and forwarded session datagrams — all encrypted
per-hop.
FMP is the boundary between opaque transport addresses and identified peers.
Below FMP, everything is transport-specific addresses (IP:port, MAC, .onion).
Below FMP, everything is transport-specific addresses (host:port, MAC, .onion).
Above FMP, everything is peers identified by public keys and routable by
node_addr. The transport layer never sees FIPS-level structure; FSP never sees
transport addresses or routing details.
@@ -280,7 +280,7 @@ authenticated peer regardless of what transport address the packet arrived
from. FMP updates the peer's current address to the packet's source address,
and subsequent outbound packets use the updated address.
This allows peers to change transport addresses (e.g., IP:port for UDP)
This allows peers to change transport addresses (e.g., host:port for UDP)
without session interruption. The mechanism is:
1. Packet arrives from a different address than expected

View File

@@ -330,7 +330,9 @@ fresh SessionSetup re-warms transit caches (still within their 300s TTL).
## Discovery Protocol
Discovery resolves a destination's tree coordinates so that multi-hop routing
can proceed.
can proceed. Requests are forwarded using **bloom-guided tree routing**
only to tree peers (parent + children) whose bloom filter contains the
target — producing single-path forwarding through the spanning tree.
### When Discovery Is Needed
@@ -347,17 +349,70 @@ The source creates a LookupRequest containing:
- **origin**: The requester's node_addr
- **origin_coords**: The requester's current tree coordinates (so the
response can route back)
- **TTL**: Bounds the flood radius
- **TTL**: Bounds the forwarding radius
The request floods through the mesh: each node decrements TTL, adds itself
to a visited filter (preventing loops on a single path), and forwards to all
peers not in the visited filter. Bloom filters may help direct the flood
toward likely candidates.
### Bloom-Guided Tree Routing
**Deduplication**: Nodes maintain a short-lived request_id dedup cache
(default 10s window) to drop convergent duplicates (the same request
arriving via different paths). This is a protocol requirement, not an
optimization.
Rather than flooding to all peers, the request is forwarded only to **tree
peers** (parent + children) whose bloom filter contains the target. Because
bloom filters propagate along tree edges with split-horizon exclusion,
typically only one tree peer matches — producing a single directed path
through the spanning tree toward the target's subtree. This reduces
discovery traffic by roughly 90% compared to flooding.
If no tree peer's bloom filter matches the target, the request falls back
to **non-tree peers** whose bloom filter contains the target. This recovers
from dead ends caused by stale bloom filters, tree restructuring, or transit
node failures. If no peer at all has a bloom match, the request is dropped
at that node.
**Loop prevention**: The spanning tree is inherently loop-free, so tree-only
forwarding cannot loop. The `request_id` dedup cache (default 10s window)
provides defense-in-depth, catching edge cases during tree restructuring
where a request might arrive via both tree and fallback paths.
### Retry Logic
Single-path forwarding is more fragile than flooding — if any transit node
on the path has a stale bloom filter or loses a link, the request fails.
To compensate, the originator retries:
- **T=0**: Initial lookup sent
- **T=5s**: Retry if no response (configurable via `retry_interval_secs`)
- **T=10s**: Timeout, fail (configurable via `timeout_secs`)
The default `max_attempts` is 2 (initial + one retry). Each retry generates
a fresh `request_id` and re-evaluates bloom filter matches, so it can take
a different path if the tree has restructured.
### Originator Backoff
After a lookup times out or no peer's bloom filter contains the target, the
originator enters **exponential backoff** before re-attempting discovery for
the same target:
- **Base delay**: 30s (configurable via `backoff_base_secs`)
- **Multiplier**: 2x per consecutive failure
- **Cap**: 300s (configurable via `backoff_max_secs`)
Backoff is **reset on topology changes** that might make previously
unreachable targets reachable: parent switch, new peer connection, first
RTT measurement from MMP, or peer reconnection.
### Bloom Filter Pre-Check
Before initiating a lookup, the originator checks whether *any* peer's
bloom filter contains the target. If no peer advertises reachability, the
lookup is skipped entirely and recorded as a failure for backoff purposes.
This avoids wasting network resources when the target is not in the mesh.
### Transit-Side Rate Limiting
Transit nodes enforce a per-target minimum interval (default 2s, configurable
via `forward_min_interval_secs`) for forwarded lookups. This is
defense-in-depth against misbehaving nodes that generate fresh `request_id`s
at high rate to bypass dedup. The rate limiter collapses rapid-fire lookups
for the same target regardless of `request_id`.
### LookupResponse
@@ -373,14 +428,21 @@ direct peer), a LookupResponse is created containing:
authenticates that the response is genuine and the target holds the
claimed tree position
The response routes back to the requester using reverse-path routing as the
primary mechanism: each transit node looks up the request_id in its
The response routes back to the requester using **reverse-path routing** as
the primary mechanism: each transit node looks up the `request_id` in its
`recent_requests` table to find the peer that forwarded the original request,
and sends the response back through that peer. This ensures the response
follows the same path as the request. Greedy tree routing toward the
origin_coords is used only as a fallback if the reverse-path entry has
`origin_coords` is used only as a fallback if the reverse-path entry has
expired.
**Response-forwarded flag**: Each `recent_requests` entry tracks whether a
response has already been forwarded for that `request_id`. If a second
response arrives (e.g., from convergent request paths that reached the
target via different routes), the transit node drops it. This prevents
response routing loops where multiple responses for the same request
circulate through the network.
**Proof verification**: The source verifies the Schnorr proof upon receipt,
confirming that the target actually signed the response. The proof covers
`(request_id || target || target_coords)` — coordinates are included because
@@ -390,12 +452,14 @@ annotation modified at each hop.
### Discovery Outcome
On receiving a LookupResponse, the source caches the target's coordinates.
Subsequent routing to that destination can proceed via the normal
`find_next_hop()` priority chain.
On receiving a verified LookupResponse, the source caches the target's
coordinates and clears any backoff state for that target. Subsequent routing
to that destination can proceed via the normal `find_next_hop()` priority
chain.
If discovery times out (no response), queued packets receive ICMPv6
Destination Unreachable.
If discovery times out (no response after all retry attempts), queued
packets receive ICMPv6 Destination Unreachable and the target enters
backoff.
## SessionSetup Self-Bootstrapping
@@ -467,7 +531,7 @@ coordinates for the destination. It cannot make a forwarding decision.
2. Reset CP warmup counter — subsequent data packets piggyback coordinates
when possible, or trigger additional CoordsWarmup messages when
piggybacking would exceed the transport MTU
3. Initiate discovery (LookupRequest flood) for the destination
3. Initiate discovery (bloom-guided LookupRequest) for the destination
4. When discovery completes, warmup counter resets again (covers timing gap)
The crypto session remains active throughout — only routing state is
@@ -553,8 +617,9 @@ sequence:
identity cache with NodeAddr + PublicKey
2. **Session initiation attempt**: Fails because no coordinates are cached
for the destination
3. **Discovery**: LookupRequest floods through the mesh; LookupResponse
returns the destination's coordinates
3. **Discovery**: LookupRequest routes through the spanning tree via
bloom-guided forwarding; LookupResponse returns the destination's
coordinates
4. **Session establishment**: SessionSetup carries coordinates, warming
transit caches along the path
5. **Warmup**: First N data packets include CP flag, reinforcing transit
@@ -643,7 +708,7 @@ routing decisions but retains its own end-to-end encryption and identity.
| ------- | ------------ | ---- | ---------- |
| TreeAnnounce | Variable (depth-dependent) | Topology changes | No (peer-to-peer) |
| FilterAnnounce | ~1 KB | Topology changes | No (peer-to-peer) |
| LookupRequest | ~300 bytes | First contact, recovery | Yes (flood) |
| LookupRequest | ~300 bytes | First contact, recovery | Yes (bloom-guided tree) |
| LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) |
| SessionDatagram + SessionSetup | ~232402 bytes | Session establishment | Yes (routed) |
| SessionDatagram + SessionAck | ~170 bytes | Session confirmation | Yes (routed) |
@@ -696,10 +761,14 @@ recovery).
| Flap dampening (hysteresis + hold-down) | **Implemented** |
| Link liveness (dead timeout) | **Implemented** |
| Discovery request deduplication | **Implemented** |
| Discovery bloom-guided tree routing | **Implemented** |
| Discovery retry logic | **Implemented** |
| Discovery originator backoff | **Implemented** |
| Discovery transit-side rate limiting | **Implemented** |
| Discovery response-forwarded dedup | **Implemented** |
| Leaf-only operation | Under development |
| Link cost in parent selection (ETX) | **Implemented** |
| Link cost in candidate ranking | **Implemented** |
| Discovery path accumulation | Future direction |
## References

View File

@@ -14,7 +14,7 @@ address, deliver the datagram to that address, and push inbound datagrams up
to the FIPS Mesh Protocol (FMP) above.
The transport layer deals exclusively in **transport addresses** — IP:port
tuples, MAC addresses, .onion identifiers, radio device addresses. These are
or hostname:port addresses, MAC addresses, .onion identifiers, radio device addresses. These are
opaque to every layer above FMP. The mapping from transport address to FIPS
identity happens at the link layer after the Noise IK link handshake completes.
The word "peer" belongs to the link layer and above; the transport layer
@@ -69,10 +69,30 @@ forwarding and LookupResponse transit annotation.
For connection-oriented transports, manage the underlying connection: TCP
handshake, Tor circuit establishment, Bluetooth pairing. FMP cannot begin
the Noise IK link handshake until the transport-layer connection is established.
the Noise IK link handshake until the transport-layer connection is
established.
Connectionless transports (UDP, raw Ethernet) skip this — datagrams can flow
immediately to any reachable address.
Connection-oriented transports expose a non-blocking connect interface.
`connect(addr)` initiates the connection in a background task and returns
immediately. `connection_state(addr)` reports the current status:
```text
ConnectionState {
None No connection attempt in progress
Connecting Background task running
Connected Ready for send()
Failed(msg) Error message from failed attempt
}
```
Connectionless transports (UDP, raw Ethernet) return `Connected`
immediately — no async work needed.
At the node level, `PendingConnect` entries track links waiting for
transport connection. `poll_pending_connects()` runs each tick, checks
`connection_state()`, and calls `start_handshake()` on success or
`schedule_retry()` on failure. This decouples transport-layer connection
(which may take seconds for Tor circuits) from the FMP event loop.
### Discovery (Optional)
@@ -95,8 +115,8 @@ for internet connectivity:
| Transport | Addressing | MTU | Reliability | Notes |
| --------- | ---------- | --- | ----------- | ----- |
| UDP/IP | IP:port | 12801472 | Unreliable | Primary internet transport |
| TCP/IP | IP:port | Stream | Reliable | Requires length-prefix framing |
| UDP/IP | host:port | 12801472 | Unreliable | Primary internet transport |
| TCP/IP | host:port | Stream | Reliable | Requires length-prefix framing |
| WebSocket | URL | Stream | Reliable | Browser-compatible |
| Tor | .onion | Stream | Reliable | High latency, strong anonymity |
@@ -326,8 +346,7 @@ Ethernet transport started name=eth0 interface=eth0 mac=aa:bb:cc:dd:ee:ff mtu=14
## TCP/IP: Firewall Traversal Transport
For networks where UDP is blocked but TCP port 443 is open, the TCP
transport provides an alternative path. It also serves as the foundation
for the future Tor transport.
transport provides an alternative path.
FIPS protocols (FMP, FSP, MMP) are all unreliable datagrams. Running them
over TCP introduces head-of-line blocking, which adds latency jitter. MMP
@@ -338,16 +357,18 @@ penalizes TCP links (higher SRTT leads to higher link cost). ETX will be
### Architecture
Unlike UDP (one socket serves all peers), TCP requires one `TcpStream` per
peer. The transport maintains a connection pool (`HashMap<TransportAddr,
TcpConnection>`) plus an optional `TcpListener` for inbound connections.
peer. The transport maintains two pools: a `ConnectingPool` for background
connection attempts in progress, and an established connection pool
(`HashMap<TransportAddr, TcpConnection>`) for active connections, plus an
optional `TcpListener` for inbound connections.
| Property | Value |
| -------- | ----- |
| Addressing | IP:port (same as UDP) |
| Addressing | host:port — IP address or DNS hostname |
| Default MTU | 1400 bytes |
| Per-link MTU | Derived from `TCP_MAXSEG` socket option |
| Framing | FMP header-based (zero overhead) |
| Connection model | Connect-on-send, optional listener |
| Connection model | Non-blocking connect, connect-on-send fallback, optional listener |
| Platform | Cross-platform (no `#[cfg]` gates) |
### FMP Header-Based Framing
@@ -364,29 +385,36 @@ packet boundaries:
This provides zero framing overhead and built-in phase validation. The
stream reader is implemented in a separate module (`stream.rs`) for reuse
by the future Tor transport.
by the Tor transport.
### Connect-on-Send
### Connection Establishment
When `send(addr, data)` is called with no existing connection:
TCP connections use a non-blocking connect model. When FMP needs to reach
a configured peer address, the node calls `connect(addr)` on the transport,
which spawns a background tokio task to perform the TCP handshake and socket
configuration (TCP_NODELAY, keepalive, buffer sizes, TCP_MAXSEG query). The
call returns immediately without blocking the event loop.
1. Connect with configurable timeout (default 5s)
2. Configure socket: `TCP_NODELAY`, keepalive, buffer sizes
3. Read `TCP_MAXSEG` for per-connection MTU
4. Split stream into read/write halves
5. Spawn per-connection receive task
6. Store connection in pool
7. Write packet directly to stream
The node tracks each pending connection in a `PendingConnect` entry. On
every tick, `poll_pending_connects()` calls `connection_state(addr)` to
check progress. When the transport reports `Connected`, the completed
connection is promoted to the established pool (stream split into
read/write halves, per-connection receive task spawned), and the node
initiates the Noise IK link handshake. If the transport reports `Failed`,
the node schedules a retry with exponential backoff.
If connect fails, return error. The node's handshake retry mechanism
handles re-attempts.
As a fallback, `send(addr, data)` still performs synchronous
connect-on-send if no connection exists — this handles the case where a
send arrives before the node-level connect path runs. The non-blocking
path is the primary mechanism for configured peers.
### Session Independence
TCP connection loss does **not** tear down the FIPS peer. Noise keys, MMP
state, and FSP sessions are bound to the peer's npub, not the TCP
connection. The transport reconnects transparently on the next send via
connect-on-send. MMP liveness timeout is the sole authority for peer death.
connection. The transport reconnects transparently via the non-blocking
connect path or connect-on-send fallback. MMP liveness timeout is the sole
authority for peer death.
### Connection Deduplication
@@ -408,13 +436,231 @@ transports:
recv_buf_size: 2097152 # SO_RCVBUF (2 MB)
send_buf_size: 2097152 # SO_SNDBUF (2 MB)
max_inbound_connections: 256 # Resource protection limit
socks5_proxy: "127.0.0.1:9050" # SOCKS5 for outbound (deferred)
```
If `bind_addr` is configured, the transport accepts inbound connections.
Without it, the transport operates in outbound-only mode (no listener
socket is created).
## Tor: The Anonymity Transport
The Tor transport routes FIPS traffic through the Tor network, hiding
a node's IP address from its peers. A node behind Tor connects outbound
through a local Tor SOCKS5 proxy; the remote peer sees the Tor exit
node's IP, not the initiator's. After the Noise IK handshake, the remote
peer knows the initiator's FIPS identity (npub) but not its network
location.
Like TCP, Tor is connection-oriented and reliable. The same TCP-over-TCP
considerations apply — MMP correctly measures the elevated latency and
cost-based parent selection naturally deprioritizes Tor links.
### Architecture
The Tor transport is a separate `TorTransport` implementation, not a TCP
variant, because it manages SOCKS5 proxy negotiation, has different
address semantics (.onion vs IP:port), and has significantly different
latency characteristics. It reuses the FMP header-based stream reader
(`tcp/stream.rs`) for packet framing on the underlying TCP connection.
The transport maintains two pools (same pattern as TCP): a
`ConnectingPool` for background SOCKS5 connection attempts, and an
established pool of `TorConnection` entries. Each `TorConnection` holds
a write half, a per-connection receive task, the negotiated MTU, and
a connection timestamp.
| Property | Value |
| -------- | ----- |
| Addressing | .onion:port or IP:port |
| Default MTU | 1400 bytes |
| Framing | FMP header-based (shared with TCP) |
| Connection model | Non-blocking connect, outbound SOCKS5 + inbound via onion service |
| Platform | Cross-platform (requires external Tor daemon) |
### Address Types
The Tor transport accepts three address formats, parsed into a `TorAddr`
enum:
- **Onion**: `.onion:port` — connects to a Tor hidden service. Both
sides anonymous. (e.g., `abcdef...xyz.onion:8443`)
- **Clearnet IP**: `IP:port` — connects through a Tor exit node to a
remote TCP listener. Hides the initiator's IP; the remote peer sees
the exit node's IP.
- **Clearnet Hostname**: `hostname:port` — hostname is passed through
SOCKS5 for Tor-side DNS resolution, avoiding local DNS leaks. Compatible
with SafeSocks 1. (e.g., `fips.example.com:8443`)
All address types are routed through the same SOCKS5 proxy.
### Connection Establishment
Connection setup follows the same non-blocking pattern as TCP. When FMP
needs to reach a peer, the node calls `connect(addr)` on the transport.
The transport spawns a background tokio task that:
1. Opens a SOCKS5 connection through the local Tor proxy
2. Configures the socket: `TCP_NODELAY`, keepalive (30s)
3. Returns the connected stream
The call returns immediately. `connection_state(addr)` reports progress.
Tor circuit establishment typically takes 1060 seconds (vs milliseconds
for TCP), making non-blocking connect essential — a blocking connect
would stall the entire FMP event loop.
The connect timeout defaults to 120 seconds (vs 5 seconds for TCP),
accounting for Tor circuit setup time. As a fallback, `send(addr, data)`
performs synchronous connect-on-send if no connection exists.
### Inbound via Onion Service (Directory Mode)
In `directory` mode (recommended for production), Tor manages the onion
service via `HiddenServiceDir` in `torrc`. FIPS reads the `.onion` address
from the hostname file at startup and binds a local TCP listener that the
Tor daemon forwards inbound connections to.
This mode enables Tor's `Sandbox 1` (seccomp-bpf) — the strongest single
hardening option — because no control port interaction is required for
onion service management. Tor handles key generation and persistence
directly through the `HiddenServiceDir`.
The inbound accept loop mirrors the TCP transport's pattern: accept
connection, configure socket (TCP_NODELAY, keepalive), spawn a
per-connection receive loop using the shared FMP stream reader. Inbound
connections arrive from `127.0.0.1` (Tor daemon's local forwarding); peer
identity is resolved during the Noise IK handshake, not from the transport
address.
Configuration requires coordinating `torrc` and `fips.yaml`:
```text
# torrc
HiddenServiceDir /var/lib/tor/fips
HiddenServicePort 8443 127.0.0.1:8444
# fips.yaml tor section
mode: "directory"
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
```
The `HiddenServicePort` external port (8443) is what peers connect to.
The bind_addr must match the `HiddenServicePort` target address.
### Session Independence
Same as TCP: Tor connection loss does **not** tear down the FIPS peer.
Noise keys, MMP state, and FSP sessions survive reconnection.
### Bridge Node Pattern
A node running both Tor and UDP transports acts as a bridge between
anonymous and clearnet portions of the mesh:
```text
[Anonymous node] --tor--> [Bridge node] --udp--> [Clearnet node]
```
No special code is needed — FIPS multi-transport routing handles it.
Anonymous nodes connect to the bridge via Tor; the bridge forwards
traffic to clearnet peers over UDP. Clearnet peers never see the
anonymous node's IP.
### Latency Characteristics
Tor adds 200ms2s RTT per circuit. First-packet latency after connection
is higher (~2.8s) due to circuit warm-up. MMP measures this elevated
latency, and cost-based parent selection penalizes Tor links (high SRTT
→ high link cost). ETX is 1.0 since TCP handles retransmission.
Tor throughput is typically 15 Mbps — adequate for control plane and
moderate data transfer, not for bulk transfer.
### Monitoring
In `control_port` mode and optionally in `directory` mode (when
`control_addr` is configured), the transport spawns a background
monitoring task that polls the Tor daemon every 10 seconds via the
control port. The cached monitoring data is exposed through the
`show_transports` control socket query and displayed in fipstop.
Monitoring data includes:
- **Bootstrap progress** (0100%) with INFO logging at milestones
(25/50/75/100%) and WARN if stalled >60s
- **Circuit status** (whether Tor has a working circuit)
- **Network liveness** (up/down) with WARN on transitions
- **Dormant mode** detection with WARN on entry
- **Tor daemon version** and **traffic counters** (bytes read/written)
The control port connection uses cookie authentication by default
(reading from `/var/run/tor/control.authcookie`). Unix socket
connections (`/run/tor/control`) are preferred over TCP for security.
### Configuration
```yaml
transports:
tor:
mode: "socks5" # "socks5", "control_port", or "directory"
socks5_addr: "127.0.0.1:9050" # SOCKS5 proxy address
connect_timeout_ms: 120000 # Connect timeout (120s for Tor circuits)
mtu: 1400 # Default MTU
# control_port mode: monitoring via Tor control port (no inbound)
# control_addr: "/run/tor/control" # Unix socket (preferred) or host:port
# control_auth: "cookie" # "cookie" or "password:<secret>"
# cookie_path: "/var/run/tor/control.authcookie"
# directory mode: inbound via Tor-managed HiddenServiceDir
# directory_service:
# hostname_file: "/var/lib/tor/fips/hostname"
# bind_addr: "127.0.0.1:8444"
# max_inbound_connections: 64
```
Three modes are available:
- **`socks5`** (default): Outbound-only through a SOCKS5 proxy. No
control port, no inbound connections.
- **`control_port`**: Outbound via SOCKS5 plus control port connection
for Tor daemon monitoring. No inbound connections.
- **`directory`** (recommended for inbound): Outbound via SOCKS5 plus
inbound via Tor-managed `HiddenServiceDir` onion service. Optionally
connects to the control port for monitoring when `control_addr` is set.
Enables Tor's `Sandbox 1` for maximum security.
The Tor transport requires an external Tor daemon. Named instances are
supported for multiple proxy endpoints.
### Implementation Roadmap
- Outbound SOCKS5 connections to .onion, clearnet IP, and clearnet
hostname addresses *(implemented)*
- Inbound connections via Tor onion service using `HiddenServiceDir`
directory mode *(implemented)*
- Operator visibility: cached monitoring snapshot, control socket
exposure, fipstop display, bootstrap/liveness logging *(implemented)*
- Embedded `arti` (Rust Tor implementation) for self-contained operation
without an external Tor daemon *(future)*
### Statistics
The transport tracks per-instance statistics:
| Counter | Description |
| ------- | ----------- |
| `packets_sent` / `bytes_sent` | Successful sends |
| `packets_recv` / `bytes_recv` | Successful receives |
| `send_errors` / `recv_errors` | Send/receive failures |
| `connections_established` | Successful SOCKS5 connections |
| `connect_timeouts` | Connection timeout count |
| `connect_refused` | Connection refused count |
| `socks5_errors` | SOCKS5 protocol errors |
| `mtu_exceeded` | Packets rejected for MTU violation |
| `connections_accepted` | Accepted inbound connections via onion service |
| `connections_rejected` | Rejected inbound connections (limit exceeded) |
| `control_errors` | Tor control port errors |
## Discovery
Discovery determines that a FIPS-capable endpoint is reachable at a given
@@ -423,7 +669,7 @@ detection — a new TCP connection or UDP packet from an unknown source is not
discovery; a FIPS-specific announcement or response is.
Discovery is an optional transport capability. Transports that don't support
it (configured UDP endpoints, TCP) simply don't provide discovery events.
it (configured UDP endpoints, TCP, Tor) simply don't provide discovery events.
FMP handles both cases uniformly: with discovery, it waits for events then
initiates link setup; without discovery, it initiates link setup directly to
configured addresses.
@@ -453,7 +699,7 @@ X." FMP does not need to distinguish beacons from query responses.
For internet-reachable transports, a node publishes a signed Nostr event
containing its FIPS discovery information — public key and reachable
transport endpoints (UDP IP:port, TCP IP:port, .onion address). Other FIPS
transport endpoints (UDP host:port, TCP host:port, .onion address). Other FIPS
nodes subscribing on the same relays learn about available peers.
Nostr relay discovery is not a transport — it is a discovery service that
@@ -472,13 +718,13 @@ Key properties:
### Current State
> **Implemented**: UDP and TCP peers are configured via YAML. Ethernet
> peers are discovered via beacon broadcast — the `discover()` trait
> method returns newly seen endpoints, and per-transport `auto_connect()`
> / `accept_connections()` policies control whether discovered peers are
> connected automatically or require explicit configuration. TCP has no
> discovery mechanism (peers are configured). Nostr relay discovery is
> not yet implemented.
> **Implemented**: UDP, TCP, Tor, and Ethernet peers can be configured
> statically via YAML. Ethernet peers can also be discovered via beacon
> broadcast — the `discover()` trait method returns newly seen endpoints,
> and per-transport `auto_connect()` / `accept_connections()` policies
> control whether discovered peers are connected automatically or require
> explicit configuration. TCP and Tor have no discovery mechanism.
> Nostr relay discovery is not yet implemented.
## Transport Interface
@@ -496,6 +742,8 @@ link_mtu(addr) → u16 Per-link MTU (defaults to mtu())
start() → lifecycle Bring transport up (bind socket, open device)
stop() → lifecycle Bring transport down
send(addr, data) → delivery Send datagram to transport address
connect(addr) → () Initiate non-blocking connection (connection-oriented only)
connection_state(addr)→ ConnectionState Poll connection status (None/Connecting/Connected/Failed)
close_connection(addr)→ () Close a specific connection (no-op for connectionless)
congestion() → TransportCongestion Local congestion indicators (optional)
discover() → Vec<DiscoveredPeer> Report discovered FIPS endpoints (optional)
@@ -553,13 +801,14 @@ on all forwarded datagrams.
| Transport | Congestion Source | Mechanism |
| --------- | ----------------- | --------- |
| UDP | `SO_RXQ_OVFL` kernel drop counter | `recvmsg()` ancillary data on every packet |
| TCP | Not yet implemented | Returns `None` (TCP handles congestion internally) |
| Ethernet | Not yet implemented | Returns `None` |
| TCP | Not implemented | Returns `None` (TCP handles congestion internally) |
| Tor | Not implemented | Returns `None` (TCP handles congestion internally) |
| Ethernet | Not implemented | Returns `None` |
### Transport Addresses
Transport addresses (`TransportAddr`) are opaque byte vectors. The transport
layer interprets them (e.g., UDP parses "ip:port" strings); all layers above
layer interprets them (e.g., UDP/TCP resolve "host:port" strings (IP fast path, DNS fallback with 60s cache for UDP)); all layers above
treat them as opaque handles passed back to the transport for sending.
### Transport State Machine
@@ -579,10 +828,10 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| Transport | Status | Notes |
| --------- | ------ | ----- |
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
| TCP/IP | **Implemented** | FMP header-based framing, connect-on-send, per-connection MSS MTU |
| TCP/IP | **Implemented** | FMP header-based framing, non-blocking connect, per-connection MSS MTU |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
| WiFi | Future direction | Infrastructure mode = Ethernet driver |
| Tor | Future direction | High latency, .onion addressing |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| BLE | Future direction | ATT_MTU negotiation, per-link MTU |
| Radio | Future direction | Constrained MTU (51222 bytes) |
| Serial | Future direction | SLIP/COBS framing, point-to-point |

View File

@@ -21,7 +21,8 @@ Datagram-oriented transports (UDP, raw Ethernet, radio) preserve natural
packet boundaries and require no additional framing. Stream-oriented
transports (TCP, WebSocket, Tor) must delineate FIPS packets within the
byte stream; the common prefix `payload_len` field provides this
framing directly.
framing directly. TCP and Tor share a common stream reader
(`tcp/stream.rs`) that implements this framing.
**Ethernet data frame header.** The Ethernet transport prepends a 3-byte
header before the FMP payload on data frames: a 1-byte frame type
@@ -254,7 +255,11 @@ With link overhead: 1,072 bytes.
### LookupRequest (0x30)
Coordinate discovery request, flooded through the mesh.
Coordinate discovery request, routed through the spanning tree via
bloom-filter-guided forwarding. Each transit node forwards only to tree
peers (parent + children) whose bloom filter contains the target.
Request_id dedup in recent_requests handles edge cases from tree
restructuring.
![LookupRequest](diagrams/lookup-request.svg)
@@ -268,20 +273,19 @@ Coordinate discovery request, flooded through the mesh.
| 42 | min_mtu | 2 bytes LE | Minimum transport MTU the origin requires (0 = no requirement) |
| 44 | origin_coords_cnt | 2 bytes LE | Number of coordinate entries |
| 46 | origin_coords | 16 x n bytes | Requester's ancestry (NodeAddr only) |
| 46 + 16n | visited_hash_cnt | 1 byte | Hash count for visited filter |
| 47 + 16n | visited_bits | 256 bytes | Compact bloom of visited nodes |
**Size**: `303 + (n x 16)` bytes, where n = origin depth + 1
**Size**: `46 + (n x 16)` bytes, where n = origin depth + 1
| Origin Depth | Payload |
| ------------ | ------- |
| 3 | 351 bytes |
| 5 | 383 bytes |
| 10 | 463 bytes |
| 3 | 110 bytes |
| 5 | 142 bytes |
| 10 | 222 bytes |
### LookupResponse (0x31)
Coordinate discovery response, greedy-routed back to requester.
Coordinate discovery response, reverse-path routed back to the
requester via the transit nodes that forwarded the request.
![LookupResponse](diagrams/lookup-response.svg)

BIN
docs/logos/fips_banner.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

BIN
docs/logos/fips_logo.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

View File

@@ -0,0 +1,11 @@
target/
.git/
.github/
testing/chaos/
testing/static/
testing/sidecar/
testing/tor/
fips
fipsctl
../sidecar/fips
../sidecar/fipsctl

View File

@@ -0,0 +1,63 @@
# -----------------------------------------------------------------------------
# Stage 1 — build
# Compile fips and fipsctl inside a Rust toolchain image so no local Rust
# install is needed and the image is fully reproducible.
# -----------------------------------------------------------------------------
FROM rust:slim-trixie AS builder
RUN apt-get update && \
apt-get install -y --no-install-recommends \
pkg-config \
libssl-dev && \
rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Copy the full workspace source. .dockerignore filters target/ and other
# large paths so the context stays small.
COPY . .
# Build only the two binaries needed at runtime; skip the TUI (fipstop)
# to avoid pulling in ratatui and crossterm. The daemon does not require
# any default features to run.
RUN cargo build --release --no-default-features \
--bin fips \
--bin fipsctl
# -----------------------------------------------------------------------------
# Stage 2 — runtime
# Minimal Debian image with only the packages required at runtime.
# -----------------------------------------------------------------------------
FROM debian:trixie-slim
RUN apt-get update && \
apt-get install -y --no-install-recommends \
iproute2 \
iptables \
dnsmasq \
procps \
curl && \
rm -rf /var/lib/apt/lists/*
# dnsmasq: forward .fips queries to the FIPS daemon DNS resolver;
# forward everything else to the upstream resolver passed in at runtime.
# The upstream placeholder is replaced by entrypoint.sh with the pod's
# original nameserver(s) from /etc/resolv.conf before dnsmasq starts.
RUN printf '%s\n' \
'port=53' \
'listen-address=127.0.0.1' \
'bind-interfaces' \
'server=/fips/127.0.0.1#5354' \
'no-resolv' \
> /etc/dnsmasq.d/fips.conf
COPY --from=builder /build/target/release/fips /usr/local/bin/fips
COPY --from=builder /build/target/release/fipsctl /usr/local/bin/fipsctl
COPY examples/k8s-sidecar/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD fipsctl show status
ENTRYPOINT ["/entrypoint.sh"]

View File

@@ -0,0 +1,176 @@
# FIPS Kubernetes Sidecar
Run FIPS as a sidecar container in a Kubernetes Pod, injecting the FIPS mesh
network interface into every other container in the Pod. All containers in
the Pod share the same network namespace, so once the sidecar creates `fips0`
it is immediately visible to the app container(s) without any further
configuration.
## Quick Start
```bash
# 1. Build the sidecar image
cd examples/k8s-sidecar
./scripts/build.sh --tag fips-k8s-sidecar:latest
# 2. Push to your registry (replace with your own)
docker tag fips-k8s-sidecar:latest registry.example.com/fips-k8s-sidecar:latest
docker push registry.example.com/fips-k8s-sidecar:latest
# 3. Create the identity secret
kubectl create secret generic fips-identity \
--from-literal=nsec=$(openssl rand -hex 32)
# 4. Deploy
kubectl apply -f pod.yaml
```
## How It Works
In Kubernetes, all containers in a Pod share the same network namespace. The
fips sidecar:
1. Generates `/etc/fips/fips.yaml` from environment variables.
2. Rewrites `/etc/resolv.conf` to route `.fips` DNS through dnsmasq (which
forwards `.fips` names to the FIPS daemon's built-in DNS resolver and
everything else to the original cluster DNS).
3. Applies `iptables` isolation rules so that the pod's physical interface
(`eth0`) only carries FIPS transport traffic.
4. Starts dnsmasq and then `exec`s the FIPS daemon.
The app container starts concurrently and immediately sees `lo`, `eth0`, and
`fips0`. DNS for `<npub>.fips` names resolves to `fd::/8` addresses via the
dnsmasq → FIPS daemon pipeline.
```text
┌────────────────────────────────────────────────────┐
│ Kubernetes Pod (shared network namespace) │
│ │
│ ┌─────────────────┐ ┌───────────────────────────┐│
│ │ fips (sidecar) │ │ app container(s) ││
│ │ │ │ ││
│ │ fips daemon │ │ your workload ││
│ │ fipsctl │ │ sees: lo, eth0, fips0 ││
│ │ dnsmasq │ │ ││
│ └─────────────────┘ └───────────────────────────┘│
│ │
│ Interfaces: │
│ lo — loopback (unrestricted) │
│ eth0 — pod CNI interface (iptables: FIPS only) │
│ fips0 — FIPS TUN (fd00::/8, unrestricted) │
└────────────────────────────────────────────────────┘
```
## Network Isolation
`FIPS_ISOLATE` controls whether the sidecar locks down `eth0`:
| Value | Behaviour |
|---|---|
| `false` **(default)** | `fips0` is added alongside normal cluster networking. `eth0` continues to work — services, DNS, other pods, and the internet are all reachable as normal. Application traffic to FIPS mesh peers uses `fips0`. |
| `true` | All traffic on `eth0` is dropped except FIPS transport (UDP 2121). The app container can **only** communicate via `fips0`. Use this for deployments where the workload must never bypass the mesh. |
> **Common gotcha**: if you deploy the sidecar and find that services or
> other pods are suddenly unreachable, check that `FIPS_ISOLATE` is not
> set to `true`. The mesh-only mode is intentionally strict and will break
> normal cluster connectivity.
## Requirements
| Requirement | Notes |
|---|---|
| `NET_ADMIN` capability | Required for TUN creation and iptables |
| `/dev/net/tun` | HostPath volume mount (see `pod.yaml`) |
| Linux kernel ≥ 4.9 | Standard on all current distributions |
| No gVisor / kata-containers | Requires real kernel TUN support |
## Environment Variables
| Variable | Default | Description |
|---|---|---|
| `FIPS_NSEC` | *(required)* | Node secret key (hex or nsec1 bech32) |
| `FIPS_PEER_NPUB` | *(empty)* | Single peer npub |
| `FIPS_PEER_ADDR` | *(empty)* | Single peer transport address (`host:port`) |
| `FIPS_PEER_ALIAS` | `peer` | Single peer alias |
| `FIPS_PEER_TRANSPORT` | `udp` | Single peer transport type |
| `FIPS_PEERS_JSON` | *(empty)* | JSON array of peer objects (overrides single-peer vars). **Note**: the included parser handles simple host:port addresses only; IPv6 addresses with brackets may not parse correctly. |
| `FIPS_UDP_BIND` | `0.0.0.0:2121` | UDP transport bind address |
| `FIPS_UDP_PORT` | *(from `FIPS_UDP_BIND`)* | UDP port for iptables rules |
| `FIPS_TCP_BIND` | *(disabled)* | TCP transport bind address |
| `FIPS_TUN_NAME` | `fips0` | TUN interface name |
| `FIPS_TUN_MTU` | `1280` | TUN MTU |
| `FIPS_ISOLATE` | `false` | Apply iptables isolation (blocks all eth0 traffic except FIPS transport) |
| `FIPS_POD_IFACE` | `eth0` | Pod physical interface name |
| `FIPS_REWRITE_DNS` | `true` | Rewrite `/etc/resolv.conf` |
| `RUST_LOG` | `info` | FIPS log level |
### Multiple Peers
Use `FIPS_PEERS_JSON` to configure more than one peer:
```yaml
- name: FIPS_PEERS_JSON
value: |
[
{"npub":"npub1abc...","alias":"gw1","addr":"203.0.113.10:2121","transport":"udp"},
{"npub":"npub1def...","alias":"gw2","addr":"198.51.100.5:2121","transport":"udp"}
]
```
Each object supports the keys: `npub` (required), `addr` (required),
`alias`, `transport`, `priority`.
Alternatively, mount a hand-crafted `fips.yaml` and set `FIPS_NSEC` to
anything (the mount takes precedence because the entrypoint writes to
`/etc/fips/fips.yaml` which is then overridden by the volume):
```yaml
volumes:
- name: fips-config
configMap:
name: my-fips-config
containers:
- name: fips
volumeMounts:
- name: fips-config
mountPath: /etc/fips/fips.yaml
subPath: fips.yaml
```
### JSON Parsing Limitations
`FIPS_PEERS_JSON` supports parsing with a simple awk-based parser. It handles common `host:port` addresses but does not support IPv6 addresses with brackets (e.g., `[::1]:2121`) or values containing embedded commas/colons. For complex configs, install `jq` in the container and modify the entrypoint, or use a mounted `fips.yaml`.
## Files
| File | Purpose |
|---|---|
| `Dockerfile` | Builds the sidecar image |
| `entrypoint.sh` | Generates config, rewrites DNS, applies iptables, starts fips |
| `pod.yaml` | Example Pod manifest (single app container + fips sidecar) |
| `scripts/build.sh` | Compiles FIPS and builds the Docker image |
## Troubleshooting
**`FIPS_NSEC is required`** — The secret was not injected. Verify the
`secretKeyRef` name and key match the secret you created with `kubectl
create secret`.
**`fips0` not visible in app container** — Check that the fips sidecar
started successfully: `kubectl logs <pod> -c fips`. The sidecar must start
and run the daemon before `fips0` appears. Add a startup probe or
`postStart` lifecycle hook to your app container if you need to wait.
**iptables errors** — The `NET_ADMIN` capability and `/dev/net/tun` volume
mount are both required. Verify they are present in the Pod spec.
**`.fips` DNS not resolving** — Check that `FIPS_REWRITE_DNS=true` and that
dnsmasq started: `kubectl exec <pod> -c fips -- pgrep dnsmasq`. Verify the
FIPS DNS listener: `kubectl exec <pod> -c fips -- fipsctl show status`.
**CNI uses a different interface name** — Set `FIPS_POD_IFACE` to match your
CNI's interface name (e.g. `ens3`, `net1` for Multus secondary interfaces).
**gVisor / kata-containers** — These sandboxed runtimes intercept syscalls
and do not support `AF_PACKET` or `/dev/net/tun` in the same way as a
standard kernel. Use a standard RuntimeClass for FIPS sidecar pods.

View File

@@ -0,0 +1,309 @@
#!/bin/bash
# FIPS Kubernetes sidecar entrypoint.
#
# Generates /etc/fips/fips.yaml from environment variables, rewrites the
# pod's resolv.conf to route .fips DNS through dnsmasq, applies iptables
# isolation rules so that the shared pod network namespace can only reach
# the outside world via the FIPS mesh, then launches the FIPS daemon.
#
# Required environment variables:
# FIPS_NSEC Node secret key (hex or nsec1 bech32)
#
# Optional environment variables (single peer shorthand):
# FIPS_PEER_NPUB Peer's npub
# FIPS_PEER_ADDR Peer's transport address (e.g. 203.0.113.10:2121)
# FIPS_PEER_ALIAS Human-readable peer name (default: peer)
# FIPS_PEER_TRANSPORT Transport type (default: udp)
#
# Multiple peers via JSON (overrides single-peer vars when set):
# FIPS_PEERS_JSON JSON array of peer objects, e.g.:
# '[{"npub":"npub1...","alias":"gw","addr":"1.2.3.4:2121","transport":"udp"}]'
#
# Transport / TUN tuning:
# FIPS_UDP_BIND UDP bind address (default: 0.0.0.0:2121)
# FIPS_UDP_PORT UDP transport port (derived from FIPS_UDP_BIND when unset)
# FIPS_TCP_BIND TCP bind address (default: disabled)
# FIPS_TUN_NAME TUN interface name (default: fips0)
# FIPS_TUN_MTU TUN interface MTU (default: 1280, IPv6 minimum)
#
# Network isolation:
# FIPS_ISOLATE Apply iptables isolation (default: false)
# Set to "true" for mesh-only deployments where the app
# must not communicate outside the FIPS mesh.
# Set to "false" to add fips0 alongside normal cluster
# networking without restricting eth0.
# FIPS_POD_IFACE Pod network interface (default: eth0)
# Adjust if your CNI uses a different name (e.g. ens3).
#
# DNS:
# FIPS_REWRITE_DNS Rewrite /etc/resolv.conf (default: true)
# Set to "false" if you manage DNS via another mechanism.
#
# Logging:
# RUST_LOG FIPS log level (default: info)
set -euo pipefail
# ---------------------------------------------------------------------------
# IPv6 kernel configuration
#
# net.ipv6.conf.all.disable_ipv6, net.ipv6.conf.default.disable_ipv6, and
# net.ipv6.bindv6only are set via the pod securityContext.sysctls field in
# pod.yaml so that kubelet applies them before any container starts.
# Nothing to do here.
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# Defaults
# ---------------------------------------------------------------------------
FIPS_NSEC="${FIPS_NSEC:?FIPS_NSEC is required — provide your node secret key}"
FIPS_UDP_BIND="${FIPS_UDP_BIND:-0.0.0.0:2121}"
FIPS_TCP_BIND="${FIPS_TCP_BIND:-}"
FIPS_TUN_NAME="${FIPS_TUN_NAME:-fips0}"
FIPS_TUN_MTU="${FIPS_TUN_MTU:-1280}"
FIPS_ISOLATE="${FIPS_ISOLATE:-false}"
FIPS_POD_IFACE="${FIPS_POD_IFACE:-eth0}"
FIPS_REWRITE_DNS="${FIPS_REWRITE_DNS:-true}"
FIPS_PEER_TRANSPORT="${FIPS_PEER_TRANSPORT:-udp}"
FIPS_PEER_ALIAS="${FIPS_PEER_ALIAS:-peer}"
# Derive the UDP port for iptables rules from FIPS_UDP_BIND (last colon-field).
FIPS_UDP_PORT="${FIPS_UDP_PORT:-${FIPS_UDP_BIND##*:}}"
mkdir -p /etc/fips
# ---------------------------------------------------------------------------
# Build peers YAML section
# ---------------------------------------------------------------------------
build_peers_yaml() {
# FIPS_PEERS_JSON wins when set (supports multiple peers).
if [ -n "${FIPS_PEERS_JSON:-}" ]; then
# Parse the JSON array with awk — no jq required.
# Expected element keys: npub, alias, addr, transport, priority
#
# Note: This parser handles simple host:port addresses only.
# IPv6 addresses with brackets ([::1]:2121) or values containing
# commas/colons may not parse correctly. Use single-peer env vars
# for complex configs or install jq and replace this block.
echo "$FIPS_PEERS_JSON" | awk '
BEGIN { RS="}"; FS="," }
{
npub=""; alias="peer"; addr=""; transport="udp"; priority=100
for (i=1; i<=NF; i++) {
gsub(/[{}\[\] \t\n\r]/, "", $i)
if ($i ~ /"npub"/) { split($i,a,":"); npub=a[2]; gsub(/"/,"",npub) }
if ($i ~ /"alias"/) { split($i,a,":"); alias=a[2]; gsub(/"/,"",alias) }
if ($i ~ /"addr"/) { n=split($i,a,":"); addr=a[2]":"a[3]; gsub(/"/,"",addr) }
if ($i ~ /"transport"/) { split($i,a,":"); transport=a[2]; gsub(/"/,"",transport) }
if ($i ~ /"priority"/) { split($i,a,":"); priority=a[2]; gsub(/"/,"",priority) }
}
if (npub != "" && addr != "") {
printf " - npub: \"%s\"\n alias: \"%s\"\n addresses:\n - transport: %s\n addr: \"%s\"\n priority: %s\n connect_policy: auto_connect\n auto_reconnect: true\n", npub, alias, transport, addr, priority
}
}'
return
fi
# Single-peer shorthand via individual env vars.
if [ -n "${FIPS_PEER_NPUB:-}" ] && [ -n "${FIPS_PEER_ADDR:-}" ]; then
cat <<EOF
- npub: "${FIPS_PEER_NPUB}"
alias: "${FIPS_PEER_ALIAS}"
addresses:
- transport: ${FIPS_PEER_TRANSPORT}
addr: "${FIPS_PEER_ADDR}"
connect_policy: auto_connect
auto_reconnect: true
EOF
return
fi
# No peers configured — start standalone (useful for a new node joining
# via discovery or when peers are added later via fipsctl).
echo " []"
}
# ---------------------------------------------------------------------------
# Build optional TCP transport stanza
# ---------------------------------------------------------------------------
build_tcp_yaml() {
if [ -n "$FIPS_TCP_BIND" ]; then
cat <<EOF
tcp:
bind_addr: "${FIPS_TCP_BIND}"
EOF
fi
}
# ---------------------------------------------------------------------------
# Generate /etc/fips/fips.yaml
# ---------------------------------------------------------------------------
PEERS_YAML="$(build_peers_yaml)"
TCP_YAML="$(build_tcp_yaml)"
cat > /etc/fips/fips.yaml <<EOF
node:
identity:
nsec: "${FIPS_NSEC}"
control:
enabled: true
socket_path: "/run/fips/control.sock"
tun:
enabled: true
name: ${FIPS_TUN_NAME}
mtu: ${FIPS_TUN_MTU}
dns:
enabled: true
bind_addr: "127.0.0.1"
port: 5354
transports:
udp:
bind_addr: "${FIPS_UDP_BIND}"
${TCP_YAML}
peers:
${PEERS_YAML}
EOF
echo "[fips-sidecar] Generated /etc/fips/fips.yaml"
# ---------------------------------------------------------------------------
# Rewrite /etc/resolv.conf to route .fips DNS through dnsmasq
#
# Strategy:
# 1. Harvest the nameservers the kubelet injected.
# 2. Write dnsmasq upstream rules for each harvested nameserver.
# 3. Point /etc/resolv.conf at 127.0.0.1 (dnsmasq).
# 4. dnsmasq forwards .fips → FIPS daemon (127.0.0.1:5354),
# everything else → original cluster DNS (typically 169.254.20.10 or
# the kube-dns ClusterIP injected by kubelet).
# ---------------------------------------------------------------------------
if [ "$FIPS_REWRITE_DNS" = "true" ]; then
# Collect existing upstream nameservers before we overwrite resolv.conf.
UPSTREAM_NS=()
while IFS= read -r line; do
if [[ "$line" =~ ^nameserver[[:space:]]+(.+)$ ]]; then
ns="${BASH_REMATCH[1]}"
# Skip loopback — we're about to replace it.
if [[ "$ns" != "127."* ]] && [[ "$ns" != "::1" ]]; then
UPSTREAM_NS+=("$ns")
fi
fi
done < /etc/resolv.conf
# Append upstream server lines to dnsmasq fips config.
# (The placeholder in the image has no upstream server= lines yet.)
for ns in "${UPSTREAM_NS[@]}"; do
echo "server=${ns}" >> /etc/dnsmasq.d/fips.conf
done
# If no upstream was found fall back to Google (better than no resolution).
if [ "${#UPSTREAM_NS[@]}" -eq 0 ]; then
echo "[fips-sidecar] WARNING: no upstream nameserver found in /etc/resolv.conf; falling back to 8.8.8.8"
echo "server=8.8.8.8" >> /etc/dnsmasq.d/fips.conf
fi
# Preserve the search/domain lines so pod DNS short-names still work.
SEARCH_LINE=""
while IFS= read -r line; do
if [[ "$line" =~ ^(search|domain)[[:space:]] ]]; then
SEARCH_LINE="$line"
break
fi
done < /etc/resolv.conf
{
echo "nameserver 127.0.0.1"
[ -n "$SEARCH_LINE" ] && echo "$SEARCH_LINE"
} > /etc/resolv.conf
echo "[fips-sidecar] DNS rewritten: upstream ${UPSTREAM_NS[*]:-8.8.8.8} → dnsmasq → fips/cluster"
fi
# ---------------------------------------------------------------------------
# iptables isolation
#
# Goal: only FIPS transport traffic may leave/enter the pod via the physical
# interface. All other pod-to-external traffic is dropped. Traffic on the
# fips0 TUN and loopback is unrestricted. The app container sharing this
# network namespace therefore can only communicate over the FIPS mesh.
#
# Skip when FIPS_ISOLATE=false (e.g. already handled by a NetworkPolicy,
# or when the operator wants unrestricted egress alongside mesh traffic).
# ---------------------------------------------------------------------------
if [ "$FIPS_ISOLATE" = "true" ]; then
IFACE="$FIPS_POD_IFACE"
UDP_PORT="$FIPS_UDP_PORT"
# IPv4: allow loopback + FIPS UDP transport; drop everything else on
# the pod's physical interface.
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o "$IFACE" -p udp --dport "$UDP_PORT" -j ACCEPT
iptables -A OUTPUT -o "$IFACE" -p udp --sport "$UDP_PORT" -j ACCEPT
iptables -A INPUT -i "$IFACE" -p udp --dport "$UDP_PORT" -j ACCEPT
iptables -A INPUT -i "$IFACE" -p udp --sport "$UDP_PORT" -j ACCEPT
# Allow TCP transport port when configured.
if [ -n "$FIPS_TCP_BIND" ]; then
TCP_PORT="${FIPS_TCP_BIND##*:}"
iptables -A OUTPUT -o "$IFACE" -p tcp --dport "$TCP_PORT" -j ACCEPT
iptables -A INPUT -i "$IFACE" -p tcp --sport "$TCP_PORT" -j ACCEPT
iptables -A OUTPUT -o "$IFACE" -p tcp --sport "$TCP_PORT" -j ACCEPT
iptables -A INPUT -i "$IFACE" -p tcp --dport "$TCP_PORT" -j ACCEPT
fi
# Allow established/related so existing connections aren't torn down mid-flight.
iptables -A INPUT -i "$IFACE" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o "$IFACE" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o "$IFACE" -j DROP
iptables -A INPUT -i "$IFACE" -j DROP
# IPv6: allow loopback and fips TUN; block physical interface entirely.
ip6tables -A OUTPUT -o lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o "$FIPS_TUN_NAME" -j ACCEPT
ip6tables -A INPUT -i "$FIPS_TUN_NAME" -j ACCEPT
ip6tables -A OUTPUT -o "$IFACE" -j DROP
ip6tables -A INPUT -i "$IFACE" -j DROP
echo "[fips-sidecar] iptables isolation applied on ${IFACE} (UDP ${UDP_PORT})"
fi
# ---------------------------------------------------------------------------
# TCP MSS clamping on fips0
#
# The kernel derives MSS from the TUN MTU: 1280 - 60 = 1220. But FIPS adds
# FIPS_IPV6_OVERHEAD (77) bytes of encapsulation that the kernel doesn't know
# Clamp the MSS in TCP SYN packets traversing fips0 to the path MTU so that
# segments fit within the effective MTU after FIPS encapsulation.
# This is applied regardless of FIPS_ISOLATE mode since it's needed for
# correct TCP negotiation even when normal network access is allowed.
# ---------------------------------------------------------------------------
ip6tables -t mangle -A FORWARD -o "$FIPS_TUN_NAME" -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
ip6tables -t mangle -A FORWARD -i "$FIPS_TUN_NAME" -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
ip6tables -t mangle -A OUTPUT -o "$FIPS_TUN_NAME" -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
echo "[fips-sidecar] TCP MSS clamped to PMTU on ${FIPS_TUN_NAME}"
# ---------------------------------------------------------------------------
# Start dnsmasq and launch FIPS daemon
# ---------------------------------------------------------------------------
dnsmasq --conf-dir=/etc/dnsmasq.d
echo "[fips-sidecar] dnsmasq started"
exec fips --config /etc/fips/fips.yaml

View File

@@ -0,0 +1,207 @@
# FIPS Kubernetes sidecar — example Pod manifest
#
# WARNING: The Secret below contains a placeholder nsec. Replace it with your
# real key before applying. Do NOT commit real keys to version control.
# Generate a fresh key with: openssl rand -hex 32
#
# Apply:
# kubectl apply -f pod.yaml
#
# Requirements on the node:
# - /dev/net/tun must exist (standard on all modern Linux kernels)
# - The container runtime must honour NET_ADMIN (true for containerd/CRI-O
# with default settings; may require a RuntimeClass tweak for gVisor)
#
# Kubernetes version: 1.28+
# -----------------------------------------------------------------------------
# Secret — node identity
# Replace the nsec value with your real key (hex or nsec1 bech32).
# In production, manage this with a secrets operator (External Secrets,
# Sealed Secrets, Vault Agent, etc.) rather than a plaintext manifest.
# -----------------------------------------------------------------------------
apiVersion: v1
kind: Secret
metadata:
name: fips-identity
type: Opaque
stringData:
nsec: "replace-this-with-your-hex-or-nsec1-secret-key"
---
apiVersion: v1
kind: Pod
metadata:
name: fips-sidecar-example
labels:
app: fips-sidecar-example
spec:
# Containers in the same Pod share the same network namespace automatically.
# The fips sidecar creates fips0; the app container sees it immediately.
#
# Start order: initContainers run first, then all containers start
# concurrently. If your app needs the TUN interface to be ready before it
# starts, use a readiness probe on the sidecar and a dependency in the app
# (or use a startup init container that waits for fips0 to appear).
# net.ipv6.* sysctls are namespaced per-pod in Linux. Declaring them here
# allows the fips sidecar entrypoint to set them at runtime via sysctl(8).
# bindv6only=0 means apps listening on 0.0.0.0 will also receive IPv6
# connections arriving via fips0, without needing to be rewritten.
# These are "unsafe" sysctls and require the kubelet --allowed-unsafe-sysctls
# flag or a corresponding RuntimeClass / PodSecurityPolicy exception.
securityContext:
sysctls:
- name: net.ipv6.conf.all.disable_ipv6
value: "0"
- name: net.ipv6.conf.default.disable_ipv6
value: "0"
- name: net.ipv6.bindv6only
value: "0"
volumes:
- name: tun-device
hostPath:
path: /dev/net/tun
type: CharDevice
# Optional: mount a pre-built fips.yaml instead of generating from env.
# - name: fips-config
# configMap:
# name: fips-config
containers:
# -----------------------------------------------------------------------
# FIPS sidecar — owns the network namespace setup
# -----------------------------------------------------------------------
- name: fips
image: your-registry.example.com/fips-sidecar:latest
imagePullPolicy: Always
securityContext:
capabilities:
add:
- NET_ADMIN # required: TUN creation and iptables
runAsNonRoot: false # fips needs root for iptables/TUN
runAsUser: 0
volumeMounts:
- name: tun-device
mountPath: /dev/net/tun
env:
# --- Identity (required) ---
- name: FIPS_NSEC
valueFrom:
secretKeyRef:
name: fips-identity
key: nsec
# --- Single peer shorthand ---
# Remove these and use FIPS_PEERS_JSON below for multiple peers.
- name: FIPS_PEER_NPUB
value: "" # e.g. npub1abc...
- name: FIPS_PEER_ADDR
value: "" # e.g. 203.0.113.10:2121
- name: FIPS_PEER_ALIAS
value: "gateway"
- name: FIPS_PEER_TRANSPORT
value: "udp"
# --- Multiple peers via JSON (overrides single-peer vars when non-empty) ---
# - name: FIPS_PEERS_JSON
# value: |
# [
# {"npub":"npub1abc...","alias":"gw1","addr":"203.0.113.10:2121","transport":"udp"},
# {"npub":"npub1def...","alias":"gw2","addr":"198.51.100.5:2121","transport":"udp"}
# ]
# --- Transport tuning ---
- name: FIPS_UDP_BIND
value: "0.0.0.0:2121"
# Uncomment to enable TCP transport (useful on networks that block UDP):
# - name: FIPS_TCP_BIND
# value: "0.0.0.0:8443"
# --- TUN interface ---
- name: FIPS_TUN_NAME
value: "fips0"
- name: FIPS_TUN_MTU
value: "1280"
# --- Network isolation ---
# "false" — fips0 is added alongside normal cluster networking. eth0
# continues to work as usual (services, DNS, other pods all reachable).
# "true" — eth0 is locked to FIPS transport only; app can only talk
# via fips0. Use for high-security mesh-only deployments.
- name: FIPS_ISOLATE
value: "false"
# Adjust if your CNI names the pod interface differently (e.g. ens3, net1).
- name: FIPS_POD_IFACE
value: "eth0"
# --- DNS ---
# Set to "false" if you manage pod DNS externally.
- name: FIPS_REWRITE_DNS
value: "true"
# --- Logging ---
- name: RUST_LOG
value: "info"
ports:
- name: fips-udp
containerPort: 2121
protocol: UDP
# - name: fips-tcp
# containerPort: 8443
# protocol: TCP
readinessProbe:
exec:
command: ["fipsctl", "show", "status"]
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
livenessProbe:
exec:
command: ["fipsctl", "show", "status"]
initialDelaySeconds: 15
periodSeconds: 30
failureThreshold: 3
resources:
requests:
cpu: "50m"
memory: "32Mi"
limits:
cpu: "500m"
memory: "128Mi"
# -----------------------------------------------------------------------
# App container — shares the network namespace created by the fips sidecar
# -----------------------------------------------------------------------
- name: app
image: your-app:latest
imagePullPolicy: IfNotPresent
# No special capabilities needed — app uses fips0 like a normal interface.
securityContext:
runAsNonRoot: true
# The app sees: lo, eth0 (isolated by iptables), fips0 (mesh).
# All outbound traffic from the app flows through fips0.
# DNS resolution for <npub>.fips names works via dnsmasq on 127.0.0.1.
command: ["sleep", "infinity"] # replace with your actual workload command
resources:
requests:
cpu: "100m"
memory: "64Mi"
limits:
cpu: "1000m"
memory: "512Mi"
# Restart policy for the Pod. Use "Always" for long-running workloads.
restartPolicy: Always

View File

@@ -0,0 +1,37 @@
#!/bin/bash
# Build the FIPS Kubernetes sidecar Docker image.
# The build happens entirely inside Docker (multi-arch friendly).
# Usage: ./scripts/build.sh [--tag TAG] [-- <extra docker buildx args>]
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
DOCKER_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
PROJECT_ROOT="$(cd "$DOCKER_DIR/../.." && pwd)"
IMAGE_TAG="fips-k8s-sidecar:latest"
while [[ $# -gt 0 ]]; do
case "$1" in
--tag) IMAGE_TAG="$2"; shift 2 ;;
--) shift; break ;;
*) echo "Unknown option: $1" >&2; exit 1 ;;
esac
done
if [ ! -f "$PROJECT_ROOT/Cargo.toml" ]; then
echo "Error: Cannot find Cargo.toml at $PROJECT_ROOT" >&2
echo "Expected layout: <project-root>/examples/k8s-sidecar/scripts/build.sh" >&2
exit 1
fi
echo "Building Docker image: $IMAGE_TAG"
docker build \
-t "$IMAGE_TAG" \
-f "$DOCKER_DIR/Dockerfile" \
"$@" \
"$PROJECT_ROOT"
echo ""
echo "Done. Image: $IMAGE_TAG"
echo ""
echo "Push to your registry, then apply the example manifest:"
echo " kubectl apply -f examples/k8s-sidecar/pod.yaml"

View File

@@ -0,0 +1,12 @@
# FIPS sidecar default configuration.
# Override these values or create a .env.local file.
# Node identity — generate with: fipsctl keygen
# Must be set before running: export FIPS_NSEC=<your-nsec>
FIPS_NSEC=
# Peer configuration (leave empty for standalone operation)
FIPS_PEER_NPUB=npub1qmc3cvfz0yu2hx96nq3gp55zdan2qclealn7xshgr448d3nh6lks7zel98
FIPS_PEER_ADDR=217.77.8.91:2121
FIPS_PEER_ALIAS=vps
FIPS_PEER_TRANSPORT=udp

View File

@@ -0,0 +1,3 @@
fips
fipsctl
fipstop

View File

@@ -2,11 +2,31 @@ FROM debian:trixie-slim
RUN apt-get update && \
apt-get install -y --no-install-recommends \
ca-certificates \
iproute2 iputils-ping dnsutils dnsmasq iptables \
openssh-client openssh-server python3 \
tcpdump netcat-openbsd curl iperf3 && \
rm -rf /var/lib/apt/lists/*
# Install nak (Nostr Army Knife) — detect arch and download the correct binary.
# Asset naming: nak-<version>-linux-<arch>
RUN ARCH=$(dpkg --print-architecture) && \
case "$ARCH" in \
amd64) NAK_ARCH="linux-amd64" ;; \
arm64) NAK_ARCH="linux-arm64" ;; \
armhf) NAK_ARCH="linux-arm" ;; \
*) echo "Unsupported arch: $ARCH" && exit 1 ;; \
esac && \
NAK_VERSION=$(curl -sSL --retry 3 \
"https://api.github.com/repos/fiatjaf/nak/releases/latest" \
| grep '"tag_name"' | head -1 | sed 's/.*"tag_name": *"\(.*\)".*/\1/') && \
echo "Installing nak ${NAK_VERSION} for ${NAK_ARCH}" && \
curl -sSL --retry 3 \
"https://github.com/fiatjaf/nak/releases/download/${NAK_VERSION}/nak-${NAK_VERSION}-${NAK_ARCH}" \
-o /usr/local/bin/nak && \
chmod +x /usr/local/bin/nak && \
nak --version
# Setup SSH server with no authentication (test only!)
RUN mkdir -p /var/run/sshd && \
ssh-keygen -A && \

View File

@@ -0,0 +1,49 @@
# strfry: official multi-arch image (linux/amd64 + linux/arm64)
# replaces the nostr-rs-relay source build which was amd64-only.
#
# IMPORTANT: use alpine (same musl-based libc as the strfry source image) as
# the final stage. Copying the strfry binary into a glibc-based image such as
# debian:bookworm-slim causes "not found" at exec time because the musl dynamic
# linker (/lib/ld-musl-*.so.1) and its shared libraries are absent there.
FROM ghcr.io/hoytech/strfry:latest AS strfry
FROM alpine:3.18
RUN apk add --no-cache nginx
# nginx: reverse proxy port 80 (IPv4 + IPv6) → strfry 127.0.0.1:7777
RUN printf 'server {\n\
listen 80;\n\
listen [::]:80;\n\
location / {\n\
proxy_pass http://127.0.0.1:7777;\n\
proxy_http_version 1.1;\n\
proxy_read_timeout 1d;\n\
proxy_send_timeout 1d;\n\
proxy_set_header Upgrade $http_upgrade;\n\
proxy_set_header Connection "Upgrade";\n\
proxy_set_header Host $host;\n\
}\n\
}\n' > /etc/nginx/http.d/nostr-relay.conf && \
rm -f /etc/nginx/http.d/default.conf
COPY --from=strfry /app/strfry /usr/local/bin/strfry
# Copy musl shared libraries that strfry depends on from the source image.
COPY --from=strfry \
/usr/lib/liblmdb.so.0 \
/usr/lib/libcrypto.so.50 \
/usr/lib/libssl.so.53 \
/usr/lib/libsecp256k1.so.2 \
/usr/lib/libzstd.so.1 \
/usr/lib/libstdc++.so.6 \
/usr/lib/libgcc_s.so.1 \
/usr/lib/
COPY --from=strfry /lib/libz.so.1 /lib/
WORKDIR /usr/src/app
RUN mkdir -p strfry-db
RUN printf '#!/bin/sh\nnginx\nexec strfry relay\n' > /entrypoint-app.sh && \
chmod +x /entrypoint-app.sh
ENTRYPOINT ["/entrypoint-app.sh"]

View File

@@ -0,0 +1,289 @@
# FIPS Nostr Relay Sidecar
Runs a [strfry](https://github.com/hoytech/strfry) Nostr relay reachable
exclusively over the FIPS mesh. The relay container shares the FIPS
sidecar's network namespace and is isolated from the host network by
iptables — it can only be reached via the node's `.fips` name.
## How to Run
### 1. Build the FIPS binaries
From the repo root, compile FIPS for Linux and copy the binaries into the
Docker build context:
```bash
cd examples/sidecar-nostr-relay
./scripts/build.sh
```
Cross-compilation from macOS is supported via `cargo-zigbuild`.
### 2. Set your node identity
The relay needs a unique FIPS identity. Generate one with:
```bash
fipsctl keygen
```
Then set it in `.env`:
```bash
# .env
FIPS_NSEC=nsec1... # paste your nsec here
```
`FIPS_NSEC` is required — the container will refuse to start without it.
### 3. Start the stack
```bash
docker compose up -d
```
This starts two containers that share a network namespace:
- **fips** — FIPS daemon + dnsmasq. Owns the namespace, creates `fips0`.
- **app** — strfry relay + nginx. Joins the namespace via `network_mode: service:fips`.
### 4. Verify
```bash
# FIPS node is up and has a mesh address:
docker exec sidecar-nostr-relay-fips-1 fipsctl show status
# Relay is listening (should show nginx on :80 and strfry on :7777):
docker exec sidecar-nostr-relay-fips-1 ss -tlnp
# Peer link to the public node is established:
docker exec sidecar-nostr-relay-fips-1 fipsctl show peers
# Or check the logs:
docker compose logs -f
```
### 5. Connect to the relay
Your node's npub (and therefore its `.fips` name) is derived from its keypair:
```bash
docker exec sidecar-nostr-relay-fips-1 fipsctl show status
```
Connect from any FIPS-peered client using the node's npub:
```
ws://npub1xxxx.fips:80
```
## Security Model
The sidecar pattern enforces strict network isolation on the app container:
- **No IPv4 access**: iptables blocks all eth0 traffic except FIPS UDP
transport (port 2121) and TCP transport (port 443). The app container
cannot reach the Docker bridge, the host network, or any IPv4 address.
- **No IPv6 on eth0**: ip6tables blocks all IPv6 traffic on eth0. The app
container cannot use link-local or any Docker-assigned IPv6 addresses.
- **FIPS mesh only**: The only routable network path is through `fips0`
(`fd::/8`). All application traffic traverses the FIPS mesh with
end-to-end encryption.
- **Loopback allowed**: `lo` is unrestricted for inter-process communication
within the shared namespace.
This means the app container treats the FIPS mesh as its sole network. Even
if the application is compromised, it cannot bypass the mesh or communicate
with the transport layer directly.
## Architecture
```text
┌───────────────────────────────────────────────────┐
│ Shared network namespace │
│ │
│ ┌───────────────┐ ┌──────────────────────────┐ │
│ │ fips-sidecar │ │ fips-app │ │
│ │ │ │ │ │
│ │ fips daemon │ │ your workload │ │
│ │ fipsctl │ │ │ │
│ │ dnsmasq │ │ │ │
│ └───────────────┘ └──────────────────────────┘ │
│ │
│ Interfaces: │
│ lo — loopback (unrestricted) │
│ eth0 — Docker bridge (iptables: FIPS only) │
│ fips0 — FIPS TUN (fd::/8, unrestricted) │
└───────────────────────────────────────────────────┘
```
The FIPS sidecar owns the network namespace and creates the `fips0` TUN
interface. The app container joins via `network_mode: service:fips` and
sees the same interfaces. The entrypoint script applies iptables rules
before launching the FIPS daemon:
**IPv4 rules** (iptables):
- ACCEPT on `lo` (both directions)
- ACCEPT UDP sport/dport 2121 on `eth0` (FIPS UDP transport)
- ACCEPT TCP dport 443 / sport 443 on `eth0` (FIPS TCP transport)
- DROP everything else on `eth0`
**IPv6 rules** (ip6tables):
- ACCEPT on `lo` (both directions)
- ACCEPT on `fips0` (both directions)
- DROP everything on `eth0`
### DNS Resolution
DNS inside the container is handled by dnsmasq (127.0.0.1:53):
- `.fips` queries are forwarded to the FIPS daemon's built-in DNS resolver
(127.0.0.1:5354), which resolves npub-based names to `fd::/8` addresses
- All other queries are forwarded to Docker's embedded DNS (127.0.0.11)
The `resolv.conf` mount points the container's resolver at 127.0.0.1,
where dnsmasq handles the routing.
## Build
```bash
cd examples/sidecar-nostr-relay
./scripts/build.sh
```
This compiles FIPS for Linux, copies the binaries into the Docker context,
and builds the sidecar and app images. Cross-compilation from macOS is
supported via `cargo-zigbuild`.
## Run with Peers
To connect the sidecar to an existing mesh, provide the peer's npub and
transport address:
```bash
FIPS_PEER_NPUB=npub1... \
FIPS_PEER_ADDR=203.0.113.10:2121 \
FIPS_PEER_ALIAS=gateway \
docker compose up -d
```
Verify the peer link:
```bash
docker exec sidecar-nostr-relay-fips-1 fipsctl show peers
docker exec sidecar-nostr-relay-fips-1 fipsctl show links
```
## Verify Connectivity and Isolation
From the app container:
```bash
# Ping a mesh node by npub (resolves via .fips DNS):
docker exec sidecar-nostr-relay-app-1 ping6 -c3 npub1xxxx.fips
# Fetch a web page from a mesh node over FIPS:
docker exec sidecar-nostr-relay-app-1 curl -6 "http://npub1xxxx.fips:8000/"
# Docker bridge is blocked — this should fail:
docker exec sidecar-nostr-relay-app-1 ping -c1 -W2 172.20.0.13
# Loopback is allowed:
docker exec sidecar-nostr-relay-app-1 ping -c1 127.0.0.1
```
## Environment Variables
| Variable | Default | Description |
|---|---|---|
| `FIPS_NSEC` | *(required)* | Node secret key (hex or nsec1 bech32) |
| `FIPS_PEER_NPUB` | *(empty)* | Peer's npub to connect to |
| `FIPS_PEER_ADDR` | *(empty)* | Peer's transport address (e.g. `203.0.113.10:2121`) |
| `FIPS_PEER_ALIAS` | `peer` | Human-readable peer name |
| `FIPS_UDP_BIND` | `0.0.0.0:2121` | UDP transport bind address |
| `FIPS_TCP_BIND` | `0.0.0.0:8443` | TCP transport bind address |
| `FIPS_PEER_TRANSPORT` | `udp` | Peer transport type (`udp` or `tcp`) |
| `FIPS_TUN_MTU` | `1280` | TUN interface MTU |
| `FIPS_NETWORK` | `fips-sidecar-net` | Docker network name (set to join external network) |
| `FIPS_SUBNET` | `172.20.1.0/24` | Docker network subnet |
| `FIPS_IPV4` | `172.20.1.20` | Sidecar's IPv4 address on the Docker network |
| `RUST_LOG` | `info` | FIPS log level |
## Troubleshooting
**`FIPS_NSEC is required`** — The `FIPS_NSEC` environment variable is not
set. Either add it to `.env` or pass it on the command line. Generate a
random key with: `openssl rand -hex 32`
**`fips0` interface not appearing** — The FIPS daemon needs `/dev/net/tun`
and `NET_ADMIN` capability. Check that the compose file includes both:
```yaml
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
```
**No peer connection established** — Verify the peer address is reachable
from the sidecar container (`docker exec sidecar-nostr-relay-fips-1 ping -c1 <peer-ip>`).
If joining an external Docker network, ensure `FIPS_NETWORK`, `FIPS_SUBNET`,
and `FIPS_IPV4` match the target network. Check logs with
`docker logs sidecar-nostr-relay-fips-1`.
**DNS not resolving `.fips` names** — Verify dnsmasq is running:
`docker exec sidecar-nostr-relay-fips-1 pgrep dnsmasq`. Check that `resolv.conf` is
mounted (should contain `nameserver 127.0.0.1`). Verify the FIPS DNS
resolver is listening: `docker exec sidecar-nostr-relay-fips-1 dig @127.0.0.1 -p 5354 <npub>.fips AAAA`.
**iptables errors in entrypoint** — The sidecar container requires
`NET_ADMIN` capability for iptables. Without it, the isolation rules
cannot be applied and the entrypoint will fail.
## Production Considerations
**Secrets management**: The default `.env` contains a hardcoded nsec for
development. In production, use Docker secrets, a vault, or inject the key
via a secure CI/CD pipeline. Never commit production keys to version control.
**Logging**: Set `RUST_LOG` to control log verbosity (`debug`, `info`,
`warn`, `error`). For production, configure the Docker logging driver with
size limits:
```yaml
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
```
**Resource limits**: Add memory and CPU constraints in the compose file:
```yaml
deploy:
resources:
limits:
memory: 256M
cpus: "0.5"
```
**Multiple peers**: The entrypoint supports a single peer via environment
variables. For multiple peers, mount a custom `fips.yaml` directly:
```yaml
volumes:
- ./my-fips.yaml:/etc/fips/fips.yaml:ro
```
**Health checks**: Add a Docker health check using `fipsctl`:
```yaml
healthcheck:
test: ["CMD", "fipsctl", "show", "status"]
interval: 30s
timeout: 5s
retries: 3
```

View File

@@ -0,0 +1,6 @@
# Override: use a pre-existing external network instead of creating one.
# Used by test-sidecar.sh for multi-instance testing on a shared network.
networks:
fips-net:
external: true
name: ${FIPS_NETWORK:-fips-sidecar-net}

View File

@@ -0,0 +1,54 @@
networks:
fips-net:
name: ${FIPS_NETWORK:-fips-sidecar-net}
driver: bridge
ipam:
config:
- subnet: ${FIPS_SUBNET:-172.20.1.0/24}
services:
fips:
build:
context: .
hostname: fips-sidecar
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
restart: "no"
ports:
- "2121:2121/udp" # FIPS UDP transport
- "8443:8443/tcp" # FIPS TCP transport
- "80:80/tcp" # Nostr relay WebSocket (via nginx)
environment:
- RUST_LOG=${RUST_LOG:-info}
- FIPS_NSEC=${FIPS_NSEC}
- FIPS_PEER_NPUB=${FIPS_PEER_NPUB:-}
- FIPS_PEER_ADDR=${FIPS_PEER_ADDR:-}
- FIPS_PEER_ALIAS=${FIPS_PEER_ALIAS:-peer}
- FIPS_UDP_BIND=${FIPS_UDP_BIND:-0.0.0.0:2121}
- FIPS_TUN_MTU=${FIPS_TUN_MTU:-1280}
- FIPS_PEER_TRANSPORT=${FIPS_PEER_TRANSPORT:-udp}
volumes:
- ./resolv.conf:/etc/resolv.conf:ro
networks:
fips-net:
ipv4_address: ${FIPS_IPV4:-172.20.1.20}
app:
build:
context: .
dockerfile: Dockerfile.app
network_mode: "service:fips"
restart: unless-stopped
depends_on:
- fips
volumes:
- relay-data:/usr/src/app/strfry-db
- ./relay/strfry.conf:/usr/src/app/strfry.conf:ro
- ./resolv.conf:/etc/resolv.conf:ro
volumes:
relay-data:

View File

@@ -6,6 +6,7 @@ set -e
FIPS_NSEC="${FIPS_NSEC:?FIPS_NSEC is required}"
FIPS_UDP_BIND="${FIPS_UDP_BIND:-0.0.0.0:2121}"
FIPS_TCP_BIND="${FIPS_TCP_BIND:-0.0.0.0:8443}"
FIPS_TUN_MTU="${FIPS_TUN_MTU:-1280}"
FIPS_PEER_TRANSPORT="${FIPS_PEER_TRANSPORT:-udp}"
@@ -41,7 +42,8 @@ transports:
udp:
bind_addr: "${FIPS_UDP_BIND}"
mtu: 1472
tcp: {}
tcp:
bind_addr: "${FIPS_TCP_BIND}"
peers:
${PEERS_SECTION:- []}

View File

@@ -0,0 +1,49 @@
# nostr-rs-relay configuration for FIPS mesh deployment.
# Full reference: https://github.com/scsibug/nostr-rs-relay
[info]
# Shown in NIP-11 relay information document.
relay_url = "" # Set to ws://[<your-fips-ipv6>]:8080 after first start
name = "FIPS Nostr Relay"
description = "A Nostr relay accessible over the FIPS mesh network."
pubkey = "" # Optional: your npub (hex) as relay operator
contact = "" # Optional: contact address
[database]
# SQLite database path inside the container (mapped to relay-data volume).
data_directory = "/usr/src/app/db"
[network]
# Bind on all interfaces (IPv4 + IPv6) — FIPS delivers traffic via the fips0
# TUN as IPv6 (fd00::/8). Using 0.0.0.0 with port 8080; the relay also needs
# to accept IPv6 connections so we set port separately and rely on the OS
# dual-stack socket (IPV6_V6ONLY=0).
# nostr-rs-relay address field is just the IP, port is separate.
address = "0.0.0.0"
port = 8080
ping_interval_seconds = 300
[limits]
# Adjust to taste. These are conservative defaults for a personal relay.
messages_per_sec = 10
subscriptions_per_min = 20
max_blocking_threads = 4
max_event_bytes = 131072 # 128 KiB per event
max_ws_message_bytes = 131072
max_ws_frame_bytes = 131072
broadcast_buffer = 16384
event_persist_buffer = 4096
[authorization]
# Set to true to require NIP-42 AUTH before accepting events.
# Useful for a private relay — only authenticated users can publish.
nip42_auth = false
nip42_dms = false
[verified_users]
# NIP-05 verification (optional).
mode = "disabled"
[pay_to_relay]
# Lightning payments for relay access (optional).
enabled = false

View File

@@ -0,0 +1,71 @@
##
## strfry configuration for FIPS mesh deployment.
## Full reference: https://github.com/hoytech/strfry
##
db = "/usr/src/app/strfry-db/"
dbParams {
# Maximum size of the database (bytes). 10 GiB is a safe default.
mapsize = 10737418240
}
relay {
# Bind on all interfaces so the FIPS TUN (IPv4 + IPv6) can reach it.
bind = "0.0.0.0"
port = 7777
nofiles = 0
info {
name = "FIPS Nostr Relay"
description = "A Nostr relay accessible over the FIPS mesh network."
pubkey = ""
contact = ""
}
# Maximum size of an inbound WebSocket message (bytes).
maxWebsocketPayloadSize = 131072
# Send a ping every N seconds to keep connections alive.
autoPingSeconds = 55
# Enable per-message compression.
enableTcpNoDelay = false
rejectFutureEventsSeconds = 900
rejectEphemeralEventsOlderThanSeconds = 60
rejectEventsNewerThanSeconds = 900
maxFilterLimit = 500
maxSubsPerConnection = 20
writePolicy {
# Plugin executable for write-policy decisions (leave empty to allow all).
plugin = ""
}
compression {
enabled = true
slidingWindow = true
}
logging {
dumpInAll = false
dumpInEvents = false
dumpInReqs = false
dbScanPerf = false
}
numThreads {
ingester = 3
reqWorker = 3
reqMonitor = 3
negentropy = 2
}
negentropy {
enabled = true
maxSyncEvents = 1000000
}
}

View File

@@ -10,7 +10,7 @@ DOCKER_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
PROJECT_ROOT="$(cd "$DOCKER_DIR/../.." && pwd)"
if [ ! -f "$PROJECT_ROOT/Cargo.toml" ]; then
echo "Error: Cannot find Cargo.toml at $PROJECT_ROOT" >&2
echo "Expected layout: <project-root>/testing/sidecar/scripts/build.sh" >&2
echo "Expected layout: <project-root>/examples/sidecar-nostr-relay/scripts/build.sh" >&2
exit 1
fi
@@ -54,4 +54,4 @@ echo ""
echo "Building Docker image..."
docker compose -f "$DOCKER_DIR/docker-compose.yml" build
echo ""
echo "Ready: cd testing/sidecar && docker compose up -d"
echo "Ready: cd examples/sidecar-nostr-relay && docker compose up -d"

31
packaging/Makefile Normal file
View File

@@ -0,0 +1,31 @@
# FIPS Packaging Makefile
#
# Builds release packages for supported target platforms.
# All outputs are placed in deploy/ at the project root.
#
# Usage:
# make deb Build a Debian/Ubuntu .deb package
# make tarball Build a systemd install tarball
# make ipk Build an OpenWrt .ipk package
# make all Build deb and tarball (default)
# make clean Remove deploy/ directory
SHELL := /bin/bash
PACKAGING_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
PROJECT_ROOT := $(abspath $(PACKAGING_DIR)/..)
.PHONY: all deb tarball ipk clean
all: deb tarball
deb:
@bash $(PACKAGING_DIR)/debian/build-deb.sh
tarball:
@bash $(PACKAGING_DIR)/systemd/build-tarball.sh
ipk:
@bash $(PACKAGING_DIR)/openwrt/build-ipk.sh
clean:
rm -rf $(PROJECT_ROOT)/deploy

86
packaging/README.md Normal file
View File

@@ -0,0 +1,86 @@
# FIPS Packaging
This directory contains packaging for all supported target platforms.
All build outputs go to `deploy/` at the project root.
## Quick Start
```sh
make deb # Debian/Ubuntu .deb
make tarball # systemd install tarball
make ipk # OpenWrt .ipk
make all # deb + tarball (default)
```
## Directory Structure
```text
packaging/
common/ Shared assets (default config, hosts file)
debian/ Debian/Ubuntu .deb packaging via cargo-deb
systemd/ Generic Linux systemd tarball packaging
openwrt/ OpenWrt .ipk packaging via cargo-zigbuild
```
## Formats
### Debian/Ubuntu (`.deb`)
Built with [cargo-deb](https://github.com/kornelski/cargo-deb). Installs
`fips`, `fipsctl`, and `fipstop` to `/usr/bin/`, places config at
`/etc/fips/fips.yaml` (preserved on upgrade), and enables the systemd
service.
```sh
# Build
make deb
# Install
sudo dpkg -i deploy/fips_<version>_<arch>.deb
# Remove (preserves config and keys)
sudo dpkg -r fips
# Purge (removes config and identity keys)
sudo dpkg -P fips
```
### systemd Tarball
A self-contained tarball with binaries and an `install.sh` script for
any systemd-based Linux distribution.
```sh
# Build
make tarball
# Install (on target host)
tar -xzf deploy/fips-<version>-linux-<arch>.tar.gz
sudo ./fips-<version>-linux-<arch>/install.sh
```
See [systemd/README.install.md](systemd/README.install.md) for full
installation and configuration instructions.
### OpenWrt (`.ipk`)
Cross-compiled with cargo-zigbuild and assembled as a standard `.ipk`
archive. Supports aarch64, mipsel, mips, arm, and x86\_64 targets.
```sh
# Build (default: aarch64)
make ipk
# Build for a specific architecture
bash packaging/openwrt/build-ipk.sh --arch mipsel
```
See [openwrt/README.md](openwrt/README.md) for router-specific
installation instructions.
## Shared Assets
`common/` contains assets used across packaging formats:
- `fips.yaml` — default configuration (ephemeral identity, UDP/TCP/TUN/DNS)
- `hosts` — static hostname-to-npub mappings for `.fips` DNS resolution

View File

@@ -43,5 +43,5 @@ peers: []
# alias: "gateway"
# addresses:
# - transport: udp
# addr: "217.77.8.91:2121" # public FIPS testing node
# addr: "217.77.8.91:2121" # IP or hostname (e.g., "peer.example.com:2121")
# connect_policy: auto_connect

View File

@@ -19,6 +19,11 @@ if ! command -v cargo-deb &>/dev/null; then
exit 1
fi
# Derive SOURCE_DATE_EPOCH from git if not already set (reproducible builds)
if [ -z "${SOURCE_DATE_EPOCH:-}" ]; then
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)
fi
# Build the .deb package
echo "Building .deb package..."
cargo deb

View File

@@ -10,11 +10,10 @@ Restart=on-failure
RestartSec=5
# Logging: RUST_LOG controls verbosity.
# Use "info" for production, "debug" for troubleshooting.
Environment=RUST_LOG=info
Environment=RUST_LOG=debug
# Control socket directory (/run/fips/).
# Group-readable so 'fips' group members can use fipsctl/fipstop.
# Group-accessible so 'fips' group members can use fipsctl/fipstop.
RuntimeDirectory=fips
RuntimeDirectoryMode=0750

View File

@@ -244,7 +244,11 @@ fi
ipk_tar() {
# ipk_tar <output.tar.gz> <source-dir> [paths...]
local out="$1" src="$2"; shift 2
COPYFILE_DISABLE=1 "$TAR_CMD" $TAR_EXTRA_FLAGS -czf "$out" -C "$src" "$@"
local mtime_flags=""
if [ -n "${SOURCE_DATE_EPOCH:-}" ]; then
mtime_flags="--mtime=@$SOURCE_DATE_EPOCH"
fi
COPYFILE_DISABLE=1 "$TAR_CMD" $TAR_EXTRA_FLAGS $mtime_flags -czf "$out" -C "$src" "$@"
}
ipk_tar "$IPK_WORK/control.tar.gz" "$CONTROL_DIR" .

View File

@@ -61,4 +61,4 @@ peers: []
# alias: "gateway"
# addresses:
# - transport: udp
# addr: "1.2.3.4:2121"
# addr: "1.2.3.4:2121" # IP or hostname

View File

@@ -72,7 +72,7 @@ peers:
alias: "gateway"
addresses:
- transport: udp
addr: "217.77.8.91:2121" # public FIPS testing node
addr: "217.77.8.91:2121" # IP or hostname (e.g., "peer.example.com:2121")
connect_policy: auto_connect
```

View File

@@ -45,9 +45,20 @@ cp "${SCRIPT_DIR}/README.install.md" "${STAGING_DIR}/"
chmod +x "${STAGING_DIR}/install.sh" "${STAGING_DIR}/uninstall.sh"
# Create tarball
# Create tarball (reproducible: normalize timestamps, ownership, and sort order)
cd "${DEPLOY_DIR}"
tar czf "${TARBALL_NAME}.tar.gz" "${TARBALL_NAME}/"
# Default SOURCE_DATE_EPOCH to git commit timestamp if not set
if [ -z "${SOURCE_DATE_EPOCH:-}" ]; then
SOURCE_DATE_EPOCH=$(git -C "${PROJECT_ROOT}" log -1 --format=%ct)
export SOURCE_DATE_EPOCH
fi
TAR_REPRO_FLAGS="--mtime=@${SOURCE_DATE_EPOCH} --sort=name"
if tar --version 2>/dev/null | grep -q 'GNU tar'; then
TAR_REPRO_FLAGS="${TAR_REPRO_FLAGS} --numeric-owner --owner=0 --group=0"
fi
COPYFILE_DISABLE=1 tar ${TAR_REPRO_FLAGS} -czf "${TARBALL_NAME}.tar.gz" "${TARBALL_NAME}/"
rm -rf "${STAGING_DIR}"
echo ""

View File

@@ -10,11 +10,10 @@ Restart=on-failure
RestartSec=5
# Logging: RUST_LOG controls verbosity.
# Use "info" for production, "debug" for troubleshooting.
Environment=RUST_LOG=info
Environment=RUST_LOG=debug
# Control socket directory (/run/fips/).
# Group-readable so 'fips' group members can use fipsctl/fipstop.
# Group-accessible so 'fips' group members can use fipsctl/fipstop.
RuntimeDirectory=fips
RuntimeDirectoryMode=0750

83
packaging/torrc.fips Normal file
View File

@@ -0,0 +1,83 @@
### FIPS-specific Tor configuration ###
###
### Reference torrc for running Tor alongside the FIPS daemon.
### Uses HiddenServiceDir (directory mode) for the onion service,
### which enables Sandbox mode — the strongest single hardening option.
###
### Install: cp torrc.fips /etc/tor/torrc
### Verify: tor --verify-config -f /etc/tor/torrc
###
### The corresponding fips.yaml transport section should be:
###
### transports:
### tor:
### mode: "directory"
### directory_service:
### hostname_file: "/var/lib/tor/fips_onion_service/hostname"
### bind_addr: "127.0.0.1:8443"
## Identity
DataDirectory /var/lib/tor
User debian-tor
## SOCKS proxy (for outbound peer connections)
## IsolateSOCKSAuth: username/password fields serve as circuit isolation
## keys — each FIPS peer destination gets its own Tor circuit.
## IsolateDestAddr + IsolateDestPort: additional isolation by destination.
## IsolateSOCKSAuth: required — FIPS uses SOCKS5 username/password fields
## as per-destination circuit isolation keys (not real authentication).
##
## SafeSocks 1 rejects SOCKS5 CONNECT to raw IP addresses. FIPS supports
## DNS hostnames in peer addresses — when clearnet peers use hostnames,
## they are passed through SOCKS5 for Tor-side resolution (no local DNS
## leak). Enable SafeSocks 1 when all clearnet peers use hostnames or
## when running .onion-only. Not enabled by default for compatibility
## with IP-addressed peers.
SocksPort 127.0.0.1:9050 IsolateSOCKSAuth
#TestSocks 1 # Enable during development only
## Control access (Unix socket, not TCP — filesystem permission control)
## Only needed if fips uses control_port mode. In directory mode, the
## control socket is not required but can be useful for monitoring.
ControlSocket /run/tor/control GroupWritable RelaxDirModeCheck
ControlSocketsGroupWritable 1
## Do NOT use ControlPort 9051 — TCP control ports are accessible to any
## local process that authenticates. Use Unix socket only.
## Authentication
CookieAuthentication 1
CookieAuthFileGroupReadable 1
CookieAuthFile /run/tor/control.authcookie
## Onion service (persistent, filesystem-managed by Tor)
## Tor manages the key in HiddenServiceDir. FIPS reads the .onion
## address from the hostname file at startup.
HiddenServiceDir /var/lib/tor/fips_onion_service
HiddenServicePort 8443 127.0.0.1:8443
## Onion service hardening
HiddenServiceMaxStreams 100
HiddenServiceMaxStreamsCloseCircuit 1
## Onion service DoS protection (Tor 0.4.8+)
## Intro point rate limiting — prevents introduction cell floods.
HiddenServiceEnableIntroDoSDefense 1
HiddenServiceEnableIntroDoSRatePerSec 25
HiddenServiceEnableIntroDoSBurstPerSec 200
## Proof-of-work defense — auto-scaling computational puzzle for clients.
## Minimal cost to legitimate peers, expensive for flood attacks.
HiddenServicePoWDefensesEnabled 1
HiddenServicePoWQueueRate 250
HiddenServicePoWQueueBurst 2500
## Security
ExitRelay 0
ExitPolicy reject *:*
Sandbox 1
VanguardsLiteEnabled 1
ConnectionPadding 1
SafeLogging 1
## Logging
Log notice syslog

2
rust-toolchain.toml Normal file
View File

@@ -0,0 +1,2 @@
[toolchain]
channel = "1.94.0"

View File

@@ -1,10 +1,11 @@
//! fipsctl — FIPS control client
//!
//! Connects to the FIPS daemon's Unix domain control socket, sends a
//! query command, and pretty-prints the JSON response.
//! Connects to the FIPS daemon's Unix domain control socket, sends
//! commands, and pretty-prints the JSON response.
use clap::{Parser, Subcommand};
use fips::config::{write_key_file, write_pub_file};
use fips::upper::hosts::HostMap;
use fips::version;
use fips::{encode_nsec, Identity};
use std::io::{BufRead, BufReader, Write};
@@ -18,7 +19,7 @@ use std::time::Duration;
name = "fipsctl",
version = version::short_version(),
long_version = version::long_version(),
about = "Query a running FIPS daemon"
about = "Control a running FIPS daemon"
)]
struct Cli {
/// Control socket path override
@@ -48,6 +49,20 @@ enum Commands {
#[arg(short = 's', long = "stdout")]
stdout: bool,
},
/// Connect to a peer
Connect {
/// Peer identifier: npub (bech32) or hostname from /etc/fips/hosts
peer: String,
/// Transport address (e.g., "192.168.1.1:2121")
address: String,
/// Transport type: udp, tcp, tor, ethernet
transport: String,
},
/// Disconnect a peer
Disconnect {
/// Peer identifier: npub (bech32) or hostname from /etc/fips/hosts
peer: String,
},
}
#[derive(Subcommand, Debug)]
@@ -108,123 +123,58 @@ fn default_socket_path() -> PathBuf {
}
}
fn main() {
let cli = Cli::parse();
// Commands that don't require a running daemon
match &cli.command {
Commands::Keygen {
dir,
force,
stdout,
} => {
let identity = Identity::generate();
let nsec = encode_nsec(&identity.keypair().secret_key());
let npub = identity.npub();
if *stdout {
println!("{}", nsec);
println!("{}", npub);
return;
}
let key_path = dir.join("fips.key");
let pub_path = dir.join("fips.pub");
if key_path.exists() && !force {
eprintln!("error: key file already exists: {}", key_path.display());
eprintln!("Use --force to overwrite.");
std::process::exit(1);
}
if let Err(e) = std::fs::create_dir_all(dir) {
eprintln!("error: cannot create directory {}: {}", dir.display(), e);
std::process::exit(1);
}
if let Err(e) = write_key_file(&key_path, &nsec) {
eprintln!("error: failed to write key file: {}", e);
std::process::exit(1);
}
if let Err(e) = write_pub_file(&pub_path, &npub) {
eprintln!("error: failed to write pub file: {}", e);
std::process::exit(1);
}
eprintln!("{}", npub);
eprintln!("Key files written to: {}/", dir.display());
eprintln!();
eprintln!("NOTE: Set 'node.identity.persistent: true' in fips.yaml");
eprintln!(" or these keys will be overwritten on next daemon start.");
return;
}
Commands::Show { .. } => {}
}
let socket_path = cli.socket.unwrap_or_else(default_socket_path);
let command_name = match &cli.command {
Commands::Show { what } => what.command_name(),
Commands::Keygen { .. } => unreachable!(),
};
// Connect to the control socket
let mut stream = match UnixStream::connect(&socket_path) {
Ok(s) => s,
Err(e) => {
eprintln!(
"error: cannot connect to {}: {}",
/// Send a JSON request to the control socket and return the response.
fn send_request(socket_path: &Path, request_json: &str) -> Result<serde_json::Value, String> {
let mut stream = UnixStream::connect(socket_path).map_err(|e| {
if e.kind() == std::io::ErrorKind::PermissionDenied {
format!(
"cannot connect to {}: {}\n\
Hint: add your user to the 'fips' group: sudo usermod -aG fips $USER\n\
Then log out and back in for the change to take effect.",
socket_path.display(),
e
);
if e.kind() == std::io::ErrorKind::PermissionDenied {
eprintln!(
"Hint: add your user to the 'fips' group: sudo usermod -aG fips $USER"
);
eprintln!("Then log out and back in for the change to take effect.");
} else {
eprintln!("Is the FIPS daemon running?");
}
std::process::exit(1);
)
} else {
format!(
"cannot connect to {}: {}\nIs the FIPS daemon running?",
socket_path.display(),
e
)
}
};
})?;
// Set timeouts
let timeout = Duration::from_secs(2);
let timeout = Duration::from_secs(5);
let _ = stream.set_read_timeout(Some(timeout));
let _ = stream.set_write_timeout(Some(timeout));
// Send request
let request = format!("{{\"command\":\"{}\"}}\n", command_name);
if let Err(e) = stream.write_all(request.as_bytes()) {
eprintln!("error: failed to send request: {}", e);
std::process::exit(1);
}
// Shutdown write half to signal end of request
stream
.write_all(request_json.as_bytes())
.map_err(|e| format!("failed to send request: {e}"))?;
let _ = stream.shutdown(std::net::Shutdown::Write);
// Read response
let reader = BufReader::new(&stream);
let response_line = match reader.lines().next() {
Some(Ok(line)) => line,
Some(Err(e)) => {
eprintln!("error: failed to read response: {}", e);
std::process::exit(1);
}
None => {
eprintln!("error: no response from daemon");
std::process::exit(1);
}
};
let line = reader
.lines()
.next()
.ok_or("no response from daemon")?
.map_err(|e| format!("failed to read response: {e}"))?;
// Parse and pretty-print
let Ok(value) = serde_json::from_str::<serde_json::Value>(&response_line) else {
// Not JSON, print raw
println!("{}", response_line);
return;
};
serde_json::from_str(&line).map_err(|e| format!("invalid response JSON: {e}"))
}
/// Build a request JSON string for a simple command (no params).
fn build_query(command: &str) -> String {
format!("{{\"command\":\"{command}\"}}\n")
}
/// Build a request JSON string for a command with params.
fn build_command(command: &str, params: serde_json::Value) -> String {
let req = serde_json::json!({"command": command, "params": params});
format!("{}\n", serde_json::to_string(&req).unwrap())
}
/// Print a control socket response, handling error status.
fn print_response(value: &serde_json::Value) {
let status = value
.get("status")
.and_then(|v| v.as_str())
@@ -235,15 +185,121 @@ fn main() {
.get("message")
.and_then(|v| v.as_str())
.unwrap_or("unknown error");
eprintln!("error: {}", msg);
eprintln!("error: {msg}");
std::process::exit(1);
}
// Pretty-print the data field (or whole response if no data field)
let output = if let Some(data) = value.get("data") {
serde_json::to_string_pretty(data).unwrap_or(response_line)
serde_json::to_string_pretty(data)
} else {
serde_json::to_string_pretty(&value).unwrap_or(response_line)
serde_json::to_string_pretty(value)
};
println!("{}", output);
println!("{}", output.unwrap_or_else(|_| format!("{value}")));
}
/// Resolve a peer identifier to an npub.
///
/// If the identifier starts with "npub1", it's returned as-is.
/// Otherwise, it's looked up as a hostname in /etc/fips/hosts.
fn resolve_peer(peer: &str) -> String {
if peer.starts_with("npub1") {
return peer.to_string();
}
let hosts = HostMap::load_hosts_file(Path::new(fips::upper::hosts::DEFAULT_HOSTS_PATH));
match hosts.lookup_npub(peer) {
Some(npub) => npub.to_string(),
None => {
eprintln!("error: unknown host '{peer}'");
eprintln!("Not found in /etc/fips/hosts and not an npub.");
std::process::exit(1);
}
}
}
fn main() {
let cli = Cli::parse();
// Commands that don't require a running daemon
if let Commands::Keygen {
dir,
force,
stdout,
} = &cli.command
{
let identity = Identity::generate();
let nsec = encode_nsec(&identity.keypair().secret_key());
let npub = identity.npub();
if *stdout {
println!("{nsec}");
println!("{npub}");
return;
}
let key_path = dir.join("fips.key");
let pub_path = dir.join("fips.pub");
if key_path.exists() && !force {
eprintln!("error: key file already exists: {}", key_path.display());
eprintln!("Use --force to overwrite.");
std::process::exit(1);
}
if let Err(e) = std::fs::create_dir_all(dir) {
eprintln!("error: cannot create directory {}: {e}", dir.display());
std::process::exit(1);
}
if let Err(e) = write_key_file(&key_path, &nsec) {
eprintln!("error: failed to write key file: {e}");
std::process::exit(1);
}
if let Err(e) = write_pub_file(&pub_path, &npub) {
eprintln!("error: failed to write pub file: {e}");
std::process::exit(1);
}
eprintln!("{npub}");
eprintln!("Key files written to: {}/", dir.display());
eprintln!();
eprintln!("NOTE: Set 'node.identity.persistent: true' in fips.yaml");
eprintln!(" or these keys will be overwritten on next daemon start.");
return;
}
let socket_path = cli.socket.unwrap_or_else(default_socket_path);
let request = match &cli.command {
Commands::Show { what } => build_query(what.command_name()),
Commands::Connect {
peer,
address,
transport,
} => {
let npub = resolve_peer(peer);
build_command(
"connect",
serde_json::json!({
"npub": npub,
"address": address,
"transport": transport,
}),
)
}
Commands::Disconnect { peer } => {
let npub = resolve_peer(peer);
build_command("disconnect", serde_json::json!({"npub": npub}))
}
Commands::Keygen { .. } => unreachable!(),
};
match send_request(&socket_path, &request) {
Ok(value) => print_response(&value),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
}

View File

@@ -104,7 +104,9 @@ fn draw_routing_stats(frame: &mut Frame, data: &serde_json::Value, area: Rect) {
helpers::kv_line("Deduplicated", &helpers::nested_u64(data, "discovery", "req_deduplicated")),
helpers::kv_line("Target Is Us", &helpers::nested_u64(data, "discovery", "req_target_is_us")),
helpers::kv_line("Duplicate", &helpers::nested_u64(data, "discovery", "req_duplicate")),
helpers::kv_line("Already Visited", &helpers::nested_u64(data, "discovery", "req_already_visited")),
helpers::kv_line("Bloom Miss", &helpers::nested_u64(data, "discovery", "req_bloom_miss")),
helpers::kv_line("Backoff Suppressed", &helpers::nested_u64(data, "discovery", "req_backoff_suppressed")),
helpers::kv_line("Fwd Rate Limited", &helpers::nested_u64(data, "discovery", "req_forward_rate_limited")),
helpers::kv_line("TTL Exhausted", &helpers::nested_u64(data, "discovery", "req_ttl_exhausted")),
helpers::kv_line("Decode Error", &helpers::nested_u64(data, "discovery", "req_decode_error")),
Line::from(""),

View File

@@ -160,6 +160,24 @@ fn draw_table(
let addr = t.get("local_addr").and_then(|v| v.as_str()).unwrap_or("");
let label = if !name.is_empty() {
format!("{indicator}{typ} {name}")
} else if typ == "tor" {
let mode = t
.get("tor_mode")
.and_then(|v| v.as_str())
.unwrap_or("socks5");
let onion_hint = t
.get("onion_address")
.and_then(|v| v.as_str())
.map(|a| {
let short = if a.len() > 16 {
&a[..16]
} else {
a
};
format!(" {short}..")
})
.unwrap_or_default();
format!("{indicator}tor({mode}){onion_hint}")
} else if !addr.is_empty() {
format!("{indicator}{typ} {addr}")
} else {
@@ -328,6 +346,14 @@ fn draw_transport_detail(frame: &mut Frame, app: &App, area: Rect, t: &serde_jso
lines.push(helpers::kv_line("Local Addr", addr));
}
// Tor-specific info
if let Some(mode) = t.get("tor_mode").and_then(|v| v.as_str()) {
lines.push(helpers::kv_line("Tor Mode", mode));
}
if let Some(onion) = t.get("onion_address").and_then(|v| v.as_str()) {
lines.push(helpers::kv_line("Onion Address", onion));
}
// Transport stats
if let Some(stats) = t.get("stats") {
let typ = helpers::str_field(t, "type");
@@ -424,6 +450,42 @@ fn draw_transport_detail(frame: &mut Frame, app: &App, area: Rect, t: &serde_jso
&helpers::nested_u64(t, "stats", "connect_refused"),
));
}
"tor" => {
lines.push(helpers::kv_line(
"MTU Exceeded",
&helpers::nested_u64(t, "stats", "mtu_exceeded"),
));
lines.push(helpers::kv_line(
"SOCKS5 Errors",
&helpers::nested_u64(t, "stats", "socks5_errors"),
));
lines.push(helpers::kv_line(
"Control Errors",
&helpers::nested_u64(t, "stats", "control_errors"),
));
lines.push(Line::from(""));
lines.push(helpers::section_header("Connections"));
lines.push(helpers::kv_line(
"Established",
&helpers::nested_u64(t, "stats", "connections_established"),
));
lines.push(helpers::kv_line(
"Accepted",
&helpers::nested_u64(t, "stats", "connections_accepted"),
));
lines.push(helpers::kv_line(
"Rejected",
&helpers::nested_u64(t, "stats", "connections_rejected"),
));
lines.push(helpers::kv_line(
"Timeouts",
&helpers::nested_u64(t, "stats", "connect_timeouts"),
));
lines.push(helpers::kv_line(
"Refused",
&helpers::nested_u64(t, "stats", "connect_refused"),
));
}
"ethernet" => {
lines.push(Line::from(""));
lines.push(helpers::section_header("Beacons"));
@@ -448,6 +510,54 @@ fn draw_transport_detail(frame: &mut Frame, app: &App, area: Rect, t: &serde_jso
}
_ => {}
}
// Tor daemon monitoring (when control port data is available)
if let Some(mon) = t.get("tor_monitoring") {
lines.push(Line::from(""));
lines.push(helpers::section_header("Tor Daemon"));
lines.push(helpers::kv_line(
"Bootstrap",
&format!("{}%", helpers::nested_u64(t, "tor_monitoring", "bootstrap")),
));
lines.push(helpers::kv_line(
"Circuit",
if mon
.get("circuit_established")
.and_then(|v| v.as_bool())
.unwrap_or(false)
{
"established"
} else {
"none"
},
));
lines.push(helpers::kv_line(
"Version",
&helpers::nested_str(t, "tor_monitoring", "version"),
));
lines.push(helpers::kv_line(
"Network",
&helpers::nested_str(t, "tor_monitoring", "network_liveness"),
));
lines.push(helpers::kv_line("Dormant", helpers::bool_field(mon, "dormant")));
let tor_read = mon
.get("traffic_read")
.and_then(|v| v.as_u64())
.unwrap_or(0);
let tor_written = mon
.get("traffic_written")
.and_then(|v| v.as_u64())
.unwrap_or(0);
lines.push(helpers::kv_line(
"Tor Read",
&helpers::format_bytes(tor_read),
));
lines.push(helpers::kv_line(
"Tor Written",
&helpers::format_bytes(tor_written),
));
}
}
let detail_scroll = app.detail_view.as_ref().map(|d| d.scroll).unwrap_or(0);

View File

@@ -34,7 +34,7 @@ pub use node::{
TreeConfig,
};
pub use peer::{ConnectPolicy, PeerAddress, PeerConfig};
pub use transport::{EthernetConfig, TcpConfig, TransportInstances, TransportsConfig, UdpConfig};
pub use transport::{DirectoryServiceConfig, EthernetConfig, TcpConfig, TorConfig, TransportInstances, TransportsConfig, UdpConfig};
/// Default config filename.
const CONFIG_FILENAME: &str = "fips.yaml";

View File

@@ -166,6 +166,27 @@ pub struct DiscoveryConfig {
/// Dedup cache expiry in seconds (`node.discovery.recent_expiry_secs`).
#[serde(default = "DiscoveryConfig::default_recent_expiry_secs")]
pub recent_expiry_secs: u64,
/// Base backoff after first lookup failure in seconds (`node.discovery.backoff_base_secs`).
/// Doubles per consecutive failure up to `backoff_max_secs`.
#[serde(default = "DiscoveryConfig::default_backoff_base_secs")]
pub backoff_base_secs: u64,
/// Maximum backoff cap in seconds (`node.discovery.backoff_max_secs`).
#[serde(default = "DiscoveryConfig::default_backoff_max_secs")]
pub backoff_max_secs: u64,
/// Minimum interval between forwarded lookups for the same target in seconds
/// (`node.discovery.forward_min_interval_secs`).
/// Defense-in-depth against misbehaving nodes.
#[serde(default = "DiscoveryConfig::default_forward_min_interval_secs")]
pub forward_min_interval_secs: u64,
/// Retry interval within the timeout window in seconds
/// (`node.discovery.retry_interval_secs`).
/// After this interval without a response, resend the lookup.
#[serde(default = "DiscoveryConfig::default_retry_interval_secs")]
pub retry_interval_secs: u64,
/// Maximum attempts per lookup (`node.discovery.max_attempts`).
/// 1 = no retry, 2 = one retry, etc.
#[serde(default = "DiscoveryConfig::default_max_attempts")]
pub max_attempts: u8,
}
impl Default for DiscoveryConfig {
@@ -174,6 +195,11 @@ impl Default for DiscoveryConfig {
ttl: 64,
timeout_secs: 10,
recent_expiry_secs: 10,
backoff_base_secs: 30,
backoff_max_secs: 300,
forward_min_interval_secs: 2,
retry_interval_secs: 5,
max_attempts: 2,
}
}
}
@@ -182,6 +208,11 @@ impl DiscoveryConfig {
fn default_ttl() -> u8 { 64 }
fn default_timeout_secs() -> u64 { 10 }
fn default_recent_expiry_secs() -> u64 { 10 }
fn default_backoff_base_secs() -> u64 { 30 }
fn default_backoff_max_secs() -> u64 { 300 }
fn default_forward_min_interval_secs() -> u64 { 2 }
fn default_retry_interval_secs() -> u64 { 5 }
fn default_max_attempts() -> u8 { 2 }
}
/// Spanning tree (`node.tree.*`).

View File

@@ -35,9 +35,9 @@ pub struct PeerAddress {
/// Transport-specific address string.
///
/// Format depends on transport type:
/// - UDP: "host:port" (e.g., "192.168.1.1:2121")
/// - Tor: "onion_address:port" (e.g., "xyz...abc.onion:2121")
/// - Ethernet: "interface/mac" (future)
/// - UDP/TCP: "host:port" — IP address or DNS hostname
/// (e.g., "192.168.1.1:2121" or "peer1.example.com:2121")
/// - Ethernet: "interface/mac" (e.g., "eth0/aa:bb:cc:dd:ee:ff")
pub addr: String,
/// Priority for address selection (lower = preferred).

View File

@@ -293,10 +293,6 @@ pub struct TcpConfig {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub send_buf_size: Option<usize>,
/// SOCKS5 proxy for outbound connections (placeholder; not yet implemented).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub socks5_proxy: Option<String>,
/// Maximum simultaneous inbound connections. Defaults to 256.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_inbound_connections: Option<usize>,
@@ -339,6 +335,169 @@ impl TcpConfig {
}
}
// ============================================================================
// Tor Transport Configuration
// ============================================================================
/// Default Tor SOCKS5 proxy address.
const DEFAULT_TOR_SOCKS5_ADDR: &str = "127.0.0.1:9050";
/// Default Tor control port address.
const DEFAULT_TOR_CONTROL_ADDR: &str = "/run/tor/control";
/// Default Tor control cookie file path (Debian standard location).
const DEFAULT_TOR_COOKIE_PATH: &str = "/var/run/tor/control.authcookie";
/// Default Tor connect timeout in milliseconds (120s — Tor circuit
/// establishment can take 30-60s on first connect, plus SOCKS5 handshake).
const DEFAULT_TOR_CONNECT_TIMEOUT_MS: u64 = 120_000;
/// Default Tor MTU (same as TCP).
const DEFAULT_TOR_MTU: u16 = 1400;
/// Default max inbound connections via onion service.
const DEFAULT_TOR_MAX_INBOUND: usize = 64;
/// Default HiddenServiceDir hostname file path.
const DEFAULT_HOSTNAME_FILE: &str = "/var/lib/tor/fips_onion_service/hostname";
/// Default directory mode bind address.
const DEFAULT_DIRECTORY_BIND_ADDR: &str = "127.0.0.1:8443";
/// Tor transport instance configuration.
///
/// Supports three modes:
/// - `socks5`: Outbound-only connections through a Tor SOCKS5 proxy.
/// - `control_port`: Full bidirectional support — outbound via SOCKS5
/// plus inbound via Tor onion service managed through the control port.
/// - `directory`: Full bidirectional support — outbound via SOCKS5,
/// inbound via a Tor-managed `HiddenServiceDir` onion service. No
/// control port needed. Enables Tor `Sandbox 1` mode.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct TorConfig {
/// Tor access mode: "socks5", "control_port", or "directory".
/// Default: "socks5".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mode: Option<String>,
/// SOCKS5 proxy address (host:port). Defaults to "127.0.0.1:9050".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub socks5_addr: Option<String>,
/// Outbound connect timeout in milliseconds. Defaults to 120000 (120s).
/// Tor circuit establishment can take 30-60s, so this must be generous.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub connect_timeout_ms: Option<u64>,
/// Default MTU for Tor connections. Defaults to 1400.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mtu: Option<u16>,
/// Control port address: a Unix socket path (`/run/tor/control`) or
/// TCP address (`host:port`). Unix sockets are preferred for security.
/// Defaults to "/run/tor/control".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub control_addr: Option<String>,
/// Control port authentication method:
/// `"cookie"` (read from default path),
/// `"cookie:/path/to/cookie"` (read from specified path), or
/// `"password:secret"` (password auth). Default: `"cookie"`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub control_auth: Option<String>,
/// Path to the Tor control cookie file. Used when control_auth is "cookie".
/// Defaults to "/var/run/tor/control.authcookie".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cookie_path: Option<String>,
/// Maximum number of inbound connections via onion service. Default: 64.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_inbound_connections: Option<usize>,
/// Directory-mode onion service configuration. Only valid in
/// "directory" mode. Tor manages the onion service via HiddenServiceDir
/// in torrc; fips reads the .onion hostname from a file.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub directory_service: Option<DirectoryServiceConfig>,
}
/// Directory-mode onion service configuration.
///
/// In `directory` mode, Tor manages the onion service via `HiddenServiceDir`
/// in torrc. FIPS reads the `.onion` address from the hostname file and
/// binds a local TCP listener for Tor to forward inbound connections to.
/// This mode requires no control port and enables Tor's `Sandbox 1`.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DirectoryServiceConfig {
/// Path to the Tor-managed hostname file containing the .onion address.
/// Defaults to "/var/lib/tor/fips_onion_service/hostname".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub hostname_file: Option<String>,
/// Local bind address for the listener that Tor forwards inbound
/// connections to. Must match the target in torrc's `HiddenServicePort`.
/// Defaults to "127.0.0.1:8443".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bind_addr: Option<String>,
}
impl DirectoryServiceConfig {
/// Path to the hostname file. Default: "/var/lib/tor/fips_onion_service/hostname".
pub fn hostname_file(&self) -> &str {
self.hostname_file.as_deref().unwrap_or(DEFAULT_HOSTNAME_FILE)
}
/// Local bind address for the listener. Default: "127.0.0.1:8443".
pub fn bind_addr(&self) -> &str {
self.bind_addr.as_deref().unwrap_or(DEFAULT_DIRECTORY_BIND_ADDR)
}
}
impl TorConfig {
/// Get the access mode. Default: "socks5".
pub fn mode(&self) -> &str {
self.mode.as_deref().unwrap_or("socks5")
}
/// Get the SOCKS5 proxy address. Default: "127.0.0.1:9050".
pub fn socks5_addr(&self) -> &str {
self.socks5_addr.as_deref().unwrap_or(DEFAULT_TOR_SOCKS5_ADDR)
}
/// Get the control port address. Default: "/run/tor/control".
pub fn control_addr(&self) -> &str {
self.control_addr.as_deref().unwrap_or(DEFAULT_TOR_CONTROL_ADDR)
}
/// Get the control auth string. Default: "cookie".
pub fn control_auth(&self) -> &str {
self.control_auth.as_deref().unwrap_or("cookie")
}
/// Get the cookie file path. Default: "/var/run/tor/control.authcookie".
pub fn cookie_path(&self) -> &str {
self.cookie_path.as_deref().unwrap_or(DEFAULT_TOR_COOKIE_PATH)
}
/// Get the connect timeout in milliseconds. Default: 120000.
pub fn connect_timeout_ms(&self) -> u64 {
self.connect_timeout_ms.unwrap_or(DEFAULT_TOR_CONNECT_TIMEOUT_MS)
}
/// Get the default MTU. Default: 1400.
pub fn mtu(&self) -> u16 {
self.mtu.unwrap_or(DEFAULT_TOR_MTU)
}
/// Get the max inbound connections. Default: 64.
pub fn max_inbound_connections(&self) -> usize {
self.max_inbound_connections.unwrap_or(DEFAULT_TOR_MAX_INBOUND)
}
}
// ============================================================================
// TransportsConfig
// ============================================================================
@@ -360,6 +519,10 @@ pub struct TransportsConfig {
/// TCP transport instances.
#[serde(default, skip_serializing_if = "is_transport_empty")]
pub tcp: TransportInstances<TcpConfig>,
/// Tor transport instances.
#[serde(default, skip_serializing_if = "is_transport_empty")]
pub tor: TransportInstances<TorConfig>,
}
/// Helper for skip_serializing_if on TransportInstances.
@@ -370,7 +533,7 @@ fn is_transport_empty<T>(instances: &TransportInstances<T>) -> bool {
impl TransportsConfig {
/// Check if any transports are configured.
pub fn is_empty(&self) -> bool {
self.udp.is_empty() && self.ethernet.is_empty() && self.tcp.is_empty()
self.udp.is_empty() && self.ethernet.is_empty() && self.tcp.is_empty() && self.tor.is_empty()
}
/// Merge another TransportsConfig into this one.
@@ -386,5 +549,8 @@ impl TransportsConfig {
if !other.tcp.is_empty() {
self.tcp = other.tcp;
}
if !other.tor.is_empty() {
self.tor = other.tor;
}
}
}

68
src/control/commands.rs Normal file
View File

@@ -0,0 +1,68 @@
//! Mutating control socket commands.
//!
//! Commands that modify node state (connect, disconnect) are handled here,
//! separate from read-only queries in `queries.rs`.
use super::protocol::Response;
use crate::node::Node;
use serde_json::Value;
use tracing::info;
/// Dispatch a mutating command to the appropriate handler.
pub async fn dispatch(node: &mut Node, command: &str, params: Option<&Value>) -> Response {
match command {
"connect" => connect(node, params).await,
"disconnect" => disconnect(node, params),
_ => Response::error(format!("unknown command: {command}")),
}
}
/// Connect to a peer.
///
/// Params: `{"npub": "npub1...", "address": "host:port", "transport": "udp"}`
async fn connect(node: &mut Node, params: Option<&Value>) -> Response {
let Some(params) = params else {
return Response::error("missing params for connect");
};
let npub = match params.get("npub").and_then(|v| v.as_str()) {
Some(v) => v,
None => return Response::error("missing 'npub' parameter"),
};
let address = match params.get("address").and_then(|v| v.as_str()) {
Some(v) => v,
None => return Response::error("missing 'address' parameter"),
};
let transport = match params.get("transport").and_then(|v| v.as_str()) {
Some(v) => v,
None => return Response::error("missing 'transport' parameter"),
};
info!(npub = %npub, address = %address, transport = %transport, "API connect requested");
match node.api_connect(npub, address, transport).await {
Ok(data) => Response::ok(data),
Err(msg) => Response::error(msg),
}
}
/// Disconnect a peer.
///
/// Params: `{"npub": "npub1..."}`
fn disconnect(node: &mut Node, params: Option<&Value>) -> Response {
let Some(params) = params else {
return Response::error("missing params for disconnect");
};
let npub = match params.get("npub").and_then(|v| v.as_str()) {
Some(v) => v,
None => return Response::error("missing 'npub' parameter"),
};
info!(npub = %npub, "API disconnect requested");
match node.api_disconnect(npub) {
Ok(data) => Response::ok(data),
Err(msg) => Response::error(msg),
}
}

View File

@@ -1,9 +1,10 @@
//! Control socket for runtime observability.
//! Control socket for runtime management and observability.
//!
//! Provides a Unix domain socket that accepts query commands and returns
//! structured JSON data about the node's current state. Read-only queries
//! only — no state mutation through this channel.
//! Provides a Unix domain socket that accepts commands and returns
//! structured JSON responses. Supports both read-only queries (show_*)
//! and mutating commands (connect, disconnect).
pub mod commands;
pub mod protocol;
pub mod queries;
@@ -54,6 +55,16 @@ impl ControlSocket {
}
let listener = UnixListener::bind(&socket_path)?;
// Make the socket and its parent directory group-accessible so
// 'fips' group members can use fipsctl/fipstop without root.
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(&socket_path, std::fs::Permissions::from_mode(0o770))?;
Self::chown_to_fips_group(&socket_path);
if let Some(parent) = socket_path.parent() {
Self::chown_to_fips_group(parent);
}
info!(path = %socket_path.display(), "Control socket listening");
Ok(Self {
@@ -85,6 +96,34 @@ impl ControlSocket {
}
}
/// Set group ownership of a path to the 'fips' group (best-effort).
fn chown_to_fips_group(path: &Path) {
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
// Look up the 'fips' group
let group_name = CString::new("fips").unwrap();
let grp = unsafe { libc::getgrnam(group_name.as_ptr()) };
if grp.is_null() {
debug!("'fips' group not found, skipping chown for {}", path.display());
return;
}
let gid = unsafe { (*grp).gr_gid };
let c_path = match CString::new(path.as_os_str().as_bytes()) {
Ok(p) => p,
Err(_) => return,
};
let ret = unsafe { libc::chown(c_path.as_ptr(), u32::MAX, gid) };
if ret != 0 {
warn!(
path = %path.display(),
error = %std::io::Error::last_os_error(),
"Failed to chown control socket to 'fips' group"
);
}
}
/// Run the accept loop, forwarding requests to the main event loop via mpsc.
///
/// Each accepted connection is handled in a spawned task:

View File

@@ -8,8 +8,11 @@ use serde::{Deserialize, Serialize};
/// A control request from a client.
#[derive(Debug, Deserialize)]
pub struct Request {
/// The command to execute (e.g., "show_status", "show_peers").
/// The command to execute (e.g., "show_status", "connect").
pub command: String,
/// Optional parameters for mutating commands.
#[serde(default)]
pub params: Option<serde_json::Value>,
}
/// A control response to a client.
@@ -81,6 +84,24 @@ mod tests {
assert_eq!(req.command, "show_peers");
}
#[test]
fn test_deserialize_request_with_params() {
let json = r#"{"command": "connect", "params": {"npub": "npub1abc", "address": "1.2.3.4:2121", "transport": "udp"}}"#;
let req: Request = serde_json::from_str(json).unwrap();
assert_eq!(req.command, "connect");
let params = req.params.unwrap();
assert_eq!(params["npub"], "npub1abc");
assert_eq!(params["transport"], "udp");
}
#[test]
fn test_deserialize_request_without_params() {
let json = r#"{"command": "show_status"}"#;
let req: Request = serde_json::from_str(json).unwrap();
assert_eq!(req.command, "show_status");
assert!(req.params.is_none());
}
#[test]
fn test_deserialize_malformed_request() {
let json = r#"{"not_command": "foo"}"#;

View File

@@ -490,6 +490,18 @@ pub fn show_transports(node: &Node) -> Value {
t_json["local_addr"] = json!(format!("{}", addr));
}
// Tor-specific fields
if let Some(mode) = handle.tor_mode() {
t_json["tor_mode"] = json!(mode);
}
if let Some(onion) = handle.onion_address() {
t_json["onion_address"] = json!(onion);
}
if let Some(monitoring) = handle.tor_monitoring() {
t_json["tor_monitoring"] =
serde_json::to_value(&monitoring).unwrap_or_default();
}
t_json["stats"] = handle.transport_stats();
t_json

View File

@@ -26,7 +26,7 @@ pub use identity::{
};
// Re-export config types
pub use config::{Config, ConfigError, IdentityConfig, UdpConfig};
pub use config::{Config, ConfigError, IdentityConfig, TorConfig, UdpConfig};
pub use upper::config::{DnsConfig, TunConfig};
// Re-export tree types

View File

@@ -192,6 +192,11 @@ impl OwdTrendDetector {
}
}
/// Clear all samples, keeping the same capacity.
pub fn clear(&mut self) {
self.samples.clear();
}
/// Add an OWD sample.
///
/// `seq` is a monotonic sequence number (e.g., truncated frame counter).

View File

@@ -45,9 +45,39 @@ pub struct MmpMetrics {
prev_rr_reorder: u32,
/// Time of previous ReceiverReport (for goodput rate computation).
prev_rr_time: Option<Instant>,
/// Whether we have a previous ReceiverReport for delta computation.
has_prev_rr: bool,
// --- State for reverse delivery ratio delta computation ---
/// Previous reverse-side cumulative packets received (our receiver state).
prev_reverse_packets: u64,
/// Previous reverse-side highest counter (our receiver state).
prev_reverse_highest: u64,
/// Whether we have a previous reverse-side snapshot for delta computation.
has_prev_reverse: bool,
}
impl MmpMetrics {
/// Reset state derived from ReceiverReport counters for rekey cutover.
///
/// The new session starts with counter 0, so the prev_rr deltas must
/// be reset to avoid computing bogus loss/goodput from the counter
/// discontinuity. RTT (SRTT) is preserved since it remains valid.
pub fn reset_for_rekey(&mut self) {
self.prev_rr_cum_packets = 0;
self.prev_rr_cum_bytes = 0;
self.prev_rr_highest_counter = 0;
self.prev_rr_ecn_ce = 0;
self.prev_rr_reorder = 0;
self.prev_rr_time = None;
self.has_prev_rr = false;
self.delivery_ratio_forward = 1.0;
self.prev_reverse_packets = 0;
self.prev_reverse_highest = 0;
self.has_prev_reverse = false;
// Keep srtt, etx, trends, goodput_bps — they'll refresh from data
}
pub fn new() -> Self {
Self {
srtt: SrttEstimator::new(),
@@ -66,6 +96,10 @@ impl MmpMetrics {
prev_rr_ecn_ce: 0,
prev_rr_reorder: 0,
prev_rr_time: None,
has_prev_rr: false,
prev_reverse_packets: 0,
prev_reverse_highest: 0,
has_prev_reverse: false,
}
}
@@ -73,7 +107,12 @@ impl MmpMetrics {
///
/// `our_timestamp_ms` is the current session-relative time in ms (for RTT).
/// `now` is the current monotonic time (for goodput rate computation).
pub fn process_receiver_report(&mut self, rr: &ReceiverReport, our_timestamp_ms: u32, now: Instant) {
///
/// Returns `true` if this report produced the first SRTT measurement
/// (transition from uninitialized to initialized).
pub fn process_receiver_report(&mut self, rr: &ReceiverReport, our_timestamp_ms: u32, now: Instant) -> bool {
let had_srtt = self.srtt.initialized();
// --- RTT from timestamp echo ---
// RTT = now - echoed_timestamp - dwell_time
if rr.timestamp_echo > 0 {
@@ -98,7 +137,7 @@ impl MmpMetrics {
// --- Loss rate from cumulative counters ---
// Delta: frames the peer should have received vs. actually received
if self.prev_rr_highest_counter > 0 {
if self.has_prev_rr {
let counter_span = rr.highest_counter.saturating_sub(self.prev_rr_highest_counter);
let packets_delta = rr.cumulative_packets_recv.saturating_sub(self.prev_rr_cum_packets);
@@ -113,7 +152,7 @@ impl MmpMetrics {
}
// --- Goodput from cumulative bytes + time delta ---
if self.prev_rr_cum_bytes > 0 {
if self.has_prev_rr {
let bytes_delta = rr.cumulative_bytes_recv.saturating_sub(self.prev_rr_cum_bytes);
self.goodput_trend.update(bytes_delta as f64);
@@ -143,13 +182,31 @@ impl MmpMetrics {
self.prev_rr_ecn_ce = rr.ecn_ce_count;
self.prev_rr_reorder = rr.cumulative_reorder_count;
self.prev_rr_time = Some(now);
self.has_prev_rr = true;
!had_srtt && self.srtt.initialized()
}
/// Update the reverse delivery ratio (from our own receiver state about the peer's traffic).
pub fn set_delivery_ratio_reverse(&mut self, ratio: f64) {
self.delivery_ratio_reverse = ratio.clamp(0.0, 1.0);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
/// Update the reverse delivery ratio from our own receiver state.
///
/// Computes a per-interval delta (same as forward ratio) rather than
/// a lifetime cumulative ratio, so ETX responds to recent conditions.
pub fn update_reverse_delivery(&mut self, our_recv_packets: u64, peer_highest: u64) {
if self.has_prev_reverse {
let counter_span = peer_highest.saturating_sub(self.prev_reverse_highest);
let packets_delta = our_recv_packets.saturating_sub(self.prev_reverse_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_reverse = delivery.clamp(0.0, 1.0);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
self.prev_reverse_packets = our_recv_packets;
self.prev_reverse_highest = peer_highest;
self.has_prev_reverse = true;
}
/// Current smoothed RTT in milliseconds, or `None` if not yet measured.
@@ -272,9 +329,15 @@ mod tests {
let mut m = MmpMetrics::new();
assert_eq!(m.etx, 1.0); // initial: perfect
// Simulate some loss
// Simulate some loss via forward ratio
m.delivery_ratio_forward = 0.9;
m.set_delivery_ratio_reverse(0.95);
// First call establishes the baseline (no ETX update yet)
m.update_reverse_delivery(100, 100);
assert_eq!(m.etx, 1.0); // still perfect — baseline only
// Second call: 190 of 200 frames received (5% loss)
m.update_reverse_delivery(290, 300);
assert!(m.etx > 1.0);
assert!(m.etx < 2.0);
}
@@ -319,4 +382,47 @@ mod tests {
assert!(m.goodput_bps() > 90_000.0, "goodput={}, expected ~100000", m.goodput_bps());
assert!(m.goodput_bps() < 110_000.0, "goodput={}, expected ~100000", m.goodput_bps());
}
#[test]
fn test_reverse_delivery_delta() {
let mut m = MmpMetrics::new();
// First call: baseline only, no ratio update
m.update_reverse_delivery(100, 100);
assert_eq!(m.delivery_ratio_reverse, 1.0); // unchanged from default
// Second call: perfect delivery (200 new frames, all received)
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Third call: 50% loss (100 frames sent, 50 received)
m.update_reverse_delivery(350, 400);
assert!((m.delivery_ratio_reverse - 0.5).abs() < 0.001,
"reverse={}, expected 0.5", m.delivery_ratio_reverse);
}
#[test]
fn test_reverse_delivery_rekey_reset() {
let mut m = MmpMetrics::new();
// Establish baseline and one measurement
m.update_reverse_delivery(100, 100);
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Rekey resets reverse state
m.reset_for_rekey();
// First call after rekey: baseline only
m.update_reverse_delivery(50, 50);
// delivery_ratio_reverse was reset to 1.0 by reset_for_rekey's
// clearing of delivery_ratio_forward; reverse is not explicitly
// reset — but the delta state is, so next call computes fresh.
assert_eq!(m.delivery_ratio_reverse, 1.0);
// Second call after rekey: 80% delivery
m.update_reverse_delivery(90, 100);
assert!((m.delivery_ratio_reverse - 0.8).abs() < 0.001,
"reverse={}, expected 0.8", m.delivery_ratio_reverse);
}
}

View File

@@ -197,6 +197,12 @@ impl MmpPeerState {
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
@@ -256,6 +262,12 @@ impl MmpSessionState {
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode

View File

@@ -10,6 +10,14 @@ use crate::mmp::report::ReceiverReport;
use crate::mmp::{DEFAULT_COLD_START_INTERVAL_MS, DEFAULT_OWD_WINDOW_SIZE,
MAX_REPORT_INTERVAL_MS, MIN_REPORT_INTERVAL_MS};
/// Grace period after rekey before resuming jitter calculation.
///
/// During rekey cutover, frames from the old session may still arrive via the
/// drain window (DRAIN_WINDOW_SECS = 10s). These carry large sender timestamps
/// from the old session, producing enormous transit deltas that spike the EWMA
/// jitter estimator. We suppress jitter updates for drain window + 5s margin.
const REKEY_JITTER_GRACE_SECS: u64 = 15;
// ============================================================================
// Gap Tracker (burst loss detection)
// ============================================================================
@@ -161,6 +169,11 @@ pub struct ReceiverState {
/// Local time when the most recent frame was received (for dwell computation).
last_recv_time: Option<Instant>,
// --- Rekey grace ---
/// When set, jitter updates are suppressed until this instant passes.
/// Prevents drain-window frames from spiking the jitter estimator.
rekey_jitter_grace_until: Option<Instant>,
// --- Report timing ---
last_report_time: Option<Instant>,
report_interval: Duration,
@@ -192,12 +205,37 @@ impl ReceiverState {
ecn_ce_count: 0,
last_sender_timestamp: 0,
last_recv_time: None,
rekey_jitter_grace_until: None,
last_report_time: None,
report_interval: Duration::from_millis(cold_start_ms),
interval_has_data: false,
}
}
/// Reset counter-dependent state for rekey cutover.
///
/// After cutover, the new session starts with counter 0 and reset
/// timestamps. Without resetting, the old `highest_counter` and
/// `GapTracker.expected_next` cause false reorder/loss detection.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.highest_counter = 0;
self.cumulative_reorder_count = 0;
self.gap_tracker = GapTracker::new();
self.interval_packets_recv = 0;
self.interval_bytes_recv = 0;
self.jitter = JitterEstimator::new();
self.owd_trend.clear();
self.owd_seq = 0;
self.last_sender_timestamp = 0;
self.last_recv_time = None;
self.rekey_jitter_grace_until =
Some(now + Duration::from_secs(REKEY_JITTER_GRACE_SECS));
self.ecn_ce_count = 0;
self.interval_has_data = false;
// Keep cumulative_packets_recv, cumulative_bytes_recv (lifetime stats)
// Keep last_report_time, report_interval (report scheduling)
}
/// Record a received frame from this peer.
///
/// Called on the RX path after AEAD decryption, before message dispatch.
@@ -242,11 +280,18 @@ impl ReceiverState {
let sender_us = (sender_timestamp_ms as i64) * 1000;
// We can't get absolute µs from Instant, but we can compute the delta
// between consecutive transits using relative Instant differences.
if let Some(prev_recv) = self.last_recv_time {
let recv_delta_us = now.duration_since(prev_recv).as_micros() as i64;
let send_delta_us = sender_us - (self.last_sender_timestamp as i64 * 1000);
let transit_delta = (recv_delta_us - send_delta_us) as i32;
self.jitter.update(transit_delta);
// Skip during post-rekey grace period to avoid drain-window spikes.
let in_grace = self.rekey_jitter_grace_until
.is_some_and(|deadline| now < deadline);
if !in_grace {
self.rekey_jitter_grace_until = None; // clear expired grace
if let Some(prev_recv) = self.last_recv_time {
let recv_delta_us = now.duration_since(prev_recv).as_micros() as i64;
let send_delta_us =
sender_us - (self.last_sender_timestamp as i64 * 1000);
let transit_delta = (recv_delta_us - send_delta_us) as i32;
self.jitter.update(transit_delta);
}
}
// OWD trend: use sender timestamp as a proxy for send time
@@ -531,4 +576,36 @@ mod tests {
r.update_report_interval_from_srtt(500_000);
assert_eq!(r.report_interval(), Duration::from_millis(500));
}
#[test]
fn test_rekey_jitter_grace_suppresses_spikes() {
let mut r = ReceiverState::new(32);
let t0 = Instant::now();
// Establish baseline with two frames so jitter starts updating
r.record_recv(1, 1000, 100, false, t0);
r.record_recv(2, 2000, 100, false, t0 + Duration::from_secs(1));
assert_eq!(r.jitter_us(), 0); // perfect 1s spacing → 0 jitter
// Simulate rekey: reset, then send a frame with a large old-session
// timestamp followed by a new-session timestamp near zero.
// Without grace, this would produce a huge jitter spike.
r.reset_for_rekey(t0 + Duration::from_secs(2));
// Frame arrives during grace period with old-session timestamp
r.record_recv(0, 120_000, 100, false, t0 + Duration::from_secs(3));
// Next frame with new-session timestamp near zero
r.record_recv(1, 100, 100, false, t0 + Duration::from_secs(4));
// Jitter should still be zero — updates suppressed during grace
assert_eq!(r.jitter_us(), 0);
// After grace expires, jitter updates resume
let after_grace = t0 + Duration::from_secs(2)
+ Duration::from_secs(REKEY_JITTER_GRACE_SECS + 1);
r.record_recv(2, 200, 100, false, after_grace);
r.record_recv(3, 300, 100, false, after_grace + Duration::from_millis(100));
// Now jitter should be updating (non-zero or zero depending on timing)
// The key assertion is that it's not a multi-second spike
assert!(r.jitter_us() < 1_000_000); // less than 1 second
}
}

View File

@@ -0,0 +1,378 @@
//! Discovery protocol rate limiting and backoff.
//!
//! Two complementary mechanisms:
//!
//! - **`DiscoveryBackoff`** (originator-side): Exponential backoff for failed
//! lookups. After a lookup times out, suppresses re-initiation with
//! increasing delays (30s → 60s → 300s cap). Reset on topology changes
//! (parent change, new peer, first RTT, reconnection).
//!
//! - **`DiscoveryForwardRateLimiter`** (transit-side): Per-target minimum
//! interval for forwarded requests. Defense-in-depth against misbehaving
//! nodes generating fresh request_ids at high rate.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
// ============================================================================
// Originator-side: Discovery Backoff
// ============================================================================
/// Default base backoff after first lookup failure.
const DEFAULT_BACKOFF_BASE_SECS: u64 = 30;
/// Default maximum backoff cap.
const DEFAULT_BACKOFF_MAX_SECS: u64 = 300;
/// Backoff multiplier per consecutive failure.
const BACKOFF_MULTIPLIER: u64 = 2;
/// Exponential backoff for failed discovery lookups.
///
/// Tracks targets whose lookups have timed out and suppresses
/// re-initiation with increasing delays. Cleared on topology changes.
pub struct DiscoveryBackoff {
/// Maps target → (suppress_until, consecutive_failures).
entries: HashMap<NodeAddr, BackoffEntry>,
/// Base backoff duration (first failure).
base: Duration,
/// Maximum backoff cap.
max: Duration,
}
struct BackoffEntry {
/// Don't re-initiate until this instant.
suppress_until: Instant,
/// Consecutive failures (drives exponential backoff).
failures: u32,
}
impl DiscoveryBackoff {
/// Create with default parameters (30s base, 300s cap).
pub fn new() -> Self {
Self::with_params(DEFAULT_BACKOFF_BASE_SECS, DEFAULT_BACKOFF_MAX_SECS)
}
/// Create with custom base and max backoff in seconds.
pub fn with_params(base_secs: u64, max_secs: u64) -> Self {
Self {
entries: HashMap::new(),
base: Duration::from_secs(base_secs),
max: Duration::from_secs(max_secs),
}
}
/// Check if a lookup for this target is suppressed.
///
/// Returns true if the target is in backoff and should not be
/// looked up yet.
pub fn is_suppressed(&self, target: &NodeAddr) -> bool {
if let Some(entry) = self.entries.get(target) {
Instant::now() < entry.suppress_until
} else {
false
}
}
/// Record a lookup failure (timeout) for a target.
///
/// Increments the failure count and sets the next suppression
/// window using exponential backoff.
pub fn record_failure(&mut self, target: &NodeAddr) {
let now = Instant::now();
let failures = self
.entries
.get(target)
.map_or(0, |e| e.failures)
+ 1;
let backoff_secs = self
.base
.as_secs()
.saturating_mul(BACKOFF_MULTIPLIER.saturating_pow(failures.saturating_sub(1)));
let backoff = Duration::from_secs(backoff_secs.min(self.max.as_secs()));
self.entries.insert(
*target,
BackoffEntry {
suppress_until: now + backoff,
failures,
},
);
}
/// Record a successful lookup — remove backoff for this target.
pub fn record_success(&mut self, target: &NodeAddr) {
self.entries.remove(target);
}
/// Clear all backoff entries.
///
/// Called on topology changes that might make previously-unreachable
/// targets reachable (parent change, new peer, first RTT, reconnection).
pub fn reset_all(&mut self) {
self.entries.clear();
}
/// Whether any entries exist.
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// Current number of entries.
pub fn entry_count(&self) -> usize {
self.entries.len()
}
/// Get the failure count for a target (for logging).
pub fn failure_count(&self, target: &NodeAddr) -> u32 {
self.entries.get(target).map_or(0, |e| e.failures)
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.entries.len()
}
}
impl Default for DiscoveryBackoff {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Transit-side: Discovery Forward Rate Limiter
// ============================================================================
/// Default minimum interval between forwarded lookups for the same target.
const DEFAULT_FORWARD_MIN_INTERVAL: Duration = Duration::from_secs(2);
/// Maximum age of entries before cleanup.
const FORWARD_MAX_AGE: Duration = Duration::from_secs(60);
/// Rate limiter for forwarded discovery requests.
///
/// Tracks the last time a LookupRequest was forwarded for each target
/// and enforces a minimum interval to prevent floods from misbehaving
/// nodes generating fresh request_ids.
pub struct DiscoveryForwardRateLimiter {
last_forwarded: HashMap<NodeAddr, Instant>,
min_interval: Duration,
max_age: Duration,
}
impl DiscoveryForwardRateLimiter {
/// Create with default parameters (2s interval).
pub fn new() -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval: DEFAULT_FORWARD_MIN_INTERVAL,
max_age: FORWARD_MAX_AGE,
}
}
/// Create with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval,
max_age: FORWARD_MAX_AGE,
}
}
/// Check if we should forward a lookup for this target.
///
/// Returns true if enough time has passed since the last forward
/// for this target. Updates internal state when returning true.
pub fn should_forward(&mut self, target: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_forwarded.get(target)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_forwarded.insert(*target, now);
self.cleanup(now);
true
}
/// Replace the minimum interval (e.g., set to zero to disable).
#[cfg(test)]
pub fn set_interval(&mut self, interval: Duration) {
self.min_interval = interval;
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_forwarded
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_forwarded.len()
}
}
impl Default for DiscoveryForwardRateLimiter {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
// --- DiscoveryBackoff tests ---
#[test]
fn test_backoff_not_suppressed_initially() {
let backoff = DiscoveryBackoff::new();
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_suppressed_after_failure() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
// Different target not affected
assert!(!backoff.is_suppressed(&addr(2)));
}
#[test]
fn test_backoff_cleared_on_success() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
backoff.record_success(&addr(1));
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_reset_all() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
backoff.record_failure(&addr(2));
assert_eq!(backoff.len(), 2);
backoff.reset_all();
assert_eq!(backoff.len(), 0);
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_exponential() {
let mut backoff = DiscoveryBackoff::with_params(1, 300);
// First failure: 1s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 1);
// Second failure: 2s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 2);
// Third failure: 4s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 3);
}
#[test]
fn test_backoff_expires() {
let mut backoff = DiscoveryBackoff::with_params(0, 0);
backoff.record_failure(&addr(1));
// With 0s backoff, should not be suppressed
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_capped() {
let mut backoff = DiscoveryBackoff::with_params(1, 10);
// Record many failures
for _ in 0..20 {
backoff.record_failure(&addr(1));
}
// Backoff should be capped at max (10s), not overflow
let entry = backoff.entries.get(&addr(1)).unwrap();
let remaining = entry.suppress_until.duration_since(Instant::now());
assert!(remaining <= Duration::from_secs(11));
}
// --- DiscoveryForwardRateLimiter tests ---
#[test]
fn test_forward_first_allowed() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_rapid_rate_limited() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_different_targets_independent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(2)));
}
#[test]
fn test_forward_allowed_after_interval() {
let mut limiter =
DiscoveryForwardRateLimiter::with_interval(Duration::from_millis(100));
assert!(limiter.should_forward(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_cleanup_removes_old() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(61);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_forward_cleanup_preserves_recent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
}

View File

@@ -1,25 +1,25 @@
//! LookupRequest/LookupResponse discovery protocol handlers.
//!
//! Handles coordinate discovery requests: flood-based lookup with TTL,
//! visited filter for loop prevention, and reverse-path forwarding for
//! responses.
//! Handles coordinate discovery via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::{Node, RecentRequest};
use crate::protocol::{LookupRequest, LookupResponse};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, trace, warn};
use tracing::{debug, info, trace, warn};
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup)
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. Check visited filter (loop prevention)
/// 6. If we're the target, generate and send response
/// 7. If TTL > 0, forward to peers not in visited filter
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(
&mut self,
from: &NodeAddr,
@@ -38,10 +38,12 @@ impl Node {
let now_ms = Self::now_ms();
// Dedup: drop if we've already seen this request_id
// Dedup: drop if we've already seen this request_id.
// Also serves as loop protection — tree routing is loop-free,
// but request_id dedup catches edge cases during tree restructuring.
if self.recent_requests.contains_key(&request.request_id) {
self.stats_mut().discovery.req_duplicate += 1;
trace!(
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
@@ -58,17 +60,6 @@ impl Node {
// Lazy purge expired entries
self.purge_expired_requests(now_ms);
// Loop prevention: drop if we've already been visited
if request.was_visited(self.node_addr()) {
self.stats_mut().discovery.req_already_visited += 1;
trace!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Already visited, dropping LookupRequest"
);
return;
}
// Are we the target?
if request.target == *self.node_addr() {
self.stats_mut().discovery.req_target_is_us += 1;
@@ -83,14 +74,25 @@ impl Node {
// Forward if TTL permits
if request.can_forward() {
// Transit-side rate limit: collapse rapid-fire lookups for the
// same target from misbehaving nodes generating fresh request_ids.
if !self.discovery_forward_limiter.should_forward(&request.target) {
self.stats_mut().discovery.req_forward_rate_limited += 1;
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
return;
}
self.stats_mut().discovery.req_forwarded += 1;
self.forward_lookup_request(request).await;
} else {
self.stats_mut().discovery.req_ttl_exhausted += 1;
trace!(
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted, not forwarding"
"LookupRequest TTL exhausted"
);
}
}
@@ -121,7 +123,19 @@ impl Node {
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
if let Some(recent) = self.recent_requests.get(&response.request_id) {
if let Some(recent) = self.recent_requests.get_mut(&response.request_id) {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
if recent.response_forwarded {
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
return;
}
recent.response_forwarded = true;
// Transit node: reverse-path forward
let from_peer = recent.from_peer;
self.stats_mut().discovery.resp_forwarded += 1;
@@ -195,12 +209,15 @@ impl Node {
self.stats_mut().discovery.resp_accepted += 1;
debug!(
// Clear backoff on success — target is reachable
self.discovery_backoff.record_success(&target);
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Received LookupResponse, proof verified, caching route"
"Discovery succeeded, proof verified, route cached"
);
self.coord_cache.insert_with_path_mtu(
@@ -214,9 +231,6 @@ impl Node {
self.pending_lookups.remove(&target);
// If an established session exists, reset the warmup counter.
// Discovery has completed and transit nodes along the response
// path now have fresh coords. Reset warmup so the next N
// data packets include COORDS_PRESENT to re-warm the forward path.
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
@@ -232,22 +246,18 @@ impl Node {
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if self.pending_tun_packets.contains_key(&target) {
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
}
/// Generate and send a LookupResponse when we are the target.
///
/// Signs a proof using our identity and routes the response back
/// toward the origin via reverse-path forwarding. The first hop
/// uses the `recent_requests` entry (which records who sent us the
/// request), ensuring the response follows the same path the
/// request took. This is critical because greedy tree routing
/// might send the response to a peer that never forwarded the
/// request and thus has no `recent_requests` entry, causing the
/// response to be discarded.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
@@ -262,10 +272,7 @@ impl Node {
proof,
);
// Route toward origin via reverse path. The recent_requests entry
// was recorded before we got here (line 49-51), so from_peer is
// the node that forwarded the request to us — the correct first
// hop for the response's reverse path.
// Route toward origin via reverse path.
let next_hop_addr = if let Some(recent) = self.recent_requests.get(&request.request_id) {
recent.from_peer
} else {
@@ -299,37 +306,71 @@ impl Node {
}
}
/// Forward a LookupRequest to peers not in the visited filter.
/// Forward a LookupRequest to eligible peers.
///
/// Decrements TTL, adds self to visited, and sends to all eligible peers.
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
if !request.forward(self.node_addr()) {
if !request.forward() {
return;
}
// Collect peers not in visited filter
// Collect tree peers whose bloom filter contains the target
let forward_to: Vec<NodeAddr> = self
.peers
.keys()
.filter(|addr| !request.was_visited(addr))
.copied()
.iter()
.filter(|(addr, peer)| {
self.is_tree_peer(addr) && peer.may_reach(&request.target)
})
.map(|(addr, _)| *addr)
.collect();
if forward_to.is_empty() {
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
return;
}
// Fallback: if no tree peer matches, try non-tree bloom-matching peers
let (forward_to, used_fallback) = if forward_to.is_empty() {
let fallback: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| {
!self.is_tree_peer(addr) && peer.may_reach(&request.target)
})
.map(|(addr, _)| *addr)
.collect();
if fallback.is_empty() {
self.stats_mut().discovery.req_no_tree_peer += 1;
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
return;
}
(fallback, true)
} else {
(forward_to, false)
};
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest"
);
if used_fallback {
self.stats_mut().discovery.req_fallback_forwarded += 1;
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest"
);
}
let encoded = request.encode();
@@ -346,30 +387,42 @@ impl Node {
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and floods it to all peers. The originator
/// does NOT record the request_id in recent_requests, so when the
/// response arrives, it's recognized as "our request" and the
/// target's coordinates are cached in coord_cache.
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) {
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.stats_mut().discovery.req_initiated += 1;
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let mut request = LookupRequest::generate(*target, origin, origin_coords, ttl, 0);
let request = LookupRequest::generate(*target, origin, origin_coords, ttl, 0);
// Add ourselves to the visited filter so forwarding nodes
// won't send the request back to us
request.visited.insert(&origin);
// Send only to tree peers whose bloom filter contains the target
let peer_addrs: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| {
self.is_tree_peer(addr) && peer.may_reach(target)
})
.map(|(addr, _)| *addr)
.collect();
debug!(
let peer_count = peer_addrs.len();
info!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
"Initiating LookupRequest"
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
// Send to all peers (flood)
let peer_addrs: Vec<NodeAddr> = self.peers.keys().copied().collect();
if peer_count == 0 {
return 0;
}
let encoded = request.encode();
for peer_addr in peer_addrs {
@@ -381,43 +434,136 @@ impl Node {
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Deduplicates lookups using `pending_lookups` with a timeout. If a
/// lookup was recently initiated and hasn't timed out, this is a no-op.
/// Checks: pending dedup, backoff, bloom filter pre-check. If all pass,
/// initiates the lookup. If no tree peers have the target in their bloom
/// filter, the lookup is skipped (bloom miss) and recorded as a failure
/// for backoff purposes.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
let lookup_timeout_ms = self.config.node.discovery.timeout_secs * 1000;
if let Some(&initiated_at) = self.pending_lookups.get(dest)
&& now_ms.saturating_sub(initiated_at) < lookup_timeout_ms
{
self.stats_mut().discovery.req_deduplicated += 1;
// Check pending lookup dedup (in-flight)
if let Some(entry) = self.pending_lookups.get(dest) {
let age_ms = now_ms.saturating_sub(entry.initiated_ms);
let attempt = entry.attempt;
if age_ms < lookup_timeout_ms {
self.stats_mut().discovery.req_deduplicated += 1;
debug!(
target_node = %self.peer_display_name(dest),
age_ms = age_ms,
attempt = attempt,
"Discovery lookup deduplicated, already pending"
);
return;
}
}
// Check backoff from previous failures
if self.discovery_backoff.is_suppressed(dest) {
self.stats_mut().discovery.req_backoff_suppressed += 1;
debug!(
target_node = %self.peer_display_name(dest),
failures = self.discovery_backoff.failure_count(dest),
"Discovery lookup suppressed by backoff"
);
return;
}
self.pending_lookups.insert(*dest, now_ms);
// Bloom filter pre-check: if no peer's filter contains the target,
// it's not in the mesh — skip the lookup and record as failure.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
if !reachable {
self.stats_mut().discovery.req_bloom_miss += 1;
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
return;
}
self.pending_lookups.insert(*dest, PendingLookup::new(now_ms));
let ttl = self.config.node.discovery.ttl;
self.initiate_lookup(dest, ttl).await;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
self.pending_lookups.remove(dest);
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
/// Remove timed-out pending lookups and drain their queued packets.
/// Check pending lookups for retry or timeout.
///
/// Called periodically from the tick handler. For each timed-out lookup,
/// sends ICMPv6 Destination Unreachable for any queued TUN packets and
/// removes them from the pending queue.
pub(in crate::node) fn purge_stale_lookups(&mut self, now_ms: u64) {
let timed_out: Vec<NodeAddr> = self
.pending_lookups
.iter()
.filter(|&(_, &ts)| now_ms.saturating_sub(ts) >= self.config.node.discovery.timeout_secs * 1000)
.map(|(addr, _)| *addr)
.collect();
/// Called periodically from the tick handler. For each pending lookup:
/// - If retry interval elapsed and attempts remain: resend
/// - If total timeout elapsed: fail, record backoff, send ICMP unreachable
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let timeout_ms = self.config.node.discovery.timeout_secs * 1000;
let retry_ms = self.config.node.discovery.retry_interval_secs * 1000;
for addr in timed_out {
// Collect targets needing action
let mut to_retry: Vec<NodeAddr> = Vec::new();
let mut to_timeout: Vec<NodeAddr> = Vec::new();
for (&target, entry) in &self.pending_lookups {
let age = now_ms.saturating_sub(entry.initiated_ms);
if age >= timeout_ms {
to_timeout.push(target);
} else if entry.attempt < self.config.node.discovery.max_attempts
&& now_ms.saturating_sub(entry.last_sent_ms) >= retry_ms
{
to_retry.push(target);
}
}
// Process retries
for target in to_retry {
if let Some(entry) = self.pending_lookups.get_mut(&target) {
entry.attempt += 1;
entry.last_sent_ms = now_ms;
let attempt = entry.attempt;
let ttl = self.config.node.discovery.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
}
// Process timeouts
for addr in to_timeout {
self.stats_mut().discovery.resp_timed_out += 1;
self.pending_lookups.remove(&addr);
if let Some(packets) = self.pending_tun_packets.remove(&addr) {
// Record failure for backoff
self.discovery_backoff.record_failure(&addr);
let failures = self.discovery_backoff.failure_count(&addr);
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
@@ -425,11 +571,41 @@ impl Node {
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_discovery_backoff(&mut self) {
if !self.discovery_backoff.is_empty() {
debug!(
entries = self.discovery_backoff.entry_count(),
"Resetting discovery backoff on topology change"
);
self.discovery_backoff.reset_all();
}
}
/// Remove expired entries from the recent_requests cache.
fn purge_expired_requests(&mut self, current_time_ms: u64) {
let expiry_ms = self.config.node.discovery.recent_expiry_secs * 1000;
self.recent_requests
.retain(|_, entry| !entry.is_expired(current_time_ms, expiry_ms));
}
}
/// Tracks a pending discovery lookup with retry state.
pub(crate) struct PendingLookup {
/// When the lookup was first initiated.
pub initiated_ms: u64,
/// When the last attempt was sent.
pub last_sent_ms: u64,
/// Current attempt number (1 = initial, 2 = first retry, ...).
pub attempt: u8,
}
impl PendingLookup {
pub fn new(now_ms: u64) -> Self {
Self {
initiated_ms: now_ms,
last_sent_ms: now_ms,
attempt: 1,
}
}
}

View File

@@ -27,7 +27,7 @@ impl Node {
}
0x02 => {
// ReceiverReport
self.handle_receiver_report(from, payload);
self.handle_receiver_report(from, payload).await;
}
0x10 => {
// TreeAnnounce
@@ -109,13 +109,29 @@ impl Node {
}
// MMP teardown log (before we drop the peer)
let peer_name = self.peer_aliases.get(node_addr)
.cloned()
.unwrap_or_else(|| peer.identity().short_npub());
if let Some(mmp) = peer.mmp() {
let name = self.peer_aliases.get(node_addr)
.cloned()
.unwrap_or_else(|| peer.identity().short_npub());
Self::log_mmp_teardown(&name, mmp);
Self::log_mmp_teardown(&peer_name, mmp);
}
// Remove any end-to-end session associated with this peer.
//
// Sessions are tracked separately from peers (self.sessions vs self.peers).
// Leaving a stale session alive after removing the peer causes:
// 1. check_session_mmp_reports() keeps logging stale "MMP session metrics"
// with frozen counters until purge_idle_sessions() eventually fires.
// 2. initiate_session() finds is_established() == true on the stale entry
// and silently returns Ok(()), preventing a new session from being
// established even after the link layer reconnects successfully.
if let Some(session_entry) = self.sessions.remove(node_addr)
&& let Some(mmp) = session_entry.mmp()
{
Self::log_session_mmp_teardown(&peer_name, mmp);
}
self.pending_tun_packets.remove(node_addr);
let link_id = peer.link_id();
let transport_id = peer.transport_id();

View File

@@ -5,7 +5,7 @@ use crate::node::Node;
use crate::node::wire::{EncryptedHeader, strip_inner_header, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP};
use crate::transport::ReceivedPacket;
use std::time::Instant;
use tracing::{debug, info, warn};
use tracing::{debug, info, trace, warn};
/// Force-remove a peer after this many consecutive decryption failures.
const DECRYPT_FAILURE_THRESHOLD: u32 = 20;
@@ -32,7 +32,7 @@ impl Node {
let node_addr = match self.peers_by_index.get(&key) {
Some(id) => *id,
None => {
debug!(
trace!(
receiver_idx = %header.receiver_idx,
transport_id = %packet.transport_id,
"Unknown session index, dropping"
@@ -64,13 +64,16 @@ impl Node {
);
let peer = self.peers.get_mut(&node_addr).unwrap();
if let Some(_old_our_index) = peer.handle_peer_kbit_flip()
&& let (Some(transport_id), Some(new_our_index)) =
(peer.transport_id(), peer.our_index())
{
self.peers_by_index.insert(
(transport_id, new_our_index.as_u32()),
node_addr,
if let Some(_old_our_index) = peer.handle_peer_kbit_flip() {
// New index was pre-registered in peers_by_index during
// msg1 handling (handshake.rs). Verify, don't duplicate.
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(
&(peer.transport_id().unwrap(), peer.our_index().unwrap().as_u32())
),
"peers_by_index should contain pre-registered new index after K-bit flip"
);
}
}
@@ -218,7 +221,13 @@ impl Node {
consecutive_failures = count,
"Excessive decryption failures, removing peer"
);
let addr = *node_addr;
self.remove_active_peer(node_addr);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
}
}
}

View File

@@ -176,6 +176,11 @@ impl Node {
"Peer restart detected (epoch mismatch), removing stale session"
);
self.remove_active_peer(&peer_node_addr);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(peer_node_addr, now_ms);
// Fall through to process as new connection
}
_ => {
@@ -196,6 +201,55 @@ impl Node {
&& existing_peer.is_healthy()
&& session_age_secs >= 30
{
// Guard: already have a pending session from a completed
// rekey (waiting for K-bit cutover). Don't overwrite it
// with a new handshake — drop this msg1.
if existing_peer.pending_new_session().is_some() {
debug!(
peer = %self.peer_display_name(&peer_node_addr),
"Rekey msg1 received but already have pending session, dropping"
);
self.connections.remove(&link_id);
self.links.remove(&link_id);
self.msg1_rate_limiter.complete_handshake();
return;
}
// Dual-initiation detection: both sides sent msg1
// simultaneously. Apply tie-breaker — smaller NodeAddr
// wins as initiator (same as cross-connection resolution).
if existing_peer.rekey_in_progress() {
let our_addr = self.identity.node_addr();
if our_addr < &peer_node_addr {
// We win as initiator — drop their msg1.
// Our msg2 will arrive at peer, who completes
// as our responder.
debug!(
peer = %self.peer_display_name(&peer_node_addr),
"Dual rekey initiation: we win (smaller addr), dropping their msg1"
);
self.connections.remove(&link_id);
self.links.remove(&link_id);
self.msg1_rate_limiter.complete_handshake();
return;
}
// We lose — abandon our rekey, become responder below.
debug!(
peer = %self.peer_display_name(&peer_node_addr),
"Dual rekey initiation: we lose (larger addr), abandoning ours"
);
if let Some(peer) = self.peers.get_mut(&peer_node_addr)
&& let Some(idx) = peer.abandon_rekey()
{
if let Some(tid) = peer.transport_id() {
self.peers_by_index.remove(&(tid, idx.as_u32()));
self.pending_outbound.remove(&(tid, idx.as_u32()));
}
let _ = self.index_allocator.free(idx);
}
// Fall through to respond as responder
}
// Rekey: process as responder, store new session as pending
let noise_session = conn.take_session();
let our_new_index = match self.index_allocator.allocate() {
@@ -378,6 +432,7 @@ impl Node {
}
// Schedule filter announce (sent on next tick via debounce)
self.bloom_state.mark_update_needed(node_addr);
self.reset_discovery_backoff();
}
PromotionResult::CrossConnectionWon { loser_link_id, node_addr } => {
// Store msg2 on peer for resend on duplicate msg1
@@ -405,6 +460,7 @@ impl Node {
}
// Schedule filter announce (sent on next tick via debounce)
self.bloom_state.mark_update_needed(node_addr);
self.reset_discovery_backoff();
}
PromotionResult::CrossConnectionLost { winner_link_id } => {
// Close the losing TCP connection (no-op for connectionless)
@@ -537,6 +593,9 @@ impl Node {
"Rekey msg2 processing failed"
);
if let Some(idx) = peer.abandon_rekey() {
if let Some(tid) = peer.transport_id() {
self.peers_by_index.remove(&(tid, idx.as_u32()));
}
let _ = self.index_allocator.free(idx);
}
}
@@ -708,6 +767,7 @@ impl Node {
}
// Schedule filter announce (sent on next tick via debounce)
self.bloom_state.mark_update_needed(peer_node_addr);
self.reset_discovery_backoff();
return;
}
@@ -729,6 +789,7 @@ impl Node {
}
// Schedule filter announce (sent on next tick via debounce)
self.bloom_state.mark_update_needed(node_addr);
self.reset_discovery_backoff();
}
PromotionResult::CrossConnectionWon { loser_link_id, node_addr } => {
// Close the losing TCP connection (no-op for connectionless)
@@ -757,6 +818,7 @@ impl Node {
}
// Schedule filter announce (sent on next tick via debounce)
self.bloom_state.mark_update_needed(node_addr);
self.reset_discovery_backoff();
}
PromotionResult::CrossConnectionLost { winner_link_id } => {
// Close the losing TCP connection (no-op for connectionless)

View File

@@ -72,7 +72,7 @@ impl Node {
///
/// The peer is telling us about what they received from us. We feed
/// this to our metrics to compute RTT, loss rate, and trend indicators.
pub(in crate::node) fn handle_receiver_report(&mut self, from: &NodeAddr, payload: &[u8]) {
pub(in crate::node) async fn handle_receiver_report(&mut self, from: &NodeAddr, payload: &[u8]) {
let rr = match ReceiverReport::decode(payload) {
Ok(rr) => rr,
Err(e) => {
@@ -101,7 +101,7 @@ impl Node {
// Process the report: computes RTT from timestamp echo, updates
// loss rate, goodput rate, jitter trend, and ETX.
let now = Instant::now();
mmp.metrics.process_receiver_report(&rr, our_timestamp_ms, now);
let first_rtt = mmp.metrics.process_receiver_report(&rr, our_timestamp_ms, now);
// Feed SRTT back to sender/receiver report interval tuning
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -111,13 +111,10 @@ impl Node {
}
// Update reverse delivery ratio from our own receiver state
// (what fraction of peer's frames we received).
// (what fraction of peer's frames we received), using per-interval deltas.
let our_recv_packets = mmp.receiver.cumulative_packets_recv();
let peer_highest = mmp.receiver.highest_counter();
if peer_highest > 0 {
let reverse_ratio = (our_recv_packets as f64) / (peer_highest as f64);
mmp.metrics.set_delivery_ratio_reverse(reverse_ratio);
}
mmp.metrics.update_reverse_delivery(our_recv_packets, peer_highest);
trace!(
from = %peer_name,
@@ -126,6 +123,48 @@ impl Node {
etx = format_args!("{:.2}", mmp.metrics.etx),
"Processed ReceiverReport"
);
// First RTT sample — peer is now eligible for parent selection.
// Trigger re-evaluation so the node doesn't wait for the next
// periodic tick or TreeAnnounce.
if first_rtt {
let peer_costs: std::collections::HashMap<crate::NodeAddr, f64> = self.peers.iter()
.filter(|(_, p)| p.has_srtt())
.map(|(a, p)| (*a, p.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
if let Err(e) = self.tree_state.sign_declaration(&self.identity) {
warn!(error = %e, "Failed to sign declaration after first-RTT parent eval");
return;
}
self.tree_state.recompute_coords();
self.coord_cache.clear();
self.reset_discovery_backoff();
self.stats_mut().tree.parent_switched += 1;
self.stats_mut().tree.parent_switches += 1;
info!(
new_parent = %self.peer_display_name(&new_parent),
new_seq = new_seq,
new_root = %self.tree_state.root(),
depth = self.tree_state.my_coords().depth(),
trigger = "first-rtt",
"Parent switched after first RTT measurement"
);
if flap_dampened {
self.stats_mut().tree.flap_dampened += 1;
warn!("Flap dampening engaged: excessive parent switches detected");
}
self.send_tree_announce_to_all().await;
let all_peers: Vec<crate::NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
}
}
}
/// Check all peers for pending MMP reports and send them.

View File

@@ -1,6 +1,6 @@
//! RX event loop and message handlers.
mod discovery;
pub(crate) mod discovery;
mod dispatch;
mod encrypted;
mod forwarding;

View File

@@ -19,6 +19,10 @@ const DRAIN_WINDOW_SECS: u64 = 10;
/// a peer's rekey msg1.
const REKEY_DAMPENING_SECS: u64 = 30;
/// Delay FSP initiator cutover after handshake completion to allow
/// XK msg3 to reach the responder before K-bit-flipped data arrives.
const FSP_CUTOVER_DELAY_MS: u64 = 2000;
impl Node {
/// Periodic rekey check. Called from the tick loop.
///
@@ -79,14 +83,16 @@ impl Node {
if let Some(peer) = self.peers.get_mut(&node_addr)
&& let Some(_old_our_index) = peer.cutover_to_new_session()
{
if let (Some(transport_id), Some(new_our_index)) =
(peer.transport_id(), peer.our_index())
{
self.peers_by_index.insert(
(transport_id, new_our_index.as_u32()),
node_addr,
);
}
// New index was pre-registered in peers_by_index during
// msg2 handling (handshake.rs). Verify, don't duplicate.
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(
&(peer.transport_id().unwrap(), peer.our_index().unwrap().as_u32())
),
"peers_by_index should contain pre-registered new index after cutover"
);
info!(
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
@@ -281,10 +287,14 @@ impl Node {
continue;
}
// 1. Initiator-side cutover: completed rekey, pending session ready
// 1. Initiator-side cutover: completed rekey, pending session ready.
// Defer cutover until msg3 has had time to reach the responder.
// Without this delay, K-bit-flipped data can arrive before
// msg3, causing decryption failures on the responder.
if entry.pending_new_session().is_some()
&& !entry.has_rekey_in_progress()
&& entry.is_rekey_initiator()
&& now_ms.saturating_sub(entry.rekey_completed_ms()) >= FSP_CUTOVER_DELAY_MS
{
sessions_to_cutover.push(*node_addr);
continue;

View File

@@ -1,6 +1,6 @@
//! RX event loop and packet dispatch.
use crate::control::ControlSocket;
use crate::control::{commands, ControlSocket};
use crate::control::queries;
use crate::node::{Node, NodeError};
use crate::transport::ReceivedPacket;
@@ -98,7 +98,15 @@ impl Node {
self.register_identity(identity.node_addr, identity.pubkey);
}
Some((request, response_tx)) = control_rx.recv() => {
let response = queries::dispatch(self, &request.command);
let response = if request.command.starts_with("show_") {
queries::dispatch(self, &request.command)
} else {
commands::dispatch(
self,
&request.command,
request.params.as_ref(),
).await
};
let _ = response_tx.send(response);
}
_ = tick.tick() => {
@@ -107,6 +115,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.poll_pending_connects().await;
self.resend_pending_handshakes(now_ms).await;
self.resend_pending_rekeys(now_ms).await;
self.resend_pending_session_handshakes(now_ms).await;
@@ -120,7 +129,7 @@ impl Node {
self.check_link_heartbeats().await;
self.check_rekey().await;
self.check_session_rekey().await;
self.purge_stale_lookups(now_ms);
self.check_pending_lookups(now_ms).await;
self.poll_transport_discovery().await;
self.sample_transport_congestion();
}

View File

@@ -410,7 +410,37 @@ impl Node {
} else if existing.is_established() {
// Rekey: if rekey enabled, treat as rekey for key rotation.
// The existing established session remains active for traffic.
if self.config.node.rekey.enabled && !existing.has_rekey_in_progress() {
if self.config.node.rekey.enabled {
let rekey_in_progress = existing.has_rekey_in_progress();
let has_pending = existing.pending_new_session().is_some();
// Dual-initiation detection: both sides sent SessionSetup
// simultaneously. Apply tie-breaker — smaller NodeAddr
// wins as initiator (same as initial session setup).
if rekey_in_progress {
if self.identity.node_addr() < src_addr {
// We win as initiator — drop their msg1.
debug!(
src = %self.peer_display_name(src_addr),
"Dual FSP rekey initiation: we win (smaller addr), dropping their msg1"
);
return;
}
// We lose — abandon our rekey, become responder below.
debug!(
src = %self.peer_display_name(src_addr),
"Dual FSP rekey initiation: we lose (larger addr), abandoning ours"
);
let entry = self.sessions.get_mut(src_addr).unwrap();
entry.abandon_rekey();
} else if has_pending {
// Guard: already have a pending session waiting for K-bit cutover
debug!(
src = %self.peer_display_name(src_addr),
"FSP rekey msg1 received but already have pending session, dropping"
);
return;
}
let our_keypair = self.identity.keypair();
let mut handshake = HandshakeState::new_xk_responder(our_keypair);
handshake.set_local_epoch(self.startup_epoch);
@@ -595,6 +625,7 @@ impl Node {
};
entry.set_pending_session(session);
entry.set_rekey_completed_ms(Self::now_ms());
self.sessions.insert(*src_addr, entry);
debug!(
@@ -866,13 +897,10 @@ impl Node {
mmp.path_mtu.update_interval_from_srtt(srtt_ms);
}
// Update reverse delivery ratio from our own receiver state
// Update reverse delivery ratio from our own receiver state, using per-interval deltas.
let our_recv_packets = mmp.receiver.cumulative_packets_recv();
let peer_highest = mmp.receiver.highest_counter();
if peer_highest > 0 {
let reverse_ratio = (our_recv_packets as f64) / (peer_highest as f64);
mmp.metrics.set_delivery_ratio_reverse(reverse_ratio);
}
mmp.metrics.update_reverse_delivery(our_recv_packets, peer_highest);
trace!(
src = %peer_name,
@@ -1634,7 +1662,6 @@ impl Node {
/// misconfigured applications sending repeated oversized packets.
pub(in crate::node) fn send_icmpv6_packet_too_big(&mut self, original_packet: &[u8], mtu: u32) {
use crate::upper::icmp::build_packet_too_big;
use crate::FipsAddress;
use std::net::Ipv6Addr;
// Extract source address for rate limiting
@@ -1652,18 +1679,20 @@ impl Node {
return;
}
let our_ipv6 = FipsAddress::from_node_addr(self.node_addr()).to_ipv6();
if let Some(response) = build_packet_too_big(original_packet, mtu, our_ipv6)
// Use the original packet's *destination* as the ICMP source so the
// kernel sees the PTB coming from a remote router, not from itself.
// Linux ignores PTBs whose source matches a local address, which
// causes a PMTUD blackhole when both src and ICMP-src are local.
let dest_addr = Ipv6Addr::from(<[u8; 16]>::try_from(&original_packet[24..40]).unwrap());
if let Some(response) = build_packet_too_big(original_packet, mtu, dest_addr)
&& let Some(tun_tx) = &self.tun_tx
{
debug!(
src = %src_addr,
dst = %our_ipv6,
original_src = %src_addr,
original_dst = %dest_addr,
packet_size = original_packet.len(),
reported_mtu = mtu,
"Sending ICMP Packet Too Big (ICMP src={}, dst={})",
our_ipv6,
src_addr
"Sending ICMP Packet Too Big"
);
let _ = tun_tx.send(response);
}

View File

@@ -51,7 +51,7 @@ impl Node {
if conn.is_outbound()
&& let Some(identity) = conn.expected_identity()
{
self.schedule_retry(*identity.node_addr(), now_ms, false);
self.schedule_retry(*identity.node_addr(), now_ms);
}
}
self.cleanup_stale_connection(link_id, now_ms);

View File

@@ -3,7 +3,7 @@
use super::{Node, NodeError, NodeState};
use crate::peer::PeerConnection;
use crate::protocol::{Disconnect, DisconnectReason};
use crate::transport::{packet_channel, Link, LinkDirection, TransportAddr, TransportId};
use crate::transport::{packet_channel, Link, LinkDirection, LinkId, TransportAddr, TransportId};
use crate::upper::tun::{run_tun_reader, shutdown_tun_interface, TunDevice, TunState};
use crate::node::wire::build_msg1;
use crate::{NodeAddr, PeerIdentity};
@@ -17,15 +17,29 @@ impl Node {
/// For each peer configured with AutoConnect policy, creates a link and
/// peer entry, then starts the Noise handshake by sending the first message.
pub(super) async fn initiate_peer_connections(&mut self) {
// Build display name map from all configured peers (alias or short npub)
for peer_config in self.config.peers() {
if let Ok(identity) = PeerIdentity::from_npub(&peer_config.npub) {
let name = peer_config
.alias
.clone()
.unwrap_or_else(|| identity.short_npub());
self.peer_aliases.insert(*identity.node_addr(), name);
}
// Build display name map from all configured peers (alias or short npub),
// and pre-seed the identity cache from each peer's npub so that TUN packets
// addressed to a configured peer can be dispatched (and trigger session
// initiation) immediately on startup — without waiting for the link-layer
// handshake to complete first.
let peer_identities: Vec<(PeerIdentity, Option<String>)> = self
.config
.peers()
.iter()
.filter_map(|pc| {
PeerIdentity::from_npub(&pc.npub)
.ok()
.map(|id| (id, pc.alias.clone()))
})
.collect();
for (identity, alias) in peer_identities {
let name = alias.unwrap_or_else(|| identity.short_npub());
self.peer_aliases.insert(*identity.node_addr(), name);
// Pre-seed identity cache. The parity may be wrong (npub is x-only)
// but will be corrected to the real value when the peer is promoted
// after a successful Noise handshake.
self.register_identity(*identity.node_addr(), identity.pubkey_full());
}
// Collect peer configs to avoid borrow conflicts
@@ -143,9 +157,14 @@ impl Node {
/// Initiate a connection to a peer on a specific transport and address.
///
/// Allocates a link, starts the Noise IK handshake, sends msg1, and
/// registers the connection for msg2 dispatch. Used by both static peer
/// config and transport discovery auto-connect paths.
/// For connectionless transports (UDP, Ethernet): allocates a link, starts
/// the Noise IK handshake, sends msg1, and registers the connection for
/// msg2 dispatch.
///
/// For connection-oriented transports (TCP, Tor): allocates a link and
/// starts a non-blocking transport connect. The handshake is deferred
/// until the transport connection is established — the tick handler
/// polls `connection_state()` and initiates the handshake when ready.
pub(super) async fn initiate_connection(
&mut self,
transport_id: TransportId,
@@ -154,15 +173,14 @@ impl Node {
) -> Result<(), NodeError> {
let peer_node_addr = *peer_identity.node_addr();
let is_connection_oriented = self.transports.get(&transport_id)
.map(|t| t.transport_type().connection_oriented)
.unwrap_or(false);
// Allocate link ID and create link
let link_id = self.allocate_link_id();
// Use Link::new() for connection-oriented transports (Connecting state)
// and Link::connectionless() for connectionless transports (Connected state)
let link = if self.transports.get(&transport_id)
.map(|t| t.transport_type().connection_oriented)
.unwrap_or(false)
{
let link = if is_connection_oriented {
Link::new(
link_id,
transport_id,
@@ -186,6 +204,53 @@ impl Node {
self.addr_to_link
.insert((transport_id, remote_addr.clone()), link_id);
if is_connection_oriented {
// Connection-oriented: start non-blocking connect, defer handshake
if let Some(transport) = self.transports.get(&transport_id) {
match transport.connect(&remote_addr).await {
Ok(()) => {
debug!(
peer = %self.peer_display_name(&peer_node_addr),
transport_id = %transport_id,
remote_addr = %remote_addr,
link_id = %link_id,
"Transport connect initiated (non-blocking)"
);
self.pending_connects.push(super::PendingConnect {
link_id,
transport_id,
remote_addr,
peer_identity,
});
}
Err(e) => {
// Clean up link
self.links.remove(&link_id);
self.addr_to_link.remove(&(transport_id, remote_addr));
return Err(NodeError::TransportError(e.to_string()));
}
}
}
Ok(())
} else {
// Connectionless: proceed with immediate handshake
self.start_handshake(link_id, transport_id, remote_addr, peer_identity).await
}
}
/// Start the Noise handshake on a link and send msg1.
///
/// Called immediately for connectionless transports, or after the
/// transport connection is established for connection-oriented transports.
pub(super) async fn start_handshake(
&mut self,
link_id: LinkId,
transport_id: TransportId,
remote_addr: TransportAddr,
peer_identity: PeerIdentity,
) -> Result<(), NodeError> {
let peer_node_addr = *peer_identity.node_addr();
// Create connection in handshake phase (outbound knows expected identity)
let current_time_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
@@ -336,6 +401,99 @@ impl Node {
}
}
/// Poll pending transport connects and initiate handshakes for ready ones.
///
/// Called from the tick handler. For each pending connect, queries the
/// transport's connection state. When a connection is established,
/// marks the link as Connected and starts the Noise handshake.
/// Failed connections are cleaned up and scheduled for retry.
pub(super) async fn poll_pending_connects(&mut self) {
if self.pending_connects.is_empty() {
return;
}
let mut completed = Vec::new();
for (i, pending) in self.pending_connects.iter().enumerate() {
let state = if let Some(transport) = self.transports.get(&pending.transport_id) {
transport.connection_state(&pending.remote_addr)
} else {
crate::transport::ConnectionState::Failed("transport removed".into())
};
match state {
crate::transport::ConnectionState::Connected => {
completed.push((i, true, None));
}
crate::transport::ConnectionState::Failed(reason) => {
completed.push((i, false, Some(reason)));
}
crate::transport::ConnectionState::Connecting => {
// Still in progress, check on next tick
}
crate::transport::ConnectionState::None => {
// Shouldn't happen — treat as failure
completed.push((i, false, Some("no connection attempt found".into())));
}
}
}
// Process completions in reverse order to preserve indices
for (i, success, reason) in completed.into_iter().rev() {
let pending = self.pending_connects.remove(i);
if success {
// Mark link as Connected
if let Some(link) = self.links.get_mut(&pending.link_id) {
link.set_connected();
}
debug!(
peer = %self.peer_display_name(pending.peer_identity.node_addr()),
transport_id = %pending.transport_id,
remote_addr = %pending.remote_addr,
link_id = %pending.link_id,
"Transport connected, starting handshake"
);
// Start the handshake now that the transport is connected
if let Err(e) = self.start_handshake(
pending.link_id,
pending.transport_id,
pending.remote_addr.clone(),
pending.peer_identity,
).await {
warn!(
link_id = %pending.link_id,
error = %e,
"Failed to start handshake after transport connect"
);
// Clean up link on handshake failure
self.remove_link(&pending.link_id);
}
} else {
let reason = reason.unwrap_or_default();
warn!(
peer = %self.peer_display_name(pending.peer_identity.node_addr()),
transport_id = %pending.transport_id,
remote_addr = %pending.remote_addr,
link_id = %pending.link_id,
reason = %reason,
"Transport connect failed"
);
// Clean up link and schedule retry
self.remove_link(&pending.link_id);
self.links.remove(&pending.link_id);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_retry(*pending.peer_identity.node_addr(), now_ms);
}
}
}
// === State Transitions ===
/// Start the node.
@@ -581,4 +739,76 @@ impl Node {
info!(sent, total = peer_addrs.len(), reason = %reason, "Sent disconnect notifications");
}
// === Control API methods ===
/// Connect to a peer via the control API.
///
/// Creates an ephemeral peer connection (not persisted to config, no
/// auto-reconnect). Reuses the same connection path as auto-connect
/// peers. Returns JSON data on success or an error message.
pub(crate) async fn api_connect(
&mut self,
npub: &str,
address: &str,
transport: &str,
) -> Result<serde_json::Value, String> {
let peer_config = crate::config::PeerConfig {
npub: npub.to_string(),
alias: None,
addresses: vec![crate::config::PeerAddress::new(transport, address)],
connect_policy: crate::config::ConnectPolicy::Manual,
auto_reconnect: false,
};
// Pre-seed identity cache (same as initiate_peer_connections does)
if let Ok(identity) = PeerIdentity::from_npub(npub) {
self.peer_aliases
.insert(*identity.node_addr(), identity.short_npub());
self.register_identity(*identity.node_addr(), identity.pubkey_full());
}
self.initiate_peer_connection(&peer_config)
.await
.map(|()| {
info!(
npub = %npub,
address = %address,
transport = %transport,
"API connect initiated"
);
serde_json::json!({
"npub": npub,
"address": address,
"transport": transport,
})
})
.map_err(|e| e.to_string())
}
/// Disconnect a peer via the control API.
///
/// Removes the peer and suppresses auto-reconnect.
pub(crate) fn api_disconnect(&mut self, npub: &str) -> Result<serde_json::Value, String> {
let peer_identity = PeerIdentity::from_npub(npub)
.map_err(|e| format!("invalid npub '{npub}': {e}"))?;
let node_addr = *peer_identity.node_addr();
if !self.peers.contains_key(&node_addr) {
return Err(format!("peer not found: {npub}"));
}
// Remove the peer (full cleanup: sessions, indices, links, tree, bloom)
self.remove_active_peer(&node_addr);
// Suppress any pending auto-reconnect
self.retry_pending.remove(&node_addr);
info!(npub = %npub, "API disconnect completed");
Ok(serde_json::json!({
"npub": npub,
"disconnected": true,
}))
}
}

View File

@@ -8,6 +8,7 @@ mod bloom;
mod handlers;
mod lifecycle;
mod retry;
mod discovery_rate_limit;
mod rate_limit;
mod routing_error_rate_limit;
pub(crate) mod session;
@@ -23,6 +24,7 @@ use crate::cache::CoordCache;
use crate::utils::index::IndexAllocator;
use crate::node::session::SessionEntry;
use crate::peer::{ActivePeer, PeerConnection};
use self::discovery_rate_limit::{DiscoveryBackoff, DiscoveryForwardRateLimiter};
use self::rate_limit::HandshakeRateLimiter;
use self::routing_error_rate_limit::RoutingErrorRateLimiter;
use crate::transport::{
@@ -30,6 +32,7 @@ use crate::transport::{
};
use crate::transport::udp::UdpTransport;
use crate::transport::tcp::TcpTransport;
use crate::transport::tor::TorTransport;
#[cfg(target_os = "linux")]
use crate::transport::ethernet::EthernetTransport;
use crate::tree::TreeState;
@@ -119,6 +122,9 @@ pub enum NodeError {
#[error("handshake failed: {0}")]
HandshakeFailed(String),
#[error("transport error: {0}")]
TransportError(String),
}
/// Node operational state.
@@ -171,12 +177,17 @@ impl fmt::Display for NodeState {
/// When a LookupRequest is forwarded through a node, the node stores the
/// request_id and which peer sent it. When the corresponding LookupResponse
/// arrives, it's forwarded back to that peer (reverse-path forwarding).
/// The `response_forwarded` flag prevents response routing loops.
#[derive(Clone, Debug)]
pub(crate) struct RecentRequest {
/// The peer who sent this request to us.
pub(crate) from_peer: NodeAddr,
/// When we received this request (Unix milliseconds).
pub(crate) timestamp_ms: u64,
/// Whether we've already forwarded a response for this request.
/// Prevents response routing loops when convergent request paths
/// create bidirectional entries in recent_requests.
pub(crate) response_forwarded: bool,
}
impl RecentRequest {
@@ -184,6 +195,7 @@ impl RecentRequest {
Self {
from_peer,
timestamp_ms,
response_forwarded: false,
}
}
@@ -208,6 +220,22 @@ struct TransportDropState {
dropping: bool,
}
/// State for a link waiting for transport-level connection establishment.
///
/// For connection-oriented transports (TCP, Tor), the transport connect runs
/// asynchronously. This struct holds the data needed to complete the handshake
/// once the connection is ready.
struct PendingConnect {
/// The link that was created for this connection.
link_id: LinkId,
/// Which transport is being used.
transport_id: TransportId,
/// The remote address being connected to.
remote_addr: TransportAddr,
/// The peer identity (for handshake initiation).
peer_identity: PeerIdentity,
}
/// A running FIPS node instance.
///
/// This is the top-level container holding all node state.
@@ -303,7 +331,7 @@ pub struct Node {
// === Pending Discovery Lookups ===
/// Tracks in-flight discovery lookups. Maps target NodeAddr to the
/// initiation timestamp (Unix ms). Prevents duplicate flood queries.
pending_lookups: HashMap<NodeAddr, u64>,
pending_lookups: HashMap<NodeAddr, handlers::discovery::PendingLookup>,
// === Resource Limits ===
/// Maximum connections (0 = unlimited).
@@ -362,6 +390,17 @@ pub struct Node {
routing_error_rate_limiter: RoutingErrorRateLimiter,
/// Rate limiter for source-side CoordsRequired/PathBroken responses.
coords_response_rate_limiter: RoutingErrorRateLimiter,
/// Backoff for failed discovery lookups (originator-side).
discovery_backoff: DiscoveryBackoff,
/// Rate limiter for forwarded discovery requests (transit-side).
discovery_forward_limiter: DiscoveryForwardRateLimiter,
// === Pending Transport Connects ===
/// Links waiting for transport-level connection establishment before
/// sending handshake msg1. For connection-oriented transports (TCP, Tor),
/// the transport connect runs in the background; the tick handler polls
/// connection_state() and initiates the handshake when connected.
pending_connects: Vec<PendingConnect>,
// === Connection Retry ===
/// Retry state for peers whose outbound connections have failed.
@@ -445,6 +484,9 @@ impl Node {
let max_peers = config.node.limits.max_peers;
let max_links = config.node.limits.max_links;
let coords_response_interval_ms = config.node.session.coords_response_interval_ms;
let backoff_base_secs = config.node.discovery.backoff_base_secs;
let backoff_max_secs = config.node.discovery.backoff_max_secs;
let forward_min_interval_secs = config.node.discovery.forward_min_interval_secs;
let mut host_map = HostMap::from_peer_configs(config.peers());
let hosts_file = HostMap::load_hosts_file(std::path::Path::new(
@@ -499,6 +541,14 @@ impl Node {
coords_response_rate_limiter: RoutingErrorRateLimiter::with_interval(
std::time::Duration::from_millis(coords_response_interval_ms),
),
discovery_backoff: DiscoveryBackoff::with_params(
backoff_base_secs,
backoff_max_secs,
),
discovery_forward_limiter: DiscoveryForwardRateLimiter::with_interval(
std::time::Duration::from_secs(forward_min_interval_secs),
),
pending_connects: Vec::new(),
retry_pending: HashMap::new(),
last_parent_reeval: None,
last_congestion_log: None,
@@ -601,6 +651,9 @@ impl Node {
coords_response_rate_limiter: RoutingErrorRateLimiter::with_interval(
std::time::Duration::from_millis(coords_response_interval_ms),
),
discovery_backoff: DiscoveryBackoff::new(),
discovery_forward_limiter: DiscoveryForwardRateLimiter::new(),
pending_connects: Vec::new(),
retry_pending: HashMap::new(),
last_parent_reeval: None,
last_congestion_log: None,
@@ -676,6 +729,21 @@ impl Node {
transports.push(TransportHandle::Tcp(tcp));
}
// Create Tor transport instances
let tor_instances: Vec<_> = self
.config
.transports
.tor
.iter()
.map(|(name, config)| (name.map(|s| s.to_string()), config.clone()))
.collect();
for (name, tor_config) in tor_instances {
let transport_id = self.allocate_transport_id();
let tor = TorTransport::new(transport_id, name, tor_config, packet_tx.clone());
transports.push(TransportHandle::Tor(tor));
}
transports
}
@@ -1178,6 +1246,13 @@ impl Node {
// === End-to-End Sessions ===
/// Get a session by remote NodeAddr.
/// Disable the discovery forward rate limiter (for tests).
#[cfg(test)]
pub(crate) fn disable_discovery_forward_rate_limit(&mut self) {
self.discovery_forward_limiter
.set_interval(std::time::Duration::ZERO);
}
#[cfg(test)]
pub(crate) fn get_session(&self, remote: &NodeAddr) -> Option<&SessionEntry> {
self.sessions.get(remote)

View File

@@ -59,11 +59,10 @@ impl Node {
&mut self,
node_addr: NodeAddr,
now_ms: u64,
reconnect: bool,
) {
let retry_cfg = &self.config.node.retry;
let max_retries = retry_cfg.max_retries;
if max_retries == 0 && !reconnect {
if max_retries == 0 {
return;
}
@@ -112,12 +111,11 @@ impl Node {
if let Some(pc) = peer_config {
let mut state = RetryState::new(pc);
state.retry_count = 1;
state.reconnect = reconnect;
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %self.peer_display_name(&node_addr),
reconnect = reconnect,
delay_secs = delay / 1000,
"First connection attempt failed, scheduling retry"
);
@@ -131,6 +129,12 @@ impl Node {
///
/// Looks up the peer in auto-connect config and checks `auto_reconnect`.
/// If enabled, feeds the peer into the retry system with unlimited retries.
///
/// If a retry entry already exists (e.g. from a previous failed handshake
/// attempt during an earlier reconnect cycle), the existing retry count is
/// preserved and incremented rather than reset to zero. This ensures
/// exponential backoff accumulates across repeated link-dead events instead
/// of resetting to the base interval on every peer removal.
pub(super) fn schedule_reconnect(&mut self, node_addr: NodeAddr, now_ms: u64) {
// Find peer in auto-connect config
let peer_config = self
@@ -157,6 +161,24 @@ impl Node {
let base_interval_ms = self.config.node.retry.base_interval_secs * 1000;
let max_backoff_ms = self.config.node.retry.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
// If we already have accumulated backoff from previous failed attempts,
// preserve and bump it rather than resetting to zero. This prevents the
// exponential backoff from being discarded on each link-dead cycle.
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.reconnect = true;
state.retry_count += 1;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
info!(
peer = %peer_name,
retry = state.retry_count,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal (backoff preserved)"
);
return;
}
let mut state = RetryState::new(pc);
state.reconnect = true;
@@ -164,7 +186,7 @@ impl Node {
state.retry_after_ms = now_ms + delay;
info!(
peer = %self.peer_display_name(&node_addr),
peer = %peer_name,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal"
);
@@ -237,7 +259,7 @@ impl Node {
);
// Immediate failure counts as an attempt — schedule next retry
// (reconnect flag is preserved on existing retry_pending entry)
self.schedule_retry(node_addr, now_ms, false);
self.schedule_retry(node_addr, now_ms);
}
}
}

View File

@@ -5,6 +5,8 @@
//! (SessionSetup/SessionAck/SessionMsg3) carried inside SessionDatagram
//! envelopes through the mesh.
use std::time::Instant;
use crate::config::SessionMmpConfig;
use crate::mmp::MmpSessionState;
use crate::noise::{HandshakeState, NoiseSession};
@@ -107,6 +109,9 @@ pub(crate) struct SessionEntry {
rekey_initiator: bool,
/// Dampening: last time peer sent us a rekey msg1 (Unix ms).
last_peer_rekey_ms: u64,
/// When the FSP rekey handshake completed (initiator sent msg3, Unix ms).
/// Used to defer cutover until msg3 has time to reach the responder.
rekey_completed_ms: u64,
}
impl SessionEntry {
@@ -142,6 +147,7 @@ impl SessionEntry {
pending_new_session: None,
rekey_initiator: false,
last_peer_rekey_ms: 0,
rekey_completed_ms: 0,
}
}
@@ -369,6 +375,16 @@ impl SessionEntry {
}
}
/// When the FSP rekey handshake completed (initiator sent msg3).
pub(crate) fn rekey_completed_ms(&self) -> u64 {
self.rekey_completed_ms
}
/// Record when the FSP rekey handshake completed (initiator side).
pub(crate) fn set_rekey_completed_ms(&mut self, ms: u64) {
self.rekey_completed_ms = ms;
}
/// Store a completed rekey session.
pub(crate) fn set_pending_session(&mut self, session: NoiseSession) {
self.pending_new_session = Some(session);
@@ -408,6 +424,13 @@ impl SessionEntry {
self.session_start_ms = now_ms;
self.rekey_state = None;
self.rekey_initiator = false;
self.rekey_completed_ms = 0;
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
}
true
}
@@ -430,6 +453,12 @@ impl SessionEntry {
self.session_start_ms = now_ms;
self.rekey_state = None;
self.rekey_initiator = false;
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
}
true
}

View File

@@ -107,12 +107,16 @@ pub struct DiscoveryStats {
pub req_received: u64,
pub req_decode_error: u64,
pub req_duplicate: u64,
pub req_already_visited: u64,
pub req_target_is_us: u64,
pub req_forwarded: u64,
pub req_ttl_exhausted: u64,
pub req_initiated: u64,
pub req_deduplicated: u64,
pub req_backoff_suppressed: u64,
pub req_forward_rate_limited: u64,
pub req_bloom_miss: u64,
pub req_no_tree_peer: u64,
pub req_fallback_forwarded: u64,
// Response counters
pub resp_received: u64,
pub resp_decode_error: u64,
@@ -129,12 +133,16 @@ impl DiscoveryStats {
req_received: self.req_received,
req_decode_error: self.req_decode_error,
req_duplicate: self.req_duplicate,
req_already_visited: self.req_already_visited,
req_target_is_us: self.req_target_is_us,
req_forwarded: self.req_forwarded,
req_ttl_exhausted: self.req_ttl_exhausted,
req_initiated: self.req_initiated,
req_deduplicated: self.req_deduplicated,
req_backoff_suppressed: self.req_backoff_suppressed,
req_forward_rate_limited: self.req_forward_rate_limited,
req_bloom_miss: self.req_bloom_miss,
req_no_tree_peer: self.req_no_tree_peer,
req_fallback_forwarded: self.req_fallback_forwarded,
resp_received: self.resp_received,
resp_decode_error: self.resp_decode_error,
resp_forwarded: self.resp_forwarded,
@@ -342,12 +350,16 @@ pub struct DiscoveryStatsSnapshot {
pub req_received: u64,
pub req_decode_error: u64,
pub req_duplicate: u64,
pub req_already_visited: u64,
pub req_target_is_us: u64,
pub req_forwarded: u64,
pub req_ttl_exhausted: u64,
pub req_initiated: u64,
pub req_deduplicated: u64,
pub req_backoff_suppressed: u64,
pub req_forward_rate_limited: u64,
pub req_bloom_miss: u64,
pub req_no_tree_peer: u64,
pub req_fallback_forwarded: u64,
pub resp_received: u64,
pub resp_decode_error: u64,
pub resp_forwarded: u64,

View File

@@ -228,6 +228,94 @@ async fn test_disconnect_chain_partition() {
cleanup_nodes(&mut nodes).await;
}
/// Removing a peer via disconnect must also remove the associated end-to-end session.
///
/// Regression test for issue #5: `remove_active_peer` previously left the
/// `SessionEntry` alive in `self.sessions` after evicting the peer from
/// `self.peers`. This caused:
/// 1. Stale "MMP session metrics" logs with frozen counters until
/// `purge_idle_sessions` eventually fired (up to idle_timeout_secs later).
/// 2. `initiate_session` silently returning `Ok(())` on the stale Established
/// entry's guard check, preventing a new session from being created even
/// after the link layer reconnected successfully.
#[tokio::test]
async fn test_disconnect_clears_session() {
use crate::identity::Identity;
use crate::node::session::{EndToEndState, SessionEntry};
use crate::noise::HandshakeState;
// Two-node topology: 0 -- 1.
let edges = vec![(0, 1)];
let mut nodes = run_tree_test(2, &edges, false).await;
verify_tree_convergence(&nodes);
let node0_addr = *nodes[0].node.node_addr();
let node1_addr = *nodes[1].node.node_addr();
// Inject a synthetic Established session entry into node 1's session table
// to simulate the state after a completed XK handshake with node 0.
let remote_identity = Identity::generate();
{
let our_identity = nodes[1].node.identity();
let mut initiator = HandshakeState::new_initiator(
our_identity.keypair(),
remote_identity.pubkey_full(),
);
let mut responder = HandshakeState::new_responder(remote_identity.keypair());
let mut init_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut init_epoch);
initiator.set_local_epoch(init_epoch);
let mut resp_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut resp_epoch);
responder.set_local_epoch(resp_epoch);
let msg1 = initiator.write_message_1().unwrap();
responder.read_message_1(&msg1).unwrap();
let msg2 = responder.write_message_2().unwrap();
initiator.read_message_2(&msg2).unwrap();
let session = initiator.into_session().unwrap();
let entry = SessionEntry::new(
node0_addr,
remote_identity.pubkey_full(),
EndToEndState::Established(session),
1_000,
true,
);
nodes[1].node.sessions.insert(node0_addr, entry);
}
assert_eq!(nodes[1].node.session_count(), 1, "Session should exist before disconnect");
assert_eq!(nodes[1].node.peer_count(), 1, "Peer should exist before disconnect");
// Node 0 sends Disconnect to node 1.
let disconnect = crate::protocol::Disconnect::new(DisconnectReason::Shutdown);
nodes[0]
.node
.send_encrypted_link_message(&node1_addr, &disconnect.encode())
.await
.expect("Failed to send disconnect");
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(&mut nodes).await;
// Peer must be gone.
assert_eq!(
nodes[1].node.peer_count(), 0,
"Peer should be removed after disconnect"
);
// Session must also be gone — core regression check for issue #5.
// Before the fix, session_count() would still be 1 here because
// remove_active_peer didn't remove self.sessions[node0_addr].
assert_eq!(
nodes[1].node.session_count(), 0,
"Session must be cleaned up when peer is removed (regression: issue #5)"
);
cleanup_nodes(&mut nodes).await;
}
/// Verify that different disconnect reasons are handled correctly.
///
/// Sends each reason code and verifies the peer is removed regardless.

View File

@@ -1,8 +1,8 @@
//! Discovery protocol tests: LookupRequest and LookupResponse.
//!
//! Unit tests for handler logic (dedup, visited filter, TTL, response
//! caching) and integration tests for multi-node forwarding and
//! reverse-path response routing.
//! Unit tests for handler logic (dedup, TTL, response caching) and
//! integration tests for multi-node forwarding and reverse-path
//! response routing.
use super::*;
use crate::node::RecentRequest;
@@ -46,26 +46,6 @@ async fn test_request_dedup() {
assert_eq!(node.recent_requests.len(), 1);
}
#[tokio::test]
async fn test_request_visited_filter_self() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let target = make_node_addr(0xBB);
let origin = make_node_addr(0xCC);
let coords = TreeCoordinate::from_addrs(vec![origin, make_node_addr(0)]).unwrap();
let mut request = LookupRequest::new(888, target, origin, coords, 5, 0);
// Mark ourselves as already visited
request.visited.insert(node.node_addr());
let payload = &request.encode()[1..];
node.handle_lookup_request(&from, payload).await;
// Request was recorded (dedup happens before visited check)
// but the handler should have stopped after detecting self in visited filter
assert!(node.recent_requests.contains_key(&888));
}
#[tokio::test]
async fn test_request_target_is_self() {
let mut node = make_node();
@@ -379,13 +359,13 @@ async fn test_recent_request_expiry() {
#[tokio::test]
async fn test_request_forwarding_two_node() {
// Set up a two-node topology: node0 — node1
// Send a LookupRequest from node0 targeting some unknown node.
// Send a LookupRequest from node0 targeting node1's address.
// Node1 should receive the forwarded request.
let edges = vec![(0, 1)];
let mut nodes = run_tree_test(2, &edges, false).await;
let node0_addr = *nodes[0].node.node_addr();
let target = make_node_addr(0xEE); // unknown node
let target = *nodes[1].node.node_addr(); // target node1 (in bloom filters)
let root = make_node_addr(0);
let coords = TreeCoordinate::from_addrs(vec![node0_addr, root]).unwrap();
@@ -499,20 +479,22 @@ async fn test_request_three_node_chain() {
#[tokio::test]
async fn test_request_dedup_convergent_paths() {
// Topology: triangle (node0 — node1, node0 — node2, node1 — node2)
// A request from node0 reaches node2 via two paths: 0→1→2 and 0→2.
// The second arrival at node2 should be deduped.
// A request from node0 targeting node2 may reach it via two paths
// depending on bloom filter state. If both paths deliver the request,
// the second arrival at node2 should be deduped.
let edges = vec![(0, 1), (0, 2), (1, 2)];
let mut nodes = run_tree_test(3, &edges, false).await;
let node0_addr = *nodes[0].node.node_addr();
let target = make_node_addr(0xEE);
let target = *nodes[2].node.node_addr(); // target node2 (in bloom filters)
let root = make_node_addr(0);
let coords = TreeCoordinate::from_addrs(vec![node0_addr, root]).unwrap();
let request = LookupRequest::new(300, target, node0_addr, coords, 5, 0);
let payload = &request.encode()[1..];
// Node0 handles the request (forwards to both node1 and node2)
// Node0 handles the request (forwards to peers whose bloom filter
// contains node2 — bloom-guided, not flooding)
nodes[0]
.node
.handle_lookup_request(&node0_addr, payload)
@@ -524,12 +506,16 @@ async fn test_request_dedup_convergent_paths() {
process_available_packets(&mut nodes).await;
}
// Both node1 and node2 should have recorded the request
assert!(nodes[1].node.recent_requests.contains_key(&300));
assert!(nodes[2].node.recent_requests.contains_key(&300));
// Node2 (the target) must have received the request
assert!(
nodes[2].node.recent_requests.contains_key(&300),
"Node 2 (target) should have received the request"
);
// The request should appear exactly once in each node's recent_requests
// (dedup prevents duplicate processing via convergent paths)
// If node1 also received and forwarded it, node2 would have seen a
// duplicate — verify dedup counter reflects convergent arrivals.
// With bloom-guided routing, node1 may or may not receive the request
// depending on filter state, so we only assert the target received it.
cleanup_nodes(&mut nodes).await;
}
@@ -547,11 +533,18 @@ async fn test_discovery_100_nodes() {
const NUM_NODES: usize = 100;
const TARGET_EDGES: usize = 250;
const SEED: u64 = 42;
const TTL: u8 = 15; // generous TTL for network diameter
const TTL: u8 = 20; // must exceed tree diameter (can reach 17+ hops)
let edges = generate_random_edges(NUM_NODES, TARGET_EDGES, SEED);
let mut nodes = run_tree_test(NUM_NODES, &edges, false).await;
verify_tree_convergence(&nodes);
// Disable forward rate limiting: in this test all 100 nodes look up
// the same 10 targets in <1s wall time. The 2s per-target rate limit
// would suppress nearly all transit forwarding.
for tn in nodes.iter_mut() {
tn.node.disable_discovery_forward_rate_limit();
}
// Collect all node addresses and public keys for lookup targets
let all_addrs: Vec<NodeAddr> = nodes
.iter()
@@ -588,9 +581,8 @@ async fn test_discovery_100_nodes() {
let total_lookups = lookup_pairs.len();
// Process one source node at a time. Each node initiates ~10 lookups,
// which flood through the network. We drain until quiescent before
// moving to the next node. This avoids overwhelming UDP buffers
// while still testing concurrent lookups from the same origin.
// which route through the tree via bloom filters. We drain until
// quiescent before moving to the next node.
for src in 0..NUM_NODES {
// Initiate all lookups for this source node
let mut initiated = false;
@@ -607,14 +599,17 @@ async fn test_discovery_100_nodes() {
continue;
}
// Drain packets until quiescent
// Drain packets until quiescent. With single-path tree routing,
// a packet forwarded by node X may land in node Y's queue where
// Y < X in iteration order, causing a zero-count round even though
// packets are in flight. Use a higher idle threshold to handle this.
let mut idle_rounds = 0;
for _ in 0..40 {
tokio::time::sleep(Duration::from_millis(10)).await;
for _ in 0..80 {
tokio::time::sleep(Duration::from_millis(5)).await;
let count = process_available_packets(&mut nodes).await;
if count == 0 {
idle_rounds += 1;
if idle_rounds >= 2 {
if idle_rounds >= 5 {
break;
}
} else {

View File

@@ -1779,6 +1779,14 @@ async fn test_tun_outbound_path_mtu_generates_ptb() {
assert_eq!(ptb[40], 2, "ICMPv6 type should be Packet Too Big (2)");
assert_eq!(ptb[41], 0, "ICMPv6 code should be 0");
// Verify PTB source is the *remote peer* (original packet's destination),
// NOT the local node. Linux ignores PTBs whose source matches a local
// address, causing a PMTUD blackhole.
let ptb_src = std::net::Ipv6Addr::from(<[u8; 16]>::try_from(&ptb[8..24]).unwrap());
let ptb_dst = std::net::Ipv6Addr::from(<[u8; 16]>::try_from(&ptb[24..40]).unwrap());
assert_eq!(ptb_src, dst_fips.to_ipv6(), "PTB source must be remote peer (original dst), not local node");
assert_eq!(ptb_dst, src_fips.to_ipv6(), "PTB destination must be local node (original src)");
// Verify reported MTU (32-bit field at ICMPv6 header bytes 4-7)
let reported_mtu = u32::from_be_bytes([ptb[44], ptb[45], ptb[46], ptb[47]]);
assert_eq!(reported_mtu, reduced_ipv6_mtu as u32, "Reported MTU should match path IPv6 MTU");
@@ -1904,6 +1912,14 @@ async fn test_multihop_pmtud_heterogeneous_mtu() {
assert_eq!(ptb[40], 2, "ICMPv6 type should be Packet Too Big (2)");
assert_eq!(ptb[41], 0, "ICMPv6 code should be 0");
// Verify PTB source is the *remote peer* (original packet's destination),
// NOT the local node. Linux ignores PTBs whose source matches a local
// address, causing a PMTUD blackhole.
let ptb_src = std::net::Ipv6Addr::from(<[u8; 16]>::try_from(&ptb[8..24]).unwrap());
let ptb_dst = std::net::Ipv6Addr::from(<[u8; 16]>::try_from(&ptb[24..40]).unwrap());
assert_eq!(ptb_src, dst_fips.to_ipv6(), "PTB source must be remote peer (original dst), not local node");
assert_eq!(ptb_dst, src_fips.to_ipv6(), "PTB destination must be local node (original src)");
// Verify reported MTU is the path MTU (not local MTU)
let reported_mtu = u32::from_be_bytes([ptb[44], ptb[45], ptb[46], ptb[47]]);
let expected_ipv6_mtu = crate::upper::icmp::effective_ipv6_mtu(path_mtu) as u32;

View File

@@ -568,11 +568,12 @@ fn test_schedule_retry_creates_entry() {
assert!(node.retry_pending.is_empty());
node.schedule_retry(peer_node_addr, 1000, false);
node.schedule_retry(peer_node_addr, 1000);
assert_eq!(node.retry_pending.len(), 1);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert_eq!(state.retry_count, 1);
assert!(state.reconnect, "Auto-connect peers always get reconnect=true");
// Default base = 5s, 2^1 = 10s, but first retry is 2^0... let me check:
// retry_count is set to 1, backoff_ms(5000) = 5000 * 2^1 = 10000
assert_eq!(state.retry_after_ms, 1000 + 10_000);
@@ -595,20 +596,20 @@ fn test_schedule_retry_increments() {
let mut node = Node::new(config).unwrap();
// First failure
node.schedule_retry(peer_node_addr, 1000, false);
node.schedule_retry(peer_node_addr, 1000);
assert_eq!(node.retry_pending.get(&peer_node_addr).unwrap().retry_count, 1);
// Second failure
node.schedule_retry(peer_node_addr, 11_000, false);
node.schedule_retry(peer_node_addr, 11_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert_eq!(state.retry_count, 2);
// backoff_ms(5000) with retry_count=2 = 5000 * 4 = 20000
assert_eq!(state.retry_after_ms, 11_000 + 20_000);
}
/// Test that schedule_retry gives up after max_retries.
/// Test that auto-connect peers retry indefinitely (never exhaust).
#[test]
fn test_schedule_retry_max_retries_exhausted() {
fn test_schedule_retry_auto_connect_never_exhausts() {
let peer_identity = Identity::generate();
let peer_npub = peer_identity.npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
@@ -623,19 +624,20 @@ fn test_schedule_retry_max_retries_exhausted() {
let mut node = Node::new(config).unwrap();
// Attempts 1 and 2 should schedule retries
node.schedule_retry(peer_node_addr, 1000, false);
// All attempts should keep the entry alive despite max_retries=2
node.schedule_retry(peer_node_addr, 1000);
assert!(node.retry_pending.contains_key(&peer_node_addr));
node.schedule_retry(peer_node_addr, 2000, false);
node.schedule_retry(peer_node_addr, 2000);
assert!(node.retry_pending.contains_key(&peer_node_addr));
// Attempt 3 exceeds max_retries=2, should remove entry
node.schedule_retry(peer_node_addr, 3000, false);
// Attempt 3 would have exhausted before, but now retries indefinitely
node.schedule_retry(peer_node_addr, 3000);
assert!(
!node.retry_pending.contains_key(&peer_node_addr),
"Should be removed after max retries exhausted"
node.retry_pending.contains_key(&peer_node_addr),
"Auto-connect peers should never exhaust retries"
);
assert_eq!(node.retry_pending.get(&peer_node_addr).unwrap().retry_count, 3);
}
/// Test that schedule_retry does nothing when max_retries is 0.
@@ -655,7 +657,7 @@ fn test_schedule_retry_disabled() {
let mut node = Node::new(config).unwrap();
node.schedule_retry(peer_node_addr, 1000, false);
node.schedule_retry(peer_node_addr, 1000);
assert!(
node.retry_pending.is_empty(),
"No retry should be scheduled when max_retries=0"
@@ -671,7 +673,7 @@ fn test_schedule_retry_ignores_non_autoconnect() {
// No peers configured at all
let mut node = make_node();
node.schedule_retry(peer_node_addr, 1000, false);
node.schedule_retry(peer_node_addr, 1000);
assert!(
node.retry_pending.is_empty(),
"No retry for unconfigured peer"
@@ -693,13 +695,97 @@ fn test_schedule_retry_skips_connected_peer() {
assert_eq!(node.peer_count(), 1);
// Scheduling a retry for an already-connected peer should be a no-op
node.schedule_retry(node_addr, 3000, false);
node.schedule_retry(node_addr, 3000);
assert!(
node.retry_pending.is_empty(),
"No retry for already-connected peer"
);
}
/// Test that schedule_reconnect preserves accumulated backoff across link-dead cycles.
///
/// Regression test for issue #5: previously `schedule_reconnect` always created a
/// fresh `RetryState` with `retry_count=0`, discarding any backoff accumulated by
/// prior failed handshake attempts. On repeated link-dead evictions the node would
/// restart exponential backoff from the base interval every time instead of
/// continuing to back off.
#[test]
fn test_schedule_reconnect_preserves_backoff() {
let peer_identity = Identity::generate();
let peer_npub = peer_identity.npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
let mut config = Config::new();
config.peers.push(crate::config::PeerConfig::new(
peer_npub,
"udp",
"10.0.0.2:2121",
));
let mut node = Node::new(config).unwrap();
// Simulate two stale handshake timeouts incrementing the retry count.
node.schedule_retry(peer_node_addr, 1_000); // count=1, delay=10s
node.schedule_retry(peer_node_addr, 11_000); // count=2, delay=20s
{
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert_eq!(state.retry_count, 2, "Two failures should yield count=2");
}
// Now simulate a link-dead removal triggering schedule_reconnect.
// The existing retry entry (count=2) should be preserved and bumped to 3,
// NOT reset to 0 as it was before the fix.
node.schedule_reconnect(peer_node_addr, 31_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert!(
state.reconnect,
"Entry should be marked as reconnect"
);
assert_eq!(
state.retry_count, 3,
"schedule_reconnect should increment existing count (was 2), not reset to 0 (regression: issue #5)"
);
// With count=3, backoff should be 5s * 2^3 = 40s.
let base_ms = node.config.node.retry.base_interval_secs * 1000;
let max_ms = node.config.node.retry.max_backoff_secs * 1000;
let expected_delay = state.backoff_ms(base_ms, max_ms);
assert_eq!(
state.retry_after_ms, 31_000 + expected_delay,
"retry_after_ms should reflect count=3 backoff"
);
}
/// Test that schedule_reconnect on a fresh peer (no prior retry entry) starts at count=0.
#[test]
fn test_schedule_reconnect_fresh_state() {
let peer_identity = Identity::generate();
let peer_npub = peer_identity.npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
let mut config = Config::new();
config.peers.push(crate::config::PeerConfig::new(
peer_npub,
"udp",
"10.0.0.2:2121",
));
let mut node = Node::new(config).unwrap();
// No prior retry entry — first reconnect should use base delay.
node.schedule_reconnect(peer_node_addr, 1_000);
let state = node.retry_pending.get(&peer_node_addr).unwrap();
assert!(state.reconnect, "Entry should be marked as reconnect");
assert_eq!(state.retry_count, 0, "Fresh reconnect should start at count=0");
// Base delay: 5s * 2^0 = 5s
let base_ms = node.config.node.retry.base_interval_secs * 1000;
let max_ms = node.config.node.retry.max_backoff_secs * 1000;
let expected_delay = state.backoff_ms(base_ms, max_ms);
assert_eq!(state.retry_after_ms, 1_000 + expected_delay);
}
/// Test that promote_connection clears retry_pending.
#[test]
fn test_promote_clears_retry_pending() {

View File

@@ -211,8 +211,11 @@ impl Node {
self.bloom_state.mark_update_needed(*from);
}
// Re-evaluate parent selection with current link costs
// Re-evaluate parent selection with current link costs.
// Exclude peers without MMP RTT data — they are not yet eligible
// as parent candidates (prevents oscillation from optimistic defaults).
let peer_costs: HashMap<NodeAddr, f64> = self.peers.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
@@ -229,6 +232,7 @@ impl Node {
}
self.tree_state.recompute_coords();
self.coord_cache.clear();
self.reset_discovery_backoff();
self.stats_mut().tree.parent_switched += 1;
self.stats_mut().tree.parent_switches += 1;
@@ -263,6 +267,7 @@ impl Node {
"Parent ancestry contains us — loop detected, dropping parent"
);
let peer_costs: HashMap<NodeAddr, f64> = self.peers.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
if self.tree_state.handle_parent_lost(&peer_costs) {
@@ -271,6 +276,7 @@ impl Node {
return;
}
self.coord_cache.clear();
self.reset_discovery_backoff();
self.send_tree_announce_to_all().await;
}
return;
@@ -295,6 +301,7 @@ impl Node {
}
self.tree_state.recompute_coords();
self.coord_cache.clear();
self.reset_discovery_backoff();
let new_root = *self.tree_state.root();
let new_depth = self.tree_state.my_coords().depth();
@@ -354,6 +361,7 @@ impl Node {
self.last_parent_reeval = Some(now);
let peer_costs: HashMap<NodeAddr, f64> = self.peers.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
@@ -371,6 +379,7 @@ impl Node {
}
self.tree_state.recompute_coords();
self.coord_cache.clear();
self.reset_discovery_backoff();
self.stats_mut().tree.parent_switched += 1;
self.stats_mut().tree.parent_switches += 1;
@@ -410,6 +419,7 @@ impl Node {
if was_parent {
self.stats_mut().tree.parent_losses += 1;
let peer_costs: HashMap<NodeAddr, f64> = self.peers.iter()
.filter(|(_, peer)| peer.has_srtt())
.map(|(addr, peer)| (*addr, peer.link_cost()))
.collect();
let changed = self.tree_state.handle_parent_lost(&peer_costs);

View File

@@ -593,6 +593,12 @@ impl ActivePeer {
}
}
/// Whether this peer has at least one MMP RTT measurement.
pub fn has_srtt(&self) -> bool {
self.mmp()
.is_some_and(|mmp| mmp.metrics.srtt_ms().is_some())
}
/// When this peer was authenticated.
pub fn authenticated_at(&self) -> u64 {
self.authenticated_at
@@ -880,6 +886,12 @@ impl ActivePeer {
self.rekey_in_progress = false;
self.reset_replay_suppressed();
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
}
self.previous_our_index
}
@@ -909,6 +921,12 @@ impl ActivePeer {
self.rekey_in_progress = false;
self.reset_replay_suppressed();
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
}
self.previous_our_index
}

View File

@@ -1,6 +1,5 @@
//! Discovery messages: LookupRequest and LookupResponse.
use crate::bloom::BloomFilter;
use crate::protocol::error::ProtocolError;
use crate::protocol::session::{decode_coords, encode_coords};
use crate::tree::TreeCoordinate;
@@ -9,8 +8,9 @@ use secp256k1::schnorr::Signature;
/// Request to discover a node's coordinates.
///
/// Flooded through the network with TTL limiting scope. The visited
/// filter prevents routing loops.
/// Routed through the spanning tree via bloom-filter-guided forwarding.
/// Each transit node forwards only to tree peers whose bloom filter
/// contains the target. TTL limits propagation depth.
#[derive(Clone, Debug)]
pub struct LookupRequest {
/// Unique request identifier.
@@ -26,8 +26,6 @@ pub struct LookupRequest {
/// Minimum transport MTU the origin requires for a viable route.
/// 0 means no requirement.
pub min_mtu: u16,
/// Visited nodes filter (loop prevention).
pub visited: BloomFilter,
}
impl LookupRequest {
@@ -40,8 +38,6 @@ impl LookupRequest {
ttl: u8,
min_mtu: u16,
) -> Self {
// Small filter for visited tracking
let visited = BloomFilter::with_params(256 * 8, 5).expect("valid params");
Self {
request_id,
target,
@@ -49,7 +45,6 @@ impl LookupRequest {
origin_coords,
ttl,
min_mtu,
visited,
}
}
@@ -66,15 +61,14 @@ impl LookupRequest {
Self::new(request_id, target, origin, origin_coords, ttl, min_mtu)
}
/// Decrement TTL and add self to visited.
/// Decrement TTL for forwarding.
///
/// Returns false if TTL was already 0.
pub fn forward(&mut self, my_node_addr: &NodeAddr) -> bool {
pub fn forward(&mut self) -> bool {
if self.ttl == 0 {
return false;
}
self.ttl -= 1;
self.visited.insert(my_node_addr);
true
}
@@ -83,19 +77,12 @@ impl LookupRequest {
self.ttl > 0
}
/// Check if a node was already visited.
pub fn was_visited(&self, node_addr: &NodeAddr) -> bool {
self.visited.contains(node_addr)
}
/// Encode as wire format (includes msg_type byte).
///
/// Format: `[0x30][request_id:8][target:16][origin:16][ttl:1][min_mtu:2]`
/// `[origin_coords_cnt:2][origin_coords:16×n]`
/// `[visited_hash_cnt:1][visited_bits:256]`
pub fn encode(&self) -> Vec<u8> {
let visited_bytes = self.visited.as_bytes();
let mut buf = Vec::with_capacity(46 + self.origin_coords.depth() * 16 + 1 + visited_bytes.len());
let mut buf = Vec::with_capacity(46 + self.origin_coords.depth() * 16);
buf.push(0x30); // msg_type
buf.extend_from_slice(&self.request_id.to_le_bytes());
@@ -104,8 +91,6 @@ impl LookupRequest {
buf.push(self.ttl);
buf.extend_from_slice(&self.min_mtu.to_le_bytes());
encode_coords(&self.origin_coords, &mut buf);
buf.push(self.visited.hash_count());
buf.extend_from_slice(visited_bytes);
buf
}
@@ -113,10 +98,10 @@ impl LookupRequest {
/// Decode from wire format (after msg_type byte has been consumed).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
// Minimum: request_id(8) + target(16) + origin(16) + ttl(1) + min_mtu(2)
// + coords_count(2) + hash_count(1) = 46 bytes
if payload.len() < 46 {
// + coords_count(2) = 45 bytes
if payload.len() < 45 {
return Err(ProtocolError::MessageTooShort {
expected: 46,
expected: 45,
got: payload.len(),
});
}
@@ -150,25 +135,7 @@ impl LookupRequest {
);
pos += 2;
let (origin_coords, consumed) = decode_coords(&payload[pos..])?;
pos += consumed;
if payload.len() < pos + 1 {
return Err(ProtocolError::MessageTooShort {
expected: pos + 1,
got: payload.len(),
});
}
let hash_count = payload[pos];
pos += 1;
let filter_bytes = &payload[pos..];
if filter_bytes.is_empty() {
return Err(ProtocolError::Malformed("visited filter missing".into()));
}
let visited = BloomFilter::from_slice(filter_bytes, hash_count)
.map_err(|e| ProtocolError::Malformed(format!("bad visited filter: {e}")))?;
let (origin_coords, _consumed) = decode_coords(&payload[pos..])?;
Ok(Self {
request_id,
@@ -177,7 +144,6 @@ impl LookupRequest {
origin_coords,
ttl,
min_mtu,
visited,
})
}
}
@@ -323,17 +289,12 @@ mod tests {
let target = make_node_addr(1);
let origin = make_node_addr(2);
let coords = make_coords(&[2, 0]);
let forwarder = make_node_addr(3);
let mut request = LookupRequest::new(123, target, origin, coords, 5, 0);
assert!(request.can_forward());
assert!(!request.was_visited(&forwarder));
assert!(request.forward(&forwarder));
assert!(request.forward());
assert_eq!(request.ttl, 4);
assert!(request.was_visited(&forwarder));
}
#[test]
@@ -344,9 +305,9 @@ mod tests {
let mut request = LookupRequest::new(123, target, origin, coords, 1, 0);
assert!(request.forward(&make_node_addr(3)));
assert!(request.forward());
assert!(!request.can_forward());
assert!(!request.forward(&make_node_addr(4)));
assert!(!request.forward());
}
#[test]
@@ -384,8 +345,8 @@ mod tests {
let origin = make_node_addr(20);
let coords = make_coords(&[20, 0]);
let mut request = LookupRequest::new(12345, target, origin, coords.clone(), 8, 1386);
request.forward(&make_node_addr(30));
let mut request = LookupRequest::new(12345, target, origin, coords, 8, 1386);
request.forward();
let encoded = request.encode();
assert_eq!(encoded[0], 0x30);
@@ -396,7 +357,6 @@ mod tests {
assert_eq!(decoded.origin, origin);
assert_eq!(decoded.ttl, 7); // decremented by forward()
assert_eq!(decoded.min_mtu, 1386);
assert!(decoded.was_visited(&make_node_addr(30)));
}
#[test]
@@ -438,7 +398,7 @@ mod tests {
let digest: [u8; 32] = sha2::Sha256::digest(&proof_data).into();
let sig = secp.sign_schnorr(&digest, &keypair);
let response = LookupResponse::new(999, target, coords.clone(), sig);
let response = LookupResponse::new(999, target, coords, sig);
// Default path_mtu should be u16::MAX
assert_eq!(response.path_mtu, u16::MAX);

View File

@@ -6,6 +6,7 @@
pub mod udp;
pub mod tcp;
pub mod tor;
#[cfg(target_os = "linux")]
pub mod ethernet;
@@ -13,9 +14,12 @@ pub mod ethernet;
use secp256k1::XOnlyPublicKey;
use udp::UdpTransport;
use tcp::TcpTransport;
use tor::control::TorMonitoringInfo;
use tor::TorTransport;
#[cfg(target_os = "linux")]
use ethernet::EthernetTransport;
use std::fmt;
use std::net::SocketAddr;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use thiserror::Error;
@@ -362,9 +366,8 @@ impl fmt::Display for LinkDirection {
/// Opaque transport-specific address.
///
/// Each transport type interprets this differently:
/// - UDP: "ip:port"
/// - UDP/TCP: "host:port" (IP address or DNS hostname)
/// - Ethernet: MAC address (6 bytes)
/// - Tor: ".onion:port"
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct TransportAddr(Vec<u8>);
@@ -791,6 +794,26 @@ pub trait Transport {
}
}
// ============================================================================
// Connection State (for non-blocking connect)
// ============================================================================
/// State of a transport-level connection attempt.
///
/// Used by connection-oriented transports (TCP, Tor) to report the progress
/// of a background connection attempt initiated by `connect()`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ConnectionState {
/// No connection attempt in progress for this address.
None,
/// Connection attempt is in progress (background task running).
Connecting,
/// Connection is established and ready for send().
Connected,
/// Connection attempt failed with the given error message.
Failed(String),
}
// ============================================================================
// Transport Congestion
// ============================================================================
@@ -822,6 +845,8 @@ pub enum TransportHandle {
Ethernet(EthernetTransport),
/// TCP/IP transport.
Tcp(TcpTransport),
/// Tor transport (via SOCKS5).
Tor(TorTransport),
}
impl TransportHandle {
@@ -832,6 +857,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.start_async().await,
TransportHandle::Tcp(t) => t.start_async().await,
TransportHandle::Tor(t) => t.start_async().await,
}
}
@@ -842,6 +868,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.stop_async().await,
TransportHandle::Tcp(t) => t.stop_async().await,
TransportHandle::Tor(t) => t.stop_async().await,
}
}
@@ -852,6 +879,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.send_async(addr, data).await,
TransportHandle::Tcp(t) => t.send_async(addr, data).await,
TransportHandle::Tor(t) => t.send_async(addr, data).await,
}
}
@@ -862,6 +890,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.transport_id(),
TransportHandle::Tcp(t) => t.transport_id(),
TransportHandle::Tor(t) => t.transport_id(),
}
}
@@ -872,6 +901,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.name(),
TransportHandle::Tcp(t) => t.name(),
TransportHandle::Tor(t) => t.name(),
}
}
@@ -882,6 +912,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.transport_type(),
TransportHandle::Tcp(t) => t.transport_type(),
TransportHandle::Tor(t) => t.transport_type(),
}
}
@@ -892,6 +923,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.state(),
TransportHandle::Tcp(t) => t.state(),
TransportHandle::Tor(t) => t.state(),
}
}
@@ -902,6 +934,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.mtu(),
TransportHandle::Tcp(t) => t.mtu(),
TransportHandle::Tor(t) => t.mtu(),
}
}
@@ -915,16 +948,18 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.link_mtu(addr),
TransportHandle::Tcp(t) => t.link_mtu(addr),
TransportHandle::Tor(t) => t.link_mtu(addr),
}
}
/// Get the local bound address (UDP only, returns None for other transports).
/// Get the local bound address (UDP/TCP only, returns None for other transports).
pub fn local_addr(&self) -> Option<std::net::SocketAddr> {
match self {
TransportHandle::Udp(t) => t.local_addr(),
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(_) => None,
TransportHandle::Tcp(t) => t.local_addr(),
TransportHandle::Tor(_) => None,
}
}
@@ -935,6 +970,31 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => Some(t.interface_name()),
TransportHandle::Tcp(_) => None,
TransportHandle::Tor(_) => None,
}
}
/// Get the onion service address (Tor only, returns None for other transports).
pub fn onion_address(&self) -> Option<&str> {
match self {
TransportHandle::Tor(t) => t.onion_address(),
_ => None,
}
}
/// Get cached Tor daemon monitoring info (Tor only).
pub fn tor_monitoring(&self) -> Option<TorMonitoringInfo> {
match self {
TransportHandle::Tor(t) => t.cached_monitoring(),
_ => None,
}
}
/// Get the Tor transport mode (Tor only).
pub fn tor_mode(&self) -> Option<&str> {
match self {
TransportHandle::Tor(t) => Some(t.mode()),
_ => None,
}
}
@@ -945,6 +1005,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.discover(),
TransportHandle::Tcp(t) => t.discover(),
TransportHandle::Tor(t) => t.discover(),
}
}
@@ -955,6 +1016,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.auto_connect(),
TransportHandle::Tcp(t) => t.auto_connect(),
TransportHandle::Tor(t) => t.auto_connect(),
}
}
@@ -965,12 +1027,45 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.accept_connections(),
TransportHandle::Tcp(t) => t.accept_connections(),
TransportHandle::Tor(t) => t.accept_connections(),
}
}
/// Initiate a non-blocking connection to a remote address.
///
/// For connection-oriented transports (TCP, Tor), spawns a background
/// task to establish the connection. For connectionless transports
/// (UDP, Ethernet), this is a no-op that returns Ok immediately.
///
/// Poll `connection_state()` to check when the connection is ready.
pub async fn connect(&self, addr: &TransportAddr) -> Result<(), TransportError> {
match self {
TransportHandle::Udp(_) => Ok(()), // connectionless
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(_) => Ok(()), // connectionless
TransportHandle::Tcp(t) => t.connect_async(addr).await,
TransportHandle::Tor(t) => t.connect_async(addr).await,
}
}
/// Query the state of a connection attempt to a remote address.
///
/// For connectionless transports, always returns `ConnectionState::Connected`
/// (they are always "connected"). For connection-oriented transports, returns
/// the current state of the background connection attempt.
pub fn connection_state(&self, addr: &TransportAddr) -> ConnectionState {
match self {
TransportHandle::Udp(_) => ConnectionState::Connected,
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(_) => ConnectionState::Connected,
TransportHandle::Tcp(t) => t.connection_state_sync(addr),
TransportHandle::Tor(t) => t.connection_state_sync(addr),
}
}
/// Close a specific connection on this transport.
///
/// No-op for connectionless transports. For TCP, removes the
/// No-op for connectionless transports. For TCP/Tor, removes the
/// connection from the pool and drops the stream.
pub async fn close_connection(&self, addr: &TransportAddr) {
match self {
@@ -978,6 +1073,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(t) => t.close_connection(addr),
TransportHandle::Tcp(t) => t.close_connection_async(addr).await,
TransportHandle::Tor(t) => t.close_connection_async(addr).await,
}
}
@@ -997,6 +1093,7 @@ impl TransportHandle {
#[cfg(target_os = "linux")]
TransportHandle::Ethernet(_) => TransportCongestion::default(),
TransportHandle::Tcp(_) => TransportCongestion::default(),
TransportHandle::Tor(_) => TransportCongestion::default(),
}
}
@@ -1027,10 +1124,50 @@ impl TransportHandle {
TransportHandle::Tcp(t) => {
serde_json::to_value(t.stats().snapshot()).unwrap_or_default()
}
TransportHandle::Tor(t) => {
serde_json::to_value(t.stats().snapshot()).unwrap_or_default()
}
}
}
}
// ============================================================================
// DNS Resolution
// ============================================================================
/// Resolve a TransportAddr to a SocketAddr.
///
/// Fast path: if the address parses as a numeric IP:port, returns
/// immediately with no DNS lookup. Otherwise, treats the address as
/// `hostname:port` and performs async DNS resolution via the system
/// resolver.
pub(crate) async fn resolve_socket_addr(
addr: &TransportAddr,
) -> Result<SocketAddr, TransportError> {
let s = addr
.as_str()
.ok_or_else(|| TransportError::InvalidAddress("not valid UTF-8".into()))?;
// Fast path: numeric IP address — no DNS lookup
if let Ok(sock_addr) = s.parse::<SocketAddr>() {
return Ok(sock_addr);
}
// Slow path: DNS resolution
tokio::net::lookup_host(s)
.await
.map_err(|e| {
TransportError::InvalidAddress(format!("DNS resolution failed for {}: {}", s, e))
})?
.next()
.ok_or_else(|| {
TransportError::InvalidAddress(format!(
"DNS resolution returned no addresses for {}",
s
))
})
}
// ============================================================================
// Tests
// ============================================================================

View File

@@ -25,14 +25,16 @@
pub mod stats;
pub mod stream;
use super::resolve_socket_addr;
use super::{
DiscoveredPeer, PacketTx, ReceivedPacket, Transport, TransportAddr, TransportError,
TransportId, TransportState, TransportType,
ConnectionState, DiscoveredPeer, PacketTx, ReceivedPacket, Transport, TransportAddr,
TransportError, TransportId, TransportState, TransportType,
};
use crate::config::TcpConfig;
use stats::TcpStats;
use stream::read_fmp_packet;
use futures::FutureExt;
use socket2::TcpKeepalive;
use std::collections::HashMap;
use std::net::SocketAddr;
@@ -67,6 +69,18 @@ struct TcpConnection {
/// Shared connection pool.
type ConnectionPool = Arc<Mutex<HashMap<TransportAddr, TcpConnection>>>;
/// A pending background connection attempt.
///
/// Holds the JoinHandle for a spawned TCP connect task. The task
/// produces a configured `TcpStream` and MSS-derived MTU on success.
struct ConnectingEntry {
/// Background task performing TCP connect + socket configuration.
task: JoinHandle<Result<(TcpStream, u16), TransportError>>,
}
/// Map of addresses with background connection attempts in progress.
type ConnectingPool = Arc<Mutex<HashMap<TransportAddr, ConnectingEntry>>>;
// ============================================================================
// TCP Transport
// ============================================================================
@@ -85,8 +99,10 @@ pub struct TcpTransport {
config: TcpConfig,
/// Current state.
state: TransportState,
/// Connection pool: addr -> per-connection state.
/// Connection pool: addr -> established connections.
pool: ConnectionPool,
/// Pending connection attempts: addr -> background connect task.
connecting: ConnectingPool,
/// Channel for delivering received packets to Node.
packet_tx: PacketTx,
/// Accept loop task handle (if listener bound).
@@ -111,6 +127,7 @@ impl TcpTransport {
config,
state: TransportState::Configured,
pool: Arc::new(Mutex::new(HashMap::new())),
connecting: Arc::new(Mutex::new(HashMap::new())),
packet_tx,
accept_task: None,
local_addr: None,
@@ -212,7 +229,19 @@ impl TcpTransport {
let _ = task.await;
}
// Close all connections
// Abort pending connection attempts
let mut connecting = self.connecting.lock().await;
for (addr, entry) in connecting.drain() {
entry.task.abort();
debug!(
transport_id = %self.transport_id,
remote_addr = %addr,
"TCP connect aborted (transport stopping)"
);
}
drop(connecting);
// Close all established connections
let mut pool = self.pool.lock().await;
for (addr, conn) in pool.drain() {
conn.recv_task.abort();
@@ -311,7 +340,7 @@ impl TcpTransport {
&self,
addr: &TransportAddr,
) -> Result<Arc<Mutex<OwnedWriteHalf>>, TransportError> {
let socket_addr = parse_socket_addr(addr)?;
let socket_addr = resolve_socket_addr(addr).await?;
let timeout_ms = self.config.connect_timeout_ms();
// Connect with timeout
@@ -396,6 +425,239 @@ impl TcpTransport {
);
}
}
/// Initiate a non-blocking connection to a remote address.
///
/// Spawns a background task that performs TCP connect with timeout,
/// configures socket options, and reads MSS. The connection becomes
/// available for `send_async()` once the task completes successfully.
///
/// Poll `connection_state_sync()` to check progress.
pub async fn connect_async(&self, addr: &TransportAddr) -> Result<(), TransportError> {
if !self.state.is_operational() {
return Err(TransportError::NotStarted);
}
// Already established?
{
let pool = self.pool.lock().await;
if pool.contains_key(addr) {
return Ok(());
}
}
// Already connecting?
{
let connecting = self.connecting.lock().await;
if connecting.contains_key(addr) {
return Ok(());
}
}
// Validate address is UTF-8 before spawning (fail fast on bad input)
let addr_string = addr
.as_str()
.ok_or_else(|| TransportError::InvalidAddress("not valid UTF-8".into()))?
.to_string();
let timeout_ms = self.config.connect_timeout_ms();
let config = self.config.clone();
let transport_id = self.transport_id;
let remote_addr = addr.clone();
debug!(
transport_id = %transport_id,
remote_addr = %remote_addr,
timeout_ms,
"Initiating background TCP connect"
);
let task = tokio::spawn(async move {
// Resolve address (may involve DNS for hostnames)
let socket_addr: SocketAddr = if let Ok(sa) = addr_string.parse() {
sa
} else {
tokio::net::lookup_host(&addr_string)
.await
.map_err(|e| {
TransportError::InvalidAddress(format!(
"DNS resolution failed for {}: {}",
addr_string, e
))
})?
.next()
.ok_or_else(|| {
TransportError::InvalidAddress(format!(
"DNS resolution returned no addresses for {}",
addr_string
))
})?
};
// Connect with timeout
let stream = match tokio::time::timeout(
Duration::from_millis(timeout_ms),
TcpStream::connect(socket_addr),
)
.await
{
Ok(Ok(stream)) => stream,
Ok(Err(e)) => {
debug!(
transport_id = %transport_id,
remote_addr = %remote_addr,
error = %e,
"Background TCP connect refused"
);
return Err(TransportError::ConnectionRefused);
}
Err(_) => {
debug!(
transport_id = %transport_id,
remote_addr = %remote_addr,
"Background TCP connect timed out"
);
return Err(TransportError::Timeout);
}
};
// Configure socket options via socket2
let std_stream = stream.into_std()
.map_err(|e| TransportError::StartFailed(format!("into_std: {}", e)))?;
configure_socket(&std_stream, &config)?;
// Read TCP_MAXSEG for per-connection MTU
let mss_mtu = read_mss_mtu(&std_stream, config.mtu());
// Convert back to tokio
let stream = TcpStream::from_std(std_stream)
.map_err(|e| TransportError::StartFailed(format!("from_std: {}", e)))?;
Ok((stream, mss_mtu))
});
let mut connecting = self.connecting.lock().await;
connecting.insert(addr.clone(), ConnectingEntry { task });
Ok(())
}
/// Query the state of a connection to a remote address.
///
/// Checks both established and connecting pools. If a background
/// connect task has completed, promotes it to the established pool
/// (spawning a receive loop) or reports the failure.
///
/// This method is synchronous but uses `try_lock` internally.
/// Returns `ConnectionState::Connecting` if locks can't be acquired.
pub fn connection_state_sync(&self, addr: &TransportAddr) -> ConnectionState {
// Check established pool first
if let Ok(pool) = self.pool.try_lock() {
if pool.contains_key(addr) {
return ConnectionState::Connected;
}
} else {
return ConnectionState::Connecting; // can't tell, assume still going
}
// Check connecting pool
let mut connecting = match self.connecting.try_lock() {
Ok(c) => c,
Err(_) => return ConnectionState::Connecting,
};
let entry = match connecting.get_mut(addr) {
Some(e) => e,
None => return ConnectionState::None,
};
// Check if the background task has completed
if !entry.task.is_finished() {
return ConnectionState::Connecting;
}
// Task is done — take the result and remove from connecting pool.
// We need to poll the finished task. Since it's finished, we use
// now_or_never to get the result without blocking.
let addr_clone = addr.clone();
let task = connecting.remove(&addr_clone).unwrap().task;
// Use futures::FutureExt::now_or_never or block_on for the finished task.
// Since the task is finished, we can safely poll it.
match task.now_or_never() {
Some(Ok(Ok((stream, mss_mtu)))) => {
// Promote to established pool
self.promote_connection(addr, stream, mss_mtu);
ConnectionState::Connected
}
Some(Ok(Err(e))) => {
ConnectionState::Failed(format!("{}", e))
}
Some(Err(e)) => {
// JoinError (panic or cancel)
ConnectionState::Failed(format!("task failed: {}", e))
}
None => {
// Shouldn't happen since is_finished() was true
ConnectionState::Connecting
}
}
}
/// Promote a completed background connection to the established pool.
///
/// Splits the stream, spawns a receive loop, and inserts into the pool.
/// Called from `connection_state_sync()` when a background task completes.
fn promote_connection(&self, addr: &TransportAddr, stream: TcpStream, mss_mtu: u16) {
let (read_half, write_half) = stream.into_split();
let writer = Arc::new(Mutex::new(write_half));
let transport_id = self.transport_id;
let packet_tx = self.packet_tx.clone();
let pool = self.pool.clone();
let recv_stats = self.stats.clone();
let remote_addr = addr.clone();
let recv_task = tokio::spawn(async move {
tcp_receive_loop(
read_half,
transport_id,
remote_addr.clone(),
packet_tx,
pool,
mss_mtu,
recv_stats,
)
.await;
});
let conn = TcpConnection {
writer,
recv_task,
mtu: mss_mtu,
established_at: Instant::now(),
};
// Use try_lock since we're in a sync context and the pool
// should be available (connection_state_sync already checked it)
if let Ok(mut pool) = self.pool.try_lock() {
pool.insert(addr.clone(), conn);
self.stats.record_connection_established();
debug!(
transport_id = %self.transport_id,
remote_addr = %addr,
mtu = mss_mtu,
"TCP connection established (background connect)"
);
} else {
// Pool locked — abort the recv task, connection will be retried
conn.recv_task.abort();
warn!(
transport_id = %self.transport_id,
remote_addr = %addr,
"Failed to promote connection (pool locked)"
);
}
}
}
impl Transport for TcpTransport {
@@ -667,14 +929,6 @@ async fn tcp_receive_loop(
// Socket Configuration Helpers
// ============================================================================
/// Parse a TransportAddr as SocketAddr.
fn parse_socket_addr(addr: &TransportAddr) -> Result<SocketAddr, TransportError> {
addr.as_str()
.ok_or_else(|| TransportError::InvalidAddress("not valid UTF-8".into()))?
.parse()
.map_err(|e| TransportError::InvalidAddress(format!("{}", e)))
}
/// Configure a TCP socket with the transport's settings.
fn configure_socket(
stream: &std::net::TcpStream,
@@ -1120,4 +1374,243 @@ mod tests {
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
#[tokio::test]
async fn test_connect_async_success() {
let (tx1, mut rx1) = packet_channel(100);
let (tx2, _rx2) = packet_channel(100);
let mut t1 = TcpTransport::new(TransportId::new(1), None, make_outbound_config(), tx1);
let mut t2 = TcpTransport::new(TransportId::new(2), None, make_config(), tx2);
t1.start_async().await.unwrap();
t2.start_async().await.unwrap();
let addr2 = t2.local_addr().unwrap();
let remote = TransportAddr::from_string(&addr2.to_string());
// State should be None before connect
assert_eq!(t1.connection_state_sync(&remote), ConnectionState::None);
// Initiate non-blocking connect
t1.connect_async(&remote).await.unwrap();
// Wait for the background connect to complete
tokio::time::sleep(Duration::from_millis(200)).await;
// Poll state — should be Connected now
let state = t1.connection_state_sync(&remote);
assert_eq!(state, ConnectionState::Connected);
// Now send should work (connection already established)
let mut msg1 = vec![0xAA; 114];
msg1[0] = 0x01;
msg1[1] = 0x00;
msg1[2..4].copy_from_slice(&110u16.to_le_bytes());
t1.send_async(&remote, &msg1).await.unwrap();
let packet = timeout(Duration::from_secs(2), rx1.recv())
.await;
// We receive on rx1 but that's the wrong receiver — t2's rx gets the packet
// Just verify send didn't error
drop(packet);
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
#[tokio::test]
async fn test_connect_async_timeout() {
let (tx, _rx) = packet_channel(100);
let config = TcpConfig {
bind_addr: None,
connect_timeout_ms: Some(100), // Very short timeout
..Default::default()
};
let mut transport = TcpTransport::new(TransportId::new(1), None, config, tx);
transport.start_async().await.unwrap();
let remote = TransportAddr::from_string("192.0.2.1:2121");
transport.connect_async(&remote).await.unwrap();
// Wait for timeout
tokio::time::sleep(Duration::from_millis(500)).await;
let state = transport.connection_state_sync(&remote);
assert!(matches!(state, ConnectionState::Failed(_)));
transport.stop_async().await.unwrap();
}
#[tokio::test]
async fn test_connect_async_not_started() {
let (tx, _rx) = packet_channel(100);
let transport = TcpTransport::new(TransportId::new(1), None, make_config(), tx);
let result = transport
.connect_async(&TransportAddr::from_string("127.0.0.1:9999"))
.await;
assert!(matches!(result, Err(TransportError::NotStarted)));
}
#[tokio::test]
async fn test_connect_async_already_connected() {
let (tx1, _rx1) = packet_channel(100);
let (tx2, _rx2) = packet_channel(100);
let mut t1 = TcpTransport::new(TransportId::new(1), None, make_outbound_config(), tx1);
let mut t2 = TcpTransport::new(TransportId::new(2), None, make_config(), tx2);
t1.start_async().await.unwrap();
t2.start_async().await.unwrap();
let addr2 = t2.local_addr().unwrap();
let remote = TransportAddr::from_string(&addr2.to_string());
// Connect first time
t1.connect_async(&remote).await.unwrap();
tokio::time::sleep(Duration::from_millis(200)).await;
assert_eq!(t1.connection_state_sync(&remote), ConnectionState::Connected);
// Second connect should be a no-op (already connected)
t1.connect_async(&remote).await.unwrap();
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
#[tokio::test]
async fn test_connect_async_then_send_recv() {
let (tx1, _rx1) = packet_channel(100);
let (tx2, mut rx2) = packet_channel(100);
let mut t1 = TcpTransport::new(TransportId::new(1), None, make_outbound_config(), tx1);
let mut t2 = TcpTransport::new(TransportId::new(2), None, make_config(), tx2);
t1.start_async().await.unwrap();
t2.start_async().await.unwrap();
let addr2 = t2.local_addr().unwrap();
let remote = TransportAddr::from_string(&addr2.to_string());
// Connect first, then send
t1.connect_async(&remote).await.unwrap();
tokio::time::sleep(Duration::from_millis(200)).await;
assert_eq!(t1.connection_state_sync(&remote), ConnectionState::Connected);
// Build valid FMP msg1 frame
let mut msg1 = vec![0xAA; 114];
msg1[0] = 0x01;
msg1[1] = 0x00;
msg1[2..4].copy_from_slice(&110u16.to_le_bytes());
// Send using the pre-established connection
t1.send_async(&remote, &msg1).await.unwrap();
let packet = timeout(Duration::from_secs(2), rx2.recv())
.await
.expect("timeout")
.expect("channel closed");
assert_eq!(packet.data, msg1);
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
#[test]
fn test_connection_state_none_for_unknown() {
let (tx, _rx) = packet_channel(100);
let transport = TcpTransport::new(TransportId::new(1), None, make_config(), tx);
let state = transport.connection_state_sync(
&TransportAddr::from_string("unknown:1234"),
);
assert_eq!(state, ConnectionState::None);
}
#[tokio::test]
async fn test_connect_ip_string() {
let (tx1, _rx1) = packet_channel(100);
let (tx2, mut rx2) = packet_channel(100);
let mut t1 = TcpTransport::new(TransportId::new(1), None, make_config(), tx1);
let mut t2 = TcpTransport::new(
TransportId::new(2),
None,
TcpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
..Default::default()
},
tx2,
);
t1.start_async().await.unwrap();
t2.start_async().await.unwrap();
let port2 = t2.local_addr().unwrap().port();
// Connect using IP string — build a valid FMP frame (114 bytes)
let addr = TransportAddr::from_string(&format!("127.0.0.1:{}", port2));
let mut frame = vec![0xAA; 114];
frame[0] = 0x01; // ver=0, phase=1
frame[1] = 0x00; // flags
frame[2..4].copy_from_slice(&110u16.to_le_bytes()); // payload_len
t1.send_async(&addr, &frame).await.unwrap();
// Receive on t2
let packet = tokio::time::timeout(Duration::from_secs(5), rx2.recv())
.await
.expect("timeout")
.expect("channel closed");
assert_eq!(packet.data, frame);
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
#[tokio::test]
async fn test_connect_async_ip_string() {
let (tx1, _rx1) = packet_channel(100);
let (tx2, _rx2) = packet_channel(100);
let mut t1 = TcpTransport::new(TransportId::new(1), None, make_config(), tx1);
let mut t2 = TcpTransport::new(
TransportId::new(2),
None,
TcpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
..Default::default()
},
tx2,
);
t1.start_async().await.unwrap();
t2.start_async().await.unwrap();
let port2 = t2.local_addr().unwrap().port();
let addr = TransportAddr::from_string(&format!("127.0.0.1:{}", port2));
// Non-blocking connect via IP string
t1.connect_async(&addr).await.unwrap();
// Poll until connected
for _ in 0..50 {
let state = t1.connection_state_sync(&addr);
if state == ConnectionState::Connected {
break;
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
assert_eq!(
t1.connection_state_sync(&addr),
ConnectionState::Connected,
);
t1.stop_async().await.unwrap();
t2.stop_async().await.unwrap();
}
}

View File

@@ -0,0 +1,776 @@
//! Tor control port client.
//!
//! Minimal async client for the Tor control protocol (control-spec).
//! Implements AUTHENTICATE and GETINFO for monitoring the Tor daemon.
use std::fmt;
use std::path::{Path, PathBuf};
use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWrite, AsyncWriteExt, BufReader};
use tokio::net::TcpStream;
#[cfg(unix)]
use tokio::net::UnixStream;
use serde::Serialize;
use tracing::debug;
// ============================================================================
// Error Type
// ============================================================================
/// Errors from the Tor control port client.
#[derive(Debug)]
pub enum TorControlError {
/// Failed to connect to the control port.
ConnectionFailed(String),
/// Authentication failed.
AuthFailed(String),
/// Protocol-level error (unexpected response format).
ProtocolError(String),
/// I/O error.
Io(std::io::Error),
}
impl fmt::Display for TorControlError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::ConnectionFailed(msg) => write!(f, "control port connection failed: {}", msg),
Self::AuthFailed(msg) => write!(f, "control port auth failed: {}", msg),
Self::ProtocolError(msg) => write!(f, "control protocol error: {}", msg),
Self::Io(e) => write!(f, "control port I/O error: {}", e),
}
}
}
impl std::error::Error for TorControlError {}
impl From<std::io::Error> for TorControlError {
fn from(e: std::io::Error) -> Self {
Self::Io(e)
}
}
// ============================================================================
// Authentication
// ============================================================================
/// Control port authentication method.
#[derive(Debug, Clone)]
pub enum ControlAuth {
/// Cookie authentication — reads 32-byte cookie from file, sends as hex.
Cookie(PathBuf),
/// Password authentication — sends AUTHENTICATE "password".
Password(String),
}
impl ControlAuth {
/// Parse a control_auth config string into a ControlAuth value.
///
/// - `"cookie"` or `"cookie:/path/to/cookie"` → Cookie auth
/// - `"password:secret"` → Password auth
pub fn from_config(
auth_str: &str,
default_cookie_path: &str,
) -> Result<Self, TorControlError> {
if auth_str == "cookie" {
Ok(Self::Cookie(PathBuf::from(default_cookie_path)))
} else if let Some(path) = auth_str.strip_prefix("cookie:") {
Ok(Self::Cookie(PathBuf::from(path)))
} else if let Some(password) = auth_str.strip_prefix("password:") {
Ok(Self::Password(password.to_string()))
} else {
Err(TorControlError::AuthFailed(format!(
"unknown control_auth format '{}': expected 'cookie', 'cookie:/path', or 'password:secret'",
auth_str
)))
}
}
}
// ============================================================================
// Monitoring Info
// ============================================================================
/// Snapshot of Tor daemon status collected via control port GETINFO queries.
#[derive(Debug, Clone, Serialize)]
pub struct TorMonitoringInfo {
/// Bootstrap progress (0-100).
pub bootstrap: u8,
/// Whether Tor has at least one working circuit.
pub circuit_established: bool,
/// Total bytes read by Tor since startup.
pub traffic_read: u64,
/// Total bytes written by Tor since startup.
pub traffic_written: u64,
/// Network liveness: "up" or "down".
pub network_liveness: String,
/// Tor daemon version string.
pub version: String,
/// Whether Tor is in dormant mode (no recent activity).
pub dormant: bool,
}
// ============================================================================
// Client
// ============================================================================
/// Async Tor control port client.
///
/// Maintains a persistent connection to the Tor daemon's control port.
/// Supports both TCP (`host:port`) and Unix socket (`/path/to/socket`)
/// connections. The connection must stay alive for the lifetime of
/// ephemeral onion services (unless created with detach=true).
pub struct TorControlClient {
reader: BufReader<Box<dyn AsyncRead + Unpin + Send>>,
writer: Box<dyn AsyncWrite + Unpin + Send>,
}
impl fmt::Debug for TorControlClient {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("TorControlClient").finish_non_exhaustive()
}
}
impl TorControlClient {
/// Connect to a Tor control port.
///
/// The address can be either:
/// - A TCP address (`host:port` or `IP:port`) for TCP connections
/// - A filesystem path (starting with `/` or `./`) for Unix socket connections
///
/// Unix sockets are preferred for security: they provide filesystem
/// permission-based access control and are not reachable from containers
/// unless explicitly mounted. The Debian default is `/run/tor/control`.
pub async fn connect(addr: &str) -> Result<Self, TorControlError> {
if is_unix_socket_path(addr) {
Self::connect_unix(addr).await
} else {
Self::connect_tcp(addr).await
}
}
/// Connect via TCP to a control port at `host:port`.
async fn connect_tcp(addr: &str) -> Result<Self, TorControlError> {
let stream = TcpStream::connect(addr).await.map_err(|e| {
TorControlError::ConnectionFailed(format!(
"failed to connect to control port {}: {}",
addr, e
))
})?;
let (read_half, write_half) = stream.into_split();
debug!(addr = %addr, transport = "tcp", "Connected to Tor control port");
Ok(Self {
reader: BufReader::new(Box::new(read_half)),
writer: Box::new(write_half),
})
}
/// Connect via Unix socket to a control port at the given path.
#[cfg(unix)]
async fn connect_unix(path: &str) -> Result<Self, TorControlError> {
let stream = UnixStream::connect(path).await.map_err(|e| {
TorControlError::ConnectionFailed(format!(
"failed to connect to control socket {}: {}",
path, e
))
})?;
let (read_half, write_half) = stream.into_split();
debug!(path = %path, transport = "unix", "Connected to Tor control port");
Ok(Self {
reader: BufReader::new(Box::new(read_half)),
writer: Box::new(write_half),
})
}
#[cfg(not(unix))]
async fn connect_unix(path: &str) -> Result<Self, TorControlError> {
Err(TorControlError::ConnectionFailed(format!(
"Unix sockets not supported on this platform: {}",
path
)))
}
/// Authenticate with the Tor daemon.
pub async fn authenticate(&mut self, auth: &ControlAuth) -> Result<(), TorControlError> {
let command = match auth {
ControlAuth::Cookie(path) => {
let cookie = read_cookie_file(path)?;
format!("AUTHENTICATE {}\r\n", hex::encode(cookie))
}
ControlAuth::Password(password) => {
// Escape quotes in password
let escaped = password.replace('\\', "\\\\").replace('"', "\\\"");
format!("AUTHENTICATE \"{}\"\r\n", escaped)
}
};
self.send_command(&command).await?;
let response = self.read_response().await?;
if response.code != 250 {
return Err(TorControlError::AuthFailed(format!(
"AUTHENTICATE failed: {} {}",
response.code, response.message
)));
}
debug!("Authenticated with Tor control port");
Ok(())
}
// ========================================================================
// Monitoring Queries
// ========================================================================
/// Issue a GETINFO query and return the value for the given key.
///
/// Tor responds with `250-key=value` data lines. This extracts the
/// value for the requested key.
async fn getinfo(&mut self, key: &str) -> Result<String, TorControlError> {
let command = format!("GETINFO {}\r\n", key);
self.send_command(&command).await?;
let response = self.read_response().await?;
if response.code != 250 {
return Err(TorControlError::ProtocolError(format!(
"GETINFO {} failed: {} {}",
key, response.code, response.message
)));
}
let prefix = format!("{}=", key);
for line in &response.data_lines {
if let Some(value) = line.strip_prefix(&prefix) {
return Ok(value.to_string());
}
}
Err(TorControlError::ProtocolError(format!(
"GETINFO response missing key '{}'",
key
)))
}
/// Query Tor's bootstrap progress (0-100).
pub async fn get_bootstrap_phase(&mut self) -> Result<u8, TorControlError> {
let raw = self.getinfo("status/bootstrap-phase").await?;
// Value looks like: NOTICE BOOTSTRAP PROGRESS=100 TAG=done SUMMARY="Done"
if let Some(progress_start) = raw.find("PROGRESS=") {
let after = &raw[progress_start + 9..];
let digits: String = after.chars().take_while(|c| c.is_ascii_digit()).collect();
if let Ok(progress) = digits.parse::<u8>() {
return Ok(progress);
}
}
Err(TorControlError::ProtocolError(
"could not parse bootstrap progress".into(),
))
}
/// Check whether Tor has established circuits (health check).
///
/// Returns true if Tor has at least one working circuit, false otherwise.
pub async fn is_circuit_established(&mut self) -> Result<bool, TorControlError> {
let value = self.getinfo("status/circuit-established").await?;
Ok(value.trim() == "1")
}
/// Query total bytes read by Tor since startup.
pub async fn traffic_read(&mut self) -> Result<u64, TorControlError> {
let value = self.getinfo("traffic/read").await?;
value.trim().parse::<u64>().map_err(|_| {
TorControlError::ProtocolError(format!(
"invalid traffic/read value: '{}'",
value
))
})
}
/// Query total bytes written by Tor since startup.
pub async fn traffic_written(&mut self) -> Result<u64, TorControlError> {
let value = self.getinfo("traffic/written").await?;
value.trim().parse::<u64>().map_err(|_| {
TorControlError::ProtocolError(format!(
"invalid traffic/written value: '{}'",
value
))
})
}
/// Query whether Tor considers the network reachable.
///
/// Returns `"up"` or `"down"`.
pub async fn network_liveness(&mut self) -> Result<String, TorControlError> {
self.getinfo("network-liveness").await
}
/// Query the Tor daemon version string.
pub async fn version(&mut self) -> Result<String, TorControlError> {
self.getinfo("version").await
}
/// Query whether Tor is in dormant mode (no recent activity).
pub async fn is_dormant(&mut self) -> Result<bool, TorControlError> {
let value = self.getinfo("dormant").await?;
Ok(value.trim() == "1")
}
/// Query Tor's SOCKS listener addresses.
///
/// Returns a list of addresses Tor is listening on for SOCKS connections.
pub async fn socks_listeners(&mut self) -> Result<Vec<String>, TorControlError> {
let value = self.getinfo("net/listeners/socks").await?;
Ok(value.split_whitespace().map(|s| s.trim_matches('"').to_string()).collect())
}
/// Collect all monitoring info in a single batch of queries.
pub async fn monitoring_snapshot(&mut self) -> Result<TorMonitoringInfo, TorControlError> {
let bootstrap = self.get_bootstrap_phase().await.unwrap_or(0);
let circuit_established = self.is_circuit_established().await.unwrap_or(false);
let traffic_read = self.traffic_read().await.unwrap_or(0);
let traffic_written = self.traffic_written().await.unwrap_or(0);
let network_liveness = self.network_liveness().await.unwrap_or_else(|_| "unknown".into());
let version = self.version().await.unwrap_or_else(|_| "unknown".into());
let dormant = self.is_dormant().await.unwrap_or(false);
Ok(TorMonitoringInfo {
bootstrap,
circuit_established,
traffic_read,
traffic_written,
network_liveness,
version,
dormant,
})
}
// ========================================================================
// Protocol Helpers
// ========================================================================
/// Send a raw command string to the control port.
async fn send_command(&mut self, command: &str) -> Result<(), TorControlError> {
self.writer.write_all(command.as_bytes()).await?;
self.writer.flush().await?;
Ok(())
}
/// Read a complete response from the control port.
///
/// Tor responses are line-based:
/// - `250-key=value` — mid-reply data line (more lines follow)
/// - `250 OK` — final line of a successful reply
/// - `5xx message` — error
///
/// Returns the status code and collected data lines.
async fn read_response(&mut self) -> Result<ControlResponse, TorControlError> {
let mut data_lines = Vec::new();
let mut line_buf = String::new();
loop {
line_buf.clear();
let n = self.reader.read_line(&mut line_buf).await?;
if n == 0 {
return Err(TorControlError::ProtocolError(
"control port connection closed".into(),
));
}
let line = line_buf.trim_end_matches(['\r', '\n']);
if line.len() < 4 {
return Err(TorControlError::ProtocolError(format!(
"response line too short: '{}'",
line
)));
}
let code: u16 = line[..3].parse().map_err(|_| {
TorControlError::ProtocolError(format!(
"invalid response code in: '{}'",
line
))
})?;
let separator = line.as_bytes()[3];
let content = &line[4..];
match separator {
b'-' => {
// Mid-reply data line
data_lines.push(content.to_string());
}
b' ' => {
// Final line
return Ok(ControlResponse {
code,
message: content.to_string(),
data_lines,
});
}
b'+' => {
// Multi-line data (dot-encoded). Read until lone "."
data_lines.push(content.to_string());
loop {
line_buf.clear();
let n = self.reader.read_line(&mut line_buf).await?;
if n == 0 {
return Err(TorControlError::ProtocolError(
"connection closed during multi-line response".into(),
));
}
let dot_line =
line_buf.trim_end_matches(['\r', '\n']);
if dot_line == "." {
break;
}
// Strip leading dot-escape
let unescaped = dot_line.strip_prefix('.').unwrap_or(dot_line);
data_lines.push(unescaped.to_string());
}
}
_ => {
return Err(TorControlError::ProtocolError(format!(
"unexpected separator '{}' in: '{}'",
separator as char, line
)));
}
}
}
}
}
/// Parsed control port response.
struct ControlResponse {
/// Status code (250 = success, 5xx = error).
code: u16,
/// Message from the final line.
message: String,
/// Data lines from mid-reply (250-) lines.
data_lines: Vec<String>,
}
// ============================================================================
// Cookie File
// ============================================================================
/// Read a Tor control cookie file (32 bytes of raw binary).
fn read_cookie_file(path: &Path) -> Result<Vec<u8>, TorControlError> {
let data = std::fs::read(path).map_err(|e| {
TorControlError::AuthFailed(format!("failed to read cookie file '{}': {}", path.display(), e))
})?;
if data.len() != 32 {
return Err(TorControlError::AuthFailed(format!(
"cookie file '{}' has {} bytes, expected 32",
path.display(),
data.len()
)));
}
Ok(data)
}
// ============================================================================
// Unix Socket Detection
// ============================================================================
/// Detect whether a control address string is a Unix socket path.
///
/// Returns true if the string starts with `/` or `./`, indicating a
/// filesystem path rather than a `host:port` TCP address.
fn is_unix_socket_path(addr: &str) -> bool {
addr.starts_with('/') || addr.starts_with("./")
}
// ============================================================================
// Hex Encoding (minimal, no dependency)
// ============================================================================
mod hex {
const HEX_CHARS: &[u8; 16] = b"0123456789abcdef";
pub fn encode(data: Vec<u8>) -> String {
let mut s = String::with_capacity(data.len() * 2);
for byte in data {
s.push(HEX_CHARS[(byte >> 4) as usize] as char);
s.push(HEX_CHARS[(byte & 0x0f) as usize] as char);
}
s
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use crate::transport::tor::mock_control::{self, MockTorControlServer};
use tempfile::TempDir;
// === ControlAuth parsing ===
#[test]
fn test_control_auth_cookie_default() {
let auth = ControlAuth::from_config("cookie", "/var/run/tor/cookie").unwrap();
match auth {
ControlAuth::Cookie(path) => assert_eq!(path, Path::new("/var/run/tor/cookie")),
_ => panic!("expected Cookie"),
}
}
#[test]
fn test_control_auth_cookie_custom_path() {
let auth = ControlAuth::from_config("cookie:/tmp/my_cookie", "/default").unwrap();
match auth {
ControlAuth::Cookie(path) => assert_eq!(path, Path::new("/tmp/my_cookie")),
_ => panic!("expected Cookie"),
}
}
#[test]
fn test_control_auth_password() {
let auth = ControlAuth::from_config("password:mypass", "/default").unwrap();
match auth {
ControlAuth::Password(p) => assert_eq!(p, "mypass"),
_ => panic!("expected Password"),
}
}
#[test]
fn test_control_auth_invalid() {
let result = ControlAuth::from_config("unknown", "/default");
assert!(result.is_err());
}
// === Hex encoding ===
#[test]
fn test_hex_encode() {
assert_eq!(hex::encode(vec![0xde, 0xad, 0xbe, 0xef]), "deadbeef");
assert_eq!(hex::encode(vec![0x00, 0xff]), "00ff");
}
// === Unix socket path detection ===
#[test]
fn test_is_unix_socket_path() {
assert!(is_unix_socket_path("/run/tor/control"));
assert!(is_unix_socket_path("/var/run/tor/control"));
assert!(is_unix_socket_path("./tor-control.sock"));
assert!(!is_unix_socket_path("127.0.0.1:9051"));
assert!(!is_unix_socket_path("tor-daemon:9051"));
assert!(!is_unix_socket_path("localhost:9051"));
}
#[tokio::test]
async fn test_connect_unix_socket_nonexistent() {
let result = TorControlClient::connect("/tmp/nonexistent-tor-control.sock").await;
assert!(result.is_err());
let err = format!("{}", result.unwrap_err());
assert!(err.contains("control socket"));
}
#[tokio::test]
async fn test_connect_unix_socket_roundtrip() {
// Create a Unix socket listener, accept a connection, respond to AUTHENTICATE
let dir = TempDir::new().unwrap();
let sock_path = dir.path().join("control.sock");
let sock_path_str = sock_path.to_str().unwrap().to_string();
let listener = tokio::net::UnixListener::bind(&sock_path).unwrap();
// Spawn a minimal control handler
let handle = tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
let (reader, mut writer) = stream.into_split();
let mut reader = tokio::io::BufReader::new(reader);
let mut line = String::new();
// Read AUTHENTICATE
reader.read_line(&mut line).await.unwrap();
assert!(line.starts_with("AUTHENTICATE"));
use tokio::io::AsyncWriteExt;
writer.write_all(b"250 OK\r\n").await.unwrap();
writer.flush().await.unwrap();
});
let mut client = TorControlClient::connect(&sock_path_str).await.unwrap();
let auth = ControlAuth::Password("test".to_string());
client.authenticate(&auth).await.unwrap();
handle.await.unwrap();
}
// === Cookie file ===
#[test]
fn test_read_cookie_file_valid() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("cookie");
let cookie_data = vec![0xAA; 32];
std::fs::write(&path, &cookie_data).unwrap();
let loaded = read_cookie_file(&path).unwrap();
assert_eq!(loaded, cookie_data);
}
#[test]
fn test_read_cookie_file_wrong_size() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("cookie");
std::fs::write(&path, [0u8; 16]).unwrap();
assert!(read_cookie_file(&path).is_err());
}
#[test]
fn test_read_cookie_file_nonexistent() {
assert!(read_cookie_file(Path::new("/nonexistent/cookie")).is_err());
}
// === Control protocol (requires mock server) ===
#[tokio::test]
async fn test_authenticate_password() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
let auth = ControlAuth::Password("testpass".to_string());
client.authenticate(&auth).await.unwrap();
}
#[tokio::test]
async fn test_authenticate_cookie() {
let mock = MockTorControlServer::start().await;
// Create a cookie file
let dir = TempDir::new().unwrap();
let cookie_path = dir.path().join("cookie");
std::fs::write(&cookie_path, [0xAA; 32]).unwrap();
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
let auth = ControlAuth::Cookie(cookie_path);
client.authenticate(&auth).await.unwrap();
}
#[tokio::test]
async fn test_get_bootstrap_phase() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
let auth = ControlAuth::Password("testpass".to_string());
client.authenticate(&auth).await.unwrap();
let progress = client.get_bootstrap_phase().await.unwrap();
assert_eq!(progress, 100);
}
#[tokio::test]
async fn test_auth_failure() {
let mock = MockTorControlServer::start_with_options(mock_control::MockOptions {
reject_auth: true,
})
.await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
let auth = ControlAuth::Password("wrongpass".to_string());
let result = client.authenticate(&auth).await;
assert!(result.is_err());
}
#[tokio::test]
async fn test_connect_to_closed_port() {
// Bind and immediately drop to get a port that's closed
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
drop(listener);
let result = TorControlClient::connect(&addr.to_string()).await;
assert!(result.is_err());
}
// === Monitoring queries ===
#[tokio::test]
async fn test_is_circuit_established() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
assert!(client.is_circuit_established().await.unwrap());
}
#[tokio::test]
async fn test_traffic_counters() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
assert_eq!(client.traffic_read().await.unwrap(), 1048576);
assert_eq!(client.traffic_written().await.unwrap(), 524288);
}
#[tokio::test]
async fn test_network_liveness() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
assert_eq!(client.network_liveness().await.unwrap(), "up");
}
#[tokio::test]
async fn test_version() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
assert_eq!(client.version().await.unwrap(), "0.4.8.10");
}
#[tokio::test]
async fn test_dormant() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
assert!(!client.is_dormant().await.unwrap());
}
#[tokio::test]
async fn test_socks_listeners() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
let listeners = client.socks_listeners().await.unwrap();
assert_eq!(listeners, vec!["127.0.0.1:9050"]);
}
#[tokio::test]
async fn test_monitoring_snapshot() {
let mock = MockTorControlServer::start().await;
let mut client = TorControlClient::connect(&mock.addr().to_string()).await.unwrap();
client.authenticate(&ControlAuth::Password("test".into())).await.unwrap();
let info = client.monitoring_snapshot().await.unwrap();
assert_eq!(info.bootstrap, 100);
assert!(info.circuit_established);
assert_eq!(info.traffic_read, 1048576);
assert_eq!(info.traffic_written, 524288);
assert_eq!(info.network_liveness, "up");
assert_eq!(info.version, "0.4.8.10");
assert!(!info.dormant);
}
}

View File

@@ -0,0 +1,121 @@
//! Mock Tor control port server for testing.
//!
//! Implements enough of the Tor control protocol to validate
//! AUTHENTICATE and GETINFO commands.
use std::net::SocketAddr;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
/// Options for configuring mock behavior.
#[derive(Default)]
pub struct MockOptions {
/// If true, reject all AUTHENTICATE attempts.
pub reject_auth: bool,
}
/// A mock Tor control port server.
///
/// Accepts a single client connection and responds to control protocol
/// commands with valid-looking responses for testing.
pub struct MockTorControlServer {
addr: SocketAddr,
_handle: JoinHandle<()>,
}
impl MockTorControlServer {
/// Start a mock control server with default options.
pub async fn start() -> Self {
Self::start_with_options(MockOptions::default()).await
}
/// Start a mock control server with custom options.
pub async fn start_with_options(options: MockOptions) -> Self {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind mock control");
let addr = listener.local_addr().expect("local addr");
let handle = tokio::spawn(async move {
let (stream, _) = listener.accept().await.expect("accept");
let (reader, mut writer) = stream.into_split();
let mut reader = BufReader::new(reader);
let mut line = String::new();
let mut authenticated = false;
loop {
line.clear();
let n = match reader.read_line(&mut line).await {
Ok(n) => n,
Err(_) => break,
};
if n == 0 {
break;
}
let cmd = line.trim();
if cmd.starts_with("AUTHENTICATE") {
if options.reject_auth {
let _ = writer.write_all(b"515 Authentication failed\r\n").await;
} else {
authenticated = true;
let _ = writer.write_all(b"250 OK\r\n").await;
}
} else if !authenticated {
let _ = writer
.write_all(b"514 Authentication required\r\n")
.await;
} else if cmd.starts_with("GETINFO status/bootstrap-phase") {
let _ = writer.write_all(
b"250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=100 TAG=done SUMMARY=\"Done\"\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO status/circuit-established") {
let _ = writer.write_all(
b"250-status/circuit-established=1\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO traffic/read") {
let _ = writer.write_all(
b"250-traffic/read=1048576\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO traffic/written") {
let _ = writer.write_all(
b"250-traffic/written=524288\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO network-liveness") {
let _ = writer.write_all(
b"250-network-liveness=up\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO version") {
let _ = writer.write_all(
b"250-version=0.4.8.10\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO dormant") {
let _ = writer.write_all(
b"250-dormant=0\r\n250 OK\r\n",
).await;
} else if cmd.starts_with("GETINFO net/listeners/socks") {
let _ = writer.write_all(
b"250-net/listeners/socks=\"127.0.0.1:9050\"\r\n250 OK\r\n",
).await;
} else {
let _ = writer
.write_all(b"510 Unrecognized command\r\n")
.await;
}
let _ = writer.flush().await;
}
});
Self {
addr,
_handle: handle,
}
}
/// Get the address the mock server is listening on.
pub fn addr(&self) -> SocketAddr {
self.addr
}
}

View File

@@ -0,0 +1,157 @@
//! Mock SOCKS5 server for testing.
//!
//! Implements just enough of the SOCKS5 protocol (RFC 1928) to support
//! the username/password auth + CONNECT flow used by TorTransport.
//! Proxies bytes bidirectionally between the SOCKS5 client and a real
//! TCP target.
use std::net::SocketAddr;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
/// SOCKS5 protocol constants.
const SOCKS_VERSION: u8 = 0x05;
const AUTH_NONE: u8 = 0x00;
const AUTH_PASSWORD: u8 = 0x02;
const CMD_CONNECT: u8 = 0x01;
const ATYP_IPV4: u8 = 0x01;
const ATYP_DOMAIN: u8 = 0x03;
const REP_SUCCESS: u8 = 0x00;
/// Username/password auth sub-negotiation version (RFC 1929).
const AUTH_SUBNEG_VERSION: u8 = 0x01;
const AUTH_SUBNEG_SUCCESS: u8 = 0x00;
/// A minimal mock SOCKS5 proxy server for testing.
///
/// Accepts a single connection, performs the SOCKS5 handshake (supporting
/// both no-auth and username/password auth), then connects to a fixed
/// target address and proxies bytes bidirectionally.
pub struct MockSocks5Server {
/// Address the mock proxy is listening on.
addr: SocketAddr,
/// The real target address to connect to (ignores SOCKS5 requested target).
target_addr: SocketAddr,
/// Listener handle.
listener: Option<TcpListener>,
}
impl MockSocks5Server {
/// Create a new mock SOCKS5 server that forwards to the given target.
///
/// Binds to `127.0.0.1:0` (OS-assigned port).
pub async fn new(target_addr: SocketAddr) -> std::io::Result<Self> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
Ok(Self {
addr,
target_addr,
listener: Some(listener),
})
}
/// Get the proxy's listen address (for TorConfig.socks5_addr).
pub fn addr(&self) -> SocketAddr {
self.addr
}
/// Run the proxy, accepting one connection and proxying it.
///
/// Returns a JoinHandle that completes when the proxied connection ends.
pub fn spawn(mut self) -> JoinHandle<()> {
let listener = self.listener.take().expect("listener already consumed");
let target_addr = self.target_addr;
tokio::spawn(async move {
// Accept one SOCKS5 client
let (mut client, _) = listener.accept().await.expect("accept failed");
// === Method negotiation ===
// Client sends: [version, nmethods, methods...]
let mut ver_nmethods = [0u8; 2];
client.read_exact(&mut ver_nmethods).await.expect("read version+nmethods");
assert_eq!(ver_nmethods[0], SOCKS_VERSION, "expected SOCKS5");
let nmethods = ver_nmethods[1] as usize;
let mut methods = vec![0u8; nmethods];
client.read_exact(&mut methods).await.expect("read methods");
// Prefer username/password auth if offered, fall back to no-auth
let selected = if methods.contains(&AUTH_PASSWORD) {
AUTH_PASSWORD
} else if methods.contains(&AUTH_NONE) {
AUTH_NONE
} else {
panic!("no supported auth method offered");
};
// Reply: [version, selected_method]
client.write_all(&[SOCKS_VERSION, selected]).await.expect("write method reply");
// === Username/password sub-negotiation (RFC 1929) ===
if selected == AUTH_PASSWORD {
// Client sends: [ver(1), ulen(1), uname(ulen), plen(1), passwd(plen)]
let mut subneg_header = [0u8; 2];
client.read_exact(&mut subneg_header).await.expect("read subneg header");
assert_eq!(subneg_header[0], AUTH_SUBNEG_VERSION, "expected auth subneg v1");
let ulen = subneg_header[1] as usize;
let mut uname = vec![0u8; ulen];
client.read_exact(&mut uname).await.expect("read username");
let mut plen_buf = [0u8; 1];
client.read_exact(&mut plen_buf).await.expect("read plen");
let plen = plen_buf[0] as usize;
let mut passwd = vec![0u8; plen];
client.read_exact(&mut passwd).await.expect("read password");
// Always accept (Tor uses these as isolation keys, not real auth)
client.write_all(&[AUTH_SUBNEG_VERSION, AUTH_SUBNEG_SUCCESS])
.await.expect("write subneg reply");
}
// === Connect request ===
// Client sends: [version, cmd, rsv, atyp, addr..., port]
let mut header = [0u8; 4];
client.read_exact(&mut header).await.expect("read connect header");
assert_eq!(header[0], SOCKS_VERSION);
assert_eq!(header[1], CMD_CONNECT);
// Read and skip the address (we connect to target_addr regardless)
match header[3] {
ATYP_IPV4 => {
let mut addr_port = [0u8; 6]; // 4 IP + 2 port
client.read_exact(&mut addr_port).await.expect("read IPv4 addr");
}
ATYP_DOMAIN => {
let mut len_buf = [0u8; 1];
client.read_exact(&mut len_buf).await.expect("read domain len");
let domain_len = len_buf[0] as usize;
let mut domain_port = vec![0u8; domain_len + 2]; // domain + 2 port
client.read_exact(&mut domain_port).await.expect("read domain addr");
}
other => panic!("unsupported ATYP: {}", other),
}
// Connect to the real target
let mut target = tokio::net::TcpStream::connect(target_addr)
.await
.expect("connect to target");
// Reply: success, bind addr = 0.0.0.0:0
let reply = [
SOCKS_VERSION,
REP_SUCCESS,
0x00, // RSV
ATYP_IPV4,
0, 0, 0, 0, // bind addr
0, 0, // bind port
];
client.write_all(&reply).await.expect("write connect reply");
// Proxy bytes bidirectionally
let _ = tokio::io::copy_bidirectional(&mut client, &mut target).await;
})
}
}

1869
src/transport/tor/mod.rs Normal file

File diff suppressed because it is too large Load Diff

155
src/transport/tor/stats.rs Normal file
View File

@@ -0,0 +1,155 @@
//! Tor transport statistics.
use std::sync::atomic::{AtomicU64, Ordering};
use serde::Serialize;
/// Statistics for a Tor transport instance.
///
/// Uses atomic counters for lock-free updates from per-connection
/// receive loops and the send path concurrently.
pub struct TorStats {
pub packets_sent: AtomicU64,
pub bytes_sent: AtomicU64,
pub packets_recv: AtomicU64,
pub bytes_recv: AtomicU64,
pub send_errors: AtomicU64,
pub recv_errors: AtomicU64,
pub mtu_exceeded: AtomicU64,
pub connections_established: AtomicU64,
pub connect_timeouts: AtomicU64,
pub connect_refused: AtomicU64,
pub socks5_errors: AtomicU64,
pub connections_accepted: AtomicU64,
pub connections_rejected: AtomicU64,
pub control_errors: AtomicU64,
}
impl TorStats {
/// Create a new stats instance with all counters at zero.
pub fn new() -> Self {
Self {
packets_sent: AtomicU64::new(0),
bytes_sent: AtomicU64::new(0),
packets_recv: AtomicU64::new(0),
bytes_recv: AtomicU64::new(0),
send_errors: AtomicU64::new(0),
recv_errors: AtomicU64::new(0),
mtu_exceeded: AtomicU64::new(0),
connections_established: AtomicU64::new(0),
connect_timeouts: AtomicU64::new(0),
connect_refused: AtomicU64::new(0),
socks5_errors: AtomicU64::new(0),
connections_accepted: AtomicU64::new(0),
connections_rejected: AtomicU64::new(0),
control_errors: AtomicU64::new(0),
}
}
/// Record a successful send.
pub fn record_send(&self, bytes: usize) {
self.packets_sent.fetch_add(1, Ordering::Relaxed);
self.bytes_sent.fetch_add(bytes as u64, Ordering::Relaxed);
}
/// Record a successful receive.
pub fn record_recv(&self, bytes: usize) {
self.packets_recv.fetch_add(1, Ordering::Relaxed);
self.bytes_recv.fetch_add(bytes as u64, Ordering::Relaxed);
}
/// Record a send error.
pub fn record_send_error(&self) {
self.send_errors.fetch_add(1, Ordering::Relaxed);
}
/// Record a receive error.
pub fn record_recv_error(&self) {
self.recv_errors.fetch_add(1, Ordering::Relaxed);
}
/// Record an MTU exceeded rejection.
pub fn record_mtu_exceeded(&self) {
self.mtu_exceeded.fetch_add(1, Ordering::Relaxed);
}
/// Record a successful outbound connection.
pub fn record_connection_established(&self) {
self.connections_established.fetch_add(1, Ordering::Relaxed);
}
/// Record a connect timeout.
pub fn record_connect_timeout(&self) {
self.connect_timeouts.fetch_add(1, Ordering::Relaxed);
}
/// Record a connection refused.
pub fn record_connect_refused(&self) {
self.connect_refused.fetch_add(1, Ordering::Relaxed);
}
/// Record a SOCKS5 protocol error.
pub fn record_socks5_error(&self) {
self.socks5_errors.fetch_add(1, Ordering::Relaxed);
}
/// Record a successful inbound connection via onion service.
pub fn record_connection_accepted(&self) {
self.connections_accepted.fetch_add(1, Ordering::Relaxed);
}
/// Record a rejected inbound connection (max_inbound limit).
pub fn record_connection_rejected(&self) {
self.connections_rejected.fetch_add(1, Ordering::Relaxed);
}
/// Record a control port error.
pub fn record_control_error(&self) {
self.control_errors.fetch_add(1, Ordering::Relaxed);
}
/// Take a snapshot of all counters.
pub fn snapshot(&self) -> TorStatsSnapshot {
TorStatsSnapshot {
packets_sent: self.packets_sent.load(Ordering::Relaxed),
bytes_sent: self.bytes_sent.load(Ordering::Relaxed),
packets_recv: self.packets_recv.load(Ordering::Relaxed),
bytes_recv: self.bytes_recv.load(Ordering::Relaxed),
send_errors: self.send_errors.load(Ordering::Relaxed),
recv_errors: self.recv_errors.load(Ordering::Relaxed),
mtu_exceeded: self.mtu_exceeded.load(Ordering::Relaxed),
connections_established: self.connections_established.load(Ordering::Relaxed),
connect_timeouts: self.connect_timeouts.load(Ordering::Relaxed),
connect_refused: self.connect_refused.load(Ordering::Relaxed),
socks5_errors: self.socks5_errors.load(Ordering::Relaxed),
connections_accepted: self.connections_accepted.load(Ordering::Relaxed),
connections_rejected: self.connections_rejected.load(Ordering::Relaxed),
control_errors: self.control_errors.load(Ordering::Relaxed),
}
}
}
impl Default for TorStats {
fn default() -> Self {
Self::new()
}
}
/// Point-in-time snapshot of Tor stats (non-atomic, copyable).
#[derive(Clone, Debug, Default, Serialize)]
pub struct TorStatsSnapshot {
pub packets_sent: u64,
pub bytes_sent: u64,
pub packets_recv: u64,
pub bytes_recv: u64,
pub send_errors: u64,
pub recv_errors: u64,
pub mtu_exceeded: u64,
pub connections_established: u64,
pub connect_timeouts: u64,
pub connect_refused: u64,
pub socks5_errors: u64,
pub connections_accepted: u64,
pub connections_rejected: u64,
pub control_errors: u64,
}

Some files were not shown because too many files have changed in this diff Show More