mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-22 07:48:26 +00:00
data-plane perf overhaul: off-task encrypt + decrypt, GSO, connected UDP
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per packet) plus the sendmsg syscall off the rx_loop task onto a per-shard worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb into N on-the-wire datagrams in a single TX-stack walk) when packets in a batch are uniform-size. Same kernel primitive WireGuard's in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream. Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1: A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms) A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms) E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms) Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD + sendmmsg now run on a separate core in exchange. What lands: - src/node/encrypt_worker.rs: std::thread + crossbeam_channel workers; hash-by-destination dispatch pins a TCP flow to one worker so wire ordering is preserved; per-worker sendmmsg(2) batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when packets in a group are uniform-size. - src/node/decrypt_worker.rs: receive-side mirror. Each shard owns its session's recv cipher + replay window in a thread-local HashMap (no shared RwLock/Mutex). Sessions are handed off at promote_connection and re-registered on K-bit flip / rekey cutover. - src/node/handlers/session.rs try_send_session_data_pipelined: FSP+FMP both seal in-place in the worker on one wire-buffer alloc; no intermediate inner_plaintext / fsp_payload Vecs. - src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen socket too — without that, EADDRINUSE on activation and every packet falls back to the wildcard path); the worker sends with msg_name=NULL and the kernel uses its cached 5-tuple. Tick- driven activation in handlers/connected_udp.rs, idempotent. - src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer instead of buf.to_vec() per packet — single pointer swap, no MTU-sized memcpy. - src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view used by handle_session_datagram for the bulk local-delivery path; handle_session_payload takes the borrowed payload directly (no payload[35..].to_vec()). - src/transport/mod.rs TransportAddr::from_socket_addr: collapses the two-alloc from_string(addr.to_string()) pattern to one. - src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted ahead of packet_rx in the select! (TCP ACK starvation fix); interleaved fallback drain every 32 packets inside the rx burst loop. - noise::Session: send_cipher_clone / recv_cipher_clone / recv_replay_snapshot_owned / take_send_counter / accept_replay so off-task workers can hold a cloned cipher + reserved counter while the dispatcher keeps replay/counter sequencing serial. CipherState::cipher_clone returns a refcount-bumped LessSafeKey. AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg without going through the tokio reactor. - Worker pool sizing: both default to num_cpus, overridable via FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer connected UDP can be disabled via FIPS_CONNECTED_UDP=0. - src/perf_profile.rs: optional per-stage timing reporter under FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero overhead when disabled. - All cfg(unix)-gated. Windows continues on the existing tokio- based send/recv. Decrypt worker session lifecycle: - Node::unregister_decrypt_worker_session mirrors the existing register helper. Wired at the two natural sites that already iterate peers_by_index: the rekey drain-completion block in handlers/rekey.rs (drops the worker entry for the old our_index once the drain window has expired and the cache_key is unreachable to any in-flight OLD-K packet), and remove_active_peer in handlers/dispatch.rs (drops the worker entry for each of the four index slots: current, rekey, pending, previous). Only our_index is normally registered; unregister_session is fire- and-forget for missing entries, so calling unconditionally on all four slots is correct and bounds the cleanup without per- slot accounting. Without these callers the per-worker sessions HashMap and the Node's decrypt_registered_sessions set would grow monotonically per rekey on long-lived peers. Testing: - testing/static/scripts/bench-multirun.sh: multi-run iperf3 + ping bench. N reruns (default 5), median / min / max / CoV % / per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit total. Plain client→dest labels + topology header. Pre-bench peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15); per-path route verification via stats.bytes_sent deltas — fails fast if traffic exits via a non-static-peer link. - testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS / FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B benchmarking without rebuilds. - testing/static/scripts/iperf-test.sh: same plain client→dest labels + topology header (was multihop/direct/N hop, which conflated topology distance with on-wire path). - .config/nextest.toml: synthetic UDP node tests serialized through a max-threads=1 test group. Localhost handshakes drop on shared CI runners under parallel load; one-at-a-time keeps assertions reliable. - src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes — retries up to 5 times for synthetic edges whose msg1 was dropped, with a drain after each edge retry instead of after each attempt's full burst. - src/node/decrypt_worker.rs::tests: two unit tests asserting WorkerMsg::UnregisterSession removes the worker-thread session HashMap entry (handle_msg_unregister_session_removes_entry) and is a no-op for never-seen cache_keys (handle_msg_unregister_session_idempotent_on_unknown_key), which is the safety invariant the unconditional unregister calls at the four index slots in remove_active_peer rely on. - src/node/encrypt_worker.rs::unix_tests pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors the encoder geometry of try_send_session_data_pipelined (no coords, the common established-session path), runs the worker's real seal + send via flush_direct_batch_sync, and decodes the resulting wire packet using only canonical receive-side decoders (EncryptedHeader::parse, SessionDatagramRef::decode, FSP header parse, noise::open). Any divergence between the hand-rolled encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the decoders fails at one of the parse / open / decode steps before the inner-plaintext assertion fires. Complements the existing fsp_preseal_runs_before_outer_fmp_seal test which covers the seal-ordering invariant with synthetic headers but does not exercise the wire-layout invariant. CHANGELOG.md [Unreleased] # Changed entry added describing the worker-pool threading model, hash-by-destination dispatch, sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env vars, and the bench numbers above. Cherry-picks from mmalmi/master (paths translated from crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e, 3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5, 1d01ada, 8c37008, e12469e, 6eb2860. Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
This commit is contained in:
Martti Malmi
committed by
Johnathan Corgan
Johnathan Corgan
parent
6e7e44c8ff
commit
0a5c367edc
@@ -1,2 +1,24 @@
|
||||
[profile.ci]
|
||||
junit = { path = "junit.xml" }
|
||||
junit = { path = "junit.xml" }
|
||||
# Synthetic node tests build 250-edge meshes with one-shot UDP
|
||||
# handshakes; on shared CI runners the localhost stack still drops the
|
||||
# occasional msg1 under burst load even with the per-edge repair loop.
|
||||
# Allow a retry rather than failing the whole CI run on a single
|
||||
# dropped packet.
|
||||
retries = 2
|
||||
|
||||
[test-groups]
|
||||
node-synthetic = { max-threads = 1 }
|
||||
|
||||
# nextest runs each test in a separate process, so in-process Tokio mutexes
|
||||
# can't serialize the synthetic localhost UDP node tests on CI. Those tests
|
||||
# send one-shot handshakes without production reconnect timers; under runner
|
||||
# load even small topologies drop the lone msg1. Group all node tests so
|
||||
# they run mutually exclusive — slower CI, reliable assertions.
|
||||
[[profile.default.overrides]]
|
||||
filter = 'test(node::tests::)'
|
||||
test-group = 'node-synthetic'
|
||||
|
||||
[[profile.ci.overrides]]
|
||||
filter = 'test(node::tests::)'
|
||||
test-group = 'node-synthetic'
|
||||
|
||||
30
CHANGELOG.md
30
CHANGELOG.md
@@ -100,6 +100,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
behavioral change; the goal is to keep `cargo test` and
|
||||
`cargo clippy` clean on cross-platform builds so unrelated
|
||||
warning fixes don't get bundled into behavioral PRs.
|
||||
- Data-plane: AEAD encrypt and AEAD decrypt now run on per-shard
|
||||
worker-pool threads (`std::thread` + `crossbeam_channel`), off the
|
||||
rx_loop. Hash-by-destination dispatch pins each TCP flow to one
|
||||
worker so wire ordering is preserved; per-worker `sendmmsg(2)`
|
||||
batches up to 32 outbound packets per syscall, with UDP_GSO
|
||||
(`UDP_SEGMENT`) when the batch is uniform-sized — the same kernel
|
||||
primitive WireGuard's in-kernel module and Cloudflare's userspace
|
||||
BoringTun use to hit multi-Gbps single-stream rates. On Linux +
|
||||
macOS each established UDP peer also gets a dedicated `connect(2)`-
|
||||
ed kernel socket bound to the same wildcard listen port via
|
||||
`SO_REUSEPORT`, so the kernel caches per-packet route + neighbor
|
||||
lookup and the worker sends with `msg_name = NULL`. The receive
|
||||
side mirrors: per-shard thread-local `HashMap` owns each session's
|
||||
recv cipher + replay window, replacing the previous shared
|
||||
`RwLock`. Sessions are re-registered with the decrypt pool on
|
||||
K-bit flip and rekey cutover, and unregistered on rekey drain
|
||||
completion and peer removal so the per-shard tables stay bounded.
|
||||
New `crossbeam-channel = "0.5"` dependency. Worker counts default
|
||||
to `num_cpus`; both pools are overridable via
|
||||
`FIPS_ENCRYPT_WORKERS` and `FIPS_DECRYPT_WORKERS` (the latter
|
||||
accepts `0` to disable the pool and fall back to in-line decrypt
|
||||
in rx_loop). Per-peer connected UDP can be disabled via
|
||||
`FIPS_CONNECTED_UDP=0`. Optional per-stage timing reporter
|
||||
available via `FIPS_PERF=1` (or `FIPS_PIPELINE_TRACE=1`); detailed
|
||||
knob documentation is a follow-up at
|
||||
`docs/how-to/tune-worker-pools.md`. Bench (5 × 15 s × 1 stream
|
||||
medians, Linux x86_64, docker-bridge mesh): A→D 1379→2708 Mbps
|
||||
(1.96×), A→E 1394→2663 Mbps (1.91×), E→A 1406→2624 Mbps (1.87×);
|
||||
RTT +0.11–0.19 ms from the worker queue handoff. Windows
|
||||
continues on the existing tokio-based send/recv path.
|
||||
|
||||
### Fixed
|
||||
|
||||
|
||||
10
Cargo.lock
generated
10
Cargo.lock
generated
@@ -661,6 +661,15 @@ dependencies = [
|
||||
"itertools 0.13.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crossbeam-channel"
|
||||
version = "0.5.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2"
|
||||
dependencies = [
|
||||
"crossbeam-utils",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crossbeam-deque"
|
||||
version = "0.8.6"
|
||||
@@ -1058,6 +1067,7 @@ dependencies = [
|
||||
"bluer",
|
||||
"clap",
|
||||
"criterion",
|
||||
"crossbeam-channel",
|
||||
"dirs",
|
||||
"futures",
|
||||
"hex",
|
||||
|
||||
@@ -15,6 +15,7 @@ sha2 = "0.10"
|
||||
hkdf = "0.12"
|
||||
ring = "0.17"
|
||||
rand = "0.10.1"
|
||||
crossbeam-channel = "0.5"
|
||||
thiserror = "2.0"
|
||||
bech32 = "0.11"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
|
||||
@@ -15,6 +15,7 @@ pub mod mmp;
|
||||
pub mod node;
|
||||
pub mod noise;
|
||||
pub mod peer;
|
||||
pub mod perf_profile;
|
||||
pub mod protocol;
|
||||
pub mod transport;
|
||||
pub mod tree;
|
||||
|
||||
774
src/node/decrypt_worker.rs
Normal file
774
src/node/decrypt_worker.rs
Normal file
@@ -0,0 +1,774 @@
|
||||
//! Off-task FMP + FSP decrypt + delivery worker.
|
||||
//!
|
||||
//! First incremental step of the data-plane shard restructure (per the
|
||||
//! architectural plan): each worker now **owns its session state
|
||||
//! directly** in a local `HashMap`, with no `Arc<RwLock<HashMap>>`
|
||||
//! cache on the Node side and no `Arc<Mutex<ReplayWindow>>` shared
|
||||
//! with the rx_loop. The worker is the sole authority over the replay
|
||||
//! window and the recv-side ciphers for every session it owns.
|
||||
//!
|
||||
//! Dispatch is **deterministic by session key**: rx_loop computes
|
||||
//! `worker_idx = hash(cache_key) % N` and routes both
|
||||
//! `RegisterSession` control messages and per-packet `Job` messages
|
||||
//! through the same hash, so a session always lands on the same shard.
|
||||
//!
|
||||
//! Three message types travel through the per-worker `crossbeam_channel`:
|
||||
//!
|
||||
//! - **`RegisterSession`** — sent once on the first successful legacy
|
||||
//! decrypt for a session. Hands the worker an owned snapshot of the
|
||||
//! recv cipher + replay window for both FMP and FSP layers.
|
||||
//! - **`Job`** — per-packet bulk decrypt + deliver. The worker looks
|
||||
//! up the session in its local HashMap; if absent (registration
|
||||
//! hasn't arrived yet, or session was unregistered), the packet is
|
||||
//! bounced back to rx_loop via the fallback channel.
|
||||
//! - **`UnregisterSession`** — sent on rekey / peer drop so the worker
|
||||
//! releases the owned cipher + replay state.
|
||||
//!
|
||||
//! Only the **bulk-data** path (FMP DataPacket → FSP EndpointData) is
|
||||
//! handled by the worker. Anything else (handshakes, MMP reports,
|
||||
//! routing errors, IPv6-shim packets going to TUN) is bounced back to
|
||||
//! the rx_loop via a fallback channel so the existing slow paths
|
||||
//! continue to work.
|
||||
|
||||
// **Unix only at the call sites.** On Windows nothing constructs an
|
||||
// `OwnedSessionState` or spawns the pool (see `lifecycle.rs`), so
|
||||
// every field + function in here becomes dead. Silence the warnings
|
||||
// rather than gate them individually.
|
||||
#![cfg_attr(not(unix), allow(dead_code))]
|
||||
|
||||
use crate::NodeAddr;
|
||||
use crate::transport::{TransportAddr, TransportId};
|
||||
use crossbeam_channel::{Receiver, Sender, TrySendError, bounded};
|
||||
use ring::aead::{Aad, LessSafeKey, Nonce};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use tokio::sync::mpsc::UnboundedSender;
|
||||
use tracing::{debug, trace, warn};
|
||||
|
||||
// `endpoint_event_tx` used to ride on every `DecryptJob` so the worker
|
||||
// could deliver inbound EndpointData straight to the API layer,
|
||||
// bypassing rx_loop. After the FMP-only refactor (correctness fix —
|
||||
// see the long comment in `handle_job`'s phase-2 block) the worker
|
||||
// bounces ALL link messages back to rx_loop, so the sender went
|
||||
// unused. It's been removed: it bloated `DecryptJob` (an extra Arc
|
||||
// clone per packet on the rx_loop hot path) and — worse — its
|
||||
// presence was used as the production-path predicate in
|
||||
// `handle_encrypted_frame`, which silently disabled the entire
|
||||
// worker for TUN-only configurations that never call
|
||||
// `endpoint_data_io()`.
|
||||
|
||||
use crate::noise::ReplayWindow;
|
||||
|
||||
const WORKER_CHANNEL_CAP: usize = 32768;
|
||||
|
||||
/// Owning recv-side state for one established FMP session. Lives
|
||||
/// **inside the worker thread that owns this session** — never
|
||||
/// shared, never behind a mutex.
|
||||
///
|
||||
/// **FMP only** — the worker exclusively handles the FMP layer
|
||||
/// (decrypt + replay accept), then bounces the FMP plaintext back to
|
||||
/// rx_loop for FSP-layer dispatch. This split is what makes
|
||||
/// register-at-FMP-establishment correct: the worker doesn't need
|
||||
/// the FSP cipher / replay window, and can therefore be the
|
||||
/// authoritative recv path for a peer the moment FMP is up — well
|
||||
/// before the FSP handshake completes.
|
||||
///
|
||||
/// Built at FMP-session establishment time (`promote_connection`)
|
||||
/// and shipped to the assigned worker via `WorkerMsg::RegisterSession`.
|
||||
pub(crate) struct OwnedSessionState {
|
||||
pub fmp_cipher: LessSafeKey,
|
||||
pub fmp_replay: ReplayWindow,
|
||||
pub source_npub: Option<String>,
|
||||
}
|
||||
|
||||
/// Pre-cooked decrypt + dispatch job. Built on rx_loop after parsing
|
||||
/// the FMP header; the worker pulls its session state from its own
|
||||
/// local HashMap (keyed by `cache_key`) instead of receiving a
|
||||
/// `WorkerSessionState` clone per packet.
|
||||
pub(crate) struct DecryptJob {
|
||||
/// The raw packet bytes (incl. the 16-byte FMP outer header).
|
||||
/// Mutated in place during AEAD open — must reach the worker
|
||||
/// with the full ciphertext + tag intact.
|
||||
pub packet_data: Vec<u8>,
|
||||
/// Lookup key into the worker's owned session HashMap. Mirrors the
|
||||
/// `peers_by_index` key on the Node side: `(transport_id,
|
||||
/// receiver_idx)`.
|
||||
pub cache_key: (TransportId, u32),
|
||||
/// Source kernel transport. Forwarded into the bounced
|
||||
/// `DecryptFallback` so rx_loop can update per-peer last-seen +
|
||||
/// link stats (otherwise the MMP link-dead timer fires at 30s
|
||||
/// because the worker handles packets without ever calling
|
||||
/// `peer.touch()` / `record_recv()`).
|
||||
pub _transport_id: TransportId,
|
||||
pub _remote_addr: TransportAddr,
|
||||
pub timestamp_ms: u64,
|
||||
/// Source NodeAddr (looked up via `peers_by_index` on rx_loop).
|
||||
/// Needed to attach to the bounced `DecryptFallback` so rx_loop
|
||||
/// can dispatch its legacy link-message handler.
|
||||
pub source_node_addr: NodeAddr,
|
||||
/// Counter from the FMP outer header. Used both as nonce input
|
||||
/// and to update the replay window.
|
||||
pub fmp_counter: u64,
|
||||
/// Flag byte from the FMP outer header. Carried through the
|
||||
/// fallback so the rx_loop bounce arm can extract `CE` and `SP`
|
||||
/// for ECN propagation, MMP stats, and spin-bit RTT
|
||||
/// observation — these used to be dropped on the worker path
|
||||
/// because the bounce hardcoded `fmp_flags: 0`.
|
||||
pub fmp_flags: u8,
|
||||
/// 16-byte FMP outer header used as AAD during AEAD open.
|
||||
pub fmp_header: [u8; 16],
|
||||
/// Offset within `packet_data` where the FMP ciphertext+tag begins.
|
||||
pub fmp_ciphertext_offset: usize,
|
||||
|
||||
/// Anything that's NOT bulk EndpointData gets bounced back to the
|
||||
/// rx_loop via this channel along with its now-decrypted plaintext.
|
||||
/// The rx_loop drains this in a select! arm and runs the legacy
|
||||
/// dispatch (handshakes, MMP reports, routing errors, IPv6-shim →
|
||||
/// TUN). Keeps the slow paths working unchanged.
|
||||
pub fallback_tx: UnboundedSender<DecryptWorkerEvent>,
|
||||
}
|
||||
|
||||
/// Result of a successful FMP decrypt + replay accept, when the
|
||||
/// worker has decided this packet isn't on the EndpointData fast
|
||||
/// path and is bouncing it back to rx_loop for the legacy slow path.
|
||||
#[allow(dead_code)] // fmp_counter / fmp_flags retained for future debug paths
|
||||
pub(crate) struct DecryptFallback {
|
||||
pub source_node_addr: NodeAddr,
|
||||
/// Transport this packet arrived on — used by rx_loop's bounce
|
||||
/// arm to call `peer.set_current_addr()` so address rotation +
|
||||
/// MMP link-dead tracking continue to see updates for packets
|
||||
/// handled by the worker.
|
||||
pub transport_id: TransportId,
|
||||
/// Remote transport address — companion to `transport_id`.
|
||||
pub remote_addr: TransportAddr,
|
||||
pub timestamp_ms: u64,
|
||||
/// Length of the wire packet that produced this bounce. Used
|
||||
/// by rx_loop to call `peer.link_stats_mut().record_recv()` so
|
||||
/// per-peer stats + MMP last-seen + link-dead detection see
|
||||
/// progress for worker-handled packets. Without this update,
|
||||
/// MMP's 30-second link-dead timer fires even though packets
|
||||
/// are arriving fine.
|
||||
pub packet_len: usize,
|
||||
pub fmp_counter: u64,
|
||||
pub fmp_flags: u8,
|
||||
/// Original received wire buffer, mutated in place by the FMP
|
||||
/// AEAD open. Bytes `[fmp_plaintext_offset ..
|
||||
/// fmp_plaintext_offset+fmp_plaintext_len]` are the decrypted
|
||||
/// FMP plaintext: a 4-byte session timestamp followed by the
|
||||
/// link-layer message (FSP frame when
|
||||
/// `phase == FSP_PHASE_ESTABLISHED`). rx_loop slices into this
|
||||
/// Vec for FSP decrypt + dispatch and only allocates on the
|
||||
/// actual delivery hop.
|
||||
///
|
||||
/// **Why packet_data + offset, not `Vec<u8>` of the plaintext:**
|
||||
/// the pre-fix bounce did `packet_data[a..b].to_vec()` per
|
||||
/// packet, which is one fresh ~1500-byte allocation on every
|
||||
/// inbound bulk frame. At 150k pps that's ~225 MB/sec of
|
||||
/// memory bandwidth on the worker + rx_loop hot path, and a
|
||||
/// per-packet allocator round-trip. Passing the original Vec
|
||||
/// through unmodified lets the consumer borrow a slice; zero
|
||||
/// alloc, zero memcpy.
|
||||
pub packet_data: Vec<u8>,
|
||||
pub fmp_plaintext_offset: usize,
|
||||
pub fmp_plaintext_len: usize,
|
||||
}
|
||||
|
||||
/// Report from the decrypt worker when a registered FMP session fails
|
||||
/// AEAD authentication. Routed back to rx_loop so peer/session recovery
|
||||
/// decisions stay in one place instead of being silently dropped inside
|
||||
/// the worker thread.
|
||||
pub(crate) struct DecryptFailureReport {
|
||||
pub source_node_addr: NodeAddr,
|
||||
pub fmp_counter: u64,
|
||||
pub fmp_replay_highest: u64,
|
||||
}
|
||||
|
||||
/// Event emitted by the decrypt worker to the rx_loop.
|
||||
pub(crate) enum DecryptWorkerEvent {
|
||||
Plaintext(DecryptFallback),
|
||||
DecryptFailure(DecryptFailureReport),
|
||||
}
|
||||
|
||||
/// Messages travelling through the per-worker crossbeam channel.
|
||||
/// `Job` is the per-packet hot path; `RegisterSession` /
|
||||
/// `UnregisterSession` are control plane events sent at session
|
||||
/// establishment / teardown.
|
||||
///
|
||||
/// The `Job` variant is intentionally much larger than the control
|
||||
/// variants (it carries the whole packet buffer + cipher clone). The
|
||||
/// alternative — boxing `Job` — adds a per-packet alloc on the hot
|
||||
/// path, which is the exact thing this module is designed to avoid.
|
||||
#[allow(clippy::large_enum_variant)]
|
||||
pub(crate) enum WorkerMsg {
|
||||
Job(DecryptJob),
|
||||
RegisterSession {
|
||||
cache_key: (TransportId, u32),
|
||||
state: OwnedSessionState,
|
||||
},
|
||||
UnregisterSession {
|
||||
cache_key: (TransportId, u32),
|
||||
},
|
||||
}
|
||||
|
||||
/// Handle to the decrypt worker pool. Shard-style: each worker is one
|
||||
/// OS thread that owns its sessions outright. Dispatch is
|
||||
/// deterministic on `cache_key` so a session always reaches the same
|
||||
/// shard.
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct DecryptWorkerPool {
|
||||
senders: Arc<[Sender<WorkerMsg>]>,
|
||||
}
|
||||
|
||||
impl DecryptWorkerPool {
|
||||
pub fn spawn(n: usize) -> Self {
|
||||
let n = n.max(1);
|
||||
let mut senders = Vec::with_capacity(n);
|
||||
for i in 0..n {
|
||||
let (tx, rx) = bounded::<WorkerMsg>(WORKER_CHANNEL_CAP);
|
||||
std::thread::Builder::new()
|
||||
.name(format!("fips-decrypt-{i}"))
|
||||
.spawn(move || run_worker(i, rx))
|
||||
.expect("failed to spawn fips-decrypt OS thread");
|
||||
senders.push(tx);
|
||||
}
|
||||
Self {
|
||||
senders: senders.into(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Stable hash from session key → worker index. Same hash is used
|
||||
/// for session registration and per-packet dispatch so packets and
|
||||
/// registration arrive at the same shard.
|
||||
fn worker_idx_for(&self, cache_key: (TransportId, u32)) -> usize {
|
||||
use std::hash::{Hash, Hasher};
|
||||
let mut h = std::collections::hash_map::DefaultHasher::new();
|
||||
cache_key.hash(&mut h);
|
||||
(h.finish() as usize) % self.senders.len()
|
||||
}
|
||||
|
||||
/// Dispatch a per-packet decrypt job. Drops if the per-worker
|
||||
/// channel is full (sustained rate overrun); the rx_loop's drain
|
||||
/// caps inbound at the same scale upstream so the cliff is
|
||||
/// bounded.
|
||||
pub fn dispatch_job(&self, job: DecryptJob) {
|
||||
if self.senders.is_empty() {
|
||||
return;
|
||||
}
|
||||
let idx = self.worker_idx_for(job.cache_key);
|
||||
match self.senders[idx].try_send(WorkerMsg::Job(job)) {
|
||||
Ok(()) => {}
|
||||
Err(TrySendError::Full(_)) => {
|
||||
static FULL_COUNT: AtomicU64 = AtomicU64::new(0);
|
||||
let n = FULL_COUNT.fetch_add(1, Ordering::Relaxed);
|
||||
if n < 8 || n.is_multiple_of(10000) {
|
||||
warn!(
|
||||
worker = idx,
|
||||
drops = n + 1,
|
||||
"DecryptWorker channel full; dropping inbound packet"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(TrySendError::Disconnected(_)) => {
|
||||
debug!(worker = idx, "DecryptWorker thread gone; dropping job");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Hand ownership of a session's recv-side state to its assigned
|
||||
/// worker. Called once per session, from the rx_loop, on the
|
||||
/// first authentic legacy-path decrypt — the worker thereafter is
|
||||
/// the sole authority over the replay window and the cipher
|
||||
/// clones for this session.
|
||||
///
|
||||
/// Returns `true` iff the registration message was actually
|
||||
/// queued. Callers MUST gate any "this session is now worker-
|
||||
/// owned" state on the returned bool — the previous version
|
||||
/// fire-and-forget'd the `try_send` and the caller unconditionally
|
||||
/// marked the session as registered on its side, so under
|
||||
/// sustained queue pressure rx_loop believed the worker owned a
|
||||
/// session that had never received the cipher + replay state.
|
||||
/// Subsequent `dispatch_job` packets then arrived at a worker
|
||||
/// shard without that session in its local `HashMap` and were
|
||||
/// silently dropped (the "session unregistered mid-flight"
|
||||
/// fallback path in `handle_job`). The caller's normal retry —
|
||||
/// "re-register on a later event" — is documented at the only
|
||||
/// call site (`register_decrypt_worker_session`).
|
||||
#[must_use = "registration may have failed under queue pressure; caller must gate its own session-registered flag on the returned bool"]
|
||||
pub fn register_session(
|
||||
&self,
|
||||
cache_key: (TransportId, u32),
|
||||
state: OwnedSessionState,
|
||||
) -> bool {
|
||||
if self.senders.is_empty() {
|
||||
return false;
|
||||
}
|
||||
let idx = self.worker_idx_for(cache_key);
|
||||
match self.senders[idx].try_send(WorkerMsg::RegisterSession { cache_key, state }) {
|
||||
Ok(()) => true,
|
||||
Err(TrySendError::Full(_)) => {
|
||||
warn!(
|
||||
worker = idx,
|
||||
"DecryptWorker channel full at session registration; will retry on next packet"
|
||||
);
|
||||
false
|
||||
}
|
||||
Err(TrySendError::Disconnected(_)) => {
|
||||
debug!(
|
||||
worker = idx,
|
||||
"DecryptWorker thread gone; ignoring registration"
|
||||
);
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Drop a session from its worker (rekey, peer removed). Fire and
|
||||
/// forget — if the worker is gone we don't care.
|
||||
pub fn unregister_session(&self, cache_key: (TransportId, u32)) {
|
||||
if self.senders.is_empty() {
|
||||
return;
|
||||
}
|
||||
let idx = self.worker_idx_for(cache_key);
|
||||
let _ = self.senders[idx].try_send(WorkerMsg::UnregisterSession { cache_key });
|
||||
}
|
||||
}
|
||||
|
||||
fn run_worker(idx: usize, rx: Receiver<WorkerMsg>) {
|
||||
trace!(worker = idx, "FMP+FSP decrypt worker thread starting");
|
||||
|
||||
// The shard's owned session table. Lives entirely on this OS
|
||||
// thread — never observed by any other thread.
|
||||
let mut sessions: HashMap<(TransportId, u32), OwnedSessionState> = HashMap::new();
|
||||
|
||||
while let Ok(msg) = rx.recv() {
|
||||
handle_msg(idx, &mut sessions, msg);
|
||||
// Drain follow-ons before parking again. Keeps the thread
|
||||
// on-core for a burst (typical recvmmsg batch is 5–30 packets
|
||||
// delivered very close together).
|
||||
while let Ok(m) = rx.try_recv() {
|
||||
handle_msg(idx, &mut sessions, m);
|
||||
}
|
||||
}
|
||||
trace!(worker = idx, "FMP+FSP decrypt worker thread exiting");
|
||||
}
|
||||
|
||||
fn handle_msg(
|
||||
idx: usize,
|
||||
sessions: &mut HashMap<(TransportId, u32), OwnedSessionState>,
|
||||
msg: WorkerMsg,
|
||||
) {
|
||||
match msg {
|
||||
WorkerMsg::Job(job) => {
|
||||
if let Err(err) = handle_job(sessions, job) {
|
||||
debug!(worker = idx, error = %err, "decrypt worker job failed");
|
||||
}
|
||||
}
|
||||
WorkerMsg::RegisterSession { cache_key, state } => {
|
||||
trace!(worker = idx, ?cache_key, "DecryptWorker: register session");
|
||||
sessions.insert(cache_key, state);
|
||||
}
|
||||
WorkerMsg::UnregisterSession { cache_key } => {
|
||||
trace!(
|
||||
worker = idx,
|
||||
?cache_key,
|
||||
"DecryptWorker: unregister session"
|
||||
);
|
||||
sessions.remove(&cache_key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn handle_job(
|
||||
sessions: &mut HashMap<(TransportId, u32), OwnedSessionState>,
|
||||
job: DecryptJob,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let DecryptJob {
|
||||
mut packet_data,
|
||||
cache_key,
|
||||
_transport_id: transport_id,
|
||||
_remote_addr: remote_addr,
|
||||
timestamp_ms,
|
||||
source_node_addr,
|
||||
fmp_counter,
|
||||
fmp_flags,
|
||||
fmp_header,
|
||||
fmp_ciphertext_offset,
|
||||
fallback_tx,
|
||||
} = job;
|
||||
// Capture the wire packet length BEFORE decrypt mutates the
|
||||
// buffer — it'll be the same number either way (in-place AEAD
|
||||
// open doesn't change Vec::len), but documenting the intent.
|
||||
let packet_len = packet_data.len();
|
||||
|
||||
// Look up the shard-owned session state. If absent (session not
|
||||
// yet registered, or unregistered mid-flight), bounce the raw
|
||||
// packet to rx_loop so it can run its legacy decrypt + populate
|
||||
// the session via RegisterSession on success.
|
||||
let state = match sessions.get_mut(&cache_key) {
|
||||
Some(s) => s,
|
||||
None => {
|
||||
// The legacy rx_loop already has the ciphertext bytes
|
||||
// (worker owns `packet_data` here), but it can re-do the
|
||||
// decrypt from scratch since this is the first-packet
|
||||
// path. Bounce by sending the **encrypted** FMP frame
|
||||
// back wrapped in a fallback — rx_loop's
|
||||
// `dispatch_link_message` won't recognise it though, so
|
||||
// we just drop instead. This is a transient state on a
|
||||
// brand-new session; subsequent packets land after
|
||||
// registration.
|
||||
let _ = fallback_tx; // explicitly ignore — drop path
|
||||
let _ = source_node_addr;
|
||||
let _ = packet_data;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
|
||||
// === Phase 1: FMP decrypt ===
|
||||
let _t_fmp = crate::perf_profile::Timer::start(crate::perf_profile::Stage::FmpDecrypt);
|
||||
|
||||
// Replay-window check before AEAD work to avoid wasting CPU on
|
||||
// replays. **Direct &mut access** — no Arc<Mutex> lock acquire.
|
||||
let fmp_replay_highest = state.fmp_replay.highest();
|
||||
if !state.fmp_replay.check(fmp_counter) {
|
||||
return Ok(()); // replay; drop silently
|
||||
}
|
||||
|
||||
let mut nonce_bytes = [0u8; 12];
|
||||
nonce_bytes[4..12].copy_from_slice(&fmp_counter.to_le_bytes());
|
||||
let nonce = Nonce::assume_unique_for_key(nonce_bytes);
|
||||
let buf = &mut packet_data[fmp_ciphertext_offset..];
|
||||
let plaintext_len = match state
|
||||
.fmp_cipher
|
||||
.open_in_place(nonce, Aad::from(&fmp_header), buf)
|
||||
{
|
||||
Ok(p) => p.len(),
|
||||
Err(_) => {
|
||||
let _ = fallback_tx.send(DecryptWorkerEvent::DecryptFailure(DecryptFailureReport {
|
||||
source_node_addr,
|
||||
fmp_counter,
|
||||
fmp_replay_highest,
|
||||
}));
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
|
||||
// FMP decrypt succeeded — accept the counter into the replay window.
|
||||
state.fmp_replay.accept(fmp_counter);
|
||||
drop(_t_fmp);
|
||||
|
||||
// The FMP plaintext lives in packet_data[fmp_ciphertext_offset..
|
||||
// fmp_ciphertext_offset + plaintext_len]. It carries a 4-byte
|
||||
// session-relative timestamp prefix, then the link-layer message.
|
||||
let fmp_plaintext_start = fmp_ciphertext_offset;
|
||||
let fmp_plaintext_end = fmp_ciphertext_offset + plaintext_len;
|
||||
const INNER_TIMESTAMP_LEN: usize = 4;
|
||||
if plaintext_len < INNER_TIMESTAMP_LEN + 1 {
|
||||
return Ok(());
|
||||
}
|
||||
let link_msg_start = fmp_plaintext_start + INNER_TIMESTAMP_LEN;
|
||||
let link_msg_end = fmp_plaintext_end;
|
||||
let link_msg = &packet_data[link_msg_start..link_msg_end];
|
||||
|
||||
// === Phase 2: bounce ALL link messages back to rx_loop ===
|
||||
//
|
||||
// **Why no FSP fast path here:** previous design did FSP decrypt
|
||||
// + replay-accept for SessionDatagram (link msg_type 0x00), then
|
||||
// checked the inner FSP msg_type. If it was EndpointData (0x11),
|
||||
// delivered directly to the endpoint event channel. Otherwise
|
||||
// (heartbeats, MMP reports, IPv6-shim, etc.) bounced the
|
||||
// **decrypted-in-place** FMP plaintext back to rx_loop.
|
||||
//
|
||||
// Two problems with that path:
|
||||
// 1. After the shard-owned-sessions refactor (01f6c62), the FSP
|
||||
// replay window is owned by **this worker thread**. Once we
|
||||
// `state.fsp_replay.accept(fsp_counter)`, the rx_loop's
|
||||
// `noise::Session::replay_window` is stale — it still has
|
||||
// old counters. When rx_loop tries to FSP-decrypt the
|
||||
// bounced control frame, its legacy path's replay check
|
||||
// passes (the counter wasn't in its window) but the AEAD
|
||||
// tag check fails because the FSP bytes in `packet_data`
|
||||
// were already decrypted in place (now plaintext + 16
|
||||
// garbage tag bytes).
|
||||
// 2. Even if we didn't accept the worker's replay window for
|
||||
// non-EndpointData, the in-place mutation of `packet_data`
|
||||
// means the legacy path can't re-decrypt — the ciphertext
|
||||
// is gone.
|
||||
//
|
||||
// The bug manifests in benches as link death: heartbeats never
|
||||
// make it through the worker, the link-dead timer fires at 30s,
|
||||
// peer is removed and re-handshakes, repeating forever.
|
||||
//
|
||||
// **Fix:** worker handles only the FMP layer. ALL link messages
|
||||
// (SessionDatagram, heartbeats, control) bounce back to rx_loop
|
||||
// with the FMP plaintext intact. The legacy rx_loop path does
|
||||
// FSP-decrypt as usual. Net cost vs the broken fast path: we
|
||||
// give up the rx_loop bypass for EndpointData, but the worker
|
||||
// still offloads the FMP AEAD (~half the per-packet decrypt
|
||||
// CPU). Correctness over micro-optimisation.
|
||||
//
|
||||
// The DataShard end-state (per the architectural plan) re-
|
||||
// introduces the EndpointData fast path correctly by having the
|
||||
// shard worker also own the rx_loop side for its sessions — at
|
||||
// that point there's no "rx_loop legacy path" for the worker to
|
||||
// conflict with.
|
||||
// Pass the buffer through by ownership + offset/length. No
|
||||
// per-packet allocation; rx_loop slices into `packet_data`.
|
||||
let _ = link_msg; // sanity-check borrow before sending buffer onward
|
||||
let _ = fallback_tx.send(DecryptWorkerEvent::Plaintext(DecryptFallback {
|
||||
source_node_addr,
|
||||
transport_id,
|
||||
remote_addr,
|
||||
timestamp_ms,
|
||||
packet_len,
|
||||
fmp_counter,
|
||||
fmp_flags,
|
||||
packet_data,
|
||||
fmp_plaintext_offset: fmp_plaintext_start,
|
||||
fmp_plaintext_len: plaintext_len,
|
||||
}));
|
||||
// Suppress unused-variable warnings for the (now-removed) FSP
|
||||
// fast path. The `state` lookup is still needed for the FMP
|
||||
// cipher + replay window above.
|
||||
let _ = (link_msg_start, link_msg_end, &state.source_npub);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::noise::ReplayWindow;
|
||||
use ring::aead::{LessSafeKey, UnboundKey};
|
||||
|
||||
/// `DecryptJob.fmp_flags` must survive the worker bounce as
|
||||
/// `DecryptFallback.fmp_flags`. Pre-fix the worker hardcoded
|
||||
/// `fmp_flags: 0`, dropping CE / SP on every packet handled by
|
||||
/// the production worker path (i.e. every bulk-data packet).
|
||||
/// Loss of CE wrecks ECN propagation; loss of SP wrecks
|
||||
/// spin-bit RTT observation.
|
||||
///
|
||||
/// Drives the worker's `handle_job` directly: build an FMP wire
|
||||
/// packet sealed with a known cipher, ship a `DecryptJob` with
|
||||
/// non-zero flags through, observe the resulting `DecryptFallback`.
|
||||
#[test]
|
||||
fn worker_preserves_fmp_flags_through_fallback() {
|
||||
let key_bytes = [0u8; 32];
|
||||
let unbound = UnboundKey::new(&ring::aead::CHACHA20_POLY1305, &key_bytes).unwrap();
|
||||
// Both the sealing cipher (for building the test packet) and
|
||||
// the worker's owning cipher are clones of the same key.
|
||||
let seal_cipher = LessSafeKey::new(unbound);
|
||||
let unbound2 = UnboundKey::new(&ring::aead::CHACHA20_POLY1305, &key_bytes).unwrap();
|
||||
let open_cipher = LessSafeKey::new(unbound2);
|
||||
|
||||
let counter: u64 = 7;
|
||||
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
|
||||
// Build a wire packet `[16-byte header][4-byte inner ts][1 byte link msg]`
|
||||
// with capacity for the trailing AEAD tag. Header bytes
|
||||
// double as AAD and as the on-wire prefix.
|
||||
let mut wire = Vec::with_capacity(HDR + 4 + 1 + 16);
|
||||
// Header: fill the flags byte (the second byte) with both
|
||||
// FLAG_CE and FLAG_SP set; the rest is uninterpreted by the
|
||||
// worker (it just AADs the whole 16 bytes).
|
||||
let flags_byte = crate::node::wire::FLAG_CE | crate::node::wire::FLAG_SP;
|
||||
let mut header = [0u8; HDR];
|
||||
header[1] = flags_byte;
|
||||
wire.extend_from_slice(&header);
|
||||
wire.extend_from_slice(&[0u8; 4]); // inner ts placeholder
|
||||
wire.push(0xAB); // a single byte of "link message" payload
|
||||
|
||||
let mut nonce_bytes = [0u8; 12];
|
||||
nonce_bytes[4..12].copy_from_slice(&counter.to_le_bytes());
|
||||
let nonce = ring::aead::Nonce::assume_unique_for_key(nonce_bytes);
|
||||
let (hdr_slice, payload_slice) = wire.split_at_mut(HDR);
|
||||
let tag = seal_cipher
|
||||
.seal_in_place_separate_tag(nonce, ring::aead::Aad::from(&*hdr_slice), payload_slice)
|
||||
.unwrap();
|
||||
wire.extend_from_slice(tag.as_ref());
|
||||
|
||||
// Owning state held by the worker for this session.
|
||||
let cache_key = (TransportId::new(1), 99u32);
|
||||
let mut sessions: HashMap<(TransportId, u32), OwnedSessionState> = HashMap::new();
|
||||
sessions.insert(
|
||||
cache_key,
|
||||
OwnedSessionState {
|
||||
fmp_cipher: open_cipher,
|
||||
fmp_replay: ReplayWindow::new(),
|
||||
source_npub: None,
|
||||
},
|
||||
);
|
||||
|
||||
let (fallback_tx, mut fallback_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<DecryptWorkerEvent>();
|
||||
|
||||
let job = DecryptJob {
|
||||
packet_data: wire,
|
||||
cache_key,
|
||||
_transport_id: TransportId::new(1),
|
||||
_remote_addr: crate::transport::TransportAddr::from_string("127.0.0.1:1234"),
|
||||
timestamp_ms: 1_000,
|
||||
source_node_addr: crate::NodeAddr::from_bytes([0u8; 16]),
|
||||
fmp_counter: counter,
|
||||
fmp_flags: flags_byte,
|
||||
fmp_header: header,
|
||||
fmp_ciphertext_offset: HDR,
|
||||
fallback_tx,
|
||||
};
|
||||
|
||||
handle_job(&mut sessions, job).expect("worker job handled");
|
||||
|
||||
let event = fallback_rx.try_recv().expect("fallback delivered");
|
||||
let fallback = match event {
|
||||
DecryptWorkerEvent::Plaintext(fallback) => fallback,
|
||||
DecryptWorkerEvent::DecryptFailure(_) => panic!("expected plaintext fallback event"),
|
||||
};
|
||||
assert_eq!(
|
||||
fallback.fmp_flags, flags_byte,
|
||||
"fmp_flags must round-trip from DecryptJob to DecryptFallback"
|
||||
);
|
||||
assert!(
|
||||
fallback.fmp_flags & crate::node::wire::FLAG_CE != 0,
|
||||
"FLAG_CE bit lost on worker path"
|
||||
);
|
||||
assert!(
|
||||
fallback.fmp_flags & crate::node::wire::FLAG_SP != 0,
|
||||
"FLAG_SP bit lost on worker path"
|
||||
);
|
||||
}
|
||||
|
||||
/// Build a minimal `OwnedSessionState` for tests that only need
|
||||
/// the worker's HashMap to contain *something* keyed on cache_key
|
||||
/// (e.g. unregister-removal tests). The cipher key is all-zero
|
||||
/// and the replay window is fresh; not suitable for any test
|
||||
/// that exercises an actual AEAD open.
|
||||
fn make_test_session_state() -> OwnedSessionState {
|
||||
let key_bytes = [0u8; 32];
|
||||
let unbound = UnboundKey::new(&ring::aead::CHACHA20_POLY1305, &key_bytes).unwrap();
|
||||
OwnedSessionState {
|
||||
fmp_cipher: LessSafeKey::new(unbound),
|
||||
fmp_replay: ReplayWindow::new(),
|
||||
source_npub: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// `WorkerMsg::UnregisterSession` must remove the session's entry
|
||||
/// from the worker-owned HashMap. Without this, every rekey
|
||||
/// cutover on a long-lived peer would leave a stale entry behind,
|
||||
/// growing the worker's `sessions` map unboundedly.
|
||||
#[test]
|
||||
fn handle_msg_unregister_session_removes_entry() {
|
||||
let mut sessions: HashMap<(TransportId, u32), OwnedSessionState> = HashMap::new();
|
||||
let cache_key = (TransportId::new(1), 42u32);
|
||||
sessions.insert(cache_key, make_test_session_state());
|
||||
assert!(
|
||||
sessions.contains_key(&cache_key),
|
||||
"precondition: session should be registered"
|
||||
);
|
||||
|
||||
handle_msg(0, &mut sessions, WorkerMsg::UnregisterSession { cache_key });
|
||||
assert!(
|
||||
!sessions.contains_key(&cache_key),
|
||||
"UnregisterSession must drop the entry"
|
||||
);
|
||||
}
|
||||
|
||||
/// `WorkerMsg::UnregisterSession` for a cache_key the worker has
|
||||
/// never seen must be a no-op (no panic, no effect on unrelated
|
||||
/// sessions). This is required for the unconditional unregister
|
||||
/// calls at all four index slots in `remove_active_peer`, where
|
||||
/// most slots are typically `None` and the registered ones may
|
||||
/// have moved between slots before removal.
|
||||
#[test]
|
||||
fn handle_msg_unregister_session_idempotent_on_unknown_key() {
|
||||
let mut sessions: HashMap<(TransportId, u32), OwnedSessionState> = HashMap::new();
|
||||
let key1 = (TransportId::new(1), 1u32);
|
||||
let key2 = (TransportId::new(1), 2u32);
|
||||
|
||||
sessions.insert(key1, make_test_session_state());
|
||||
|
||||
// Unregister never-seen key — must not panic, must not affect key1.
|
||||
handle_msg(
|
||||
0,
|
||||
&mut sessions,
|
||||
WorkerMsg::UnregisterSession { cache_key: key2 },
|
||||
);
|
||||
assert!(
|
||||
sessions.contains_key(&key1),
|
||||
"unrelated session must remain after unregistering an unknown key"
|
||||
);
|
||||
assert!(
|
||||
!sessions.contains_key(&key2),
|
||||
"unknown key must not have been inserted"
|
||||
);
|
||||
|
||||
// First unregister of key1 — removes the entry.
|
||||
handle_msg(
|
||||
0,
|
||||
&mut sessions,
|
||||
WorkerMsg::UnregisterSession { cache_key: key1 },
|
||||
);
|
||||
assert!(!sessions.contains_key(&key1));
|
||||
|
||||
// Double-unregister of key1 — also no-op, no panic.
|
||||
handle_msg(
|
||||
0,
|
||||
&mut sessions,
|
||||
WorkerMsg::UnregisterSession { cache_key: key1 },
|
||||
);
|
||||
assert!(!sessions.contains_key(&key1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn worker_reports_fmp_aead_failure_to_rx_loop() {
|
||||
let key_bytes = [0u8; 32];
|
||||
let unbound = UnboundKey::new(&ring::aead::CHACHA20_POLY1305, &key_bytes).unwrap();
|
||||
let open_cipher = LessSafeKey::new(unbound);
|
||||
|
||||
let counter: u64 = 11;
|
||||
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
|
||||
let header = [0u8; HDR];
|
||||
let mut wire = Vec::with_capacity(HDR + 4 + 1 + 16);
|
||||
wire.extend_from_slice(&header);
|
||||
wire.extend_from_slice(&[0u8; 4]);
|
||||
wire.push(0xAB);
|
||||
wire.extend_from_slice(&[0u8; 16]); // invalid AEAD tag
|
||||
|
||||
let cache_key = (TransportId::new(1), 77u32);
|
||||
let mut sessions: HashMap<(TransportId, u32), OwnedSessionState> = HashMap::new();
|
||||
sessions.insert(
|
||||
cache_key,
|
||||
OwnedSessionState {
|
||||
fmp_cipher: open_cipher,
|
||||
fmp_replay: ReplayWindow::new(),
|
||||
source_npub: None,
|
||||
},
|
||||
);
|
||||
|
||||
let (fallback_tx, mut fallback_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<DecryptWorkerEvent>();
|
||||
let source_node_addr = crate::NodeAddr::from_bytes([9u8; 16]);
|
||||
let job = DecryptJob {
|
||||
packet_data: wire,
|
||||
cache_key,
|
||||
_transport_id: TransportId::new(1),
|
||||
_remote_addr: crate::transport::TransportAddr::from_string("127.0.0.1:1234"),
|
||||
timestamp_ms: 1_000,
|
||||
source_node_addr,
|
||||
fmp_counter: counter,
|
||||
fmp_flags: 0,
|
||||
fmp_header: header,
|
||||
fmp_ciphertext_offset: HDR,
|
||||
fallback_tx,
|
||||
};
|
||||
|
||||
handle_job(&mut sessions, job).expect("worker job handled");
|
||||
|
||||
let event = fallback_rx.try_recv().expect("failure delivered");
|
||||
match event {
|
||||
DecryptWorkerEvent::DecryptFailure(report) => {
|
||||
assert_eq!(report.source_node_addr, source_node_addr);
|
||||
assert_eq!(report.fmp_counter, counter);
|
||||
}
|
||||
DecryptWorkerEvent::Plaintext(_) => panic!("expected decrypt failure report"),
|
||||
}
|
||||
}
|
||||
}
|
||||
2407
src/node/encrypt_worker.rs
Normal file
2407
src/node/encrypt_worker.rs
Normal file
File diff suppressed because it is too large
Load Diff
232
src/node/handlers/connected_udp.rs
Normal file
232
src/node/handlers/connected_udp.rs
Normal file
@@ -0,0 +1,232 @@
|
||||
//! Lifecycle for per-peer connected UDP sockets.
|
||||
//!
|
||||
//! Tick-driven, idempotent, **on by default** for established UDP peers on
|
||||
//! Linux and macOS:
|
||||
//!
|
||||
//! - **Tick-driven:** every node tick, scan established UDP peers
|
||||
//! that don't yet have a connected socket installed and try to
|
||||
//! open one. No need to thread an activation call through every
|
||||
//! handshake-completion code path.
|
||||
//! - **Idempotent:** if `peer.connected_udp()` is already `Some`,
|
||||
//! skip. Replaces stale sockets lazily by clearing them on
|
||||
//! address change / rekey from elsewhere (see
|
||||
//! `deregister_session_index` and the rekey handler).
|
||||
//!
|
||||
//! Implementation note: only the **listen socket → wildcard** demux
|
||||
//! path delivers the very first packets of a session (handshakes).
|
||||
//! Once the peer's session is established, Linux/macOS install the connected
|
||||
//! socket; from that moment on the kernel routes that peer's traffic
|
||||
//! to it (most-specific 5-tuple match wins under `SO_REUSEPORT`), and
|
||||
//! the drain thread feeds the existing `packet_tx` just like the
|
||||
//! wildcard listen socket does. The rx_loop dispatch sees no
|
||||
//! difference.
|
||||
//!
|
||||
//! macOS originally defaulted to the wildcard UDP socket because early
|
||||
//! Darwin tests found liveness regressions under load. Later testing
|
||||
//! showed the problem was mismatched listener/peer `SO_REUSE*` state:
|
||||
//! with the live listener and connected sibling in the same reuse group,
|
||||
//! the connected `send(2)` path improves the MacBook Wi-Fi sender case
|
||||
//! and is now the default. Operators can still disable it with
|
||||
//! `FIPS_MACOS_CONNECTED_UDP=0` or `FIPS_CONNECTED_UDP=0` for A/B tests.
|
||||
|
||||
use crate::NodeAddr;
|
||||
use crate::node::Node;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
use crate::transport::TransportHandle;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
impl Node {
|
||||
/// Tick-driven activation of per-peer connected UDP sockets.
|
||||
/// Scans established UDP peers that don't yet have a connected
|
||||
/// socket and opens one. No-op when there are no eligible peers
|
||||
/// (e.g. only non-UDP transports). Enabled on Linux and macOS:
|
||||
/// both kernels route a matching peer 5-tuple to the connected
|
||||
/// socket when it shares the wildcard listen port via SO_REUSEPORT.
|
||||
pub(in crate::node) async fn activate_connected_udp_sessions(&mut self) {
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
{
|
||||
// No-op on platforms without the connected-UDP fast path.
|
||||
}
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
if !connected_udp_enabled() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Collect candidate NodeAddrs first so we can iterate
|
||||
// without holding the &mut on self.peers across awaits.
|
||||
let candidates: Vec<NodeAddr> = self
|
||||
.peers
|
||||
.iter()
|
||||
.filter_map(|(addr, peer)| {
|
||||
let has_session = peer.noise_session().is_some();
|
||||
let has_transport = peer.transport_id().is_some();
|
||||
let has_addr = peer.current_addr().is_some();
|
||||
let already_active = peer.connected_udp().is_some();
|
||||
if has_session && has_transport && has_addr && !already_active {
|
||||
Some(*addr)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
for addr in candidates {
|
||||
if let Err(e) = self.activate_connected_udp_for_peer(&addr).await {
|
||||
static FAILURES: AtomicU64 = AtomicU64::new(0);
|
||||
crate::perf_profile::record_event(
|
||||
crate::perf_profile::Event::ConnectedUdpActivationFailed,
|
||||
);
|
||||
let n = FAILURES.fetch_add(1, Relaxed);
|
||||
if n < 8 || n.is_multiple_of(1000) {
|
||||
warn!(peer = %addr, error = %e, failures = n + 1, "connected UDP activation deferred");
|
||||
} else {
|
||||
debug!(peer = %addr, error = %e, "connected UDP activation deferred");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Open the connected UDP socket + spawn its drain thread for
|
||||
/// one peer. Idempotent — re-checks the eligibility conditions
|
||||
/// inside the &mut so a race with peer drop doesn't install on a
|
||||
/// freshly-removed peer. Returns `Ok(())` on success or if the
|
||||
/// peer is no longer eligible (treated as benign).
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
async fn activate_connected_udp_for_peer(
|
||||
&mut self,
|
||||
node_addr: &NodeAddr,
|
||||
) -> Result<(), String> {
|
||||
// Read-only pass: figure out which transport + remote addr we need.
|
||||
let (transport_id, peer_transport_addr) = {
|
||||
let Some(peer) = self.peers.get(node_addr) else {
|
||||
return Ok(());
|
||||
};
|
||||
if peer.connected_udp().is_some() {
|
||||
return Ok(()); // already activated
|
||||
}
|
||||
let Some(tid) = peer.transport_id() else {
|
||||
return Ok(());
|
||||
};
|
||||
let Some(addr) = peer.current_addr().cloned() else {
|
||||
return Ok(());
|
||||
};
|
||||
(tid, addr)
|
||||
};
|
||||
|
||||
// Resolve the peer's TransportAddr → kernel SocketAddr via
|
||||
// the UDP transport's DNS cache. This may await on a DNS
|
||||
// lookup the very first time we see a hostname; subsequent
|
||||
// calls hit the cache.
|
||||
let (peer_socket_addr, local_addr, recv_buf, send_buf, packet_tx) = {
|
||||
let Some(transport) = self.transports.get(&transport_id) else {
|
||||
return Ok(());
|
||||
};
|
||||
let udp = match transport {
|
||||
TransportHandle::Udp(u) => u,
|
||||
_ => return Ok(()), // not a UDP transport — feature N/A
|
||||
};
|
||||
let peer_sa = udp
|
||||
.resolve_for_off_task(&peer_transport_addr)
|
||||
.await
|
||||
.map_err(|e| format!("address resolve: {e}"))?;
|
||||
let local = udp
|
||||
.local_addr()
|
||||
.ok_or_else(|| "udp transport not started".to_string())?;
|
||||
let recv_buf = udp.recv_buf_size();
|
||||
let send_buf = udp.send_buf_size();
|
||||
let tx = udp.clone_packet_tx();
|
||||
(peer_sa, local, recv_buf, send_buf, tx)
|
||||
};
|
||||
|
||||
// Open the connected socket on the kernel side.
|
||||
let socket = std::sync::Arc::new(
|
||||
crate::transport::udp::connected_peer::ConnectedPeerSocket::open(
|
||||
local_addr,
|
||||
peer_socket_addr,
|
||||
recv_buf,
|
||||
send_buf,
|
||||
)
|
||||
.map_err(|e| format!("ConnectedPeerSocket::open: {e}"))?,
|
||||
);
|
||||
|
||||
// Spawn the drain thread. It feeds `packet_tx` exactly like
|
||||
// the wildcard listen socket — rx_loop dispatches identically.
|
||||
let drain = crate::transport::udp::peer_drain::PeerRecvDrain::spawn(
|
||||
socket.clone(),
|
||||
transport_id,
|
||||
peer_socket_addr,
|
||||
packet_tx,
|
||||
)
|
||||
.map_err(|e| format!("PeerRecvDrain::spawn: {e}"))?;
|
||||
|
||||
// Install on the peer, idempotent re-check.
|
||||
if let Some(peer) = self.peers.get_mut(node_addr) {
|
||||
if peer.connected_udp().is_some() {
|
||||
// Lost the race — somebody else activated us first.
|
||||
// Drop the new socket + drain so we don't leak.
|
||||
drop(drain);
|
||||
drop(socket);
|
||||
return Ok(());
|
||||
}
|
||||
peer.set_connected_udp(socket, drain);
|
||||
crate::perf_profile::record_event(crate::perf_profile::Event::ConnectedUdpInstalled);
|
||||
info!(
|
||||
peer = %self.peer_display_name(node_addr),
|
||||
peer_addr = %peer_socket_addr,
|
||||
"connected UDP socket installed"
|
||||
);
|
||||
} else {
|
||||
// Peer disappeared between read-only pass and now.
|
||||
drop(drain);
|
||||
drop(socket);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Clear the per-peer connected UDP socket + drain for a peer.
|
||||
/// Called on peer disconnect / removal. The drain thread exits
|
||||
/// via self-pipe; the kernel fd closes when the last `Arc`
|
||||
/// drops.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[allow(dead_code)] // wired by session-deregister + rekey teardown follow-up
|
||||
pub(in crate::node) fn clear_connected_udp_for_peer(&mut self, node_addr: &NodeAddr) {
|
||||
if let Some(peer) = self.peers.get_mut(node_addr)
|
||||
&& peer.connected_udp().is_some()
|
||||
{
|
||||
peer.clear_connected_udp();
|
||||
debug!(peer = %self.peer_display_name(node_addr), "connected UDP socket cleared");
|
||||
}
|
||||
}
|
||||
|
||||
/// No-op shim for non-Linux builds so the rx_loop tick site can
|
||||
/// call us unconditionally.
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
#[allow(dead_code)] // wired by session-deregister + rekey teardown follow-up
|
||||
pub(in crate::node) fn clear_connected_udp_for_peer(&mut self, _node_addr: &NodeAddr) {}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
fn connected_udp_enabled() -> bool {
|
||||
env_flag("FIPS_CONNECTED_UDP").unwrap_or(true)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn connected_udp_enabled() -> bool {
|
||||
env_flag("FIPS_MACOS_CONNECTED_UDP")
|
||||
.or_else(|| env_flag("FIPS_CONNECTED_UDP"))
|
||||
.unwrap_or(true)
|
||||
}
|
||||
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
fn env_flag(name: &str) -> Option<bool> {
|
||||
let value = std::env::var(name).ok()?;
|
||||
match value.trim().to_ascii_lowercase().as_str() {
|
||||
"1" | "true" | "yes" | "on" => Some(true),
|
||||
"0" | "false" | "no" | "off" => Some(false),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
@@ -155,20 +155,32 @@ impl Node {
|
||||
// Free session indices (current, rekey, pending, previous)
|
||||
if let Some(tid) = transport_id {
|
||||
if let Some(idx) = peer.our_index() {
|
||||
self.peers_by_index.remove(&(tid, idx.as_u32()));
|
||||
let cache_key = (tid, idx.as_u32());
|
||||
self.peers_by_index.remove(&cache_key);
|
||||
#[cfg(unix)]
|
||||
self.unregister_decrypt_worker_session(cache_key);
|
||||
let _ = self.index_allocator.free(idx);
|
||||
}
|
||||
if let Some(idx) = peer.rekey_our_index() {
|
||||
self.pending_outbound.remove(&(tid, idx.as_u32()));
|
||||
self.peers_by_index.remove(&(tid, idx.as_u32()));
|
||||
let cache_key = (tid, idx.as_u32());
|
||||
self.pending_outbound.remove(&cache_key);
|
||||
self.peers_by_index.remove(&cache_key);
|
||||
#[cfg(unix)]
|
||||
self.unregister_decrypt_worker_session(cache_key);
|
||||
let _ = self.index_allocator.free(idx);
|
||||
}
|
||||
if let Some(idx) = peer.pending_our_index() {
|
||||
self.peers_by_index.remove(&(tid, idx.as_u32()));
|
||||
let cache_key = (tid, idx.as_u32());
|
||||
self.peers_by_index.remove(&cache_key);
|
||||
#[cfg(unix)]
|
||||
self.unregister_decrypt_worker_session(cache_key);
|
||||
let _ = self.index_allocator.free(idx);
|
||||
}
|
||||
if let Some(idx) = peer.previous_our_index() {
|
||||
self.peers_by_index.remove(&(tid, idx.as_u32()));
|
||||
let cache_key = (tid, idx.as_u32());
|
||||
self.peers_by_index.remove(&cache_key);
|
||||
#[cfg(unix)]
|
||||
self.unregister_decrypt_worker_session(cache_key);
|
||||
let _ = self.index_allocator.free(idx);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,7 +64,8 @@ impl Node {
|
||||
);
|
||||
|
||||
let peer = self.peers.get_mut(&node_addr).unwrap();
|
||||
if let Some(_old_our_index) = peer.handle_peer_kbit_flip() {
|
||||
let did_flip = peer.handle_peer_kbit_flip().is_some();
|
||||
if did_flip {
|
||||
// New index was pre-registered in peers_by_index during
|
||||
// msg1 handling (handshake.rs). Verify, don't duplicate.
|
||||
debug_assert!(
|
||||
@@ -77,6 +78,52 @@ impl Node {
|
||||
"peers_by_index should contain pre-registered new index after K-bit flip"
|
||||
);
|
||||
}
|
||||
// Re-register the (now-promoted) session with the decrypt
|
||||
// worker: cache_key = (transport_id, our_index) changed at
|
||||
// the flip, so the old worker entry is stranded and every
|
||||
// packet on the new session would miss the worker's
|
||||
// HashMap lookup. Without this, throughput drops back to
|
||||
// the inline-decrypt path after each rekey.
|
||||
#[cfg(unix)]
|
||||
if did_flip {
|
||||
self.register_decrypt_worker_session(&node_addr);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Decrypt-worker fast path (unix) ─────────────────────────
|
||||
// Once the session has been registered with a decrypt shard
|
||||
// (at FMP-establishment in `promote_connection`), the worker
|
||||
// owns the FMP recv cipher + replay window. Dispatch the
|
||||
// packet and return; the worker will run AEAD off-task and
|
||||
// bounce the plaintext back via `decrypt_fallback_tx` for
|
||||
// rx_loop to do the post-decrypt side-effects.
|
||||
//
|
||||
// The in-line decrypt below is the **synchronous test-mode
|
||||
// path** for unit tests that construct `Node` without
|
||||
// `lifecycle::start_async`; in production every established
|
||||
// session is dispatched to the worker.
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let cache_key = (packet.transport_id, header.receiver_idx.as_u32());
|
||||
if let Some(workers) = self.decrypt_workers.as_ref().cloned()
|
||||
&& self.decrypt_registered_sessions.contains(&cache_key)
|
||||
{
|
||||
let job = crate::node::decrypt_worker::DecryptJob {
|
||||
packet_data: packet.data,
|
||||
cache_key,
|
||||
_transport_id: packet.transport_id,
|
||||
_remote_addr: packet.remote_addr,
|
||||
timestamp_ms: packet.timestamp_ms,
|
||||
source_node_addr: node_addr,
|
||||
fmp_counter: header.counter,
|
||||
fmp_flags: header.flags,
|
||||
fmp_header: header.header_bytes,
|
||||
fmp_ciphertext_offset: header.ciphertext_offset(),
|
||||
fallback_tx: self.decrypt_fallback_tx.clone(),
|
||||
};
|
||||
workers.dispatch_job(job);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -214,6 +261,197 @@ impl Node {
|
||||
}
|
||||
}
|
||||
|
||||
/// Canonical post-FMP-decrypt side-effect site. Used by both the
|
||||
/// inline rx_loop decrypt path and the decrypt-worker bounce path
|
||||
/// so the per-peer bookkeeping (stats, MMP, spin-bit RTT, ECN
|
||||
/// propagation, address-rotation handling, link-message dispatch)
|
||||
/// happens in exactly one place.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(in crate::node) async fn process_authentic_fmp_plaintext(
|
||||
&mut self,
|
||||
node_addr: &crate::NodeAddr,
|
||||
transport_id: crate::transport::TransportId,
|
||||
remote_addr: &crate::transport::TransportAddr,
|
||||
packet_timestamp_ms: u64,
|
||||
packet_len: usize,
|
||||
fmp_counter: u64,
|
||||
ce_flag: bool,
|
||||
sp_flag: bool,
|
||||
fmp_plaintext: &[u8],
|
||||
) {
|
||||
const INNER_TIMESTAMP_LEN: usize = 4;
|
||||
let inner_ts = if fmp_plaintext.len() >= INNER_TIMESTAMP_LEN {
|
||||
u32::from_le_bytes([
|
||||
fmp_plaintext[0],
|
||||
fmp_plaintext[1],
|
||||
fmp_plaintext[2],
|
||||
fmp_plaintext[3],
|
||||
])
|
||||
} else {
|
||||
return;
|
||||
};
|
||||
let now = Instant::now();
|
||||
let mut address_changed = false;
|
||||
if let Some(peer) = self.peers.get_mut(node_addr) {
|
||||
peer.reset_decrypt_failures();
|
||||
address_changed = peer.set_current_addr(transport_id, remote_addr.clone());
|
||||
peer.link_stats_mut()
|
||||
.record_recv(packet_len, packet_timestamp_ms);
|
||||
peer.touch(packet_timestamp_ms);
|
||||
if let Some(mmp) = peer.mmp_mut() {
|
||||
mmp.receiver
|
||||
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now);
|
||||
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now);
|
||||
}
|
||||
}
|
||||
// Address rotation invalidates the per-peer connect()-ed UDP
|
||||
// socket. Drop the connected socket + drain so the wildcard
|
||||
// listen socket takes over until the new 5-tuple settles.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if address_changed {
|
||||
self.clear_connected_udp_for_peer(node_addr);
|
||||
}
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
{
|
||||
let _ = address_changed;
|
||||
}
|
||||
let link_message = &fmp_plaintext[INNER_TIMESTAMP_LEN..];
|
||||
self.dispatch_link_message(node_addr, link_message, ce_flag)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Process a decrypt-worker bounce (FMP plaintext only — the
|
||||
/// worker has already done the AEAD + replay check).
|
||||
#[cfg(unix)]
|
||||
pub(in crate::node) async fn process_decrypt_fallback(
|
||||
&mut self,
|
||||
fallback: crate::node::decrypt_worker::DecryptFallback,
|
||||
) {
|
||||
let ce_flag = fallback.fmp_flags & FLAG_CE != 0;
|
||||
let sp_flag = fallback.fmp_flags & FLAG_SP != 0;
|
||||
let plaintext = &fallback.packet_data[fallback.fmp_plaintext_offset
|
||||
..fallback.fmp_plaintext_offset + fallback.fmp_plaintext_len];
|
||||
self.process_authentic_fmp_plaintext(
|
||||
&fallback.source_node_addr,
|
||||
fallback.transport_id,
|
||||
&fallback.remote_addr,
|
||||
fallback.timestamp_ms,
|
||||
fallback.packet_len,
|
||||
fallback.fmp_counter,
|
||||
ce_flag,
|
||||
sp_flag,
|
||||
plaintext,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Process a decrypt-worker failure event.
|
||||
#[cfg(unix)]
|
||||
pub(in crate::node) async fn process_decrypt_failure_report(
|
||||
&mut self,
|
||||
report: crate::node::decrypt_worker::DecryptFailureReport,
|
||||
) {
|
||||
debug!(
|
||||
peer = %self.peer_display_name(&report.source_node_addr),
|
||||
counter = report.fmp_counter,
|
||||
replay_highest = report.fmp_replay_highest,
|
||||
"Worker FMP AEAD decryption failed"
|
||||
);
|
||||
self.handle_decrypt_failure(&report.source_node_addr);
|
||||
}
|
||||
|
||||
/// Dispatch a decrypt-worker event (plaintext bounce or failure
|
||||
/// report) to the appropriate handler.
|
||||
#[cfg(unix)]
|
||||
pub(in crate::node) async fn process_decrypt_worker_event(
|
||||
&mut self,
|
||||
event: crate::node::decrypt_worker::DecryptWorkerEvent,
|
||||
) {
|
||||
match event {
|
||||
crate::node::decrypt_worker::DecryptWorkerEvent::Plaintext(fallback) => {
|
||||
self.process_decrypt_fallback(fallback).await;
|
||||
}
|
||||
crate::node::decrypt_worker::DecryptWorkerEvent::DecryptFailure(report) => {
|
||||
self.process_decrypt_failure_report(report).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Hand a session's FMP recv cipher + replay window off to a shard
|
||||
/// of the decrypt worker pool. Idempotent on rekey: re-registering
|
||||
/// the same cache_key overwrites the worker's entry. Gates the
|
||||
/// `decrypt_registered_sessions` insert on actual worker acceptance
|
||||
/// so a `TrySendError::Full` on the per-worker channel doesn't
|
||||
/// black-hole the session.
|
||||
#[cfg(unix)]
|
||||
pub(in crate::node) fn register_decrypt_worker_session(&mut self, node_addr: &crate::NodeAddr) {
|
||||
let Some(workers) = self.decrypt_workers.as_ref().cloned() else {
|
||||
return;
|
||||
};
|
||||
let (cache_key, state) = {
|
||||
let Some(peer) = self.peers.get(node_addr) else {
|
||||
return;
|
||||
};
|
||||
let Some(transport_id) = peer.transport_id() else {
|
||||
return;
|
||||
};
|
||||
let Some(our_index) = peer.our_index() else {
|
||||
return;
|
||||
};
|
||||
let cache_key = (transport_id, our_index.as_u32());
|
||||
let Some(state) = self.build_owned_session_state(node_addr) else {
|
||||
return;
|
||||
};
|
||||
(cache_key, state)
|
||||
};
|
||||
if workers.register_session(cache_key, state) {
|
||||
self.decrypt_registered_sessions.insert(cache_key);
|
||||
}
|
||||
}
|
||||
|
||||
/// Drop a session from the decrypt worker pool. Mirror of
|
||||
/// `register_decrypt_worker_session`. Idempotent: safe to call on a
|
||||
/// `cache_key` that isn't registered, and safe to call when the
|
||||
/// worker pool is disabled (`FIPS_DECRYPT_WORKERS=0`).
|
||||
///
|
||||
/// Called from two sites that already iterate `peers_by_index`:
|
||||
/// the rekey drain-completion block (after the drain window has
|
||||
/// expired, the old `our_index` is unreachable to any in-flight
|
||||
/// OLD-K packet) and `remove_active_peer` (terminal peer cleanup).
|
||||
/// Without these callers, the per-worker `sessions` HashMap and
|
||||
/// the Node's `decrypt_registered_sessions` set would grow
|
||||
/// monotonically per rekey on long-lived peers.
|
||||
#[cfg(unix)]
|
||||
pub(in crate::node) fn unregister_decrypt_worker_session(
|
||||
&mut self,
|
||||
cache_key: (crate::transport::TransportId, u32),
|
||||
) {
|
||||
if let Some(workers) = self.decrypt_workers.as_ref() {
|
||||
workers.unregister_session(cache_key);
|
||||
}
|
||||
self.decrypt_registered_sessions.remove(&cache_key);
|
||||
}
|
||||
|
||||
/// Snapshot the per-peer FMP recv cipher + replay window for the
|
||||
/// decrypt worker. Returns `None` if the peer / session isn't
|
||||
/// ready. After hand-off the worker is the sole FMP replay-window
|
||||
/// authority for this session.
|
||||
#[cfg(unix)]
|
||||
fn build_owned_session_state(
|
||||
&self,
|
||||
node_addr: &crate::NodeAddr,
|
||||
) -> Option<crate::node::decrypt_worker::OwnedSessionState> {
|
||||
let peer = self.peers.get(node_addr)?;
|
||||
let fmp_session = peer.noise_session()?;
|
||||
let fmp_cipher = fmp_session.recv_cipher_clone()?;
|
||||
let fmp_replay = fmp_session.recv_replay_snapshot_owned();
|
||||
Some(crate::node::decrypt_worker::OwnedSessionState {
|
||||
fmp_cipher,
|
||||
fmp_replay,
|
||||
source_npub: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// Increment decrypt failure counter and force-remove peer if threshold exceeded.
|
||||
pub(in crate::node) fn handle_decrypt_failure(&mut self, node_addr: &crate::NodeAddr) {
|
||||
if let Some(peer) = self.peers.get_mut(node_addr) {
|
||||
|
||||
@@ -1145,6 +1145,14 @@ impl Node {
|
||||
"Connection promoted to active peer"
|
||||
);
|
||||
|
||||
// Hand the FMP recv cipher + replay window to the
|
||||
// decrypt shard worker. From this point on the worker
|
||||
// is the sole authority on FMP replay protection for
|
||||
// this session. No-op when the worker pool isn't
|
||||
// spawned (unit-test path or `FIPS_DECRYPT_WORKERS=0`).
|
||||
#[cfg(unix)]
|
||||
self.register_decrypt_worker_session(&peer_node_addr);
|
||||
|
||||
Ok(PromotionResult::Promoted(peer_node_addr))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
//! RX event loop and message handlers.
|
||||
|
||||
#[cfg(unix)]
|
||||
pub(crate) mod connected_udp;
|
||||
pub(crate) mod discovery;
|
||||
mod dispatch;
|
||||
mod encrypted;
|
||||
|
||||
@@ -85,35 +85,57 @@ impl Node {
|
||||
|
||||
// Execute cutover for initiator side
|
||||
for node_addr in peers_to_cutover {
|
||||
if let Some(peer) = self.peers.get_mut(&node_addr)
|
||||
&& let Some(_old_our_index) = peer.cutover_to_new_session()
|
||||
{
|
||||
// New index was pre-registered in peers_by_index during
|
||||
// msg2 handling (handshake.rs). Verify, don't duplicate.
|
||||
debug_assert!(
|
||||
peer.transport_id().is_some()
|
||||
&& peer.our_index().is_some()
|
||||
&& self.peers_by_index.contains_key(&(
|
||||
peer.transport_id().unwrap(),
|
||||
peer.our_index().unwrap().as_u32()
|
||||
)),
|
||||
"peers_by_index should contain pre-registered new index after cutover"
|
||||
);
|
||||
debug!(
|
||||
peer = %self.peer_display_name(&node_addr),
|
||||
"Rekey cutover complete (initiator), K-bit flipped"
|
||||
);
|
||||
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
|
||||
if let Some(_old_our_index) = peer.cutover_to_new_session() {
|
||||
// New index was pre-registered in peers_by_index
|
||||
// during msg2 handling (handshake.rs).
|
||||
debug_assert!(
|
||||
peer.transport_id().is_some()
|
||||
&& peer.our_index().is_some()
|
||||
&& self.peers_by_index.contains_key(&(
|
||||
peer.transport_id().unwrap(),
|
||||
peer.our_index().unwrap().as_u32()
|
||||
)),
|
||||
"peers_by_index should contain pre-registered new index after cutover"
|
||||
);
|
||||
debug!(
|
||||
peer = %self.peer_display_name(&node_addr),
|
||||
"Rekey cutover complete (initiator), K-bit flipped"
|
||||
);
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
} else {
|
||||
false
|
||||
};
|
||||
// Re-register the new session with the decrypt worker — the
|
||||
// cache_key (transport_id, our_index) just changed, so the
|
||||
// old worker entry is stale and every packet on the new
|
||||
// session would miss the worker's HashMap lookup.
|
||||
#[cfg(unix)]
|
||||
if did_cutover {
|
||||
self.register_decrypt_worker_session(&node_addr);
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
let _ = did_cutover;
|
||||
}
|
||||
|
||||
// Execute drain completion
|
||||
for node_addr in peers_to_drain {
|
||||
if let Some(peer) = self.peers.get_mut(&node_addr)
|
||||
&& let Some(old_our_index) = peer.complete_drain()
|
||||
{
|
||||
if let Some(transport_id) = peer.transport_id() {
|
||||
self.peers_by_index
|
||||
.remove(&(transport_id, old_our_index.as_u32()));
|
||||
// Extract the old index and transport_id under the peer
|
||||
// borrow, then drop the borrow so the cache_key cleanup
|
||||
// below can take &mut self for unregister_decrypt_worker_session.
|
||||
let drained = self
|
||||
.peers
|
||||
.get_mut(&node_addr)
|
||||
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
|
||||
if let Some((old_our_index, transport_id)) = drained {
|
||||
if let Some(tid) = transport_id {
|
||||
let cache_key = (tid, old_our_index.as_u32());
|
||||
self.peers_by_index.remove(&cache_key);
|
||||
#[cfg(unix)]
|
||||
self.unregister_decrypt_worker_session(cache_key);
|
||||
}
|
||||
let _ = self.index_allocator.free(old_our_index);
|
||||
trace!(
|
||||
|
||||
@@ -10,6 +10,18 @@ use crate::transport::ReceivedPacket;
|
||||
use std::time::Duration;
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
/// Inside the packet_rx burst drain, run a fallback drain every
|
||||
/// N packets so bounced FMP plaintexts can't sit behind a full
|
||||
/// 256-packet UDP burst. Used on unix; on Windows the decrypt-worker
|
||||
/// pool isn't spawned so the fallback channel is always empty —
|
||||
/// hold the constants at module scope anyway so the burst-loop
|
||||
/// dispatch in `run_rx_loop` doesn't need a `#[cfg]` on every site.
|
||||
const FALLBACK_INTERLEAVE_EVERY: usize = 32;
|
||||
/// How many fallback events to drain per interleave step. Bounded so
|
||||
/// the inner loop can keep making forward progress on packet_rx.
|
||||
#[cfg(unix)]
|
||||
const FALLBACK_INTERLEAVE_BUDGET: usize = 32;
|
||||
|
||||
impl Node {
|
||||
/// Run the receive event loop.
|
||||
///
|
||||
@@ -78,11 +90,64 @@ impl Node {
|
||||
// Drop unused sender to avoid keeping channel open if control is disabled
|
||||
drop(control_tx);
|
||||
|
||||
// Decrypt-worker fallback receiver. The worker pushes each
|
||||
// authenticated FMP plaintext here so rx_loop can finish the
|
||||
// per-peer side-effects (stats, MMP, ECN, link dispatch).
|
||||
// Always declared so the `tokio::select!` arm doesn't need
|
||||
// a `cfg` (which the macro doesn't support); on Windows the
|
||||
// channel just never sees events.
|
||||
let (mut decrypt_fallback_rx, _decrypt_fallback_guard) = {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
match self.decrypt_fallback_rx.take() {
|
||||
Some(rx) => (rx, None),
|
||||
None => {
|
||||
let (tx, rx) = tokio::sync::mpsc::unbounded_channel();
|
||||
(rx, Some(tx))
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
// On non-unix nothing ever sends, but the macro arm
|
||||
// still needs an existing rx. Keep the sender alive to
|
||||
// avoid the channel closing into an Err loop.
|
||||
let (tx, rx) = tokio::sync::mpsc::unbounded_channel::<()>();
|
||||
(rx, Some(tx))
|
||||
}
|
||||
};
|
||||
|
||||
info!("RX event loop started");
|
||||
// Optional per-stage perf profiler (FIPS_PERF=1). No-op otherwise.
|
||||
crate::perf_profile::maybe_spawn_reporter();
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
biased;
|
||||
// Decrypt-worker fallback drains FIRST. Under sustained
|
||||
// inbound bursts the packet_rx drain (up to 256 packets)
|
||||
// can starve fallback work for tens of ms — TCP doesn't
|
||||
// tolerate that (late ACKs → dup-ACK fast retransmits →
|
||||
// cwnd collapse). Promoting fallback gives the kernel's
|
||||
// TCP machinery a fair chance to ACK in time.
|
||||
Some(event) = decrypt_fallback_rx.recv() => {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
self.process_decrypt_worker_event(event).await;
|
||||
let mut drained = 0;
|
||||
while drained < 255 {
|
||||
match decrypt_fallback_rx.try_recv() {
|
||||
Ok(ev) => {
|
||||
self.process_decrypt_worker_event(ev).await;
|
||||
drained += 1;
|
||||
}
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
let _ = event;
|
||||
}
|
||||
packet = packet_rx.recv() => {
|
||||
match packet {
|
||||
Some(p) => self.process_packet(p).await,
|
||||
@@ -95,8 +160,35 @@ impl Node {
|
||||
// datagrams available per wake. Caps at a batch
|
||||
// boundary so other branches (tick, control) eventually
|
||||
// get a turn even under sustained load.
|
||||
let mut drained = 0;
|
||||
//
|
||||
// **Interleave fallback drain** every N packets so
|
||||
// bounced FMP plaintexts (heartbeats, post-FMP-
|
||||
// decrypt forwarding payloads, control frames) don't
|
||||
// sit in the fallback queue for a full 256-packet
|
||||
// burst. Even with the priority-first ordering of
|
||||
// the outer select!, once we're inside this inner
|
||||
// loop only this interleave can free queued
|
||||
// fallbacks. On multihop forwarding paths this is
|
||||
// the difference between back-to-back encrypt-
|
||||
// worker dispatches happening promptly vs piling up
|
||||
// behind the rx burst.
|
||||
let mut drained: usize = 1; // count the packet processed above
|
||||
while drained < 256 {
|
||||
if drained.is_multiple_of(FALLBACK_INTERLEAVE_EVERY) {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let mut fb_drained = 0;
|
||||
while fb_drained < FALLBACK_INTERLEAVE_BUDGET {
|
||||
match decrypt_fallback_rx.try_recv() {
|
||||
Ok(ev) => {
|
||||
self.process_decrypt_worker_event(ev).await;
|
||||
fb_drained += 1;
|
||||
}
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
match packet_rx.try_recv() {
|
||||
Ok(p) => {
|
||||
self.process_packet(p).await;
|
||||
@@ -105,6 +197,22 @@ impl Node {
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
// Trailing fallback drain so the last bounced
|
||||
// packets of the burst aren't held up by the
|
||||
// next select! iteration.
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let mut fb_drained = 0;
|
||||
while fb_drained < 256 {
|
||||
match decrypt_fallback_rx.try_recv() {
|
||||
Ok(ev) => {
|
||||
self.process_decrypt_worker_event(ev).await;
|
||||
fb_drained += 1;
|
||||
}
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(ipv6_packet) = tun_outbound_rx.recv() => {
|
||||
self.handle_tun_outbound(ipv6_packet).await;
|
||||
@@ -161,6 +269,8 @@ impl Node {
|
||||
self.check_pending_lookups(now_ms).await;
|
||||
self.poll_transport_discovery().await;
|
||||
self.sample_transport_congestion();
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
self.activate_connected_udp_sessions().await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,20 +15,45 @@ use crate::node::session_wire::{
|
||||
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
|
||||
fsp_strip_inner_header, parse_encrypted_coords,
|
||||
};
|
||||
#[cfg(unix)]
|
||||
use crate::node::wire::{
|
||||
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
|
||||
};
|
||||
use crate::node::{Node, NodeError};
|
||||
use crate::noise::{
|
||||
HandshakeState, XK_HANDSHAKE_MSG1_SIZE, XK_HANDSHAKE_MSG2_SIZE, XK_HANDSHAKE_MSG3_SIZE,
|
||||
};
|
||||
#[cfg(unix)]
|
||||
use crate::protocol::LinkMessageType;
|
||||
#[cfg(unix)]
|
||||
use crate::protocol::SESSION_DATAGRAM_HEADER_SIZE;
|
||||
use crate::protocol::{
|
||||
CoordsRequired, FspInnerFlags, MtuExceeded, PathBroken, PathMtuNotification, SessionAck,
|
||||
SessionDatagram, SessionMessageType, SessionMsg3, SessionReceiverReport, SessionSenderReport,
|
||||
SessionSetup,
|
||||
};
|
||||
use crate::protocol::{coords_wire_size, encode_coords};
|
||||
#[cfg(unix)]
|
||||
use crate::transport::TransportHandle;
|
||||
use crate::upper::icmp::FIPS_OVERHEAD;
|
||||
use secp256k1::PublicKey;
|
||||
use tracing::{debug, info, trace, warn};
|
||||
|
||||
/// Inputs to `try_send_session_data_pipelined` — the FSP+FMP pipelined
|
||||
/// fast path that hands both AEAD operations to the encrypt worker
|
||||
/// in a single dispatch.
|
||||
#[cfg(unix)]
|
||||
struct PipelinedSend<'a> {
|
||||
dest_addr: &'a NodeAddr,
|
||||
payload: &'a [u8],
|
||||
now_ms: u64,
|
||||
timestamp: u32,
|
||||
fsp_flags: u8,
|
||||
inner_plaintext: &'a [u8],
|
||||
my_coords: Option<&'a crate::tree::TreeCoordinate>,
|
||||
dest_coords: Option<&'a crate::tree::TreeCoordinate>,
|
||||
}
|
||||
|
||||
impl Node {
|
||||
/// Handle a locally-delivered session datagram payload.
|
||||
///
|
||||
@@ -1392,6 +1417,29 @@ impl Node {
|
||||
flags |= FSP_FLAG_K;
|
||||
}
|
||||
|
||||
// ── Pipelined FSP+FMP fast path (unix + UDP) ──
|
||||
// Both AEAD layers + the sendmmsg syscall run on the encrypt
|
||||
// worker. The rx_loop only builds the wire buffer + reserves
|
||||
// counters on both sessions. Falls through to the legacy
|
||||
// sync FSP encrypt + send_session_datagram path on any
|
||||
// prereq miss (non-UDP, no worker, no cipher, …).
|
||||
#[cfg(unix)]
|
||||
if self
|
||||
.try_send_session_data_pipelined(PipelinedSend {
|
||||
dest_addr,
|
||||
payload,
|
||||
now_ms,
|
||||
timestamp,
|
||||
fsp_flags: flags,
|
||||
inner_plaintext: &inner_plaintext,
|
||||
my_coords: my_coords.as_ref(),
|
||||
dest_coords: dest_coords.as_ref(),
|
||||
})
|
||||
.await?
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Borrow session for counter + encryption (after potential standalone send)
|
||||
let entry = self
|
||||
.sessions
|
||||
@@ -1450,6 +1498,282 @@ impl Node {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Pipelined send: FSP+FMP AEAD + sendmsg on the encrypt worker.
|
||||
///
|
||||
/// Returns `Ok(true)` when the worker accepted the job (caller is
|
||||
/// done), `Ok(false)` on prereq miss (non-UDP, no worker, no
|
||||
/// cipher) so the caller falls back to the legacy synchronous
|
||||
/// path. `Err` on routing / state errors that prevent any send.
|
||||
///
|
||||
/// Wire layout built directly (no intermediate `inner_plaintext`
|
||||
/// or `fsp_payload` Vecs):
|
||||
/// ```text
|
||||
/// [16 FMP header][8 link-ts][1 SessionDatagram][1 ttl][2 mtu]
|
||||
/// [16 src_addr][16 dest_addr][12 FSP header][coords?]
|
||||
/// [FSP plaintext] <- worker seals here, appends FSP tag,
|
||||
/// then seals the full FMP plaintext and
|
||||
/// appends the FMP tag.
|
||||
/// ```
|
||||
#[cfg(unix)]
|
||||
async fn try_send_session_data_pipelined(
|
||||
&mut self,
|
||||
send: PipelinedSend<'_>,
|
||||
) -> Result<bool, NodeError> {
|
||||
let dest_addr = send.dest_addr;
|
||||
let Some(workers) = self.encrypt_workers.as_ref().cloned() else {
|
||||
return Ok(false);
|
||||
};
|
||||
|
||||
let Some(next_hop_addr) = self.find_next_hop(dest_addr).map(|peer| *peer.node_addr())
|
||||
else {
|
||||
return Err(NodeError::SendFailed {
|
||||
node_addr: *dest_addr,
|
||||
reason: "no route to destination".into(),
|
||||
});
|
||||
};
|
||||
|
||||
// Read the next hop's transport / link MTU for the
|
||||
// SessionDatagram's path_mtu field. Saves a path_mtu round-
|
||||
// trip vs the legacy path where we'd seed in
|
||||
// send_session_datagram.
|
||||
let mut path_mtu = u16::MAX;
|
||||
if let Some(peer) = self.peers.get(&next_hop_addr)
|
||||
&& let Some(tid) = peer.transport_id()
|
||||
&& let Some(transport) = self.transports.get(&tid)
|
||||
{
|
||||
if let Some(addr) = peer.current_addr() {
|
||||
path_mtu = path_mtu.min(transport.link_mtu(addr));
|
||||
} else {
|
||||
path_mtu = path_mtu.min(transport.mtu());
|
||||
}
|
||||
}
|
||||
|
||||
// Extract next-hop info + FMP cipher in one peers/sessions borrow scope.
|
||||
let (their_index, transport_id, remote_addr, timestamp_ms, fmp_flags, fmp_cipher) = {
|
||||
let peer = self
|
||||
.peers
|
||||
.get_mut(&next_hop_addr)
|
||||
.ok_or(NodeError::PeerNotFound(next_hop_addr))?;
|
||||
let their_index = peer.their_index().ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: "no their_index".into(),
|
||||
})?;
|
||||
let transport_id = peer.transport_id().ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: "no transport_id".into(),
|
||||
})?;
|
||||
let remote_addr =
|
||||
peer.current_addr()
|
||||
.cloned()
|
||||
.ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: "no current_addr".into(),
|
||||
})?;
|
||||
let timestamp_ms = peer.session_elapsed_ms();
|
||||
let sp_flag = peer.mmp().map(|m| m.spin_bit.tx_bit()).unwrap_or(false);
|
||||
let mut fmp_flags = if sp_flag { FLAG_SP } else { 0 };
|
||||
if peer.current_k_bit() {
|
||||
fmp_flags |= FLAG_KEY_EPOCH;
|
||||
}
|
||||
let session = peer
|
||||
.noise_session_mut()
|
||||
.ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: "no noise session".into(),
|
||||
})?;
|
||||
let Some(fmp_cipher) = session.send_cipher_clone() else {
|
||||
return Ok(false);
|
||||
};
|
||||
(
|
||||
their_index,
|
||||
transport_id,
|
||||
remote_addr,
|
||||
timestamp_ms,
|
||||
fmp_flags,
|
||||
fmp_cipher,
|
||||
)
|
||||
};
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let connected_socket = self
|
||||
.peers
|
||||
.get(&next_hop_addr)
|
||||
.and_then(|peer| peer.connected_udp());
|
||||
|
||||
// Need a UDP transport (this whole path is sendmsg-on-raw-fd).
|
||||
let transport = self
|
||||
.transports
|
||||
.get(&transport_id)
|
||||
.ok_or(NodeError::TransportNotFound(transport_id))?;
|
||||
let TransportHandle::Udp(udp) = transport else {
|
||||
return Ok(false);
|
||||
};
|
||||
let socket_addr_opt = {
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
match connected_socket.as_ref() {
|
||||
Some(s) => Some(s.peer_addr()),
|
||||
None => udp.resolve_for_off_task(&remote_addr).await.ok(),
|
||||
}
|
||||
}
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
{
|
||||
udp.resolve_for_off_task(&remote_addr).await.ok()
|
||||
}
|
||||
};
|
||||
let Some(socket_addr) = socket_addr_opt else {
|
||||
return Ok(false);
|
||||
};
|
||||
let Some(socket) = udp.async_socket() else {
|
||||
return Ok(false);
|
||||
};
|
||||
|
||||
// FSP cipher + counter — separate session from next-hop FMP session.
|
||||
let (fsp_counter, fsp_cipher) = {
|
||||
let entry = self
|
||||
.sessions
|
||||
.get_mut(dest_addr)
|
||||
.ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: *dest_addr,
|
||||
reason: "no session".into(),
|
||||
})?;
|
||||
if let Some(mmp) = entry.mmp_mut() {
|
||||
mmp.path_mtu.seed_source_mtu(path_mtu);
|
||||
}
|
||||
let session = match entry.state_mut() {
|
||||
EndToEndState::Established(s) => s,
|
||||
_ => {
|
||||
return Err(NodeError::SendFailed {
|
||||
node_addr: *dest_addr,
|
||||
reason: "session not established".into(),
|
||||
});
|
||||
}
|
||||
};
|
||||
let Some(fsp_cipher) = session.send_cipher_clone() else {
|
||||
return Ok(false);
|
||||
};
|
||||
let counter = session
|
||||
.take_send_counter()
|
||||
.map_err(|e| NodeError::SendFailed {
|
||||
node_addr: *dest_addr,
|
||||
reason: format!("session counter reservation failed: {}", e),
|
||||
})?;
|
||||
(counter, fsp_cipher)
|
||||
};
|
||||
|
||||
// FSP header (the AAD for the inner AEAD seal).
|
||||
let fsp_header = build_fsp_header(
|
||||
fsp_counter,
|
||||
send.fsp_flags,
|
||||
send.inner_plaintext.len() as u16,
|
||||
);
|
||||
|
||||
let coords_size = match (send.my_coords, send.dest_coords) {
|
||||
(Some(src), Some(dst)) => coords_wire_size(src) + coords_wire_size(dst),
|
||||
_ => 0,
|
||||
};
|
||||
// FMP inner = [link-ts u64][SessionDatagram-encoded after msg_type] + [FSP-encrypted blob]
|
||||
// SessionDatagram-encoded includes: [0x00 type][ttl][mtu][src][dest][fsp_payload]
|
||||
// fsp_payload = [fsp_header][coords][fsp_plaintext][16-byte FSP tag]
|
||||
let link_plaintext_len = SESSION_DATAGRAM_HEADER_SIZE
|
||||
+ FSP_HEADER_SIZE
|
||||
+ coords_size
|
||||
+ send.inner_plaintext.len();
|
||||
let fmp_inner_len = 4 + link_plaintext_len + crate::noise::TAG_SIZE;
|
||||
|
||||
// FMP counter + header (the AAD for the outer AEAD seal).
|
||||
let fmp_counter = {
|
||||
let peer = self
|
||||
.peers
|
||||
.get_mut(&next_hop_addr)
|
||||
.ok_or(NodeError::PeerNotFound(next_hop_addr))?;
|
||||
let session = peer
|
||||
.noise_session_mut()
|
||||
.ok_or_else(|| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: "no noise session".into(),
|
||||
})?;
|
||||
session
|
||||
.take_send_counter()
|
||||
.map_err(|e| NodeError::SendFailed {
|
||||
node_addr: next_hop_addr,
|
||||
reason: format!("counter reservation failed: {}", e),
|
||||
})?
|
||||
};
|
||||
let fmp_header =
|
||||
build_established_header(their_index, fmp_counter, fmp_flags, fmp_inner_len as u16);
|
||||
|
||||
// Build the wire buffer once, in place. The two FSP/FMP tags
|
||||
// are appended by the worker after the seals — reserve their
|
||||
// capacity here so the worker doesn't have to re-grow.
|
||||
let wire_capacity = ESTABLISHED_HEADER_SIZE + fmp_inner_len + crate::noise::TAG_SIZE;
|
||||
let mut wire_buf = Vec::with_capacity(wire_capacity);
|
||||
wire_buf.extend_from_slice(&fmp_header);
|
||||
wire_buf.extend_from_slice(×tamp_ms.to_le_bytes());
|
||||
// SessionDatagram-encoded layout (matches `SessionDatagram::encode`):
|
||||
wire_buf.push(LinkMessageType::SessionDatagram.to_byte());
|
||||
wire_buf.push(self.config.node.session.default_ttl);
|
||||
wire_buf.extend_from_slice(&path_mtu.to_le_bytes());
|
||||
wire_buf.extend_from_slice(self.node_addr().as_bytes());
|
||||
wire_buf.extend_from_slice(dest_addr.as_bytes());
|
||||
// FSP layer (worker seals on `inner_plaintext` portion):
|
||||
let fsp_aad_offset = wire_buf.len();
|
||||
wire_buf.extend_from_slice(&fsp_header);
|
||||
if let (Some(src), Some(dst)) = (send.my_coords, send.dest_coords) {
|
||||
encode_coords(src, &mut wire_buf);
|
||||
encode_coords(dst, &mut wire_buf);
|
||||
}
|
||||
let fsp_plaintext_offset = wire_buf.len();
|
||||
wire_buf.extend_from_slice(send.inner_plaintext);
|
||||
|
||||
// Stats update — predict size exactly (ChaCha20-Poly1305 tag
|
||||
// is constant 16 bytes).
|
||||
let predicted_bytes = wire_capacity;
|
||||
if let Some(peer) = self.peers.get_mut(&next_hop_addr) {
|
||||
peer.link_stats_mut().record_sent(predicted_bytes);
|
||||
if let Some(mmp) = peer.mmp_mut() {
|
||||
mmp.sender
|
||||
.record_sent(fmp_counter, timestamp_ms, predicted_bytes);
|
||||
}
|
||||
}
|
||||
self.stats_mut()
|
||||
.forwarding
|
||||
.record_originated(link_plaintext_len + crate::noise::TAG_SIZE);
|
||||
|
||||
if let Some(entry) = self.sessions.get_mut(dest_addr) {
|
||||
entry.record_sent(send.payload.len());
|
||||
if let Some(mmp) = entry.mmp_mut() {
|
||||
mmp.sender.record_sent(
|
||||
fsp_counter,
|
||||
send.timestamp,
|
||||
send.inner_plaintext.len() + crate::noise::TAG_SIZE,
|
||||
);
|
||||
}
|
||||
entry.touch(send.now_ms);
|
||||
}
|
||||
|
||||
workers.dispatch(crate::node::encrypt_worker::FmpSendJob {
|
||||
cipher: fmp_cipher,
|
||||
counter: fmp_counter,
|
||||
wire_buf,
|
||||
fsp_seal: Some(crate::node::encrypt_worker::FspSealJob {
|
||||
cipher: fsp_cipher,
|
||||
counter: fsp_counter,
|
||||
aad_offset: fsp_aad_offset,
|
||||
plaintext_offset: fsp_plaintext_offset,
|
||||
}),
|
||||
socket,
|
||||
dest_addr: socket_addr,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
connected_socket,
|
||||
// Bulk endpoint data: drop on UDP backpressure so the
|
||||
// worker queue keeps moving instead of stranding under
|
||||
// sustained congestion.
|
||||
drop_on_backpressure: true,
|
||||
queued_at: None,
|
||||
});
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Send an IPv6 packet through the IPv6 shim (port 256) with header compression.
|
||||
///
|
||||
/// Compresses the IPv6 header (format 0x00), then sends via `send_session_data`
|
||||
|
||||
@@ -680,6 +680,52 @@ impl Node {
|
||||
info!(count = self.transports.len(), "Transports initialized");
|
||||
}
|
||||
|
||||
// Spawn the off-task FMP-encrypt + UDP-send worker pool.
|
||||
// Unix only — the worker issues sendmmsg(2) / sendmsg+UDP_GSO
|
||||
// calls on raw fds via `AsRawFd`, a unix-only trait. Worker
|
||||
// count defaults to num_cpus, overridable via FIPS_ENCRYPT_WORKERS.
|
||||
// Hash-by-destination pins a TCP flow to one worker (preserves
|
||||
// wire ordering); additional workers light up under multi-flow load.
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let cpu_default = std::thread::available_parallelism()
|
||||
.map(|n| n.get())
|
||||
.unwrap_or(1)
|
||||
.max(1);
|
||||
let encrypt_worker_count: usize = std::env::var("FIPS_ENCRYPT_WORKERS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.unwrap_or(cpu_default)
|
||||
.max(1);
|
||||
self.encrypt_workers = Some(super::encrypt_worker::EncryptWorkerPool::spawn(
|
||||
encrypt_worker_count,
|
||||
));
|
||||
info!(
|
||||
workers = encrypt_worker_count,
|
||||
"Spawned FMP-encrypt worker pool"
|
||||
);
|
||||
|
||||
// `FIPS_DECRYPT_WORKERS=0` disables the pool entirely and
|
||||
// forces the in-line rx_loop decrypt path (useful as an A/B
|
||||
// against the worker pipeline). Any non-zero value (env or
|
||||
// default) spawns the shard-owned decrypt pool.
|
||||
let decrypt_worker_count: usize = std::env::var("FIPS_DECRYPT_WORKERS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.unwrap_or(cpu_default);
|
||||
if decrypt_worker_count == 0 {
|
||||
info!("FIPS_DECRYPT_WORKERS=0 → in-line decrypt in rx_loop");
|
||||
} else {
|
||||
self.decrypt_workers = Some(super::decrypt_worker::DecryptWorkerPool::spawn(
|
||||
decrypt_worker_count,
|
||||
));
|
||||
info!(
|
||||
workers = decrypt_worker_count,
|
||||
"Spawned FMP-decrypt worker pool"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if self.config.node.discovery.nostr.enabled {
|
||||
match NostrDiscovery::start(&self.identity, self.config.node.discovery.nostr.clone())
|
||||
.await
|
||||
|
||||
164
src/node/mod.rs
164
src/node/mod.rs
@@ -6,7 +6,11 @@
|
||||
|
||||
mod acl;
|
||||
mod bloom;
|
||||
#[cfg(unix)]
|
||||
pub(crate) mod decrypt_worker;
|
||||
mod discovery_rate_limit;
|
||||
#[cfg(unix)]
|
||||
pub(crate) mod encrypt_worker;
|
||||
mod handlers;
|
||||
mod lifecycle;
|
||||
mod rate_limit;
|
||||
@@ -32,8 +36,8 @@ use self::routing_error_rate_limit::RoutingErrorRateLimiter;
|
||||
/// `node.rekey.after_secs` remains the nominal interval (mean preserved).
|
||||
pub(crate) const REKEY_JITTER_SECS: i64 = 15;
|
||||
use self::wire::{
|
||||
FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, build_encrypted, build_established_header,
|
||||
prepend_inner_header,
|
||||
ESTABLISHED_HEADER_SIZE, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, build_encrypted,
|
||||
build_established_header, prepend_inner_header,
|
||||
};
|
||||
use crate::bloom::BloomState;
|
||||
use crate::cache::CoordCache;
|
||||
@@ -497,6 +501,40 @@ pub struct Node {
|
||||
/// Static hostname → npub mapping for DNS resolution.
|
||||
/// Built at construction from peer aliases and /etc/fips/hosts.
|
||||
host_map: Arc<HostMap>,
|
||||
|
||||
/// Off-task FMP-encrypt + UDP-send worker pool. Unix-only —
|
||||
/// the worker issues direct sendmmsg(2) / sendmsg+UDP_GSO calls
|
||||
/// on raw fds via `AsRawFd`. None on Windows or when the worker
|
||||
/// pool failed to spawn.
|
||||
#[cfg(unix)]
|
||||
pub(crate) encrypt_workers: Option<encrypt_worker::EncryptWorkerPool>,
|
||||
|
||||
/// Off-task FMP decrypt worker pool — receiver-side mirror of
|
||||
/// `encrypt_workers`. Workers are shards: each owns its session
|
||||
/// state directly in a thread-local `HashMap` (no `RwLock`,
|
||||
/// no `Mutex` per packet). Hash-by-cache-key dispatch.
|
||||
#[cfg(unix)]
|
||||
pub(crate) decrypt_workers: Option<decrypt_worker::DecryptWorkerPool>,
|
||||
|
||||
/// Sessions whose recv cipher + replay window have been handed
|
||||
/// off to a decrypt shard worker. Lookup gate on the hot receive
|
||||
/// path: if the cache-key is in here, dispatch to worker; else
|
||||
/// fall through to the legacy synchronous decrypt (test mode +
|
||||
/// not-yet-registered first packets).
|
||||
#[cfg(unix)]
|
||||
pub(crate) decrypt_registered_sessions: std::collections::HashSet<(TransportId, u32)>,
|
||||
|
||||
/// Decrypt worker fallback channel: workers bounce
|
||||
/// authenticated-FMP-plaintext back here for the rx_loop to
|
||||
/// finish the per-peer side-effects (stats, MMP, ECN
|
||||
/// propagation, dispatch_link_message). `Option` so the receive
|
||||
/// end can be `take()`-en by the rx_loop arm.
|
||||
#[cfg(unix)]
|
||||
pub(crate) decrypt_fallback_rx:
|
||||
Option<tokio::sync::mpsc::UnboundedReceiver<decrypt_worker::DecryptWorkerEvent>>,
|
||||
#[cfg(unix)]
|
||||
pub(crate) decrypt_fallback_tx:
|
||||
tokio::sync::mpsc::UnboundedSender<decrypt_worker::DecryptWorkerEvent>,
|
||||
}
|
||||
|
||||
impl Node {
|
||||
@@ -569,6 +607,10 @@ impl Node {
|
||||
hosts_path,
|
||||
);
|
||||
|
||||
#[cfg(unix)]
|
||||
let (decrypt_fallback_tx, decrypt_fallback_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<decrypt_worker::DecryptWorkerEvent>();
|
||||
|
||||
Ok(Self {
|
||||
identity,
|
||||
startup_epoch,
|
||||
@@ -638,6 +680,16 @@ impl Node {
|
||||
peer_acl,
|
||||
host_map,
|
||||
path_mtu_lookup: Arc::new(std::sync::RwLock::new(HashMap::new())),
|
||||
#[cfg(unix)]
|
||||
encrypt_workers: None,
|
||||
#[cfg(unix)]
|
||||
decrypt_workers: None,
|
||||
#[cfg(unix)]
|
||||
decrypt_registered_sessions: std::collections::HashSet::new(),
|
||||
#[cfg(unix)]
|
||||
decrypt_fallback_rx: Some(decrypt_fallback_rx),
|
||||
#[cfg(unix)]
|
||||
decrypt_fallback_tx,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -702,6 +754,10 @@ impl Node {
|
||||
std::path::PathBuf::from(crate::upper::hosts::DEFAULT_HOSTS_PATH),
|
||||
);
|
||||
|
||||
#[cfg(unix)]
|
||||
let (decrypt_fallback_tx, decrypt_fallback_rx) =
|
||||
tokio::sync::mpsc::unbounded_channel::<decrypt_worker::DecryptWorkerEvent>();
|
||||
|
||||
Ok(Self {
|
||||
identity,
|
||||
startup_epoch,
|
||||
@@ -769,6 +825,16 @@ impl Node {
|
||||
peer_acl,
|
||||
host_map,
|
||||
path_mtu_lookup: Arc::new(std::sync::RwLock::new(HashMap::new())),
|
||||
#[cfg(unix)]
|
||||
encrypt_workers: None,
|
||||
#[cfg(unix)]
|
||||
decrypt_workers: None,
|
||||
#[cfg(unix)]
|
||||
decrypt_registered_sessions: std::collections::HashSet::new(),
|
||||
#[cfg(unix)]
|
||||
decrypt_fallback_rx: Some(decrypt_fallback_rx),
|
||||
#[cfg(unix)]
|
||||
decrypt_fallback_tx,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1936,6 +2002,12 @@ impl Node {
|
||||
flags |= FLAG_KEY_EPOCH;
|
||||
}
|
||||
|
||||
// Snapshot the per-peer connect()-ed UDP socket BEFORE the
|
||||
// session borrow so the encrypt-worker dispatch can refcount-
|
||||
// clone the Arc without re-borrowing self.peers later.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let connected_socket = peer.connected_udp();
|
||||
|
||||
let session = peer
|
||||
.noise_session_mut()
|
||||
.ok_or_else(|| NodeError::SendFailed {
|
||||
@@ -1943,12 +2015,96 @@ impl Node {
|
||||
reason: "no noise session".into(),
|
||||
})?;
|
||||
|
||||
// Inner plaintext: [timestamp:4 LE][msg_type][payload...]
|
||||
// ── Off-task encrypt + sendmmsg/GSO fast path (unix + UDP) ──
|
||||
// Build the wire buffer directly as
|
||||
// `[16-byte header][4-byte timestamp][plaintext]` with
|
||||
// TAG_SIZE trailing capacity for the AEAD tag — one alloc,
|
||||
// one extend, no intermediate `inner_plaintext` Vec. The
|
||||
// worker `seal_in_place_separate_tag`s on `wire_buf[16..]`
|
||||
// and appends the tag — buffer IS the wire packet.
|
||||
const INNER_TS_LEN: usize = 4;
|
||||
let inner_len = INNER_TS_LEN + plaintext.len();
|
||||
let payload_len = inner_len as u16;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let send_cipher_opt = session.send_cipher_clone();
|
||||
if let Some(fmp_cipher) = send_cipher_opt
|
||||
&& let Some(workers) = self.encrypt_workers.as_ref().cloned()
|
||||
&& let Some(transport) = self.transports.get(&transport_id)
|
||||
&& let TransportHandle::Udp(udp) = transport
|
||||
&& let Some(socket) = udp.async_socket()
|
||||
{
|
||||
// Skip per-packet DNS resolve on the steady-state path
|
||||
// when the connected socket already knows the peer
|
||||
// address (kernel 5-tuple cache wins over re-parsing
|
||||
// the configured TransportAddr).
|
||||
let socket_addr_opt = {
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
match connected_socket.as_ref() {
|
||||
Some(s) => Some(s.peer_addr()),
|
||||
None => udp.resolve_for_off_task(&remote_addr).await.ok(),
|
||||
}
|
||||
}
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
{
|
||||
udp.resolve_for_off_task(&remote_addr).await.ok()
|
||||
}
|
||||
};
|
||||
if let Some(dest_socket_addr) = socket_addr_opt {
|
||||
let counter =
|
||||
session
|
||||
.take_send_counter()
|
||||
.map_err(|e| NodeError::SendFailed {
|
||||
node_addr: *node_addr,
|
||||
reason: format!("counter reservation failed: {}", e),
|
||||
})?;
|
||||
let header = build_established_header(their_index, counter, flags, payload_len);
|
||||
let wire_capacity =
|
||||
ESTABLISHED_HEADER_SIZE + inner_len + crate::noise::TAG_SIZE;
|
||||
let mut wire_buf = Vec::with_capacity(wire_capacity);
|
||||
wire_buf.extend_from_slice(&header);
|
||||
wire_buf.extend_from_slice(×tamp_ms.to_le_bytes());
|
||||
wire_buf.extend_from_slice(plaintext);
|
||||
|
||||
let predicted_bytes = wire_capacity;
|
||||
// Drop bulk endpoint data on UDP backpressure to
|
||||
// keep the queue moving; control frames retry.
|
||||
let drop_on_backpressure = plaintext.first().is_some_and(|t| *t == 0x00);
|
||||
workers.dispatch(crate::node::encrypt_worker::FmpSendJob {
|
||||
cipher: fmp_cipher,
|
||||
counter,
|
||||
wire_buf,
|
||||
fsp_seal: None,
|
||||
socket,
|
||||
dest_addr: dest_socket_addr,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
connected_socket,
|
||||
drop_on_backpressure,
|
||||
queued_at: None,
|
||||
});
|
||||
|
||||
if let Some(peer) = self.peers.get_mut(node_addr) {
|
||||
peer.link_stats_mut().record_sent(predicted_bytes);
|
||||
if let Some(mmp) = peer.mmp_mut() {
|
||||
mmp.sender
|
||||
.record_sent(counter, timestamp_ms, predicted_bytes);
|
||||
}
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Legacy inline path: only reached for non-UDP transports or
|
||||
// unit-test mode (no worker pool spawned). Materialise the
|
||||
// inner plaintext lazily here so the worker path above
|
||||
// avoids the alloc.
|
||||
let inner_plaintext = prepend_inner_header(timestamp_ms, plaintext);
|
||||
|
||||
// Build 16-byte outer header (used as AAD for AEAD)
|
||||
let counter = session.current_send_counter();
|
||||
let payload_len = inner_plaintext.len() as u16;
|
||||
let header = build_established_header(their_index, counter, flags, payload_len);
|
||||
|
||||
// Encrypt with AAD binding to the outer header
|
||||
|
||||
@@ -331,6 +331,72 @@ pub(super) async fn drain_all_packets(nodes: &mut [TestNode], verbose: bool) ->
|
||||
total
|
||||
}
|
||||
|
||||
/// Repair synthetic test edges whose one-shot UDP handshake packet was dropped.
|
||||
///
|
||||
/// The large topology tests create 250 links by sending exactly one msg1 per
|
||||
/// edge, bypassing the normal node reconnect timers. On slower CI runners that
|
||||
/// burst can still drop a localhost UDP datagram, so retry only edges that did
|
||||
/// not produce bidirectional peers before asserting tree/session behavior. The
|
||||
/// retry path drains after each edge instead of sending a second burst, since
|
||||
/// the repair is meant to remove harness pressure rather than recreate it.
|
||||
async fn repair_missing_edge_handshakes(
|
||||
nodes: &mut [TestNode],
|
||||
edges: &[(usize, usize)],
|
||||
verbose: bool,
|
||||
) -> usize {
|
||||
let mut retries = 0;
|
||||
|
||||
for attempt in 0..5 {
|
||||
let mut missing = Vec::new();
|
||||
for &(i, j) in edges {
|
||||
let j_addr = *nodes[j].node.node_addr();
|
||||
let i_addr = *nodes[i].node.node_addr();
|
||||
let i_has_j = nodes[i].node.get_peer(&j_addr).is_some();
|
||||
let j_has_i = nodes[j].node.get_peer(&i_addr).is_some();
|
||||
if !i_has_j || !j_has_i {
|
||||
missing.push((i, j, i_has_j, j_has_i));
|
||||
}
|
||||
}
|
||||
|
||||
if missing.is_empty() {
|
||||
break;
|
||||
}
|
||||
|
||||
if verbose {
|
||||
eprintln!(
|
||||
" Repairing {} missing synthetic edge handshake(s), attempt {}",
|
||||
missing.len(),
|
||||
attempt + 1
|
||||
);
|
||||
}
|
||||
|
||||
for (i, j, i_has_j, j_has_i) in missing {
|
||||
if !i_has_j {
|
||||
initiate_handshake(nodes, i, j).await;
|
||||
retries += 1;
|
||||
let _ = drain_all_packets(nodes, false).await;
|
||||
}
|
||||
|
||||
let j_addr = *nodes[j].node.node_addr();
|
||||
let i_addr = *nodes[i].node.node_addr();
|
||||
let j_still_missing_i = nodes[j].node.get_peer(&i_addr).is_none();
|
||||
let i_still_missing_j = nodes[i].node.get_peer(&j_addr).is_none();
|
||||
|
||||
if !j_has_i && j_still_missing_i {
|
||||
initiate_handshake(nodes, j, i).await;
|
||||
retries += 1;
|
||||
let _ = drain_all_packets(nodes, false).await;
|
||||
} else if i_still_missing_j {
|
||||
initiate_handshake(nodes, i, j).await;
|
||||
retries += 1;
|
||||
let _ = drain_all_packets(nodes, false).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
retries
|
||||
}
|
||||
|
||||
/// Generate a connected random graph with deterministic topology.
|
||||
///
|
||||
/// First builds a random spanning tree to ensure connectivity,
|
||||
@@ -588,9 +654,14 @@ pub(super) async fn run_tree_test(
|
||||
// Drain packets until convergence (handles rate-limited announces)
|
||||
let total = drain_all_packets(&mut nodes, verbose).await;
|
||||
assert!(total > 0, "Should have processed at least some packets");
|
||||
let repaired = repair_missing_edge_handshakes(&mut nodes, edges, verbose).await;
|
||||
|
||||
if verbose {
|
||||
eprintln!("\n Total packets processed: {}", total);
|
||||
if repaired > 0 {
|
||||
eprintln!(" Synthetic handshake retries: {}", repaired);
|
||||
print_tree_snapshot("After synthetic handshake repair", &nodes);
|
||||
}
|
||||
}
|
||||
|
||||
// Verify all edges established bidirectional peers
|
||||
@@ -636,6 +707,7 @@ pub(super) async fn run_tree_test_with_mtus(
|
||||
|
||||
let total = drain_all_packets(&mut nodes, false).await;
|
||||
assert!(total > 0, "Should have processed at least some packets");
|
||||
let _ = repair_missing_edge_handshakes(&mut nodes, edges, false).await;
|
||||
|
||||
for &(i, j) in edges {
|
||||
let j_addr = *nodes[j].node.node_addr();
|
||||
|
||||
@@ -394,6 +394,21 @@ impl CipherState {
|
||||
pub fn has_key(&self) -> bool {
|
||||
self.has_key
|
||||
}
|
||||
|
||||
/// Clone the underlying keyed AEAD, for off-task AEAD workers.
|
||||
///
|
||||
/// Returns `None` if no key. The cloned `LessSafeKey` pairs with
|
||||
/// `decrypt_with_counter[_and_aad]` / `encrypt_with_counter[_and_aad]`
|
||||
/// or with bare `ring::aead::LessSafeKey::seal_in_place_separate_tag` —
|
||||
/// the worker holds the cipher and a pre-assigned counter, while the
|
||||
/// dispatcher keeps the replay window and counter assignment sequential.
|
||||
pub fn cipher_clone(&self) -> Option<LessSafeKey> {
|
||||
if self.has_key {
|
||||
Self::build_cipher(&self.key)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl fmt::Debug for CipherState {
|
||||
@@ -429,7 +444,7 @@ fn seal(
|
||||
/// Decrypt `ciphertext` (with appended tag) with the given keyed AEAD,
|
||||
/// counter, and AAD, returning the plaintext as a `Vec<u8>`. Truncates
|
||||
/// in place to drop the AEAD tag.
|
||||
fn open(
|
||||
pub(crate) fn open(
|
||||
cipher: Option<&LessSafeKey>,
|
||||
counter: u64,
|
||||
aad: &[u8],
|
||||
|
||||
@@ -148,6 +148,43 @@ impl NoiseSession {
|
||||
self.replay_window.highest()
|
||||
}
|
||||
|
||||
/// Clone the recv-side AEAD instance, for off-task decrypt workers.
|
||||
pub fn recv_cipher_clone(&self) -> Option<ring::aead::LessSafeKey> {
|
||||
self.recv_cipher.cipher_clone()
|
||||
}
|
||||
|
||||
/// Snapshot the current replay-window state as an **owned**
|
||||
/// `ReplayWindow`, for hand-off to a shard-owning decrypt worker.
|
||||
/// After this snapshot, the worker becomes the sole authority for
|
||||
/// replay protection on the session.
|
||||
pub fn recv_replay_snapshot_owned(&self) -> ReplayWindow {
|
||||
self.replay_window.clone()
|
||||
}
|
||||
|
||||
/// Clone the send-side AEAD instance, for off-task encrypt workers.
|
||||
/// Pair with `take_send_counter` to keep counter assignment serial
|
||||
/// under the session's `&mut`.
|
||||
pub fn send_cipher_clone(&self) -> Option<ring::aead::LessSafeKey> {
|
||||
self.send_cipher.cipher_clone()
|
||||
}
|
||||
|
||||
/// Reserve and return the next send counter, advancing the internal
|
||||
/// nonce. For pipelined encrypt paths.
|
||||
pub fn take_send_counter(&mut self) -> Result<u64, NoiseError> {
|
||||
if self.send_cipher.nonce == u64::MAX {
|
||||
return Err(NoiseError::NonceOverflow);
|
||||
}
|
||||
let counter = self.send_cipher.nonce;
|
||||
self.send_cipher.nonce += 1;
|
||||
Ok(counter)
|
||||
}
|
||||
|
||||
/// Accept a counter into the replay window after a successful out-of-task
|
||||
/// decrypt. Caller is responsible for verifying decrypt success first.
|
||||
pub fn accept_replay(&mut self, counter: u64) {
|
||||
self.replay_window.accept(counter);
|
||||
}
|
||||
|
||||
/// Reset the replay window (use when rekeying).
|
||||
pub fn reset_replay_window(&mut self) {
|
||||
self.replay_window.reset();
|
||||
|
||||
@@ -195,6 +195,23 @@ pub struct ActivePeer {
|
||||
rekey_msg1: Option<Vec<u8>>,
|
||||
/// In-progress rekey: next resend timestamp (Unix ms).
|
||||
rekey_msg1_next_resend: u64,
|
||||
|
||||
/// Unix UDP fast-path: per-peer `connect()`-ed socket (paired with
|
||||
/// the listen socket via `SO_REUSEPORT`). The kernel demux prefers
|
||||
/// the connected 5-tuple, so inbound packets land here; the
|
||||
/// encrypt-worker send path sends with `msg_name = NULL`, skipping
|
||||
/// per-packet sockaddr handling + route lookup. Behind an `Arc` so
|
||||
/// in-flight worker jobs survive rekey/address-change rotations.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
connected_udp:
|
||||
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
|
||||
|
||||
/// Per-peer recv drain thread. Always paired with `connected_udp`:
|
||||
/// the kernel routes inbound packets from this peer to the
|
||||
/// connected socket, so it *must* be drained or the kernel recv
|
||||
/// buffer fills. Drop signals shutdown via self-pipe.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
peer_recv_drain: Option<crate::transport::udp::peer_drain::PeerRecvDrain>,
|
||||
}
|
||||
|
||||
impl ActivePeer {
|
||||
@@ -247,6 +264,10 @@ impl ActivePeer {
|
||||
rekey_our_index: None,
|
||||
rekey_msg1: None,
|
||||
rekey_msg1_next_resend: 0,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
connected_udp: None,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
peer_recv_drain: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -328,9 +349,53 @@ impl ActivePeer {
|
||||
rekey_our_index: None,
|
||||
rekey_msg1: None,
|
||||
rekey_msg1_next_resend: 0,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
connected_udp: None,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
peer_recv_drain: None,
|
||||
}
|
||||
}
|
||||
|
||||
// === Connected-UDP fast path ===
|
||||
|
||||
/// Refcount the per-peer `connect()`-ed UDP socket if installed.
|
||||
/// Encrypt-worker send path uses this to bypass the wildcard
|
||||
/// listen socket's per-packet sockaddr handling.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
pub(crate) fn connected_udp(
|
||||
&self,
|
||||
) -> Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>> {
|
||||
self.connected_udp.clone()
|
||||
}
|
||||
|
||||
/// Install a per-peer `connect()`-ed UDP socket with its paired
|
||||
/// recv drain thread. The two own each other's lifetime: the drain
|
||||
/// is the only consumer of packets on this socket.
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
pub(crate) fn set_connected_udp(
|
||||
&mut self,
|
||||
socket: std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>,
|
||||
drain: crate::transport::udp::peer_drain::PeerRecvDrain,
|
||||
) {
|
||||
// Drop the old drain BEFORE the old socket so its last fd
|
||||
// reference is released cleanly.
|
||||
self.peer_recv_drain = None;
|
||||
self.connected_udp = None;
|
||||
self.connected_udp = Some(socket);
|
||||
self.peer_recv_drain = Some(drain);
|
||||
}
|
||||
|
||||
/// Clear the per-peer connected UDP socket + drain. The drain
|
||||
/// exits via self-pipe signal; the kernel fd closes on last `Arc`
|
||||
/// drop (any in-flight worker jobs holding the old `Arc` stay
|
||||
/// valid until they complete).
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[allow(dead_code)] // called from session-deregister + rekey follow-up
|
||||
pub(crate) fn clear_connected_udp(&mut self) {
|
||||
self.peer_recv_drain = None;
|
||||
self.connected_udp = None;
|
||||
}
|
||||
|
||||
// === Identity Accessors ===
|
||||
|
||||
/// Get the peer's verified identity.
|
||||
@@ -455,9 +520,15 @@ impl ActivePeer {
|
||||
/// Update the current address (for roaming support).
|
||||
///
|
||||
/// Called when we receive a valid authenticated packet from a new address.
|
||||
pub fn set_current_addr(&mut self, transport_id: TransportId, addr: TransportAddr) {
|
||||
/// Returns `true` if `(transport_id, addr)` actually changed — callers
|
||||
/// use this to invalidate per-peer `connect(2)`-ed UDP sockets whose
|
||||
/// 5-tuple just went stale.
|
||||
pub fn set_current_addr(&mut self, transport_id: TransportId, addr: TransportAddr) -> bool {
|
||||
let changed =
|
||||
self.transport_id != Some(transport_id) || self.current_addr.as_ref() != Some(&addr);
|
||||
self.transport_id = Some(transport_id);
|
||||
self.current_addr = Some(addr);
|
||||
changed
|
||||
}
|
||||
|
||||
// === Handshake Resend ===
|
||||
|
||||
402
src/perf_profile.rs
Normal file
402
src/perf_profile.rs
Normal file
@@ -0,0 +1,402 @@
|
||||
// Some entry points (e.g. `stamp`, `record_since`) are only called from
|
||||
// paths that aren't yet wired up in this PR (FSP-pipelined dispatch,
|
||||
// per-stage worker telemetry). Keep them in tree so the follow-up wiring
|
||||
// PR is a pure call-site change.
|
||||
#![allow(dead_code)]
|
||||
|
||||
//! Runtime perf profiler for the FMP/FSP hot path and queue handoffs.
|
||||
//!
|
||||
//! Avoids external dependencies (`perf`, samply, etc.) by instrumenting
|
||||
//! the key stages directly with `AtomicU64` ns counters, histograms,
|
||||
//! and packet counts. A background task prints a per-stage breakdown
|
||||
//! every `FIPS_PERF_INTERVAL_SECS` seconds when `FIPS_PERF=1` or
|
||||
//! `FIPS_PIPELINE_TRACE=1` is set at runtime.
|
||||
//!
|
||||
//! Enabling adds `Instant::now()` plus a few relaxed atomics per
|
||||
//! measured stage, so the measured numbers are slightly pessimistic vs
|
||||
//! production. The relative picture is the point: it shows whether a
|
||||
//! run is spending time in crypto, syscalls, or scheduler/channel
|
||||
//! waits.
|
||||
//!
|
||||
//! Stages tracked, inbound:
|
||||
//! * `UDP_RECV` — recvmmsg syscall + per-message buffer copy
|
||||
//! * `FMP_DECRYPT` — outer AEAD open + replay window
|
||||
//! * `LINK_DISPATCH` — `dispatch_link_message` excluding FSP work
|
||||
//! * `FSP_DECRYPT` — inner AEAD open + replay window
|
||||
//! * `TUN_WRITE` — IPv6 shim decompress + tun_tx.send
|
||||
//!
|
||||
//! Stages tracked, outbound:
|
||||
//! * `FSP_ENCRYPT` — inner AEAD seal (`send_session_data`)
|
||||
//! * `FMP_ENCRYPT` — outer AEAD seal (`send_encrypted_link_message`)
|
||||
//! * `UDP_SEND` — sendmmsg/sendmsg/sendto flush
|
||||
//!
|
||||
//! Handoff waits tracked:
|
||||
//! * `TRANSPORT_QUEUE_WAIT` — UDP/transport receive loop → rx_loop
|
||||
//! * `ENDPOINT_COMMAND_WAIT` — FipsEndpoint send → node command loop
|
||||
//! * `FMP_WORKER_QUEUE_WAIT` — rx_loop FMP job dispatch → worker
|
||||
//! * `ENDPOINT_EVENT_WAIT` — rx_loop endpoint delivery → endpoint recv
|
||||
|
||||
use std::sync::OnceLock;
|
||||
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
|
||||
use std::time::Instant;
|
||||
|
||||
/// Number of measurement buckets. Indices match `Stage`.
|
||||
const N_STAGES: usize = 16;
|
||||
const N_EVENTS: usize = 6;
|
||||
const HIST_BUCKETS: usize = 48;
|
||||
|
||||
/// Stage identifier. `as usize` indexes into the counter arrays.
|
||||
#[derive(Copy, Clone, Debug)]
|
||||
#[repr(usize)]
|
||||
pub enum Stage {
|
||||
UdpRecv = 0,
|
||||
FmpDecrypt = 1,
|
||||
LinkDispatch = 2,
|
||||
FspDecrypt = 3,
|
||||
TunWrite = 4,
|
||||
FspEncrypt = 5,
|
||||
FmpEncrypt = 6,
|
||||
UdpSend = 7,
|
||||
/// Whole `Node::process_packet` body. Anchor for "what fraction of
|
||||
/// the receive hot path is in the non-AEAD parts of the pipeline".
|
||||
ProcessPacket = 8,
|
||||
/// Just the `endpoint_event_tx.send()` for inbound application
|
||||
/// payloads — wakes the embedded-endpoint consumer task.
|
||||
EndpointDeliver = 9,
|
||||
/// Whole `handle_encrypted_session_msg` (FSP receive path) minus
|
||||
/// the `FspDecrypt` sub-span. Surfaces dispatch + ipv6_shim +
|
||||
/// `Vec::drain` cost on the inner session layer.
|
||||
FspHandle = 10,
|
||||
/// Whole `handle_endpoint_data_command` body — the SENDER's
|
||||
/// per-packet "do everything to push one outbound packet"
|
||||
/// dispatch. Compare against the sum of `FspEncrypt`,
|
||||
/// `FmpEncrypt`, and `UdpSend` to see how much of the sender
|
||||
/// hot path is in state-touching dispatch (sessions/peers
|
||||
/// lookups, MMP/stats updates, Vec allocs) vs the AEAD/syscall
|
||||
/// work that's a natural fit for an off-task worker.
|
||||
EndpointSend = 11,
|
||||
/// Time spent waiting after `FipsEndpoint::send`/`blocking_send`
|
||||
/// creates a node command until `rx_loop` starts handling it.
|
||||
EndpointCommandWait = 12,
|
||||
/// Time spent waiting after `rx_loop` creates an FMP encrypt/send
|
||||
/// worker job until the worker thread starts encrypting it.
|
||||
FmpWorkerQueueWait = 13,
|
||||
/// Time spent waiting after a transport receives a packet until
|
||||
/// `rx_loop` starts processing it.
|
||||
TransportQueueWait = 14,
|
||||
/// Time spent waiting after `rx_loop` delivers endpoint data until
|
||||
/// the embedded endpoint consumer receives it.
|
||||
EndpointEventWait = 15,
|
||||
}
|
||||
|
||||
impl Stage {
|
||||
const fn name(self) -> &'static str {
|
||||
match self {
|
||||
Stage::UdpRecv => "udp_recv",
|
||||
Stage::FmpDecrypt => "fmp_decrypt",
|
||||
Stage::LinkDispatch => "link_dispatch",
|
||||
Stage::FspDecrypt => "fsp_decrypt",
|
||||
Stage::TunWrite => "tun_write",
|
||||
Stage::FspEncrypt => "fsp_encrypt",
|
||||
Stage::FmpEncrypt => "fmp_encrypt",
|
||||
Stage::UdpSend => "udp_send",
|
||||
Stage::ProcessPacket => "process_packet",
|
||||
Stage::EndpointDeliver => "endpoint_deliver",
|
||||
Stage::FspHandle => "fsp_handle",
|
||||
Stage::EndpointSend => "endpoint_send",
|
||||
Stage::EndpointCommandWait => "endpoint_command_wait",
|
||||
Stage::FmpWorkerQueueWait => "fmp_worker_queue_wait",
|
||||
Stage::TransportQueueWait => "transport_queue_wait",
|
||||
Stage::EndpointEventWait => "endpoint_event_wait",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn stage_from_index(idx: usize) -> Stage {
|
||||
match idx {
|
||||
0 => Stage::UdpRecv,
|
||||
1 => Stage::FmpDecrypt,
|
||||
2 => Stage::LinkDispatch,
|
||||
3 => Stage::FspDecrypt,
|
||||
4 => Stage::TunWrite,
|
||||
5 => Stage::FspEncrypt,
|
||||
6 => Stage::FmpEncrypt,
|
||||
7 => Stage::UdpSend,
|
||||
8 => Stage::ProcessPacket,
|
||||
9 => Stage::EndpointDeliver,
|
||||
10 => Stage::FspHandle,
|
||||
11 => Stage::EndpointSend,
|
||||
12 => Stage::EndpointCommandWait,
|
||||
13 => Stage::FmpWorkerQueueWait,
|
||||
14 => Stage::TransportQueueWait,
|
||||
15 => Stage::EndpointEventWait,
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Count-only events that clarify which hot-path variant is active.
|
||||
#[derive(Copy, Clone, Debug)]
|
||||
#[repr(usize)]
|
||||
pub enum Event {
|
||||
UdpSendConnected = 0,
|
||||
UdpSendWildcard = 1,
|
||||
UdpSendBackpressure = 2,
|
||||
ConnectedUdpInstalled = 3,
|
||||
ConnectedUdpActivationFailed = 4,
|
||||
UdpSendBackpressureSleep = 5,
|
||||
}
|
||||
|
||||
impl Event {
|
||||
const fn name(self) -> &'static str {
|
||||
match self {
|
||||
Event::UdpSendConnected => "udp_send_connected",
|
||||
Event::UdpSendWildcard => "udp_send_wildcard",
|
||||
Event::UdpSendBackpressure => "udp_send_backpressure",
|
||||
Event::ConnectedUdpInstalled => "connected_udp_installed",
|
||||
Event::ConnectedUdpActivationFailed => "connected_udp_activation_failed",
|
||||
Event::UdpSendBackpressureSleep => "udp_send_backpressure_sleep",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn event_from_index(idx: usize) -> Event {
|
||||
match idx {
|
||||
0 => Event::UdpSendConnected,
|
||||
1 => Event::UdpSendWildcard,
|
||||
2 => Event::UdpSendBackpressure,
|
||||
3 => Event::ConnectedUdpInstalled,
|
||||
4 => Event::ConnectedUdpActivationFailed,
|
||||
5 => Event::UdpSendBackpressureSleep,
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
static TOTAL_NS: [AtomicU64; N_STAGES] = [const { AtomicU64::new(0) }; N_STAGES];
|
||||
static COUNT: [AtomicU64; N_STAGES] = [const { AtomicU64::new(0) }; N_STAGES];
|
||||
static MAX_NS: [AtomicU64; N_STAGES] = [const { AtomicU64::new(0) }; N_STAGES];
|
||||
static HIST: [AtomicU64; N_STAGES * HIST_BUCKETS] =
|
||||
[const { AtomicU64::new(0) }; N_STAGES * HIST_BUCKETS];
|
||||
static EVENTS: [AtomicU64; N_EVENTS] = [const { AtomicU64::new(0) }; N_EVENTS];
|
||||
|
||||
/// True iff perf/pipeline tracing is enabled. Read once at startup so
|
||||
/// the per-packet check is a single cached load.
|
||||
pub(crate) fn enabled() -> bool {
|
||||
static ENABLED: OnceLock<bool> = OnceLock::new();
|
||||
*ENABLED.get_or_init(|| {
|
||||
["FIPS_PERF", "FIPS_PIPELINE_TRACE"].into_iter().any(|key| {
|
||||
std::env::var(key)
|
||||
.map(|s| s == "1" || s.eq_ignore_ascii_case("true"))
|
||||
.unwrap_or(false)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/// Capture a timestamp for a future queue-wait measurement. Returns
|
||||
/// `None` when tracing is disabled so callers can store it cheaply in
|
||||
/// packet/job structs without paying `Instant::now()` in production.
|
||||
#[inline]
|
||||
pub(crate) fn stamp() -> Option<Instant> {
|
||||
if enabled() {
|
||||
Some(Instant::now())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Record time elapsed since a previously captured stamp.
|
||||
#[inline]
|
||||
pub(crate) fn record_since(stage: Stage, start: Option<Instant>) {
|
||||
if let Some(start) = start {
|
||||
record(stage, start.elapsed().as_nanos() as u64);
|
||||
}
|
||||
}
|
||||
|
||||
/// Record `elapsed_ns` for the given stage. No-op when disabled.
|
||||
pub fn record(stage: Stage, elapsed_ns: u64) {
|
||||
if !enabled() {
|
||||
return;
|
||||
}
|
||||
let idx = stage as usize;
|
||||
let elapsed_ns = elapsed_ns.max(1);
|
||||
TOTAL_NS[idx].fetch_add(elapsed_ns, Relaxed);
|
||||
COUNT[idx].fetch_add(1, Relaxed);
|
||||
MAX_NS[idx].fetch_max(elapsed_ns, Relaxed);
|
||||
HIST[(idx * HIST_BUCKETS) + bucket_for_ns(elapsed_ns)].fetch_add(1, Relaxed);
|
||||
}
|
||||
|
||||
#[inline]
|
||||
pub fn record_event(event: Event) {
|
||||
record_event_count(event, 1);
|
||||
}
|
||||
|
||||
pub fn record_event_count(event: Event, count: u64) {
|
||||
if !enabled() || count == 0 {
|
||||
return;
|
||||
}
|
||||
EVENTS[event as usize].fetch_add(count, Relaxed);
|
||||
}
|
||||
|
||||
/// RAII timer — `drop` records the elapsed time into the stage.
|
||||
/// Use:
|
||||
/// ```ignore
|
||||
/// let _t = profile::Timer::start(Stage::FmpDecrypt);
|
||||
/// // ... AEAD work ...
|
||||
/// ```
|
||||
pub struct Timer {
|
||||
stage: Stage,
|
||||
start: Option<Instant>,
|
||||
}
|
||||
|
||||
impl Timer {
|
||||
#[inline]
|
||||
pub fn start(stage: Stage) -> Self {
|
||||
let start = if enabled() {
|
||||
Some(Instant::now())
|
||||
} else {
|
||||
None
|
||||
};
|
||||
Self { stage, start }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Timer {
|
||||
fn drop(&mut self) {
|
||||
if let Some(t0) = self.start {
|
||||
let ns = t0.elapsed().as_nanos() as u64;
|
||||
record(self.stage, ns);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn a background task that prints a per-stage breakdown every
|
||||
/// `FIPS_PERF_INTERVAL_SECS` seconds (default 5). Idempotent — only
|
||||
/// the first call spawns. No-op when profiling isn't enabled.
|
||||
pub fn maybe_spawn_reporter() {
|
||||
if !enabled() {
|
||||
return;
|
||||
}
|
||||
static STARTED: OnceLock<()> = OnceLock::new();
|
||||
if STARTED.set(()).is_err() {
|
||||
return;
|
||||
}
|
||||
let interval = std::env::var("FIPS_PERF_INTERVAL_SECS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.unwrap_or(5)
|
||||
.max(1);
|
||||
tokio::spawn(async move {
|
||||
let mut prev_total = [0u64; N_STAGES];
|
||||
let mut prev_count = [0u64; N_STAGES];
|
||||
let mut prev_hist = [0u64; N_STAGES * HIST_BUCKETS];
|
||||
let mut prev_events = [0u64; N_EVENTS];
|
||||
loop {
|
||||
tokio::time::sleep(std::time::Duration::from_secs(interval)).await;
|
||||
let mut line = format!("[pipe {}s]", interval);
|
||||
for i in 0..N_STAGES {
|
||||
let t = TOTAL_NS[i].load(Relaxed);
|
||||
let c = COUNT[i].load(Relaxed);
|
||||
let dt = t.saturating_sub(prev_total[i]);
|
||||
let dc = c.saturating_sub(prev_count[i]);
|
||||
prev_total[i] = t;
|
||||
prev_count[i] = c;
|
||||
|
||||
let base = i * HIST_BUCKETS;
|
||||
let mut hist_delta = [0u64; HIST_BUCKETS];
|
||||
for (bucket, delta) in hist_delta.iter_mut().enumerate().take(HIST_BUCKETS) {
|
||||
let idx = base + bucket;
|
||||
let current = HIST[idx].load(Relaxed);
|
||||
*delta = current.saturating_sub(prev_hist[idx]);
|
||||
prev_hist[idx] = current;
|
||||
}
|
||||
if dc == 0 {
|
||||
continue;
|
||||
}
|
||||
let stage = stage_from_index(i);
|
||||
let avg_ns = if dc > 0 { dt / dc } else { 0 };
|
||||
let pps = if interval > 0 { dc / interval } else { 0 };
|
||||
let p50 = percentile_ns(&hist_delta, dc, 50);
|
||||
let p95 = percentile_ns(&hist_delta, dc, 95);
|
||||
let p99 = percentile_ns(&hist_delta, dc, 99);
|
||||
let approx_max = interval_max_ns(&hist_delta);
|
||||
let lifetime_max = MAX_NS[i].load(Relaxed);
|
||||
line.push_str(&format!(
|
||||
" {}={}/s avg={} p50<={} p95<={} p99<={} max<={} allmax={}",
|
||||
stage.name(),
|
||||
pps,
|
||||
fmt_ns(avg_ns),
|
||||
fmt_ns(p50),
|
||||
fmt_ns(p95),
|
||||
fmt_ns(p99),
|
||||
fmt_ns(approx_max),
|
||||
fmt_ns(lifetime_max),
|
||||
));
|
||||
}
|
||||
for i in 0..N_EVENTS {
|
||||
let current = EVENTS[i].load(Relaxed);
|
||||
let delta = current.saturating_sub(prev_events[i]);
|
||||
prev_events[i] = current;
|
||||
if delta == 0 {
|
||||
continue;
|
||||
}
|
||||
let event = event_from_index(i);
|
||||
let per_sec = delta / interval;
|
||||
line.push_str(&format!(" {}={}/s", event.name(), per_sec));
|
||||
}
|
||||
// eprintln so it always lands regardless of RUST_LOG.
|
||||
eprintln!("{}", line);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn bucket_for_ns(ns: u64) -> usize {
|
||||
if ns <= 1 {
|
||||
return 0;
|
||||
}
|
||||
((u64::BITS - (ns - 1).leading_zeros()) as usize).min(HIST_BUCKETS - 1)
|
||||
}
|
||||
|
||||
fn bucket_upper_ns(bucket: usize) -> u64 {
|
||||
if bucket == 0 {
|
||||
1
|
||||
} else if bucket >= 63 {
|
||||
u64::MAX
|
||||
} else {
|
||||
1u64 << bucket
|
||||
}
|
||||
}
|
||||
|
||||
fn percentile_ns(hist_delta: &[u64; HIST_BUCKETS], total: u64, pct: u64) -> u64 {
|
||||
if total == 0 {
|
||||
return 0;
|
||||
}
|
||||
let target = total.saturating_mul(pct).saturating_add(99) / 100;
|
||||
let mut seen = 0u64;
|
||||
for (idx, count) in hist_delta.iter().enumerate() {
|
||||
seen = seen.saturating_add(*count);
|
||||
if seen >= target {
|
||||
return bucket_upper_ns(idx);
|
||||
}
|
||||
}
|
||||
bucket_upper_ns(HIST_BUCKETS - 1)
|
||||
}
|
||||
|
||||
fn interval_max_ns(hist_delta: &[u64; HIST_BUCKETS]) -> u64 {
|
||||
for idx in (0..HIST_BUCKETS).rev() {
|
||||
if hist_delta[idx] != 0 {
|
||||
return bucket_upper_ns(idx);
|
||||
}
|
||||
}
|
||||
0
|
||||
}
|
||||
|
||||
fn fmt_ns(ns: u64) -> String {
|
||||
if ns >= 1_000_000_000 {
|
||||
format!("{:.1}s", ns as f64 / 1_000_000_000.0)
|
||||
} else if ns >= 1_000_000 {
|
||||
format!("{:.1}ms", ns as f64 / 1_000_000.0)
|
||||
} else if ns >= 1_000 {
|
||||
format!("{:.1}us", ns as f64 / 1_000.0)
|
||||
} else {
|
||||
format!("{ns}ns")
|
||||
}
|
||||
}
|
||||
@@ -402,7 +402,6 @@ impl TransportAddr {
|
||||
/// Create a UDP/TCP transport address directly from a socket address.
|
||||
pub fn from_socket_addr(addr: std::net::SocketAddr) -> Self {
|
||||
use std::io::Write;
|
||||
|
||||
let mut buf = Vec::with_capacity(56);
|
||||
write!(&mut buf, "{addr}").expect("Vec<u8>::write_fmt is infallible");
|
||||
Self(buf)
|
||||
|
||||
436
src/transport/udp/connected_peer.rs
Normal file
436
src/transport/udp/connected_peer.rs
Normal file
@@ -0,0 +1,436 @@
|
||||
// The connected-UDP fast path is infra-ready but not yet wired into the
|
||||
// encrypt-worker dispatch site (a follow-up PR will refcount-clone the
|
||||
// socket into each FmpSendJob). Keep the API surface in tree.
|
||||
#![allow(dead_code)]
|
||||
|
||||
//! Connected per-peer UDP socket.
|
||||
//!
|
||||
//! One of the levers boringtun uses to hit 2.5–3.2 Gbps on a real
|
||||
//! NIC: after a peer is established, give them their **own UDP socket
|
||||
//! `connect()`-ed to their address**. The kernel then:
|
||||
//!
|
||||
//! - Routes inbound packets *from that peer* directly to the
|
||||
//! connected socket (most-specific-match wins over the wildcard
|
||||
//! listen socket), so the demux happens once at socket-receive
|
||||
//! time instead of repeatedly at the application layer.
|
||||
//! - Lets us `sendmsg(2)` with `msg_name = NULL` (or `send(2)`),
|
||||
//! skipping the per-packet sockaddr copy + route lookup + neighbor
|
||||
//! resolve that the kernel otherwise repeats for every datagram on
|
||||
//! an unconnected socket.
|
||||
//! - Combines cleanly with UDP_GSO: the connected socket sends one
|
||||
//! super-skb to one cached destination, and the kernel skips
|
||||
//! per-segment route lookups.
|
||||
//!
|
||||
//! Multiple connected sockets coexist with the wildcard listen socket
|
||||
//! via `SO_REUSEPORT` plus `SO_REUSEADDR`. The UDP demux picks the
|
||||
//! most specific match (5-tuple of connected sockets beats the
|
||||
//! wildcard), so traffic from new / unknown peers continues to land on
|
||||
//! the listen socket (handshakes / discovery), while established peers'
|
||||
//! steady-state traffic goes directly to their dedicated socket.
|
||||
//!
|
||||
//! **Scope of this module:** infrastructure only — the FD lifecycle
|
||||
//! (open / close), buffer sizing (matches the listen socket's
|
||||
//! `SO_RCVBUF` / `SO_SNDBUF` via `FORCE` variants where possible),
|
||||
//! and unit tests exercising the open + bind + connect path.
|
||||
|
||||
#![cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
|
||||
use std::io;
|
||||
use std::net::SocketAddr;
|
||||
use std::os::unix::io::{AsRawFd, RawFd};
|
||||
|
||||
/// A `connect()`-ed UDP socket for one established peer.
|
||||
///
|
||||
/// Owns the raw fd and closes it on drop. Configured with:
|
||||
/// - `SO_REUSEADDR` and `SO_REUSEPORT` so it can share the listen port
|
||||
/// with the wildcard socket and any other peers' connected sockets.
|
||||
/// - The receive / send buffer sizes inherited from the configured
|
||||
/// UDP transport (best-effort via `*BUFFORCE` variants — the kernel
|
||||
/// silently falls back to the normal `*BUF` ceiling if our process
|
||||
/// lacks `CAP_NET_ADMIN`).
|
||||
/// - `O_NONBLOCK` so callers that drive it from an OS-thread shard
|
||||
/// loop don't accidentally block the entire shard on a single
|
||||
/// recv / send.
|
||||
/// - `connect()`-ed to the peer's `SocketAddr`, locking in the
|
||||
/// per-packet kernel-side route + ARP / neighbor cache so neither
|
||||
/// needs to be redone on the data path.
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct ConnectedPeerSocket {
|
||||
fd: RawFd,
|
||||
peer_addr: SocketAddr,
|
||||
local_addr: SocketAddr,
|
||||
}
|
||||
|
||||
impl ConnectedPeerSocket {
|
||||
/// Open a new peer-connected UDP socket.
|
||||
///
|
||||
/// `local_addr` is the wildcard bind address (e.g. `0.0.0.0:51820`
|
||||
/// or `[::]:51820`) — the same address the listen socket bound
|
||||
/// to. `peer_addr` is the kernel `SocketAddr` of the established
|
||||
/// peer's UDP endpoint. `recv_buf` / `send_buf` are the requested
|
||||
/// buffer sizes; they're applied with `SO_*BUFFORCE` first and
|
||||
/// fall back to the normal `SO_*BUF` if the process can't bypass
|
||||
/// the kernel ceiling.
|
||||
pub fn open(
|
||||
local_addr: SocketAddr,
|
||||
peer_addr: SocketAddr,
|
||||
recv_buf: usize,
|
||||
send_buf: usize,
|
||||
) -> io::Result<Self> {
|
||||
// Family must match between local and peer.
|
||||
if local_addr.is_ipv4() != peer_addr.is_ipv4() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidInput,
|
||||
"ConnectedPeerSocket: local + peer address families differ",
|
||||
));
|
||||
}
|
||||
|
||||
let domain = if local_addr.is_ipv4() {
|
||||
libc::AF_INET
|
||||
} else {
|
||||
libc::AF_INET6
|
||||
};
|
||||
// Linux accepts SOCK_NONBLOCK | SOCK_CLOEXEC directly. Darwin
|
||||
// does not, so we set the equivalent fd flags with fcntl below.
|
||||
#[cfg(target_os = "linux")]
|
||||
let typ = libc::SOCK_DGRAM | libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC;
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let typ = libc::SOCK_DGRAM;
|
||||
let fd = unsafe { libc::socket(domain, typ, libc::IPPROTO_UDP) };
|
||||
if fd < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
// Take ownership of the fd so we close it on any error below.
|
||||
let sock = ConnectedPeerSocket {
|
||||
fd,
|
||||
peer_addr,
|
||||
local_addr,
|
||||
};
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
sock.set_nonblocking_cloexec()?;
|
||||
|
||||
// SO_REUSEADDR lets us bind to the same local port the listen
|
||||
// socket already holds. SO_REUSEPORT lets the UDP demux permit
|
||||
// several sockets bound to the same address and route the peer
|
||||
// 5-tuple to the connected sibling.
|
||||
sock.set_sockopt_int(libc::SOL_SOCKET, libc::SO_REUSEADDR, 1)?;
|
||||
sock.set_sockopt_int(libc::SOL_SOCKET, libc::SO_REUSEPORT, 1)?;
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
crate::transport::udp::darwin_sockopts::apply_udp_socket_tuning(
|
||||
sock.fd,
|
||||
"connected-udp-peer",
|
||||
);
|
||||
|
||||
// Buffer sizes — try the FORCE variants first (succeed if we
|
||||
// have CAP_NET_ADMIN), then fall back to the ceiling-clamped
|
||||
// normal variants. The ceiling-clamped path always succeeds
|
||||
// even if it gives us less than we asked for.
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
sock.set_buf_size(libc::SO_RCVBUFFORCE, libc::SO_RCVBUF, recv_buf);
|
||||
sock.set_buf_size(libc::SO_SNDBUFFORCE, libc::SO_SNDBUF, send_buf);
|
||||
}
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
{
|
||||
sock.set_buf_size(libc::SO_RCVBUF, recv_buf);
|
||||
sock.set_buf_size(libc::SO_SNDBUF, send_buf);
|
||||
}
|
||||
|
||||
// Bind to the wildcard local address (same port as listen socket).
|
||||
let local_sa: socket2::SockAddr = local_addr.into();
|
||||
let bind_r = unsafe {
|
||||
libc::bind(
|
||||
sock.fd,
|
||||
local_sa.as_ptr() as *const libc::sockaddr,
|
||||
local_sa.len(),
|
||||
)
|
||||
};
|
||||
if bind_r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
|
||||
// Connect to the peer — locks in the per-packet kernel route.
|
||||
let peer_sa: socket2::SockAddr = peer_addr.into();
|
||||
let conn_r = unsafe {
|
||||
libc::connect(
|
||||
sock.fd,
|
||||
peer_sa.as_ptr() as *const libc::sockaddr,
|
||||
peer_sa.len(),
|
||||
)
|
||||
};
|
||||
if conn_r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
|
||||
Ok(sock)
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
fn set_nonblocking_cloexec(&self) -> io::Result<()> {
|
||||
let flags = unsafe { libc::fcntl(self.fd, libc::F_GETFL) };
|
||||
if flags < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if unsafe { libc::fcntl(self.fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
|
||||
let fd_flags = unsafe { libc::fcntl(self.fd, libc::F_GETFD) };
|
||||
if fd_flags < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if unsafe { libc::fcntl(self.fd, libc::F_SETFD, fd_flags | libc::FD_CLOEXEC) } < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Set an integer-valued socket option. Returns the kernel error
|
||||
/// on failure but doesn't `?`-propagate caller-side because most
|
||||
/// callers want to log + continue rather than fail the whole open.
|
||||
fn set_sockopt_int(
|
||||
&self,
|
||||
level: libc::c_int,
|
||||
name: libc::c_int,
|
||||
value: libc::c_int,
|
||||
) -> io::Result<()> {
|
||||
let r = unsafe {
|
||||
libc::setsockopt(
|
||||
self.fd,
|
||||
level,
|
||||
name,
|
||||
&value as *const _ as *const libc::c_void,
|
||||
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
if r < 0 {
|
||||
Err(io::Error::last_os_error())
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Try `SO_*BUFFORCE` first (bypasses the rmem/wmem ceiling) and
|
||||
/// fall back to `SO_*BUF` if that fails. Returns silently — buffer
|
||||
/// sizing is best-effort.
|
||||
#[cfg(target_os = "linux")]
|
||||
fn set_buf_size(&self, force_name: libc::c_int, normal_name: libc::c_int, size: usize) {
|
||||
let value: libc::c_int = size as libc::c_int;
|
||||
let r = unsafe {
|
||||
libc::setsockopt(
|
||||
self.fd,
|
||||
libc::SOL_SOCKET,
|
||||
force_name,
|
||||
&value as *const _ as *const libc::c_void,
|
||||
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
if r < 0 {
|
||||
// Fall back to non-force — kernel may clamp.
|
||||
let _ = unsafe {
|
||||
libc::setsockopt(
|
||||
self.fd,
|
||||
libc::SOL_SOCKET,
|
||||
normal_name,
|
||||
&value as *const _ as *const libc::c_void,
|
||||
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
fn set_buf_size(&self, normal_name: libc::c_int, size: usize) {
|
||||
let value: libc::c_int = size as libc::c_int;
|
||||
let _ = unsafe {
|
||||
libc::setsockopt(
|
||||
self.fd,
|
||||
libc::SOL_SOCKET,
|
||||
normal_name,
|
||||
&value as *const _ as *const libc::c_void,
|
||||
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
}
|
||||
|
||||
pub fn peer_addr(&self) -> SocketAddr {
|
||||
self.peer_addr
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // wired up by future per-peer recv loops
|
||||
pub fn local_addr(&self) -> SocketAddr {
|
||||
self.local_addr
|
||||
}
|
||||
}
|
||||
|
||||
impl AsRawFd for ConnectedPeerSocket {
|
||||
fn as_raw_fd(&self) -> RawFd {
|
||||
self.fd
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ConnectedPeerSocket {
|
||||
fn drop(&mut self) {
|
||||
// Best-effort close. Ignore the result — if close fails the
|
||||
// kernel has already done what it can; we don't want to panic
|
||||
// in Drop.
|
||||
unsafe {
|
||||
libc::close(self.fd);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::net::UdpSocket;
|
||||
|
||||
/// Open a connected peer socket against a fresh loopback UDP
|
||||
/// listener and exercise the round-trip: connected socket sends
|
||||
/// without msg_name → listener receives → listener replies →
|
||||
/// connected socket receives without parsing msg_name. Validates
|
||||
/// reuse flags + `bind` + `connect` + `O_NONBLOCK`.
|
||||
#[test]
|
||||
fn open_send_recv_loopback() {
|
||||
// Peer (the "remote") side: a regular blocking UDP socket on
|
||||
// loopback. We'll have our connected socket send to it.
|
||||
let peer = UdpSocket::bind("127.0.0.1:0").expect("bind peer");
|
||||
let peer_addr = peer.local_addr().expect("peer local_addr");
|
||||
peer.set_read_timeout(Some(std::time::Duration::from_millis(500)))
|
||||
.expect("set_read_timeout");
|
||||
|
||||
// Our side: a wildcard listen address (use 127.0.0.1:0 to
|
||||
// avoid colliding with any real local service). Connect to the
|
||||
// peer. Linux requires that we bind before connect — the
|
||||
// ConnectedPeerSocket constructor does both.
|
||||
let local_addr: SocketAddr = "127.0.0.1:0".parse().unwrap();
|
||||
let sock = ConnectedPeerSocket::open(
|
||||
local_addr,
|
||||
peer_addr,
|
||||
/* recv_buf */ 1 << 20,
|
||||
/* send_buf */ 1 << 20,
|
||||
)
|
||||
.expect("ConnectedPeerSocket::open");
|
||||
|
||||
// Confirm the socket is in fact connected: `send(2)` should
|
||||
// succeed without specifying a destination.
|
||||
let payload = b"hello-from-connected-socket";
|
||||
let r = unsafe {
|
||||
libc::send(
|
||||
sock.as_raw_fd(),
|
||||
payload.as_ptr() as *const libc::c_void,
|
||||
payload.len(),
|
||||
0,
|
||||
)
|
||||
};
|
||||
assert!(r >= 0, "send failed: {}", std::io::Error::last_os_error());
|
||||
assert_eq!(r as usize, payload.len());
|
||||
|
||||
let mut recv_buf = [0u8; 64];
|
||||
let (len, from) = peer.recv_from(&mut recv_buf).expect("peer recv");
|
||||
assert_eq!(len, payload.len());
|
||||
assert_eq!(&recv_buf[..len], payload);
|
||||
|
||||
// Reply back from the peer. Since our socket is connected to
|
||||
// peer_addr, the kernel UDP demux should route this packet to
|
||||
// our connected socket (most-specific-match) and `recv(2)`
|
||||
// without sockaddr should pick it up.
|
||||
let reply = b"hello-back";
|
||||
peer.send_to(reply, from).expect("peer send_to");
|
||||
|
||||
// Drain on the connected socket. Spin briefly because
|
||||
// O_NONBLOCK + a tiny one-shot recv would race with the
|
||||
// kernel's veth-less loopback delivery.
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_millis(500);
|
||||
loop {
|
||||
let mut buf = [0u8; 64];
|
||||
let r = unsafe {
|
||||
libc::recv(
|
||||
sock.as_raw_fd(),
|
||||
buf.as_mut_ptr() as *mut libc::c_void,
|
||||
buf.len(),
|
||||
0,
|
||||
)
|
||||
};
|
||||
if r >= 0 {
|
||||
assert_eq!(r as usize, reply.len());
|
||||
assert_eq!(&buf[..r as usize], reply);
|
||||
break;
|
||||
}
|
||||
let err = std::io::Error::last_os_error();
|
||||
if err.kind() == std::io::ErrorKind::WouldBlock {
|
||||
if std::time::Instant::now() >= deadline {
|
||||
panic!("connected socket never received reply");
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(2));
|
||||
continue;
|
||||
}
|
||||
panic!("recv failed: {err}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Two connected sockets coexisting on the same local port via
|
||||
/// `SO_REUSEPORT`, each connected to a different peer.
|
||||
#[test]
|
||||
fn two_connected_sockets_share_listen_port() {
|
||||
let peer_a = UdpSocket::bind("127.0.0.1:0").expect("bind peer_a");
|
||||
let peer_b = UdpSocket::bind("127.0.0.1:0").expect("bind peer_b");
|
||||
let peer_a_addr = peer_a.local_addr().expect("peer_a local_addr");
|
||||
let peer_b_addr = peer_b.local_addr().expect("peer_b local_addr");
|
||||
|
||||
// Anchor a shared local port via a wildcard socket on a
|
||||
// non-zero ephemeral port, then open two connected sockets
|
||||
// bound to the same port.
|
||||
let anchor = UdpSocket::bind("127.0.0.1:0").expect("bind anchor");
|
||||
let shared_port = anchor.local_addr().expect("anchor local_addr").port();
|
||||
let shared_local: SocketAddr = format!("127.0.0.1:{shared_port}").parse().unwrap();
|
||||
// Drop the anchor so the only thing holding the port is the
|
||||
// connected sockets' reuse semantics.
|
||||
drop(anchor);
|
||||
|
||||
let sock_a = ConnectedPeerSocket::open(shared_local, peer_a_addr, 1 << 20, 1 << 20)
|
||||
.expect("open sock_a");
|
||||
let sock_b = ConnectedPeerSocket::open(shared_local, peer_b_addr, 1 << 20, 1 << 20)
|
||||
.expect("open sock_b");
|
||||
|
||||
assert_eq!(sock_a.peer_addr(), peer_a_addr);
|
||||
assert_eq!(sock_b.peer_addr(), peer_b_addr);
|
||||
}
|
||||
|
||||
/// The production fast path keeps the wildcard UDP listener bound
|
||||
/// while opening a sibling socket connected to a peer. This catches
|
||||
/// the Darwin regression where the adopted traversal socket used a
|
||||
/// different reuse mode than the connected-peer socket and every
|
||||
/// activation failed with EADDRINUSE.
|
||||
#[test]
|
||||
fn connected_socket_shares_live_listener_port() {
|
||||
let peer = UdpSocket::bind("127.0.0.1:0").expect("bind peer");
|
||||
let peer_addr = peer.local_addr().expect("peer local_addr");
|
||||
|
||||
let listener = socket2::Socket::new(
|
||||
socket2::Domain::IPV4,
|
||||
socket2::Type::DGRAM,
|
||||
Some(socket2::Protocol::UDP),
|
||||
)
|
||||
.expect("create listener");
|
||||
listener
|
||||
.set_reuse_address(true)
|
||||
.expect("listener reuseaddr");
|
||||
listener.set_reuse_port(true).expect("listener reuseport");
|
||||
listener
|
||||
.bind(&"0.0.0.0:0".parse::<SocketAddr>().unwrap().into())
|
||||
.expect("bind listener");
|
||||
let local = listener
|
||||
.local_addr()
|
||||
.expect("listener local addr")
|
||||
.as_socket()
|
||||
.expect("ip socket");
|
||||
|
||||
let sock = ConnectedPeerSocket::open(local, peer_addr, 1 << 20, 1 << 20)
|
||||
.expect("open connected sibling");
|
||||
|
||||
assert_eq!(sock.local_addr(), local);
|
||||
assert_eq!(sock.peer_addr(), peer_addr);
|
||||
}
|
||||
}
|
||||
197
src/transport/udp/darwin_sockopts.rs
Normal file
197
src/transport/udp/darwin_sockopts.rs
Normal file
@@ -0,0 +1,197 @@
|
||||
// Applied at `ConnectedPeerSocket::open` (dormant in this PR; see
|
||||
// `connected_peer.rs`). Linux toolchain only checks gates — keep
|
||||
// the module visible on Linux so clippy doesn't lose track.
|
||||
#![allow(dead_code)]
|
||||
|
||||
//! Darwin UDP socket tuning.
|
||||
//!
|
||||
//! macOS does not expose UDP GSO/TSO to userspace tunnels, so high-rate
|
||||
//! Wi-Fi sends are often limited by per-datagram kernel scheduling. The
|
||||
//! service type is the one low-cost Darwin hint that can change how the
|
||||
//! socket is queued by the host networking stack and Wi-Fi WMM.
|
||||
|
||||
#![cfg(target_os = "macos")]
|
||||
|
||||
use std::io;
|
||||
use std::os::fd::RawFd;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
use tracing::{debug, warn};
|
||||
|
||||
const SO_NET_SERVICE_TYPE: libc::c_int = 0x1116;
|
||||
|
||||
const NET_SERVICE_TYPE_BE: libc::c_int = 0;
|
||||
const NET_SERVICE_TYPE_BK: libc::c_int = 1;
|
||||
const NET_SERVICE_TYPE_SIG: libc::c_int = 2;
|
||||
const NET_SERVICE_TYPE_VI: libc::c_int = 3;
|
||||
const NET_SERVICE_TYPE_VO: libc::c_int = 4;
|
||||
const NET_SERVICE_TYPE_RV: libc::c_int = 5;
|
||||
const NET_SERVICE_TYPE_AV: libc::c_int = 6;
|
||||
const NET_SERVICE_TYPE_OAM: libc::c_int = 7;
|
||||
const NET_SERVICE_TYPE_RD: libc::c_int = 8;
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
struct NetServiceType {
|
||||
name: &'static str,
|
||||
value: libc::c_int,
|
||||
}
|
||||
|
||||
const VPN_SERVICE_TYPE: NetServiceType = NetServiceType {
|
||||
name: "oam",
|
||||
value: NET_SERVICE_TYPE_OAM,
|
||||
};
|
||||
|
||||
#[derive(Debug)]
|
||||
struct Tuning {
|
||||
service_type: Option<NetServiceType>,
|
||||
}
|
||||
|
||||
static TUNING: OnceLock<Tuning> = OnceLock::new();
|
||||
|
||||
pub(crate) fn apply_udp_socket_tuning(fd: RawFd, context: &'static str) {
|
||||
if let Some(service_type) = tuning().service_type {
|
||||
match set_sockopt_int(
|
||||
fd,
|
||||
libc::SOL_SOCKET,
|
||||
SO_NET_SERVICE_TYPE,
|
||||
service_type.value,
|
||||
) {
|
||||
Ok(()) => {
|
||||
debug!(
|
||||
context,
|
||||
service_type = service_type.name,
|
||||
value = service_type.value,
|
||||
"set Darwin UDP SO_NET_SERVICE_TYPE"
|
||||
);
|
||||
}
|
||||
Err(err) => {
|
||||
warn!(
|
||||
context,
|
||||
service_type = service_type.name,
|
||||
value = service_type.value,
|
||||
%err,
|
||||
"failed to set Darwin UDP SO_NET_SERVICE_TYPE"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn tuning() -> &'static Tuning {
|
||||
TUNING.get_or_init(|| {
|
||||
let service_type = match std::env::var("FIPS_MACOS_NET_SERVICE_TYPE") {
|
||||
Ok(raw) => match parse_net_service_type(&raw) {
|
||||
Ok(service_type) => service_type,
|
||||
Err(()) => {
|
||||
warn!(
|
||||
value = %raw,
|
||||
default = "off",
|
||||
"invalid FIPS_MACOS_NET_SERVICE_TYPE, using default"
|
||||
);
|
||||
None
|
||||
}
|
||||
},
|
||||
// Apple documents NET_SERVICE_TYPE_OAM as fitting VPN tunnels, but
|
||||
// measured MacBook Wi-Fi sends regressed badly with OAM/RD/VI
|
||||
// marking in 2026-05 local LAN tests. Leave Darwin UDP sockets at
|
||||
// the kernel default unless an experiment opts in via env.
|
||||
Err(_) => None,
|
||||
};
|
||||
|
||||
Tuning { service_type }
|
||||
})
|
||||
}
|
||||
|
||||
fn parse_net_service_type(raw: &str) -> Result<Option<NetServiceType>, ()> {
|
||||
let normalized = raw.trim().to_ascii_lowercase().replace(['-', '_'], "");
|
||||
let service_type = match normalized.as_str() {
|
||||
"" | "off" | "none" | "disabled" | "disable" | "unset" | "default" => None,
|
||||
"vpn" | "oam" => Some(VPN_SERVICE_TYPE),
|
||||
"be" | "besteffort" => Some(NetServiceType {
|
||||
name: "be",
|
||||
value: NET_SERVICE_TYPE_BE,
|
||||
}),
|
||||
"bk" | "background" => Some(NetServiceType {
|
||||
name: "bk",
|
||||
value: NET_SERVICE_TYPE_BK,
|
||||
}),
|
||||
"sig" | "signaling" => Some(NetServiceType {
|
||||
name: "sig",
|
||||
value: NET_SERVICE_TYPE_SIG,
|
||||
}),
|
||||
"vi" | "interactivevideo" | "video" => Some(NetServiceType {
|
||||
name: "vi",
|
||||
value: NET_SERVICE_TYPE_VI,
|
||||
}),
|
||||
"vo" | "interactivevoice" | "voice" => Some(NetServiceType {
|
||||
name: "vo",
|
||||
value: NET_SERVICE_TYPE_VO,
|
||||
}),
|
||||
"rv" | "responsivemultimedia" | "responsivemultimediavideo" => Some(NetServiceType {
|
||||
name: "rv",
|
||||
value: NET_SERVICE_TYPE_RV,
|
||||
}),
|
||||
"av" | "multimedia" | "audiovideo" => Some(NetServiceType {
|
||||
name: "av",
|
||||
value: NET_SERVICE_TYPE_AV,
|
||||
}),
|
||||
"rd" | "responsivedata" => Some(NetServiceType {
|
||||
name: "rd",
|
||||
value: NET_SERVICE_TYPE_RD,
|
||||
}),
|
||||
_ => return Err(()),
|
||||
};
|
||||
Ok(service_type)
|
||||
}
|
||||
|
||||
fn set_sockopt_int(
|
||||
fd: RawFd,
|
||||
level: libc::c_int,
|
||||
name: libc::c_int,
|
||||
value: libc::c_int,
|
||||
) -> io::Result<()> {
|
||||
let r = unsafe {
|
||||
libc::setsockopt(
|
||||
fd,
|
||||
level,
|
||||
name,
|
||||
&value as *const _ as *const libc::c_void,
|
||||
std::mem::size_of::<libc::c_int>() as libc::socklen_t,
|
||||
)
|
||||
};
|
||||
if r < 0 {
|
||||
Err(io::Error::last_os_error())
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parses_service_type_aliases() {
|
||||
assert_eq!(parse_net_service_type("off").unwrap(), None);
|
||||
assert_eq!(parse_net_service_type("default").unwrap(), None);
|
||||
assert_eq!(
|
||||
parse_net_service_type("vpn").unwrap(),
|
||||
Some(VPN_SERVICE_TYPE)
|
||||
);
|
||||
assert_eq!(
|
||||
parse_net_service_type("interactive_video").unwrap(),
|
||||
Some(NetServiceType {
|
||||
name: "vi",
|
||||
value: NET_SERVICE_TYPE_VI
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
parse_net_service_type("responsive-data").unwrap(),
|
||||
Some(NetServiceType {
|
||||
name: "rd",
|
||||
value: NET_SERVICE_TYPE_RD
|
||||
})
|
||||
);
|
||||
assert!(parse_net_service_type("wat").is_err());
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,13 @@ use super::{
|
||||
DiscoveredPeer, PacketTx, ReceivedPacket, Transport, TransportAddr, TransportError,
|
||||
TransportId, TransportState, TransportType,
|
||||
};
|
||||
mod socket;
|
||||
#[cfg(unix)]
|
||||
pub(crate) mod connected_peer;
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(crate) mod darwin_sockopts;
|
||||
#[cfg(unix)]
|
||||
pub(crate) mod peer_drain;
|
||||
pub(crate) mod socket;
|
||||
mod stats;
|
||||
use super::resolve_socket_addr;
|
||||
use crate::config::UdpConfig;
|
||||
@@ -83,11 +89,46 @@ impl UdpTransport {
|
||||
self.local_addr
|
||||
}
|
||||
|
||||
/// Configured recv buffer size — used when opening per-peer
|
||||
/// `ConnectedPeerSocket`s so they get the same buffer ceiling as
|
||||
/// the wildcard listen socket.
|
||||
pub fn recv_buf_size(&self) -> usize {
|
||||
self.config.recv_buf_size()
|
||||
}
|
||||
|
||||
/// Configured send buffer size — companion to `recv_buf_size`.
|
||||
pub fn send_buf_size(&self) -> usize {
|
||||
self.config.send_buf_size()
|
||||
}
|
||||
|
||||
/// Clone the `PacketTx` end of the packet channel for off-task
|
||||
/// receive paths (per-peer connected-socket drains).
|
||||
pub fn clone_packet_tx(&self) -> PacketTx {
|
||||
self.packet_tx.clone()
|
||||
}
|
||||
|
||||
/// Get the transport statistics.
|
||||
pub fn stats(&self) -> &Arc<UdpStats> {
|
||||
&self.stats
|
||||
}
|
||||
|
||||
/// Resolve a transport address (numeric `1.2.3.4:5678` or hostname)
|
||||
/// to a `SocketAddr` via the per-transport DNS cache. Public
|
||||
/// companion to `async_socket()` for off-task workers.
|
||||
pub async fn resolve_for_off_task(
|
||||
&self,
|
||||
addr: &TransportAddr,
|
||||
) -> Result<SocketAddr, TransportError> {
|
||||
self.resolve_cached(addr).await
|
||||
}
|
||||
|
||||
/// Clone the underlying async UDP socket. Returns `None` if the
|
||||
/// transport hasn't been started yet. The clone is just an `Arc`
|
||||
/// refcount bump on `AsyncFd<UdpRawSocket>`.
|
||||
pub fn async_socket(&self) -> Option<AsyncUdpSocket> {
|
||||
self.socket.clone()
|
||||
}
|
||||
|
||||
/// Resolve a transport address, using cached results for hostnames.
|
||||
///
|
||||
/// Numeric IP addresses bypass the cache entirely. Hostnames are
|
||||
@@ -456,6 +497,8 @@ async fn udp_receive_loop(
|
||||
};
|
||||
stats.record_recv(len);
|
||||
|
||||
// Peek before swap — punch probes / acks are
|
||||
// discarded without consuming a buffer move.
|
||||
if is_punch_packet(&backing[i][..len]) {
|
||||
trace!(
|
||||
transport_id = %transport_id,
|
||||
@@ -466,6 +509,13 @@ async fn udp_receive_loop(
|
||||
continue;
|
||||
}
|
||||
|
||||
// Move the filled buffer out of the slot and
|
||||
// refill with a fresh one. `mem::replace`
|
||||
// returns the OLD Vec and writes the new one —
|
||||
// single pointer swap, no per-packet memcpy of
|
||||
// the ~MTU-sized payload (previously
|
||||
// `buf.to_vec()` cost ~150 MB/sec of memory
|
||||
// bandwidth on the RX hot path at 100 kpps).
|
||||
let mut data = std::mem::replace(&mut backing[i], vec![0u8; buf_size]);
|
||||
data.truncate(len);
|
||||
let addr = TransportAddr::from_socket_addr(remote_addr);
|
||||
@@ -520,7 +570,7 @@ async fn udp_receive_loop(
|
||||
}
|
||||
|
||||
let data = buf[..len].to_vec();
|
||||
let addr = TransportAddr::from_string(&remote_addr.to_string());
|
||||
let addr = TransportAddr::from_socket_addr(remote_addr);
|
||||
let packet = ReceivedPacket::new(transport_id, addr, data);
|
||||
|
||||
trace!(
|
||||
|
||||
457
src/transport/udp/peer_drain.rs
Normal file
457
src/transport/udp/peer_drain.rs
Normal file
@@ -0,0 +1,457 @@
|
||||
// Paired with `connected_peer.rs`: dormant in this PR until the
|
||||
// activation handler is wired into the node tick (follow-up).
|
||||
#![allow(dead_code)]
|
||||
|
||||
//! Recv-side drain thread for a per-peer connected UDP socket.
|
||||
//!
|
||||
//! Once a UDP socket is `connect()`-ed to a peer, Linux and Darwin
|
||||
//! UDP demux preferentially route inbound packets matching the peer's
|
||||
//! 5-tuple to that socket (most-specific match wins over the wildcard
|
||||
//! listen socket under `SO_REUSEPORT`). So a connected socket **must**
|
||||
//! be drained, or packets pile up in its recv buffer until it overflows
|
||||
//! and the kernel drops them silently.
|
||||
//!
|
||||
//! This module owns the drain side: spawn one OS thread per connected
|
||||
//! socket, drain into a fixed-size batch (`recvmmsg(2)` on Linux,
|
||||
//! repeated nonblocking `recv(2)` on Darwin), push each packet into
|
||||
//! the existing `packet_tx` (the same channel that the wildcard listen
|
||||
//! socket feeds), and exit cleanly when the parent signals shutdown
|
||||
//! via a self-pipe.
|
||||
//!
|
||||
//! Future: when the full data-plane shard lands, this per-peer thread
|
||||
//! becomes a `epoll_wait` arm inside the shard's event loop instead
|
||||
//! of a dedicated OS thread. The drain *function* `drain_loop` stays
|
||||
//! useful in either shape; only the wakeup mechanism differs.
|
||||
|
||||
#![cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
|
||||
use super::super::{ReceivedPacket, TransportAddr, TransportId};
|
||||
use super::PacketTx;
|
||||
use super::connected_peer::ConnectedPeerSocket;
|
||||
use std::io;
|
||||
use std::net::SocketAddr;
|
||||
use std::os::unix::io::{AsRawFd, RawFd};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use tracing::{debug, trace, warn};
|
||||
|
||||
/// Handle to a running per-peer drain thread. Drops the thread (and
|
||||
/// closes its self-pipe) on drop; the thread exits next time it
|
||||
/// returns from `poll(2)`.
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct PeerRecvDrain {
|
||||
/// Write end of the shutdown self-pipe. Write a single byte to
|
||||
/// wake the drain thread out of `poll(2)` so it sees the stop
|
||||
/// flag and exits.
|
||||
stop_pipe_tx: RawFd,
|
||||
/// Atomic stop signal — primary mechanism for the drain thread
|
||||
/// to know it should exit. Set before writing to `stop_pipe_tx`
|
||||
/// so the thread observes the flag once woken.
|
||||
stop: Arc<AtomicBool>,
|
||||
/// Joined on drop; the thread is cheap (just exits after the
|
||||
/// next `poll` returns) so the wait is bounded.
|
||||
join: Option<std::thread::JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl PeerRecvDrain {
|
||||
/// Spawn a drain thread for the given connected socket.
|
||||
///
|
||||
/// The thread holds an `Arc<ConnectedPeerSocket>` to keep the
|
||||
/// kernel fd alive while it's running. When this handle drops,
|
||||
/// the stop pipe fires; the thread exits; its `Arc` releases.
|
||||
/// If the parent also releases its `Arc`, the socket's `Drop`
|
||||
/// closes the kernel fd.
|
||||
pub fn spawn(
|
||||
socket: Arc<ConnectedPeerSocket>,
|
||||
transport_id: TransportId,
|
||||
peer_addr: SocketAddr,
|
||||
packet_tx: PacketTx,
|
||||
) -> io::Result<Self> {
|
||||
// Self-pipe for shutdown signaling. The drain thread polls
|
||||
// (socket_fd | pipe_rx) so a write to pipe_tx wakes it.
|
||||
let (pipe_rx, pipe_tx) = make_pipe()?;
|
||||
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
|
||||
let stop_clone = stop.clone();
|
||||
let socket_clone = socket.clone();
|
||||
let thread = std::thread::Builder::new()
|
||||
.name(format!("fips-peer-drain-{}", socket.peer_addr()))
|
||||
.spawn(move || {
|
||||
drain_loop(
|
||||
socket_clone,
|
||||
transport_id,
|
||||
peer_addr,
|
||||
packet_tx,
|
||||
pipe_rx,
|
||||
stop_clone,
|
||||
);
|
||||
// Drain thread cleans up the read end of the pipe on exit.
|
||||
unsafe { libc::close(pipe_rx) };
|
||||
});
|
||||
|
||||
match thread {
|
||||
Ok(join) => Ok(Self {
|
||||
stop_pipe_tx: pipe_tx,
|
||||
stop,
|
||||
join: Some(join),
|
||||
}),
|
||||
Err(e) => {
|
||||
unsafe {
|
||||
libc::close(pipe_rx);
|
||||
libc::close(pipe_tx);
|
||||
}
|
||||
Err(io::Error::other(format!(
|
||||
"failed to spawn peer drain thread: {e}"
|
||||
)))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for PeerRecvDrain {
|
||||
fn drop(&mut self) {
|
||||
// 1. Set the stop flag.
|
||||
self.stop.store(true, Ordering::Release);
|
||||
// 2. Wake the drain thread by writing to the self-pipe. One
|
||||
// byte is enough; the thread's poll will return on
|
||||
// POLLIN, observe the stop flag, and exit.
|
||||
let byte = 1u8;
|
||||
let _ = unsafe { libc::write(self.stop_pipe_tx, &byte as *const _ as *const _, 1) };
|
||||
// 3. Join — bounded wait, the thread exits within one
|
||||
// poll-iteration of seeing the stop flag.
|
||||
if let Some(j) = self.join.take() {
|
||||
let _ = j.join();
|
||||
}
|
||||
// 4. Close the write end of the pipe.
|
||||
unsafe { libc::close(self.stop_pipe_tx) };
|
||||
}
|
||||
}
|
||||
|
||||
/// The drain thread's main loop. Runs until `stop` is set + the
|
||||
/// stop-pipe is written to (Drop does both in order).
|
||||
fn drain_loop(
|
||||
socket: Arc<ConnectedPeerSocket>,
|
||||
transport_id: TransportId,
|
||||
peer_addr: SocketAddr,
|
||||
packet_tx: PacketTx,
|
||||
stop_pipe_rx: RawFd,
|
||||
stop: Arc<AtomicBool>,
|
||||
) {
|
||||
let socket_fd = socket.as_raw_fd();
|
||||
trace!(
|
||||
transport_id = %transport_id,
|
||||
peer_addr = %peer_addr,
|
||||
"fips-peer-drain: starting"
|
||||
);
|
||||
|
||||
const BATCH: usize = 32;
|
||||
const BUF_SIZE: usize = 1600; // covers any practical FIPS MTU.
|
||||
let mut backing: Vec<Vec<u8>> = (0..BATCH).map(|_| vec![0u8; BUF_SIZE]).collect();
|
||||
let mut lens: [usize; BATCH] = [0; BATCH];
|
||||
let packet_addr = TransportAddr::from_socket_addr(peer_addr);
|
||||
|
||||
loop {
|
||||
if stop.load(Ordering::Acquire) {
|
||||
break;
|
||||
}
|
||||
|
||||
// poll(2) on the socket + stop pipe. -1 timeout = block
|
||||
// until at least one is readable; the stop pipe wake-up
|
||||
// guarantees forward progress under Drop.
|
||||
let mut pfds = [
|
||||
libc::pollfd {
|
||||
fd: socket_fd,
|
||||
events: libc::POLLIN,
|
||||
revents: 0,
|
||||
},
|
||||
libc::pollfd {
|
||||
fd: stop_pipe_rx,
|
||||
events: libc::POLLIN,
|
||||
revents: 0,
|
||||
},
|
||||
];
|
||||
let r = unsafe { libc::poll(pfds.as_mut_ptr(), 2, -1) };
|
||||
if r < 0 {
|
||||
let err = io::Error::last_os_error();
|
||||
if err.kind() == io::ErrorKind::Interrupted {
|
||||
continue;
|
||||
}
|
||||
warn!(error = %err, "fips-peer-drain: poll failed; exiting");
|
||||
break;
|
||||
}
|
||||
if pfds[1].revents != 0 {
|
||||
// Stop pipe fired. We may or may not also have data on
|
||||
// the socket; check the flag and exit if set.
|
||||
if stop.load(Ordering::Acquire) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if pfds[0].revents & libc::POLLIN == 0 {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Drain whatever is currently queued in the kernel.
|
||||
let n = drain_packets(socket_fd, &mut backing, &mut lens);
|
||||
let count = match n {
|
||||
Ok(c) => c,
|
||||
Err(err) if err.kind() == io::ErrorKind::WouldBlock => continue,
|
||||
Err(err) => {
|
||||
debug!(error = %err, "fips-peer-drain: recv failed; exiting");
|
||||
break;
|
||||
}
|
||||
};
|
||||
|
||||
for i in 0..count {
|
||||
let len = lens[i];
|
||||
if len == 0 {
|
||||
continue;
|
||||
}
|
||||
// Move the filled buffer out, refill the slot with a
|
||||
// fresh one. Same zero-copy pattern the wildcard listen
|
||||
// socket uses (see `transport/udp/mod.rs::run_receive_loop`).
|
||||
let mut data = std::mem::replace(&mut backing[i], vec![0u8; BUF_SIZE]);
|
||||
data.truncate(len);
|
||||
let packet = ReceivedPacket::new(transport_id, packet_addr.clone(), data);
|
||||
// Drain runs on a std::thread, packet_tx is tokio::mpsc::Sender;
|
||||
// `send` returns a Future. `blocking_send` blocks the OS thread
|
||||
// until the channel has capacity — exactly what we want here.
|
||||
if packet_tx.blocking_send(packet).is_err() {
|
||||
trace!("fips-peer-drain: packet channel closed; exiting");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
trace!(
|
||||
transport_id = %transport_id,
|
||||
peer_addr = %peer_addr,
|
||||
"fips-peer-drain: stopped"
|
||||
);
|
||||
}
|
||||
|
||||
fn make_pipe() -> io::Result<(RawFd, RawFd)> {
|
||||
let mut pipe_fds = [0i32; 2];
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
let r = unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC | libc::O_NONBLOCK) };
|
||||
if r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
}
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
{
|
||||
let r = unsafe { libc::pipe(pipe_fds.as_mut_ptr()) };
|
||||
if r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if let Err(err) = set_nonblocking_cloexec(pipe_fds[0]) {
|
||||
unsafe {
|
||||
libc::close(pipe_fds[0]);
|
||||
libc::close(pipe_fds[1]);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
if let Err(err) = set_nonblocking_cloexec(pipe_fds[1]) {
|
||||
unsafe {
|
||||
libc::close(pipe_fds[0]);
|
||||
libc::close(pipe_fds[1]);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
Ok((pipe_fds[0], pipe_fds[1]))
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
fn set_nonblocking_cloexec(fd: RawFd) -> io::Result<()> {
|
||||
let flags = unsafe { libc::fcntl(fd, libc::F_GETFL) };
|
||||
if flags < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if unsafe { libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
|
||||
let fd_flags = unsafe { libc::fcntl(fd, libc::F_GETFD) };
|
||||
if fd_flags < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
if unsafe { libc::fcntl(fd, libc::F_SETFD, fd_flags | libc::FD_CLOEXEC) } < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
fn drain_packets(fd: RawFd, backing: &mut [Vec<u8>], lens: &mut [usize]) -> io::Result<usize> {
|
||||
recvmmsg_drain(fd, backing, lens)
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
fn drain_packets(fd: RawFd, backing: &mut [Vec<u8>], lens: &mut [usize]) -> io::Result<usize> {
|
||||
recv_drain(fd, backing, lens)
|
||||
}
|
||||
|
||||
/// One-shot `recvmmsg(2)` on a non-blocking fd. Returns the number of
|
||||
/// datagrams received (0 on no data ready). Same minimal-overhead
|
||||
/// shape as the wildcard listen socket's `recv_batch` helper but
|
||||
/// without the kernel-drop counter cmsg (the listen socket samples
|
||||
/// that for the congestion detector; per-peer sockets share the
|
||||
/// kernel-wide UDP socket-buffer accounting already).
|
||||
#[cfg(target_os = "linux")]
|
||||
fn recvmmsg_drain(fd: RawFd, backing: &mut [Vec<u8>], lens: &mut [usize]) -> io::Result<usize> {
|
||||
const BATCH: usize = 32;
|
||||
let n = backing.len().min(lens.len()).min(BATCH);
|
||||
if n == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
let mut iovs: [libc::iovec; BATCH] = unsafe { std::mem::zeroed() };
|
||||
let mut storages: [libc::sockaddr_storage; BATCH] = unsafe { std::mem::zeroed() };
|
||||
let mut msgs: [libc::mmsghdr; BATCH] = unsafe { std::mem::zeroed() };
|
||||
for i in 0..n {
|
||||
iovs[i].iov_base = backing[i].as_mut_ptr() as *mut libc::c_void;
|
||||
iovs[i].iov_len = backing[i].len();
|
||||
msgs[i].msg_hdr.msg_name = &mut storages[i] as *mut _ as *mut libc::c_void;
|
||||
msgs[i].msg_hdr.msg_namelen =
|
||||
std::mem::size_of::<libc::sockaddr_storage>() as libc::socklen_t;
|
||||
msgs[i].msg_hdr.msg_iov = &mut iovs[i];
|
||||
// `msg_iovlen` is `usize` on glibc / `i32` on musl.
|
||||
msgs[i].msg_hdr.msg_iovlen = 1 as _;
|
||||
msgs[i].msg_len = 0;
|
||||
}
|
||||
|
||||
// `MSG_DONTWAIT` is `c_int` (i32) on glibc but `u32` on musl;
|
||||
// `as _` resolves to whichever the recvmmsg signature wants.
|
||||
let r = unsafe {
|
||||
libc::recvmmsg(
|
||||
fd,
|
||||
msgs.as_mut_ptr(),
|
||||
n as libc::c_uint,
|
||||
libc::MSG_DONTWAIT as _,
|
||||
std::ptr::null_mut(),
|
||||
)
|
||||
};
|
||||
if r < 0 {
|
||||
return Err(io::Error::last_os_error());
|
||||
}
|
||||
let count = r as usize;
|
||||
for i in 0..count {
|
||||
lens[i] = msgs[i].msg_len as usize;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
fn recv_drain(fd: RawFd, backing: &mut [Vec<u8>], lens: &mut [usize]) -> io::Result<usize> {
|
||||
let n = backing.len().min(lens.len());
|
||||
if n == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
let mut count = 0usize;
|
||||
while count < n {
|
||||
let r = unsafe {
|
||||
libc::recv(
|
||||
fd,
|
||||
backing[count].as_mut_ptr() as *mut libc::c_void,
|
||||
backing[count].len(),
|
||||
0,
|
||||
)
|
||||
};
|
||||
if r < 0 {
|
||||
let err = io::Error::last_os_error();
|
||||
if err.kind() == io::ErrorKind::Interrupted {
|
||||
continue;
|
||||
}
|
||||
if err.kind() == io::ErrorKind::WouldBlock && count > 0 {
|
||||
return Ok(count);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
lens[count] = r as usize;
|
||||
count += 1;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::net::UdpSocket;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
/// End-to-end: open a ConnectedPeerSocket, spawn a drain thread
|
||||
/// on it, send packets at it from a remote, verify they land in
|
||||
/// the packet_tx mpsc with the correct transport_id + peer_addr.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn drain_delivers_packets_to_packet_tx() {
|
||||
// Peer (remote) — sends packets at our connected socket.
|
||||
let peer = UdpSocket::bind("127.0.0.1:0").expect("bind peer");
|
||||
let peer_addr = peer.local_addr().expect("peer local_addr");
|
||||
|
||||
// Our connected socket. Use an ephemeral local port so we
|
||||
// don't conflict with anything else on the test host.
|
||||
let local_addr: SocketAddr = "127.0.0.1:0".parse().unwrap();
|
||||
let socket = Arc::new(
|
||||
ConnectedPeerSocket::open(local_addr, peer_addr, 1 << 20, 1 << 20)
|
||||
.expect("ConnectedPeerSocket::open"),
|
||||
);
|
||||
|
||||
// packet_tx for the drain thread to push into.
|
||||
let (tx, mut rx) = mpsc::channel::<ReceivedPacket>(64);
|
||||
let transport_id = TransportId::new(42);
|
||||
|
||||
// Find out what local_addr the kernel assigned to our socket
|
||||
// so the peer can sendto() it. Use getsockname; cast the
|
||||
// returned sockaddr_storage to sockaddr_in (we only test on
|
||||
// IPv4 loopback here, so this is safe).
|
||||
let our_local_addr: SocketAddr = {
|
||||
let mut storage: libc::sockaddr_storage = unsafe { std::mem::zeroed() };
|
||||
let mut len = std::mem::size_of::<libc::sockaddr_storage>() as libc::socklen_t;
|
||||
let r = unsafe {
|
||||
libc::getsockname(
|
||||
socket.as_raw_fd(),
|
||||
&mut storage as *mut _ as *mut libc::sockaddr,
|
||||
&mut len,
|
||||
)
|
||||
};
|
||||
assert!(r >= 0, "getsockname failed");
|
||||
assert_eq!(
|
||||
storage.ss_family as i32,
|
||||
libc::AF_INET,
|
||||
"test assumes IPv4 loopback"
|
||||
);
|
||||
let sin: &libc::sockaddr_in =
|
||||
unsafe { &*(&storage as *const _ as *const libc::sockaddr_in) };
|
||||
let port = u16::from_be(sin.sin_port);
|
||||
let ip = std::net::Ipv4Addr::from(u32::from_be(sin.sin_addr.s_addr));
|
||||
SocketAddr::from((ip, port))
|
||||
};
|
||||
|
||||
// Spawn the drain.
|
||||
let _drain = PeerRecvDrain::spawn(socket.clone(), transport_id, peer_addr, tx)
|
||||
.expect("PeerRecvDrain::spawn");
|
||||
|
||||
// Send a couple of packets from the peer to our socket.
|
||||
for i in 0u8..5 {
|
||||
let payload = [i, 0xAA, 0xBB, 0xCC];
|
||||
peer.send_to(&payload, our_local_addr).expect("peer sendto");
|
||||
}
|
||||
|
||||
// Verify the drain picked them up.
|
||||
for i in 0u8..5 {
|
||||
let pkt = tokio::time::timeout(Duration::from_millis(500), rx.recv())
|
||||
.await
|
||||
.unwrap_or_else(|_| panic!("timeout waiting for packet {i}"))
|
||||
.expect("packet channel closed");
|
||||
assert_eq!(pkt.transport_id, transport_id);
|
||||
assert_eq!(pkt.data.len(), 4);
|
||||
assert_eq!(pkt.data[0], i, "packet {i} payload mismatch");
|
||||
}
|
||||
// Drop the drain handle — should stop the thread within one
|
||||
// poll iteration.
|
||||
}
|
||||
}
|
||||
@@ -93,6 +93,16 @@ mod platform {
|
||||
TransportError::StartFailed(format!("set nonblocking failed: {}", e))
|
||||
})?;
|
||||
|
||||
// SO_REUSEPORT lets per-peer `ConnectedPeerSocket`s bind
|
||||
// to the same wildcard port the listen socket holds. Must
|
||||
// be set BEFORE bind. Without this, the connected-UDP
|
||||
// activation handler fails with EADDRINUSE on Linux and
|
||||
// every outbound packet falls back to the wildcard listen
|
||||
// socket — losing the kernel 5-tuple cache benefit and
|
||||
// most of the multihop forwarding throughput gain.
|
||||
let _ = sock.set_reuse_port(true);
|
||||
let _ = sock.set_reuse_address(true);
|
||||
|
||||
sock.bind(&bind_addr.into())
|
||||
.map_err(|e| TransportError::StartFailed(format!("bind failed: {}", e)))?;
|
||||
|
||||
@@ -494,6 +504,12 @@ mod platform {
|
||||
inner: Arc<AsyncFd<UdpRawSocket>>,
|
||||
}
|
||||
|
||||
impl AsRawFd for AsyncUdpSocket {
|
||||
fn as_raw_fd(&self) -> RawFd {
|
||||
self.inner.get_ref().as_raw_fd()
|
||||
}
|
||||
}
|
||||
|
||||
impl AsyncUdpSocket {
|
||||
/// Send a payload to a destination address.
|
||||
pub async fn send_to(
|
||||
|
||||
@@ -25,6 +25,12 @@ x-fips-common: &fips-common
|
||||
- ./generated-configs/npubs.env
|
||||
environment:
|
||||
- RUST_LOG=info
|
||||
# Passthrough for A/B benchmarking — set on the host shell before
|
||||
# `docker compose up` to override the worker-pool defaults.
|
||||
- FIPS_ENCRYPT_WORKERS=${FIPS_ENCRYPT_WORKERS:-}
|
||||
- FIPS_DECRYPT_WORKERS=${FIPS_DECRYPT_WORKERS:-}
|
||||
- FIPS_PERF=${FIPS_PERF:-0}
|
||||
- FIPS_PERF_INTERVAL_SECS=${FIPS_PERF_INTERVAL_SECS:-5}
|
||||
|
||||
services:
|
||||
# ── Mesh topology ──────────────────────────────────────────────
|
||||
|
||||
427
testing/static/scripts/bench-multirun.sh
Executable file
427
testing/static/scripts/bench-multirun.sh
Executable file
@@ -0,0 +1,427 @@
|
||||
#!/bin/bash
|
||||
# Multi-run iperf3 + ping bench for FIPS mesh. Repeats each test N times
|
||||
# (default 3), reports median + min/max + flags outliers >20% from
|
||||
# median. Adds round-trip latency (ping) and a TCP-retransmit count
|
||||
# (proxy for packet loss across the FIPS overlay) per path.
|
||||
#
|
||||
# Usage:
|
||||
# ./bench-multirun.sh [mesh|chain]
|
||||
#
|
||||
# Environment:
|
||||
# FIPS_BENCH_RUNS=3 Number of iperf3 repetitions per path
|
||||
# FIPS_BENCH_DURATION=15 Seconds per iperf3 run
|
||||
# FIPS_BENCH_PARALLEL=1 Parallel TCP streams (1 = single-stream)
|
||||
# FIPS_BENCH_PING_COUNT=30 ICMP probes per path
|
||||
# FIPS_BENCH_OUTLIER_PCT=20 Flag run as outlier if Δ from median > N%
|
||||
# FIPS_BENCH_OUTPUT=json|text (default text; json emits NDJSON)
|
||||
|
||||
set -e
|
||||
trap 'echo ""; echo "Bench interrupted"; exit 130' INT
|
||||
|
||||
PROFILE="${1:-mesh}"
|
||||
RUNS="${FIPS_BENCH_RUNS:-5}"
|
||||
DURATION="${FIPS_BENCH_DURATION:-15}"
|
||||
PARALLEL="${FIPS_BENCH_PARALLEL:-1}"
|
||||
PING_COUNT="${FIPS_BENCH_PING_COUNT:-30}"
|
||||
OUTLIER_PCT="${FIPS_BENCH_OUTLIER_PCT:-20}"
|
||||
OUTPUT="${FIPS_BENCH_OUTPUT:-text}"
|
||||
|
||||
if ! [[ "$RUNS" =~ ^[1-9][0-9]*$ ]] || [ "$RUNS" -lt 1 ]; then
|
||||
echo "FIPS_BENCH_RUNS must be ≥ 1" >&2; exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="$SCRIPT_DIR/../generated-configs/npubs.env"
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "Error: $ENV_FILE not found. Run generate-configs.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
# shellcheck source=../generated-configs/npubs.env
|
||||
source "$ENV_FILE"
|
||||
|
||||
# ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
# Run iperf3 once, print "<mbps> <retransmits>" on success or "FAIL" on
|
||||
# failure. Uses JSON output so we don't have to scrape human-readable
|
||||
# SI prefixes.
|
||||
iperf_once() {
|
||||
local client="$1" dest_npub="$2"
|
||||
local out
|
||||
if ! out=$(docker exec "fips-$client" iperf3 -c "${dest_npub}.fips" \
|
||||
-t "$DURATION" -P "$PARALLEL" -J 2>&1); then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
python3 - "$out" <<'PYEOF'
|
||||
import json, sys
|
||||
try:
|
||||
d = json.loads(sys.argv[1])
|
||||
bps = d['end']['sum_received']['bits_per_second']
|
||||
# iperf3 reports TCP retransmits in 'sum_sent.retransmits' on
|
||||
# the client side; missing on UDP or for the server-summary view.
|
||||
retr = d.get('end', {}).get('sum_sent', {}).get('retransmits', 0)
|
||||
print(f"{bps/1e6:.2f} {retr}")
|
||||
except Exception as e:
|
||||
print("FAIL")
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# Median + min + max + CoV% + outlier indices over a whitespace-
|
||||
# separated list of floats. CoV% = (stddev / mean) * 100, i.e. the
|
||||
# coefficient of variation in percent — directly comparable across
|
||||
# paths regardless of absolute throughput. Single-value runs print
|
||||
# CoV=0.
|
||||
# Prints: "<median> <min> <max> <cov_pct> <outliers_csv>"
|
||||
stats() {
|
||||
python3 - "$OUTLIER_PCT" "$@" <<'PYEOF'
|
||||
import math
|
||||
import sys
|
||||
pct = float(sys.argv[1])
|
||||
vals = [float(x) for x in sys.argv[2:]]
|
||||
n = len(vals)
|
||||
s = sorted(vals)
|
||||
median = s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2
|
||||
lo, hi = s[0], s[-1]
|
||||
mean = sum(vals) / n if n else 0.0
|
||||
if n > 1 and mean > 0:
|
||||
variance = sum((v - mean) ** 2 for v in vals) / (n - 1) # sample stddev
|
||||
cov_pct = math.sqrt(variance) / mean * 100
|
||||
else:
|
||||
cov_pct = 0.0
|
||||
outliers = []
|
||||
if median > 0:
|
||||
for i, v in enumerate(vals):
|
||||
if abs(v - median) / median * 100 > pct:
|
||||
outliers.append(str(i + 1))
|
||||
print(
|
||||
f"{median:.2f} {lo:.2f} {hi:.2f} {cov_pct:.1f}% "
|
||||
f"{','.join(outliers) if outliers else '-'}"
|
||||
)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# Ping once, print "<min_ms> <avg_ms> <max_ms> <mdev_ms> <loss_pct>"
|
||||
# or "FAIL".
|
||||
ping_path() {
|
||||
local client="$1" dest_npub="$2"
|
||||
local out
|
||||
if ! out=$(docker exec "fips-$client" \
|
||||
ping -c "$PING_COUNT" -i 0.2 -w "$((PING_COUNT * 2))" \
|
||||
-q "${dest_npub}.fips" 2>&1); then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
# "min/avg/max/mdev = 0.123/0.456/0.789/0.012 ms"
|
||||
local rtt loss
|
||||
rtt=$(echo "$out" | awk -F' = ' '/min\/avg\/max\/mdev/ {print $2}' | awk '{print $1}')
|
||||
loss=$(echo "$out" | awk -F', ' '/packet loss/ {for (i=1;i<=NF;i++) if ($i ~ /packet loss/) print $i}' | awk '{print $1}')
|
||||
if [ -z "$rtt" ]; then
|
||||
echo FAIL
|
||||
return
|
||||
fi
|
||||
echo "${rtt//\// } ${loss:-N/A}"
|
||||
}
|
||||
|
||||
# ── Path definitions ───────────────────────────────────────────────────────
|
||||
#
|
||||
# Each entry is `<client-container> <dest-npub> <label>`. Labels are
|
||||
# plain `client→dest` — the bench measures iperf3 throughput between
|
||||
# the two named nodes. The static topology is printed at start so the
|
||||
# operator can see the configured routing context.
|
||||
#
|
||||
# IMPORTANT — labels DO NOT encode hop count. With peer discovery
|
||||
# enabled and all nodes on the same docker-bridge subnet, every node
|
||||
# pair establishes a direct UDP path within a few ticks regardless of
|
||||
# the static `peers:` list. The bench measures the post-convergence
|
||||
# steady state, which is what real-world FIPS deployments see. To
|
||||
# force on-wire multihop, isolate nodes on distinct docker networks.
|
||||
case "$PROFILE" in
|
||||
mesh|mesh-public)
|
||||
# Static-peer paths only — without mDNS / a Nostr relay in the
|
||||
# test container set, non-adjacent pairs (A↔B, A↔C) can't
|
||||
# establish direct UDP and the bench would either fail
|
||||
# convergence or measure multihop forwarding (apples-to-oranges
|
||||
# vs builds that do have mDNS). Stick to pairs that are in
|
||||
# each other's static `peers:` list so the measurement is
|
||||
# deterministic and the same on every build.
|
||||
PATHS=(
|
||||
"node-a $NPUB_D A→D"
|
||||
"node-a $NPUB_E A→E"
|
||||
"node-e $NPUB_A E→A"
|
||||
)
|
||||
TOPOLOGY_LINES=(
|
||||
"A peers with: D, E"
|
||||
"D peers with: A, C, E"
|
||||
"E peers with: A, C, D"
|
||||
"(Static-peer pairs A↔D, A↔E only — non-adjacent pairs"
|
||||
" are skipped because plain mesh.yaml has no discovery"
|
||||
" transport that converges them onto direct UDP.)"
|
||||
)
|
||||
;;
|
||||
chain)
|
||||
# Chain topology forces intentional multihop forwarding for
|
||||
# non-adjacent pairs: A peers only with B, B peers with A+C,
|
||||
# etc. We bench just A→B (1 hop, direct static peer) so the
|
||||
# comparison is again apples-to-apples without discovery.
|
||||
PATHS=(
|
||||
"node-a $NPUB_B A→B"
|
||||
)
|
||||
TOPOLOGY_LINES=(
|
||||
"Static chain: A — B — C — D — E"
|
||||
"(A↔B is the only static-peer pair we benchmark; multi-hop"
|
||||
" forwarding to C/D/E needs reply-learned routing or mDNS"
|
||||
" to settle, which is not deterministic on a fresh mesh.)"
|
||||
)
|
||||
;;
|
||||
*)
|
||||
echo "Unknown profile: $PROFILE" >&2; exit 1 ;;
|
||||
esac
|
||||
|
||||
# ── Run ────────────────────────────────────────────────────────────────────
|
||||
|
||||
echo "=== FIPS multi-run bench ($PROFILE, ${RUNS}×${DURATION}s, P=${PARALLEL}) ==="
|
||||
echo "Outlier flag: Δ from median > ${OUTLIER_PCT}%"
|
||||
echo ""
|
||||
echo "Topology (static \`peers:\` from $PROFILE.yaml):"
|
||||
for line in "${TOPOLOGY_LINES[@]}"; do
|
||||
echo " $line"
|
||||
done
|
||||
echo ""
|
||||
|
||||
# Wait for peer discovery to converge before measuring — otherwise
|
||||
# the bench randomly measures either direct UDP (post-convergence
|
||||
# steady state) or multihop forwarding through static peers (only
|
||||
# until discovery converges). On the same docker-bridge subnet
|
||||
# convergence should take seconds, not tens of seconds; this is the
|
||||
# tight default. Bump FIPS_BENCH_CONVERGE_SECS for slower test setups
|
||||
# (no Nostr relay, no mDNS, etc.).
|
||||
#
|
||||
# If any path hasn't converged by the deadline, the bench FAILS with
|
||||
# a clear list of unconverged pairs — that's almost always a topology
|
||||
# / discovery misconfiguration that would make the numbers noise.
|
||||
CONVERGE_SECS="${FIPS_BENCH_CONVERGE_SECS:-15}"
|
||||
|
||||
peer_is_direct() {
|
||||
local client="$1" dest_npub="$2"
|
||||
docker exec "fips-$client" fipsctl show peers 2>/dev/null \
|
||||
| python3 -c '
|
||||
import json, sys
|
||||
target = sys.argv[1]
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
for p in d.get("peers", []):
|
||||
if p.get("npub") == target:
|
||||
print("yes" if p.get("connectivity") == "connected" else "no")
|
||||
sys.exit(0)
|
||||
print("no")
|
||||
except Exception:
|
||||
print("no")
|
||||
' "$dest_npub" 2>/dev/null
|
||||
}
|
||||
|
||||
# Per-peer `stats.bytes_sent` snapshot for the given client. Used to
|
||||
# verify that the iperf3 traffic actually went out the intended next-
|
||||
# hop (i.e. the routing protocol picked the static-peer link, not a
|
||||
# multihop alternative).
|
||||
peer_bytes_sent_snapshot() {
|
||||
local client="$1"
|
||||
docker exec "fips-$client" fipsctl show peers 2>/dev/null \
|
||||
| python3 -c '
|
||||
import json, sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
for p in d.get("peers", []):
|
||||
npub = p.get("npub") or "?"
|
||||
sent = (p.get("stats") or {}).get("bytes_sent", 0)
|
||||
print(f"{npub} {sent}")
|
||||
except Exception:
|
||||
pass
|
||||
'
|
||||
}
|
||||
|
||||
# Compute deltas between two snapshots and identify which peer
|
||||
# received the most bytes_sent growth, plus the absolute bytes for
|
||||
# the requested target. Prints "<top_npub> <top_delta> <target_delta>".
|
||||
peer_bytes_delta_winner() {
|
||||
local target="$1"; shift
|
||||
local before="$1"; shift
|
||||
local after="$1"; shift
|
||||
python3 - "$target" "$before" "$after" <<'PYEOF'
|
||||
import sys
|
||||
target, before, after = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
b = {}
|
||||
for line in before.splitlines():
|
||||
if not line.strip(): continue
|
||||
k, v = line.split()
|
||||
b[k] = int(v)
|
||||
a = {}
|
||||
for line in after.splitlines():
|
||||
if not line.strip(): continue
|
||||
k, v = line.split()
|
||||
a[k] = int(v)
|
||||
deltas = {k: a.get(k, 0) - b.get(k, 0) for k in a}
|
||||
target_delta = deltas.get(target, 0)
|
||||
if deltas:
|
||||
top_npub = max(deltas, key=deltas.get)
|
||||
top_delta = deltas[top_npub]
|
||||
else:
|
||||
top_npub, top_delta = "-", 0
|
||||
print(f"{top_npub} {top_delta} {target_delta}")
|
||||
PYEOF
|
||||
}
|
||||
|
||||
declare -A PEER_STATE
|
||||
echo "Waiting for peer convergence (up to ${CONVERGE_SECS}s)…"
|
||||
WAIT_START=$(date +%s)
|
||||
while :; do
|
||||
all_done=1
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
if [ "${PEER_STATE[$label]:-}" = "direct" ]; then
|
||||
continue
|
||||
fi
|
||||
if [ "$(peer_is_direct "$client" "$npub")" = "yes" ]; then
|
||||
PEER_STATE[$label]=direct
|
||||
else
|
||||
all_done=0
|
||||
fi
|
||||
done
|
||||
if [ "$all_done" = 1 ]; then
|
||||
break
|
||||
fi
|
||||
elapsed=$(( $(date +%s) - WAIT_START ))
|
||||
if [ "$elapsed" -ge "$CONVERGE_SECS" ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
unconverged=()
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
state="${PEER_STATE[$label]:-via-forward}"
|
||||
PEER_STATE[$label]="$state"
|
||||
printf ' %-10s %s\n' "$label" "$state"
|
||||
if [ "$state" != "direct" ]; then
|
||||
unconverged+=("$label")
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
if [ "${#unconverged[@]}" -gt 0 ]; then
|
||||
echo "ERROR: peer discovery did not converge for ${#unconverged[@]} of ${#PATHS[@]} paths within ${CONVERGE_SECS}s:" >&2
|
||||
for p in "${unconverged[@]}"; do
|
||||
echo " - $p" >&2
|
||||
done
|
||||
echo "" >&2
|
||||
echo "On a single docker-bridge subnet convergence should take seconds." >&2
|
||||
echo "Check that: (a) all node IPs are reachable peer-to-peer, (b) the" >&2
|
||||
echo "test build includes a working discovery transport (Nostr relay /" >&2
|
||||
echo "mDNS / etc.), (c) FIPS_BENCH_CONVERGE_SECS is high enough for" >&2
|
||||
echo "this setup. Bench aborted to avoid reporting noisy mixed-state" >&2
|
||||
echo "measurements." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Header for text output. All bandwidth columns are Mbits/sec
|
||||
# (Mbps). CoV% = sample coefficient of variation (stddev/mean) over
|
||||
# the N runs, in percent — directly comparable across paths.
|
||||
if [ "$OUTPUT" = "text" ]; then
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
path session 'median Mbps' 'min Mbps' 'max Mbps' 'CoV %' outliers \
|
||||
'avg RTT ms' 'loss %' 'TCP retr'
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
'------' '------' '------' '------' '------' '------' '------' \
|
||||
'------' '------' '------'
|
||||
fi
|
||||
|
||||
FAIL_COUNT=0
|
||||
|
||||
ROUTE_ERRORS=()
|
||||
|
||||
for path in "${PATHS[@]}"; do
|
||||
read -r client npub label <<<"$path"
|
||||
|
||||
# Snapshot per-peer bytes_sent before iperf3 — used after the
|
||||
# runs to verify the routing protocol actually pushed the test
|
||||
# traffic out the intended static-peer link (and not, say,
|
||||
# via an alternative multihop route the cost-based router could
|
||||
# in principle pick). Same defensive check the iperf-test.sh
|
||||
# path measurement assumes implicitly.
|
||||
bytes_before=$(peer_bytes_sent_snapshot "$client")
|
||||
|
||||
# iperf3 runs
|
||||
bw_runs=()
|
||||
retr_runs=()
|
||||
for i in $(seq 1 "$RUNS"); do
|
||||
result=$(iperf_once "$client" "$npub")
|
||||
if [ "$result" = "FAIL" ]; then
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
bw_runs+=("0")
|
||||
retr_runs+=("0")
|
||||
else
|
||||
bw=$(echo "$result" | awk '{print $1}')
|
||||
retr=$(echo "$result" | awk '{print $2}')
|
||||
bw_runs+=("$bw")
|
||||
retr_runs+=("$retr")
|
||||
fi
|
||||
done
|
||||
|
||||
bytes_after=$(peer_bytes_sent_snapshot "$client")
|
||||
# `delta_winner` is "<npub_with_largest_bytes_sent_delta> <its_bytes> <target_dest_bytes>"
|
||||
delta_winner=$(peer_bytes_delta_winner "$npub" "$bytes_before" "$bytes_after")
|
||||
read -r winning_npub winning_delta target_delta <<<"$delta_winner"
|
||||
if [ "$winning_npub" != "$npub" ]; then
|
||||
ROUTE_ERRORS+=("$label: traffic exited via $winning_npub ($winning_delta B) instead of $npub ($target_delta B)")
|
||||
fi
|
||||
|
||||
# Stats on bandwidth (median / min / max / coefficient-of-variation / outlier indices)
|
||||
bw_stats=$(stats "${bw_runs[@]}")
|
||||
read -r bw_median bw_min bw_max bw_cov bw_outliers <<<"$bw_stats"
|
||||
# Sum retransmits across runs (a proxy for packet loss volume).
|
||||
retr_sum=$(python3 -c "print(sum(int(x) for x in '${retr_runs[*]}'.split()))")
|
||||
|
||||
# Ping (single run; ping gives its own internal stats already).
|
||||
ping_result=$(ping_path "$client" "$npub")
|
||||
if [ "$ping_result" = "FAIL" ]; then
|
||||
rtt_avg="N/A"
|
||||
loss="N/A"
|
||||
else
|
||||
# min avg max mdev loss
|
||||
read -r _rtt_min rtt_avg _rtt_max _rtt_mdev loss <<<"$ping_result"
|
||||
fi
|
||||
|
||||
session_state="${PEER_STATE[$label]:-via-forward}"
|
||||
if [ "$OUTPUT" = "json" ]; then
|
||||
printf '{"path":"%s","client":"%s","dest_npub":"%s","session":"%s","mbps_runs":[%s],"mbps_median":%s,"mbps_min":%s,"mbps_max":%s,"mbps_cov_pct":"%s","outlier_runs":"%s","rtt_avg_ms":"%s","loss":"%s","tcp_retr_total":%s}\n' \
|
||||
"$label" "$client" "$npub" "$session_state" \
|
||||
"$(IFS=, ; echo "${bw_runs[*]}")" \
|
||||
"$bw_median" "$bw_min" "$bw_max" "$bw_cov" "$bw_outliers" \
|
||||
"$rtt_avg" "$loss" "$retr_sum"
|
||||
else
|
||||
printf '%-10s %-12s %12s %12s %12s %8s %10s | %10s %8s %10s\n' \
|
||||
"$label" "$session_state" "$bw_median" "$bw_min" "$bw_max" \
|
||||
"$bw_cov" "$bw_outliers" \
|
||||
"$rtt_avg" "$loss" "$retr_sum"
|
||||
if [ "$bw_outliers" != "-" ]; then
|
||||
printf '%-10s runs (Mbps): %s\n' " ↑outliers" "${bw_runs[*]}"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
if [ "${#ROUTE_ERRORS[@]}" -gt 0 ]; then
|
||||
echo "ERROR: traffic for ${#ROUTE_ERRORS[@]} of ${#PATHS[@]} paths did not exit via the intended static-peer link:" >&2
|
||||
for e in "${ROUTE_ERRORS[@]}"; do
|
||||
echo " - $e" >&2
|
||||
done
|
||||
echo "" >&2
|
||||
echo "The numbers above measured a different route than the path label" >&2
|
||||
echo "claims. Inspect peer cost / tree topology before trusting them." >&2
|
||||
exit 3
|
||||
fi
|
||||
if [ "$FAIL_COUNT" -gt 0 ]; then
|
||||
echo "WARN: $FAIL_COUNT iperf3 runs failed"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK"
|
||||
@@ -84,40 +84,48 @@ iperf_test() {
|
||||
echo "=== FIPS iperf3 Bandwidth Test ($PROFILE topology) ==="
|
||||
echo ""
|
||||
|
||||
# Print topology so the operator can see the configured routing
|
||||
# context. The bench measures iperf3 throughput between the named
|
||||
# nodes — labels DO NOT encode hop count, since peer discovery
|
||||
# converges every same-subnet pair onto a direct UDP path within a
|
||||
# few ticks regardless of the static `peers:` list.
|
||||
if [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then
|
||||
echo "Topology (static \`peers:\` from mesh.yaml):"
|
||||
echo " A peers with: D, E"
|
||||
echo " B peers with: C"
|
||||
echo " C peers with: B, D, E"
|
||||
echo " D peers with: A, C, E"
|
||||
echo " E peers with: A, C, D"
|
||||
echo " (Same docker-bridge subnet — discovery converges every"
|
||||
echo " pair onto a direct UDP path; static \`peers:\` only seeds"
|
||||
echo " the initial mesh, not the steady-state routing.)"
|
||||
elif [ "$PROFILE" = "chain" ]; then
|
||||
echo "Topology (static chain): A — B — C — D — E"
|
||||
echo " (Same docker-bridge subnet — see mesh note above.)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Wait for nodes to converge
|
||||
echo "Waiting ${SETTLE_SECONDS}s for mesh convergence..."
|
||||
sleep "$SETTLE_SECONDS"
|
||||
|
||||
if [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then
|
||||
# Test key paths in mesh topology
|
||||
echo ""
|
||||
echo "Testing mesh topology paths:"
|
||||
|
||||
# Direct peer links (client on A, server on D/E)
|
||||
iperf_test node-d node-a "$NPUB_D" "A → D (direct peer)"
|
||||
iperf_test node-e node-a "$NPUB_E" "A → E (direct peer)"
|
||||
|
||||
# Multi-hop paths (client on A, server on B/C)
|
||||
iperf_test node-b node-a "$NPUB_B" "A → B (multi-hop)"
|
||||
iperf_test node-c node-a "$NPUB_C" "A → C (multi-hop)"
|
||||
|
||||
# Reverse test (client on E, server on A)
|
||||
iperf_test node-a node-e "$NPUB_A" "E → A (direct peer)"
|
||||
iperf_test node-d node-a "$NPUB_D" "A→D"
|
||||
iperf_test node-e node-a "$NPUB_E" "A→E"
|
||||
iperf_test node-b node-a "$NPUB_B" "A→B"
|
||||
iperf_test node-c node-a "$NPUB_C" "A→C"
|
||||
iperf_test node-a node-e "$NPUB_A" "E→A"
|
||||
|
||||
elif [ "$PROFILE" = "chain" ]; then
|
||||
echo ""
|
||||
echo "Testing chain topology paths:"
|
||||
|
||||
# Adjacent hop (client on A, server on B)
|
||||
iperf_test node-b node-a "$NPUB_B" "A → B (1 hop)"
|
||||
|
||||
# Multi-hop tests (client on A, server on C/D/E)
|
||||
iperf_test node-c node-a "$NPUB_C" "A → C (2 hops)"
|
||||
iperf_test node-d node-a "$NPUB_D" "A → D (3 hops)"
|
||||
iperf_test node-e node-a "$NPUB_E" "A → E (4 hops)"
|
||||
|
||||
# Reverse multi-hop (client on E, server on A)
|
||||
iperf_test node-a node-e "$NPUB_A" "E → A (4 hops)"
|
||||
iperf_test node-b node-a "$NPUB_B" "A→B"
|
||||
iperf_test node-c node-a "$NPUB_C" "A→C"
|
||||
iperf_test node-d node-a "$NPUB_D" "A→D"
|
||||
iperf_test node-e node-a "$NPUB_E" "A→E"
|
||||
iperf_test node-a node-e "$NPUB_A" "E→A"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
Reference in New Issue
Block a user