mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-30 15:48:34 +00:00
Johnathan Corgan
9f8b266e5d
The decrypt worker's module doc and its fallback-path comment framed the code as a step in an architectural plan this repository does not carry, and named a "DataShard end-state" that exists nowhere in the tree. State the property directly instead: each worker owns its session state, and a worker that also owned the rx_loop side could restore the EndpointData fast path. The reworded comments are rewrapped to the surrounding width. The metrics registry docs described off-thread reads as a later step, but the control read handle already serves show_metrics from the registry off the rx_loop task. Put both the module doc and the Padded doc in the present tense, with the single writer the code has. Three source comments cited commits by hash (an abandoned republish design, the change that made sessions shard-owned, and the fix that moved encrypt dispatch off round-robin). None of those hashes resolves in this repository, so a reader cannot follow them. Say what the commit did instead. The one hash cited in the routing tests is on every branch and stays. The bloom-storm scenario README pointed readers at a reproduction harness and results file outside this repository, told them to check out a commit and a branch that are not here, and named the regressing commit, which is not here either. Describe the harness and the regressed build without those references and anchor the confirmation procedure on the fix commit, which every branch contains. The off-loop snapshot dispatch test's name carried a stage label from an outside plan. Rename it to say what it checks: show_acl, show_stats_peers and show_stats_history_all_peers are served off-loop through snapshot_dispatch and render byte-identically to their on-loop forms. Nothing refers to the old name.
176 lines
6.7 KiB
Markdown
176 lines
6.7 KiB
Markdown
# bloom-storm — bloom announce storm regression scenario
|
||
|
||
Six-node depth-4 mesh with an induced upstream parent flap. Asserts a
|
||
trailing-window ceiling on per-node `stats.bloom.sent` to catch a
|
||
regression class where a localized spanning-tree update at an
|
||
internal mid-chain edge fails to be properly contained and instead
|
||
propagates as a bloom announce storm down to the leaves.
|
||
|
||
## Topology
|
||
|
||
```text
|
||
n01 (root, depth 0)
|
||
/ \
|
||
n02 n03 (parent candidates pa/pb, depth 1)
|
||
\ /
|
||
n04 (flap, depth 2 — induced parent flap)
|
||
|
|
||
n05 (leaf, depth 3)
|
||
|
|
||
n06 (tail, depth 4)
|
||
```
|
||
|
||
n01 is expected to win the root election under the deterministic key
|
||
derivation used by the chaos runner; the run's tree snapshots
|
||
(`tree-snapshot-warmup.json`, `tree-snapshot-final.json`) record the
|
||
actual assignment.
|
||
|
||
## Bug class guarded against
|
||
|
||
A spanning-tree update that changes only an internal path edge — no
|
||
root change, no depth change — must not produce a sustained bloom
|
||
announce storm at downstream nodes. The original regression
|
||
(since rolled back, and fixed in master `4cdf382`) had this property:
|
||
in the field, a single mid-chain ancestor swap on an upstream node
|
||
caused every downstream node in its subtree to issue a bloom
|
||
announce on every parent re-evaluation tick of the upstream node,
|
||
resulting in a ~480x mesh-wide elevation in `FilterAnnounce` traffic
|
||
and a perfect bimodal flip on `est_entries`.
|
||
|
||
## Mechanism
|
||
|
||
The new `link_swap` chaos primitive deterministically alternates the
|
||
netem delay on `n02-n04` (5ms vs 100ms) and `n03-n04` (100ms vs 5ms)
|
||
every 4 seconds. Combined with the FIPS overrides
|
||
(`parent_hysteresis: 0.0`, `reeval_interval_secs: 1`,
|
||
`flap_threshold: 9999`, `hold_down_secs: 0`), this forces n04 to
|
||
switch parents on every swap. The whole point of the scenario is to
|
||
*force* sustained parent flapping at n04 and assert that the bloom
|
||
layer doesn't amplify it.
|
||
|
||
## Assertions
|
||
|
||
```yaml
|
||
assertions:
|
||
bloom_send_rate:
|
||
window_secs: 30
|
||
max_per_node: 40
|
||
min_parent_switches:
|
||
min_total: 10
|
||
```
|
||
|
||
`bloom_send_rate` is the load-bearing assertion: per-node delta of
|
||
`stats.bloom.sent` over the trailing 30s of the run must be at most
|
||
40. Per-node deltas and the offending node IDs are written to
|
||
`assertions.txt` and the runner exits 3 on failure.
|
||
|
||
`min_parent_switches` is a sanity guard. It fails if the run did not
|
||
record at least 10 parent switches across all nodes, which would
|
||
mean the harness fired its link swaps but the topology never produced
|
||
a real parent-switch event (e.g., wrong root election made the flap
|
||
target's parent candidates structurally non-equivalent). Without this
|
||
guard, the bloom-rate assertion would trivially pass on any binary,
|
||
including a regressed one.
|
||
|
||
## Threshold derivation
|
||
|
||
The original reproduction harness, kept outside this repository, measured
|
||
(90s flap window, ~21 induced parent switches at the mid-chain
|
||
node):
|
||
|
||
| binary | tail bloom_sent / 90s | rate scaled to 30s |
|
||
| ----------- | --------------------: | -----------------: |
|
||
| pre-fix | 21 | ~7 |
|
||
| current fix | 0 | 0 |
|
||
|
||
Observed on this scenario at master `db5b6b1` (180s run, 35 parent
|
||
switches, 41 link swaps), per-node `bloom_sent` deltas over the
|
||
trailing 30s:
|
||
|
||
```text
|
||
n01=5 n02=5 n03=4 n04=12 n05=6 n06=0
|
||
```
|
||
|
||
n04 (the flapping node) is the highest because it is legitimately
|
||
re-sending its filter on its own parent changes. n06 (the depth-4
|
||
"tail") sees 0, matching the calm post-fix behavior that harness recorded
|
||
for its `fix2` variant.
|
||
|
||
In the field, the regression's mesh-wide rate scaled ~480x above
|
||
steady state. A `30 / 30s / node` ceiling sits ~2.5x above the
|
||
observed maximum on fixed master and well below the
|
||
deployment-scale storm rate, giving headroom for harness jitter
|
||
without losing the ability to fail loud on the regression class.
|
||
|
||
If `link_swap.interval_secs` or the netem delta is changed,
|
||
recalibrate. The threshold is calibrated against the values in
|
||
`scenarios/bloom-storm.yaml` as committed and the `seed: 31` pin.
|
||
|
||
### Lab-data ceiling re-tune (2026-05-24)
|
||
|
||
The ceiling was bumped from 30 to 40 after a 59-rep characterization run
|
||
under the `github-runner-equivalent` pressure profile with per-container
|
||
CPU pinning to `cpuset=0,1` (mimicking a 2-core `ubuntu-latest` GitHub
|
||
runner). Combined pinned distribution on n04 (the structural max-spike
|
||
node — flap target at depth 2):
|
||
|
||
| metric | value |
|
||
| ------ | ----: |
|
||
| mean | 24.4 |
|
||
| sd | 4.7 |
|
||
| P90 | 28 |
|
||
| P95 | 29 |
|
||
| P99 | 29 |
|
||
| max | 30 |
|
||
|
||
The lab's structural ceiling at 30 corresponds to the bloom-advertise
|
||
rate-limit token bucket's steady-state cap of ~1 send per second over the
|
||
trailing 30 s assertion window. GHA fires at n04=34 represent transient
|
||
release of queued bloom-sends during flap-recovery windows and do not
|
||
reproduce on this lab host even with CPU-pinning sidecar (`cpuset=0,1`)
|
||
applied to every chaos-spawned container.
|
||
|
||
Rationale for ceiling = 40: lab max 30 + ~2σ headroom (≈ 39.4) rounds to
|
||
40, giving 33 % margin over the observed lab maximum while still firing
|
||
loud on a regression-class storm (the original regression
|
||
scaled mesh-wide bloom traffic ~480× above steady state, far above any
|
||
plausible jitter band).
|
||
|
||
## Limitations
|
||
|
||
- The bloom-storm regression has not been confirmed-failing here
|
||
on a regressed binary in this harness directly; the threshold is
|
||
inferred from the values measured in the dedicated post-mortem
|
||
harness against a regressed build whose commit is not in this
|
||
repository. To gain that confirmation, build a binary that
|
||
carries the regression and predates the `4cdf382` fix, copy
|
||
binaries into `testing/docker/`, and rerun
|
||
this scenario; the bloom-rate assertion is expected to fail loud
|
||
with n05/n06 deltas well above 40.
|
||
|
||
- Root-election outcome is sensitive to the seed (smallest
|
||
`NodeAddr` wins, where `NodeAddr = SHA-256(pubkey)[..16]`). The
|
||
seed value `31` is pinned for this reason. The
|
||
`min_parent_switches` assertion catches drift if the seed is
|
||
changed without re-validating the topology.
|
||
|
||
## Running locally
|
||
|
||
```bash
|
||
# From the source repo root, with binaries already built and copied
|
||
# into testing/docker/ (see testing/scripts/build.sh).
|
||
./testing/chaos/scripts/chaos.sh bloom-storm
|
||
```
|
||
|
||
Run output is in `testing/chaos/sim-results/<timestamp>-bloom-storm/`.
|
||
Key artifacts:
|
||
|
||
- `analysis.txt` — log analysis (panics, errors, parent switches).
|
||
- `assertions.txt` — per-assertion pass/fail with per-node deltas.
|
||
- `tree-snapshot-warmup.json`, `tree-snapshot-final.json` — control
|
||
socket tree state at warmup end and at run end.
|
||
- `runner.log` — full orchestration log.
|
||
|
||
Total runtime: ~3.5 minutes (25s warmup + 180s scenario + ~30s
|
||
teardown).
|