amethyst/quartz/plans/2026-07-02-nostrclient-receiver-perf.md

# NostrClient receiver performance — review + production measurements

**Date:** 2026-07-02
**Status:** findings + measurement harness; no production code changed yet
**Harness:** `quartz/src/jvmTest/.../relay/prodbench/ProductionReceiverBenchmark.kt`
(run with `./gradlew :quartz:jvmTest --tests "*.ProductionReceiverBenchmark" -PprodRelayBench=1`)

## Question

Is the NostrClient's single-threaded receiver making us significantly slower?
And how do our filters actually behave against production relays?

## What the receiver actually is (review)

The receive path is single-threaded **per relay connection**, not globally:

```
OkHttp reader thread (per socket)
  └─ trySendBlocking → Channel(UNLIMITED)            [BasicOkHttpWebSocket / app OkHttpWebSocket]
       └─ ONE consumer coroutine per connection (Dispatchers.IO)
            ├─ OptimizedJsonMapper.fromJsonToMessage  [JSON parse]        ~32µs/msg (JVM)
            ├─ PoolRequests.onIncomingMessage         [spin-lock + state]
            │    └─ SubscriptionListener.onEvent      [assemblers, inline]
            └─ NostrClient.listeners.forEach          [EventCollector, loggers…]
                 └─ LocalCache.justConsume(…, wasVerified = false)
                      └─ dedup by id → justVerify()   [Schnorr verify]    ~75µs/event (JVM)
                      └─ note creation, index updates, flow emissions
```

Everything downstream of the channel — parse, subscription state machine, every
listener, signature verification, and LocalCache consumption — runs serially on
that one consumer coroutine. Different relays run on different coroutines
(cross-relay parallelism exists), but they contend on shared state: the
`PoolRequests` spin lock, `LocalCache`'s maps, and (on Android) the same
`Dispatchers.IO` pool as everything else.

Notably, the **relay-server side already solved this** on the write path:
`IngestQueue` batches incoming EVENTs and runs Schnorr verification for a batch
in parallel on `Dispatchers.Default` (`parallelVerify`), precisely because "an
event's `verify()` is CPU-bound and parallelisable". The client receiver has no
equivalent.

## Production measurements (2026-07-02)

Environment: JVM 21, 4-core x86 cloud container (server CPU — phones are
5–10× slower per core), relays `relay.damus.io`, `nos.lol`, `relay.primal.net`,
`nostr.wine`. All scenarios ran after a discarded warmup pass (JIT warm).
`INLINE` = verify on the receiver coroutine (what the app does today via
`CacheClientConnector`); `PARALLEL` = verify offloaded to a
`Dispatchers.Default` pool.

### Filter behavior

| Scenario | received | unique | duplicates | notes |
|---|---|---|---|---|
| kind-1 firehose, limit 500 ×4 relays | 2013 | 1738 | 14% | all relays honored limit; EOSE 0.3–1.0s after REQ |
| notifications (p=busy pubkey, kinds 1,6,7,9735) | 1552 | 986 | 36% | primal returned only 52 (thin index for this query) |
| metadata burst (kinds 0,10002, 300 authors) | 1146 | 491 | 57% | each profile fetched ~2.3× across 4 relays |

- Time from `subscribe()` to REQ-on-the-wire is dominated by connect/TLS
  (~0.7–1.5s cold); local dispatch is negligible.
- Event sizes vary wildly by relay: damus kind-1 averaged **8.7 KB**/event
  (4.4 MB for 500 events) vs ~0.8 KB on nos.lol — parse cost tracks size
  (p90 proc 1.5ms on damus vs 0.18ms on nos.lol).
- Duplicate factor grows with fan-out: the metadata pattern wastes >half the
  received bytes on duplicates. Dedup happens *before* verify (LocalCache
  checks the id first), so duplicates cost parse + dedup lookup, not a verify.

### Receiver queue behavior (the actual question)

Queue delay = how long a frame sat in the per-connection channel before the
consumer picked it up. This is the direct symptom of a saturated receiver.

kind-1 firehose (large events), per relay:

| | INLINE p50 / p90 / max | PARALLEL p50 / p90 / max |
|---|---|---|
| relay.damus.io | 6.8ms / 26.6ms / 77.9ms | 5.4ms / 38.1ms / 41.5ms |
| nos.lol | 4.3ms / 11.2ms / 15.9ms | 1.3ms / 3.8ms / 4.5ms |
| relay.primal.net | 5.4ms / 24.5ms / 28.1ms | 1.2ms / 1.8ms / 7.7ms |
| nostr.wine | 9.4ms / 30.3ms / 34.8ms | 2.4ms / 8.4ms / 20.7ms |

metadata burst (small events, fastest arrival — damus delivered at 3300–8400 msg/s):

| | INLINE p50 / max | PARALLEL p50 / max |
|---|---|---|
| relay.damus.io | 3.0ms / 10.7ms | 0.14ms / 0.48ms |

Consumer busy fraction peaked at **53%** (nostr.wine, INLINE, 3000 msg/s) on
this fast CPU — i.e. a single relay bursting at production speed already
consumes half of one core with inline verification, on hardware much faster
than a phone.

### Single-thread ceilings (offline, captured production frames)

- JSON parse: **31,000 msg/s** (32µs/msg, mixed sizes incl. damus 8.7KB events)
- Schnorr verify: **13,200 events/s** sequential (75µs/event);
  **34,400 events/s** across 4 cores (2.6–3.0× speedup)
- Combined parse+verify ceiling for one receiver coroutine: **~9,000 events/s**
  on this CPU. On a phone core (5–10× slower): **~1,000–2,000 events/s**, and
  that is *before* LocalCache's note creation, index updates and flow emissions,
  which the harness does not model and which run on the same coroutine.

## Answer

1. **The receiver is not globally single-threaded** — it is one consumer per
   relay. Cross-relay parallelism already exists. The per-relay serialization
   is what caps throughput.

2. **For steady-state browsing the receiver is not the bottleneck.** Queue
   delays with today's inline design are single-digit-to-tens of ms; connect
   latency and relay EOSE times (300ms–1.5s) dominate what the user perceives.

3. **For burst ingest (app start, feed switch, big profile sync) it is a real
   cost, and it scales linearly with burst size.** A 5,000-event burst from one
   fast relay serializes ~375ms of parse+verify on this benchmark machine —
   plausibly **2–4s per relay on a mid-range phone**, during which that relay's
   EOSE, OKs and live events all sit behind the backlog. Verification is
   60–70% of that inline cost and is embarrassingly parallel (measured 2.6–3×
   on 4 cores; phones have 8).

4. **Offloading verification cuts queue latency 2–5× and raises the per-relay
   ceiling ~3×** with no protocol or API change — the exact pattern
   `IngestQueue.parallelVerify` already uses on the server side.

## Dispatch stage (post-parse, pre-verify) microbenchmark

**Harness:** `quartz/src/jvmTest/.../relay/prodbench/DispatchStageBenchmark.kt`
(offline, ungated: `./gradlew :quartz:jvmTest --tests "*.DispatchStageBenchmark"`)

Measures everything between the JSON parse and where verification would start:
`NostrClient.onIncomingMessage` → `PoolRequests` (spin lock + sub state) →
listeners → id-dedup → handoff to a verify stage. 30k unique synthetic events,
2 overlapping subs per relay, delivered by 1 feeder thread (one busy relay) and
by 4 (four relays bursting into the shared client). JVM 21, 4 cores.

| variant | 1 feeder (deliveries/s) | 4 feeders (aggregate deliveries/s) |
|---|---|---|
| PoolRequests-only | 11.1M | 3.6M |
| full dispatch, no-op sink | 10.2M | 4.3M |
| + dedup (CHM keySet) | 6.8M | 3.5M |
| + dedup + per-event channel handoff | 3.6M | 3.1M |
| + dedup + batched handoff (64) | 6.8M | 2.8M |
| **early dedup + batched handoff** | 7.2M | **11.4M** |

Findings:

1. **Uncontended, the dispatch stage is ~100ns/message** — 300× cheaper than
   parse (32µs) and 750× cheaper than verify (75µs). One relay can never
   saturate it; nothing to fix for the single-relay case.
2. **It scales negatively under concurrency.** Four feeder threads deliver
   *less* aggregate throughput (3.6–4.3M/s) than one thread alone (10–11M/s):
   the `PoolRequests` busy-wait spin lock serializes every EVENT frame from
   every relay and burns the other cores spinning. Isolated `PoolRequests`
   shows the same collapse, so the lock (not listeners or dedup) is the cause.
   At production message rates (~3k msg/s/relay) this is not yet the
   bottleneck, but it wastes cores the verify pool would want, and it is the
   structural ceiling once verify moves off the receiver.
3. **Handoff granularity matters:** a per-event `Channel.send` costs ~180ns
   extra per event (halves single-feeder throughput); a 64-event batch makes
   the handoff essentially free — same conclusion the server's `IngestQueue`
   already embodies.
4. **Early dedup is the biggest lever under multi-relay load:** checking the
   seen-ids set right after parse — before entering the locked dispatch path —
   let duplicate frames (75% of deliveries in the 4-relay setup; production
   showed 14–57% dups) skip the contended section entirely: 2.7–4× the
   aggregate throughput of every other 4-feeder variant. Semantic caveat: a
   short-circuited duplicate no longer bumps that sub's per-relay
   `onNewEvent`/stats counters, so paging/EOSE bookkeeping would need the
   cheap counters kept ahead of the skip.

## Bulk download: N-million events from one relay (download + parse only)

**Harness:** `quartz/src/jvmTest/.../relay/prodbench/BulkDownloadBenchmark.kt`
(gated: `./gradlew :quartz:jvmTest --tests "*.BulkDownloadBenchmark" -PprodRelayBench=1`)

### Local ceilings (geode on localhost TCP, 100k seeded events, 4 cores)

| strategy | events/s | wall for 100k |
|---|---|---|
| quartz, 1 conn, one giant REQ | 476 | 210s (and 2 events silently missing) |
| quartz, 4 conns time-sharded, giant REQs | 2,001 | 50s |
| quartz, 1 conn, paged 1000/page | 12,480 | 8.0s |
| **quartz, 4 conns time-sharded + paged** | **24,210** | **4.1s** |
| raw socket (no parse), one giant REQ | ~676 | timed out at 120s (81k/100k) |

- **Giant single REQs are a trap.** The raw (no-parse) variant proves it's
  server-side: geode streams a 100k-event response at ~700 events/s
  (per-frame cost in the session pump / query streaming path — needs its own
  investigation, `WebSocketSessionPump`), and the quartz run came back 2
  events short. Public relays are worse: they clamp limits and may drop
  frames or the connection under output backpressure. Paged cursors through
  the very same server ran 26× faster.
- **Sharding the `created_at` range across connections stacks with paging:**
  4 sharded + paged connections ≈ 2× one paged connection locally (24k/s),
  and each connection gets its own receiver coroutine, so parse parallelizes
  for free.

### Per-frame strategies, offline (50k frames, ~580B each)

| strategy | events/s | µs/frame |
|---|---|---|
| full `fromJsonToMessage`, 1 thread | 276k | 3.6 |
| full parse, 4 threads | 657k | 1.5 (aggregate) |
| id-scan only (archive raw, parse lazily) | 2.96M | 0.34 |

Parse of small events is ~3.6µs; the earlier production capture (mixed sizes,
damus 8.7KB events) measured 32µs. Either way **parse is not the bottleneck
for bulk download**: one core parses 10M small events in ~36s, and a 4-core
fan-out does it in ~15s. The wire and the relay's page cadence dominate.

### Production test case: kinds=[30382] on nip85.nosfabrica.com (cap 20k)

| strategy | wall | events/s | notes |
|---|---|---|---|
| paged download (until-cursor) | 5.4s | 3,711 | 41 pages ≈ 500/page (relay clamp) |
| NIP-77 negentropy sync | 12.7s | 1,575 | full set = 31,241 ids, 9 reconcile windows |

Negentropy correctly enumerated the whole 31,241-id set (splitting 9 windows
around the relay's `max_sync_events` cap) and streamed id-batch REQs — but
for a **cold** download it was 2.4× slower than plain paging: the reconcile
rounds and by-id lookups cost more than sequential pages when you need
*everything anyway*. Negentropy's win is **incremental re-sync**: once the
10M events are local, the next sync transfers only fingerprints + the diff
instead of re-paging the world. (kind 30382 events carry empty `content` —
all data in tags — so the MB/s column reads 0.)

### Per-connection wall: relay pacing, not client parse (2026-07-03)

A user report proposed "parallel parse + strictly-ordered dispatch" on the
theory that the per-connection serial parse caps a single connection at
~4k events/s (fetch-by-id, 12 idle cores, no gain past ~4 concurrent REQs,
throughput scaling only with connections). The discriminating test
(`BulkDownloadBenchmark.perConnectionWall`) pages the same production query
on one connection twice — once through a raw socket that does NO JSON parse
(only an `indexOf` scan for `created_at`), once through the full quartz
stack:

| | events/s (nip85.nosfabrica.com, kinds 30382, 20k cap) |
|---|---|
| raw socket, no parse | 3,382 |
| full quartz stack, parse + dispatch | 3,754 |

Identical within noise. **The ~3.5–4k/s single-connection ceiling is the
relay's per-connection page/response cadence, not client CPU** — at 465B
events the parser is ~1% busy at this rate (single-thread parse ceiling:
276k/s small events, 31k/s for the large-frame production mix). The same
relay served our negentropy fetch-by-id at only 1.6k/s with 8 concurrent
REQs and an idle client — also server-side. This matches every symptom in
the report: per-connection pacing explains "more REQs on one connection
don't help" and "throughput scales with connections" just as well as a
client-side serial stage would, and "CPU idle" is what a network wall looks
like.

Implications for the proposed parallel-parse pipeline:

- It will NOT lift the reported 4k/s ceiling on relays with this pacing; the
  proven lever is **more connections** (created_at- or id-range sharding).
- It IS still worth having for the cases where the consumer genuinely
  saturates: large-frame relays (32µs/frame mix) on phones (5–10× slower
  cores → ~3–6k/s parse ceiling) being streamed by fast relays, and any
  setup where verify/consume stays inline on the consumer.
- The proposal's ordering constraints are correct and non-negotiable:
  NEG-MSG/NEG-ERR handshake order, EOSE-only-after-its-EVENTs,
  per-subscription EVENT order, OK/CLOSED/NOTICE/AUTH stream order. A
  sequence-numbered reorder buffer after a parallel parse pool satisfies all
  four (dispatch order = receive order).
- The bounded-pipeline half of the proposal is worth doing regardless — the
  UNLIMITED per-connection channel is the unbounded-memory risk under any
  slow consumer (see below), and bounding it converts overload into TCP
  backpressure. One caveat: blocking OkHttp's reader thread also delays its
  PING/PONG handling, so sustained backpressure can trip relay-side ping
  timeouts — the bound should be sized generously.

### Parallel fetch-by-id matrix — full corpus, no cap (2026-07-03)

**Harness:** `quartz/src/jvmTest/.../relay/prodbench/ByIdFetchBenchmark.kt`.
Enumerates every kind-30382 id on nip85.nosfabrica.com (33,000 that day), then
each cell re-downloads the FULL corpus by 250-id REQ batches from a shared
queue. Connections are pre-established before timing; every cell completed
with zero missing ids and zero timed-out batches.

| cell | events/s |
|---|---|
| 1 conn × 1 REQ | 1,970 |
| 1 conn × 2 REQs | 3,423 |
| 1 conn × 4 REQs | 8,026 |
| 1 conn × 8 REQs | 17,214 |
| 1 conn × 16 REQs | **30,735** |
| 2 conns × 4 REQs | 15,557 |
| 4 conns × 4 REQs | 27,251 |
| 8 conns × 4 REQs | **40,402** |

This corrects both the user report AND this doc's own earlier "relay
per-connection pacing" phrasing:

1. **No wall at ~4 concurrent REQs** — one connection scales near-linearly to
   16 in-flight REQs (1,970 → 30,735 events/s). The relay is happy to serve
   30k+/s down a single socket when the client keeps requests in flight.
2. **Connections and per-connection REQs are interchangeable**: 1×16 ≈ 4×4.
   The real variable is **total in-flight REQs** (Little's law: throughput ≈
   in-flight ÷ per-REQ latency; a 250-id batch serves in ~130ms here).
   Diminishing returns start around 32 in-flight (~40k/s) — likely relay
   query capacity.
3. **Every slow configuration measured so far is a serialization problem,
   not a bandwidth/CPU one**: until-cursor paging is 1-in-flight by
   construction (~2–3.7k/s ≈ the 1×1 cell); `negentropySync` measured
   1.6k/s with `maxConcurrentReqs = 8` because its download workers starve —
   the reconcile rounds pace id production (bounded `idBatches` buffer of
   `workerCount` batches) and overflow windows recurse **sequentially**
   (`syncWindow` lo-half then hi-half).
4. The reported "stops helping past ~4 REQs" is consistent with
   `negentropySync`'s internal throttling (or a relay-side subscription cap,
   or a slow per-event sink) — not with independent by-id REQs on this relay.
5. Client parse at 30.7k/s through ONE connection's consumer coroutine was
   ~11% of a core (3.6µs × 30.7k) on this machine — still not the wall. On a
   phone (5–10× slower) 15–30k/s IS where the single consumer saturates, so
   the parallel-parse proposal becomes relevant on mobile at exactly the
   rates this fan-out unlocks.

**Concrete quartz improvements this implies** (in `negentropySync`):
raise `maxConcurrentReqs` (16 measured safe here), deepen the `idBatches`
buffer so downloads don't starve on reconcile cadence, and reconcile
overflow-split windows concurrently instead of recursing sequentially.
Together these should move it from 1.6k/s toward the ~30k/s the same
connection demonstrably sustains. At 40k/s, 10M events by id is ~4 minutes
(plus id enumeration).

### Extended matrix on a 2.6M corpus — saturation found (2026-07-03, later)

Between the two runs the relay was **backfilled from 33k to 2,628,328
kind-30382 events** (it's an actively-loading NIP-85 dataset — cross-run
comparisons must account for corpus size). Rerun with 125-id batches, cells
capped at a 120s budget (all cells partial by design at this corpus size):

| cell | events/s | effective per-REQ latency |
|---|---|---|
| 1 conn × 1 REQ | 1,088 | ~115ms |
| 1 conn × 4 REQs | 3,953 | ~126ms |
| 1 conn × 8 REQs | 5,420 | ~184ms |
| 1 conn × 16 REQs | 5,480 | ~365ms |
| 1 conn × 20 REQs (relay cap) | 5,464 | ~458ms |
| 2 conns × 10 REQs | 10,508 | ~285ms |
| 4 conns × 10 REQs | **14,757** | ~542ms |
| 8 conns × 10 REQs | 14,617 | ~1.1s |
| 16 conns × 10 REQs | 14,921 | ~2.2s |

Also learned the hard way (first extended run wedged): the relay's NIP-11
advertises `limitation.max_subscriptions = 20` — REQs 21+ on one connection
aren't just rejected, they wedged the connection, and with the client's
5-minute reconnect backoff every subsequent batch burned its full timeout.
**A bulk downloader must read NIP-11 and stay under the caps.**

What changed vs the 33k-corpus run:

1. **On the big corpus there IS a per-connection wall — at ~8 in-flight
   REQs / ~5.5k events/s**, where the small corpus scaled linearly to 16 /
   30.7k/s. Past 8, added in-flight only inflates per-REQ latency (queueing,
   textbook Little's law). The user report's "stops helping past ~4" is
   therefore *scale-dependent*: wrong on a hot 33k dataset, roughly right
   (off by 2×) on a cold 2.6M one — by-id lookups on the larger index cost
   more and strfry's per-connection service saturates.
2. **The relay-wide ceiling is ~15k events/s at ~40 total in-flight** (vs
   ~40k/s on the small corpus). 4 conns × 10 REQs already reaches it; 16
   conns adds nothing but latency. More connections cannot beat the server's
   aggregate query capacity.
3. Client-side cost remains negligible at these rates — every wall in this
   table is server-side.

Practical shape for a bulk-by-id downloader, from the data: read NIP-11 →
open ~4 connections × ~10 REQs — and make concurrency **adaptive** (grow
in-flight until events/s stops improving / per-REQ latency inflates, then
back off), since the same relay's sweet spot moved 8× in one day as its
dataset grew. At today's ~15k/s ceiling: 2.6M events ≈ 3 min; 10M ≈ 11 min.

### negentropySync pipelining (implemented, 2026-07-03)

`NostrClientNegentropySyncExt` was restructured around the findings — no
NIP-11 auto-detection, the budget knobs are the caller's:

- **Global download-worker pool** spanning the whole sync (was per-window
  with a join between windows, so the connection idled while the next
  window's NEG rounds ran).
- **`reconcileConcurrency` parameter** — overflow-split windows are
  reconciled by N concurrent NEG sessions from a shared work queue (was a
  strictly sequential lo-half/hi-half recursion). Default 1.
- **`idBufferBatches` parameter** — depth of the bounded buffer between
  reconciliation and download workers (was hardcoded to `workerCount`).
- KDoc contract: peak subscriptions = `maxConcurrentReqs +
  reconcileConcurrency + 1`; sizing it under the relay's
  `limitation.max_subscriptions` is the caller's responsibility.

All 44 negentropy tests (unit + end-to-end against the in-process relay +
geode interop) pass unchanged. Production shootout on the 2.6M corpus,
100k-event cap, same day, single connection
(`BulkDownloadBenchmark.negentropyPipelineShootout`):

| variant | events/s | note |
|---|---|---|
| old implementation (previous day, 31k corpus) | 1,575 | not same-day comparable |
| new pipeline, old-equivalent params (8 reqs, seq windows) | 3,601 | global pool already overlaps windows |
| new pipeline, tuned (12 reqs, 4 reconcilers, 96-batch buffer) | **4,497** | 17 subs ≤ strfry's 20 cap |

Tuned reaches **~82% of the measured ~5.5k/s single-connection by-id
ceiling**; the 4 reconcilers also produced ids 2.3× faster (need=171,812
enumerated vs 109,336 in less wall time). The remaining gap is reconcile
burstiness plus by-id query cost. Going past ~5.5k/s on this relay requires
multiple connections (caller-level: disjoint `created_at` windows per
client), which the matrix caps at ~15k/s relay-wide.

### negentropyReconcile: the composable half (implemented, 2026-07-03)

To let callers own the loading strategy (fan the download across several
connections/clients, prioritize, or push local events up), the reconcile is
now available standalone in `NostrClientNegentropySyncExt`:

- **`negentropyReconcile(relay, filter, localEntries, …)`** — pure NIP-77
  diff, no downloads/uploads. Streams both directions in `batchSize` chunks
  as rounds arrive: `onNeedIds` (relay has, local set lacks → download) and
  `onHaveIds` (local has, relay lacks → publish). Back-pressured: the
  callbacks suspend the reconcile round that produced them. Local state goes
  in as `List<IdAndTime>` (the same 40 B/entry projection
  `IEventStore.snapshotIdsForNegentropy` returns), and window splits slice
  it by `createdAt` via binary search, so both sides always reconcile the
  same timeline slice. Overflow windows split exactly like `negentropySync`,
  with the same `reconcileConcurrency` knob.
- **`negentropyReconcileIds(…): NegentropyIdDiff`** — convenience that
  materializes `needIds` + `haveIds` lists (KDoc warns about heap on
  multi-million diffs; the streaming form is the scalable one).
- `NegentropySession`'s primary constructor now takes `List<IdAndTime>`
  (JVM erasure forbids overloading on `List<Event>`); the old event-list
  form moved to `NegentropySession.fromEvents(…)`, mirroring
  `NegentropyServerSession` — call sites in `cli`, `geode` tests and
  `NegentropyManager` migrated. **API note for external quartz users:**
  `NegentropySession(subId, filter, events)` must become
  `NegentropySession.fromEvents(subId, filter, events)`.

`negentropySync` itself now delegates to the same window engine. Covered by
`NostrClientNegentropyReconcileTest` (empty-local, partial overlap both
ways, identical sets, batch streaming, since/until window slicing) — 49
negentropy tests green.

**`amy sync` migrated onto it (2026-07-03):** the CLI's hand-rolled raw
WebSocket negotiate loop (single un-windowed session; a strfry
`max_sync_events` overflow was a hard error) is gone. `SyncCommand` now
calls `negentropyReconcile` on amy's own `NostrClient` and pipelines the
loop-closing: need-id batches feed 4 concurrent by-id `drain`s and have-ids
feed an uploader, both overlapping the remaining reconcile rounds (peak 7
subs on the relay, under the common cap of 20). Downloads still funnel
through amy's verify-and-store path. Output field `rounds` (protocol
round-trips) became `windows` (created_at splits). Verified end-to-end
against two embedded geode relays: down-only 25/25, up-only 5/5, and
bidirectional re-runs converge to a zero diff.

### Answer for "10M events from one relay, fastest"

At nosfabrica's measured page cadence, one connection ≈ 3.7k events/s → 10M
in ~45 min. To improve, in order:

1. **Page with until-cursors — never one giant REQ** (silent drops, server
   slow paths, relay clamps).
2. **Shard the `created_at` range across K connections** to the same relay
   (each shard pages independently; no cursor dependency between shards).
   Local: 2× at K=4; WAN, where RTT dominates page turnaround, closer to
   linear until the relay rate-limits per-IP.
3. **Don't optimize parse first** — it's 2–5% of the budget. If the goal is
   archival, id-scan + store raw frames (0.34µs/frame) and parse lazily
   in parallel later.
4. **Use negentropy for the second sync onward**, not the first.
5. **Memory:** the per-connection channel is UNLIMITED — at 10M events a
   sink slower than the socket accumulates heap without bound. A bulk
   downloader should bound the channel (blocking the OkHttp reader thread is
   fine — that's TCP backpressure doing its job) and stream events to disk,
   never hold the set.

## Implemented optimizations (2026-07-03)

### PoolRequests lock sharding (done)

The global spin lock moved into `RequestSubscriptionState` — one lock per
subscription — since every compound mutation is scoped to one subId and
different subs share no wire state. `decideCommandLocked` now takes the state
instance so it never re-enters the (non-reentrant) lock, and the all-subs
iterations (connect/disconnect/cannot-connect) lock one sub at a time.
`withLock` is inline to keep the per-EVENT hot path allocation-free.

Validation (DispatchStageBenchmark, PoolRequests-only, 4 feeder threads vs
1): scaling flipped from **negative** (11.1M → 3.6M deliveries/s, 0.33×) to
**positive** (3.4–7.3× across runs; absolute numbers on the shared CI box are
noisy, the scaling direction is consistent across every run). All
PoolRequests concurrency + NostrClient + negentropy suites pass.

### CachingEventDecoder: duplicates skip the re-parse (done)

`BasicRelayClient`'s per-frame decode step is now a pluggable
`MessageDecoder` (default: full parse, unchanged). The opt-in
`CachingEventDecoder` scans an EVENT frame for its id (~0.3µs; the scan is
JSON-escape-safe, so reposts embedding event JSON in `content` can't confuse
it, and ANY irregularity falls back to a full parse) and, on a hit in its
generational id→Event cache, synthesizes the `EventMessage` from the
already-parsed `Event` with the frame's own subId. Dispatch semantics are
IDENTICAL — every subscription still gets its delivery, per-relay
bookkeeping still happens — only the redundant parse is skipped. Wire it via
`NostrClient(builder, decoder = CachingEventDecoder())`.

Validation (`DedupDecodeBenchmark`, 60k frames, 67% duplicates — the
production-measured dup share is 14–57%): full parse 10.0µs/frame vs caching
decoder 1.5µs/frame — **6.5×**, with duplicates costing ~0.3µs instead of
10µs. 7 scan-safety unit tests (`CachingEventDecoderTest`) cover repost
embedding, escaped subIds, malformed ids, cache rotation, non-EVENT frames.

Follow-up (2026-07-03): the decoder's spin lock was replaced with lock-free
concurrent maps — a new `ConcurrentHashCache` expect/actual
(`ConcurrentHashMap` on JVM/Android, `CacheMap` on Apple, copy-on-write on
Linux, mirroring `LargeCache`'s per-platform choices) plus atomic counters.
Rotation races are deliberately tolerated (documented: every failure mode is
a redundant re-parse, never a wrong message).
`CachingEventDecoderConcurrencyTest` hammers one shared decoder from 8
threads (80k duplicate-heavy frames, capacity 256 so rotations fire
constantly) and asserts zero wrong messages with exact counter accounting;
the benchmark assertion still holds post-refactor.

Same-JVM A/B against the resurrected spin-lock implementation (3 rounds,
interleaved): **1 thread — parity** (0.79–1.02×, noise around 1.0; the
uncontended lock was already ~free), **8 threads sharing one decoder —
lock-free wins 3.0–4.5×** (e.g. 127ms → 28ms for 128k frames). The lock was
serializing the concurrent hit path exactly the way the old global
PoolRequests lock did; ConcurrentHashMap removes it.

**Adopted in all four front-end clients (2026-07-03):** the Android app's
pool (`AppModules.kt`), the Android crawl client
(`AccountViewModel.buildCrawlClient` — Event Sync / Cashu discovery, the
duplicate-heaviest path), the desktop `RelayConnectionManager`, and amy's
`Context`. Each passes `CachingEventDecoder()` at `NostrClient`
construction; no other behavior change.

### Bounded per-connection receive buffer (REVERTED — deliberate design choice)

A 4096-frame bound on the reader→consumer channel was tried (backpressure
via TCP flow control instead of unbounded heap; proven lossless by a
slow-consumer test) and then **reverted by explicit maintainer decision**,
for two reasons that outweigh the heap tail-risk:

1. **The remote infrastructure isn't ours.** Backpressure parks the backlog
   in the RELAY's outbound buffers/TCP window — a client should release the
   relay from its duties as fast as the relay can send, and own the
   buffering itself.
2. **The app holds 2000+ simultaneous relay connections.** A bounded buffer
   under a slow consumer blocks OkHttp reader THREADS; at that connection
   count, blocked readers are a thread-starvation hazard far worse than the
   heap growth they prevent.

The `Channel.UNLIMITED` receive queues now carry an explicit "do not bound"
comment with this rationale. The slow-consumer risk is addressed from the
other side instead: make the consumer fast enough that backlogs don't form
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).

### ParallelEventVerifier: batched verify off the receiver coroutine (done)

The client-side mirror of `IngestQueue.parallelVerify`:
`accessories/ParallelEventVerifier`. `submit(event, context)` is a cheap
bounded-channel send from the receiver coroutine; a drain loop batches
greedily (up to 256) and fans each batch across `Dispatchers.Default` in
core-sized CHUNKS — a per-event `async` measured ~40µs/event of scheduling
overhead that swallowed the entire parallel gain, and the per-batch
fork/join barrier at batch 64 still cost half of it (64 → 1.2×, 256 → 2.2×,
1024 → 3.2× on 4 cores). Callbacks dispatch in submission order, a
`preVerified` hook short-circuits already-trusted ids, and the bounded
submit channel backpressures the socket.

Also learned while measuring: `Event` caches derived state after its first
verification, so any verify benchmark must use disjoint event sets per pass
(the first version accidentally measured warm objects and reported 0.94–1.2×).

Validation: `ParallelVerifyBenchmark` (4k fresh signed events, 8 signers) —
sequential 50.6µs/event vs pipeline 28.1µs/event, **1.8×** on the noisy
4-core CI box with an in-benchmark ≥1.5× assertion (crypto-only parallel
ceiling measured 2.89× there). 4 correctness tests: valid/tampered routing,
submission-order preservation, preVerified short-circuit, callback-crash
resilience.

**App-side scope correction (2026-07-03): do NOT wire this into
`CacheClientConnector`.** Code review confirmed the mobile app's
verification is ALREADY parallel across relays: each connection's
`OkHttpWebSocket` owns its own consumer coroutine on `Dispatchers.IO`, and
nothing downstream funnels — `NostrClient.onIncomingMessage` →
`EventCollector` → `LocalCache.justConsume` → `justVerify` all run inline
on the delivering connection's coroutine, `LargeCache` is a
`ConcurrentSkipListMap`, and the one global serializer (`PoolRequests`'
lock) is now per-subscription. With 2000+ connections a multi-relay burst
already saturates the cores; the verifier would only reshuffle the work.
Its real scope is where a SINGLE connection's stream serializes verifies:
bulk flows (`fetchAllPages`-style downloads, each client's leg inside
`negentropySyncFanOut`, amy/CLI syncs). One second-order note for a future
on-device profile: today's verifies run on `Dispatchers.IO` (64 threads on
~8 phone cores), so a large burst can oversubscribe cores with CPU-bound
crypto and starve the IO pool — if profiling ever shows that, moving verify
to a core-bounded dispatcher (which this accessory does) is the fix.

### negentropySyncFanOut: multi-connection sync (done)

`accessories/NostrClientNegentropyFanOutExt`: one reconcile feeding by-id
download batches to N clients (one socket each) × `reqsPerClient` workers,
with reconcile windows ALSO round-robined across the connections. Events
funnel through a bounded channel to a single consumer (exact `maxEvents`,
single-threaded `onEvent`); everything backpressures; `localEntries` diffing
and have-reporting work as in `negentropyReconcile`. 4 in-process
multi-client tests (full download, cap, local-diff, single-client
degradation) — 18 negentropy tests green.

Production shootout (100k cap, kind 30382, same-run pairs): fan-out 4×8 vs
tuned single client measured **+18% to +64%** (7.3k vs 6.1k/s; 7.5k vs
4.5k/s) with heavy relay-side variance — the relay is being actively
backfilled and its NEG snapshot caching favors whichever variant runs
second. The honest bound: **this relay produces reconcile ids at only
~9–10k/s regardless of connection count** (server-side snapshot build), so
end-to-end fan-out gains cap there even though the download stage alone
scales 2.7× with connections (the by-id matrix). On download-bound syncs
(bigger events, relays with faster NEG production, phone-CPU clients) the
gain approaches the matrix ratio.

**Descoped with evidence — adaptive in-flight control (#6):** an AIMD window
on download REQs would idle-down against today's reconcile-bound relay and
the benchmarks could not demonstrate a win (the criterion for shipping);
revisit if a relay shows download-bound behavior with volatile capacity.

### permessage-deflate: verified active (no change needed)

The per-connection-wall test now prints the negotiated
`Sec-WebSocket-Extensions`; against nip85.nosfabrica.com (strfry) OkHttp
negotiates `permessage-deflate; client_no_context_takeover` out of the box.
Compression was on the suspect list as a 3–5× bandwidth lever for mobile —
it's already engaged; nothing to code.

### Geode giant-REQ crawl + wedge: fixed (two masked bugs)

The "giant single REQ streams at ~700 events/s and came back 2 events
short" finding decomposed into two server bugs hiding each other:

1. **O(n²) replay dedup** (`LiveEventStore.query`): the historical-replay
   dedup set was an immutable Kotlin `Set` under an `AtomicReference` with
   copy-on-add — `set + id` copies the whole set per streamed event (100k
   events ≈ 5 billion hash inserts). Fixed with a spin-lock-guarded mutable
   `HashSet` (same read/write threads as before; lock sections are a single
   contains/add).
2. **Session-pump slow-client policy misfire** (`WebSocketSessionPump`):
   with the throttle gone, a fast replay instantly overflowed the
   8192-frame backlog cap — which conflated "client is slow" with "replay
   outruns the socket writer" (normal for bulk) — and the "drop" only
   closed the internal queue, leaving the socket half-dead: the client hung
   with no EOSE/close and silently missed the response tail (the likely
   story behind the original 99,998/100,000). Now the producer is PACED
   against a full backlog (bounded blocking wait for the writer, consistent
   with the documented ingest-fanout behavior) and only a client still
   behind after 30s is dropped — by actually cancelling the socket.

Validation: `GiantReqStreamTest` (committed regression guard) — 20k-event
single REQ over a real socket: pre-fix 8.4s (~2.4k events/s; 100k measured
476–700/s), post-fix **0.7s (~27k events/s)**, all 20k delivered with EOSE.
96 geode tests + quartz relay/server suites green.

## Recommendations (in order of value/risk)

1. **Move Schnorr verification off the receiver coroutine** in the app's
   consume path (`CacheClientConnector` → `LocalCache.justConsume`): a bounded
   verify stage on `Dispatchers.Default` (batch + `async` per batch, or a small
   worker pool), mirroring `IngestQueue`. Ordering constraint: dedup must stay
   before verify (it already is), and consumers of `justConsume`'s return value
   need an async-tolerant path.
2. **Keep parse on the receiver coroutine** (32µs/msg is cheap and keeps
   message ordering per subscription), but consider skipping re-serialization
   work for duplicates (57% of metadata-burst traffic never needs more than an
   id lookup).
3. **Fix the `PoolRequests` spin lock's negative scaling** — the dispatch
   microbenchmark shows 4 concurrent relay consumers deliver *less* than one
   thread through it. Either replace the busy-wait with `synchronized`/a
   parking lock, or shard the state by subId, and consider the early-dedup
   short-circuit so duplicate frames never enter the locked path at all.
4. Re-run this harness on-device (the same class compiles for Android
   instrumentation with minor changes) before/after any fix; the offline
   ceilings section gives the numbers to compare.