amethyst/quartz/plans/2026-07-10-graperank-connect-storm.md

GrapeRank crawl: connect once, wait once (connection storm)

Problem

A from-scratch amy graperank crawl spends most of its wall clock waiting to connect to relays, not downloading. Three compounding causes, all on our side:

1. Sockets are torn down between drains. NostrClient reconciles the relay pool against the union of active subscriptions (RelayPool.updatePool, sampled every 300ms). A Phase-B drain unit unsubscribes when it finishes, so an outbox relay with no other in-flight drain is disconnected ~300ms later — and the next round re-pays DNS + TCP + TLS + WS-upgrade for the same relay. Only the top-20 "warm pool" relays kept a do-nothing subscription open. A crawl touches ~10–15k distinct relay URLs; most get dialed many times.
2. Handshake concurrency was capped at 256 (OkHttp Dispatcher.maxRequests; the OkHttp default is 64). Every WS upgrade is an async call holding a dispatcher slot for its whole DNS + TCP + TLS + upgrade; a dead relay holds a slot for the full 7s connectTimeout. At 256 slots, dialing 5k relays where half are dead is ~20 serialized waves — minutes of pure waiting that could be one parallel wait.
3. DNS is blocking, uncached and duplicated. OkHttp's Dns.SYSTEM calls InetAddress.getAllByName (glibc getaddrinfo) on the dispatcher thread. The JVM's own cache is ~30s positive / ~10s negative — useless across a 30-minute crawl that re-dials the same hosts every round. Worse, the outbox model mints hundreds of per-user URLs on one host (filter.nostr.wine/npubA, /npubB, …) and each URL is a fresh lookup. A dead domain is the worst case: resolver timeouts (5s × 2 attempts per nameserver) can hold a thread 10–30s, and the crawl re-resolves it from every straggler's outbox, every round.

Single-server limits (what bounds "1000s of connections at once")

Measured/derived for a JVM CLI on one Linux box:

Resource	Limit	Consequence
File descriptors	ulimit -n soft (often 1024–4096; systemd default 1024 soft / 512k+ hard)	1 FD per socket. The JVM cannot raise its own rlimit — detect via UnixOperatingSystemMXBean and size the storm to fit; advise ulimit -n 16384 for big crawls.
Ephemeral ports	~28k (ip_local_port_range) per destination tuple	Non-issue: connections go to thousands of distinct relays, each gets its own tuple space.
Concurrent TCP connects	Kernel: effectively unbounded (a pending SYN is just a socket in SYN_SENT)	Dead-IP dials cost nothing in CPU but hold their FD + dispatcher slot for the connect timeout. Parallelism is the fix; refused/reset fails in 1 RTT.
DNS	glibc getaddrinfo is blocking, ~1 thread per in-flight lookup; dead domains 10–30s	Cache positives AND negatives in-process; dedupe concurrent lookups for the same host (path-URL explosion).
Threads	1 platform thread per in-flight handshake (OkHttp)	1–2k concurrent handshakes ≈ 1–2k transient threads — fine on a 64-bit JVM (~stack is virtual), but don't go to 10k.
TLS	CPU per handshake (~1ms)	Negligible next to network RTTs; OkHttp shares an SSL session cache per client.
Middleboxes	Home-router NAT/conntrack tables (~4–16k entries)	Operational caveat for residential operators; server/VPS operators unaffected (nf_conntrack_max default ~256k).

Conclusion: with FDs raised and DNS cached, a single server comfortably opens a few thousand concurrent connections; the binding constraints are the FD soft limit and the dispatcher cap, both of which we now size/raise explicitly.

Design

Three changes, all keeping completeness identical (same relays asked the same filters — only when sockets open changes):

1. Crawl-wide warm pool = mass pre-connect. The existing warm-pool trick (a never-matching ids filter that EOSEs instantly and just holds the socket) is extended from the top-20 relays to every candidate relay we know — seeded at crawl start from the NIP-66 reachability cache's live set, and refreshed every round with the relays learned from kind:10002s — capped by preconnectCap (FD-budget-aware, busiest relays first, dead relays excluded). Connections open in one parallel storm at start (and in the background as new relays are learned), stay up for the whole crawl, and every drain hits an already-open socket. This also implements the "crawl once with --max-hops to save the relay list" flow: the first (even shallow) crawl populates the store with 10002s and the reachability cache with live/dead verdicts; the next crawl pre-connects that whole universe and waits once.
2. Transport ceilings raised (CLI). Dispatcher.maxRequests 256 → 1024 (per-host stays 16 — politeness to path-multiplexed hosts is a server property), and the app's SurgeDns (moved to quartz; stale-while-revalidate, single-flight, 10-min negative TTL, persisted under ~/.amy/shared/) replaces Dns.SYSTEM. FD budget is detected at startup and sizes both the dispatcher and the default preconnectCap. Caveat: in a proxied environment (HTTP CONNECT), OkHttp sends the hostname to the proxy and never consults the client-side resolver — so the DNS layer only pays off on direct-connect deployments, and none of the A/B numbers below depend on it.
3. amy graperank probe — the relay census. Reads the full known relay universe (every relay in stored kind:10002s + everything in the reachability cache), mass-connects it in waves with a never-matching REQ, and records per-relay verdicts with real rtt-open into the NIP-66 store: connected → live (with measured RTT, however slow), cannot-connect / never-completed → dead (TTL'd, re-probed after expiry). Separates "working but slow" (kept, given the crawler's patient park path) from "not working" (skipped entirely) without burning crawl time on the distinction.

Measured A/B results (cold hop-3 crawl, fresh store per leg, same observer)

Sequential single-variable legs on the same 4-core / 4096-FD box. Completeness gate: contact_lists_fed and users_discovered must not drop (±2% network noise between runs is normal — relays come and go).

leg	wall	lists fed	users found	verdict
baseline (all connect-storm work, 24 workers)	638s	18,519	165k	reference
+ background parks don't gate convergence	579s	18,829	167k	keep
+ --drain-concurrency 48	559s	18,817	169k	keep (confirmed below)
+ --timeout 5 (shorter fast window)	735s	18,524	164k	reject — more parks/retries
+ sharded-sweep dedup	675s*	18,801	168k	keep — round wall ↓ (301s vs 336s); total noise
+ overlap tail (no convergence park-wait, agg ∥ deletions)	474s	18,816	167k	keep
+ --drain-concurrency 48 on top	396s	18,816	167k	keep → new default
+ per-relay sub cap 16→32 (AMY_RELAY_SUB_CAP)	602s	18,478	164k	reject — 27 relays demoted us + 7 rate-limited (vs 5+1 at 16); the Phase-B plateau is relay-side service rate, not client permits

Reproducibility + the recovery fixpoint

A cold hop-11 rerun matched hops 1–4 within ~3% but lost the entire hop-6/7 cluster (394k vs 621k discovered at −7.5% lists): one weak backbone hour at round 5 shifted ~30k lists from in-round arrival to the terminal aggregator pass, whose discoveries were never crawled. Fixed by iterating rounds + recovery to a fixpoint (recovery reveals in-budget pending → resume rounds → recover again). Validation, cold hop-8 on the fixpoint build: 796,459 discovered / 253,213 lists — the deepest crawl of the series (hop 5–8 populations 107k/167k/131k/72k vs 67k/110k/127k/2k before); the fixpoint fired live (+25 recovered lists → 1 pending → extra round → +0 → converge). Known follow-ups: each fixpoint iteration re-asks aggregators for the FULL straggler set (~4 min per pass at 470k stragglers — dedupe to newly-added stragglers), and the retry rounds' backbone re-fan remains the dominant tail cost (rounds 8–10 ≈ 80 min for +6.6k lists).

Final validation — cold hop-5, fresh store, winning stack (the workload that never completed pre-fixes): 85.4 min, 203,903 contact lists, 391,549 users discovered, 626,599 events, 5,794 relays. Per-hop list completeness: 90% (hop 2), 71% (hop 3), 56% (hop 4). Remaining known lever for a future pass: attempt>0 retries re-fan every user to the whole backbone inside per-batch units, so the two retry rounds cost nearly as much as first passes at a fraction of the yield — route retry backbone re-asks as round-wide chunked per-relay units instead.

Retry routing: the round-wide communal sweep (Phase C)

That "remaining lever" landed as a restructure of Phase B's shared-relay coverage. Before: every 256-user batch with retry (attempt>0) or no-outbox users fanned the SAME ~40 backbone+fallback relays, queueing thousands of small REQs against their 16-permit gates — measured as rounds 8–10 of a hop-8 crawl burning ~80 min for +6.6k lists, nearly all hot-relay head-of-line blocking. After: routeByOutbox routes only a user's OWN-signal relays (write relays; harvested hints when there's no outbox) and flags communal-coverage users into a per-round set; a new Phase C (communalSweep) asks the shared relays ONCE per round for all flagged users in author-chunked units — the same (relay, user) pairs, ~8x fewer REQs — and records answered-empty pairs so each subsequent round shrinks. Attempt accounting moves to Phase C (a user burns an attempt only after both its own relays and the shared set missed that round), shardedSweep now records answered-empty pairs too, and each recovery-fixpoint iteration only asks the aggregators for stragglers ADDED since the previous pass (a repeat full-set pass cost ~4 min at 470k stragglers for zero new answers).

Validation, cold hop-8, same observer (reference = the fixpoint build's 232.6 min / 253,213 lists / 796,459 users): 160.7 min, 292,768 lists, 1,169,194 users discovered, 7,240 relays — 31% less wall clock with +15.6% lists and +47% users. The depth cliff moved out two hops: hop-7/8 populations 319,878/292,268 vs 131,051/72,389, hop-7 lists 29,445 vs 6,117. Round-level profile: the retry rounds that defined the old tail now run in minutes (round 8: 247s total, of which Phase C fed 5,695 of its 6,080 lists), and Phase C itself costs 10–170s per round, folded out of the per-batch hot path.

* total inflated by a slow round-3 network sample + the then-serial tail; the change's own effect (straggler-round Phase A 20s → 20ms) is visible directly.

Not measurable in these legs but fixed on the way (validated separately):

• the whole crawl ran on runBlocking's single-threaded event loop — thousands of Phase-B unit coroutines starved one core while three idled, inflating batch walls ~5x (crawl() now hops to Dispatchers.IO); this also invalidated the old "64 workers is 2x slower" A/B;
• per-user store point queries (~8ms each on a multi-GB store) in harvestFromStore / sweep / consumer / aggregator paths — now one chunked author query per 300 users (hop-3 replay Phase A: 176s → 36s at identical fed counts).

Non-goals

• No change to REQ concurrency per relay (AdaptiveRelayLimiter) or the drain worker count — the 64-worker A/B showed the REQ fan-out re-floods busy hubs; this work only parallelizes and amortizes connection setup.
• No async-DNS library dependency; cached blocking lookups on OkHttp's existing threads are sufficient once deduped and negative-cached.