Broadcast Relay Batching Fix

Problem

With 600+ broadcast relays, NDK's fromRelayUrls creates 630 temporary relays and adds them all to the pool simultaneously. NDKRelaySet.publish() then fires all 630 relay publish operations in parallel via Promise.all. Each relay that isn't connected calls relay.connect() which opens a WebSocket.

Browsers limit simultaneous WebSocket connections (practically ~50-100 at a time before connections queue up). With 630 simultaneous connection attempts, most relays never get a connection slot before the per-relay timeout expires. Result: only ~13/630 relays succeed.

Solution: Chunked publishing

Instead of creating one giant NDKRelaySet with all 630 relays and calling publish() once, we split the relay URLs into batches of 50 and publish to each batch sequentially. Each batch gets its own NDKRelaySet.fromRelayUrls() call, its own publish(), and its own set of live progress listeners.

flowchart TD
  A[630 relay URLs] --> B[Split into batches of 50]
  B --> C[Batch 1: 50 relays]
  C --> D[publish to batch 1]
  D --> E{Batch 1 done?}
  E -->|yes| F[Batch 2: 50 relays]
  F --> G[publish to batch 2]
  G --> H{Batch 2 done?}
  H -->|yes| I[... continue ...]
  I --> J[Batch 13: 30 relays]
  J --> K[publish to batch 13]
  K --> L{All batches done?}
  L -->|yes| M[Final done event with total count]

Key design points

Batch size: 50 relays — small enough to stay within browser WebSocket limits, large enough to finish in a reasonable time. Each batch takes ~5-10s.
Sequential batches — wait for each batch to complete before starting the next. This keeps the total simultaneous connections at ~50, not 630.
Live progress across batches — the publishedSoFar and failedSoFar sets accumulate across all batches. The broadcastProgress events stream continuously, so the footer shows the count climbing from 0 to 630 across all batches.
Per-relay timeout: 10s per batch — each relay in a batch gets 10s to connect + publish. Since only 50 are competing for connections (not 630), this is plenty of time.
Overall deadline: removed — since batches are sequential and each has its own per-relay timeout, the overall time is naturally bounded at numBatches × perBatchTimeout. No need for a separate overall deadline.
Connection timeout: 10s — set on each relay in the batch, matching the publish timeout.

Implementation (in `www/ndk-worker.js` `handlePublish`)

Replace the current single NDKRelaySet.fromRelayUrls(allUrls, ndk) + single publish() with a batched loop:

const BATCH_SIZE = 50;
const PER_RELAY_TIMEOUT_MS = 10000;

if (isBroadcast) {
    const allUrls = [...new Set([...outboxWriteUrls, ...activeBroadcastUrls])];
    const batches = [];
    for (let i = 0; i < allUrls.length; i += BATCH_SIZE) {
        batches.push(allUrls.slice(i, i + BATCH_SIZE));
    }
    console.log(`[Worker] Broadcasting to ${allUrls.length} relays in ${batches.length} batches of ${BATCH_SIZE}`);

    // Emit start event
    broadcast({ type: 'broadcastProgress', sessionId, eventId, eventKind, phase: 'start', total: allUrls.length, successful: 0, failed: 0, latestRelay: null });

    // Attach listeners once — they accumulate across all batches
    ndkEvent.on('relay:published', (relay) => { ... publishedSoFar.add ... });
    ndkEvent.on('relay:publish:failed', (relay, err) => { ... failedSoFar.add ... });

    // Publish to each batch sequentially
    for (let i = 0; i < batches.length; i++) {
        const batch = batches[i];
        const batchSet = NDKRelaySet.fromRelayUrls(batch, ndk);
        // Set connection timeout on each relay in this batch
        for (const relay of batchSet.relays) {
            try { relay.connectionTimeout = PER_RELAY_TIMEOUT_MS; } catch (_) {}
        }
        try {
            await ndkEvent.publish(batchSet, PER_RELAY_TIMEOUT_MS, 1);
        } catch (e) {
            // Expected — some relays in this batch may fail
        }
        console.log(`[Worker] Batch ${i+1}/${batches.length} done: ${publishedSoFar.size}/${allUrls.length} total succeeded`);
    }

    // Emit final done event
    broadcast({ type: 'broadcastProgress', sessionId, eventId, eventKind, phase: 'done', total: allUrls.length, successful: publishedSoFar.size, failed: failedSoFar.size + timedOutSoFar.size, relayUrls: Array.from(publishedSoFar) });
    // Clean up listeners
    ndkEvent.removeAllListeners('relay:published');
    ndkEvent.removeAllListeners('relay:publish:failed');
}

Files touched

File	Change
`www/ndk-worker.js`	Replace single publish with batched loop in `handlePublish`

Testing

Post with a user that has 600+ broadcast relays
Watch the footer: count should climb steadily as batches complete
Final count should be much higher than 13 (ideally 500+)
Each batch of 50 should take ~5-10 seconds
Total time for 630 relays: ~13 batches × ~10s = ~130 seconds

5.0 KiB Raw Permalink Blame History Unescape Escape