Files
ginxsom/plans/fix-ginxsom-cpu-spin.md

5.1 KiB

Fix: Ginxsom 100% CPU Busy-Wait Bug

Root Cause Analysis

The ginxsom-fcgi process consumes 100% CPU due to a busy-wait spin loop in the relay management thread.

The Bug Chain

  1. relay_management_thread() in src/relay_client.c:328 runs a while loop calling nostr_relay_pool_poll() with a 1000ms timeout
  2. nostr_relay_pool_poll() in nostr_core_lib/nostr_core/core_relay_pool.c:1699 iterates over all relays, but skips disconnected relays at line 1725-1728 — it only calls nostr_ws_receive() (which actually blocks) for connected relays
  3. When no relays are connected (or all are in error/disconnected state), the entire poll function returns 0 instantly with zero blocking
  4. The caller has no sleep for the events_processed == 0 case — it immediately loops back and calls poll again
  5. This creates a tight spin loop consuming 100% of a CPU core

Additional Issues Found

  • No signal handling: main.c has zero signal(), SIGTERM, or SIGINT handling. When systemd sends SIGTERM to stop the service, the process ignores it and must be SIGKILL'd after timeout (confirmed in journal: State 'stop-sigterm' timed out. Killing.)
  • relay_client_stop() never called: The cleanup function exists but is never invoked — there's no shutdown path
  • g_relay_state.running not volatile: The running flag is read by the management thread but set by the main thread (or signal handler). Without volatile, the compiler may optimize away the check
graph TD
    A["relay_management_thread()"] -->|"while running"| B["nostr_relay_pool_poll(pool, 1000)"]
    B --> C{"Any relays connected?"}
    C -->|"Yes"| D["nostr_ws_receive() blocks up to timeout_ms"]
    C -->|"No - all disconnected"| E["Returns 0 immediately"]
    D --> F{"events_processed > 0?"}
    E --> F
    F -->|"> 0"| A
    F -->|"== 0"| G["No sleep! Loops immediately"]
    F -->|"< 0"| H["sleep(1) then loop"]
    G -->|"BUSY WAIT"| A

Fixes Required

Fix 1: Add idle sleep to poll loop (src/relay_client.c)

File: src/relay_client.c, lines 327-337

Before:

while (g_relay_state.running) {
    int events_processed = nostr_relay_pool_poll(g_relay_state.pool, 1000);
    if (events_processed < 0) {
        app_log(LOG_ERROR, "Error polling relay pool");
        sleep(1);
    }
}

After:

while (g_relay_state.running) {
    int events_processed = nostr_relay_pool_poll(g_relay_state.pool, 1000);
    if (events_processed < 0) {
        app_log(LOG_ERROR, "Error polling relay pool");
        sleep(1);
    } else if (events_processed == 0) {
        // Prevent busy-wait when no relays are connected or no events pending.
        // nostr_relay_pool_poll() returns immediately if all relays are disconnected
        // because it skips the blocking nostr_ws_receive() call for non-connected relays.
        usleep(100000); // 100ms idle sleep
    }
}

Fix 2: Make running flag volatile (src/relay_client.c)

File: src/relay_client.c, line 39

Before:

static struct {
    int enabled;
    int initialized;
    int running;
    ...

After:

static struct {
    int enabled;
    int initialized;
    volatile int running;
    ...

Fix 3: Add signal handler for graceful shutdown (src/main.c)

File: src/main.c

Add near the top (after includes):

#include <signal.h>

static volatile sig_atomic_t g_shutdown_requested = 0;

static void shutdown_handler(int signum) {
    (void)signum;
    g_shutdown_requested = 1;
}

Register the handler before the FCGI loop (before line 2664):

// Register signal handlers for graceful shutdown
signal(SIGTERM, shutdown_handler);
signal(SIGINT, shutdown_handler);

Fix 4: Call relay_client_stop() on shutdown (src/main.c)

File: src/main.c

Modify the FCGI loop condition (line 2667) to also check the shutdown flag:

while (FCGI_Accept() >= 0 && !g_shutdown_requested) {

Add cleanup after the FCGI loop exits (before return 0 at line 3068):

  // Graceful shutdown
  app_log(LOG_INFO, "Shutting down ginxsom...");
  relay_client_stop();
  app_log(LOG_INFO, "Ginxsom shutdown complete");

Also add the relay_client.h include at the top of main.c.

Summary of Changes

File Change Purpose
src/relay_client.c:328-336 Add usleep(100000) when poll returns 0 Fix 100% CPU busy-wait
src/relay_client.c:39 Make running field volatile Thread-safe flag checking
src/main.c (top) Add signal handler function Respond to SIGTERM/SIGINT
src/main.c (before FCGI loop) Register signal handlers Enable graceful shutdown
src/main.c:2667 Add && !g_shutdown_requested to loop Exit FCGI loop on signal
src/main.c (after FCGI loop) Call relay_client_stop() Clean up relay thread

Testing

  1. Build locally: make clean && make
  2. Run locally and verify CPU usage stays near 0% when idle
  3. Send SIGTERM and verify clean shutdown (no SIGKILL needed)
  4. Deploy to remote server via deploy_lt.sh
  5. Monitor with top to confirm CPU usage is normal